The Peter Principle Is Necessary

Author’s note: not sure why I decided to let this languish in drafts. It seems both basically true and basically complete. Perhaps I had more thoughts, now forgotten, that I hadn’t managed to fit in? Maybe it felt too obvious?

The Peter principle states that employees are promoted to the level of their incompetence. I suspect that this is actually a necessary consequence of the best way to run an organization, given some starting assumptions that are not particularly out there.

Givens

  1. Your organization is arranged such that some positions (classically managers) require fewer employees than some other positions (classically ICs),
  2. The positions with fewer people in them are rewarded more highly than the ones with more in them, in terms of money, or money-proxies such as vacation days or travel budget, or even just in terms of respect and esteem afforded to them,
  3. Employee promotions occur with some regularity,
  4. Maximizing employee productivity is an important goal, and
  5. Employees can be assumed to behave at least somewhat rationally with respect to their economic needs.

The Argument

Before I begin, I’m going to make some simplifying assumptions that aren’t required for the argument but save me some caveats. First, let’s assume that, for any position that is open, there is at least one employee eligible for it for whom the added marginal utility of the higher salary or other perks outweighs the costs of actually performing the job. Second, let’s assume that we’re only talking about employees who actually have some need for money and aren’t just working to scratch an itch.

Let’s go!

We have, by point 2, that there are certain obvious benefits of being in a higher position: more money, more respect, etc. All else equal, your employees would like more of all these things (5). Therefore, employees are incentivized to “go for” promotions (which, by point 3, must exist).

By point 4, the organization wants to maximize the output of its employees. The way to do this is to set up incentive structures that encourage said employees to work hard. The primary form of incentive employers can leverage over employees is money. The way for an employee to get more money from the employer is by getting promoted. Therefore the incentive we control is eligibility for promotion, and what we want is for our employees to do good work.

The obvious solution here is to promote the candidate who has done the best work at the time the promotion is going to occur. Any other solution will fail to incentivize workers the way we want.

And there we have it: whoever’s best at job level N gets promoted to job level N+1 in order to continue encouraging the efforts of the rest of the level-N employees. We never make any effort to evaluate a person’s (possibly latent) skill at job N+1, so we are selecting effectively randomly amongst the pool of candidates for job N+1 (colloquially known as managers).

Is This Universal?

No. There are many ways to break the givens above, although they may have their own problems. For example, assumption 1 can be broken with a sufficiently-flat organization. Assumption 2 can break under a commission or performance-bonus model — I would be surprised if there were many organizations where the worst-paid sales manager made more money than their best-performing salesperson with any regularity. Assumption 3 can be broken in a slow-growing organization with low turnover, or at an organization that mainly hires outside people for managerial positions. Assumption 4 can broken in some industries that just need warm bodies and performance is a commodity; the best fast-food cashier and the worst are so close in terms of actual impact to the business that incentivizing for great performance is of limited utility itself. Assumption 5 can be broken if you assume your employees are human. These are merely examples.

But can we escape the conclusion consciously, in all (or most) cases? I don’t think so, at least not without significant cost. Let’s say you want to avoid the Peter principle by always promoting the person who will be best at the job they’re up for promotion to. Now your employees are incentivized to improve their signaling for the N+1 job, which costs them effort that you would rather they put into their current job. You end up with a system of highly-skilled managers, none of whom manage anyone who’s trying to get any non-managerial work done! Say you want to avoid it by paying people based on their performance, so the promotion isn’t going to come with a hefty pay bump. You end up with people turning down promotions because they can earn more at their current job, and effectively promote the employee who is worst at their current job, because that’s the only one who will accept. And so on.

Utopian Arrangements

Some people think the future of work, that might mostly solve this problem, is extremely flat hierarchies — say one person at top who gets final veto and makes no other managerial decisions, all of which are totally self-organized. This seems to me to be mostly fantasy, or else to require a fast-increasing amount of fuzzy bonding time at the expense of productivity until everyone has to spend all day keeping up their bonds of trust with everyone else and no one produces anything. A flat organizational structure works, and solves the Peter principle problem, for 5 people, but it can’t possibly for 5000.

Author’s note 2021: the previous paragraph is probably true, but without an argument for *why* it’s true it’s pretty unconvincing, I realize.

Maybe there’s a way to make a show of promoting the best performer, and then somehow quietly moving them around until they’re back at level N somewhere else if they were bad at N+1, and finding someone new to go there, but that seems likely to be found out very quickly and thus the trust of employees lost. It also sounds like a great way to get an awful lot of people in the pattern of “work hard, get promoted, flail for a while, get demoted before finding footing, quit because of morale hit of demotion,” which isn’t great.

You could try an option where you can continue doing the level N work but make more money in line with what you’d get at level N+1, but that only goes so far; a manager can legitimately increase the output of 10 or 100 people, whereas an IC cannot, so you eventually can’t pay them more without moving into management because they only control their own output. This also leads to the problem that the only people who move into management are the people who most want to move into management, which runs into the old problem of politics: the quality of wanting power does not correlate well with the quality of being responsible with it.

Concluding Thoughts

It’s not really so bad, I think. We are not exactly living in a Malthusian dystopia, so there’s some room for slack when people don’t perform well. Managers only get promoted up to their level of incompetence, which means that managers who are bad at their jobs are generally going to manage small groups, while the managers in charge of large groups may well not be great at their jobs but have grown up through managing small and medium sized groups particularly well and thus are unlikely to be truly disastrous. And besides, the higher up the hierarchy you go, the fewer positions there are, and thus the less frequently new people get promoted. Someone becomes a VP in charge of a large organization, doesn’t do very well, and gets fired or moves on; a few more tries and you end up getting lucky and randomly getting someone who actually does have the ability to succeed. Remember, it’s not that we are selecting people because they’re not going to succeed, we’re merely not selecting them on likeliness-to-succeed.

Posted in sociology | Leave a comment

Against BCE/CE

Some people prefer to use BCE/CE for dating eras instead of BC/AD. This is extremely stupid.

I was first introduced to this convention when I was a Wee Jew, say about 10 years old, by my then-Rabbi. The explanation was something like “we’re Jews, using the Christian era-naming convention is not great because we don’t want to sprinkle references to Christ all over the place because he’s not the messiah.”

As far as I know, this is the real explanation that adults justify their use of BCE/CE with, as well. How stupid is this? Let me count the ways:

  1. You’re still sprinkling references to JC all over the place, because the system is still obviously designed around the date we’ve all sort of accepted was Christ’s birth year even though it probably isn’t.
  2. Why is calling everything after Jesus the “common era” better than saying it’s “the year of [Jesus]”? it’s still saying that Jesus was central to an entire new world order, at which point we split with the past in some ineffable but fundamental way
  3. In the real world, using BCE/CE just makes people do a mental hash look-up to BC/AD, which forces them to consciously think about the fact that our dates are references to a putative messiah, which exacerbates the problem.
  4. If you actually cared about mitigating the Christian centrality of our culture, and you’re a Rabbi informing a Wee Jew not to give Jesus a dominant position in your dating system, why not use… the fucking Hebrew calendar, which goes back 5777 years already?
  5. BCE is 3 letters, unlike BC, AD, and CE, which is stupid and asymmetrical.
    1. Some day some piece of software is going to truncate it to two letters, and then you get BC, which, lol, or you get CE, which is just wrong.
  6. People apparently don’t even agree on what CE stands for? I’ve heard “Common Era” (which is stupid? why is it common? what does that mean?”) and “Current Era,” which makes somewhat more sense but leads me to wonder when we get a third era. Except nobody has any plans on creating a third era! We all still agree that post-Jesus-life is the end of history (at least as far as dating systems go).
  7. Even if you really genuinely do think that reducing the Christian-centricity of our culture is an important goal, this has to be about the least useful way of doing that imaginable. What’s the endgame? in 2 or 3 generations, everyone writes CE on their checks instead of BCE, and has stopped using that hash lookup?

Anyway. The real point here is that randomly renaming things in an ad-hoc way is stupid and unhelpful. If you genuinely think that the date system is anchored wrong, anchor it somewhere else; don’t anchor it to the same point and then stick your fingers in your ears and shout “la la la I can’t hear you it’s totally not about Jesus anymore it’s just a coincidence that it lands in the same exact place.”

Posted in language, pedantry | Leave a comment

On Skills and Learning

Authors note: Publishing this because I’m publishing drafts today. This post is entirely true and many people would benefit substantially from internalizing its ideas. It reads more like the intro to a post though – a sequence on skills and learning is something I’d like to produce at some point, and this would be a good part of that.

There are a few things about skills and learning them that I have found hold broadly true, but which do not seem to be understood, or at least not felt deeply, not internalized, by most people. I’ll try to explain them here, as I think that they’re surprisingly important and in some ways liberating. In short, they are: everything is a skill; skills can be learned; and skills are mostly the culmination of many small things.

Everything is a Skill

I mean this very nearly literally. Most people understand some version of this, but don’t understand how broadly it generalizes. There are two extremes of this that people tend to fail to understand.

First, people underestimate the degree to which small things that feel automatic are skills. Breathing, for example, does not feel like a skill – and yet there are whole schools of meditation built around improving your skill at breathing. I have personally struggled with anxiety and panic attacks, and learning how to focus on the skill of breathing and execute it even when under duress has been of extraordinary value to me. If you can only take deep breaths when you’re calm and relaxed, but fail to be able to when you’re anxious, you can improve your skill at regular breathing. Many things, things we think of as small, fall into this category. Sitting can be seen as a skill: how’s your posture? Do you even know what it should be? Walking is a skill: I know a woman who once walked incorrectly in such a way as to cause herself significant pain. And so on. The only things that are small enough to truly fall outside the skill category, in my conception, are basically basal body procedures: digestion, heart beating, etc.

Second, on the far side of the scale, people overestimate the idea of talent and the extent to which it matters in terms of large, obviously skillful things. Many people believe they could never become good painters because they lack the talent for it, for example. And talent, which is a very vague word that I’m going to avoid delving into too deeply, definitely exists in some form. Most people, no matter how hard they work, can never become LeBron James or Flash or Picasso. But the impact of talent is minimal unless you’re competing at the very top of the ladder, and believing that talent is critical to success as a beginner is an error, and usually one deployed demotivationally – the one who says “I’m just not good at math” is justifying their unwillingness to work; they are not stating a deep truth about themselves.*

Accepting that everything is a skill is both overwhelming and liberating. On the one hand, it means that getting good at new things is mostly going to be a lot of work; but on the other, it means that getting good at new things is possible.

Skills Can Be Learned

This sort of dovetails from my digression on talent. There’s not much to say about this that isn’t straightforwardly implied by the title: skills can be learned. If you want to get good at writing, or playing guitar, or running, or whatever, you can. The issue is internalizing that belief. Too many people believe they can’t learn new things – baby boomers who have spent 20 years believing themselves incapable of learning how to use computers, teenagers who believe they will never be good at math, etc, etc. Truly believing that you can in fact learn new skills is powerful, which is why the sentence gets its own section.

Skills Are Agglomerations

(I’m working on coming up with a pithier version of this title…)

The process of learning something is dominated by the acquisition of a large number of small skills. There is a common narrative that learning consists of someone getting an epiphany about the subject at hand – all at once, a true, deep understanding of something is acquired, and with it a new level of mastery: the student has seen the code behind the matrix. This model of learning is dangerously false, and not merely false but in fact almost an exact inversion of what really goes on.

Learning generally proceeds in plateaus, where one alternates between learning small things and then integrating them into a deeper understanding. The common mistake is getting the order wrong: the sub-skill acquisition precedes the integration. Telling someone about the circle of fifths teaches them nothing unless they’ve already learned their scales; telling someone who can play all their scales in their sleep teaches about the circle of fifths unites the disparate ideas into one bigger concept that they can then employ. The same can be said for basically any large, structural idea in any subject. The formalization of an integer has no use before you know a bunch of them. The formalization of a rational number requires you understand integers well, and that you have encountered the need for ratios, and so on. Writing the fundamental theorem of calculus down for someone won’t help them – that’s the integrative part of what requires a bunch of examples and motivation to get to.

* On the extreme ends of the spectrum, there are people who will truly lack the ability to learn various skills. These are well under 1% of the population in most cases. Basically everyone without a significant mental disability can learn calculus, and yet you would be hard pressed to find a room of adults without at least one person who thinks they never could – the incidence of this kind of true inability to learn something is dramatically overestimated.

Posted in learning, pedantry | Leave a comment

The Two Buckets Theory Of Gender

Author’s note: This was written years ago. I still think it’s mostly true, but I never published it because it didn’t feel like a full explication of why this is worth thinking about. I’ve since seen similar models gestured at, so this idea is by no means unique to me. Publishing it as part of a draft dump because why not?

[Epistemic status: works on my machine]

Note: for the purposes of this article, I am assuming that there are only two genders. I’m not sure I believe that in real life, and I suspect this concept could be extended to N genders, but it makes it way, way easier to talk about.

Inspired mostly, I think, by Ozy’s excellent concept of cis by default and in particular this examination of misuses of the idea, I’ve been interrogating my own internal gender identity a lot. I’ve come up with what I’m calling the “two-bucket” theory, which feels right to my own experience.

And now I’ve come up with a new mental model for this: the two buckets model. I think this resolves a bunch of issues, but I have no idea if it will resolve yours. The idea is that, rather than having a gender, you have two gender buckets that have orthogonal amounts of gender-phlogiston in them. I have also explained this has being like the speedometer on a car, or being like attribute points in an RPG.

This feels right to me! I have a very small amount of maleness, and a very small amount of femaleness. I don’t think I would feel wrong in a female body, because I don’t think I feel right in mine; I just… don’t care much? I don’t really think I’m completely agender, and I probably lean male slightly more than female, but it’s like 10-8 whereas, like, Arnold Schwarzenegger probably has 100 points of maleness (and I don’t want to speculate on how much femaleness he has).

So what are the advantages here, besides ‘it feels right to me, personally’? Well, I think it explains a few things pretty well.

Cis-By-Default

I think “cis-by-default” is defined by people who aren’t much of either gender. Like, if your maleness:femaleness ratio is close to 1 and low in absolute value, you probably don’t care much and being gendered in line with your body is not painful. Sure, I’m a cis man. If I had been born with my same self-ness* into a female body, I’d be a cis woman. I just don’t care much one way or the other.

Non-Binary

I have, I am somewhat ashamed to admit, never really understood what “non-binary” means, and, in particular, how it is distinct from being agender. Like, how is being “in-between” genders different from just being not a gender at all? There’s no way that I’m aware of to explain the existence of both non-binary-but-gendered and totally-non-gendered in a spectrum model. In the two buckets model, a non-binary person might be like 55 male and 56 female, while an agender person might be 0/0 or 1/1 or similar, extremely small values.

Trans

One of the difficulties people have with the existence of trans folks seems to me of the form “you were born in a male body, you got by just fine being male for 25 years, and now all of a sudden you’re saying it was always wrong? You were a normal man for 25 years! You can’t have just been faking it that whole time!” I think you could model this sort of trans person as being, like, 15 male/90 female. Enough male that they would get by as cis-by-default if there weren’t a burning femaleness inside them.

* I am aware that my self-ness would have grown and developed differently if I had been born female. Maybe it would make more sense if this was a freaky-friday type of thing than a counterfactual born-female Alexander thing. But I hope my point makes sense either way.

Posted in Uncategorized | 1 Comment

Assorted Thoughts on Dragon Quest XI

Note: I wrote this a few years ago and never published it because it’s super negative and probably nobody cares. But rereading it now is kind of fun so I’m putting it out there. I liked the game but god damn it was full of problems.

I just finished Dragon Quest XI, and have some thoughts about it. This post is largely negative, because mostly I noticed things that bothered me, so I will start by saying I put about a million hours into the game and actually liked it quite well. However, it has a number of really annoying flaws that seem frankly amateurish.

Text

The text is bad. It writes out slowly, playing a little sound as it prints the letters, which is annoying. If you press a button, it writes the rest of the current block fast. However, if the block was short, it moves it off screen and starts writing the next block instead, so if you want to just be able to read things at a normal pace, you end up occasionally missing lines. This is a constant low-level annoyance throughout the entire game. The text should have printed immediately and silently and to a larger area of the screen. This makes certain things, like shops and inns, much more annoying than they should be.

The Menus

The menus are horrible. Very nearly everything about very nearly every menu sucks. The worst offense is the item menus. Each character has an individual inventory. The idea, I suppose, is to add a level of strategic depth to combat – make sure your characters have the appropriate healing items, or that your healer has some stuff she can throw at enemies to do damage. In practice, you always have a healer, and the combat isn’t very hard, even if you’re not grinding at all. So there’s a lot of complexity, and a LOT of extra menus, and places to lose track of things, for very nearly zero benefit.

There are, for some reason, 11 item menus. One per character, the “item bag,” the “equipment bag,” and “important items.” Keeping equipment separate from items might not be crazy, but it’s not interesting either. Especially since some equipment can be used in battle, making it more item-like, so the distinction isn’t really clear. It also means that you have two places to look when you get an item and don’t know what it does; it could be at the back of the item bag or the equipment bag. And some items your characters will put directly into their own inventories (healing items, mostly), so good luck if you clicked through that text. Despite having, again, 11 item menus, the game somehow fails to separate crafting materials from everything else, leading to tremendous clutter.

Equipping your characters is handled separately from the item menus; there’s an explicit Equipment menu. It’s annoying that if I am in an item menu and I realize I want to move equipment around, I can’t, but this isn’t really a big deal.

This is probably at its worst when shopping. Shopping goes as follows: You approach a shopkeeper, who says something that you don’t care about. Because of the text issues above, you press a d-pad button, or perhaps X or O if you’re feeling reckless, to skip past it. They ask if you want to buy or sell; you probably want to buy. You go through their list of items, and when you select one you have to confirm that’s what you want, then select the character whose inventory it will go into, then confirm that, and then if it’s gear, the shopkeeper will ask you if you want to equip it immediately and require another confirmation and some number of further textboxes to skip. It’s so much clicking! Most of the text is meaningless, I mostly want to do the same small number of things, and I end up putting things in the wrong inventory, or clicking X one too many times and starting the process of buying another one of whatever I just bought because I was trying to get through the menu quickly. When shopping for gear, it displays that gear’s main stat (attack, defense, charm, whatever) and how the new piece of equipment will affect that stat if equipped to a character. You have to go into the more info to see its effect on other stats which might also be relevant, and cannot at this time see the stats of the equipped gear. This is not the only time info that obviously should have been cross-referenced was not.

The main in-game menu has six options: Items, Equipment, Misc., and three options I used so little I had to look up what they were (Attributes, Magic, and Party Talk). Misc has a number of useful things hiding in it, at least one of which should have been promoted to the main menu, perhaps at the expense of Magic, which is useless.

Side note: Magic is a useless menu option because DQXI has a truly wonderful feature called “handy heal-all,” which uses your magic to heal but not all the way. I don’t know its exact rules, but it won’t overspend mana to top off your last couple life points. It felt very good, it had an easy shortcut, and I used it constantly.

The misc menu has the quest catalogue, which is a very useful list of all the side quests you have accepted, their current state, and info about how to complete them. It also tells you where you can go to accept new quests, if they’re available. Promoting this to the top menu would be reasonable. Leaving it tucked away is also reasonable, but it should have been the top or second option, since you go to it a lot. It also has the Character Builder, which is how you grow your characters. That should have made it to the top menu.

The misc menu also has the info page, which has info on everything and somehow still manages to suck. It has lists of items you’ve gathered and enemies you’ve defeated. For some reason, these things are not hyperlinked. For example, an item will show you what enemies drop it; there is no way to go from the item’s info page to the enemy’s. Why not? Because it wouldn’t be right if anything about the menu was good, I guess. The game includes crafting, and crafting materials do not list what items they are used for, either.  From the starting point of wanting to craft an item, finding out what I need to do to be able to requires me to go to the forging info menu, find the item, leave that menu, go to the item menu, find the ingredient, leave that menu, go to the monster menu, and find the monster that drops it. This is way too many steps for what is obviously a common usecase.

The Maps

There are a bunch of maps, and with one exception they all have clear flaws. There is a mini-map, which is fine, although not especially valuable. There is an area map, which is quite good. Everything else is dumb.

You can only look at the area map of wherever you are right now, and anywhere you can get to by navigating to nearby areas. The map of where you are right now is better than the maps of nearby areas, because it has info overlaid on it that is not present on other maps. Namely, people who want to talk to you, and a couple kinds of locks. There are two points in the game where you get special keys that allow you to access areas (items, really) that you couldn’t before, but it is not possible to search the maps for this information.

There is also a world map, which for some reason has no information on it except a picture of the world and dots for the places you can warp to. You cannot see a map of the place you’re thinking about warping before doing the warp, so if you forget the name of the place you’re going, you get to just try things until you get it right.

The fast travel options are weird. You can travel to major locations, except the ones you can’t. Namely, there are some islands that can only be gotten to by boat, and you have to sail to them every time. More on this in the section about linearity.

Movement and Buttons

Your character has a few options for movement. Normal walking/running (you get slower in cramped areas, which is actually kind of nice for being able to navigate inside houses and whatnot), sprinting, and horse. I wanted to sprint basically every single time I was moving; walking around between interesting areas is not the fun part of a video game. This meant I had to hold down R2 constantly, which is stupid. Sprinting should be the default, or toggled, or something. The options button sets you to auto-run, which is at the normal running speed but you don’t have to hold the control stick forward. Sprinting also has that feature. It’s one of those things where they added a nice quality of life feature that has almost no use, while missing an adjacent, hugely valuable improvement.

You can ride a horse outdoors by summoning it at a bell, which are placed by campsites and outside of towns, mostly. The horse is significantly faster than sprinting, but you have to get on and off, sometimes you’ll be doing something unremarkable like crossing a bridge and the game will stop you, walk the horse backwards a step or two, and helpfully pop up a warning that you can’t go that way mounted. So riding the horse is faster than running, except when it’s randomly annoying, and you can’t really predict when that will happen. I mostly didn’t use the horse because of the surprises. When you’re riding the horse you also have to hold R2 to dash, even though the only reason you’d ever get on the horse is because you want to go fast. Galloping through monsters knocks them out of your way instead of engaging them in combat, which is actually pretty fun. Except sometimes it doesn’t and you get into combat anyway; which monsters you can knock away and which you’ll have to engage is surprising, so I ended up just avoiding them as if I were on foot when I was on the horse. Overall the horse implementation is unreliable in ways that seem crazy, and the experience is mostly frustrating and un-fun.

When engaging enemies, you want to get close to them and then hit X, in order to strike first in battle. This is a silly thing to make me do literally hundreds of times, and has its own stupid flaws that should have been obvious to the designers. X is the button you use for everything, and in particular does three main things in the overworld: pick up items from “sparkly spots” (materials and occasionally items, they regenerate after a while), engage enemies that you’re close to, and fire your crossbow at enemies you’re a little farther away from. It is very annoying to try to grab an item, not notice where the little indicator is on your screen, and accidentally shoot an enemy instead, which causes it to come fight you. It is also annoying to try to attack an enemy and accidentally shoot it instead. Nothing bad happens but it’s a dumb way to waste a few seconds. There are four buttons that are completely unused on the overworld, by the way.

The controls are set up so that you can sort-of play one handed. The movement is on the left analog stick, of course. And menus are navigated using the d-pad. L2 is an alias for X, meaning you can interact with things and select options with only your left hand. However, there’s no alias for O, so you can’t go into a menu and then get out of it. L1 is completely unused. This is another thing where they half-assed a quality of life improvement but missed a huge, obvious part of it. L3 is technically used, but all it does is bring you to first-person view, which is, as far as I can tell, entirely for aesthetic purposes. That’s fine, but it would also have been reasonable to use L3 as an alias, and perhaps move the first-person camera into a menu. Maybe I’m the weird one, though, and other people loved this feature.

We have known since, at the very latest, 1994 that selecting moves in combat should be a consistent experience, requiring the same number of confirmations regardless of number of targets. That is, moves that do not select targets (either they hit all enemies, or are randomly targeted, or are powerups to the caster or whatever) should require some token click of “yes, I mean it” to keep the same cadence as other moves. DQXI bucks this conventional wisdom, and thus you sometimes enter a move you were still considering because you forgot that it was going to happen after N clicks rather than the standard N+1. A minor mistake, to be sure, but an amateurish one.

There are several times when a cutscene ends, you regain control, and then you walk a few feet forward and into another cutscene trigger. It’s annoying every single time. Just one more bizarre quality of life failure.

Linearity

The game is relentlessly linear. Unfortunately, it is not upfront about this. You can explore off the beaten path, but you are not allowed to have any fun if you do. You get a boat not far into the game, which provides you access to maybe ten or so new places. You can collect some new crafting materials – I didn’t track if you can make use of any of them, but you can grab them anyway. You can also see a bunch of locked doors, and talk to people who will not give you new quests or anything until the story has advanced enough that they’ll be interesting. You can also sail to a couple of whirlpools, which will glow and let you know it’s not time to go to them yet.

Much later, you get access to a flying whale (seriously) and a similar thing happens: there are handful of new areas you can go to, but you’re not allowed to have any fun until you’ve found and accepted the appropriate side quests that tell you to go to those locations. You can also see locked doors, but you get the key that opens them after you get the whale.

Side note: summoning the whale can only be done from a few locations, and takes forever. It plays a (boring) cutscene every single time you summon it. If you’re on the ground and want to go to a location you can only reach by whale, you have to warp to a place you can summon the whale from, which plays a loading screen, then watch the cutscene and then watch another loading screen. You can warp directly to on-your-boat, but you can’t warp directly to on-your-whale for some reason.

One particularly egregious thing is that there are a handful of books – reading books is a minor game mechanic – that when you read them, your character opens it up, then says something like “…but he realized he’s not ready for it quite yet.” So that’s fun. You have to just remember where they were, or else just re-explore everywhere later.

You will, in general, have a better time playing the game if you never, ever, ever go anywhere the game hasn’t explicitly told you to until the end of the postgame. What you find is uniform: crafting materials you can’t take advantage of, large clearings that are obviously going to have events later, and locked doors that you have to remember manually.

Gender Politics

I’m going to keep this short, because this post is huge and not really about the world of the game so much as its mechanics. However, it bears pointing out that the game has a constant stream of minor comments about hot girls. There are a small number of townsperson character models that are reused throughout the various towns in the game; one of them is a dancer girl. There is basically always a man watching the dancer girl and mentioning that she’s hot. There is a side quest that is “dress Jade up like a bunny for a lecherous old man.” There’s also the whole “puff-puff” thing; you can talk to various women in the game to perform the “puff-puff,” which has no effect on the story and is usually the set up for some kind of gag, but is also obviously a sex joke.

I have no issue with video game characters being hot. Costume changes aren’t very important to me, but they are cool, and some of them being hot is totally fine. But the game shouldn’t casually throw a stream of misogyny at you.

Miscellany

There are moves where it is hard to tell what stat is used to calculate damage. In particular, Rab is a character who can be either a physical or a magical fighter. Some ways into the game, he learns a move that, it turns out, uses strength rather than magical might to calculate damage. The game doesn’t tell you that, though; I had to look it up. Jade, similarly, has a number of moves based on her charm stat. Sylvando might, too; I didn’t check.

The game has a day/night cycle and a weather system. These almost never affect anything, but when they do it’s uniformly annoying. The day/night cycle is very long, and mostly you are just in day because when you rest, sleeping until dawn is the default option, and you’ll probably be sleeping enough that you’ll reset to day before it becomes night in most cases. I must have been 10+ hours into the game before I saw the day/night transition animation. Your camera is temporarily taken over and you stare into the sky, which then darkens. I thought it was some sort of important story event starting unexpectedly at first. It’s not elegant. The weather system is even more boring: sometimes, it rains. There are some enemies that only appear when it is raining, so if you are interested in fighting one of those enemies, you find a campsite and sleep over and over again until it rains. There are cows that tell you the forecast, but they’re not immediately adjacent to campsites, so they are absolutely useless – sleeping over and over is faster than running back and forth to the cows. You can sleep until four times – dawn, noon, dusk, and night. The only differences are between day and night, and sleeping until dusk does nothing but get you the weird transition animation faster, so there are four choices but only two categories.

Speaking of sleeping, it does not restore your dead characters to life. This doesn’t actually matter, because you learn Zing (the life spell in the game) reasonably early, and restoring characters to life is one of the functions of the (again, wonderful) handy heal-all. You can also pay for resurrection, but spending mana and then sleeping is always better. Sometimes, though, you’ll forget that some characters are dead, sleep, assume you’re healed up, and then start a battle down one or more characters. This is a really stupid gotcha, and provides the opportunity to have a feel-bad moment with no upside, because it doesn’t actually add strategic depth.

Churches (and statues, which are like churches but they appear outside of towns) provide a number of functions, most of which are stupid. They do saving, which is good. And later in the game they allow you to reassign skill points, which is cool. They also provide divination, which prints out about five thousand lines of text telling you how much exp each character needs. This information is freely available at all times, in a way that’s easier to get to and faster to read, but the game doesn’t tell you that, so until you stumble across it, if you care about the answer, you have to use a terrible interface for it. Reviving dead characters, as above, is handled badly. They will also remove poison and curses, which are dumb for the exact same reason revival is.

Good Things

Just to balance out the negativity here, I’d like to include some good aspects of the game.

The music is quite good. Apparently a lot of people took issue with the music, but it was good for me. I generally have on a stream or TV or something while I’m playing, and it was inoffensive and catchy as low-level background music.

A huge fraction of names (of items, monsters, abilities, etc) are puns, which is first distracting, and then sort of groan inducing, but is so thorough that it’s eventually just sort of part of the game, and the jokes are often pretty good.

The combat system is mostly a traditional turn-based JRPG system, but with a compelling tweak: different attacks target either single enemies, all enemies, or groups of enemies. Target-all abilities are in short supply, and having to think about enemies in groups is an interesting way to do combat. I’ve never played a Dragon Quest game before, so it’s possible this is an ancient innovation, but it was new to me and I liked it.

The game is arranged in three acts. You retread a lot of ground in the second act, and to some extent again in the third, but the game handles it well. New treasures appear, of course, and enemies power up. The atmosphere changes dramatically, as well, which helps it feel fresh, and you’re allowed to go through things much more quickly the second and third times. I thought it was very well done.

I fucking love the slimes. Drake slimes are my favourite, I think.drake slime

The third act, when you have access to everywhere and everything, is good. It’s a pretty nice world to run around and explore in, once it starts letting you rather than punishing you for trying.

The crafting system is good. It requires a little care, but not too much. If you’re good at it – or overleveled for whatever you’re making – you can make your crafted items stronger. It feels like something real, whereas it easily could have felt like a speedbump in the way of getting your gear. But it didn’t feel like something I had to master, so I think it hit the sweetspot pretty well. It does have one truly bizarre misstep: late in the game, you get access to a second category of crafting actions, and you simultaneously get the first action in that category. But… it’s the only action in the new category. It’s like the developers just forgot they started this new thing, or perhaps they had more options in there but got rid of them and never reworked it. This doesn’t make the game worse, but it’s a totally unforced error.

Early on in the game, your revival spell has a 50% chance of failing, which is pretty interesting. It’s a neat way to add tension to some early boss fights.

Posted in Uncategorized | Leave a comment

Performance Optimizations In Rust

Note: I wrote this a couple years ago, and abandoned it because it didn’t have a satisfying conclusion. It’s not a great story narratively but I still think it’s interesting, so here it is!

Some years ago, my coworker Zack wrote a a chess engine in Rust, to which I contributed a bit. Recently, my coworkers and I have started a new game of office chess against it, and it turns out that it runs too damn slow. On an early board position, it didnt come up with an answer after running on my macbook pro after about 8 hours when searching a 9-ply tree. And then the next day I ran it with an 8-ply search and 8 hours was once again insufficient.

Screen Shot 2019-04-22 at 21.37.50

I decided to try to make it fast, and then got sort of obsessed for a couple weeks and knocked out, depending on what task you’re running, 60-80% of the runtime. For the rest of this post, assume I’m running all timings against this position (sure hope it’s representative, lol).

An Extremely Brief Introduction To Chess Engines Using α-β Search

This isn’t what this post is about, so here’s the least you should need to know to read this post: Chess is a two-player perfect information game, so it permits an α-β search for finding moves. What you do is go through each move from a position (a ply), and then recurse; when you bottom out (however deep you wanted to search), you score the position based on some heuristics and go back up the tree. You use some algorithmic magic stuff to stop looking at trees when moves are really good or bad.

Chess games can reach the same position via multiple paths (transposition), so you want to cache positions that you’ve already scored, again to avoid looking through big trees of moves, which takes forever because of exponents.

What Not To Do

I don’t recommend taking this approach, but I had an idea for how to speed up the program before I bothered profiling anything, and it turned out I was really, really right. When we were caching the scores for various board states, we were caching the entire series of moves from the start state to the end of the recursion (the variation). That’s a ton of wasted space, and this is a very memory-hungry application in order to cache results as effectively as possible. I didn’t profile the exact mechanics of where things improved because of it, but I moved to only stashing the remaining part of the variation, so we can still stitch the full variation back together higher up in the stack, but save a ton of memory. This was worth a 30-50% speedup; you can see the implementation in this small patch.

In general, “have ideas” has been by far my least successful technique. Most of the things I’ve changed on a hunch have had non-noticeable or even negative impact on performance. Don’t be like me. Profile your application.

On The Right Track

Next up, I made a bunch of random changes and couldn’t tell if any of them helped or not. Eventually I got serious and started using tools. The first thing I tried was cargo-flamegraph, which doesn’t work very well on macs. (Foreshadowing: this story is about to come to the part where I have a second personal breakthrough). I spent a while doing horrible nonsense like booting in recovery mode to disable security settings, and ultimately found out that the tool just doesn’t follow threads on mac OS.

Next up, I found the tool cargo profiler, which installation tools start with “NOTE: This subcommand can only be used on Linux machines.” So I realized if I’m trying to profile code in an immature systems language, I need a Linux box. Fortunately, the cost of 1 Linux VM on EC2 is $0, so from here on out I was mostly developing on an Ubuntu instance.

Tools

I found that cargo flamegraph was extraordinarily valuable. Cargo profiler basically didn’t work for me, and I ended up just running valgrind myself, but the results were less useful than I had hoped. Probably this is a me problem. I mostly leaned on flamegraphs, which are wonderful.

Here’s an interesting portion of a flamegraph from this point.

Screen Shot 2019-04-22 at 22.29.54

(this isn’t the bottom of the flamegraph, but it’s a relevant part). The bottom row here, if you squint, says it’s in the α_β_negamax_search function. Everything above that is various function calls that the app spent time in while also inside the lower function; each column is sort of a cross-section of stack. So a wide column means we spent a lot of time in that function while it was also inside all the functions below it. Here you can see from the second row that while in this depth of the α_β_negamax_search call, we spent a little in a function (in orange) that I can hover over and see is called WorldState.reckless_lookahead; about 1/4 in order_movements_intuitively, and most of the rest in another recursion into α_β_negamax_search.

This is really valuable information! It was very surprising that we’re spending ~20% of our time in order_movements_intuitively, because it’s just a sort of a list with < 50 elements!

fn order_movements_intuitively(experience: &HashMap<Patch, u32>,
    commits: &mut Vec) {
    commits.sort_by(|a, b| {
        let a_feels = experience.get(&a.patch);
        let b_feels = experience.get(&b.patch);
        b_feels.cmp(&a_feels)
    });
}

The first thing I did was switch sort_by to sort_unstable_by, which the docs say is usually faster if you don’t care about stability. The docs are right! But it was still too slow. Next up, I — well, I combined a few functions together, because we were sorting and then immediately re-sorting with a different comparison function, so this part isn’t as narratively compelling — but then I pulled the cache lookups out of the comparison function. This was enough to plummet order_movements_intuitively down from 20% of the time spent in the negamax function to like 4%, which was great. And it’s the second time in a row that fixing the use of hashes was a big performance win, which it turns out is not a theme; there was just some serious low-hanging fruit there.

Rust Specific Insights

So far, all I’ve done is notice and correct what amount to a couple of significant design flaws. The tools helped me catch the second one, but as it turns out we’re out of this sort of problem. The rest is much harder.

The reason it’s hard is actually a good thing – Rust is really fast. It mostly doesn’t do stupid stuff, and just goes really fast and eats up as much CPU and memory as you’ve got lying around. This is, you’ll note, really fucking cool, and this is one of the reasons writing Rust is fun, especially for a filthy interpreted language pleb like myself. Unfortunately, for all that, it still does get harder.

The next big issue I saw was that some of our functions were spending up to half their time in Vec::extend and/or Vec::push, which themselves are spending a lot of time in low-level stuff like malloc, realloc, and free. This is the part where the project became really fun for me – I write Python in my day job, mostly, and writing python does not tend to lend itself to performance hunting on the level of “too many mallocs.” So the next step is to remove allocations.

This consisted of two main things. The first is using Vec::with_capacity to pre-allocate an appropriate amount of memory for my vectors. Sometimes I had known lengths (e.g. in the sorting of moves as seen above); sometimes I had known maximums (e.g. there will never be more than eight pawns owned by a particular player); sometimes I had good-enough guesses to feel pretty good about it (e.g. there will be more than 2 bishops in few enough positions that it doesn’t matter). Sometimes I had reasonable guesses – I’ve read that most chess positions have in the neighbourhood of 40 moves.*

The second thing is just not creating vectors in the first place. One of the main move-generation functions was delegating to a bunch of sub-functions to generate moves for each piece type and then pushing all those results onto one result vector. I changed it to pass a mutable, pre-allocated vector around and extend onto that.

These changes resulted in allocation-related time spent in the relevant function to drop from ~50% to like <5%, and some other allocations further down the stack to disappear from the perf tool’s sampling entirely.

At this point, from the starting point of right after I fixed the issue with storing full variations in the hashmap to right after I fix all these reallocs, I’m seeing time on a depth 6 search of the relevant position going from about 50-60s to about 30s.

The Bad News

The bad news is this is about where I ran out of ideas and/or energy. I had the temerity to make a couple more small changes by following basically the above pattern, but no more big wins. A couple things showed up that I thought were dumb, and I stamped them out – using a different hash algorithm for some data types was a big improvement on a micro-level but didn’t have a substantial impact on realistic workloads, for example. The program also uses immutable structs for various purposes, including the list of squares on the board and the list of pieces by team; moving those to static variables and returning them instead of re-allocating new ones when they’re needed, again, showed a pretty impressive improvement on the micro benchmark but minimal improvement on actual runs.

What’s left is that there’s just not much to strip out. The program’s threading model could be improved – we kick off one thread per move at the top level position, which frequently results in one expensive move’s calculation taking a long time after the other threads are done. I’m not sure how to make that better while still getting the α-β search benefits.

* I don’t know if this is true, or where I heard it. From unscientifically checking some positions, I’m seeing that 40 moves covers >= 90% of them.

Posted in Uncategorized | Leave a comment

How The Principle Of Charity Feels From The Inside

  1. Read something that sucks. It’s super bad, and it’s about something important to you. Your blood pressure goes through the roof, you can’t wait to show all your friends how fucking stupid this moron who doesn’t even understand the in-group is!
  2. Start composing devastating takedowns in your head. Maybe draft a tweet or even a blog post. Really nail ’em to the wall. Twist their words around, placeholders in your mind to zillions of links full of counterevidence, etc.
    1. Start collecting those links for your epic rebuttal. Notice that a few of them maybe aren’t quite as devastating as you thought in this context. They’re still pretty good, but you’ll have to be careful to word your response in such a way that they land properly.
    2. Start writing the witty rebukes. Notice that they depend on a lot of assumptions, and you wouldn’t be happy if someone reinterpreted what you said so sloppily. You’ve still got some material, but it’ll take more work to really hammer it home.
  3. Commit to writing a longer, more thought-out takedown. You can still do it, but it’ll be a little less concise. Notice that your initial response wasn’t responding to quite what the original piece said, because you felt some connotations that made you angry. Track down evidence that your victim actually does mean all the really bad stuff, so you can still be angry.
    1. Turns out they aren’t quite as bad as you thought. They’re still wrong on this point, but they’re only wrong wrong, not like burn-at-the-stake wrong.
    2. Clean your piece up so that it is responding to the actual points of discussion, and not railing about what anybody “secretly means” or what other people who agree with them on this point might believe.
  4. Notice that your “devastating takedown” is no longer either; it’s now a pretty level headed response to a bunch of points that you disagree with and that you’re pretty sure you’re correct about.
  5. Realize that nobody cares about a 3000 word rebuttal to what turns out to be fairly minor points of actual disagreement just because you have different conclusions in the end.
  6. Accept that this is no way to win the culture war, but at least you’re not one of those assholes who shoots from the hip and ends up saying a bunch of dumb false stuff that only signals group affiliation and has no basis in actual truth.
  7. Hit “save draft,” sigh, and move on.
Posted in jokes, pedantry | Leave a comment

Invisible Segmentations

People who learn about aphantasia tend to be shocked that there are people out there with a vastly different imagination than theirs; and, moreso, they are shocked that they didn’t know. Everyone understands the general meaning of the phrase “imagine a beach,” and it’s jarring to find out that some literally can’t!

Aphantasia has been actually studied by Serious People, so it definitely counts – it’s a secret segmentation of the population.

I posit that there are many others.

Some others have been studied – “psychogenic” or “emotional” shivers apparently occur while listening to music to about half of people. Are you one? I’m having trouble tracking down prevalence of emotional shivers overall, so I’m not sure which side here is more common.

Some people sometimes mess up their left and their right. I think this is because some people learn a cobweb of tricks for quickly determining left vs right, and others feel that left-ness and right-ness are physical properties of their bodies. According to studies the first thing I found online, maybe about 1/3 of people at least sometimes mix up left vs right.

Here’s one that I’m not finding any studies on, serious or otherwise: general cool twitter person QiaochuYuan posted an interesting twitter poll about experiencing tactile sensations in dreams. It appears to be pretty even! Searching the internet for this mostly brings up froufrou dream analysis nonsense and a smattering of forum posts that are mostly connected to lucid dreaming. And yet – I don’t think I’ve ever felt physical sensation in dreams; apparently about half of my fellow humans do?!

Screen Shot 2020-05-07 at 12.44.34 PM

I intend to edit this post as I come across more such segmentations. If we can find about 30 more, we can get pretty close to uniquely identifying a human based only on hidden, unobservable differences!

Posted in sociology | Leave a comment

They Should Have Sent A Movie Director

Final Fantasy 7 Remake* is a decent movie, telling a good story, awkwardly bolted onto an okay but frustrating game. It suffers from, in my opinion, two primary flaws that have similar impacts on the player, but was a pretty good time. I’ll start with the bad stuff.

Movie Envy

Movies enjoy a certain degree of prestige in our culture, and video games don’t yet. They will, of course; but in the meantime, sometimes video games that want that prestige ahead of schedule seek to co-opt it by becoming, functionally, movies. FF7R is best understood as a movie, or perhaps a miniseries (it takes upwards of 40 hours!), with brief moments of interactivity, and approximately 3 sections where you are granted some agency.

The game is too railroaded. You have an objective highlighted on your map, and you walk towards it, and periodically there’s a mandatory encounter, and periodically you lose control of your camera. You cannot explore side paths. Side paths exist to let you know that you’ll be back in this area later and railroaded onto a different path. When you try to explore, the game flashes a “no” sign at you and makes your character turn around. Sometimes, when you walk into a room, you’re not allowed to leave it, while still having “control” of your character to go walk to another place and start a cutscene.

Most sections of the game are one long hallway. This is, of course, true in a topological sense of most RPGs, but here it really feels like it. The branches you can explore are short and marked on the map and have, mostly, a commodity item. Most of the time you see something that looks interesting and isn’t in the direction of the objective, you try to walk towards it and aren’t allowed, and then you sigh and go watch a cutscene. Eventually you give up trying to play the game and just resign yourself to watching a movie punctuated by walking down a hallway and fighting a couple battles. The game resists being played at every turn.

Combat Agency

It’s well known that people feel more pain when losing something than they feel happiness when gaining the same amount. Like, having someone give you $100 feels great, but having $100 stolen from you feels much worse. In FF7R, the player is granted more control over their character in combat than in traditional JRPGs. This is good. However, that control is frequently taken away. This is proportionally worse. I understand why there is some interruption – if you were just an unstoppable force the whole time, you’d have no skin in the game. It just happens way too much.

I suspect it’s because the game wants you to really engage with the combat system, but there’s no reason to – your party is generally overpowered, and you can get away with just wailing on enemies however you like in almost all circumstances. I’m sure some people find the combat system compelling and fun, but you can ignore it and still win all the battles pretty handily while periodically getting pissed off that you can’t do what you want.** Overall, my choices seem to be doing combat the way I like to in RPGs and being annoyed, or playing what feels like an incredibly bad fighting game. Neither choice is appealing.

Perhaps more accurately, the game wants the bosses to be MMO raids. But raids are fun because you’re doing them with your group of friends and you’re working together and you can laugh about it when someone steps in the bad and you reset. Doing raid mechanics alone is just miserable.

No Fun Allowed

In general, the game isn’t very fun. It’s pretty, and as I said the story is good, but it’s not fun. Every time you want to do something for fun, the game refuses and forces you to get back on the main track and watch more cutscenes. You can never explore, and when you try you are punished. You have to do combat exactly the way the game wants, or you are punished – and not punished by losing, punished by having your control intermittently stripped away, which is much worse. The frequent removal of camera control is shockingly annoying. It sounds like such a minor point, and maybe it’s just because of all the other ways your control is removed, but it’s so annoying. I want to look at the world the developers built, and they want to stop me from doing that?

Overall, I don’t think I’ve ever played another game that is so adamantly opposed to player agency at all points.

The Good

The game is quite pretty. It’s not as pretty as it should be – it’s less pretty than, say, Horizon: Zero Dawn – but it’s still very nice to look at. It might look better on a PS4 Pro, which I don’t have. The story, as I said, is good. The quality of life is fairly high, although load times are brutal if you’re retrying a fight or whatnot. The parts that are fun are pretty fun. I suspect that people who really like movies would enjoy it more than I did. Once in a while the fights happen to hit the line of mechanics-heavy enough to be challenging, but forgiving enough to be fun. It’s rare, but it happens, and those are good.

The best part of the game is Aerith, whose cheeky personality and profound empathy and goodness burn bright. She’s true to her original character, and her voice acting is really well done. The characters generally are well done. They kept the spirit of the original – there’s a lot of humour and a lot of cursing, for example; the amount of profanity is pretty surprising compared to modern fantasy settings – while ramping up the production values to 2020 levels.

Overall, I think you should probably play it if you liked the original Final Fantasy 7 so much that you want more of it.

*I believe the full title is, in fact, Final Fantasy 7 Remake, thus not calling it “The Final Fantasy 7 remake” or whatever. I’m wondering what they’ll call parts 2 and 3, given that they didn’t name it part 1. Redux? Reignited? Remix to Ignition?

**After writing this draft and before finishing it, I went and played the Airbuster fight in hard mode. This fight is a nonstop barrage of hard-to-predict attacks that stop your characters from moving for long periods, topped off with a phase where you mostly can’t hit the boss and, as far as I could figure out, can’t dodge its strongest attack. I ended up resetting half a dozen times because the mechanics were so demotivating. It felt actively hostile to the idea of fun.

Posted in video games | Leave a comment

Another False Eschatology

You know how light always travels the fastest path, even though that’s obviously impossible and makes no fucking sense? Also, how the many-worlds interpretation is true? Bear with me, I’m going somewhere with this.

The you that experiences has to be alive, for obvious reasons. So any time there’s a world which diverges in a way such that you cease to be alive, that path isn’t taken from the perspective of the experiencing you. This feels like a continuous path through time, but what if it’s really a post-hoc picking of the worlds in which you survived? There seem to be real constraints on lifespans, but it’s possible that no one experiences dying until they’re out of possible worlds that keep them surviving. The yous that died of preventable stuff – car accidents, or whatever – aren’t around to clutter up your experience of liveness, since you can’t experience those branches. The word “you” is doing a lot of work in this paragraph, since I’m sort of divorcing it from the normal “you” concept. I apologize if this is confusing.

The obvious objection is that people do in fact die of, like, car accidents. But that’s not actually a problem! You didn’t die in a car accident. The fact that you can continue inhabiting a timeline where other people die is not a contradiction.

The other obvious objection is that this is crazy and makes no sense! That is a problem, I admit. But light traveling the fastest path is also crazy and makes no sense unless you assume that the “right” world gets chosen after the fact.*

Anyway. It’s probably not true. It’s also probably unfalsifiable. But I like it.

Even assuming this whole thing is true, it doesn’t license you to take extra risks. You can die perfectly easily in my experience, and I would rather you not.

 

* This is probably not true; wikipedia has some things to say about waves that I can’t really follow.

Posted in philosophy | 2 Comments