Weather Factory

Ryan Sumo (@RyanSumo) tweeted at 8:56 am on Tue, Jun 05, 2018:
‘I know you’ve already touched on this in a talk but practical strategies for sifting out important feedback would be much appreciated. Maybe a future blog post?’

Ryan, it’s been a while, but here it is. It’s a long’un: #5 in the bullet points at the bottom is the most useful bit, if you’re in a hurry.

—

Possibly you’ve heard of ‘Theory X’ and ‘Theory Y’. They’re two ways of looking at how people can be motivated to work, suggested by a gent called Douglas McGregor at MIT in the 50s. Theory X proposes that humans dislike work, avoid responsibility, and have to be ‘coerced, controlled, directed and threatened’ to get work out of them. Theory Y, on the other hand, proposes that work is a natural activity; that humans seek responsibility; that we do our best work when we’re allowed the most initiative.

Of course neither of these is straightforwardly true. Some jobs and some people work better in an X-ier way, some in a Y-ier way. But more than that, the same people at different times can be X-ier or Y-ier. When I was seventeen and working a summer job as a kitchen porter, I certainly wasn’t looking for initiative or responsibility.

So it is with feedback. Here’s a Theory X and Theory Y for feedback.

Theory X: Players aren’t experts in making games. Designers are. Player feedback expresses naive personal preference, not dispassionate assessment. And pressuring a designer to change their game is an attack on something they made and love – if you tell a parent that their kid is badly behaved, it’s unlikely to get a good response, even and especially if it’s true.

Theory Y: Players often know more than a designer about what it’s like to play a game. Games are engineered experiences, and getting data on how those experiences feel is of paramount importance. Feedback might sometimes be difficult to hear, but it is an unparalleled learning opportunity for designers to improve their games and their skills.

Most experienced designers, on a good day, will tip their hats to Theory Y. The same designers, on a bad day, will just want everyone to shut the hell up either because (a) the player is wrong or (b) they’re right but the designer ALREADY KNOWS.

To put it another way, all feedback is in principle useful. But some feedback is more useful than other feedback. Some feedback is more pleasant to hear than other feedback. And all feedback takes time to hear or read, and more time to respond to.

Or to put it another way, feedback is like sunlight. It’s vital, it’s generally something you want, but if you are exposed to too much of it without protective measures, it’ll take its toll.

I engage more directly with my community than some, perhaps even most, designers.  In any average work day I can easily see a dozen pieces of feedback: forum posts saying ‘I loved this  game!!’, personal insults, forum posts dissecting my game design, tweets suggesting UI enhancements, tweets insisting on UI enhancements, ten-word Steam reviews, thousand-word emails with bullet-point lists of what needs changing and what needs changing urgently, hundred-word emails gushing about how delightful an experience the game is,  fan art with an in-joke about a bug, blog posts reflecting on the game critically, Reddit posts asking plaintively how to get past a blocker…

And sometimes that’s great and sometimes that’s like eating nails. There are negative reviews, there’s personal abuse, there’s unflattering speculation about my motives… oh you know. But more than that, there’s just the matter of reading the exact same feedback like a hundred times. This is surprisingly hard to deal with, because one of the things about correct observations is that they will occur to lots of different people, and they all tell you, and it wears your patience away like dripping water.

(And a correct observation may still not be actionable criticism. It is correct that Cultist Simulator does not have a tutorial, and that this puts some people off. But that’s an intentional, if unusual, design choice on my part. It is correct that Cultist Simulator would probably work pretty well on Switch. But we can’t just push a button and put it there.)

And as Lottie has said elsewhere, it got rough in the week after launch. Cultist ended up at the top of the Steam charts, where we’d never expected to see it, and a lot of mainstream players downloaded our weird little niche game, and a lot of them had a lot of things to say about it. I saw a lot more than a dozen pieces of feedback a day.

How do I feel about that now?

I’m still overwhelmingly glad that I engaged with feedback as much as I did, before and after launch. The game is much the better for it, I’m a better designer for it, and I’m grateful to everyone who took the time to send thoughtful responses. Honestly, I think I’m grateful to people who sent mean responses too, though I’m not going to send them any Christmas cards.

Here’s my own route up Zen Mountain to Feedback City. I can’t guarantee it’ll work for you. And – vitally important – engagement with feedback should be voluntary. Neither your audience nor your manager owns your brain. It’s up to you what you decide to sign up for. But here’s what’s helped me to get the most from it.

1. Recognise that players may not mean it that way. For you, the game may be the culmination of years of sweat, passion and heartbreak. For a player, it may be a thirty-minute drive-by experience on an evening where they had a migraine and the pizza was late. It’s a casual interaction with a faceless Internet entity. They don’t know whether you’re one desperate coder-designer with an elderly laptop or whether you’re a glossy, well-funded studio of untouchable Californian tech millionaires.

2. Recognise that players often have a motive that doesn’t align with giving the most objective feedback possible. People post on forums to show off their snark to their peers. People post attacks on your design methodology because they’re frustrated designers. People send you emails full of aggressive suggestions because they secretly hope you’ll hire them. Or people are drunk. Or people very nearly loved your game but hit a bug that drove them wild with rage. Hypothesise one of these motives, and it may allow you to see past the nasty. David Foster Wallace is good on this.

3. Special mention for cultural and indeed neurological differences. If you have a player whose command of your own first language is not English, they really may not mean to be that rude. Russians speaking English, for instance, can come across as quite brusque to native English speakers. We have a lot of Russian fans, and I’m more attuned to it now, but it took me a while. I had an email last month that seemed really aggressive… and found out afterwards that the writer was autistic and hadn’t intended it that way at all. If you extend someone the benefit of the doubt, you’re doing the both of you a favour.

4. Educate your community. You can’t engage in detail with every criticism of your game; nor should you. But if you keep seeing the same point, and you have a useful point to make in response (‘our DLC policy is fair, because…’ ‘Yes, this was an unusual design choice, but here’s why…’ ‘This is a fair criticism – here’s how we messed up and what we’ll do next time…’) then consider putting the time into making a lengthy post about it. Your community can link to it, or use that information when responding to other players. Engaged communities are often respectful when a dev takes the time to give a thoughtful response, and often want to establish their credentials – or just pay it forward – when newcomers ask the same question later. This, in turn, may lead to more valuable feedback: ah, I see now what you were trying to do, says a player, but actually the way it came over to me was…

Corollary: it can be much better value engaging in detail in a public space, where many will read your response, than in an email exchange, where one person will. Consider publishing your response in a blog post or on Twitter, where it can educate your community. (Remove identifying details and/or get your correspondent’s permission, of course!)

5. Establish Gong Rules. Okay, this is the Secret Tip I’ve found most useful. Give any piece of criticism three strikes. If it hits three strikes, give yourself permission not to read any further. Decide your own list of strikeables, but here’s my top ones

• factual error
• very long post/email
• feedback I’ve seen before
• aggressive tone, sarcasm, abuse (per instance)
• the phrase ‘just bad design’

Why does this help? Firstly, it gives you a feeling of control. You’re no longer just standing there while someone sprays you – you have your hand on the hose spigot.

But secondly, you’re building an algorithm to help you allocate your time usefully. I know from experience that sometimes a snarky opening can precede some really useful feedback. Sometimes people are just vexed. But I also know that with finite time in my day, (1) snarky opening + (2) a point I’ve already seen + (3) oh my god it’s a whole page… is less likely to be worth the fifteen minutes and the personal wear and tear it will take to engage.

Make it two strikes or five or one as you prefer. Build your own list of strikeables. Specific things will piss you off, specific things will be consistent markers of likely low quality feedback. If you have a specific strategy, you’ll save time and feel better.

6. When something stings,  try to figure out why, but most importantly take time out. I have a pretty thick skin these days. But odd stuff slips past my defences. There was a Reddit response a couple months back about how Cultist Simulator was a failed experiment that left me too angry to sleep –  because it was an unexpected off-topic addition to a thread where I was trying to be good-natured. There was a Steam post the other day that was only modestly critical but which referred to me by my first name, and that felt way more personal. Understanding why they annoyed me helped, but it didn’t fix it. I found myself responding more ill-temperedly to other feedback, and I stepped away for a day. Made a big difference when I re-engaged.

This segues neatly to two parting pieces of advice.

• recognise that patience is a resource. Ration yours the way you’d ration physical strength.
• the good news: like physical strength, it can increase when exercised carefully.

Good luck. Remember most people mostly mean well.