Buzzkill
Over on Gamasutra, Richard Garriott’s Tabula Rasa‘s Richard Garriott (see what I did there?) gave an interview that touches on what was done right and wrong in the marketing of the game:
“I actually think the biggest mistake was made not by the marketing department, but by the development team. We invited too many people into the beta when the game was still too broken.”
I’ve discussed the change in perception about what beta means before, and Garriott’s interview illustrates my points nicely. Clearly the marketing imperative to get the product name out in front of the public trumped the dev team’s imperative to make a fun game. A common tale, I fear.
Which leads us to another point to consider, just so you don’t think I’m repeating myself.
The number of non-paid testers who can objectively look at the early stages of a game and not judge it as compared to a finished product is very, very small. Some can, which is why those people are excellent to bring into a friends-and-family stage of beta. But such folks are vastly outnumbered by the majority of players who want to use beta as a chance to play a game for free.
The lesson here is that, outside of your paid testers and those few you know you can rely upon for objective feedback, you shouldn’t put any phase of your product out in front of players until you know it’s fun. Realize that your whole game is going to be judged by the smallest detail of what you put in front of the audience, and your beta NDA is only as strong as its weakest link.
I wasn’t in the World of Warcraft beta, but everything I’ve heard about it suggests that the game was already fun when it got into the hands of a decent audience and it only got more polished and fun from there. Compare that with Vanguard, which suffered from negative word of mouth during its beta and saw many of its major flaws go unaddressed by launch.
Good buzz is crucial to the commercial success of any art form, and games are no exception. Rare is the book, movie, television show, or video game that overcomes negative consumer buzz to become massively popular. While it’s not unusual for the audience at large to ignore the reviews of professional critics, most people put a lot more weight upon the opinions of their friends, family, and coworkers.
If the people around you say that a game sucks, and online influencers in the communities you frequent concur, you’re probably not going to buy it. All the interviews given by visionaries about how much better the product has gotten since launch will only do so much to change that fact.
Your whole wardrobe is going to be judged by the outfit you wear to your sweet sixteen party. A lesson for us all.
I was in the WoW stress test which was one (two?) months before the closed beta and it was fun then.
It’s tough when the money men come and look you in the eye and tell you that you are hitting a date regardless of reality and say no.
“Your whole wardrobe is going to be judged by the outfit you wear to your sweet sixteen party.”
That sounds like some deep-set regret….
You’re right about word of mouth though. WoW … very unstable at launch, to the point of being nearly unplayable for some folks. It survived the heavy technical glitches because it was fun and had a lot of positive momentum and managed to remain fun as people played into the higher levels.
Other games have launched to positive buzz, only to have the buzz subside due to negative comments about the game becoming Grindy (lotr) (or other issues).
It sounds simple, right? Just make a game that everyone will talk positively about for as long as they choose to play it…
I originally wrote “coming out party,” but realized that the phrase doesn’t mean what it used to. Which, in turn, makes me old.
Sometimes I dream that my site’s pingback will work one day.
I don’t agree. The perspective is wrong.
Good buzz is crucial because it can’t be fooled or persuaded. You could have waited 6 more months to invite players to Vanguard’s beta, a full year. Good buzz just wouldn’t come out of it, because good buzz only comes out of a good game.
WoW didn’t invite players (NDA was removed in February) when it was good. You can ask players that participated to the friends & family beta and WoW was ALREADY good. That was more than one year before release. I was inside in February and I thought the game was going to be released in early June because it was basically ready.
It’s not “when”. It’s about a fact: the game was on track to be really good.
Vanguard was on track to be a trainwreck. Slightly better or worse wouldn’t make a substantial difference.
A beta is a moment of truth, where the promises fall on the ground and people can see if the game sucks or is good.
It’s not about timing, it’s about being good. Time is one factor, not even the most important. If people join beta and say the game is crap, it’s because it is (and will be proven true) unlikely that the game will substantially improve.
I can assure you that the great majority of beta testers KNOW if there’s potential in a game and if the game company can realistically address the problems there are. If the buzz is no good it’s because they are more honest and direct than all the official press releases.
It’s not the power of persuasion of the buzz, it’s the reality of mediocrity of the product.
Of course timing is a factor. No matter how great the end result, there are points in any game’s development that it would be too soon to throw it in front of the public because doing so would give a distorted impression of where the game is headed.
Sure, some testers can see and acknowledge potential, but the farther away from completeness you are, the less likely an outsider is to recognize the potential of that completeness. Some will, others won’t.
In any case it’s hard to generalize too much, because all games don’t share the same development processes. Because of the way Blizzard makes games, WoW was almost certainly fun much earlier in its cycle than most of its competitors. Ideally, this would be the norm rather than the exception.
I’m arguing the contrary.
In many game’s betas the problem is that testers are invited too late. When their feedback can’t be seriously considered and can’t have a meaningful impact because it’s just too late to change things.
There’s beta, and there’s demo-beta. The latter is used to publicize, the former to make a good game.
Naysay doesn’t begin when the release is far away. Naysay begins when players play a game two months before it’s out. And see clearly it sucks. And say so.
The problems of those examples you made is that testers were invited too late. Not too early. Beta testers for Tabula Rasa arrived like three months before release, for a game that was in development for many years.
You couldn’t invite them LATER than that.
Abalieno, the idea that beta testers can make the game better only goes so far.
Many of those same beta testers that joined Tabula Rasa “too late” were the same ones who logged in, played a little, hated the game, then logged off and waited till the NDA dropped to start talking about how much they hated the game. By the time changes were made, the word was already out that the game was bad news.
Your argument presumes that all beta testers are high quality, impartial individuals who understand big picture game design. Now, if only you could show the rest of the class where this magical “awesome beta tester tree” grows, we could all expect such results earlier on in the testing process.
I’d argue that if you can’t control the quality of the testers, you have to assume that none of your testers are really “testers” in the first place. In fact, I’d go so far as to argue that beta “tests” are no longer tests (and they haven’t been for years). A beta test is essentially a public demo of your product. You can put an NDA on the product, but unless you have a way to truly back that NDA for every anonymous review on the web, the NDA is simply worthless (as Steve said, only as strong as your weakest links – which plays back into controlling quality of testers).
We’re not talking about paid internal testing here. We’re talking about putting a sneak preview of a product in front of a public audience. Sure, you can make them “sign” a waiver…but does anyone honestly expect that to stop the leaks?
All you can hope for is that the leaks are more positive than negative. You hope that the negative leaks discuss bugs or problems with server stability – issues everyone expects to see in a beta test. The last thing I would think a studio wants is to see actual negative press regarding the gameplay itself, because that’s not a technical issue – but a perceived fundamental design flaw.
To recap: I think we’d all agree that players tend to be more forgiving of technical issues during a beta test than they are of actual game design issues. If beta testers are having to point out game design issues to designers during a beta test, that’s probably a “bad thing” (TM), especially from a marketing standpoint.
“To recap: I think we’d all agree that players tend to be more forgiving of technical issues during a beta test than they are of actual game design issues. If beta testers are having to point out game design issues to designers during a beta test, that’s probably a “bad thing†(TM), especially from a marketing standpoint.”
How true. I remember the Vanguard beta, which I honestly enjoyed, save for the fact that my machine couldn barely cut it due to the mass amount of memory leaks and bugs and such. Even though I tried to be impartial about it, there were enough basics that weren’t quite right to get in the way of me being able to actually see the technical issues around them. Honestly, I still thought it was fun for the 18-ish levels I played through, though I heard that much beyond that the content fell away and the “I’m only here to play for next to nothing” crowd had already ground up high enough to report that there wasn’t any endgame to test (since apparently that’s what everyone thought the game was supposed to be about). I’m hoping that SOE can save it, since I really do see a lot of potential in it now that a competent management team is behind it.
Tabula Rasa was easy to spot as a failure in the first week I played in “beta”. Unfortunately, it was far too late for them to implement any suggestions as to make the game better three months before release. It’s really disappointing since it seemed like it had potential (in writing) but failed utterly on execution.
A game company would be a fool to believe they are getting free technical advisers in beta. You’re getting a pre-release public “feel” for your game, and you can (and should) listen to them before it’s too late. Quality testers in beta means to most people those who will give you feedback on what is fun and what is not. NDAs are some of the funniest obstacles to this, because they encourage people to hold all their opinions back and blast you with a nice big surprise on launch day instead of voicing their opinion publicly beforehand so you can put pressure on big daddy (your publisher) to give you more funding or more time so you can make a quality product.
Instead, most developers insist on holding onto this paper (NDA) like it was a dark and secret contract bound in blood to the holder. Developers SHOULD be taking suggestions into account and sending feelers out into the community, but instead you get a strictly moderated form of communication and little to no outside viewpoints because everything is funneled to them through 17 layers of obfuscation.
To summarize:
Letting your players blog about their beta:
Good
- You can get real, non-formatted feedback directly to developers
- You could have some insightful commenters that are not in your beta (larger pool of ideas)
Bad
- The players can publicly berate your design choices
NDA restricted beta:
Good
- Your players can’t say anything negative about your game or you can sue them
- You can fix technical issues before they reach mainstream site
Bad
- Suing your players is generally the worst form of PR you can get, especially if the community sides with them
- Most players with informed opinions tend to avoid forums and “feedback forms” like the PLAGUE, your quality of feedback is very limited to those with a loud voice and too much time on their hands
Protocat.
You’re missing some fairly basic points about betas.
Firstly, the main point of beta is not only to put your game design ideas in front of an impartial audience, but because something as big and complex as an MMO with so many layers of interdependence can’t be adequately tested by the average QA department alone. There’s nothing like a few thousand players on your servers for a weekend to find bugs and glitches that your QA department didn’t spot in three months of internal testing.
Secondly, the main benefit of an NDA is that it allows the developers to control the flow of information about the game for marketing and hype purposes. Additionally there are some benefits regarding keeping stuff away from the eyes of your competitors. I can’t imagine that suing beta testers is ever going to happen except in the most extreme circumstances. The average guy who posts ‘LOLOL this gaem sux cos of…’is going to get booted from the beta and that’s about it.
Thirdly I’d disagree with your assessment about testers staying away from official beta forums and the like. Our metrics show that a very high proportion of our beta testers are active on the forums and we’re getting high quality feedback as a result. I doubt that we’re unique in that regard.
All of this of course implies an early beta phase which is more about testing rather than a marketing beta or stress test. Both of those last two stages generally don’t have an NDA anyhow.