You Beta You Beta You Bet

There’s a pretty good article over on Ten Ton Hammer offering a look at the current state of beta testing. As many others have noted, not much testing goes on in the public stages of MMO betas, and the article offers a variety of perspectives from both developers and players.

My old comrade Scott Hartsman chimes in with a voluminous set of observations (without me around to edit him, Scott will just talk forever!) on how a big-budget game can manage to make it into beta and be lacking in the fun department:

Let’s say you’re a AAA game with 3-4 years of time and money invested, enough money to support a large team having worked on it for that long.  Games like this frequently need to go for years before enough pieces come together before you can start making decisions about what’s fun and what isn’t.

By the time beta begins, you’ve made decision after decision that have compounded on each other.  Your assumptions’ assumptions’ have assumptions about what your game is.  The whole product, systems, content, operations, marketing, PR, community ramp, you name it — is built upon them.  Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month.  (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)

The game is, for the most part, what it is.  You’re capable of making shifts, but the more complex the game, the more minor the shifts you can make with any confidence.  If assumptions that you made years ago turn out to be wrong, you’re left to scramble, or in most cases, do your best to ameliorate the now-certain fallout.

If you haven’t verified your gameplay at the point of having a beta, you’ve already left your fate to chance.  (This is, of course, all presuming that your game has passed the technical bar in terms of stability, which is all too often not the case.  And, again, is another flaw with the launch-big-or-die model.)

The guy from ohai speaks true. If you start out with mediocre core gameplay and pile tons of mediocre decisions on top of it, soon you’ll find yourself in a plate of spaghetti and weaksauce you won’t be able to escape (how’s that for mixing metaphors?). For instance, if the act of killing enemies over and over is the centerpiece of your gameplay, then you better make damn sure basic combat feels interesting and cool because otherwise nothing will be able to detract attention from the suckiness.

On a more positive note, I recently had a happy beta experience and felt I should share. I installed the Chinese open beta for Aion after finding instructions on how to apply an English-language patch. Considering the hoops jumped through to get the client running in English, I expected a train wreck and was hoping I could at least get a feel for the visuals and gameplay. Instead, I found myself pleasantly surprised that the language stuff worked and the game played smoothly. Though the experience wasn’t perfect, there was a thoughtful layer of polish to the UI and other elements of the game that I found refreshing. After not expecting much more than a grindfest, I’m now looking forward to the US version.

So beta–or live preview, or whatever you choose to call it–still has the potential to positively influence the market toward your game, so long as you when you lift your skirt what you reveal to the world is solid (ugh, that metaphor is simply uncomfortable).

Look, every shipped triple-A MMO is going to have bugs. These games simply have too many systems and too much content to test every possible use case before launch, and everyone from Blizzard on down spends the first few months after a major release fixing things. And players will accept that, as long as you’re up front about the issues and the core gameplay experience is rich and enjoyable enough to mitigate the bugs until they’re stomped out.

But public perception based on that first public showing has a measurable impact on your game’s ultimate fate.  Now more than ever, major signs of weakness are going to haunt your sales potential for eternity, and even the greatest patches can only do so much to fix that lingering initial perception.

Published by


Steve Danuser, also known as Moorgard, is a a writer, editor, and game designer.

5 thoughts on “You Beta You Beta You Bet”

  1. Now more than ever, major signs of weakness are going to haunt your sales potential for eternity, and even the greatest patches can only do so much to fix that lingering initial perception


  2. Steve,

    I wonder if game developers shouldn’t look at changing the way they test & evaluate games. You mention that not much testing goes on in a beta test (certainly that has been my observation too).

    It seems to me that there are two issues here. The first issue is testing – making sure that code actually does what it is supposed to do. This is a very concrete issue – either the code works, or it doesn’t. Those of us who develop software in the business world have started to adopt Agile methodologies, particularly TDD and other automated ways of testing code, to ensure correctness (as well as guiding design and providing for regression testing, which makes code changes and refactoring so much easier). I have read about some game companies now adopting Agile methodologies and I think that’s probably a step in the right direction. Ensuring correctness is critical (players really gripe about bugs, and while we can say things like, “Hey, these games are huge and they’re going to have bugs” it doesn’t give us license to be careless with code) and this is the part that doesn’t really take place in a beta “test”. Which makes me believe that it needs to be a much more integrated part of the development process, like it is in an Agile shop. Testing, when automated, can have a big influence on the quality of a code base.

    The other issue, it seems to me, is evaluation. This is completely subjective. Is the game fun? Is this quest fun? Is this combat fun? Is this zone fun? That’s the thing that does happen in a beta test, but by then the die has pretty much been cast. The game is what it is, so what purpose does the beta test serve? (And I think this is the thrust of your post). Other than testing load balancing issues and payment processing, I’m not sure an open beta does much.

    And so I think that like code testing, evaluation has to be an integral part of the development process. It has to happen every day. If you can evaluate something in small chunks it will allow you to see the error of the design before you start piling on other assumptions… I think you’ve got to have people dedicated to the idea of evaluating the game play every day while the game is being developed. Giving feedback to team, guiding the design. Much the same way unit tests can guide the design of the code, evaluators need to be there to guide the design of the game play. I’m sure most games have those guys – you guys. But maybe they don’t. I don’t know. Never had the privilege of working on a game before.

    P.S. I’d love it if you would post some info on how you guys actually develop your game at 38 Studios. Maybe only developers like me would find it entertaining though.

  3. “As many others have noted, not much testing goes on in the public stages of MMO betas”

    Why is there even an expectation of test occuring during a public MMO beta? Are testers actually testing the product? No. Do testers have access to knowledge that make testing possible such as documents, environments, tools, code, content, the team building the product etc? No. If a company cannot stress test their system without using thousands of potential customer computers to do so they probably should not be in the MMO space.

    A possible suggestion is to call these public efforts “MMO demos” or “Warming up the Support and Live teams of an MMO” as they have nothing to do with testing.

  4. By the time a MMO gets to the beta phase all of the major gameplay mechanics should be pretty much cast in stone. Beta should really be for stress testing more then anything.

    I was fortunate to be a part of the WoW Friends & Family alpha program and believe me when I say this, WoW was very polished then. It was fun, breathtaking, engaging and simply off the charts as a MMO experience. It was light years ahead of EverQuest and DAOC at the time.

    By the time your MMO reaches beta it should be ready for the public. As Steve has said, the public will not tolerate a MMO that is not polished and ready to be shipped even at the beta stage. How many times have we seen major MMO companies violate this cardinal rule in the past few years?

    Ship it when it’s ready or don’t bother shipping it. Tell the publisher and the marketing departments to go jump in a lake if they tell you otherwise.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>