Lessons Learned Revisited – Testing

Several posts ago, I mentioned that I would be re-visiting several of my “Lessons Learned” from my NightFall post-mortem that I wrote back in 2007, and now it’s time for another one.

Really think about the best way to get critical feedback on your mod from a large audience. The important things are always gameplay mechanics and story. Playing the same game over and over warps your view on what is good and what it bad – so get your mod to a releasable state, and release it to get public opinion. Take the feedback, improve, and release again. The great strength of mods is an incremental release model. Utilize it to make sure your product is the best that it can be.

Over the past few years gaining some experience in the indie game development space, I can definitely say that a lot of the original “Lessons Learned” can be applied to game development, not just mods. The key difference between the two is obviously that your aim with a commercial game is generally to sell it and make money, so very rarely can you release a base product and continue to iterate on it, and then expect people to pay money at the end (or expect good reviews when you didn’t have a good amount of content in your first release). The obvious exception to this rule is Minecraft, but it must be said that no one in the industry understands exactly why and how Minecraft has become such a phenomenal success.
Many games have attempted to follow the success of Minecraft recently, with Desura introducing an Alpha Funding initiative that allows developers to crowdfund games while getting access to an Alpha version of the product. Probably the two standouts from this crowd are Project Zomboid and Overgrowth, although both of these games seemed to have a strong following and community involvement before being involved with the Alpha Funding program. For the most part, I believe that sales numbers from the Alpha Funding program are probably only in the thousands.

So the question remains how indie developers can get good feedback and put good testing methodologies in place for their games in order to create a polished, usable product.

The lack of ability to release early public builds means that you need to make sure you stick to a fairly strict methodology and test often. But this must come with with the understanding that it can often take a while for a product vision to come to life. In recent times, we’ve loaded up our game and really enjoyed playing it. There was an aspect of “Oh, crap, our game is really fun! How did that happen?” about it. I get the feeling that this isn’t uncommon among indie devs – often it can take a while for the pieces to come together into a product that makes sense.

The first hurdle that many indies encouter is simply creation of a test build to send to people. Good processes, such as continuous integration, and use of services such as TestFlight can make this process far easier. You’ll be far more inclined to send out a build if you can do it in a few clicks, rather than spending valuable development time packaging and distributing builds. Even something as simple as being able to zip up your game, upload it to a FTP and email a bunch of people can be frustrating to do often, but extremely easy to automate through as little as a few hundred lines of code.

Secondly and more importantly is the availability of testers, and who you get to test your game. The most crucial aspect of choosing your testers is ensuring that you can cover two groups: new players, and return players.
Every iteration, you should bring in a set of new testers who don’t know the games mechanics and see how they react. You should also use your last group to get their reaction on the changes that have been made. Both sets of groups can give you unique and useful feedback for improvement. These people should mostly consist of your target market, but its not a bad idea to get people in from outside your target market. They can often provide some valuable insights into how you may be able to gain interest from other markets, or look at your product in a different light to what others can.

This makes trade shows a very interesting prospect for the indie developer. Unfortunately I’ve only been to one – E3, which is hardly a place for the indie dev – but what I’ve heard about the critical feedback from players at PAX and GDC has been outstanding. Just being able to watch someone play your product, see their facial expressions and physical reactions can tell you so much more about your game. Trade shows attract tens of thousands of people – developers, journalists and gamers – and even if only a small fraction get to see your game, the perspective that it can offer is invaluable.

Valve have done several excellent presentations and articles about the testing process for their games, especially Portal. They highlight the importance of being able to observe your testers rather than relying on written or vocal feedback. Playtesting for Portal commenced as soon as the gun was in-game and shooting portals right through to re-development of the final boss fight 3 or 4 times. The developers were able to identify some key issues in all stages of testing primarily through observation.

For those people working with a client, these processes, while solid from an engineering and design perspective, can often be at odds with client expectations. Working with clients or managers who don’t understand the importance of a solid methodology and testing regime can be difficult and often harm the end result and quality of the product. Making changes late in development is costly or expensive and, in the end, may result in either releasing a sub-par product or blowing your deadlines and budget.

While some people might push you, try to never waiver on your development methodologies. It can be extremely hard, especially when money or desperation starts to get involved, but don’t get pushed around. If possible, get people to test your game in real life. You can gather a lot of data about where users trip up, become confused or enjoy simply by watching them and taking notes. Questionnaires once a player has finished testing the game can be great supporting material for anything you missed. Bring these players back once you resolve issues to see their reactions once you resolve specific issues they were having.

Minecraft demonstrated that games, with a bit luck, can occasionally succeed by using the public as its test platform. However, many more games – Angry Birds and Team Fortress 2 come to mind – better demonstrate that the importance is in your core mechanics and quality. You must get these right as early as possible, otherwise you may be dooming yourself to failure.

Leave a Reply