Games QA: A testing road ahead

[This feature was published in the May 2013 edition of Develop magazine, which is available through your browser and on iPad.]

Testing is something that every games should receive. But look on any retail website or app store and it’s all too clear from customer reviews that this isn’t the case.

Quality assurance firms strive to be thorough in their mission to give games the polish they need. They rid games of hurtful bugs and provide game studios with guidance to fix the holes that are holding their game back. They are the discreet lab coat-wearing assistants who ‘test ’em to destruction’ – only, without the lab coats.

Following our report on games localisation (Develop #137), which many of the same firms in this article also handle, we turn our attention to QA. With so many new games platforms, the sector is continuing to experience rapid change. The next generation is just starting to awaken, but are external QA firms excited by the new consoles and the triple-A projects they will herald, or do they see mobile as the source of their fortunes going forward?

Quality matters

First things first, however. In the past, QA has typically been viewed as the ‘unglamorous’ end of games development. Some publishers have boldly – or perhaps foolishly – decided that its not worthy of devoting time to, only for their product to feel the impact.

Testronic Labs VP of games operations and UK site manager Alastair Harsant says that things are different now: “QA has certainly matured as a service and has become an increasingly important and recognised part of the development cycle. With more social, online and mobile titles, there is an increased focus on customer retention rates as a measure of success. It is more important than ever to launch a perfect and bug-free title.”

QA firms have themselves been put to the test in new ways over the last 18 months. Unsurprisingly perhaps, it is mobile development that has proliferated many of the changes these firms face.

“Developers are focusing more on mobile quality assurance,” confirms José Ramón Sagarna, head of testing at Triple A Testing. “Not only with end of development testing, but also with a full quality assurance plan usually reserved for main console titles.”

Meanwhile, it’s the rapid uptake of social games that Babel CEO Richard Leinfellner feels has affected the business most in terms of what is now expected of QA companies.

“Console and casual have been slugging it out, with console going for very few triple-A megahits, and lots of sequels as few are taking risks. QA for these is the same, but bigger than before, especially due to multiplayer being so important,” he says.

“Social is almost the total opposite; we see everything from tiny to big budget, lots of new IP, lots of languages, tons of handsets, it’s really like the wild west. It’s really fragmented with some jobs lasting literally two hours and others more like TV soaps, which just go on and on as new content comes in on a weekly basis.

“It’s really like movies – or console – versus TV – or casual – in many ways, so we have a lot to learn from these media sources and many movie people now need to learn the cut throat world of TV.”

Tailor-fitted

Leinfellner’s comments echo that of some of his fellow QA practitioners. Based on what several firms have said, there seems to have been a marked reduction in the number of console projects that have been submitted for external testing. But this reduction has been countered by higher expectations from clients across alternative platforms and a need for more rigours testing as mobile and social titles become ever more ambitious.

This progression can be seen in the likes of the Infinity Blade series, Galaxy on Fire 2 and CSR Racing, all of them graphically-accomplished 3D games that have been created for smartphones and tablet computers. And as the number of relevant genres expands, so too has the feature set for such games.

It’s not uncommon to see online multiplayer, be it asynchronous or real-time, and other additional modes appearing in mobile titles from the get-go. And announcements by companies like Havok at GDC last month to release free tools for 3D mobile development means that we may see even more developers choosing to realise their visions in the third dimension.

What this has meant for QA is new relationships with developers and teams of varying sizes. Dealing with new clients that don’t have huge budgets to spend has led QA outfits to adapt their own services to suit specific needs, as Triple A Testing’s Sagarna explains: “Now, small developers don’t need a huge budget to develop. New platforms brought new developing and testing ways, with constant updates and the need to reinvent certain methods or procedures. It is important to be flexible and have an open mind to find a common ground between the project needs and the project costs.”

This is something that chimes with Ben Wibberley, director of games at QA specialist VMC Game Labs.

“Established models are being challenged, which is blurring the lines between the traditional roles of development, publishing and distribution. Whilst we don’t subscribe to the idea of the death of the publisher, the games business is moving from a business-to-business to a business-to-consumer model,” Wibberley tells Develop.

“New games companies and devices appear almost every week, and the business models are constantly evolving, so the key is being agile and continuing to stay on the leading edge of understanding our clients’ needs.

“For example, we’ve successfully created an independent games client programme to specifically support indie games developers and ensure the high level of service that we expect ourselves to provide to all of our games clients.”

The same is true for Testology, says project manager Harrison Baker.

“When we began to consider the opportunities with mobile platforms, it was difficult to ignore the requirements of the mobile developers,” he says.

“We wanted to ensure that our services were tailored to all client requirements – no matter the platform, or industry or team size. We wanted to be able to provide the very best QA services with every type of budget so that these companies could release applications with impressive stability and functionality.”

Evidently, QA is not longer about ‘one size fits all’, but a tailored approach to clients’ needs, whatever their situation.

Automated testing

Just how QA firms are performing their task is also evolving in terms of services they have and the technology they use.

Along with VMC’s indie programme, Testronic Labs has enlisted Player Research to bolster its UX testing, and has opened a special UX lab at its Burbank office, while Babel has launched a dedicated app testing and localisation service in TestMyApps. These services are intended to improve the flexibility and specialism on offer.

Testronic’s Harsant says its projects have longer tails, and over time they have been handling more complex projects, which has often necessitated scheduling changes. In conjunction with this, and likely as a result, he says there has been more interest in the company’s automated testing capabilities.

“Interest in automation is also increasing,” he says, though he doesn’t feel the technology is ready to replace human testers just yet. “While we have expertise in this area, automation remains a valuable aid to certain testing, but not a solution to all procedures. The human element remains prevalent, and where appropriate we provide on-site testing operatives for some major clients.”

Automated testing generally refers to code or data that is used to verify the functionality of portions of a game without any user interaction. It’s an area that Wibberley says VMC is doing a lot of work in with some
high-profile clients.

“Test automation is often sold as a ‘silver bullet’ to solve all manner of software development problems. However, we view test automation as a tool used by talented, creative people to support the development of quality games. Repetitive tasks are automated to free up valuable developer or tester time for more critical tasks.

“Ideas about effective offshoring or outsourcing are standard practice now and that will not go away, but more bodies isn’t always better. The most agile organisations are moving towards tools and test automation at pace.”

On to the next one

Automation is certainly something that will continue in step with the demands of the coming generation of games consoles. But with the rise of mobile giving QA firms a new stream of clients, the significance of console development to their businesses is in dispute.

“Our business has seen a significant percentage increase in favour of mobile platforms,” offers Testology’s Baker.

“We will anticipate Testology being heavily involved in next generation games console testing, but to what extent will be determined by demand. In the past two years we have seen a significant transition in outsourcing requirements, and the console market seems less in need of external QA services.”

A gradual move toward in-house QA teams by major publishers (See ‘Insider advantage’ below) would certainly corroborate Baker’s view.

Babel’s Leinfellner also has reservations: “Right now, it’s hard to tell where next-gen will go. The hardware is exciting and the potential for content is too. However, the development costs are also scaring developers and publishers.

“Personally, I am a little concerned that only the big guys will be able to afford it and none of them will take any risks, so expect sequels galore. That aside, it appears both Sony and Microsoft are choosing a PC architecture. This clearly will help developers make the most of their technology investment, and since we already know how to test in the connected and social PC world, it will put us in a great position.”

“With the current disruption in console, there are definitely less projects around and, of course, at this stage of the console cycle this is to be expected. We are experts in testing on all platforms and want to maintain a mixed portfolio of mobile, online and console, but maintaining that balance can be a challenge,” admits Anna Wojewodzka, operations manager of Universally Speaking.

By contrast, others have a more pragmatic opinion. Triple A Testing’s Sagarna says its a fact that consoles are “here to stay”, so training and familiarising themselves with the new architectures will be a recurring challenge. Harsant says the next-gen consoles are “very important” to Testronic’s business, while VMC’s Wibberley takes the view that “as long as there is a market and our clients are making games for any device, we will provide test support on that device”.

Rupert Young, managing director of EC-Interactive has taken stock from the current generation and feels that is “hard to envision any hardware completely reinventing the way we do QA or fundamentally changing what is expected from us as a QA provider”.

It remains to be seen whether traditional console publishers and developers will return in significant numbers to external QA firms.

But its Young’s final thought that holds true for this sector: “By far the greatest challenge we face, when we’re focusing so closely on our existing set of platforms, is to make sure we don’t lose sight or get left behind when the next big platform or QA service opportunity surfaces. Making sure our staff are continually retrained and deciding on what equipment to source is a never-ending challenge, but one that we relish.”

INSIDER ADVANTAGE

Part of the challenge for external QA firms comes from publishers own internal teams.

Reflections lead QA Joseph Rogers says there are many advantages to internal QA teams. The internal QA structure has been part of his studio for almost two decade now, so as well as experience, the team’s proximity to Ubisoft’s development team and its earlier involvement in projects means it can maintain a competitive advantage in terms of quality delivered and overall cost.

“Although in-house QA on some scale can work for all companies, it’s obviously easier for larger companies. Having a large QA structure means you have constant and continuous production feeding and supporting this structure,” he suggests, but admits there are weaknesses: “The biggest challenge is to manage change inside the structure. The benefits coming from experience and the multiple iterations of certain approaches can cause some bumps on the way to change – and there’s no doubt our industry changes continually.”