Testing times for broken games

When some of the biggest games of 2014 arrived on shelves, it was not their groundbreaking features nor their sales success that had the industry talking. Instead, it was corrupt and freakish character models, or highly cumbersome, nigh on inoperable online functionality.

The problems experienced in Assassin’s Creed: Unity, Halo: The Master Chief Collection, Driveclub and more called into question the quality standards to which the development community holds itself. Even when the dust had finally settled, issues around the recent release of Xbox indie title Ori and the Blind Forest brought the debate around launch quality of today’s games back to the fore.

Develop spoke to QA experts from around the world to ask the most pressing question: why are there so many broken games?

“Testers need to be given a fair amount of time to perform their checks, which have to be scheduled into the right moments of the development workflow,” says Adriano Majocchi, lead testing co-ordinator at Synthesis.

“The time window between a fully testable build and the project’s deadline is often squeezed to a minimum. When this happens for large and complex titles, the consequences for the final release are obvious to anyone: functional bugs – known and not known – will undoubtedly slip in.”

Tight deadlines are one of the most common complaints when it comes to QA, as well as budget cuts that often target the testing process first. Of course, the reasons behind these is well documented.

“Launch dates are announced well in advance of completion, and may be tied to marketing campaigns or product tie-ins, so development has to meet those dates,” says Scott Ferguson, senior test manager at VMC.

“Fortunately, the timeline for quality is more flexible now that publishers have the ability to make improvements post-launch.”

Bug Tracker’s Jonathan Villanueva says the fact that most of today’s console games build on established franchise mechanics may be deluding devs into thinking bugs fixed in previous iterations won’t reappear.

“When much of the gameplay is continuously recycled from previous titles, one would think the quality would just improve instead of the other way around,” he says. “But products like these are rushed through the pipelines and limited in resources.”

This is not helped by the more ambitious designs of triple-A games. Majocchi says the growing popularity of open-world games, for example, means comprehensive and thorough testing is “virtually unachievable”.

The time window between a fully testable build and the project’s deadline is often squeezed to a minimum. When this happens for large and complex titles, the consequences for the final release are obvious to anyone.

Adriano Majocchi, Synthesis

Consumers particularly expect quality from both franchises and developers that have long since established themselves.

“When you take a successful series, player expectations on game experience get higher year-on-year,” says Katsuri Rangan, director of operations for QA at Pole To Win. “But enhancing the visuals or gameplay and failing on the quality of the player experience is a failure for the overall product.”

Majocchi adds: “When the culprit is one of the big names in the gaming industry it does somehow put the rest to shame. Personally I think that, in the long-term, the integrity of the gameplay experience could and should create a new area for competition. The top video games should have to excel.”

Broken blockbusters

VMC’s Ferguson points out that the problems suffered by 2014’s triple-A hits aren’t necessary new – they’re just much better publicised.

“Not long ago, a user who had issues would have to purchase auxiliary equipment to capture their experience,” he says. “It required additional cost and effort, so fewer users publicly promoted these issues. Now there’s a Share button at our fingertips.

“Every experience – positive or not – can be easily documented or streamed. Issues become mini-celebrities, people follow conversations about a particular bug. The ease of sharing gives the bug more exposure, but that visibility doesn’t correlate to how well games are tested.”

Universally Speaking’s QA manager James Cubitt agrees, adding: “I like to believe that the industry’s quality standards are much higher, which is why these issues get so much press. They are not the norm.”

When so many titles are lampooned so publicly for their noticeable glitches, calls for accountability multiply – but they aren’t always aimed in the right direction.

“The general perception is that blame falls on QA,” says Rangan. “Sadly in most firms today, they are detection robots, who are only responsible for finding issues and don’t have the authority to resolve them. QA teams might have detected most of the critical issues in these titles, but were refused to be fixed due to launch targets.”

Villanueva adds: “If a game is sold with bugs, the developer or publisher looks to the QA team, whether in-house or outsourced. If outsourced, the QA companies are blamed for a poor quality of service. However, they tend to forget the rush in which the projects are often tested, the limitations of deadlines and budget cuts, and the constant rotation and changes in testing team sizes.”

Sharif Sakr, business development director at GameBench, says consumers are becoming more savvy about the industry’s processes. They suspect – correctly in some cases – that testing is done “at the absolute last minute” but maybe not understand the commercial pressures behind this.

“In some cases, gamers are also aware this happens because certain titles are trying to push boundaries and max out the capabilities of the new consoles,” he says. “This is fundamentally a good thing, even though it’s no excuse. Assassin’s Creed: Unity might be an example of that: its ambitious AI would have increased development time and reduced testing time, while also lowering frame rates and causing a shaky experience.”

Quality on any scale

This is by no means solely a triple-A issue. Even small or medium-sized teams face problems if their game is flawed.

“Indie developers have a different pressure,” Ferguson says. “They may not have another release coming out next quarter or even next year. With all of their eggs in one basket, they have to make sure their game is as good as it can be.

“Lower-end games also don’t have the same budget scale for rigorous QA. It’s hard to compete with the big releases, so they want to make sure people are talking about the game, not the game’s issues.”

GameBench’s Sakr adds that mobile devs are also subject to this: “If you look at low-end mobile games in the Google Play Store, you’ll see upset reviewers complaining of poor frame rates, excessive battery drain rates, and compatibility issues on their devices.

“If you look at user reviews of games that push mobile devices to their limits, such as XCom: Enemy Within, then you come away with the impression of a disconnect between what gamers expect and what mobile games currently deliver.

“The maturity of the PC market – where everybody understands the connection between hardware and performance – is totally absent in mobile, and better QA is just the beginning of a solution.”

Better testing

There are, of course, new paths testers can take that weren’t available years ago, but these are to be approached with caution. Case in point, Villanueva says, is the rise of automated QA tools.

“While automated tools can to a certain level find a large number of bugs, it is only the human factor involved during gameplay that triggers identification of unique bugs often missed during QA,” he said.

QA specialists are far more optimistic about the use of Steam Early Access and other forms of beta testing to help not only identify faults, but also raise consumer awareness of the battle testers are fighting.

“Early Access titles show the end user how long it takes to test, and fix the smallest of issues,” says Cubitt.

“This insight into the process that can only be good. It is very important the user understands the amount of work and time that is devoted to each and every title that is produced.”

Sakr agrees: “I think Steam’s Early Access program has helped developers deal with bugs and glitches, as have the various beta phases on consoles. That’s why I think external testing services are important for mobile, because the app stores aren’t really set up for beta testing and because hardware fragmentation means testing needs to be conducted on quite a broad scale.”

Lower-end games also don’t have the same budget scale for rigorous QA. It’s hard to compete with the big releases, so they want to make sure people are talking about the game, not the game’s issues.

Scott Ferguson, VMC

Several major developers have decided that the best way to secure more testing time is to delay their games. Rocksteady’s Batman: Arkham Knight and CD Projekt’s The Witcher III have both been pushed back by several months. The latter studio has even told the press that the market is now “afraid of badly polished games”.

Cubitt commends this strategy: “You never know the state of the title is until it comes to the end of the development cycle. A best case will be estimated, but with new engines and game complexity, the time required for testing can soon spiral depending on the number and complexity of the issues being reported.

“A delay is a small disappointment to the end user, but much better than having issues affecting gameplay on release.”

Ferguson argues that video games are one of the few forms of entertainment that can benefit from it: “If the film is delayed, that can have a huge impact on sales. But if a game release is delayed, that often builds buzz and generally results in improved quality. The title can have more financial and critical success despite a delay. It’s really a shift in protocol as companies see the return on investment that comes with extra polish.”

The power of patches

The increasingly connected nature of the new consoles means it is at least possible to fix a broken game after launch. Ubisoft set up a website where users could keep track of the team’s progress as fixes were issued for Assassin’s Creed: Unity.

However, Synthesis’ Majocchi argues the day one patch is “abused” by some developers, while Cubitt is stresses that studios need to become less reliant on them.

“Before this was possible, you had to ensure the release version was virtually bug free,” says Cubitt. “This meant a number of delays could be expected with titles, but the need for release day patches was non-existent.”

Rangan argues that reliance on day one patches is just the tip of the iceberg.

“Modern day games are 1,000 times more complex than games made 25 years ago, but the majority of developers still stick to old school planning strategies,” he says.

“We need a mindset change across our industry. The only standard that we should set should be offering the best possible player experience within costs allocated for production. Everyone owns quality: the producer, the VP, the CEO, the studio head, artists, developers, testers. Anyone and everyone should think about how they can give the player the best possible experience in the budget they planned.”

Villanueva agrees: “We need to raise the bar for quality standards, which is already the first problem. While the industry has evolved incredibly in the last ten years, the level of quality in the products has reduced at the same rate because of the economy-oriented nature of the business.

“QA is not only about making sure the game works, but also about making sure the game is a playable, entertaining and satisfying product.”

QA firms stress that this needs to be addressed now. The high-profile problems of Assassin’s Creed, Halo and Driveclub have left a bitter taste in many gamers’ mouths and mean future releases will be under intense scrutiny.

But greater accountability and higher standards of quality are only the beginning. With new technology on the way, developers need to be able to efficiently deal with traditional bugs and glitches in order to dedicate more time to the unexpected issues that will arise from future platforms.

“Our QA standards aren’t really keeping up with the increasing complexity or immersiveness of the games we’re creating,” says Sakr.

“As PC games push into virtual reality, and as mobile games become more console-like, then not only does QA become technically harder, but it also becomes more critical, so that the player’s sense of ‘being there’ isn’t shattered.

“It’s perhaps analogous to how make-up departments had to keep up as movies started shooting in HD and then 4K – except that I think it’s even more critical in games, because no movie-watching experience can ever be as immersive, or vulnerable, as a good gaming experience.”

—

For all the news straight to your inbox, sign up to the Develop Daily.