Doubts raised on the value of public betas

The development sector’s growing reliance on public betas has made a significant – and often troublesome – impact on the development process, according to a panel of industry experts.

Publishers are more commonly offering the public a chance to play early builds of titles such as Halo 3 and LittleBigPlanet, which in theory provide developers with vast sums of feedback.

However, taking part in a Develop panel discussion, a number of leading QA and localisation firms say open betas can often disrupt the development process if not handled with care.

Babel Media’s Keith Russell said that “using Joe Public for cheap QA is a myth”.

“What you get are hundreds of random messages, none in a useful format for the dev team, many without the steps to reproduce,” he added.

“So what you need is a good QA service that filters all that noise and turns them into correctly written, reproducible bugs for the dev team that can be regressed once fixed.”

Russell concluded that public betas hasn’t made the design, QA and localisation testing process more or less complicated per se.

“Though for high value IP,” he added, “I still wonder if people realise what they are putting out there.”

His reservations were echoed by professionals based at a number of QA and localisation groups.

Localize Direct’s Christoffer Nilsson said open betas have complicated the design process as there now exists an additional layer that must be included in the development process.

He said, “Creating an open beta currently requires an interruption to the development cycle and creates additional work for stakeholders in the process”

Nilsson also urged developers to begin QA and localisation as early as possible.

Meanwhile, Stephanie Deming, of XLOC acknowledged the “incredibly valuable” feedback from open betas, though warned of their significant impact.

“If the full direction of a game is changed, new assets written, new levels produced and new VO recorded pretty late in the development process that can have a very significant impact on both the localisation and the QA process. Text is one thing, but re-recording voiceover can be incredibly expensive, when doing so for many languages.”