Author: by Michael Steer

Who Are XNA Game Quality?

Formed in 2000 prior to the launch of the original Xbox, the XNA Game Quality team, or Xbox Certification team as they were then known, are now one of the teams that make up XNA, Microsoft's global software and services division dedicated to game development technologies.

They are very much a team that works behind the scenes, and not much is really known about them, unless of course you happen to have developed or published a game for Xbox, Xbox 360, Games for Windows or Games for Windows - LIVE. In this article, we will focus on the work the team does on the Xbox platforms, and leave their work on the Games for Windows platforms for discussion in future articles.

The job of the XNA Game Quality team is to help ensure that every game released on the Xbox and Games for Windows platforms meets Microsoft's technical quality standards, and depicts the platforms and their features in the best possible light, whilst minimizing platform security risks.

One thing they are not, and were never intended to be, is a Quality Assurance test organisation, and this is something they are keen to point out. Quality Assurance testing is the job of the game developer and publisher. XNA Game Quality does not assess game design, or how good gameplay is; only how well the game performs technically and how well it supports the platform's features.

No longer confined to just certifying Xbox or Xbox 360 games, today the team offer a much broader range of services to their publishing partners on a global scale. They now certify Games for Windows and Games for Windows - LIVE platform games, thanks to the revival that PC gaming has experienced in the last few years.

As well, a comprehensive training and education program is now offered to publishers, with the ultimate aim of helping them to improve the certification success rates of their games. Training sessions take a number of forms that include XNA Game Quality representatives visiting publishers at their own premises, inviting publishers to their nearest Microsoft campus, and online via the capabilities of Microsoft's LIVE Meeting.

Other services include representation by the team at Microsoft organised events such as XNA Gamefest (http://www.xnagamefest.com), where attendees are treated to a range of talks, presentations, and seminars covering a variety of topics from System Programming to Quality Assurance and Certification. XNA Game Quality also provides whitepapers on many aspects of the certification process, particularly those that pose significant challenges to their partners.

The team is currently made up of two areas of responsibility, Compliance and Functional, both quite different from the other, and both supported by the XNA Services' Mastering Lab. Each area performs a very different function, but equally as important as the other, and without any one of them, the quality of the certification and the services they offer to their partners would be severely compromised.

Mastering Lab

To begin with let's take a closer look at the Mastering Lab and their role in supporting XNA Game Quality. This team is the first to get their hands on the game code after a Publisher submits their game for certification. They are the "Gatekeepers" of the certification process, and no game submissions of any nature are allowed to enter Compliance or Functional testing until the Mastering Lab has processed and conducted their preliminary checks on each submission. Unlike Compliance and Functional, the majority of testing done by the Mastering Lab is automated through the use of tools.

The Mastering Lab has two core roles. Firstly they take the game code supplied by the publisher and check that it is ready to test. This ranges from checking that it boots, making sure that it has the correct images for Xbox LIVE, ensuring that the total number of Achievements and Gamerscore add up, checking that the Xbox LIVE information in the game executables matches for all versions; and ensuring that the submission doesn't contain erroneous files.

The Mastering Lab's secondary role is to process the game files, which comprises a number of different stages. First, they apply the Age Ratings to the game code, which are used by the Xbox 360 Dashboard Family Settings. The Family Settings controls can be set by parents or guardians to block access to content based on game ratings.

They then configure how the game code should interact with other versions of the same game, as well as configure how different games communicate with each other (example: some games are designed to share game save data), and whether or not a game should support the PAL-50 video standard.

The final stage of processing is when the Mastering Lab prepares the game disc images used to manufacture the final disc based product. At the same time, they also prepare the game files that both Compliance and Functional use to test the games. From here the games enter Compliance testing, which we will look at next.

Compliance

To start, let's introduce a few key concepts about certification as a whole that will help to put the following information specifically about Compliance into some context. XNA Game Quality is responsible for testing a large range of submission types that target Xbox 360 disc based products, Xbox LIVE Marketplace or in some cases, both.

These submission types include, but are not limited to boxed Retail Games, Title Updates, Marketplace Game Demos, Marketplace Game Add-ons, Compilation Discs and Xbox LIVE Arcade Games; all of which fall into two primary certification types, namely a Pre-Cert and Final-Cert.

So, what's the difference? Well, without going into too much detail, a Pre-Cert is comparable to a rehearsal for the real thing (the Final-Cert), and acts as a heads-up about the general health of a submission and potential issues that may cause problems when the Final-Cert submission enters Compliance testing. The Final-Cert, as the name suggests is the final version of a submission that is intended for public release, it's the real deal!

Importantly, Pre-Cert testing is an optional service provided to publishers that is only performed on Full Titles and isn't something that is run on other submission types, e.g. game demos. This is simply an added service that allows a publisher to determine the likelihood of their submission successfully completing Compliance testing.

Similarly, XNA Game Quality also offers an Optional Final-Cert service to publishers. This is identical in every way to a standard Final-Cert, except that it must be requested by the publisher prior to Final-Cert. An Optional Final-Cert is tested by both Compliance and Functional.

Compliance is responsible for testing all submissions, so once a game submission has completed all applicable Mastering Lab checks without issue, Compliance will execute a short list of shallow pre-test checks to catch any obvious issues. These checks are very broad, such as checking if the game functions on Xbox LIVE, posting scores to Leaderboards or starting and saving a new game. Together with some brief supported feature checking, this pre-test phase allows Compliance to build a suitable test team and determine which tests apply to a particular submission.

Once a submission has successfully completed the pre-test phase, it can officially enter a Compliance test cycle where it will be subjected to a variety of tests, a collection of which remain the same for all submissions, but some of which are tailored specifically to the platform features that are supported.

This is as good a time as any to introduce the Technical Certification Requirements (TCRs), as these form the foundation of what Compliance is all about and dictate the test approach that is taken. TCRs have existed all the way back to the start of the original Xbox program, albeit in a different form than the current TCRs for the Xbox 360. Each TCR outlines a specific rule, or a small collection of interdependent rules, that must be adhered to in order for a game to be ‘compliant'.

Collectively, these TCRs form a kind of golden-rule-book, and the intention of this rule book is to protect both the user and the platform. From a user perspective, the majority of work they do isn't always obvious, as the primary objective is not to impose on the original vision and functionality of the game, but to maintain consistency and security of the user experience across hardware (Console) and services (Xbox LIVE).

To give a simple example of this, have you ever noticed how, for the vast majority of Xbox 360 games, pressing ‘A' will progress through, and pressing ‘B' will regress through, a menu system? This is something that was once a TCR for the original Xbox some seven years ago, but its influence has persisted and has resulted in a consistent user experience for all Xbox 360 users.

From a developer perspective, there's obviously some legwork that has to go into making a submission compliant with the TCRs, and as such the TCRs are sometimes viewed as an additional hurdle in releasing a game. To clarify, each TCR has been through numerous reviews to ensure that its existence is justified and it avoids restricting game creativity and design wherever possible, while promoting the intended use of a range of system features.

So now that you know what a TCR is, how does Compliance test them? Each TCR has a corresponding Test Case which is currently maintained as a 1:1 relationship for simplicity. The aim of each Test Case is to break the requirement out into its key components and act as an independent script, or set of test steps for testing the TCR.

Some TCRs/Test Cases have a very broad scope, influencing large areas of the game and requiring a large amount of time and attention to insure full and accurate coverage. Others are more specific and less resource intensive, but no less important, and dictate how a game should react when certain events are triggered e.g. when a controller is disconnected or loses power during gameplay.

The submission is assigned to a team of Compliance testers that deal with executing the Test Cases. The team will use a tailored set of Test Cases that match the supported features of the game. For example, if testing a single player game that does not support Leaderboards, there is no need to check Leaderboard support or Game Invitations, so these will be omitted from the test cycle. With that said, there's still a lot to check, and a large proportion of the games maximize the use of platform features, so Compliance can run at a frenetic pace, all the while maintaining test accuracy and coverage.

When all of the Test Cases have been executed and the dust settles, it's time to go back and perform a regression check for all of the issues that were found during testing before finally publishing the Compliance report. In the case of a Pre-Cert, the publisher will receive a Compliance Test Complete report containing all issues that were found, and that will conclude Pre-Cert testing.

It's a different story for Final-Cert submissions, as the results of the Compliance testing will determine the submission progressing into Functional testing, which you'll read about further in this article. Regardless of the outcome, a Final Compliance report is issued upon completion of Final-Cert testing. In a nutshell, the primary responsibility of Compliance is to enforce the TCRs, which are a set of core ideals for the Xbox 360 platform. Before moving on, a final word on TCRs...

The TCRs and Test Cases represent an important tool for publishers and developers; Compliance uses the exact same TCRs and Test Cases internally as those that are published externally to the team's website, the platform's online resource for their publishing and development partners. The aim of this transparency is to promote predictable results.

Functional

So you're now a little more knowledgeable about what Mastering Lab do, and much more familiar with Compliance and the TCRs, but what about the other area within XNA Game Quality: Functional. What exactly do they do?

Well their primary role is to mimic the average gamer, and find those gameplay issues that the likes of you or I might encounter during a typical gaming session. They get to see the games for longer, and also get to play them a lot more than the other teams. Sounds like the perfect job I hear you say...

Functional's methodology for testing games is comprised of two distinctly different but combined approaches; the Functional Test Cases (FTCs), and the Game Specific Test Plan. So to begin with, let's take a closer look at the FTCs, which are a standard set of checks designed by the Functional area. Every game must adhere to them, and their aim is to ensure that a consistent technical quality bar is met by all games released on the Xbox 360 platform.

Similar in nature to Compliance's TCRs and their Test Cases, and in some instances based on particular TCRs and Test Cases that warrant the extra attention, FTCs focus on the end user experience, making sure the games interact in the ways they're intended to with the key platform differentiating features such Achievements, Rich Presence and the in-game Xbox Guide.

Whilst this duplication may be viewed by some as repetition, the principal reason for this doubling of effort is down to the simple fact that Functional have more time and resources to test these preset scenarios far deeper into a game than Compliance. The FTCs also take on a very similar look and format to the TCR Test Cases, including a title, description, remarks, preparation and test steps.

The reason for this is to maintain consistency and familiarity for the user. In addition to being used in-house in the same way as the TCRs and Test Cases, the FTCs are also published to the team's website. Their intention is to make their testing practices and procedures as transparent as possible, and to demonstrate they only have the same guidelines, requirements and tools at their disposal as their publishing and development partners have.

As well as being alike in many ways, the FTCs are a little different from the TCRs and Test Cases in other ways. The FTCs focus on testing known issues and specific user scenarios that resulted in previous games falling foul during the Functional test process. In addition, the FTCs are used by Functional to test both the processed game files, and the retail disc based media, which forms the bulk of their testing, and provides a far more "real-world" test bed.

The FTCs are sorted by category, of which there are currently several submission types and supported features. By categorising the FTCs in this way, it means they can be used in a modular fashion, allowing them to be mixed and matched to tailor the requirements of any game. It's very important to Functional that the FTCs are kept to a manageable number, not only for their own purposes, but for their partners' purposes, too; which is why an average FTC test suite takes relatively little time to complete. It's because of this that Functional conduct regular reviews of the FTCs to ensure their accuracy and efficiency at all times. Only those FTCs that test for serious issues make the final cut, as it's in no one's interests to test scenarios that don't matter.

Now let's look at the Game Specific Test Plan approach used by Functional, which is quite different to the FTCs and uses a common template, comprised of multiple sections. These include Team Setup, Test Direction, Generic Checks, Age Rating Checks and Achievement Checks. The Team Setup is used to map a broad cross section of system and game settings to the size of the assigned test team.

The point of the Test Direction is to establish the overall test goals and strategy for testing the game, and to break those down into test objectives and instructions for each tester. The Generic Checks are a set of general checks tested against every game, by game mode, game type, mission, level, etc., and ensure they very broadly test the really critical features and functionality in all areas of the game.

Age Rating Checks are used to test that the correct age ratings have been signed into the game for each regional rating board (such as PEGI in Europe), and Achievement Checks ensure that each Achievement can be obtained by the user according to its description or criteria for unlocking.

This combined approach of using the FTCs and a Game Specific Test Plan, enables Functional to achieve a massive amount of testing coverage on every game submitted. Not only can they test deep into a game's core story or campaign mode, but also hit a very wide array of options, settings, difficulties and paths in every game mode.

Another very important area of work for Functional is their reporting. Reporting is handled via an internally developed bug tracking database, which allows them to log, track, and share issues amongst any of their internal team members and partners very efficiently. The same database is also used by Compliance, meaning that a consistent look and feel is maintained throughout the reporting process for their partners, regardless of the area issuing the reports.

It also means that reports can very easily be issued to their external partners virtually at the touch of a button, as long as those partners have access to the aforementioned team's website. XNA Game Quality send out reports at the end of testing every day to update their partners with the status of their games.

Whilst games are undergoing testing they receive a daily Functional Interim Report, and then at the end of test they receive a Final Functional report in the case of a Final-Cert, or a Test Complete report in the case of an Optional Final-Cert.

Bugs are reported as a Priority 1, 2 or 3 depending on the seriousness of an issue, which takes into account the nature of a bug, its frequency and likelihood of being seen. In addition, bugs are classified by type, of which the key ones are CFR, ION and SRI. CFRs (Conditions for Resubmission) are the most seriously viewed issues, and these must be fixed before a game is resubmitted.

IONs (Issues of Note) are issues XNA Game Quality doesn't require be fixed, but views seriously enough to highlight to the publisher in case they feel differently about them. SRIs (Standard Reporting Issues) are all other issues they like to make their partners aware of.

Common issues that Functional consider to be very serious include:

Functional, along with the rest of XNA Game Quality, are always reviewing and continually striving to improve their processes and the services they offer to their publishing and development partners. It's because of this that they're keen to stress the importance of XNA Game Quality's email support service, which can be used to submit queries and feedback about any aspect of certification.

Whether it's an enquiry about a TCR, Test Case, or FTC, a question about how seriously XNA Game Quality view a particular issue, or feedback on a recent training visit, they are always happy to hear the voice of their partners and customers. The team always does their best to ensure their partners receive a timely response to their email, and if a member of the team cannot personally help, they will put them in touch with somebody who can help. So keep sending those questions in!

Final Words

Well now, if you've managed to get this far, then we hope you've found this article an informative and interesting read, whilst being an insightful glimpse into the world of XNA Game Quality. The team firmly believes the work they do to help ensure that games that operate on Microsoft's platforms and services comply with certain minimum quality standards, benefits everyone involved from their partners to the end user.

Without the great games and amazing support from their publishing and development partners and customers, the team realizes that the Xbox and Games for Windows platforms wouldn't be what they are today, and as such would like to pass on their sincerest thanks to all of these people.

If you were in any doubt before reading this article about what certification is, and what XNA Game Quality does, then we very much hope that you now have a much better appreciation of these concepts, and the great work the team does.

For more information about Games for Windows Branding and Games for Windows - LIVE platform certification, please stay tuned for future articles on both of these over the coming months.