HOME

TheInfoList



OR:

Game testing, also called quality assurance (QA) testing within the
video game industry The video game industry is the tertiary industry, tertiary and quaternary industry, quaternary sectors of the entertainment industry that specialize in the video game development, development, marketing, distribution (marketing), distribution, ...
, is a
software testing Software testing is the act of checking whether software satisfies expectations. Software testing can provide objective, independent information about the Quality (business), quality of software and the risk of its failure to a User (computin ...
process for
quality control Quality control (QC) is a process by which entities review the quality of all factors involved in production. ISO 9000 defines quality control as "a part of quality management focused on fulfilling quality requirements". This approach plac ...
of
video game A video game or computer game is an electronic game that involves interaction with a user interface or input device (such as a joystick, game controller, controller, computer keyboard, keyboard, or motion sensing device) to generate visual fe ...
s. Moore, Novak 2010, p. 95 Oxland 2004, p. 301-302 The primary function of game testing is the discovery and documentation of software defects. Interactive entertainment software testing is a highly technical field requiring
computing Computing is any goal-oriented activity requiring, benefiting from, or creating computer, computing machinery. It includes the study and experimentation of algorithmic processes, and the development of both computer hardware, hardware and softw ...
expertise, analytic competence, critical evaluation skills, and endurance. Oxland 2004, p. 301 In recent years the field of game testing has come under fire for being extremely strenuous and unrewarding, both financially and emotionally.


History

In the early days of computer and video games, the developer was in charge of all the testing. No more than one or two testers were required due to the limited scope of the games. In some cases, the programmers could handle all the testing. As games become more complex, a larger pool of QA resources, called "Quality Assessment" or "
Quality Assurance Quality assurance (QA) is the term used in both manufacturing and service industries to describe the systematic efforts taken to assure that the product(s) delivered to customer(s) meet with the contractual and other agreed upon performance, design ...
" is necessary. Most
publishers Publishing is the activities of making information, literature, music, software, and other content, physical or digital, available to the public for sale or free of charge. Traditionally, the term publishing refers to the creation and distribu ...
employ a large QA staff for testing various games from different developers. Despite the large QA infrastructure most publishers have, many developers retain a small group of testers to provide on-the-spot QA. Now most game developers rely on their highly technical and game savvy testers to find glitches and 'bugs' in either the programming code or graphic layers. Game testers usually have a background playing a variety of different games on a multitude of platforms. They must be able to notate and reference any problems they find in detailed reports, meet deadlines with assignments and have the skill level to complete the game titles on their most difficult settings. Most of the time the position of game tester is a highly stressful and competitive position with little pay yet is highly sought after for it serves as a doorway into the industry. Game testers are observant individuals and can spot minor defects in the game build. A common misconception is that all game testers enjoy alpha or beta version of the game and report occasionally found bugs. In contrast, game testing is highly focused on finding bugs using established and often tedious methodologies before alpha version.


Overview

Quality assurance is a critical component in game development, though the video game industry does not have a standard methodology. Instead developers and publishers have their own methods. Small developers do not generally have QA staff; however, large companies may employ QA teams full-time. High-profile commercial games are professionally and efficiently tested by publisher QA department. Testing starts as soon as first code is written and increases as the game progresses towards completion. Bethke 2003, p. 53 The main QA team will monitor the game from its first submission to QA until as late as post-production. Early in the game development process the testing team is small and focuses on daily feedback for new code. As the game approaches ''alpha'' stage, more team members are employed and test plans are written. Sometimes features that are not bugs are reported as bugs and sometimes the programming team fails to fix issues first time around. Bates 2004, p. 177 A good bug-reporting system may help the programmers work efficiently. As the projects enters ''beta'' stage, the testing team will have clear assignments for each day. Tester feedback may determine final decisions of exclusion or inclusion of final features. Introducing testers with fresh perspectives may help identify new bugs. Bates 2004, p. 178 At this point the lead tester communicates with the producer and department heads daily. Bates 2004, p. 179 If the developer has an external publisher, then coordination with publisher's QA team starts. For console games, a build for the console company QA team is sent. Beta testing may involve volunteers, for example, if the game is multiplayer. Testers receive scheduled uniquely identifiable game builds from the developers. The game is play-tested and testers note any uncovered errors. These may range from bugs to art glitches to logic errors and
level Level or levels may refer to: Engineering *Level (optical instrument), a device used to measure true horizontal or relative heights * Spirit level or bubble level, an instrument designed to indicate whether a surface is horizontal or vertical *C ...
bugs. Testing requires creative gameplay to discover often subtle bugs. Some bugs are easy to document, but many require detailed description so a developer can replicate or find the bug. Testers implement
concurrency control In information technology and computer science, especially in the fields of computer programming, operating systems, multiprocessors, and databases, concurrency control ensures that correct results for concurrent operations are generated, whil ...
to avoid logging bugs multiple times. Many video game companies separate technical requirement testing from functionality testing altogether since a different testing skillset is required. If a video game development enters crunch time before a deadline, the game-test team is required to test late-added features and content without delay. During this period staff from other departments may contribute to the testing—especially in multiplayer games. One example of sustained crunch, especially among the QA team, was at
Treyarch Treyarch Corporation ( ; formerly Treyarch Invention LLC) is an American video game developer based in Los Angeles with Additional studio locations in Vancouver, Canada; and Austin, Texas. Founded in 1996 by Peter Akemann and Doğan Köslü, it ...
during the development of '' Call of Duty: Black Ops 4''. Most companies rank bugs according to an estimate of their severity: Bates 2004, pp. 178-179 * ''A bugs'' are critical bugs that prevent the game from being shipped, for example, they may crash the game. * ''B bugs'' are essential problems that require attention; however, the game may still be playable. Multiple B bugs are equally severe to an A bug. * ''C bugs'' are small and obscure problems, often in form of recommendation rather than bugs.


Game tester

A game tester is a member of a development team who performs game testing.


Roles

The organization of staff differs between organizations; a typical company may employ the following roles associated with testing disciplines: * ''
Game producer A video game producer is the top person in charge of overseeing development of a video game. History The earliest documented use of the term ''producer'' in games was by Trip Hawkins, who established the position when he founded Electronic Ar ...
s'' are responsible for setting testing deadlines in coordination with marketing and quality assurance. They also manage many items outside of game testing, relating to the overall production of a title. Their approval is typically required for final submission or "
gold Gold is a chemical element; it has chemical symbol Au (from Latin ) and atomic number 79. In its pure form, it is a brightness, bright, slightly orange-yellow, dense, soft, malleable, and ductile metal. Chemically, gold is a transition metal ...
" status. * ''Lead tester'', ''test lead'' or ''QA lead'' Bethke 2003, p. 52 is the person responsible for the game working correctly and managing bug lists. A lead tester manages the QA staff. The lead tester works closely with designers and programmers, especially towards the end of the project. The lead tester is responsible for tracking bug reports and ensuring that they are fixed. They are also responsible that QA teams produce formal and complete reports. This includes discarding duplicate and erroneous bug reports, as well as requesting clarifications. As the game nears alpha and beta stages, lead tester brings more testers into the team, coordinates with external testing teams and works with management and producers. Some companies may prevent the game going gold until lead tester approves it. Lead testers are also typically responsible for compiling representative samples of game footage for submission to regulatory bodies such as the
ESRB The Entertainment Software Rating Board (ESRB) is a self-regulatory organization that assigns age and content ratings to consumer video games in Canada, the United States, and Mexico. The ESRB was established in 1994 by the Entertainment Softw ...
and
PEGI PEGI ( ), short for Pan-European Game Information, is a European video game content rating system established to help European consumers make informed decisions when buying video games or apps through the use of age recommendations and content ...
. * ''Testers'' are responsible for checking that the game works, is easy to use, has actions that make sense, and contains fun gameplay. Testers need to write accurate and specific bug reports, and if possible providing descriptions of how the bug can be reproduced. Testers may be assigned to a single game during its entire production, or brought onto other projects as demanded by the department's schedule and specific needs. * ''SDET (Software Development Engineer in Test) or Technical Testers'' are responsible for building automated test cases and frameworks as well as managing complex test problems such as overall game performance and security. These individuals usually have strong software development skills but with a focus on writing software which exposes defects in other applications. Specific roles and duties will vary between studios. Many games are developed without any Technical Testers.


Employment

Game QA is less technical than general software QA. Game testers most often require experience however occasionally only a
high school A secondary school, high school, or senior school, is an institution that provides secondary education. Some secondary schools provide both ''lower secondary education'' (ages 11 to 14) and ''upper secondary education'' (ages 14 to 18), i.e., ...
diploma and with no technical expertise, suffice. Game testing is normally a full-time job for experienced testers; however, many employees are hired as temporary staff, such as
beta Beta (, ; uppercase , lowercase , or cursive ; or ) is the second letter of the Greek alphabet. In the system of Greek numerals, it has a value of 2. In Ancient Greek, beta represented the voiced bilabial plosive . In Modern Greek, it represe ...
testers. In some cases, testers employed by a publisher may be sent to work at the developer's site. The most aggressive recruiting season is late summer/early autumn, as this is the start of the crunch period for games to be finished and shipped in time for the holiday season. Some games studios are starting to take a more technical approach to game QA that is more inline with traditional software testing. Technical Test positions are still fairly rare throughout the industry but these jobs are often full-time positions with long term career paths and require a 4-year computer science degree and significant experience with test automation. Some testers use the job as a stepping stone in the game industry. Bates 2004, p. 261 QA résumés, which display non-technical skill sets, tend towards management, than to marketing or production. Applicants for programming,
art Art is a diverse range of cultural activity centered around ''works'' utilizing creative or imaginative talents, which are expected to evoke a worthwhile experience, generally through an expression of emotional power, conceptual ideas, tec ...
, or
design A design is the concept or proposal for an object, process, or system. The word ''design'' refers to something that is or has been intentionally created by a thinking agent, and is sometimes used to refer to the inherent nature of something ...
positions need to demonstrate technical skills in these areas.


Compensation

Game testing personnel are usually paid hourly (around
US$ The United States dollar (Currency symbol, symbol: Dollar sign, $; ISO 4217, currency code: USD) is the official currency of the United States and International use of the U.S. dollar, several other countries. The Coinage Act of 1792 introdu ...
10–12 an hour). Testing management is usually more lucrative, and requires experience and often a college education. An annual survey found that testers earn an average of $39k annually. Testers with less than three years' experience earn an average of US$25k while testers with over three years' experience earn US$43k. Testing leads, with over six years' experience, earn on an average of US$71k a year.


Process

A typical bug report progression of testing process is seen below: * ''Identification''. Incorrect program behavior is analyzed and identified as a bug. * ''Reporting''. The bug is reported to the developers using a defect tracking system. The circumstances of the bug and steps to reproduce are included in the report. Developers may request additional documentation such as a real-time video of the bug's manifestation. * ''Analysis''. The developer responsible for the bug, such as an artist, programmer or
game designer Game design is the process of creating and shaping the mechanics, systems, rules, and gameplay of a game. Game design processes apply to board games, card games, dice games, casino games, role-playing games, sports, Wargame (video games), war ga ...
checks the malfunction. This is outside the scope of game tester duties, although inconsistencies in the report may require more information or evidence from the tester. * ''Verification''. After the developer fixes the issue, the tester verifies that the bug no longer occurs. Not all bugs are addressed by the developer, for example, some bugs may be claimed as features (expressed as "NAB" or "not a bug"), and may also be "waived" (given permission to be ignored) by producers,
game designer Game design is the process of creating and shaping the mechanics, systems, rules, and gameplay of a game. Game design processes apply to board games, card games, dice games, casino games, role-playing games, sports, Wargame (video games), war ga ...
s, or even lead testers, according to company policy.


Methodology

There is no standard method for game testing, and most methodologies are developed by individual
video game developer A video game developer is a software developer specializing in video game development – the process and related disciplines of creating video games. A game developer can range from one person who undertakes all tasks to a large business with em ...
s and
publisher Publishing is the activities of making information, literature, music, software, and other content, physical or digital, available to the public for sale or free of charge. Traditionally, the term publishing refers to the creation and distribu ...
s. Methodologies are continuously refined and may differ for different types of games (for example, the methodology for testing an
MMORPG A massively multiplayer online role-playing game (MMORPG) is a video game that combines aspects of a role-playing video game and a massively multiplayer online game. As in role-playing games (RPGs), the player assumes the role of a Player charac ...
will be different from testing a
casual game A casual game is a video game targeted at a mass market audience, as opposed to a hardcore game, which is targeted at hobbyist gamers. Casual games may exhibit any type of gameplay and genre. They generally involve simpler rules, shorter sessio ...
). Many methods, such as
unit testing Unit testing, component or module testing, is a form of software testing by which isolated source code is tested to validate expected behavior. Unit testing describes tests that are run at the unit-level to contrast testing at the Integration ...
, are borrowed directly from general software testing techniques. Outlined below are the most important methodologies, specific to video games. * Functionality testing is most commonly associated with the phrase "game testing", as it entails playing the game in some form. Functionality testing does not require extensive technical knowledge. Functionality testers look for general problems within the game itself or its user interface, such as stability issues, game mechanic issues, and game asset integrity. * Compliance testing is the reason for the existence of game testing labs. First-party licensors for console platforms have strict technical requirements titles licensed for their platforms. For example,
Sony is a Japanese multinational conglomerate (company), conglomerate headquartered at Sony City in Minato, Tokyo, Japan. The Sony Group encompasses various businesses, including Sony Corporation (electronics), Sony Semiconductor Solutions (i ...
publishes a ''Technical Requirements Checklist'' (TRC),
Microsoft Microsoft Corporation is an American multinational corporation and technology company, technology conglomerate headquartered in Redmond, Washington. Founded in 1975, the company became influential in the History of personal computers#The ear ...
publishes ''Xbox Requirements'' (XR), and
Nintendo is a Japanese Multinational corporation, multinational video game company headquartered in Kyoto. It develops, publishes, and releases both video games and video game consoles. The history of Nintendo began when craftsman Fusajiro Yamauchi ...
publishes a set of "guidelines" (Lotcheck). Some of these requirements are highly technical and fall outside the scope of game testing. Other parts, most notably the formatting of standard error messages, handling of memory card data, and handling of legally
trademark A trademark (also written trade mark or trade-mark) is a form of intellectual property that consists of a word, phrase, symbol, design, or a combination that identifies a Good (economics and accounting), product or Service (economics), service f ...
ed and
copyright A copyright is a type of intellectual property that gives its owner the exclusive legal right to copy, distribute, adapt, display, and perform a creative work, usually for a limited time. The creative work may be in a literary, artistic, ...
ed material, are the responsibility of the game testers. Even a single violation in submission for license approval may have the game rejected, possibly incurring additional costs in further testing and resubmission. In addition, the delay may cause the title to miss an important launch window, potentially costing the publisher even larger sums of money. :The requirements are proprietary documents released to developers and publishers under confidentiality agreements. They are not available for the general public to review, although familiarity with these standards is considered a valuable skill to have as a tester. :Compliance may also refer to regulatory bodies such as the
ESRB The Entertainment Software Rating Board (ESRB) is a self-regulatory organization that assigns age and content ratings to consumer video games in Canada, the United States, and Mexico. The ESRB was established in 1994 by the Entertainment Softw ...
and
PEGI PEGI ( ), short for Pan-European Game Information, is a European video game content rating system established to help European consumers make informed decisions when buying video games or apps through the use of age recommendations and content ...
, if the game targets a particular content rating. Testers must report objectionable content that may be inappropriate for the desired rating. Similar to licensing, games that do not receive the desired rating must be re-edited, retested, and resubmitted at additional cost. * Compatibility testing is normally required for PC titles, nearing the end of development as much of the compatibility depends on the final build of the game. Often two rounds of compatibility tests are done - early in
beta Beta (, ; uppercase , lowercase , or cursive ; or ) is the second letter of the Greek alphabet. In the system of Greek numerals, it has a value of 2. In Ancient Greek, beta represented the voiced bilabial plosive . In Modern Greek, it represe ...
to allow time for issue resolution, and late in beta or during
release candidate The software release life cycle is the process of developing, testing, and distributing a software product (e.g., an operating system). It typically consists of several stages, such as pre-alpha, alpha, beta, and release candidate, before the fi ...
. Compatibility testing team test major functionality of the game on various configurations of hardware. Usually a list of commercially important hardware is supplied by the publisher. :Compatibility testing ensures that the game runs on different configurations of hardware and
software Software consists of computer programs that instruct the Execution (computing), execution of a computer. Software also includes design documents and specifications. The history of software is closely tied to the development of digital comput ...
. The hardware encompasses brands of different manufacturers and assorted input peripherals such as
gamepad A gamepad is a type of video game controller held in two hands, where the fingers (especially thumbs) are used to provide input. They are typically the main input device for video game consoles. Features Some common additions to the standar ...
s and
joystick A joystick, sometimes called a flight stick, is an input device consisting of a stick that pivots on a base and reports its angle or direction to the device it is controlling. Also known as the control column, it is the principal control devic ...
s. :The testers also evaluate performance and results are used for game's advertised minimum
system requirements To be used efficiently, all computer software needs certain hardware components or other software resources to be present on a computer. These prerequisites are known as (computer) system requirements and are often used as a guideline as opposed ...
. Compatibility or performance issues may be either fixed by the developer or, in case of legacy hardware and software, support may be dropped. * Localization testing act as in-game text
editors Editing is the process of selecting and preparing written, visual, audible, or cinematic material used by a person or an entity to convey a message or information. The editing process can involve correction, condensation, organization, a ...
. Although general text issues are a part of functionality testing, QA departments may employ dedicated localization testers. In particular, early Japanese game translations were rife with errors, and in recent years localization testers are employed to make technical corrections and review translation work of game scripts Adams, Rollings 2003, p. 17 - catalogued collections of all the in-game text. Testers native to the region where a game is marketed may be employed to ensure the accuracy and quality of a game's localization. * Soak testing, in the context of video games, involves leaving the game running for prolonged periods time in various modes of operation, such as idling, paused, or at the title screen. This testing requires no user interaction beyond initial setup, and is usually managed by lead testers. Automated tools may be used for simulating repetitive actions, such as mouse clicks. Soaking can detect
memory leak In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in a way that memory which is no longer needed is not released. A memory leak may also happen when an objec ...
s or
rounding error In computing, a roundoff error, also called rounding error, is the difference between the result produced by a given algorithm using exact arithmetic and the result produced by the same algorithm using finite-precision, rounded arithmetic. Roun ...
s that manifest only over time. Soak tests are one of the compliance requirements. *
Beta testing Software testing is the act of checking whether software satisfies expectations. Software testing can provide objective, independent information about the quality of software and the risk of its failure to a user or sponsor. Software test ...
is done during
beta Beta (, ; uppercase , lowercase , or cursive ; or ) is the second letter of the Greek alphabet. In the system of Greek numerals, it has a value of 2. In Ancient Greek, beta represented the voiced bilabial plosive . In Modern Greek, it represe ...
stage of development. Often this refers to the first publicly available version of a game. Public betas are effective because thousands of fans may find bugs that the developer's testers did not. *
Regression testing Regression testing (rarely, ''non-regression testing'') is re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change. If not, that would be called a '' regr ...
is performed once a bug has been fixed by the
programmers A programmer, computer programmer or coder is an author of computer source code someone with skill in computer programming. The professional titles ''software developer'' and ''software engineer'' are used for jobs that require a program ...
. QA checks to see whether the bug is still there (regression) and then runs similar tests to see whether the fix broke something else. That second stage is often called "halo testing"; it involves testing all around a bug, looking for other bugs. * Load testing tests the limits of a system, such as the number of players on an MMO server, the number of sprites active on the screen, or the number of threads running in a particular program. Load testing may require a large group of testers or software that emulates heavy activity. Load testing also measures the capability of an application to function correctly under load. * Multiplayer testing may involve separate multiplayer QA team if the game has significant multiplayer portions. This testing is more common with PC games. The testers ensure that all connectivity methods (modem, LAN, Internet) are working. This allows single player and multiplayer testing to occur in parallel. * Player-experience modeling refers to attempts to mathematically model player experience and predict player's preference for or liking of a
video game A video game or computer game is an electronic game that involves interaction with a user interface or input device (such as a joystick, game controller, controller, computer keyboard, keyboard, or motion sensing device) to generate visual fe ...
.


Console hardware

For consoles, the majority of testing is not performed on a normal system or ''consumer unit''. Special test equipment is provided to developers and publishers. The most significant tools are the ''test'' or ''debug'' kits, and the ''dev'' kits. The main difference from consumer units is the ability to load games from a burned disc, USB stick, or hard drive. The console can also be set to any publishing region. This allows game developers to produce copies for testing. This functionality is not present in consumer units to combat
software piracy Online piracy or software piracy is the practice of downloading and distributing copyrighted works digitally without permission, such as music, movies or software. History Nathan Fisk traces the origins of modern online piracy back to similar ...
and grey-market imports. * ''Test kits'' have the same hardware specifications and overall appearance as a consumer unit, though often with additional
ports Ports collections (or ports trees, or just ports) are the sets of makefiles and Patch (Unix), patches provided by the BSD-based operating systems, FreeBSD, NetBSD, and OpenBSD, as a simple method of installing software or creating binary packages. T ...
and connectors for other testing equipment. Test kits contain additional options, such as running automated compliance checks, especially with regard to save data. The system software also allows the user to capture memory dumps for aid in debugging. * ''Dev kits'' are not normally used by game testers, but are used by programmers for lower-level testing. In addition to the features of a test kit, dev kits usually have higher hardware specifications, most notably increased system memory. This allows developers to estimate early game performance without worrying about optimizations. Dev kits are usually larger and look different from a test kit or consumer unit.


See also

*
Software testing Software testing is the act of checking whether software satisfies expectations. Software testing can provide objective, independent information about the Quality (business), quality of software and the risk of its failure to a User (computin ...
*
Test plan A test plan is a document detailing the objectives, resources, and processes for a specific test session for a software or hardware product. The plan typically contains a detailed understanding of the eventual workflow. Test plans A test plan docu ...
*
Game development game development (sometimes shortened to gamedev) is the process of creating a video game. It is a multidisciplinary practice, involving programming, design, art, audio, user interface, and writing. Each of those may be made up of more specialize ...
*
Software release life cycle The software release life cycle is the process of developing, testing, and distributing a software product (e.g., an operating system). It typically consists of several stages, such as pre-alpha, alpha, beta, and release candidate, before the fi ...


Notes


References

* * * *


Research

* Lahti, M., Game testing in Finnish game companies, Master's thesis, Aalto University, School of Science, 2014
Thesis


External links


Article: The Basics of Test Automation for Apps, Games And the Mobile Web

Article: Architecture and Infrastructure Aspects of Mobile Game Testing
{{DEFAULTSORT:Game Testing Video game development Software testing Video game industry labor disputes