software testing Software testing is the act of examining the artifacts and the behavior of the software under test by validation and verification. Software testing can also provide an objective, independent view of the software to allow the business to apprecia ...

, test automation is the use of

software Software is a set of computer programs and associated documentation and data. This is in contrast to hardware, from which the system is built and which actually performs the work. At the lowest programming level, executable code consists ...

separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. Test automation is critical for

continuous delivery Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, following a pipeline through a "production-like environment", withou ...

and

continuous testing Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

. There are many approaches to test automation, however below are the general approaches used widely: *

Graphical user interface testing In software engineering, graphical user interface testing is the process of testing a product's graphical user interface (GUI) to ensure it meets its specifications. This is normally done through the use of a variety of test cases. Test case gen ...

. A testing framework that generates

user interface In the industrial design field of human–computer interaction, a user interface (UI) is the space where interactions between humans and machines occur. The goal of this interaction is to allow effective operation and control of the machine f ...

events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct. * API driven testing. A testing framework that uses a programming interface to the application to validate the behaviour under test. Typically API driven testing bypasses application user interface altogether. It can also be testing public (usually) interfaces to classes, modules or libraries are tested with a variety of input arguments to validate that the results that are returned are correct. One way to generate test cases automatically is

model-based testing Model-based testing is an application of model-based design for designing and optionally also executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a system under test (SUT), or ...

through use of a model of the system for test case generation, but research continues into a variety of alternative methodologies for doing so. In some cases, the model-based approach enables non-technical users to create automated business test cases in plain English so that no programming of any kind is needed in order to configure them for multiple operating systems, browsers, and smart devices. What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make. A multi-vocal literature review of 52 practitioner and 26 academic sources found that five main factors to consider in test automation decision are: 1) System Under Test (SUT), 2) the types and numbers of tests, 3) test-tool, 4) human and organizational topics, and 5) cross-cutting factors. The most frequent individual factors identified in the study were: need for regression testing, economic factors, and maturity of SUT. A growing trend in software development is the use of

unit testing In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures&md ...

frameworks such as the

xUnit xUnit is the collective name for several unit testing frameworks that derive their structure and functionality from Smalltalk's SUnit. ''SUnit'', designed by Kent Beck in 1998, was written in a highly structured object-oriented style, which lent ...

frameworks (for example,

JUnit JUnit is a unit testing framework for the Java programming language. JUnit has been important in the development of test-driven development, and is one of a family of unit testing frameworks which is collectively known as xUnit that originated ...

and NUnit) that allow the execution of unit tests to determine whether various sections of the

code In communications and information processing, code is a system of rules to convert information—such as a letter, word, sound, image, or gesture—into another form, sometimes shortened or secret, for communication through a communication ...

are acting as expected under various circumstances.

Test case In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise ...

s describe tests that need to be run on the program to verify that the program runs as expected. Test automation, mostly using unit testing, is a key feature of

extreme programming Extreme programming (XP) is a software development methodology intended to improve software quality and responsiveness to changing customer requirements. As a type of agile software development,"Human Centred Technology Workshop 2006 ", 2006, PD ...

and

agile software development In software development, agile (sometimes written Agile) practices include requirements discovery and solutions improvement through the collaborative effort of self-organizing and cross-functional teams with their customer(s)/ end user(s), ad ...

, where it is known as

test-driven development Test-driven development (TDD) is a software development process relying on software requirements being converted to test cases before software is fully developed, and tracking all software development by repeatedly testing the software against al ...

(TDD) or test-first development. Unit tests can be written to define the functionality ''before'' the code is written. However, these unit tests evolve and are extended as coding progresses, issues are discovered and the code is subjected to refactoring. Only when all the tests for all the demanded features pass is the code considered complete. Proponents argue that it produces software that is both more reliable and less costly than code that is tested by manual exploration. It is considered more reliable because the code coverage is better, and because it is run constantly during development rather than once at the end of a

waterfall A waterfall is a point in a river or stream where water flows over a vertical drop or a series of steep drops. Waterfalls also occur where meltwater drops over the edge of a tabular iceberg or ice shelf. Waterfalls can be formed in several wa ...

development cycle. The developer discovers defects immediately upon making a change, when it is least expensive to fix. Finally,

code refactoring In computer programming and software design, code refactoring is the process of restructuring existing computer code—changing the '' factoring''—without changing its external behavior. Refactoring is intended to improve the design, structur ...

is safer when unit testing is used; transforming the code into a simpler form with less

code duplication In computer programming, duplicate code is a sequence of source code that occurs more than once, either within a program or across different programs owned or maintained by the same entity. Duplicate code is generally considered undesirable for a ...

, but equivalent behavior, is much less likely to introduce new defects when the refactored code is covered by unit tests. Some

tasks (such as extensive low-level interface

regression testing Regression testing (rarely, ''non-regression testing'') is re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change. If not, that would be called a '' regre ...

) can be laborious and time-consuming to do manually. In addition, a manual approach might not always be effective in finding certain classes of defects. Test automation offers a possibility to perform these types of testing effectively. Once automated tests have been developed, they can be run quickly and repeatedly. Many times, this can be a cost-effective method for regression testing of software products that have a long maintenance life. Even minor patches over the lifetime of the application can cause existing features to break which were working at an earlier point in time. While the reusability of automated tests is valued by software development companies, this property can also be viewed as a disadvantage. It leads to the so-called "Pesticide Paradox", where repeatedly executed scripts stop detecting errors that go beyond their frameworks. In such cases, manual testing may be a better investment. This ambiguity once again leads to the conclusion that the decision on test automation should be made individually, keeping in mind project requirements and peculiarities. Test automation tools can be expensive and are usually employed in combination with manual testing. Test automation can be made cost-effective in the long term, especially when used repeatedly in

. A good candidate for test automation is a test case for common flow of an application, as it is required to be executed (regression testing) every time an enhancement is made in the application. Test automation reduces the effort associated with manual testing. Manual effort is needed to develop and maintain automated checks, as well as reviewing test results. In automated testing, the test engineer or

software quality assurance Software quality assurance (SQA) is a means and practice of monitoring all software engineering processes, methods, and work products to ensure compliance against defined standards. It may include ensuring conformance to standards or models, suc ...

person must have software coding ability since the test cases are written in the form of source code which when run produce output according to the assertions that are a part of it. Some test automation tools allow for test authoring to be done by keywords instead of coding, which do not require programming.

API testing

API testing is also being widely used by software testers as it enables them to verify requirements independent of their GUI implementation, commonly to test them earlier in development, and to make sure the test itself adheres to clean code principles, especially the single responsibility principle. It involves directly testing

API An application programming interface (API) is a way for two or more computer programs to communicate with each other. It is a type of software interface, offering a service to other pieces of software. A document or standard that describes how ...

s as part of integration testing, to determine if they meet expectations for functionality, reliability, performance, and security.Testing APIs protects applications and reputations
by Amy Reichert, SearchSoftwareQuality March 2015 Since APIs lack a

GUI The GUI ( "UI" by itself is still usually pronounced . or ), graphical user interface, is a form of user interface that allows users to interact with electronic devices through graphical icons and audio indicator such as primary notation, inste ...

, API testing is performed at the message layer.All About API Testing: An Interview with Jonathan Cooper
by Cameron Philipp-Edmonds, Stickyminds August 19, 2014 API testing is considered critical when an API serves as the primary interface to

application logic In computer software, business logic or domain logic is the part of the program that encodes the real-world business rules that determine how data can be created, stored, and changed. It is contrasted with the remainder of the software that might ...

Continuous testing

Continuous testing Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.Part of the Pipeline: Why Continuous Testing Is Essential
by Adam Auerbach, TechWell Insights August 2015The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola
by Cameron Philipp-Edmonds, Stickyminds December 2015 For Continuous Testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.DevOps: Are You Pushing Bugs to Clients Faster
by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015

Graphical User Interface (GUI) testing

Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. The advantage of this approach is that it requires little or no

software development Software development is the process of conceiving, specifying, designing, programming, documenting, testing, and bug fixing involved in creating and maintaining applications, frameworks, or other software components. Software development invol ...

. This approach can be applied to any application that has a

graphical user interface The GUI ( "UI" by itself is still usually pronounced . or ), graphical user interface, is a form of user interface that allows users to interact with electronic devices through graphical icons and audio indicator such as primary notation, inste ...

. However, reliance on these features poses major reliability and maintainability problems. Relabelling a button or moving it to another part of the window may require the test to be re-recorded. Record and playback also often adds irrelevant activities or incorrectly records some activities. A variation on this type of tool is for testing of web sites. Here, the "interface" is the web page. However, such a framework utilizes entirely different techniques because it is rendering

HTML The HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets (CSS) and scripting languages such as JavaScri ...

and listening to DOM Events instead of operating system events.

Headless browser A headless browser is a web browser without a graphical user interface. Headless browsers provide automated control of a web page in an environment similar to popular web browsers, but they are executed via a command-line interface or using netwo ...

s or solutions based on Selenium Web Driver are normally used for this purpose.Headless Testing with PhantomJS;http://phantomjs.org/headless-testing.html Another variation of this type of test automation tool is for testing mobile applications. This is very useful given the number of different sizes, resolutions, and operating systems used on mobile phones. For this variation, a framework is used in order to instantiate actions on the mobile device and to gather results of the actions. Another variation is script-less test automation that does not use record and playback, but instead builds a model of the application and then enables the tester to create test cases by simply inserting test parameters and conditions, which requires no scripting skills.

Testing at different levels

A strategy to decide the amount of tests to automate is the test automation pyramid. This strategy suggests to write three types of tests with different granularity. The higher the level, less is the amount of tests to write.

Levels

* As a solid foundation,

Unit testing In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures&md ...

provides robustness to the software products. Testing individual parts of the code makes it easy to write and run the tests. Developers write Unit Tests as a part of each story and integrate them with CI. * The service layer refers to testing the services of an application separately from its user interface, these services are anything that the application does in response to some input or set of inputs. * At the top level we have UI testing which has fewer tests due to the different attributes that make it more complex to run, for example the fragility of the tests, where a small change in the user interface can break a lot of tests and adds maintenance effort.

Framework approach in automation

A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort. The main advantage of a

framework A framework is a generic term commonly referring to an essential supporting structure which other things are built on top of. Framework may refer to: Computing * Application framework, used to implement the structure of an application for an op ...

of assumptions, concepts and tools that provide support for automated software testing is the low cost for

maintenance Maintenance may refer to: Biological science * Maintenance of an organism * Maintenance respiration Non-technical maintenance * Alimony, also called ''maintenance'' in British English * Champerty and maintenance, two related legal doctrine ...

. If there is change to any

test case In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise ...

then only the test case file needs to be updated and the driver Script and

startup script In Unix-based computer operating systems, init (short for ''initialization'') is the first process started during booting of the computer system. Init is a daemon process that continues running until the system is shut down. It is the dire ...

will remain the same. Ideally, there is no need to update the scripts in case of changes to the application. Choosing the right framework/scripting technique helps in maintaining lower costs. The costs associated with test scripting are due to development and maintenance efforts. The approach of scripting used during test automation has effect on costs. Various framework/scripting techniques are generally used: # Linear (procedural code, possibly generated by tools like those that use record and playback) # Structured (uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements) # Data-driven (data is persisted outside of tests in a database, spreadsheet, or other mechanism) # Keyword-driven # Hybrid (two or more of the patterns above are used) # Agile automation framework The Testing framework is responsible for: # defining the format in which to express expectations # creating a mechanism to hook into or drive the application under test # executing the tests # reporting results

Test automation interface

Test automation interfaces are platforms that provide a single

workspace Workspace is a term used in various branches of engineering and economic development. Business development Workspace refers to small premises provided, often by local authorities or economic development agencies, to help new businesses to estab ...

for incorporating multiple testing tools and frameworks for System/Integration testing of application under test. The goal of Test Automation Interface is to simplify the process of mapping tests to business criteria without coding coming in the way of the process. Test automation interface are expected to improve the efficiency and flexibility of maintaining test scripts.

Test Automation Interface consists of the following core modules: * Interface Engine * Interface Environment * Object Repository

Interface engine

Interface engines are built on top of Interface Environment. Interface engine consists of a

parser Parsing, syntax analysis, or syntactic analysis is the process of analyzing a string of symbols, either in natural language, computer languages or data structures, conforming to the rules of a formal grammar. The term ''parsing'' comes from Lati ...

and a test runner. The parser is present to parse the object files coming from the object repository into the test specific scripting language. The test runner executes the test scripts using a

test harness In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the te ...

Object repository

Object repositories are a collection of UI/Application object data recorded by the testing tool while exploring the application under test.

Defining boundaries between automation framework and a testing tool

Tools are specifically designed to target some particular test environment, such as Windows and web automation tools, etc. Tools serve as a driving agent for an automation process. However, an automation framework is not a tool to perform a specific task, but rather infrastructure that provides the solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer. There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are: # Data-driven testing #

Modularity-driven testing Modularity-driven testing is a term used in the testing of software. Background The test script modularity framework A framework is a generic term commonly referring to an essential supporting structure which other things are built on top of. Fra ...

Keyword-driven testing Keyword-driven testing, also known as action word based testing (not to be confused with action driven testing), is a software testing methodology suitable for both manual and automated testing. This method separates the documentation of test cas ...

# Hybrid testing #

Model-based testing Model-based testing is an application of model-based design for designing and optionally also executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a system under test (SUT), or ...

# Code-driven testing #

Behavior driven development In software engineering, behavior-driven development (BDD) is an agile software development process that encourages collaboration among developers, quality assurance experts, and customer representatives in a software project. It encourages teams ...

What to test

Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with

test oracle In computing, software engineering, and software testing, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed. The use of oracles involves comparing the output(s) of the system under test, for a given ...

s), defect logging, etc., without necessarily automating tests in an end-to-end fashion. One must keep satisfying popular requirements when thinking of test automation: *

Platform Platform may refer to: Technology * Computing platform, a framework on which applications may be run * Platform game, a genre of video games * Car platform, a set of components shared by several vehicle models * Weapons platform, a system or ...

and OS independence * Data driven capability (Input Data, Output Data,

Metadata Metadata is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive metadata – the descriptive ...

) * Customization Reporting (DB

Data Base In computing, a database is an organized collection of data stored and accessed electronically. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases s ...

Access,

Crystal Reports Crystal Reports is a business intelligence application marketed to small- and medium-sized businesses by SAP. History Terry Cunningham and the Cunningham Group originated the software in 1984. Crystal Services Inc. marketed the product (origin ...

) * Easy debugging and logging *

Version control In software engineering, version control (also known as revision control, source control, or source code management) is a class of systems responsible for managing changes to computer programs, documents, large web sites, or other collections o ...

friendly – minimal binary files * Extensible & Customization (Open

s to be able to integrate with other tools) * Common Driver (For example, in the Java development ecosystem, that means

Ant Ants are eusocial insects of the family Formicidae and, along with the related wasps and bees, belong to the order Hymenoptera. Ants evolved from vespoid wasp ancestors in the Cretaceous period. More than 13,800 of an estimated total of 22 ...

Maven MAVEN is an American spacecraft orbiting Mars to study the loss of its atmospheric gases to space, providing insight into the history of the planet's climate and water. The spacecraft name is an acronym for "Mars Atmosphere and Volatile Evolu ...

and the popular

IDEs Ides or IDES may refer to: Calendar dates * Ides (calendar), a day in the Roman calendar that fell roughly in the middle of the month. In March, May, July, and October it was the 15th day of the month; in other months it was the 13th. **Ides of Mar ...

). This enables tests to integrate with the developers'

workflows A workflow consists of an orchestrated and repeatable pattern of activity, enabled by the systematic organization of resources into processes that transform materials, provide services, or process information. It can be depicted as a sequence of ...

. * Support unattended test runs for integration with build processes and batch runs. Continuous integration servers require this. * Email Notifications like

bounce message A bounce message or just "bounce" is an automated message from an email system, informing the sender of a previous message that the message has not been delivered (or some other delivery problem occurred). The original message is said to have "boun ...

s * Support distributed execution environment (distributed test bed) * Distributed application support (distributed SUT)

References

General references

* * * * Roman Savenkov: ''How to Become a Software Tester.'' Roman Savenkov Consulting, 2008, * * * Hayes, Linda G., "Automated Testing Handbook", Software Testing Institute, 2nd Edition, March 2004 * Kaner, Cem,
Architectures of Test Automation
, August 2000

External links

Practical Experience in Automated Testing

Test Automation: Delivering Business Value

Test Automation Snake Oil
by James Bach
When Should a Test Be Automated?
by Brian Marick
Guidelines for Test Automation framework

Advanced Test Automation

by Hans Buwalda
Automation That Learns: Making Your Computer Work for You
by Jeremy Carey-Dressler
Automation Testing Resources & Best Practices
by Joe Colantonio
List of websites to practice test automation
by Mayur S
Test Automation Starter Pack
by Bala {{DEFAULTSORT:Test Automation Software testing Automation software