Test and learn is a set of practices followed by retailers, banks and other consumer-focused companies to test ideas in a small number of locations or customers to predict impact. The process is often designed to answer three questions about any tested program before rollout: # What impact will the program have on key performance indicators if executed across the network or customer base? # Will the program have a larger impact on some stores/customers than others? # Which components of the idea are actually working?

Historical context

Test and learn has been systematically applied as far back as 1988 by

Capital One Capital One Financial Corporation is an American bank holding company specializing in credit cards, auto loans, banking, and savings accounts, headquartered in McLean, Virginia with operations primarily in the United States. It is on the li ...

. Capital One has been aggressive about testing since the firm was founded, testing everything from product design to marketing to customer selection to collection policies. In a single year, the company performs tens of thousands of tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank, CEO of Capital One, called test and learn "a marketing revolution that can be applied to many businesses".

Examples

* Retail banks running different commercials in different markets. * Fast food restaurants launching a new menu item or product for a limited time in select locations * Big-box retailers evaluating “store of the future” programs * Companies evaluating pricing strategies * Retailers testing merchandising layouts across their door base * The UK government, through research projects such as the College for Teaching and Learning's Close the Gap: Test and learn

Test and learn in practice

* Woolworths uses a test-and-learn approach to identify which stores should undergo renovations. *

Wawa Food Markets Wawa, Inc. ( ) is an American chain of convenience stores and gas stations located along the East Coast of the United States, operating in Pennsylvania, New Jersey, Delaware, Maryland, Virginia, Washington, D.C., and Florida. The company's corpor ...

, a Mid-Atlantic convenience store chain, uses Test and Learn across several aspects of its business: ** Measuring the effect of introducing a new product ** Determining the impact of adding additional staff ** Understanding how the opening a new location would impact existing nearby locations * Subway ran a promotion that lowered the price of its foot-long sandwiches to $5. Before deciding to launch the promotion nationally, Subway ran a test that compared locations offering the $5 foot-long sandwiches to a group of similar

control Control may refer to: Basic meanings Economics and business * Control (management), an element of management * Control, an element of management accounting * Comptroller (or controller), a senior financial officer in an organization * Controllin ...

restaurants that were not offering the promotion. *

Kraft Foods The second incarnation of Kraft Foods is an American food manufacturing and processing conglomerate, split from Kraft Foods Inc. in 2012 and headquartered in Chicago, Illinois. It became part of Kraft Heinz in 2015. A merger with Heinz, arran ...

, a manufacturer and distributor of packaged foods, uses testing to optimize the placement of its products on grocery store shelves: ** Determining the effect of a coffee aisle re-design on sales of premium coffees ** Measuring the impact on product sales of a change to in-store sales representatives' responsibilities *

Pier 1 Imports Pier 1 Imports, Inc. is an online retailer and former Fort Worth, Texas-based retail chain specializing in imported home furnishings and decor, particularly furniture, table-top items, decorative accessories, and seasonal decor. It was publicl ...

, a recently bankrupt retail chain specializing in home decor and furnishings, uses testing to measure the effectiveness of

search advertising In Internet marketing, search advertising is a method of placing online advertisements on web pages that show results from search engine queries. Through the same search-engine advertising services, ads can also be placed on Web pages with other pu ...

campaigns. * Test and learn is also methodically used online to test impact of various web page layouts on user behavior. Tools such as

Google Google LLC () is an American Multinational corporation, multinational technology company focusing on Search Engine, search engine technology, online advertising, cloud computing, software, computer software, quantum computing, e-commerce, ar ...

Website Analyzer provide every site with the ability to test various elements and their effect on user engagement.

Test and learn software

There are a variety of

software Software is a set of computer programs and associated software documentation, documentation and data (computing), data. This is in contrast to Computer hardware, hardware, from which the system is built and which actually performs the work. ...

tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis. Many companies, including

and

eBay eBay Inc. ( ) is an American multinational e-commerce company based in San Jose, California, that facilitates consumer-to-consumer and business-to-consumer sales through its website. eBay was founded by Pierre Omidyar in 1995 and became ...

, have developed experiment management software within the company. Alternatives to internal solutions include broad statistical analysis software tools such as SAS or test and learn focused software from Applied Predictive Technologies, Trial Run and MarketDial. There are also a variety of more narrowly tailored products, focused on particular problems or fields. In

web analytics Web analytics is the measurement, collection, analysis, and reporting of web data to understand and optimize web usage. Web analytics is not just a process for measuring web traffic but can be used as a tool for business and market research and ...

, for example, firms such as

Omniture Omniture is an online marketing and web analytics business unit in Orem, Utah. It was acquired by Adobe Systems in 2009. Until 2011, Omniture operated as a business unit within Adobe as the "Omniture Business Unit", but as of 2012 Adobe began reti ...

WebTrends Webtrends is a private company headquartered in Portland, Oregon, United States. It provides digital analytics, optimization and software related to digital marketing and e-commerce. It provides services to approximately 2,000 companies. History ...

, and

have created specialized software.

Criticism

Companies wanting to conduct Test and Learn must mobilize, track and analyze site and customer data at very granular levels. This can often be a massive and expensive undertaking. Increasingly however, an exponential increase in computing power, and corresponding decrease in its cost, have made testing more accessible. In the

Harvard Business Review ''Harvard Business Review'' (''HBR'') is a general management magazine published by Harvard Business Publishing, a wholly owned subsidiary of Harvard University. ''HBR'' is published six times a year and is headquartered in Brighton, Ma ...

February 2009 article "How to Design Smart Business Experiments", Thomas Davenport discusses the Test and Learn Process, and describes how several companies have overcome these and other barriers to testing.

References