Wayback Machine

The Wayback Machine is a digital archive of the World Wide Web founded by the Internet Archive, a nonprofit based in San Francisco, California. Created in 1996 and launched to the public in 2001, it allows the user to go "back in time" and see how websites looked in the past. Its founders, Brewster Kahle and Bruce Gilliat, developed the Wayback Machine to provide "universal access to all knowledge" by preserving archived copies of defunct web pages.

Launched on May 10, 1996, the Wayback Machine had more than 38.2 million records at the end of 2009. , the Wayback Machine had saved more than 760 billion web pages. More than 350 million web pages are added daily.

History

The Wayback Machine began archiving cached web pages in 1996. One of the earliest known pages was saved on May 10, 1996, at 2:08p.m.

Internet Archive founders Brewster Kahle and Bruce Gilliat launched the Wayback Machine in San Francisco, California, in October 2001, primarily to address the problem of web content vanishing whenever it gets changed or when a website is shut down. The service enables users to see archived versions of web pages across time, which the archive calls a "three-dimensional index". Kahle and Gilliat created the machine hoping to archive the entire Internet and provide "universal access to all knowledge". The name "Wayback Machine" is a reference to a fictional time-traveling and translation device, the " Wayback Machine", used by the characters Mister Peabody and Sherman in the animated cartoon '' The Adventures of Rocky and Bullwinkle and Friends''. In one of the cartoon's segments, "Peabody's Improbable History", the characters used the machine to witness, participate in, and often alter famous events in history.

From 1996 to 2001, the information was kept on digital tape, with Kahle occasionally allowing researchers and scientists to tap into the "clunky" database. When the archive reached its fifth anniversary in 2001, it was unveiled and opened to the public in a ceremony at the University of California, Berkeley. By the time the Wayback Machine launched, it already contained over 10 billion archived pages. The data is stored on the Internet Archive's large cluster of Linux nodes. It revisits and archives new versions of websites on occasion (see technical details below). Sites can also be captured manually by entering a website's URL into the search box, provided that the website allows the Wayback Machine to "crawl" it and save the data.

On October 30, 2020, the Wayback Machine began fact-checking content. As of January 2022, domains of ad servers are disabled from capturing.

For Internet Archive's 25th anniversary, the Wayback Machine introduced the "Wayforward Machine" which allowed users to "travel to the Internet in 2046, where knowledge is under siege".

Technical information

Software has been developed to " crawl" the Web and download all publicly accessible information and data files on webpages, the Gopher hierarchy, the Netnews (Usenet) bulletin board system, and downloadable software. The information collected by these "crawlers" does not include all the information available on the Internet, since much of the data is restricted by the publisher or stored in databases that are not accessible. To overcome inconsistencies in partially cached websites, Archive-It.org was developed in 2005 by the Internet Archive as a means of allowing institutions and content creators to voluntarily harvest and preserve collections of digital content, and create digital archives.

Crawls are contributed from various sources, some imported from third parties and others generated internally by the Archive. For example, crawls are contributed by the Sloan Foundation and Alexa, crawls run by Internet Archive on behalf of NARA and the Internet Memory Foundation, mirrors of Common Crawl. The "Worldwide Web Crawls" have been running since 2010 and capture the global Web.

Documents and resources are stored with time stamp URLs such as . Pages' individual resources such as images and style sheets and scripts, as well as outgoing hyperlinks, are linked to with the time stamp of the currently viewed page, so they are redirected automatically to their individual captures that are the closest in time.

The frequency of snapshot captures varies per website. Websites in the "Worldwide Web Crawls" are included in a "crawl list", with the site archived once per crawl. A crawl can take months or even years to complete, depending on size. For example, "Wide Crawl Number 13" started on January 9, 2015, and completed on July 11, 2016. However, there may be multiple crawls ongoing at any one time, and a site might be included in more than one crawl list, so how often a site is crawled varies widely.

Starting in October 2019, users are limited to 15 archival requests and retrievals per minute.

Storage capacity and growth

As technology has developed over the years, the storage capacity of the Wayback Machine has grown. In 2003, after only two years of public access, the Wayback Machine was growing at a rate of 12 terabytes per month. The data is stored on PetaBox rack systems custom designed by Internet Archive staff. The first 100TB rack became fully operational in June 2004, although it soon became clear that they would need much more storage than that.

The Internet Archive migrated its customized storage architecture to Sun Open Storage in 2009, and hosts a new data centre in a Sun Modular Datacenter on Sun Microsystems' California campus. , the Wayback Machine contained approximately three petabytes of data and was growing at a rate of 100 terabytes each month.

A new, improved version of the Wayback Machine, with an updated interface and a fresher index of archived content, was made available for public testing in 2011, where captures appear in a calendar layout with circles whose width visualizes the number of crawls each day, but no marking of duplicates with asterisks or an advanced search page. A top toolbar has been added to facilitate navigating between captures. A bar chart visualizes the frequency of captures per month over the years. Features like "Changes", "Summary", and a graphical site map were added subsequently.

In March that year, it was said on the Wayback Machine forum that "the Beta of the new Wayback Machine has a more complete and up-to-date index of all crawled materials into 2010, and will continue to be updated regularly. The index driving the classic Wayback Machine only has a little bit of material past 2008, and no further index updates are planned, as it will be phased out this year." Also in 2011, the Internet Archive installed their sixth pair of PetaBox racks which increased the Wayback Machine's storage capacity by 700 terabytes.

In January 2013, the company announced a ground-breaking milestone of 240 billion URLs.

In October 2013, the company introduced the "Save a Page" feature which allows any Internet user to archive the contents of a URL, and quickly generates a permanent link unlike the preceding ''liveweb'' feature.

In December 2014, the Wayback Machine contained 435 billion web pages—almost nine petabytes of data, and was growing at about 20 terabytes a week.

In March 2015, it was published that security researchers became aware of the threat posed by the service's unintentional hosting of malicious binaries from archived sites.

In July 2016, the Wayback Machine reportedly contained around 15 petabytes of data.

In September 2018, the Wayback Machine contained over 25 petabytes of data.

As of December 2020, the Wayback Machine contained over 70 petabytes of data.

Between October 2013 and March 2015, the website's global Alexa rank changed from 163 to 208. In March 2019 the rank was at 244.

Wayback Machine APIs

The Wayback Machine service offers three public APIs, SavePageNow, Availability, and CDX. SavePageNow can be used to archive web pages. Availability API for checking the archive availability status for a web page, checking whether an archive for the web page exists or not. CDX API is for complex querying, filtering, and analysis of captured data.

Website exclusion policy

Historically, the Wayback Machine has respected the robots exclusion standard (robots.txt) in determining if a website would be crawled – or if already crawled, if its archives would be publicly viewable. Website owners had the option to opt-out of Wayback Machine through the use of robots.txt. It applied robots.txt rules retroactively; if a site blocked the Internet Archive, any previously archived pages from the domain were immediately rendered unavailable as well. In addition, the Internet Archive stated that "Sometimes, a website owner will contact us directly and ask us to stop crawling or archiving a site. We comply with these requests." In addition, the website says: "The Internet Archive is not interested in preserving or offering access to Web sites or other internet documents of persons who do not want their materials in the collection."

On April 17, 2017, reports surfaced of sites that had gone defunct and became parked domains that were using robots.txt to exclude themselves from search engines, resulting in them being inadvertently excluded from the Wayback Machine. The Internet Archive changed the policy to now require an explicit exclusion request to remove it from the Wayback Machine.

Oakland Archive Policy

Wayback's retroactive exclusion policy is based in part upon ''Recommendations for Managing Removal Requests and Preserving Archival Integrity'' published by the School of Information Management and Systems at University of California, Berkeley in 2002, which gives a website owner the right to block access to the site's archives. Wayback has complied with this policy to help avoid expensive litigation.

The Wayback retroactive exclusion policy began to relax in 2017, when it stopped honoring robots on U.S. government and military web sites for both crawling and displaying web pages. As of April 2017, Wayback is ignoring robots.txt more broadly, not just for U.S. government websites.

Uses

From its public launch in 2001, the Wayback Machine has been studied by scholars both for the ways it stores and collects data as wel