Heritrix is a

web crawler A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spid ...

designed for

web archiving Web archiving is the process of collecting portions of the World Wide Web to ensure the information is preserved in an archive for future researchers, historians, and the public. Web archivists typically employ web crawlers for automated captur ...

. It was written by the

Internet Archive The Internet Archive is an American digital library with the stated mission of "universal access to all knowledge". It provides free public access to collections of digitized materials, including websites, software applications/games, music, ...

. It is available under a

free software license A free-software license is a notice that grants the recipient of a piece of software extensive rights to modify and redistribute that software. These actions are usually prohibited by copyright law, but the rights-holder (usually the author) ...

and written in

Java Java (; id, Jawa, ; jv, ꦗꦮ; su, ) is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea to the north. With a population of 151.6 million people, Java is the world's mos ...

. The main interface is accessible using a

web browser A web browser is application software for accessing websites. When a user requests a web page from a particular website, the browser retrieves its files from a web server and then displays the page on the user's screen. Browsers are used o ...

, and there is a

command-line A command-line interpreter or command-line processor uses a command-line interface (CLI) to receive commands from a user in the form of lines of text. This provides a means of setting parameters for the environment, invoking executables and pro ...

tool that can optionally be used to initiate crawls. Heritrix was developed jointly by the Internet Archive and the Nordic national libraries on specifications written in early 2003. The first official release was in January 2004, and it has been continually improved by employees of the Internet Archive and other interested parties. For many years Heritrix was not the main crawler used to crawl content for the Internet Archive's web collection. The largest contributor to the collection, as of 2011, is Alexa Internet. Alexa crawls the web for its own purposes, using a crawler named ''ia_archiver''. Alexa then donates the material to the Internet Archive. The Internet Archive itself did some of its own crawling using Heritrix, but only on a smaller scale. Starting in 2008, the Internet Archive began performance improvements to do its own wide scale crawling, and now does collect most of its content.

Projects using Heritrix

A number of organizations and national libraries are using Heritrix, among them: *

Austrian National Library The Austrian National Library (german: Österreichische Nationalbibliothek) is the largest library in Austria, with more than 12 million items in its various collections. The library is located in the Neue Burg Wing of the Hofburg in center of V ...

, Web Archiving *

Bibliotheca Alexandrina The Bibliotheca Alexandrina (Latin for "Library of Alexandria"; arz, مكتبة الإسكندرية ', ) is a major library and cultural center on the shore of the Mediterranean Sea in Alexandria, Egypt. It is a commemoration of the Library ...

's Internet Archive * Bibliothèque nationale de France *

British Library The British Library is the national library of the United Kingdom and is one of the largest libraries in the world. It is estimated to contain between 170 and 200 million items from many countries. As a legal deposit library, the British ...

* California Digital Library's Web Archiving Service *

CiteSeerX CiteSeerX (formerly called CiteSeer) is a public search engine and digital library for scientific and academic papers, primarily in the fields of computer and information science. CiteSeer's goal is to improve the dissemination and access of ac ...

* Documenting Internet2 *

Internet Memory Foundation The Internet Memory Foundation (formerly the European Archive Foundation) was a non-profitable foundation whose purpose was archiving content of the World Wide Web. It supported projects and research that included the preservation and protection ...

* Library and Archives Canada *

Library of Congress The Library of Congress (LOC) is the research library that officially serves the United States Congress and is the ''de facto'' national library of the United States. It is the oldest federal cultural institution in the country. The library ...

National and University Library of Iceland Landsbókasafn Íslands – Háskólabókasafn ( Icelandic: ; English: ''The National and University Library of Iceland'') is the national library of Iceland which also functions as the university library of the University of Iceland. The librar ...

National Library of Finland The National Library of Finland ( fi, Kansalliskirjasto, sv, Nationalbiblioteket) is the foremost research library in Finland. Administratively the library is part of the University of Helsinki. From 1919 to 1 August 2006, it was known as the ...

* National Library of New Zealand *

Royal Library of the Netherlands The Royal Library of the Netherlands (Dutch: Koninklijke Bibliotheek or KB; ''Royal Library'') is the national library of the Netherlands, based in The Hague, founded in 1798. The KB collects everything that is published in and concerning the Ne ...

(Koninklijke Bibliotheek) * Netarkivet.dk *

Smithsonian Institution Archives Smithsonian Libraries and Archives is an institutional archives and library system comprising 21 branch libraries serving the various Smithsonian Institution museums and research centers. The Libraries and Archives serve Smithsonian Institution ...

* National Library of Israel

Arc files

Older versions of Heritrix by default stored the web resources it crawls in an Arc file. This file format is wholly unrelated to

ARC (file format) ARC is a lossless data compression and archival format by System Enhancement Associates (SEA). The file format and the program were both called ARC. The format is known as the subject of controversy in the 1980s, part of important debates over w ...

. This format has been used by the Internet Archive since 1996 to store its web archives. More recently it saves by default in the WARC file format, which is similar to ARC but more precisely specified and more flexible. Heritrix can also be configured to store files in a directory format similar to the

Wget GNU Wget (or just Wget, formerly Geturl, also written as its package name, wget) is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from "World Wide Web" and " ''get''." It supports do ...

crawler that uses the URL to name the directory and filename of each resource. An Arc file stores multiple archived resources in a single file in order to avoid managing a large number of small files. The file consists of a sequence of URL records, each with a header containing metadata about how the resource was requested followed by the

HTTP header The Hypertext Transfer Protocol (HTTP) is an application layer protocol in the Internet protocol suite model for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web, w ...

and the response. Arc files range between 100 and 600 MB. Example: filedesc://IA-2006062.arc 0.0.0.0 20060622190110 text/plain 76 1 1 InternetArchive URL IP-address Archive-date Content-type Archive-length http://foo.edu:80/hello.html 127.10.100.2 19961104142103 text/html 187 HTTP/1.1 200 OK Date: Thu, 22 Jun 2006 19:01:15 GMT Server: Apache Last-Modified: Sat, 10 Jun 2006 22:33:11 GMT Content-Length: 30 Content-Type: text/html Hello World!!!

Tools for processing Arc files

Heritrix includes a command-line tool called arcreader which can be used to extract the contents of an Arc file. The following command lists all the URLs and metadata stored in the given Arc file (i
CDX
format): arcreader IA-2006062.arc The following command extracts hello.html from the above example assuming the record starts at offset 140: arcreader -o 140 -f dump IA-2006062.arc Other tools:
Arc processing tools

WERA (Web ARchive Access)

Command-line tools

Heritrix comes with several command-line tools: * htmlextractor – displays the links Heritrix would extract for a given URL * hoppath.pl – recreates the hop path (path of links) to the specified URL from a completed crawl * manifest_bundle.pl – bundles up all resources referenced by a crawl manifest file into an uncompressed or compressed tar ball * cmdline-jmxclient – enables command-line control of Heritrix * arcreader – extracts contents of ARC files (see above) Further tools are available as part of the Internet Archive's warctools project.

References

External links