HOME

TheInfoList



OR:

"\n\n\n\n\nThe robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by
website A website (also written as a web site) is a collection of web pages and related content that is identified by a common domain name and published on at least one web server. Examples of notable websites are Google Search, Google, Facebook, Amaz ...
s to indicate to visiting
web crawler A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spid ...
s and other web robots which portions of the site they are allowed to visit.\n\nThis relies on voluntary compliance. Not all robots comply with the standard; email harvesters,
spambots A spambot is a computer program designed to assist in the sending of spam. Spambots usually create accounts and send spam messages with them. Web hosts and website operators have responded by banning spammers, leading to an ongoing struggle betw ...
,
malware Malware (a portmanteau for ''malicious software'') is any software intentionally designed to cause disruption to a computer, server, client, or computer network, leak private information, gain unauthorized access to information or systems, depri ...
and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out.\n\nThe \"robots.txt\" file can be used in conjunction with
sitemaps The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about ea ...
, another robot inclusion standard for websites.\n


History

\nThe standard was proposed by
Martijn Koster Martijn Koster (born ca 1970) is a Dutch software engineer noted for his pioneering work on Internet searching. Koster created ALIWEB, the Internet's first search engine, which was announced in November 1993 while working at Nexor and presented ...
,<\/ref><\/ref> when working for
Nexor Nexor Limited is a privately held company based in Nottingham, providing product and services to safeguard government, defence and critical national infrastructure computer systems. It was originally known as X-Tel Services Limited. History Ne ...
<\/ref> in February 1994\n<\/ref> on the ''www-talk'' mailing list, the main communication channel for WWW-related activities at the time.
Charles Stross Charles David George "Charlie" Stross (born 18 October 1964) is a British writer of science fiction and fantasy. Stross specialises in hard science fiction and space opera. Between 1994 and 2004, he was also an active writer for the magazine '' ...
claims to have provoked Koster to suggest robots.txt, after he wrote a badly-behaved web crawler that inadvertently caused a
denial-of-service attack In computing, a denial-of-service attack (DoS attack) is a cyber-attack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connect ...
on Koster's server.<\/ref>\n\nIt quickly became a
de facto standard A ''de facto'' standard is a custom or convention that has achieved a dominant position by public acceptance or market forces (for example, by early entrance to the market). is a Latin phrase (literally " in fact"), here meaning "in practice b ...
that present and future web crawlers were expected to follow; most complied, including those operated by search engines such as
WebCrawler WebCrawler is a search engine, and one of the oldest surviving search engines on the web today. For many years, it operated as a metasearch engine. WebCrawler was the first web search engine to provide full text search. History Brian Pinkerto ...
,
Lycos Lycos, Inc., is a web search engine and web portal established in 1994, spun out of Carnegie Mellon University. Lycos also encompasses a network of email, web hosting, social networking, and entertainment websites. The company is based in Walth ...
, and
AltaVista AltaVista was a Web search engine established in 1995. It became one of the most-used early search engines, but lost ground to Google and was purchased by Yahoo! in 2003, which retained the brand, but based all AltaVista searches on its own sear ...
.<\/ref>\n\nOn July 1, 2019, Google announced the proposal of the Robots Exclusion Protocol as an official standard under
Internet Engineering Task Force The Internet Engineering Task Force (IETF) is a standards organization for the Internet and is responsible for the technical standards that make up the Internet protocol suite (TCP/IP). It has no formal membership roster or requirements and a ...
.<\/ref> The standard<\/ref> was published in September 2022 as RFC 9309.\n


Standard

\nWhen a site owner wishes to give instructions to web robots they place a text file called in the root of the web site hierarchy (e.g. ). This text file contains the instructions in a specific format (see examples below). Robots that ''choose'' to follow the instructions try to fetch this file and read the instructions before fetching any other file from the
website A website (also written as a web site) is a collection of web pages and related content that is identified by a common domain name and published on at least one web server. Examples of notable websites are Google Search, Google, Facebook, Amaz ...
. If this file does not exist, web robots assume that the website owner does not wish to place any limitations on crawling the entire site.\n\nA robots.txt file contains instructions for bots indicating which web pages they can and cannot access. Robots.txt files are particularly important for web crawlers from search engines such as Google.\n\nA robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operates on certain data. Links to pages listed in robots.txt can still appear in search results if they are linked to from a page that is crawled.<\/ref>\n\nA robots.txt file covers one
origin Origin(s) or The Origin may refer to: Arts, entertainment, and media Comics and manga * Origin (comics), ''Origin'' (comics), a Wolverine comic book mini-series published by Marvel Comics in 2002 * The Origin (Buffy comic), ''The Origin'' (Bu ...
. For websites with multiple subdomains, each subdomain must have its own robots.txt file. If had a robots.txt file but did not, the rules that would apply for would not apply to . In addition, each protocol and port needs its own robots.txt file; does not apply to pages under or .\n\nSome major search
engines An engine or motor is a machine designed to convert one or more forms of energy into mechanical energy. Available energy sources include potential energy (e.g. energy of the Earth's gravitational field as exploited in hydroelectric power ...
following this standard include Ask,<\/ref> AOL,<\/ref> Baidu,<\/ref> DuckDuckGo,<\/ref> Google,<\/ref> Yahoo!,<\/ref> and Yandex.<\/ref> Bing<\/ref> is still not fully compatible with the standard as it cannot inherit settings from the wildcard (*<\/code>).<\/ref>\n\nThe volunteering group
Archive Team Archive Team is a group dedicated to digital preservation and web archiving that was co-founded by Jason Scott in 2009. Its primary focus is the copying and preservation of content housed by at-risk online services. Some of its projects include t ...
explicitly ignores robots.txt directives, using it instead for discovering more links, such as
sitemap A sitemap is a list of pages of a web site within a domain. There are three primary kinds of sitemap: * Sitemaps used during the planning of a website by its designers. * Human-visible listings, typically hierarchical, of the pages on a site. * St ...
s.<\/ref> The group views it as an obsolete standard that hinders web archival efforts. According to project leader Jason Scott, \"unchecked, and left alone, the robots.txt file ensures no mirroring or reference for items that may have general use and meaning beyond the website's context.\"<\/ref>\n\nFor some years, the
Internet Archive The Internet Archive is an American digital library with the stated mission of "universal access to all knowledge". It provides free public access to collections of digitized materials, including websites, software applications/games, music, ...
did not crawl sites with robots.txt, but in April 2017, it announced that it would no longer honour directives in the robots.txt files.<\/ref> \"Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes\".<\/ref> This was in response to entire domains being tagged with robots.txt when the content became obsolete.\n


Security

\nDespite the use of the terms \"allow\" and \"disallow\", the protocol is purely advisory and relies on the compliance of the
web robot An Internet bot, web robot, robot or simply bot, is a software application that runs automated tasks (scripts) over the Internet, usually with the intent to imitate human activity on the Internet, such as messaging, on a large scale. An Internet b ...
.<\/ref> Malicious web robots are unlikely to honor robots.txt; some may even use the robots.txt as a guide to find disallowed links and go straight to them. While this is sometimes claimed to be a security risk,<\/ref> this sort of
security through obscurity Security through obscurity (or security by obscurity) is the reliance in security engineering on design or implementation secrecy as the main method of providing security to a system or component. History An early opponent of security through ob ...
is discouraged by standards bodies. The
National Institute of Standards and Technology The National Institute of Standards and Technology (NIST) is an agency of the United States Department of Commerce whose mission is to promote American innovation and industrial competitiveness. NIST's activities are organized into physical sci ...
(NIST) in the United States specifically recommends against this practice: \"System security should not depend on the secrecy of the implementation or its components.\"<\/ref> In the context of robots.txt files, security through obscurity is not recommended as a security technique.<\/ref>\n


Alternatives

\nMany robots also pass a special
user-agent In computing, a user agent is any software, acting on behalf of a user, which "retrieves, renders and facilitates end-user interaction with Web content". A user agent is therefore a special kind of software agent. Some prominent examples of use ...
to the web server when fetching content.<\/ref> A web administrator could also configure the server to automatically return failure (or pass alternative content) when it detects a connection using one of the robots.<\/ref><\/ref>\n\nSome sites, such as
Google Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...
, host a humans.txt<\/code> file that displays information meant for humans to read.<\/ref> Some sites such as
GitHub GitHub, Inc. () is an Internet hosting service for software development and version control using Git. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous ...
redirect humans.txt to an about page.<\/ref>\n\nPreviously, Google had a joke file hosted at \/killer-robots.txt<\/code> instructing
the Terminator ''The Terminator'' is a 1984 American science fiction action film directed by James Cameron. It stars Arnold Schwarzenegger as the Terminator, a cyborg assassin sent back in time from 2029 to 1984 to kill Sarah Connor (Linda Hamilton), whos ...
not to kill the company founders
Larry Page Lawrence Edward Page (born March 26, 1973) is an American business magnate, computer scientist and internet entrepreneur. He is best known for co-founding Google with Sergey Brin. Page was the chief executive officer of Google from 1997 unt ...
and
Sergey Brin Sergey Mikhailovich Brin (russian: link=no, Сергей Михайлович Брин; born August 21, 1973) is an American business magnate, computer scientist, and internet entrepreneur, who co-founded Google with Larry Page. Brin was the ...
.<\/ref><\/ref>\n


Examples

\nThis example tells all robots that they can visit all files because the wildcard *<\/code> stands for all robots and the Disallow<\/code> directive has no value, meaning no pages are disallowed.\n\n
\nUser-agent: *\nDisallow: \n<\/pre>\n\n
\nUser-agent: *\nAllow: \/\n<\/pre>\n\nThe same result can be accomplished with an empty or missing robots.txt file.\n\nThis example tells all robots to stay out of a website:\n\n
\nUser-agent: *\nDisallow: \/\n<\/pre>\n\nThis example tells all robots not to enter three directories:\n\n
\nUser-agent: *\nDisallow: \/cgi-bin\/\nDisallow: \/tmp\/\nDisallow: \/junk\/\n<\/pre>\n\nThis example tells all robots to stay away from one specific file:\n\n
\nUser-agent: *\nDisallow: \/directory\/file.html\n<\/pre>\n\nAll other files in the specified directory will be processed.\n\n
\nUser-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot\nDisallow: \/\n<\/pre>\n\nThis example tells two specific robots not to enter one specific directory:\n\n
\nUser-agent: BadBot # replace 'BadBot' with the actual user-agent of the bot\nUser-agent: Googlebot\nDisallow: \/private\/\n<\/pre>\n\nExample demonstrating how comments can be used:\n\n
\n# Comments appear after the \"#\" symbol at the start of a line, or after a directive\nUser-agent: * # match all bots\nDisallow: \/ # keep them out\n<\/pre>\n\nIt is also possible to list multiple robots with their own rules. The actual robot string is defined by the crawler. A few robot operators, such as 
Google Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...
, support several user-agent strings that allow the operator to deny access to a subset of their services by using specific user-agent strings.\n\nExample demonstrating multiple user-agents:\n\n
\nUser-agent: googlebot        # all Google services\nDisallow: \/private\/          # disallow this directory\n\nUser-agent: googlebot-news   # only the news service\nDisallow: \/                  # disallow everything\n\nUser-agent: *                # any robot\nDisallow: \/something\/        # disallow this directory\n<\/pre>\n


Nonstandard extensions

\n


Crawl-delay directive

\nThe crawl-delay value is supported by some crawlers to throttle their visits to the host. Since this value is not part of the standard, its interpretation is dependent on the crawler reading it. It is used when the multiple burst of visits from bots is slowing down the host. Yandex interprets the value as the number of seconds to wait between subsequent visits. Bing defines crawl-delay as the size of a time window (from 1 to 30 seconds) during which BingBot will access a web site only once.<\/ref> Google provides an interface in its search console for webmasters, to control the
Googlebot Googlebot is the web crawler software used by Google that collects documents from the web to build a searchable index for the Google Search engine. This name is actually used to refer to two different types of web crawlers: a desktop crawler (t ...
's subsequent visits.<\/ref>\n\n
\nUser-agent: bingbot\nAllow: \/\nCrawl-delay: 10\n<\/pre>\n


Sitemap

\nSome crawlers support a Sitemap<\/code> directive, allowing multiple
Sitemaps The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about ea ...
in the same robots.txt<\/samp> in the form Sitemap: ''full-url''<\/code>:<\/ref>\n
Sitemap: http:\/\/www.example.com\/sitemap.xml<\/pre>\n


Host

\nSome crawlers (
Yandex Yandex LLC (russian: link=no, Яндекс, p=ˈjandəks) is a Russian multinational technology company providing Internet-related products and services, including an Internet search engine, information services, e-commerce, transportation, maps ...
) support a Host<\/code> directive, allowing websites with multiple mirrors to specify their preferred domain:<\/ref>\n\n
\nHost: hosting.example.com\n<\/pre>\n\nThis is not supported by all crawlers.\n


Universal \"*\" match

\nThe ''Robot Exclusion Standard'' does not mention the \"*\" character in the Disallow:<\/code> statement.<\/ref>\n


Meta tags and headers

\nIn addition to root-level robots.txt files, robots exclusion directives can be applied at a more granular level through the use of
Robots meta tag Meta elements are tags used in HTML and XHTML documents to provide structured metadata about a Web page. They are part of a web page's head section. Multiple Meta elements with different attributes can be used on the same page. Meta elements ca ...
s and X-Robots-Tag HTTP headers. The robots meta tag cannot be used for non-HTML files such as images, text files, or PDF documents. On the other hand, the X-Robots-Tag can be added to non-HTML files by using .htaccess and

files.<\/ref>\n\n;A \"noindex\" meta tag\n\n\n\n<\/syntaxhighlight>\n\n;A \"noindex\" HTTP response header\n\n\nX-Robots-Tag: noindex\n<\/syntaxhighlight>\n\nThe X-Robots-Tag is only effective after the page has been requested and the server responds, and the robots meta tag is only effective after the page has loaded, whereas robots.txt is effective before the page is requested. Thus if a page is excluded by a robots.txt file, any robots meta tags or X-Robots-Tag headers are effectively ignored because the robot will not see them in the first place.\n\n\nMaximum Size of a Robots.txt File\n\nThe Robots Exclusion Protocol requires crawlers to parse at least 500 kibibytes (KiB) of robots.txt files,<\/ref> which Google maintains as a 500 kibibyte file size restriction for robots.txt files .<\/ref>\n


See also

\n\n*
ads.txt ads.txt (Authorized Digital Sellers) is an initiative from IAB Technology Laboratory. It specifies a text file that companies can host on their web servers, listing the other companies authorized to sell their products or services. This is desig ...
<\/code>, a standard for listing authorized ad sellers\n*
security.txt security.txt is a proposed standard for websites' security information that is meant to allow security researchers to easily report security vulnerabilities. The standard prescribes a text file called "security.txt" in the well known location, s ...
<\/code>, a file to describe the process for security researchers to follow in order to report security vulnerabilities\n*
Automated Content Access Protocol Automation describes a wide range of technologies that reduce human intervention in processes, namely by predetermining decision criteria, subprocess relationships, and related actions, as well as embodying those predeterminations in machines ...
\u2013 a failed proposal to extend robots.txt\n*
BotSeer BotSeer was a Web-based information system and search tool used for research on Web robots and trends in Robot Exclusion Protocol deployment and adherence. It was created and designed by Yang Sun, Isaac G. Councill, Ziming Zhuang and C. Lee Giles. ...
\u2013 now inactive search engine for robots.txt files\n* Distributed web crawling\n*
Focused crawler A focused crawler is a web crawler that collects Web pages that satisfy some specific property, by carefully prioritizing the crawl frontier and managing the hyperlink exploration process. Some predicates may be based on simple, deterministic and su ...
\n*
Internet Archive The Internet Archive is an American digital library with the stated mission of "universal access to all knowledge". It provides free public access to collections of digitized materials, including websites, software applications/games, music, ...
\n*
National Digital Library Program The Library of Congress National Digital Library Program (NDLP) is assembling a digital library of reproductions of primary source materials to support the study of the history and culture of the United States. Begun in 1995 after a five-year p ...
(NDLP)\n*
National Digital Information Infrastructure and Preservation Program The National Digital Information Infrastructure and Preservation Program (NDIIPP) of the United States was an archival program led by the Library of Congress to archive and provide access to digital resources. The program convened several working ...
(NDIIPP)\n*
Nofollow nofollow is a setting on a web page hyperlink that directs search engines not to use the link for page ranking calculations. It is specified in the page as a type of link relation; that is: <a rel="nofollow" ...>. Because search engines ...
\n*
Perma.cc Perma.cc is a web archiving service for legal and academic citations founded by the Harvard Library Innovation Lab in 2013. Concept Perma.cc was created in response to studies showing high incidences of link rot in both academic publications an ...
\n* Meta elements for search engines\n*
Sitemaps The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about ea ...
\n*
Spider trap A spider trap (or crawler trap) is a set of web pages that may intentionally or unintentionally be used to cause a web crawler or search bot to make an infinite number of requests or cause a poorly constructed crawler to crash. Web crawlers are a ...
\n*
Web archiving Web archiving is the process of collecting portions of the World Wide Web to ensure the information is preserved in an archive for future researchers, historians, and the public. Web archivists typically employ web crawlers for automated captur ...
\n*
Web crawler A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spid ...
\n*
noindex The noindex value of an HTML robots meta tag requests that automated Internet bots avoid indexing a web page.Robots Database (list of bot names)
n
Robots.txt checker tool
n
Microsoft bingbot official documentation
n\n\n\n{{DEFAULTSORT:Robots Exclusion Standard\n Search engine optimization"