The deep web, invisible web, or hidden web
are parts of the
World Wide Web
The World Wide Web (WWW), commonly known as the Web, is an information system enabling documents and other web resources to be accessed over the Internet.
Documents and downloadable media are made available to the network through web ...
whose contents are not
indexed by standard
web search-engine programs. This is in contrast to the "
surface web", which is accessible to anyone using the Internet.
Computer-scientist Michael K. Bergman is credited with inventing the term in 2001 as a search-indexing term.
Deep web sites can be accessed by a direct
URL or
IP address
An Internet Protocol address (IP address) is a numerical label such as that is connected to a computer network that uses the Internet Protocol for communication.. Updated by . An IP address serves two main functions: network interface ident ...
, but may require entering a password or other security information to access actual content. Such sites have uses such as
web mail,
online banking
Online banking, also known as internet banking, web banking or home banking, is an electronic payment system that enables customers of a bank or other financial institution to conduct a range of financial transactions through the financial ins ...
,
cloud storage, restricted-access
social-media pages and profiles, some
web forums
An Internet forum, or message board, is an online discussion site where people can hold conversations in the form of posted messages. They differ from chat rooms in that messages are often longer than one line of text, and are at least temporar ...
and code language that require registration for viewing content. It also includes
paywalled services such as
video on demand
Video on demand (VOD) is a media distribution system that allows users to access videos without a traditional video playback device and the constraints of a typical static broadcasting schedule. In the 20th century, broadcasting in the form of ...
and some online magazines and newspapers.
Terminology
The first conflation of the terms "deep web" with "
dark web" happened during 2009 when deep web search terminology was discussed together with illegal activities occurring on the
Freenet
Freenet is a peer-to-peer platform for censorship-resistant, anonymous communication. It uses a decentralized distributed data store to keep and deliver information, and has a suite of free software for publishing and communicating on the Web ...
and
darknet
A dark net or darknet is an overlay network within the Internet that can only be accessed with specific software, configurations, or authorization, and often uses a unique customized communication protocol. Two typical darknet types are social ne ...
. Those criminal activities include the commerce of personal passwords,
false identity documents, drugs,
firearms, and
child pornography
Child pornography (also called CP, child sexual abuse material, CSAM, child porn, or kiddie porn) is pornography that unlawfully exploits children for sexual stimulation. It may be produced with the direct involvement or sexual assault of a ...
.
Since then, after their use in the media's reporting on the black-market website
Silk Road, media outlets have generally used 'deep web'
synonymously with the
dark web or
darknet
A dark net or darknet is an overlay network within the Internet that can only be accessed with specific software, configurations, or authorization, and often uses a unique customized communication protocol. Two typical darknet types are social ne ...
, a comparison some reject as inaccurate
and consequently has become an ongoing source of confusion. ''
Wired
''Wired'' (stylized as ''WIRED'') is a monthly American magazine, published in print and online editions, that focuses on how emerging technologies affect culture, the economy, and politics. Owned by Condé Nast, it is headquartered in San ...
'' reporters
Kim Zetter
Kim Zetter is an American investigative journalist and author who has covered cybersecurity and national security since 1999. She has broken numerous stories over the years about NSA surveillance, WikiLeaks, and the hacker underground, including ...
and
Andy Greenberg
Andy Greenberg is a technology journalist serving as a senior writer at ''Wired'' magazine. He previously worked as a staff writer at ''Forbes'' magazine and as a contributor for Forbes.com. He has published the books '' This Machine Kills Secrets ...
recommend the terms be used in distinct fashions. While the deep web is a reference to any site that cannot be accessed by a traditional search engine, the dark web is a portion of the deep web that has been hidden intentionally and is inaccessible by standard browsers and methods.
Non-indexed content
Bergman, in a paper on the deep web published in ''The Journal of Electronic Publishing'', mentioned that Jill Ellsworth used the term ''
Invisible Web
The deep web, invisible web, or hidden web are parts of the World Wide Web whose contents are not indexed by standard web search-engine programs. This is in contrast to the "surface web", which is accessible to anyone using the Internet. Com ...
'' in 1994 to refer to websites that were not registered with any search engine.
Bergman cited a January 1996 article by Frank Garcia:
It would be a site that's possibly reasonably designed, but they didn't bother to register it with any of the search engines. So, no one can find them! You're hidden. I call that the invisible Web.
Another early use of the term ''Invisible Web'' was by Bruce Mount and Matthew B. Koll of
Personal Library Software
AOL (stylized as Aol., formerly a company known as AOL Inc. and originally known as America Online) is an American web portal and online service provider based in New York City. It is a brand marketed by the current incarnation of Yahoo! Inc. ...
, in a description of the No. 1 Deep Web program found in a December 1996 press release.
[@1 started with 5.7 terabytes of content, estimated to be 30 times the size of the nascent World Wide Web; PLS was acquired by AOL in 1998 and @1 was abandoned. ]
The first use of the specific term ''deep web'', now generally accepted, occurred in the aforementioned 2001 Bergman study.
[
]
Indexing methods
Methods that prevent web pages from being indexed by traditional search engines may be categorized as one or more of the following:
# Contextual web: pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).
# Dynamic content: dynamic pages, which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge
Domain knowledge is knowledge of a specific, specialized discipline or field, in contrast to general (or domain-independent) knowledge. The term is often used in reference to a more general discipline—for example, in describing a software engin ...
.
# Limited access content: sites that limit access to their pages in a technical manner (e.g., using the Robots Exclusion Standard
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the site they are allowed to visit.
Th ...
or CAPTCHA
A CAPTCHA ( , a contrived acronym for "Completely Automated Public Turing test to tell Computers and Humans Apart") is a type of challenge–response test used in computing to determine whether the user is human.
The term was coined in 2003 b ...
s, or no-store directive, which prohibit search engines from browsing them and creating cached copies). Sites may feature an internal search engine for exploring such pages.
# Non-HTML/text content: textual content encoded in multimedia (image or video) files or specific file formats
A file format is a standard way that information is encoded for storage in a computer file. It specifies how bits are used to encode information in a digital storage medium. File formats may be either proprietary or free.
Some file formats ...
not recognised by search engines.
# Private web: sites that require registration and login (password-protected resources).
# Scripted content: pages that are accessible only by links produced by JavaScript
JavaScript (), often abbreviated as JS, is a programming language that is one of the core technologies of the World Wide Web, alongside HTML and CSS. As of 2022, 98% of websites use JavaScript on the client side for webpage behavior, of ...
as well as content dynamically downloaded from Web servers via Flash
Flash, flashes, or FLASH may refer to:
Arts, entertainment, and media
Fictional aliases
* Flash (DC Comics character), several DC Comics superheroes with super speed:
** Flash (Barry Allen)
** Flash (Jay Garrick)
** Wally West, the first Kid F ...
or Ajax
Ajax may refer to:
Greek mythology and tragedy
* Ajax the Great, a Greek mythological hero, son of King Telamon and Periboea
* Ajax the Lesser, a Greek mythological hero, son of Oileus, the king of Locris
* ''Ajax'' (play), by the ancient Gree ...
solutions.
# Software: certain content is hidden intentionally from the regular Internet, accessible only with special software, such as Tor
Tor, TOR or ToR may refer to:
Places
* Tor, Pallars, a village in Spain
* Tor, former name of Sloviansk, Ukraine, a city
* Mount Tor, Tasmania, Australia, an extinct volcano
* Tor Bay, Devon, England
* Tor River, Western New Guinea, Indonesia
Sc ...
, I2P
The Invisible Internet Project (I2P) is an anonymous network layer (implemented as a mix network) that allows for censorship-resistant, peer-to-peer communication. Anonymous connections are achieved by encrypting the user's traffic (by using ...
, or other darknet software. For example, Tor allows users to access websites using the .onion
.onion is a special-use top level domain name designating an anonymous onion service, which was formerly known as a "hidden service", reachable via the Tor network. Such addresses are not actual DNS names, and the .onion TLD is not in the I ...
server address anonymously, hiding their IP address.
# Unlinked content: pages which are not linked to by other pages, which may prevent web crawling
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spid ...
programs from accessing the content. This content is referred to as pages without backlink
A backlink is a link from some other website (the referrer) to that web resource (the referent). A ''web resource'' may be (for example) a website, web page, or web directory.
A backlink is a reference comparable to a citation. The quantity, q ...
s (also known as inlinks). Also, search engines do not always detect all backlinks from searched web pages.
# Web archives: Web archival services such as the Wayback Machine
The Wayback Machine is a digital archive of the World Wide Web founded by the Internet Archive, a nonprofit based in San Francisco, California. Created in 1996 and launched to the public in 2001, it allows the user to go "back in time" and see ...
enable users to see archived versions of web pages across time, including websites that have become inaccessible and are not indexed by search engines such as Google. The Wayback Machine may be termed a program for viewing the deep web, as web archives that are not from the present cannot be indexed, as past versions of websites are impossible to view by a search. All websites are updated at some time, which is why web archives are considered Deep Web content.
Content types
While it is not always possible to discover directly a specific web server's content so that it may be indexed, a site potentially can be accessed indirectly (due to computer vulnerabilities).
To discover content on the web, search engines use web crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spid ...
s that follow hyperlinks through known protocol virtual port numbers
In computer networking, a port is a number assigned to uniquely identify a connection endpoint and to direct data to a specific service. At the software level, within an operating system, a port is a logical construct that identifies a specific ...
. This technique is ideal for discovering content on the surface web but is often ineffective at finding deep web content. For example, these crawlers do not attempt to find dynamic pages that are the result of database queries due to the indeterminate number of queries that are possible. It has been noted that this can be overcome (partially) by providing links to query results, but this could unintentionally inflate the popularity of a site of the deep web.
DeepPeep
DeepPeep was a search engine that aimed to crawl and index every database on the public Web. Unlike traditional search engines, which crawl existing webpages and their hyperlinks, DeepPeep aimed to allow access to the so-called Deep web, World ...
, Intute
Intute was a free Web service aimed at students, teachers, and researchers in UK further education and higher education. Intute provided access to online resources, via a large database of resources. Each resource was reviewed by an academic spec ...
, Deep Web Technologies, Scirus, and Ahmia.fi
Ahmia is a clearnet search engine for Tor's hidden services created by Juha Nurmi.
Overview
Developed during the 2014 Google Summer of Code with support from the Tor Project, the open source
Open source is source code that is made free ...
are a few search engines that have accessed the deep web. Intute ran out of funding and is now a temporary static archive as of July 2011. Scirus retired near the end of January 2013.
Researchers have been exploring how the deep web can be crawled in an automatic fashion, including content that can be accessed only by special software such as Tor
Tor, TOR or ToR may refer to:
Places
* Tor, Pallars, a village in Spain
* Tor, former name of Sloviansk, Ukraine, a city
* Mount Tor, Tasmania, Australia, an extinct volcano
* Tor Bay, Devon, England
* Tor River, Western New Guinea, Indonesia
Sc ...
. In 2001, Sriram Raghavan and Hector Garcia-Molina (Stanford Computer Science Department, Stanford University) presented an architectural model for a hidden-Web crawler that used important terms provided by users or collected from the query interfaces to query a Web form and crawl the Deep Web content. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho of UCLA
The University of California, Los Angeles (UCLA) is a public land-grant research university in Los Angeles, California. UCLA's academic roots were established in 1881 as a teachers college then known as the southern branch of the California ...
created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms. Several form query languages (e.g., DEQUEL) have been proposed that, besides issuing a query, also allow extraction of structured data from result pages. Another effort is DeepPeep, a project of the University of Utah
The University of Utah (U of U, UofU, or simply The U) is a public research university in Salt Lake City, Utah. It is the flagship institution of the Utah System of Higher Education. The university was established in 1850 as the University of De ...
sponsored by the National Science Foundation
The National Science Foundation (NSF) is an independent agency of the United States government that supports fundamental research and education in all the non-medical fields of science and engineering. Its medical counterpart is the National ...
, which gathered hidden-web sources (web forms) in different domains based on novel focused crawler techniques.
Commercial search engines have begun exploring alternative methods to crawl the deep web. The Sitemap Protocol (first developed, and introduced by Google in 2005) and OAI-PMH The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a protocol developed for harvesting metadata descriptions of records in an archive so that services can be built using metadata from many archives. An implementation of OAI- ...
are mechanisms that allow search engines and other interested parties to discover deep web resources on particular web servers. Both mechanisms allow web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not linked directly to the surface web. Google's deep web surfacing system computes submissions for each HTML form and adds the resulting HTML pages into the Google search engine index. The surfaced results account for a thousand queries per second to deep web content. In this system, the pre-computation of submissions is done using three algorithms:
# selecting input values for text search inputs that accept keywords,
# identifying inputs that accept only values of a specific type (e.g., date) and
# selecting a small number of input combinations that generate URLs suitable for inclusion into the Web search index.
In 2008, to facilitate users of Tor hidden services in their access and search of a hidden .onion
.onion is a special-use top level domain name designating an anonymous onion service, which was formerly known as a "hidden service", reachable via the Tor network. Such addresses are not actual DNS names, and the .onion TLD is not in the I ...
suffix, Aaron Swartz
Aaron Hillel Swartz (November 8, 1986 – January 11, 2013) was an American computer programmer, entrepreneur, writer, political organizer, and Internet hacktivist. A prolific programmer, Swartz helped develop the web feed format RSS, the tech ...
designed Tor2web
Tor2web (pronounced "Tor to Web") is a software project to allow Tor hidden services to be accessed from a standard browser without being connected to the Tor network. It was created by Aaron Swartz and Virgil Griffith.
History
Tor is a networ ...
—a proxy application able to provide access by means of common web browsers. Using this application, deep web links appear as a random sequence of letters followed by the .onion top-level domain
A top-level domain (TLD) is one of the domains at the highest level in the hierarchical Domain Name System of the Internet after the root domain. The top-level domain names are installed in the root zone of the name space. For all domains in ...
.
See also
* DARPA's Memex program
* Deep linking
In the context of the World Wide Web, deep linking is the use of a hyperlink that links to a specific, generally searchable or indexed, piece of web content on a website (e.g. "http://example.com/path/page"), rather than the website's home page (e ...
References
Further reading
* .
* .
* .
* .
* .
*
* .
*
*
*
*
* Shestakov, Denis (June 2008).
Search Interfaces on the Web: Querying and Characterizing
'. TUCS Doctoral Dissertations 104, University of Turku
* .
* .
*
External links
{{Authority control
Dark web
Internet search engines
Internet terminology
2000s neologisms