Explore relationships

Robots Exclusion Protocol

The Robots Exclusion Protocol (often referred to by the filename used to implement it, robots.txt) is a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.

Use + to expand a branch. Click a topic name to open its summary.