Robots Exclusion Protocol

References

1. ^.
2. ^ . First International Conference on the World Wide Web. Geneva. Archived from the original on 2013-09-27 2013.
3. ^.
4. ^ on October 29, 2013.
5. ^ ". Charlie's Diary. 19 June 2006. Archived from the original on 2013-11-25 2014.
6. ^ 2024.
7. ^.
8. ^.
9. ^ . RFC9309.Proposed Standard.
10. ^.
11. ^ 2026.
12. ^ .
13. ^
14. ^.
15. ^ 2015.
16. ^ 2015.
17. ^ 2025.
18. ^ . govinfo.gov. July 20, 2007 2025.
19. ^.
20. ^.
21. ^ 2013.
22. ^ 2013.
23. ^ 2013.
24. ^ 2013.
25. ^ 2017.
26. ^ 2013.
27. ^ 2013.
28. ^ 2013.
29. ^ 2024.
30. ^ 2022.
31. ^ 2017.
32. ^.
33. ^ 2017.
34. ^.
35. ^ 2026.{{}}: CS1 maint: url-status ()
36. ^ 2025.
37. ^ 2025.
38. ^ 2025.
39. ^.
40. ^ 2015.
41. ^ . National Institute of Standards and Technology. doi:10.6028/NIST.SP.800-123. Archived(PDF) from the original on 2011-10-08 2015.
42. ^ 92. ISBN . Archived from the original on 2016-04-01.
43. ^.
44. ^.
45. ^.
46. ^ 2019.
47. ^ 2019.
48. ^.
49. ^.
50. ^
51. ^ 2016.
52. ^.
53. ^ 2018.
54. ^.
55. ^. How can I ensure the Common Crawl CCBot can crawl my site effectively? The crawler supports the Sitemap Protocol and utilizes any Sitemap announced in the robots.txt file.
56. ^ 2020.
57. ^.
58. ^.
59. ^.
60. ^ .
61. ^ . RFC9309.Proposed Standard. sec. 2.5: Limits.
62. ^.