robots.txt

.txt File Types

Definition

A plain text file placed at the root of a website that instructs web crawlers which pages or sections should or should not be crawled. Follows the Robots Exclusion Protocol standard.