A text file inside the root directory that directs web robots (usually Yahoo crawlers) which web pages or portions of a website to examine and dismiss.
A text file inside the root directory that directs web robots (usually Yahoo crawlers) which web pages or portions of a website to examine and dismiss.