robots.txt puts limits on what pages the web search engine 'spiders' can read and catalog. No robots.txt or spiders ignoring it = all pages spidered, big use of resources.
10 posted on 11/02/2004 11:55:02 AM PST by flashbunny
(Every thought that enters my head requires its own vanity thread.)