Free Republic
Browse · Search
General/Chat
Topics · Post Article

To: Halls

no, it doesn't.

robots.txt puts limits on what pages the web search engine 'spiders' can read and catalog. No robots.txt or spiders ignoring it = all pages spidered, big use of resources.


10 posted on 11/02/2004 11:55:02 AM PST by flashbunny (Every thought that enters my head requires its own vanity thread.)
[ Post Reply | Private Reply | To 6 | View Replies ]


To: flashbunny
robots.txt puts limits on what pages the web search engine 'spiders' can read and catalog. No robots.txt or spiders ignoring it = all pages spidered, big use of resources.

Does that mean something like Google can't or isn't suppose to index the site?

34 posted on 11/02/2004 12:30:55 PM PST by ClintonBeGone (Take the first step in the war on terror - defeat John Kerry)
[ Post Reply | Private Reply | To 10 | View Replies ]

Free Republic
Browse · Search
General/Chat
Topics · Post Article


FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson