robots.txt
Posted: Fri Jan 28, 2005 7:22 am
A suggestion from balamm to slow down the index bot parties
[quote]I had that problem with MSN for a while but they suggested adding \"crawl delay=600\" to the robots.txt to slow down the spidering a bit and take a load off the server.
Seems to have worked.[/quote]
[quote]I had that problem with MSN for a while but they suggested adding \"crawl delay=600\" to the robots.txt to slow down the spidering a bit and take a load off the server.
Seems to have worked.[/quote]