Page is a not externally linkable
- Search Engines
-- Ask - Teoma
---- A parameter to tell the crawler how how many pages in a second!

Kaushal_Kurapati - 8:23 pm on Feb 10, 2005 (gmt 0)


We support the crawl-delay feature which allows us to fine tune the load imposed by our crawler on web site servers.

This is the specification method in robots.txt (for a 10 second delay between requests):

Crawl-Delay: 10

Thanks,
Kaushal
Search Product Manager, Ask Jeeves

Thread source:: http://www.webmasterworld.com/ask_jeeves_teoma/464.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com