Page is a not externally linkable
- Yahoo
-- Deprecated - Altavista, Alltheweb.com
---- Every week FAST's crawler fetches robots.txt & leaves without crawling


Pushycat - 2:52 pm on Aug 28, 2002 (gmt 0)


I'm glad luma's issue is solved but it bothers me that this user agent has been in my logs nearly every day lately. It always gets my index.asp page first, then it comes back later -- sometimes the next day -- to get my robots.txt file, and then it disappears for one or two days. The robots.txt file does not exclude FAST except for the directories that I exclude all user agents from.


Thread source:: http://www.webmasterworld.com/altavista/541.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com