Alta Vista and robots.txt

need to have one?

[Marcia]

I resubmitted a site that's already in AltaVista because the directory structure was changed. Scooter came by today, looked for robots.txt (there is none) and went no further - twice.

Could this be because there isn't one? Is it better to have a robots.txt in there, even if it's blank?

209.73.164.50 - - [09/Jul/2001:14:34:03 -0400] "GET /robots.txt HTTP/1.0" 404 204 "-" "Scooter/1.0"

TIA, Marcia

[agerhart]

I have had the same thing happening with one of my new sites, and I figured that the best thing to do in this case is just to make the robots.txt and put the allow=all if anything.

Can't hurt, and at least the spider will get what it is searching for.

[bartek]

Marcia, scooter sometimes requests the robots.txt and leaves without going further whether the file is there or not. Give it some time and this "recon mission" will likely be followed by a proper crawl. Um... semi-proper these days...