No, Spider! Stay!

What does it mean when they grab your robots.txt page and then leave?

[brina]

I'm a little paranoid that some important spiders aren't able to correctly spider my site.

The Wisenut bot loves the site and has gone very deep over the last few days.

However, Fast, Inktomi and the Googlebot get the robots.txt file and then leave. Almost everyday there are about 5 different spiders that never go further than the robots.txt page.

I ran the robots.txt file through the little checker and it seems fine.

Can anyone shed a little light? I apologize if this has been answered before - I can't think of a better way to search for this question.

Brina

[Receptional]

If wisenut is the only one really crawling your site, see what happens if you take off the robots.txt file for 24 hours. I'd be interested in your findings.

Dixon.

[edited by: Receptional at 5:20 pm (utc) on Nov. 22, 2002]

[crash]

Are you sure your robots.txt isn't disallowing the spiders? That would be the first thing to check.

How long has this been happening, if not that long it's normal. They tend to come and grab the robots.txt (sometimes more than once) before actually crawling a few hours or days later.

[brina]

My robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wwwstat/

It has been happening for the last week or so. I recently overhauled the whole site so alot of the pages found in Google's index currently are no longer there. Hope I'm not confusing the little guy - I do love him so...