AskJeeves visited my site yesterday - to the best of my knowledge, the first time the bot was ever there. It got robots.txt and the home page. Today it was back, but it requested a whole bunch of obsolete pages, each of which returned a 300 code (they used to be .htm, are now .php), but the bot didn't go for the new pages. I'm scratching my head as to what happened - there are no links to those old pages to be found in ANY document in my webspace. I did notice that the request for the homepage came through as HTTP/1.0, and the number of bytes was about 50 lower than requests for the same page as HTTP/1.1 (but I think this is normal? I don't understand the difference). Any ideas as to what could be going on?
About a month ago. But if AJ picked up my home page yesterday, wouldn't they use it instead of an older version? By the way, they didn't return to robots.txt today, although they did yesterday. Is that typical bot behavior (to not check robots.txt every time)? Rob