Welcome to WebmasterWorld Guest from 54.205.170.21

Forum Moderators: open

Message Too Old, No Replies

puzzling crawl

   
9:14 pm on Mar 23, 2004 (gmt 0)

10+ Year Member



AskJeeves visited my site yesterday - to the best of my knowledge, the first time the bot was ever there. It got robots.txt and the home page.
Today it was back, but it requested a whole bunch of obsolete pages, each of which returned a 300 code (they used to be .htm, are now .php), but the bot didn't go for the new pages. I'm scratching my head as to what happened - there are no links to those old pages to be found in ANY document in my webspace.
I did notice that the request for the homepage came through as HTTP/1.0, and the number of bytes was about 50 lower than requests for the same page as HTTP/1.1 (but I think this is normal? I don't understand the difference).
Any ideas as to what could be going on?
1:30 am on Mar 24, 2004 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



It is very doubtful that it was Jeeves's first visit. How long ago did you make the switch to php?
4:22 am on Mar 24, 2004 (gmt 0)

10+ Year Member



About a month ago. But if AJ picked up my home page yesterday, wouldn't they use it instead of an older version? By the way, they didn't return to robots.txt today, although they did yesterday. Is that typical bot behavior (to not check robots.txt every time)?
Rob