Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

puzzling crawl

9:14 pm on Mar 23, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 17, 2004
votes: 0

AskJeeves visited my site yesterday - to the best of my knowledge, the first time the bot was ever there. It got robots.txt and the home page.
Today it was back, but it requested a whole bunch of obsolete pages, each of which returned a 300 code (they used to be .htm, are now .php), but the bot didn't go for the new pages. I'm scratching my head as to what happened - there are no links to those old pages to be found in ANY document in my webspace.
I did notice that the request for the homepage came through as HTTP/1.0, and the number of bytes was about 50 lower than requests for the same page as HTTP/1.1 (but I think this is normal? I don't understand the difference).
Any ideas as to what could be going on?
1:30 am on Mar 24, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 108

It is very doubtful that it was Jeeves's first visit. How long ago did you make the switch to php?
4:22 am on Mar 24, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 17, 2004
votes: 0

About a month ago. But if AJ picked up my home page yesterday, wouldn't they use it instead of an older version? By the way, they didn't return to robots.txt today, although they did yesterday. Is that typical bot behavior (to not check robots.txt every time)?

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members