homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

puzzling crawl

10+ Year Member

Msg#: 390 posted 9:14 pm on Mar 23, 2004 (gmt 0)

AskJeeves visited my site yesterday - to the best of my knowledge, the first time the bot was ever there. It got robots.txt and the home page.
Today it was back, but it requested a whole bunch of obsolete pages, each of which returned a 300 code (they used to be .htm, are now .php), but the bot didn't go for the new pages. I'm scratching my head as to what happened - there are no links to those old pages to be found in ANY document in my webspace.
I did notice that the request for the homepage came through as HTTP/1.0, and the number of bytes was about 50 lower than requests for the same page as HTTP/1.1 (but I think this is normal? I don't understand the difference).
Any ideas as to what could be going on?



WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 390 posted 1:30 am on Mar 24, 2004 (gmt 0)

It is very doubtful that it was Jeeves's first visit. How long ago did you make the switch to php?


10+ Year Member

Msg#: 390 posted 4:22 am on Mar 24, 2004 (gmt 0)

About a month ago. But if AJ picked up my home page yesterday, wouldn't they use it instead of an older version? By the way, they didn't return to robots.txt today, although they did yesterday. Is that typical bot behavior (to not check robots.txt every time)?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved