homepage Welcome to WebmasterWorld Guest from 54.234.60.133
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

    
puzzling crawl
RobBroekhuis




msg:30547
 9:14 pm on Mar 23, 2004 (gmt 0)

AskJeeves visited my site yesterday - to the best of my knowledge, the first time the bot was ever there. It got robots.txt and the home page.
Today it was back, but it requested a whole bunch of obsolete pages, each of which returned a 300 code (they used to be .htm, are now .php), but the bot didn't go for the new pages. I'm scratching my head as to what happened - there are no links to those old pages to be found in ANY document in my webspace.
I did notice that the request for the homepage came through as HTTP/1.0, and the number of bytes was about 50 lower than requests for the same page as HTTP/1.1 (but I think this is normal? I don't understand the difference).
Any ideas as to what could be going on?

 

Brett_Tabke




msg:30548
 1:30 am on Mar 24, 2004 (gmt 0)

It is very doubtful that it was Jeeves's first visit. How long ago did you make the switch to php?

RobBroekhuis




msg:30549
 4:22 am on Mar 24, 2004 (gmt 0)

About a month ago. But if AJ picked up my home page yesterday, wouldn't they use it instead of an older version? By the way, they didn't return to robots.txt today, although they did yesterday. Is that typical bot behavior (to not check robots.txt every time)?
Rob

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved