homepage Welcome to WebmasterWorld Guest from 54.197.110.151
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library, Charter, Moderator: open

Deprecated - Search Engine Submission Forum

    
Search bots get stuck
After a major redesign of the web site search bots get stuck on the root
groul

10+ Year Member



 
Msg#: 350 posted 5:07 pm on Mar 29, 2002 (gmt 0)


Here's the log:

216.35.116.103 - - [25/Mar/2002:03:33:44 -0500] "GET / HTTP/1.0" 304 - "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]

or

216.239.46.180 - - [25/Mar/2002:05:18:18 -0500] "GET / HTTP/1.0" 200 259 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

and that's all! They don't index the root page, and they don't follow the links!

Any ideas?

 

eljefe3

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 350 posted 6:51 am on Mar 30, 2002 (gmt 0)

Hi groul and welcome to wmw

Have you looked at your robots.txt file and your .htaccess file. Perhaps there's something in there that is preventing them from indexing.

Looking at your two examples it looks to me like slurp crawled it and found the content hadn't changed (code 304)since it last visited the page and googlebot did crawl it (code 200) and the size of your page is 259 bytes.

groul

10+ Year Member



 
Msg#: 350 posted 5:17 pm on Apr 1, 2002 (gmt 0)

> the size of your page is 259 bytes
This helped a lot.

Thank you!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved