Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Search bots get stuck

After a major redesign of the web site search bots get stuck on the root



5:07 pm on Mar 29, 2002 (gmt 0)

Inactive Member
Account Expired


Here's the log: - - [25/Mar/2002:03:33:44 -0500] "GET / HTTP/1.0" 304 - "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]

or - - [25/Mar/2002:05:18:18 -0500] "GET / HTTP/1.0" 200 259 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

and that's all! They don't index the root page, and they don't follow the links!

Any ideas?

6:51 am on Mar 30, 2002 (gmt 0)


WebmasterWorld Administrator 10+ Year Member

joined:July 2, 2000
votes: 0

Hi groul and welcome to wmw

Have you looked at your robots.txt file and your .htaccess file. Perhaps there's something in there that is preventing them from indexing.

Looking at your two examples it looks to me like slurp crawled it and found the content hadn't changed (code 304)since it last visited the page and googlebot did crawl it (code 200) and the size of your page is 259 bytes.


5:17 pm on Apr 1, 2002 (gmt 0)

Inactive Member
Account Expired


> the size of your page is 259 bytes
This helped a lot.

Thank you!


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members