homepage Welcome to WebmasterWorld Guest from 54.234.42.16
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library : Charter : Moderator: open

Deprecated - Search Engine Submission Forum

    
Search bots get stuck
After a major redesign of the web site search bots get stuck on the root
groul




msg:702752		 5:07 pm on Mar 29, 2002 (gmt 0)


Here's the log:

216.35.116.103 - - [25/Mar/2002:03:33:44 -0500] "GET / HTTP/1.0" 304 - "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]

or

216.239.46.180 - - [25/Mar/2002:05:18:18 -0500] "GET / HTTP/1.0" 200 259 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

and that's all! They don't index the root page, and they don't follow the links!

Any ideas?

 

eljefe3




msg:702753		 6:51 am on Mar 30, 2002 (gmt 0)

Hi groul and welcome to wmw

Have you looked at your robots.txt file and your .htaccess file. Perhaps there's something in there that is preventing them from indexing.

Looking at your two examples it looks to me like slurp crawled it and found the content hadn't changed (code 304)since it last visited the page and googlebot did crawl it (code 200) and the size of your page is 259 bytes.

groul




msg:702754		 5:17 pm on Apr 1, 2002 (gmt 0)

> the size of your page is 259 bytes
This helped a lot.

Thank you!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved