Welcome to WebmasterWorld Guest from 220.127.116.11 , register , login , search , subscribe , help , library , PubCon , announcements , recent posts , open posts Pubcon Website
Search bots get stuck After a major redesign of the web site search bots get stuck on the root groul msg:702752 5:07 pm on Mar 29, 2002 (gmt 0) Here's the log:
18.104.22.168 - - [25/Mar/2002:03:33:44 -0500] "GET / HTTP/1.0" 304 - "-" "Mozilla/5.0 (Slurp/cat; email@example.com; [
22.214.171.124 - - [25/Mar/2002:05:18:18 -0500] "GET / HTTP/1.0" 200 259 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
and that's all! They don't index the root page, and they don't follow the links!
eljefe3 msg:702753 6:51 am on Mar 30, 2002 (gmt 0)
Hi groul and welcome to wmw
Have you looked at your robots.txt file and your .htaccess file. Perhaps there's something in there that is preventing them from indexing.
Looking at your two examples it looks to me like slurp crawled it and found the content hadn't changed (code 304)since it last visited the page and googlebot did crawl it (code 200) and the size of your page is 259 bytes.
groul msg:702754 5:17 pm on Apr 1, 2002 (gmt 0)
> the size of your page is 259 bytes This helped a lot.