My Spider report showing bots getting error and leaving

Forum Moderators: open

Message Too Old, No Replies

My Spider report showing bots getting error and leaving

Unknown error is keeping bots out of my site.

juniperwasting

5:06 pm on Mar 18, 2003 (gmt 0)

I keep a daily track of the bots coming and going from my site, and until the week before last things were doing ok. Now I am in big trouble, and I cannot determine the problem. According to the reports, googlebot, and all others, are reading 2 pages and leaving due to an error. I do not know what this error is, and I desperately need to find out what it is. I have made only one big change to the site, the addition of a new php menu on the product pages. This should not be the cause, as it is not on any of the department pages, nor the index page. I’m stumped, the bot reads my robots.txt (which I have validated), and my site map. Is there a way to determine what the error is? Help!

Dreamquick

5:15 pm on Mar 18, 2003 (gmt 0)

Whats the error code they are getting? (e.g. a normal page will be a 200, a redirect a 301/302 etc.)

How do you know they are getting an error and that they are not just requesting robots + another page, which is the sort of behaviour they sometimes exhibit.

- Tony

juniperwasting

5:20 pm on Mar 18, 2003 (gmt 0)

I cannot determine the code for the error. The software I am using to parse out the log files is just not giving it up to me.

jdMorgan

5:37 pm on Mar 18, 2003 (gmt 0)

Be the Googlebot... :)

Use wannabrowser with the user-agent set to "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
This is an easily-found free web-based user-agent tool.

Jim

juniperwasting

5:56 pm on Mar 18, 2003 (gmt 0)

Jim-

Thanks for the wannabrowser, but now I am more confused. The bot can see the site, down to the last tag. This error problem is almost standard on all the bots that have come to me in the last week, week and a half. I have to wonder if my software is giving me false reports.

-Will

littleman

6:12 pm on Mar 18, 2003 (gmt 0)

Hey Jim, you should get in the habit of looking at your actual logs so you could tell exactly what is going on. For instance your errors may actually be 304s which mean 'Not Modified' -- if a smart bot (like googlebot) comes in and hits the index page and gets a 304 it will often assume that your site has not been modified since the last crawl and it will save itself the work of crawling your site.

Learning a bit of shell (especially how to use grep) will make on the fly log crunching much easier.

juniperwasting

6:57 pm on Mar 18, 2003 (gmt 0)

After crunching through the log myself, I am seeing 200 error. I think this is a run time issue. Now I have to figure out what to do about that! Oi

wilderness

7:22 pm on Mar 18, 2003 (gmt 0)

"200" not an error.
A Normal page view.

This link although it says IE errors applies to logs.
http: //members.tripod.com/mr_tweaks/mytweaks/ieerrors.htm

juniperwasting

8:14 pm on Mar 18, 2003 (gmt 0)

Well I am just baffled. I have gone through my log from yestday, checking just the googlebot IP and found no errors. If 200 is a standard page view, then I should be fine. I have seen some 304 on other bots, but that should not be sending me the mad errors. Humph, I thinks me brain be tired of logs.

wilderness

8:59 pm on Mar 18, 2003 (gmt 0)

If your page is the same as your profile mail?
I see two things that stick out BIG!
1) Your page has no DOCTYPE defined in the head. (This page your reading; view source) <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

I would think if it was generated server side it would still show up?

My pages generated by FP; <meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">

2) In addition and more important. The page has no closing statement for <head> before the tables begin?
there should be a </head>

juniperwasting

9:16 pm on Mar 18, 2003 (gmt 0)

Thanks for looking into it. I can assure you there is a </head> tag on all my pages, i am not that tired. As to the DOCTYPE, these pages are hand coded templates, so no program has spit out a DOCTYPE label, and I have not heard of the importance of it. If it is truly a hinderence, then I will add one post haste....

juniperwasting

9:18 pm on Mar 18, 2003 (gmt 0)

Also, not sure if you saw the right page. I just double checked my profile, and had to re-input the page. Sticky mail me if you want to be sure. :)