| 4:58 am on Feb 24, 2002 (gmt 0)|
No it isn't your lack of a robots.txt. If you haven't modified your pages in a while that your site isn't being crawled because what you are serving up is identical to what's in google's cache. If this is the case try doing a small tweak to your index page.
| 7:55 pm on Feb 24, 2002 (gmt 0)|
You do need a robots.txt, and you need to put some text in it. My first robots.txt disallowed one (empty)libery just to have some text in it.
The reason for this is that all (most) robots ask for the robots.txt, and if yoy dosn't hav it the robot will recorde an error. Not good
| 8:03 pm on Feb 24, 2002 (gmt 0)|
Okay, so which one is it - do I or dont I need a robots.txt file. If I do need one, can someone give me an example. Also does this .txt file belong in the root directory (where my index.html is located)?
Thanks again for the help...
| 8:04 pm on Feb 24, 2002 (gmt 0)|
I have to respectfully disagree with you Bufferzone, I've worked many sites over the past four years without a robots.txt and I have never had a problem.
| 8:10 pm on Feb 24, 2002 (gmt 0)|
Actually Littleman is right- you don't need a robots.txt file to be properly indexed by Google. It's helpful for excluding spiders from your website. E.g., if you don't want Google or other spiders to visit areas of your site.
| 8:13 pm on Feb 24, 2002 (gmt 0)|
| 8:20 pm on Feb 24, 2002 (gmt 0)|
| 8:33 pm on Feb 24, 2002 (gmt 0)|
Yes, Jady, it would go in your root directory. Whether or not it is needed is irrelavent. Put it in. If the bot reads it ok, if not ok.
| 9:58 pm on Feb 24, 2002 (gmt 0)|
So Jady, going back to the original problem:
>>Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.<<
Are your pages recently modified? Googlebot does a date check, and if your pages are listed as older than the last crawl then googlebot won't do a full spiderring. Google uses a "IF_MODIFIED_SINCE" header in it's requests. Look in your logs for a 304 error. That would be a tell tail singe.