Do I need a robots.txt file in my directory?

Google Bot - Whats Up with it?

[jady]

Another question guys - Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.

My site map is well laid out, we use some (external) js and flash (embedded in the HTML). All pages also have text links to the other pages.

Why is googlebot not paying much attention to us anymore? Do I need a robots.txt file in my directory?

Thanks guys!!! :)

[littleman]

No it isn't your lack of a robots.txt. If you haven't modified your pages in a while that your site isn't being crawled because what you are serving up is identical to what's in google's cache. If this is the case try doing a small tweak to your index page.

[bufferzone]

You do need a robots.txt, and you need to put some text in it. My first robots.txt disallowed one (empty)libery just to have some text in it.

The reason for this is that all (most) robots ask for the robots.txt, and if yoy dosn't hav it the robot will recorde an error. Not good

[jady]

Okay, so which one is it - do I or dont I need a robots.txt file. If I do need one, can someone give me an example. Also does this .txt file belong in the root directory (where my index.html is located)?

Thanks again for the help...

[littleman]

I have to respectfully disagree with you Bufferzone, I've worked many sites over the past four years without a robots.txt and I have never had a problem.

[Key_Master]

Actually Littleman is right- you don't need a robots.txt file to be properly indexed by Google. It's helpful for excluding spiders from your website. E.g., if you don't want Google or other spiders to visit areas of your site.

[jady]

OKay - should I have the robots.txt file exclude some external javascript and cgi files? (google wise)

[Key_Master]

I do.

Example:

User-Agent: Googlebot
Disallow: /cgi-bin/
Disallow: /css/
Disallow: /js/

Where /css/ is the directory your CSS files are located, /js/ is where your Javascript files are located, and /cgi-bin/ is where your server side scripts are located.

[mdharrold]

Yes, Jady, it would go in your root directory. Whether or not it is needed is irrelavent. Put it in. If the bot reads it ok, if not ok.

[littleman]

So Jady, going back to the original problem:
>>Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.<<

Are your pages recently modified? Googlebot does a date check, and if your pages are listed as older than the last crawl then googlebot won't do a full spiderring. Google uses a "IF_MODIFIED_SINCE" header in it's requests. Look in your logs for a 304 error. That would be a tell tail singe.