Welcome to WebmasterWorld Guest from 54.196.94.176

Forum Moderators: goodroi

Message Too Old, No Replies

Do I need a robots.txt file in my directory?

Google Bot - Whats Up with it?

     
4:41 am on Feb 24, 2002 (gmt 0)

10+ Year Member



Another question guys - Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.

My site map is well laid out, we use some (external) js and flash (embedded in the HTML). All pages also have text links to the other pages.

Why is googlebot not paying much attention to us anymore? Do I need a robots.txt file in my directory?

Thanks guys!!! :)

4:58 am on Feb 24, 2002 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



No it isn't your lack of a robots.txt. If you haven't modified your pages in a while that your site isn't being crawled because what you are serving up is identical to what's in google's cache. If this is the case try doing a small tweak to your index page.
7:55 pm on Feb 24, 2002 (gmt 0)

10+ Year Member



You do need a robots.txt, and you need to put some text in it. My first robots.txt disallowed one (empty)libery just to have some text in it.

The reason for this is that all (most) robots ask for the robots.txt, and if yoy dosn't hav it the robot will recorde an error. Not good

8:03 pm on Feb 24, 2002 (gmt 0)

10+ Year Member



Okay, so which one is it - do I or dont I need a robots.txt file. If I do need one, can someone give me an example. Also does this .txt file belong in the root directory (where my index.html is located)?

Thanks again for the help...

8:04 pm on Feb 24, 2002 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I have to respectfully disagree with you Bufferzone, I've worked many sites over the past four years without a robots.txt and I have never had a problem.
8:10 pm on Feb 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Actually Littleman is right- you don't need a robots.txt file to be properly indexed by Google. It's helpful for excluding spiders from your website. E.g., if you don't want Google or other spiders to visit areas of your site.
8:13 pm on Feb 24, 2002 (gmt 0)

10+ Year Member



OKay - should I have the robots.txt file exclude some external javascript and cgi files? (google wise)
8:20 pm on Feb 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I do.

Example:

User-Agent: Googlebot
Disallow: /cgi-bin/
Disallow: /css/
Disallow: /js/

Where /css/ is the directory your CSS files are located, /js/ is where your Javascript files are located, and /cgi-bin/ is where your server side scripts are located.

8:33 pm on Feb 24, 2002 (gmt 0)

10+ Year Member



Yes, Jady, it would go in your root directory. Whether or not it is needed is irrelavent. Put it in. If the bot reads it ok, if not ok.
9:58 pm on Feb 24, 2002 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



So Jady, going back to the original problem:
>>Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.<<

Are your pages recently modified? Googlebot does a date check, and if your pages are listed as older than the last crawl then googlebot won't do a full spiderring. Google uses a "IF_MODIFIED_SINCE" header in it's requests. Look in your logs for a 304 error. That would be a tell tail singe.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month