Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: goodroi

Message Too Old, No Replies

Do I need a robots.txt file in my directory?

Google Bot - Whats Up with it?

     
4:41 am on Feb 24, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 9, 2002
posts:426
votes: 0


Another question guys - Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.

My site map is well laid out, we use some (external) js and flash (embedded in the HTML). All pages also have text links to the other pages.

Why is googlebot not paying much attention to us anymore? Do I need a robots.txt file in my directory?

Thanks guys!!! :)

4:58 am on Feb 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


No it isn't your lack of a robots.txt. If you haven't modified your pages in a while that your site isn't being crawled because what you are serving up is identical to what's in google's cache. If this is the case try doing a small tweak to your index page.
7:55 pm on Feb 24, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 24, 2001
posts:562
votes: 0


You do need a robots.txt, and you need to put some text in it. My first robots.txt disallowed one (empty)libery just to have some text in it.

The reason for this is that all (most) robots ask for the robots.txt, and if yoy dosn't hav it the robot will recorde an error. Not good

8:03 pm on Feb 24, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 9, 2002
posts:426
votes: 0


Okay, so which one is it - do I or dont I need a robots.txt file. If I do need one, can someone give me an example. Also does this .txt file belong in the root directory (where my index.html is located)?

Thanks again for the help...

8:04 pm on Feb 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


I have to respectfully disagree with you Bufferzone, I've worked many sites over the past four years without a robots.txt and I have never had a problem.
8:10 pm on Feb 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


Actually Littleman is right- you don't need a robots.txt file to be properly indexed by Google. It's helpful for excluding spiders from your website. E.g., if you don't want Google or other spiders to visit areas of your site.
8:13 pm on Feb 24, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 9, 2002
posts:426
votes: 0


OKay - should I have the robots.txt file exclude some external javascript and cgi files? (google wise)
8:20 pm on Feb 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


I do.

Example:

User-Agent: Googlebot
Disallow: /cgi-bin/
Disallow: /css/
Disallow: /js/

Where /css/ is the directory your CSS files are located, /js/ is where your Javascript files are located, and /cgi-bin/ is where your server side scripts are located.

8:33 pm on Feb 24, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Mar 19, 2001
posts:220
votes: 0


Yes, Jady, it would go in your root directory. Whether or not it is needed is irrelavent. Put it in. If the bot reads it ok, if not ok.
9:58 pm on Feb 24, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


So Jady, going back to the original problem:
>>Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.<<

Are your pages recently modified? Googlebot does a date check, and if your pages are listed as older than the last crawl then googlebot won't do a full spiderring. Google uses a "IF_MODIFIED_SINCE" header in it's requests. Look in your logs for a 304 error. That would be a tell tail singe.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members