homepage Welcome to WebmasterWorld Guest from 54.161.191.254
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Do I need a robots.txt file in my directory?
Google Bot - Whats Up with it?
jady




msg:1527289
 4:41 am on Feb 24, 2002 (gmt 0)

Another question guys - Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.

My site map is well laid out, we use some (external) js and flash (embedded in the HTML). All pages also have text links to the other pages.

Why is googlebot not paying much attention to us anymore? Do I need a robots.txt file in my directory?

Thanks guys!!! :)

 

littleman




msg:1527290
 4:58 am on Feb 24, 2002 (gmt 0)

No it isn't your lack of a robots.txt. If you haven't modified your pages in a while that your site isn't being crawled because what you are serving up is identical to what's in google's cache. If this is the case try doing a small tweak to your index page.

bufferzone




msg:1527291
 7:55 pm on Feb 24, 2002 (gmt 0)

You do need a robots.txt, and you need to put some text in it. My first robots.txt disallowed one (empty)libery just to have some text in it.

The reason for this is that all (most) robots ask for the robots.txt, and if yoy dosn't hav it the robot will recorde an error. Not good

jady




msg:1527292
 8:03 pm on Feb 24, 2002 (gmt 0)

Okay, so which one is it - do I or dont I need a robots.txt file. If I do need one, can someone give me an example. Also does this .txt file belong in the root directory (where my index.html is located)?

Thanks again for the help...

littleman




msg:1527293
 8:04 pm on Feb 24, 2002 (gmt 0)

I have to respectfully disagree with you Bufferzone, I've worked many sites over the past four years without a robots.txt and I have never had a problem.

Key_Master




msg:1527294
 8:10 pm on Feb 24, 2002 (gmt 0)

Actually Littleman is right- you don't need a robots.txt file to be properly indexed by Google. It's helpful for excluding spiders from your website. E.g., if you don't want Google or other spiders to visit areas of your site.

jady




msg:1527295
 8:13 pm on Feb 24, 2002 (gmt 0)

OKay - should I have the robots.txt file exclude some external javascript and cgi files? (google wise)

Key_Master




msg:1527296
 8:20 pm on Feb 24, 2002 (gmt 0)

I do.

Example:

User-Agent: Googlebot
Disallow: /cgi-bin/
Disallow: /css/
Disallow: /js/

Where /css/ is the directory your CSS files are located, /js/ is where your Javascript files are located, and /cgi-bin/ is where your server side scripts are located.

mdharrold




msg:1527297
 8:33 pm on Feb 24, 2002 (gmt 0)

Yes, Jady, it would go in your root directory. Whether or not it is needed is irrelavent. Put it in. If the bot reads it ok, if not ok.

littleman




msg:1527298
 9:58 pm on Feb 24, 2002 (gmt 0)

So Jady, going back to the original problem:
>>Most months, (throughout the entire month) our site gets hit by google 60-70 times, sometimes more. This month, it has only gotten hit 4 times, with 4 pageviews.<<

Are your pages recently modified? Googlebot does a date check, and if your pages are listed as older than the last crawl then googlebot won't do a full spiderring. Google uses a "IF_MODIFIED_SINCE" header in it's requests. Look in your logs for a 304 error. That would be a tell tail singe.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved