homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Is this robots.txt blocking anybody

 7:54 am on Feb 1, 2004 (gmt 0)

Hi this is how my robots.txt looks like:

User-agent: *

Googlebot is not getting past my homepage. I'm just making sure that my robots.txt is not the problem.
Could anybody verify this for me please?

Thank you



 8:10 am on Feb 1, 2004 (gmt 0)

Looks like you are blocking everyone with that robots.txt. Do you really want to do this?


 8:14 am on Feb 1, 2004 (gmt 0)

I'm not expert on this, but I believe your robots.txt file is set to allow all robots complete access to your site.

Allow all is:

User-agent: *

Disallow all is:

User-agent: *
Disallow: /


 9:35 am on Feb 1, 2004 (gmt 0)

MarkHutch is right you robots.txt file is not blocking any bots so your problem must be elsewhere

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /



 10:39 pm on Feb 1, 2004 (gmt 0)

Thank you all.

Is it normal for Googlebot to crawl just the home page for a week or two (my site was launched on January 15th 2004) and than do deep crawl later. Or this is something unusual?
For the last two weeks Googlebot visits my home page one or twice a day and it crawls only robots.txt and home page.



 10:51 pm on Feb 1, 2004 (gmt 0)

No it's not unusual for googlebot to do this especially if the site is new, it will visit often but only request the robots.txt file or the index page of your site.

I have noticed this on the latest sites which I have put online that it has taken a few weeks to do a deep crawl

Just got to hang on in there till it happens



 11:36 pm on Feb 1, 2004 (gmt 0)

Get some links to your site and Google will crawl your whole site instantly!


 2:20 pm on Feb 2, 2004 (gmt 0)

I did get some links. Now I have around 18 links pointing to my home page and I also have 2 deep links I obtained in last 3 days. I guess I just need keep waiting.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved