homepage Welcome to WebmasterWorld Guest from 54.226.235.222
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Is this robots.txt blocking anybody
nalavanje




msg:1527708
 7:54 am on Feb 1, 2004 (gmt 0)

Hi this is how my robots.txt looks like:

User-agent: *
Disallow:

Googlebot is not getting past my homepage. I'm just making sure that my robots.txt is not the problem.
Could anybody verify this for me please?

Thank you

 

RobbieD




msg:1527709
 8:10 am on Feb 1, 2004 (gmt 0)

Looks like you are blocking everyone with that robots.txt. Do you really want to do this?

MarkHutch




msg:1527710
 8:14 am on Feb 1, 2004 (gmt 0)

I'm not expert on this, but I believe your robots.txt file is set to allow all robots complete access to your site.

Allow all is:

User-agent: *
Disallow:

Disallow all is:

User-agent: *
Disallow: /

ncw164x




msg:1527711
 9:35 am on Feb 1, 2004 (gmt 0)

MarkHutch is right you robots.txt file is not blocking any bots so your problem must be elsewhere

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

ncw164x

nalavanje




msg:1527712
 10:39 pm on Feb 1, 2004 (gmt 0)

Thank you all.

Is it normal for Googlebot to crawl just the home page for a week or two (my site was launched on January 15th 2004) and than do deep crawl later. Or this is something unusual?
For the last two weeks Googlebot visits my home page one or twice a day and it crawls only robots.txt and home page.

nalavanje

ncw164x




msg:1527713
 10:51 pm on Feb 1, 2004 (gmt 0)

No it's not unusual for googlebot to do this especially if the site is new, it will visit often but only request the robots.txt file or the index page of your site.

I have noticed this on the latest sites which I have put online that it has taken a few weeks to do a deep crawl

Just got to hang on in there till it happens

ncw164x

moltar




msg:1527714
 11:36 pm on Feb 1, 2004 (gmt 0)

Get some links to your site and Google will crawl your whole site instantly!

nalavanje




msg:1527715
 2:20 pm on Feb 2, 2004 (gmt 0)

I did get some links. Now I have around 18 links pointing to my home page and I also have 2 deep links I obtained in last 3 days. I guess I just need keep waiting.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved