Welcome to WebmasterWorld Guest from 54.146.169.191

Forum Moderators: goodroi

Message Too Old, No Replies

Is this robots.txt blocking anybody

     
7:54 am on Feb 1, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 29, 2003
posts:69
votes: 0


Hi this is how my robots.txt looks like:

User-agent: *
Disallow:

Googlebot is not getting past my homepage. I'm just making sure that my robots.txt is not the problem.
Could anybody verify this for me please?

Thank you

8:10 am on Feb 1, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:June 29, 2003
posts:363
votes: 0


Looks like you are blocking everyone with that robots.txt. Do you really want to do this?
8:14 am on Feb 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:742
votes: 0


I'm not expert on this, but I believe your robots.txt file is set to allow all robots complete access to your site.

Allow all is:

User-agent: *
Disallow:

Disallow all is:

User-agent: *
Disallow: /

9:35 am on Feb 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


MarkHutch is right you robots.txt file is not blocking any bots so your problem must be elsewhere

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

ncw164x

10:39 pm on Feb 1, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 29, 2003
posts:69
votes: 0


Thank you all.

Is it normal for Googlebot to crawl just the home page for a week or two (my site was launched on January 15th 2004) and than do deep crawl later. Or this is something unusual?
For the last two weeks Googlebot visits my home page one or twice a day and it crawls only robots.txt and home page.

nalavanje

10:51 pm on Feb 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


No it's not unusual for googlebot to do this especially if the site is new, it will visit often but only request the robots.txt file or the index page of your site.

I have noticed this on the latest sites which I have put online that it has taken a few weeks to do a deep crawl

Just got to hang on in there till it happens

ncw164x

11:36 pm on Feb 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 18, 2003
posts:1925
votes: 0


Get some links to your site and Google will crawl your whole site instantly!
2:20 pm on Feb 2, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 29, 2003
posts:69
votes: 0


I did get some links. Now I have around 18 links pointing to my home page and I also have 2 deep links I obtained in last 3 days. I guess I just need keep waiting.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members