| 8:10 am on Feb 1, 2004 (gmt 0)|
Looks like you are blocking everyone with that robots.txt. Do you really want to do this?
| 8:14 am on Feb 1, 2004 (gmt 0)|
I'm not expert on this, but I believe your robots.txt file is set to allow all robots complete access to your site.
Allow all is:
Disallow all is:
| 9:35 am on Feb 1, 2004 (gmt 0)|
MarkHutch is right you robots.txt file is not blocking any bots so your problem must be elsewhere
This indicates that nothing is disallowed and the spider can follow all links
To allow a single robot complete access and exclude all others
This would prevent your entire web site from being indexed
| 10:39 pm on Feb 1, 2004 (gmt 0)|
Thank you all.
Is it normal for Googlebot to crawl just the home page for a week or two (my site was launched on January 15th 2004) and than do deep crawl later. Or this is something unusual?
For the last two weeks Googlebot visits my home page one or twice a day and it crawls only robots.txt and home page.
| 10:51 pm on Feb 1, 2004 (gmt 0)|
No it's not unusual for googlebot to do this especially if the site is new, it will visit often but only request the robots.txt file or the index page of your site.
I have noticed this on the latest sites which I have put online that it has taken a few weeks to do a deep crawl
Just got to hang on in there till it happens
| 11:36 pm on Feb 1, 2004 (gmt 0)|
Get some links to your site and Google will crawl your whole site instantly!
| 2:20 pm on Feb 2, 2004 (gmt 0)|
I did get some links. Now I have around 18 links pointing to my home page and I also have 2 deep links I obtained in last 3 days. I guess I just need keep waiting.