please help robots.txt

please check and see if this is the correct robots.txt

[apays14]

Hi everyone, I have been around here for a while now but this is my first post. I have a robots.txt file and I'm not sure if it is correct.

User-agent: *
Disallow:

I want all robots to visit and index as many pages on my site. Googlebot has come around the past week but doesn't follow links.

Thanks in advance, Randy

[Macguru]

Hi apays14,

And welcome to WebmasterWorld! [webmasterworld.com]

Your robots.txt file seems perfectly valid to me. It will allow any spider to go anywhere in your site.

It is normal that Googlebot does not follow the links for some time after it crawled the robots.txt file. Give it a little more time.

<added> If you have private files like some statistics files, laying into some unprotected folder, it would be a good idea to ban bots from crawling it. ie:

User-agent: *
Disallow:/stats/
Disallow:/work_in_progress/
Disallow:/private_folder/
</added>

[apays14]

Macguru,

Thanks for the help. Also, do I need a robots.txt file on my site for the robots to spider the site?

[Macguru]

Not really. This file's only purpose is to block access to well behaving spiders to given files or folders. As far as I know, it will not improve spidering in any way.

I always put one to at least free up the error log file from all the 404 caused by spiders scanning for it.

Googlebot generally waits 2 weeks after a first scan before it starts folowing the links. It can take up to 6 weeks.

[WebGuerrilla]

apays14,

Brett has a very cool robots.txt validator tool [searchengineworld.com] you might want to check out.