Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Question about robots.txt file

It is blank

1:05 am on Oct 12, 2009 (gmt 0)

New User

5+ Year Member

joined:Oct 10, 2009
votes: 0

I have a blank robots.txt file, and I was wondering what I need to put in there to allow all bots and spiders to crawl around on all the pages, or should I just leave it how it is, blank?

In my meta tags I have the line:
<META name="robots" content="index,follow"> but I also want to make sure in the robots.txt file that I allow them to crawl around.

Thanks for your help.

2:45 am on Oct 12, 2009 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 15, 2001
votes: 29

For he robots.txt the entries you add are to dissalow access, if it is blank it will not effect spidering. The bots will see this as "no restrictions"

A lot of people place a blank robots.txt file to eliminate 404 issues when bots request the non existant file.