Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Question about robots.txt file

It is blank



1:05 am on Oct 12, 2009 (gmt 0)

5+ Year Member

I have a blank robots.txt file, and I was wondering what I need to put in there to allow all bots and spiders to crawl around on all the pages, or should I just leave it how it is, blank?

In my meta tags I have the line:
<META name="robots" content="index,follow"> but I also want to make sure in the robots.txt file that I allow them to crawl around.

Thanks for your help.


2:45 am on Oct 12, 2009 (gmt 0)

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

For he robots.txt the entries you add are to dissalow access, if it is blank it will not effect spidering. The bots will see this as "no restrictions"

A lot of people place a blank robots.txt file to eliminate 404 issues when bots request the non existant file.



Featured Threads

Hot Threads This Week

Hot Threads This Month