| Question about robots.txt file
It is blank |
iGLOBAL NiCK
msg:4005377 | 1:05 am on Oct 12, 2009 (gmt 0) |
I have a blank robots.txt file, and I was wondering what I need to put in there to allow all bots and spiders to crawl around on all the pages, or should I just leave it how it is, blank?
In my meta tags I have the line:
<META name="robots" content="index,follow"> but I also want to make sure in the robots.txt file that I allow them to crawl around. Thanks for your help.
|
mack
msg:4005395 | 2:45 am on Oct 12, 2009 (gmt 0) |
For he robots.txt the entries you add are to dissalow access, if it is blank it will not effect spidering. The bots will see this as "no restrictions" A lot of people place a blank robots.txt file to eliminate 404 issues when bots request the non existant file. Mack.
|
|
|
