| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe and Support WebmasterWorld|
|Question about robots.txt file|
It is blank
| 1:05 am on Oct 12, 2009 (gmt 0)|
I have a blank robots.txt file, and I was wondering what I need to put in there to allow all bots and spiders to crawl around on all the pages, or should I just leave it how it is, blank?
In my meta tags I have the line:
<META name="robots" content="index,follow"> but I also want to make sure in the robots.txt file that I allow them to crawl around.
Thanks for your help.
| 2:45 am on Oct 12, 2009 (gmt 0)|
For he robots.txt the entries you add are to dissalow access, if it is blank it will not effect spidering. The bots will see this as "no restrictions"
A lot of people place a blank robots.txt file to eliminate 404 issues when bots request the non existant file.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved