| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|Question about robots.txt file|
It is blank
I have a blank robots.txt file, and I was wondering what I need to put in there to allow all bots and spiders to crawl around on all the pages, or should I just leave it how it is, blank?
In my meta tags I have the line:
<META name="robots" content="index,follow"> but I also want to make sure in the robots.txt file that I allow them to crawl around.
Thanks for your help.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved