| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
To use... or not to use? That is the question
I allow all content on my site to be crawled. With that said, should I simply not use robots.txt, or have one with the following lines :
Would having the latter of the two improve the number of pages crawled by the spiders?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved