| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Really necessary to add to robots.txt|
| 9:06 pm on Jul 4, 2003 (gmt 0)|
Is it really necessary to add the two lines to robots.txt like the AdSense FAQ says? As long as you are not explicitly banning their bot, why is this necessary?
| 9:54 pm on Jul 4, 2003 (gmt 0)|
It's not necessary. If you're not banning their bots (or all bots) you don't need to worry with adding anything to robots.txt.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved