Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Really necessary to add to robots.txt

9:06 pm on Jul 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:May 19, 2002
votes: 0

Is it really necessary to add the two lines to robots.txt like the AdSense FAQ says? As long as you are not explicitly banning their bot, why is this necessary?
9:54 pm on July 4, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 5, 2003
votes: 0

It's not necessary. If you're not banning their bots (or all bots) you don't need to worry with adding anything to robots.txt.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members