Welcome to WebmasterWorld Guest from 23.20.245.192
Forum Moderators: goodroi
Message Too Old, No Replies
Really necessary to add to robots.txt
Junior Member
joined:May 19, 2002
posts:61
votes: 0
Is it really necessary to add the two lines to robots.txt like the AdSense FAQ says? As long as you are not explicitly banning their bot, why is this necessary?
Preferred Member
joined:Mar 5, 2003
posts:636
votes: 0
It's not necessary. If you're not banning their bots (or all bots) you don't need to worry with adding anything to robots.txt.
Join The Conversation
- Register For Free! - Become a Pro Member!
- See forum categories - Enter the Forum
Moderators and Top Contributors
- Moderator List | Top Contributors:This Week, This Month, Mar, Feb, Archive, Top 100 All Time, Top Voted Members
Hot Threads This Week
- Google Updates and SERP Changes - April 2018
- April 2018 AdSense Earnings & Observations
- Google AdSense: Expands "Funding Choices" To Replace Lost Revenue From Ad Blocking
- What Makes Good Web Content Today
- MauiBot
- Google Updates and SERP Changes - March 2018
- Chrome 66 for Windows, Mac and Linux Rolls Out
- Facebook Goes for World-Wide ToS Changes
- Mozilla: Internet Health Report 2018
- How About an RSS Revival
Featured Threads
- Facebook Goes for World-Wide ToS Changes
- Chrome 66 for Windows, Mac and Linux Rolls Out
- Google AdSense: Expands "Funding Choices" To Replace Lost Revenue From Ad Blocking
- What Makes Good Web Content Today
- Mozilla: Internet Health Report 2018
- Facebook Launches Data Abuse Bounty
- How About an RSS Revival
- Webmasters: Distrust of Symantec PKI in Chrome 66
- A Directory is a Utility far more than it is a Business Model
- Mozilla Readies Virtual and Augmented Reality Browser, Firefox Reality