Welcome to WebmasterWorld Guest from 54.162.138.175
Forum Moderators: goodroi
Message Too Old, No Replies
Really necessary to add to robots.txt
Junior Member
joined:May 19, 2002
posts:61
votes: 0
Is it really necessary to add the two lines to robots.txt like the AdSense FAQ says? As long as you are not explicitly banning their bot, why is this necessary?
Preferred Member
joined:Mar 5, 2003
posts:636
votes: 0
It's not necessary. If you're not banning their bots (or all bots) you don't need to worry with adding anything to robots.txt.
Join The Conversation
- Register For Free! - Become a Pro Member!
- See forum categories - Enter the Forum
Moderators and Top Contributors
- Moderator List | Top Contributors:This Week, This Month, June, May, Archive, Top 100 All Time, Top Voted Members
Hot Threads This Week
- Google Updates and SERP Changes - July 2017
- July 2017 AdSense Earnings & Observations
- What would you do if Google SERPs just disappeared
- Google AdSense: No More Pop-Up and Pop Under - New Rule
- Page layout impacts on speed and indexability
- Feb 7, 2017 Google Algorithm Update - studies suggest it was Phantom
- GSC July 13-14 Decrease in Average Positions is a Reporting Change... Not a Rankings Drop
- Google Updates and SERP Changes - June 2017
- AdSense Publisher Policy Violation
- Negative SEO Is Alive and Well
Featured Threads
- WordPress: Vulnerability Scanners WPScan, WPSeku
- Google AdSense: No More Pop-Up and Pop Under - New Rule
- GSC July 13-14 Decrease in Average Positions is a Reporting Change... Not a Rankings Drop
- Report: Facebook to Test Subscription News Service
- Google Search App Gets Algorithmic News Feed
- What would you do if Google SERPs just disappeared
- Twitter Adds Advanced Filter Settings With Better Mute Controls
- Most of Your Website Traffic is Not Human
- Google Updates and SERP Changes - July 2017
- Google AdSense New Native Ads