Welcome to WebmasterWorld Guest from 35.175.120.174

Forum Moderators: buckworks & eWhisper & skibum

Message Too Old, No Replies

How much of an impact if AdBot can't crawl my landing pages?

     
4:11 pm on Nov 22, 2014 (gmt 0)

New User

5+ Year Member

joined:Mar 6, 2014
posts: 30
votes: 0


I'd like to use some of my search results pages as landing pages for certain keywords. However, my search results pages aren't crawlable and I'm not sure how much of a penalty there will be for this. I've heard it will affect my quality score, but I have a few of these keywords already running with QS of 9 or 10, so I'm not sure if it has much impact or not.

Before you ask, Nextopia hosts our search results pages, and despite telling us when we signed up with them that we could use search results pages as landing pages, they are now telling us we need to pay them $700/year to change their robots.txt. I'm trying to assess if it's worth ponying up.

The robots.txt is:
User-Agent: *
Disallow: /

I've also heard that Google will ignore this and still allow AdBot to crawl. Is this true?
9:17 pm on Nov 22, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 20, 2006
posts:2177
votes: 94


Though this is from 2011, my understanding today still matches what it says under the AdWords section:
searchengineland.com/what-ppc-practitioners-should-know-about-robots-txt-files-88670

More here:
[support.google.com...]

The QS issues are so painful, I'd say you should unblock the adsBot.

And for Nextopia, if they are holding you hostage (imo) as you described, the you should probably assess your future with them.
1:08 am on Nov 23, 2014 (gmt 0)

New User

5+ Year Member

joined:Mar 6, 2014
posts: 30
votes: 0


I agree, and if the decision was mine they would already be gone.
10:54 am on Dec 2, 2014 (gmt 0)

Moderator This Forum

WebmasterWorld Administrator ewhisper is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 13, 2003
posts:4744
votes: 8


That SEL article is still accurate today (I wrote it and keep up closely with these).

Adsbot-Google does ignore global disallows; so you should be OK.

However, Bing's bot does follow standard robots.txt conventions so if you choose to expand into Bing, you might have some QS issues.