Forum Moderators: buckworks & skibum

Message Too Old, No Replies

How much of an impact if AdBot can't crawl my landing pages?

         

lina

4:11 pm on Nov 22, 2014 (gmt 0)

10+ Year Member



I'd like to use some of my search results pages as landing pages for certain keywords. However, my search results pages aren't crawlable and I'm not sure how much of a penalty there will be for this. I've heard it will affect my quality score, but I have a few of these keywords already running with QS of 9 or 10, so I'm not sure if it has much impact or not.

Before you ask, Nextopia hosts our search results pages, and despite telling us when we signed up with them that we could use search results pages as landing pages, they are now telling us we need to pay them $700/year to change their robots.txt. I'm trying to assess if it's worth ponying up.

The robots.txt is:
User-Agent: *
Disallow: /

I've also heard that Google will ignore this and still allow AdBot to crawl. Is this true?

RhinoFish

9:17 pm on Nov 22, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Though this is from 2011, my understanding today still matches what it says under the AdWords section:
searchengineland.com/what-ppc-practitioners-should-know-about-robots-txt-files-88670

More here:
[support.google.com...]

The QS issues are so painful, I'd say you should unblock the adsBot.

And for Nextopia, if they are holding you hostage (imo) as you described, the you should probably assess your future with them.

lina

1:08 am on Nov 23, 2014 (gmt 0)

10+ Year Member



I agree, and if the decision was mine they would already be gone.

eWhisper

10:54 am on Dec 2, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That SEL article is still accurate today (I wrote it and keep up closely with these).

Adsbot-Google does ignore global disallows; so you should be OK.

However, Bing's bot does follow standard robots.txt conventions so if you choose to expand into Bing, you might have some QS issues.