Massive Bing crawl impacted performance; reason for rankings lost?

[kb73]

In the past couple of months I've seen an exponential increase in crawling from Bing - so much so that it been impacting server performance and taking sites down from time to time, definitely impacting response times for users and bots, both Bing and Google.

They aren't identifying themselves properly all of the time, either via reverse dns or user agent - however, the IP is either owned by Microsoft or originates in Redmond, Washington. Given that earlier this week they crawled over 9.7 million pages from a single site in one day it definitely appears to be Bing.

My question is - if Bing's crawling has been taking my sites down, could this be a possible reason for traffic drops from Google? Starting yesterday I've seen a 10-20% decrease in tag page traffic across 20+ sites, all in different countries, all at the same time. A few potential reasons come to mind - Panda (20+ markets and only affecting a single page type...unlikely), Google downgrade due to bad response times (again, single page type and all at the same time), or general Google algorithm change targeting 'tag' type pages..... Any thoughts would be appreciated...

[tedster]

If a site is regularly dog slow or unavailable, that can affect rankings. However, what you describe sounds a bit more like you are suffering from an algorithm change.

Since the Panda algorithm came into play, many webmasters have suspected that tag pages might be problematic - especially those that offer little content. If those tag pages that concern you offer little value to a visitor, you should at least consider removing them.

[Robert Charlton]

Panda (20+ markets and only affecting a single page type...unlikely)

kb73 - If I were looking at an algorithm change over a wide variety of markets that 'only affected a single page type, and all at the same time', I'd come to the opposite conclusion you do. I'd assume that this particular page type probably had a lot to do with the problems.

If this page type were the only type of page that was slow, I might further assume that server load could be hurting them, and I'd look to see why these pages slowed down more than others. Perhaps the sizes of the tags are computed dynamically as the pages are served, or some such. You don't say, though, that this is the case.

I think it's much more likely, as tedster suggests, that the shallow content of the tag pages is the culprit.

[kb73]

Yes - the pages are computed dynamically at the time the request comes in. It's interesting that I'm not seeing anyone else mentioning that tag pages are being downgraded. Whilst all sites appear to have been hit, it does not appear to be all at exactly the same time - some countries went together and others the day prior. A similar impact is also being seen across multiple languages as well.

About 2 months ago we removed 50%+ of the tag pages on one site as a Panda remedial action and this site has seen the drop as well.

[A1Seo]

What type of sites do you have? Do you scrape content from other sites? It could be that someone reported your site and upon investigation Google spam team decided to put some penalty on your sites.

And if none of the above, has anything changed on your site in last couple of weeks which could be the cause of it?

[Robert Charlton]

It's interesting that I'm not seeing anyone else mentioning that tag pages are being downgraded.

That's because you are perhaps many months late to the discussion. Do a search for...

site:webmasterworld.com tag pages panda

...eg, and you'll see that there have been many discussions on the topic.

[kb73]

The sites are all directory type sites. In addition to our own data we have a lot of user generated data, plus we aggregate and enhance data from a lot of sources (including our own crawling), but don't 'scrape' any content and simply repurpose it on our sites.

There has been a change to the way we display ads on our some of the sites recently and this has increased ad unit CTR by around 10% - however, this is only on about 1/3rd of the sites that have been impacted. Most of the site that have been hit haven't changed their ad configurations yet.

[tangor]

I have a more simplistic point of view: Does your host have enough pipe/bandwidth to cope? Is this a shared or dedicated? Ultimately, you get what you pay for.

Bing hits me 4-1 over Google (currently delivering HALF of my visitors) and no drop in server performance. Then again, I'm not on the cheapie hosts out there so . . .

[kb73]

I wish the answer was that easy - no cheap hosting here. All dedicated boxes on a tier one host with state of the art infrastructure.

[tangor]

This the Google SEO forum. Not Bing. Bing is aggressive these days, but that aggression relates to my previous above. Repeat query as to whether your site/host can giddyup or not, otherwise, Bing is not your culprit. But I will say that if you are "intriguing" to Bing these days, go for it. There's more than one pony in the race!

[kb73]

I appreciate everyone's comments and that this is the Google forum, not Bing. Traffic drop is from Google which is why the post is here, the conincidence is super aggressive Bing crawling. Host can deliver, maybe our app can't and Google has reacted negatively but our analysis makes this look unlikely.

[A1Seo]

You could look into blocking Bing, if the traffic is low from them. Or if you care about the traffic then you can try reducing the crawl rate via Bing webmaster tools. That "should" stop from Bing hitting that hard.

[AlyssaS]

You don't have to block Bing. Verify your website with Bing webmaster tools - they then allow you to set the crawl rate, so you can instruct them to slow down.

I have heard of rogue Bingbots before - if this keeps happening even after you've instructed Bing to slow down, then go to the Bing part of this forum and start a thread - the Bing team visits the webmasterworld and will help you. But first try instructing the bot to crawl slower.

[canuckseo]

Bing bot recognizes crawl delay robots.txt directive:

[bing.com...]

[A1Seo]

I would suggest to go for controlling crawl rate via Bing webmaster rather than crawl-delay. I have seen them defying crawl-delay since beginning.