"Some results have been removed" - Site dropped from Bing SERPs

[robzilla]

I was investigating a ~10% traffic drop on one of my sites (10+ years old), and it turns out it's no longer appearing in the search results over at Bing and all Bing-powered sites like Yahoo, AOL and DDG. A site: search returns only "Some results have been removed". In Web search, that is; an Image search still returns thousands of images from the site.

No guideline, malware or other problems reported in Webmaster Tools, no major changes to the site in the past months. I've submitted a support request, but may have to wait up to 10 days for a response, if I get one. Content is original, no scheming or spamming; at most, a bit heavier on the ads lately. The site previously ranked at the top of its niche for most keywords, as it still does on Google, and is one of the most complete and authorative, so I'm a bit stumped by what appears to be a heavy penalty.

Has anyone here had this happen to a site? A quick Google search (heh) indicates it's fairly common on Bing for sites to just disappear like that, and I have yet to find a report where the site eventually recovers.

Here's a visual from Webmaster Tools: [i.imgur.com...] (green is indexed pages, purple is clicks)

[levforever]

Thank you @OhNoSEO
On this thread I think we are all people who have been working with the web for years and years.
Honestly, I didn't even know the existence of the bing webmaster tool. It is confusing and very limited.
I believe that the assistance service is a kind of call center that is limited to forwarding standard email.

By checking the traffic before of de-index, we are talking about ten clicks a day, it is useless to waste so much time. It is clear that they don't have the additional resources to help us webmasters. It seems a very amateur for a company like microsoft.

[robzilla]

Something is wrong in Bing land.

Thanks for joining and sharing your experience, OhNoSEO. I'm glad you're all finding this thread.

It sure is odd that Site Explorer says one thing, i.e. your site has malware, while the support team literally tells you the opposite: "your site doesn't have malware". What the heck are you supposed to do with that? If it's practically impossible for malware to exist on your site, and the support team effectively confirms this, then it has to be a flaw in their system. And if that's the case, then malware may not be the reason you were penalized. Not enough backlinks can't be the reason either, that should only apply to sites not yet discovered (due to a lack of backlinks).

So we're left with those Webmaster Guidelines. I did another deep dive yesterday, reviewing all the URLs in Site Explorer, and managed to find a few small issues related to duplicate content (one of the "things to avoid" in those guidelines). I'm not sure how they got to them, but Bing has indexed a big bunch of URLs that normal users would never access but that still worked, and that had self reinforcing canonical tags that suggested those pages ought to exist, when they should have returned a 404. There was also some weird duplicative parameterization going on, e.g. /product?color=blue&color=blue&color=blue, that the site doesn't link to in any way, but that I wasn't properly responding to either.

I fixed those issues, but they affected only a small section of the site, and ironically those bad URLs are listed in Site Explorer under "Indexed URLs", not under "URLs with guideline issues". So were they a problem? Site Explorer says no, but can I trust it?

I also did a scan of all outbound links and removed a few broken ones, as well as one that redirected to a site that is similarly penalized (showing only "Some results have been removed" for a site: search). However, the source page is not excluded but "Indexed successfully" according to Site Explorer.

So I'm fixing small things here and there, that I may not have found otherwise (thanks, Bing?), but I doubt any of them (or even all together) would warrant a complete removal of the site.

Meanwhile, the number of indexed pages is suddenly dropping rapidly, while the crawl rate remains pretty stable: [imgur.com...]

I wouldn't care a single bit about Bing if they weren't also powering DDG, Ecosia, etc. Even then, I don't know how much I care.

I mentioned a 10% traffic drop in the starter post, but it's actually closer to 8%. About 6% of that is directly from Bing, the Bing-powered sites never sent much traffic. 8% is not huge, but over a year's time the loss of revenue is still significant (8% being roughly 1 of 12 months).

Long story short: robzilla, don't get your hopes up.

I'll lower my expectations for the escalated ticket even further ;-) I'm curious to see if any of the above changes will have any impact, but I highly doubt it.

[Robera]

Great thread. Glad I found it.
My site lost a lot of positions at the end of November. Img: [imgur.com...]
Created support ticket, the response was that they see nothing wrong with the site on their end, but told me that it might be an algorithm update that caused this.

Site:domain.com query also shows "Some results have been removed". I have some thoughts that the site might lost authority as Bing uses machine learning and maybe the machine got something wrong.

Something must have changed at the end of November.

[robzilla]

Site:domain.com query also shows "Some results have been removed".

Does it only show that? Or is it mixed with indexed pages? I ask because you're apparently still getting hits from Bing, which suggests the site isn't completely demoted or penalized.

Here's what I'm seeing: [imgur.com...]

Have you reviewed the data in Site Explorer? Particularly the results when filtered by "URLs with guideline issues" or "URLs with malware".

[Robera]

It's mixed with indexed pages. I see no URLs flagged as malware.
However, there are URLs with guidelines issues. It's a large site, I'm trying to clean up those URLs.

[robzilla]

OK, that might be a different issue, although it's possible for sites to be partially demoted. Has the number of indexed pages and crawl errors remained stable? If so, it seems likely that an algorithm update that cost you some traffic, as indicated by the support team. The message "Some results have been removed" can also apply to results removed for copyright violations, duplicate(-ish) content and other issues. Maybe you'll see some results from cleaning up those URLs with guideline issues.

[robzilla]

Interestingly, here's what Bing says about malware on the page behind the "Some results have been removed" link:

When there is a significant risk of serious harm to the public from purchasing unsafe, counterfeit, and other illegal drugs online (illegal pharmaceuticals) or accessing sites that are reasonably suspected of containing malware, Bing wants to help our users make informed decisions. With this goal in mind, we provide a set of warnings on Bing.com to give our users more information about the dangers of visiting unsafe sites. We will not prevent a user from visiting the sites. However, the warning will caution users of the risks, and provide links to resources where the user can learn more about selecting a safe online pharmacy.

[help.bing.microsoft.com...]

Suggesting malware is not grounds for removal, contradicting the Webmaster Guidelines:

When creating content and managing your site, make sure to not participate in phishing or installing viruses, trojans, or other badware by verifying your content, maintaining your Content Management System, keeping your operating system up to date, and by limiting access to who can publish on your site. Malicious behavior can lead to demotion or even the delisting of your website from our search results.

[levforever]

Meanwhile, the number of indexed pages is suddenly dropping rapidly, while the crawl rate remains pretty stable: [imgur.com...]

Confirm that since the day we added the sitemap the indexed pages are practically half.

[levforever]

My site lost a lot of positions at the end of November. Img: [imgur.com...]

@Robera

I compared your graph with my and the decline starts exactly from 12/13 November, Confirm?

[Robera]

Almost. Nov 14 - 15.

[jbnz]

Been away a while and have just worked my work through the thread ...

Meanwhile, the number of indexed pages is suddenly dropping rapidly, while the crawl rate remains pretty stable: - rob dec 29

Yep, that's now started to happen to me too, started Dec 14th and now down to 25% of original number.

Check if the month that your website was removed from the search results, you received traffic from this site, prod.uhrs.playmsn.com - xnokia dec 21

I only have stats for December but I was paid a visit from here on Dec 10th. The site is intriguing - Universal Human Relevance System. I initially read that as asking if humans are relevant ...

Digging further, it appears that the "Filter: URLs with Malware" in Site Explorer is giving us some hints. - Andem dec 27

This is intriguing, other than webpages and images embedded within, my visitors cannot 'download' anything from my website. However the topic of the website is a piece of software and a common search query is 'where can I download x'. In Site Explorer this filter listed some gzipped sitemap pages which intrigued me. This also led me to discover I was using two sitemap plugins, ooops! Whilst I admit bad site management, hardly a bannable offence? Who knows. Need less to say I have removed one of the plugins, the one creating the gz sitemap files listed.

Re-submitted the request and got a quicker response this time. Apparently it's now been escalated to the engineering team. - rob dec 15

This will be my next step too. I have requested further responses to my initial enquiry but have heard nothing so I shall try a fresh support request.

I am frustrated at the lack of response that any of us appear to have received. I am afraid to say, I believe it's simply because Bing doesn't really care about the small webmaster or possibly even how its search results are managed. I appreciate search engines provide free traffic, but to get no response when your site disappears from the SERPS or a non-response is pretty sad. I get much better support from many other free services.

[Robera]

Quick update.

After digging my analytics as @xnokia5200 suggested I have found a correlation between "Universal Human Relevance System" visits to my site and the traffic drop that I'm seeing. I had auto-ads enabled with Adsense, so now it all makes sense.

I hope they will give me another visit soon lol

[robzilla]

Finally, a bit of good news:

I am happy to provide you information that our Product Review Group succeeded in resolving the issue which prevented your site from showing in our index. After submitting your site to be reviewed, the team has decided to lift the block. Allow up to two to three weeks for your site to be crawled, indexed, and serving again.

I am unable to provide you the specifics of the block, as our Product Review team does not share the details of the block.

It's a [seroundtable.com] boilerplate [reddit.com] message [answers.microsoft.com] that feels a little contradictive. The team has "lift[ed] the block", suggesting either a manual penalty or the overriding of an automated penalty, but the sentence before seems to suggest they resolved an issue on their end. But I may be misreading the text, it's all a little confusingly written.

No idea if any of the changes I made recently have made a difference (none of them seemed to warrant a penalty), or if they just needed more time to get word back from the engineering team.

Anyway, a site: search is beginning to return more and more pages, but so far only less important URLs. The most popular pages have yet to return, along with the search traffic.

Meanwhile, the "URLs with malware" report in Site Explorer is still chock-full of URLs, including some now appearing for the site: search.

[levforever]

Good News! @robzilla
We are still fighting! : D. Indexed pages continue to decrease.

It is not clear, but how bing has indexed everything over the years, including noindex pages , url auto-generated and nosense url.

Your site completely back in the index?

[robzilla]

Your site completely back in the index?

Not yet, still missing most of the primary landing pages, including the homepage. But clicks and indexed pages are rising slowly: [imgur.com...]

Search traffic from Bing now at about 10% of what it was before the site was dropped.

It is not clear, but how bing has indexed everything over the years, including noindex pages , url auto-generated and nosense url.

If you look in Site Explorer, then yes lots of noindex pages are listed there as "indexed", some even that are blocked by robots.txt, but happily I don't see these in a site: search.

I also had a lot of weird URLs appear in Site Explorer, no idea where they came from, but the problem for me was actually that those were returning soft 404s (error page but not 404 HTTP status code -- oops). As soon as I fixed that, they were dropped pretty quickly.

They're also crawling and indexing a lot of pages I don't particularly want them to, like lots of URLs with all kinds of parameters, e.g. all combinations of category filters. I wish Bing Webmaster Tools still had the Ignore URL Parameters tool, now I may have to block these using robots.txt to avoid wasting crawl budget. I've been using rel=canonical on them but that's not really the right approach since the content changes quite a bit with each parameter.

So if you see weird URLs in Site Explorer listed as "indexed", make sure your server is properly responding to those requests.

[bingdude]

@robzilla I'm glad to see you are using Site Explorer [bing.com...] unique to Bing. It is indeed a great way to look at your entire site in addition to inspecting URLs.

[robzilla]

Hi there, bingdude. Good to know you're still perusing these forums.

The Site Explorer has been useful in discovering a few technical issues with the site, that I would not have easily found otherwise, so that's great. However, there's also a lot of confusing (sometimes even contradictory) data in there. We've discussed the "URLs with malware" report in this thread, for example, and frankly it's pretty useless. It doesn't give any actionable data.

Site Explorer is also frequently buggy, sometimes when combined with the Site Inspection tool. One example: when I filter by "Indexed URLs", I see a lot of redirecting URLs listed under the root domain that are labeled as "Indexed". Mostly these are URLs like /catalog?color=blue&page=2, where the canonical URL has a trailing slash, like /catalog/?color=blue&page=2. Nobody links to those URLs without a trailing slash (internally or externally), they all 301 redirect, and they have the appropriate rel=canonical. When I click "Inspect URL", I'm told "The inspected URL is not known to Bing." OK, so Bing has quite a bit of data on a URL (discovered years ago, recently crawled, HTTP code 200, page size, etc.) but it is also... not known to Bing?

In this thread there's also been mention of things like noindexed pages listed as indexed in Site Explorer, or URLs blocked by robots.txt, etc. All of that combined makes me question the validity of the data in Site Explorer, so it would be great to see some improvements there.

In the absence of the old Ignore URL Parameters tool, it would also be helpful to have an alternative for that, because clearly bingbot continues to crawl and index all possible variations of parameterized URLs.

I'm guessing there's not much you can say about this whole ordeal with sites being penalized quite aggressively.

[samwest]

Just double-checked my Bing result...#2 on page one, with a full expanded sub-page listing. Still, never get any Bing traffic, so why bother getting listed on them?

[levforever]

The stats have dropped completely to zero. We don't know what to do.

Site explorer is useful for small sites but with thousands of URLs also not parameterized and difficult to understand.

The absurd thing is that an entire site is de-indexed and there is no note in the webmater tool, no reason, nothing.

We manage millions of monthly users with our network, it is frustrating to try to understand something in this way.

[robzilla]

Still, never get any Bing traffic, so why bother getting listed on them?

If it's not worth it, it's not worth it. I don't think I would have bothered if it wouldn't have reduced my revenue by thousands of dollars a year.

Still, digging into this brought a few small issues to light that I did need to fix, so I think it was worth it even if I hadn't recovered (partially so far).

To summarize what I did in between the blocking and the unblocking:

- Fixed a faulty rel=canonical in a section (couple dozen pages) that must have been confusing the robots. The section has two different names internally, and the canonical tag wasn't programmed to serve the correct one there.

- Fixed an issue where the site returned a clear "Page not found" error page but not the corresponding 404 HTTP Status Code (it served a 200) for certain non-existent URLs. It's possible Bing can't handle soft 404s as well as Google can, and they may have considered those error pages duplicate content. This issue also kept a few spammy-looking URLs that never existed on the site in the "URLs with guideline issues" list in Site Explorer. As soon as I started returning the proper Status Code, that list cleared up within a week or so. What's odd is that for non-existent index pages, Bing would keep tacking on duplicate URL parameters like color=blue&color=blue&color=blue (Site Explorer showed me URLs with at least 20 parameter repetitions), whereas that did not happen with existent index pages.

- In an effort to understand why so many URLs are (still) listed under "URLs with malware", I got to reading about the roll-up system [blogs.bing.com] I mentioned earlier, where Bing might "roll up" a malware detection to affect a whole section of a site, or even the whole site, if the malware is either very dangerous or easily accessible (i.e. not on some paged buried deep in the site). There was one old executable file that was perfectly safe but some anti-virus programs still had an issue with, and that was hosted on a page that many other pages link to, so I figured it could perhaps cause a roll-up and so I swapped out the file with a newly generated one that every AV engine has cleared. Is it a safer file? No, but I guess it's nice to know it won't trigger any false alarms for my users.

So, a full-site penalty for an incorrect rel=canonical in a relatively small section of the site? Unlikely. Then perhaps for returning soft 404s, which Bing was unable to recognize as such and may have considered duplicate content? Possible, but very troubling if so. Finally, a roll-up of a malware detection that was most certainly a false positive? Could be, I have no way of knowing how Bing detects malware. Nothing has changed in the (pretty useless) "URLs with malware" report in Site Explorer since making that change, whereas other changes were picked up pretty quickly.

Those are the only changes I made before the support team told me the site had been cleared, but of course they wouldn't tell me if any of the recent changes I made had an effect, or if they had simply gotten word back from the engineering team and that those people had decided there were no issues after all. In the other support ticket I was told I could contact them again after I had made "significant changes" to the site. I'm not convinced the above, even when combined, should be considered "significant".

All of these "issues" had existed on the site for many years.

The stats have dropped completely to zero. We don't know what to do.

It's very frustrating, yes. I don't fully understand why they cleared my site, but if they hadn't I really wouldn't have known what else I could have done. I'm still quite convinced something is wrong on their end.

[levforever]

I read that you used fixed canonicals on parameterized urls. Google does not recommend this technique. (duplicate content)
Use the exact canonical or URL rewriting htaccess.

A question :do you use hreflang?

[robzilla]

I read that you used fixed canonicals on parameterized urls. Google does not recommend this technique. (duplicate content)
Use the exact canonical or URL rewriting htaccess.

Yes, I realized that I am applying it too broadly sometimes. I want search engines to ignore things like the sorting parameter of an index page (which affects the content on the page but not in a meaningful way for search engines), but rel=canonical is not the way to do that. Unfortunately, there aren't any good alternatives. Maybe I'll open up a new thread about that soon, to see how everyone's handling that.

A question :do you use hreflang?

Not on the site that was blocked, no. It's a single-language site.

[xnokia5200]

Hey guys, okay now when I run a full site scan, it says that the Bing bot IP is blocked so maybe I can try and whitelist them or something. I use cloudfare for DNS and Cloudways hosting. Any suggestion on how to do this? Kinda lost TBH!

[xnokia5200]

Here is the image for refernce! [paste.pics...]

[robzilla]

A different issue, of course, but blocking Bingbot is obviously not great for rankings ;-)

If you use CloudFlare only for DNS, it won't block any traffic. If you use it as a CDN or for security, i.e. when you pass all traffic through CF servers, then it might. Check your firewall rules if you have them, as well as the CloudFlare community forums (quite a few posts about Bingbot and Googlebot getting blocked, mostly due to faulty configurations).

It can also be blocked by your CMS, probably not by your web host. If you're using WordPress, check your plug-ins (e.g. WordFence).

--

Meanwhile, it's been two weeks since the block was lifted and traffic now seems to plateau at about 50% of what it used to be. Still, it's nice that have that back, at least. Will give it a few more weeks to settle before I start troubleshooting for that missing 50%.

[Ethan_Rob]

Bing's Site Scan does not use Bingbot IP, you can see this sentence at the bottom of the Site Scan page:

Site Scan crawler is not yet using Bingbot IP addresses, we plan to switch to Bingbot IP addresses later this year.

So Cloudflare's firewall blocks Bing Site Scan.

All you need to do is temporarily disable the 100202 and 100202_2 rules under Cloudflare Specials in Cloudflare's firewall. After the scan is over, you can restore the two rules.

[levforever]

The graph is completely zero. No message from support.

Since Microsoft is free to decide, we have decided to block bingbot on our network.

It is useless to waste precious time, especially in a pandemic period.

[Andem]

It's nice of @bingdude to join us on this thread. A quick update: I got a response months ago completely unrelated to our issue and the followups have been ignored, so I'm wondering what the next step is to take.

The issue seems to be pages not included because most pages have been marked as "malware" but this must be a huge bug. We don't even have any type of ads on most pages.

[robzilla]

The issue seems to be pages not included because most pages have been marked as "malware" but this must be a huge bug. We don't even have any type of ads on most pages.

Do you host any executable files or anything else that could contain malware? It doesn't have to be ads, there are many types of malware. If it's a malware roll-up, i.e. a malware detection that affects the whole site (or part thereof) rather than just the page the "malware" is on, the prime suspects are pages that many other pages link to and/or that are close to the root of the site. Maybe also run a link checker to see if you have any outdated links (which aren't necessarily suspect, of course, but they can sometimes redirect to shady URLs).

Nothing under "URLs with guideline issue"?

[Andem]

Nothing under "URLs with guideline issue"?

Yep, ads.txt and robots.txt along with a handful of harmless pages on subdomains. Virtually nothing in the "Site Explorer" makes any sense and I've been over it countless times.

I will take your advice and run a link checker.