SITE command shows links to HTTPS

... never had SSL Cerificate attached to this site

[blend27]

Several month back I took over a project on a "small brochure type" site.

It was a hacked Worpress site with 14 pages of real info, that is it. To my best knowledge it had never ever had SSL installed on it. Luckily at that point Bing did not index(or even visited according to row logs) all the additional pages created by the hack.

site:www.example.com command returned 15 pages all links to HTTP.

I cleaned it up & moved it to another host(with a static IP), different technology - handwritten code, mobile friendly, responsive design.

The prior host(also with a static IP) where the site was hosted for the past 7 years does not even offer shared SSL.

if I run site:www.example.com in Bing the 14 pages are still shown as in the index but all links point to SSL version of the links, including home page.

Not that I know many people who run SITE command but What Gives?

p.s.
GOOG only shows to links to HTTP from its SITE command for this site.

[Andy Langton]

Both Google and Bing now request HTTPS as a matter of course, which basically means asking for an HTTPS Version and crawling it if it exists. In an era of shared hosting where many sites share the same IP, this means many shared hosts will have some certificate installed, so HTTPS requests are technically valid (although the certs usually throw an error - usually for an invalid hostname).

It's a bit of a technical annoyance in many hosting situations. If https:// works on your site (even with errors) search engines are likely to crawl it. It will disappear eventually if SSL requests completely fail, as seems to be the case now as you describe it.

[blend27]

-- If https:// works on your site (even with errors) --

Unfortunately it does not, even if you add a site as an exception in a browser. After doing so it returns IIS 404.

This is something new this morning: Now if I simply type the domain name into a "Bing search bar" HTTPS version of the root domain plus HTTPS site links are returned. and it shows a CHACHED link pointing to the old site's code after several month of crawling new site.

On top of that since Bing data is fed into DuckDuckGo, those results are pointing to HTTPS now.

I do understand ALL technical aspects of it though but this is straight from an episode of South Park: What seems to the officer problem....

If the site has an invalid certificate(e.g. domain/host mismatch) what is the point of listing the entire sites URLs HTTPS in SERP if no one can connect to it?

IMHO this is disservice to a USERS of Bing first and a big headache for webmasters that hosts sites on shared hosting accounts, never mind trying to explain to a Client(whose site recently got hacked) why the users browser does say THIS SITE IS NOT SECURE(Security certificate problems may indicate an attempt to fool you or intercept any data you send to the server.).

[bakedjake]

I sympathize. It's often impossible to know how people are linking to your site or how bots discover that HTTPS is being served.

After doing so it returns IIS 404.

Your site is misconfigured. If you don't want to serve HTTPS, it needs to be disabled it completely in IIS.

If HTTPS was disabled for your site, you would not receive any type of error from the server (404 or else) when making an HTTPS request, because your server would not be responding to HTTPS requests for your site. You would receive a (local, network) error from your browser that said "could not connect to server".

As Andy said, if you want the HTTPS version to disappear, you need to stop your server from responding to HTTPS requests.

As an alternative, to save face with your client, perhaps consider setting up the secure version properly. You'll need to do it eventually [chromium.org].

[blend27]

Andy, jake,

Thanks for your input, really appreciated.

But my points are:

Why display links to a Content that was NEVER ever accessible by HTTPS and it does not now?


Why send a Bing User on a wild goose chase?


How was Bing able to index HTTPS(and show me a cache of it) without having access to it, a proper certificate?


Why not show link for FTP/NNTP protocol for that matter in SERP?

If I go to goog and search for that domain name everything is peachy.

Not like they(Bing or any other search engine) DO NOT know that it would NOT display(+generate a scary warning) in any modern browsers or am I wrong about something here?

[bakedjake]

It's difficult to give exact answers without further investigation. But to me, given your description, it sounds like the content may have been available in the past via HTTPS inadvertantly at some point and/or linked to or shared.

You don't actually need a "good" (properly issued for the domain in question, unexpired) certificate to establish an HTTPS connection. Clients (including your web browser) can be set to ignore invalid certificates and speak HTTPS anyway. That's what you do when you ignore certificate errors or set sites to your safe zone. One would think that bing would check certificate validity but I don't know if that's done in conjunction with the crawl process or at a later time, and I don't know how frequently it's updated if it's done at separate intervals.

From what you've described it sounds to me like the content was accessible via HTTPS at some point, even if the certificate was invalid, and bing hasn't reindexed the site to pick up the 404s (or the server not responding).

Other than that, I agree about the user experience. It's not great. Also I've been frustrated at times myself with HTTPS site conversions and getting bing to pick redirects up in a reasonable amount of time.

[bakedjake]

One note - I just looked at bing docs and they seem to indicate that their content removal tool will check that content has been removed from the web when you submit it:

[bing.com...]

Maybe another avenue for quick resolution - if you can submit the previously indexed HTTPS URLs that are now returning 404s, bing might pull them quickly.

[blend27]

Just did a quick research for the refs for the past year and Bing traffic is at 1.2% out of all. A link from New York Times(2.5 years old) consistently beats Bing traffic on monthly bases.

I am either going to get SSL Certificate and write off the cost and including time monkeying around time on implementation.

OR just Ban Bing from this site outright, visitors from bing cant access the referenced links anyway.

Have to think what is better for the client at this point, lets see what the clients say.

[keyplyr]

Have to think what is better for the client at this point

With that said, also consider what may be best for the client in the *future.*

Potential is often ignored when we get close to a problem. Stepping back and taking a broad look at the future possibilities is always recommended.

[blend27]

I ended up getting the Certificate for the client after speaking with them.

So far Bing bot has been sniffing around, getting 301 on few URLs but NOT following the redirect. Goog recrawled the old URLs and got all under HTTPS without being instructed in anyway.

I still don't like the way they do it though. If one of my competitors does not have SSL, well actually on a shared server with a shared SSL, I could hypothetically ruin their traffic from Bing, Yahoo and DuckDuckGo and by the time they implement the SSL on their sites they could make a lot of mistakes and it takes a lot to get it done properly.

:(

[blend27]

I guess this is turning into what to expect...!

Now all URLs are listed as, but not pointing to, the versions without forward slash at the end on Bing.

Awesome that I had that redirect in .htaccess - lets catch some Bing scrapers now!

[blend27]

Quick update, 9 days later:

Now search on Bing only returns 2 URLs on the site(same for "site" command) excluding root for the domain: One link point to HTTPS version of "About US" page and one to NONE-HTTPS version of another page that is rarely visited even by site humans,same goes for DuckDuckGo results, Yahoo lists all the pages properly .

All of Bing referring traffic is GONE. Not a single referrer from them for over a week.

Bing crawls the site twice a day, every URL that is present on the site, including images and other assets. If gets a 301 to https, follows it properly.

I tested the sites SSL setup and got an A grade on ssllabs.com/ssltest/.

Goog lists all URLs as expected.

[keyplyr]

What does it say at Bing Webmaster Tools... Sitemaps?

[blend27]

Probably something along the lines of WTF. Clean 1 to 1 redirects from http to https.

They have a 17 year old domain of a Biz that was established in 1896(yes 1896) with hundreds of natural links from authority sites, online newspapers and such, the site has its on WIKI page for 9 years, and now it is under HTTPS.

They listed all URLs under HTTPs before the move. From a technical stand point nothing changes except 301 to https. Let them($M) figure that out.

I am not creating another Microsoft Account or what ever they want to call it.

[blend27]

BTW,

Andy, Jake, KP.

Thanks for Your input.

Always Appreciated!

[keyplyr]

I am not creating another Microsoft Account or what ever they want to call it.

But that is where you point Bing to the correct version of the site. Then you authenticate.

Without a Bing Webmaster Tools account for the site, you end up where you are now... not understanding what they are doing.

[blend27]

@keyplyr,

1. They($M) listed all URLs under HTTPS from the get go without me having proper SSL/TLS enabled on this site AND without Webmaster Account. They spider HTTPS version on a daily bases with NP. I reviewed logs for the past 6 month and there were no requests on port 433, NADA, ZIP.
2. Yahoo which already serves its results from Bing lists all HTTPS links properly to the site.
3. .......They already have ONE page under HTTPS, and send no traffic any way.

I spoke with the client on this issue and they said not to waste any more time on it, as long as if some one searches for the the domain name they come up First.

....So Lets wait and see.

p.s, I have several other sites, in full HTTPS mode and NOT, that never had Bing Webmaster tool account, never. Some smaller sites and some with over 500K active URLs. Yet they do just fine & fine and fine, in Bing Serp.

[blend27]

Just an update on this.

So far every one of the URLs on the site has been crawled by BINGBOT over 50 times, literally. Every one of those requests was served proper content.

all http://example.com/ was redirected to https://www.example.com/ - with a subsequent crawl of https://www.example.com/
all http://www.example.com/ was redirected to https://www.example.com/ - with a subsequent crawl of https://www.example.com/

I have compared Bytes-Sent(sc-bites) from raw IIS log files that are being sent to humans and other bots(GoogleBot and Yandex crawler) and the number is exactly the same for all URLs.

10 days ago I have created a Bing webmaster account for https://www.example.com/ version. Verified ownership and Submitted a proper sitemap. Same sitemap was submitted to Google. I have done Fetch as BingBot for the URLs included in sitemap file. Everything looks perfect. I have tested HTTP URLs with the same tool and get a proper 301 response code with a proper location response header. There are no chained redirects. All 1 to 1. All URLs show as Mobile Friendly with a green/white check-mark next to them in Bing webmaster account.


The result Absolutely nothing.

Not a single URL shows up for SITE command. Searching for a domain name bring up one url from a sub-domain(old blog that has not been updated in 5 years). Same for Bing and Yahoo. DuckDuckGo seems to be puling data from Google(I think), all HTTPS urls are intact. Yandex shows all properly indexed content under HTTPS.

[blend27]

Finally!

Checked the logs this morning, seems like a first referrer was yesterday. Checked site command - all pages are in.

Did not do anything since Sept 23rd, so 3 month of doing nothing actually gets the site in Bing index ;) and Yahoo and DuckDuckGo.