Firefox wants to get rid of your custom 404 error pages

Official plugin takes users to wayback machine instead

[bakedjake]

Mozilla's Test Pilot add-on is rather basic when it comes to that. It reacts only when a 404 not found error is recognized by the add-on. The add-on won't display the prompt if a site returns a different kind of error, or is redirected.

The notification reads: "This page appears to be missing. View a saved version courtesy of the Wayback Machine".

You may click on the link to open the Internet Archive website to read an archived snapshot of the page on the site. Please note that you may middle-click on the link to open the page in a new tab in the Firefox browser.

[ghacks.net...]

[engine]

What! That is just stupid!

[tangor]

404 exists for a reason. This time the boffins at Mozilla have sniffed too much cat nip!

[engine]

I'm not sure it's cat nip, tangor. ;)

This is the sort of thing i'd expect on April 1.

[lucy24]

Uh.... Firefox doesn't know the difference between 404 and 410?

This is worrying.

[keyplyr]

I have not seen this.

[tangor]

Firefox knows, it's the busy boffins trying to assert new frontiers that need no exploring. Sheesh. Just produce a good browser that is fast, secure and user configurable!

(We can dream, right?)

[not2easy]

I noticed that of all the new FF Experimental Add-Ons that one has the fewest "Test Pilots". I'd guess they signed on for the ride so they could give them some of that feedback they're asking for. What's next? A scrollbar that takes you to some random page in your browser's History? That could be fun.

[tangor]

Instead of things like this, I want the "Winning Lotto Ticket" add on. The one that pays off 100% of the time. Now THAT would be something actually useful. :)

[keyplyr]

So what happens if the page is not in the Wayback Machine?

[tangor]

So what happens if the page is not in the Wayback Machine?

Also makes me wonder since I have disallowed (and removed) some sites from wayback YEARS ago. (sites dependent upon Latest and Most Recent Info and the Old Stuff was NOT DESIRED)

[keyplyr]

Tangor... As do I (reason I asked) but according to that article, to my understanding, this add-on only attempts to forward the browser to the Wayback Machine if the page does not use code to redirect to a custom 404 page. So if properly redirected, this gizmo does not kick in and won't affect my sites.

However, surely this is browser hijacking and should be challenged and stopped soon enough?

[blend27]

Pretty cool Idea.

Question is whether WayBackMachine's infrastructure is ready for it? What is next? "Ads On all over" someone else's dirty laundry?

Ye, pick on a small guy, go on.

Would You like to know More? << that came from a SyFy movie.

[motorhaven]

This is simply stupid. My sites return a special 404 page with links to major parts of the site, search box, etc. This would defeat the purpose of it, and have a poorer user experience.

[keyplyr]

My copy of Firefox just updated to 48.0 and says "up to date."

The title of this thread "Firefox wants to get rid of your custom 404 error pages" is a bit misleading IMO. Firefox 48.0 does not do this. The add-on in question, Mozilla's Test Pilot, is not default, in other words, it was not added when the browser updated.

So IMO this is a non-issue, especially if properly coded redirects to custom 404 pages are done either at the server level or the account level.

- - -

Further, when searching Firefox's Get More Add-ons for "Test Pilot" the Mozilla utility returned this message:

About this Add-on
This add-on has now been discontinued. The next update will automatically uninstall itself from user's profiles.

So voila... end of story

[bakedjake]

So IMO this is a non-issue, especially if properly coded redirects to custom 404 pages are done either at the server level or the account level.

The plugin, if installed, prompts the user to redirect to the Wayback Machine if the page is stored in their archive and your server returns a 404. No matter how you have your server configured, if you're returning a 404 status code (regardless of the HTML content you may provide back with it), the plugin prompts the user to redirect to the Wayback Machine.

Further, when searching Firefox's Get More Add-ons for "Test Pilot" the Mozilla utility returned this message:

You must be looking in the wrong place. Look here [testpilot.firefox.com]. Also you can see the source code [github.com] yourself.

The title of this thread "Firefox wants to get rid of your custom 404 error pages" is a bit misleading IMO

That this plugin has been given somewhat "official" test status indicates to me that they as an organization think the state of 404 errors are confusing for users, intend to do something about it, and are rolling out their solution to interested testers. Apologies if you were misled, but I think it's important to know about this stuff before it becomes default.

[edited by: bakedjake at 1:53 pm (utc) on Aug 6, 2016]

[keyplyr]

Well I guess so. I couldn't find that add-on page. I stopped looking when Mozilla gave the discontinued message.

Just installed Firefox for Android directly from testpilot.firefox.com. That add-on either is not included or it doesn't work on the pages I tested.

[bakedjake]

properly coded redirects to custom 404 pages

A couple more notes:

404 is a status code returned by a web server to the client. It can be returned along with HTML content that is rendered by the browser. The plugin detects when this code is returned in the HTTP headers and prompts the user to redirect to the Wayback Machine if the file is found there. Even if you're using a custom 404 page, the prompt is displayed to the user.

The plugin prompt does not appear if you 301/302 redirect in the event of file not found (although you shouldn't be doing this anyway - best practice is to serve a 404 with content).

The prompt does not appear if you serve a 200 OK with HTML content in response to a file not found event. This is not best practice either but many webmasters misconfigure their custom error pages this way. ;-)

The prompt does not appear if you serve another type of code (like 410 Gone).

I have no idea what happens if you use a META refresh tag to handle your custom error page (whether returned with 200 or 404); I didn't test that.

[bakedjake]

Also, interestingly, IA keeps the clickthrough data server side and their disclosure of the information collected through the plugin indicates that they know the clickthrough rate - they show an example here [github.com].

I'm going to write to IA and ask if they'll give me an idea of CTR.

[keyplyr]

Assuming the Mozilla discontinued message is in error and this feature goes forward, it would seem the only way to avoid having our traffic hijacked (or lured away) is to make sure the Wayback Machine does not keep archived copies of our pages.

[not2easy]

When I followed the links in the article I landed on a Test Pilot page at Mozilla. It showed 3 different add ons that volunteers could sign up to "Test Pilot" and this 404 Add On had about 3,600 volunteers at that time vs. 200k - 400k for the other two (sorry did not write it all down). It did not look like anything to worry about but interesting to know what ideas are out there.

[bakedjake]

After having this installed for a bit I think this is probably great for users even though I hate it as a webmaster. The plugin was useful to me twice for non-webmaster reasons in the first hour I had it installed. I now want this as a default feature in my day-to-day browser.

I imagine these prompts see near 100% CTR, because a 404 page (even if it's beautiful, custom, with a splendid call to action or list of useful pages) is simply not that compelling compared to a link to the content I was actually looking for!

[keyplyr]

My objection to the Wayback Machine & the so-called Internet Archive goes back many years. I could easily rant for hours about copyright infringement and their unethical tactics, but that's been done many times with these guys and they're still in operation.

When I see my favorite browser considering aiding and abetting by diverting my traffic to them, I once again feel they've somehow slipped under the door.

I do not feel this is great for users. Personally, if I take a page down, I do it for a reason. Other than bot exploit attempts, all 404s at my sites are typos by the user or poorly written links at remote sites. My custom 404 page answers that and give them a search-site tool to get to the correct page.

How long will it take the prompt to add "did you mean...? Here are some related pages from other sites at the Wayback Machine."

[super70s]

Question is whether WayBackMachine's infrastructure is ready for it?

It would have been a good idea if implemented correctly from the start. It can look like your site on acid, and the wayer back you go the worse it gets.

[blend27]

Ha!

Nice!

Notwithstanding that there is an Army of scarpers out there, na? ;)

[DeSilvaSEO]

Sounds quite a foolish thing to do on three fronts: CRO, UX and SEO.

Search engines expect us to 404 permanently removed/unavailable pages. They encourage us to return custom 404 content driven by data sniffed from the URL. I guess, so long as we start to return 410s then the plugin might not trigger....
If many FF users adopted this plugin, might search engines be more reluctant to land Firefox users on page levels of websites which they know to expire regularly? We might see a shift in entry points for FF users: screwing with CRO and UX.

Custom 404 pages can be a big win for CRO and UX.

And we have very strict bot-blockers, so wayback machine's not been able to crawl our sites for a few years.....so is this plugin going to have to pre-fetch the Wayback Machine page to find out whether there's an archived version of the page? I can't see this working very well TBH. Too many reasons why this is not a good idea IMHO.

[tangor]

Things should just be simple. If a page isn't there (for whatever reason) 404. If it is GONE, 410. Helpful idiots don't need to complicate things by trying to pretty up that side of the web.

And who selected wayback (archive.org) as the goto source? I suspect the index cache pages by the major search engines would be of better value, and more reasonably up to date (and more likely to N(OT have been blocked by the website for such historical records).

[samwest]

Why would they want top server up old outdated and likely incorrect information? Stooooopid

[JS_Harris]

Seriously?

User-agent: ia_archiver
Disallow: /

That, when placed in my robots.txt, takes care of the wayback machine and

<meta name='googlebot' content='noarchive'>

takes care of Google's cache, which I noticed Yahoo has been serving up from Google lately. I've never benefited much from having archived copies of my site out there, in fact I feel it's cost me some visitors. But to go out of their way to force visitors not to see my custom error pages that have helpful links and such in them, game on... blocked.

If they start ignoring robots.txt and meta tags too I'll file DMCAs and block IPs etc... there is no need to be hiding custom error pages, imo. This is stupid imo, what's the payoff?

Webmasters have been losing control of their own sites enough as is.

[keyplyr]

Seriously?

User-agent: ia_archiver
Disallow: /

That, when placed in my robots.txt, takes care of the wayback machine

Not quite, sadly. IA has been documented to circumvent robots.txt and use other UAs to scrape content from web sites.

If you do not want your content published from their servers without your explicit permission, IMO it is necessary to use several methods concurrently:
• disallow their UA via robots.txt
• block their IP ranges from access to your server
• block their UAs from access to your server

And keep a diligent watch to see if they are still getting your content; if so, use the legal processes designed to protect content owners from copyright infringement.