Firefox - FasterFox extension

help... it triggers "bad bot" traps

[davelms]

from [webmasterworld.com...] :

By the way if you are not using FasterFox, you should be:
[fasterfox.mozdev.org...]
Makes all the little speedup tweaks you think you should be using easy to access.

My site has a "bad bot" trap to catch bots that enter a hidden page on my site, ie to catch any bot that doesn't honour robots.txt rules.

Anyway, that has been in place for a long time and works very well - however, only in the past month has it started to trap genuine users, from the user-agent all Firefox browsers and I couldn't for the life of me work out why. The user-agent revealed nothing other than it being a normal Firefox user.

Anyway, the above thread caused me to check out the FasterFox extension and behold, I fell into the trap set as soon as I reloaded Firefox (my site being my homepage) and was denied access.

FasterFox sets up the pre-fetch functions and downloads links from the current page, and hence took all of the "invisible" pages set up as robot traps (all denied in robots.txt), triggered the trap and denied me access to the site.

Genuis, at least I understand now why; the extension (I'm guessing) must have become more popular in the past month as the trap has denied access to users, counting from zero Firefox users a month ago to 10+ in the last 4 weeks.

Anyway, my point? How I detect this and let these users gain access. The user-agent is unchanged and contains nothing at all that shows it to be a Firefox "pre-fetch" function. Does anyone have any such "bad bot" traps and how do you cater for this extension?

Example user-agent:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7

Thankyou!

[JAB Creations]

Check and see if those Firefox users are downloading Stylesheets, external JavaScript, or image files. If they are not (and you have them on the pages which they are hitting) then those are bots spoofing as Firefox.

[encyclo]

Is the browser sending a prefetch header when loading the linked pages? This is the usual behavior for the non-modified browser. You should see a HTTP header:

X-moz: prefetch

If you do not get this, then the you should file a bug with the developer of the extension, otherwise it risks triggering all sorts of spider traps.

[jdMorgan]

In order to keep Mozilla browsers with prefetch enabled from getting trapped because they prefetch 'bait' URLs on your pages, you'll need to 403 or 404 any requests if the X-moz prefetch header is sent with the request.

For Apache, the following code snippet in .htaccess will suffice:

SetEnvIf X-moz prefetch blockit
<Files *>
Deny from env=blockit
</Files>

If you already have other Allow or Deny directives in your .htaccess file, then you will need to integrate this function with your existing allow/deny logic.

Because this is a browser prefetching links, it won't read or be affected by robots.txt.

Jim

[davelms]

Because this is a browser prefetching links, it won't read or be affected by robots.txt.

Yeah I appreciated it wasn't doing anything 'untoward' just wanted to ensure I didn't exclude these valid users.

All your responses have been most welcome, thankyou for the tips, I will have a go at sorting this out.

Thanks again!

[stapel]

I had similar problems [webmasterworld.com] when Google came out with its Firefox plug-in. I put the following coding in my .htaccess file:

RewriteCond %{HTTP_X-MOZ} ^prefetch [NC,OR] 
RewriteCond %{HTTP:X-MOZ} ^prefetch [NC,OR]

I have both lines because I'm not clear on which is "the" "proper" format. But one or both of these is working, because my prefetch-in-Firefox problem has pretty much disappeared.

Hope that helps a bit.

Eliz.

[CritterNYC]

FasterFox can mess up sites that use .html as the extension for dynamic pages for logouts, adding things to carts, checkouts and things like that... since it will prefetch those pages and log you out, add things to your cart, etc. So, I generally recommend against it. Since you never know...

It does however attempt to avoid this by:

Fasterfox further limits prefetching such that only files with the extension .gif, .htm, .html, .jpeg, .jpg, .pdf, .png, .text, .txt, and .xml are prefetched. This should eliminate all possibility of Enhanced Prefetching messing things up, however, if you should notice undesirable behavior, please file a bug or email me.

So, you could use a .php or .asp or .shtml or something else as your trap. I heard they were attempting to filter out links with signout or logout or something like that... so maybe you could try that as well.