Great spambot trap - but requires a 1x1 pixel image link

Will this hurt?

[bluemi]

I found a really cool and easy to install spambot trap - works like a charm. Only problem is that it requires a 1x1 pixel image linking to a "forbidden" directory ("good" robots are excluded from it with a robots.txt file so that only "bad" bots will follow the link and then get blocked from the site). Any ideas if such a link will hurt my site?

[edited by: tedster at 2:00 am (utc) on Nov. 14, 2006]

[kamikaze Optimizer]

I use it and it seems to work well, it does not hurt you at all.

Just remember to add the deny to the robots.txt a few days before adding the remainder of the code on your site. Some bots only check the robots file every few days.

[bluemi]

Great, thanks, that's good to know. I'm still running it in "test mode" on a minor site, so no worries about the deny in the robots.txt. Looks like a great tool though.

[jpavery]

the smart scrappers obey the robots.txt file.

I did the same on my site (just collected IPs) and I got a lot of IPs.

I got concerned "pre-link fetchers" might be fetching the banned link and thus adding a regular surfer to my blocked list.

Paul

[MThiessen]

1x1 dots are common "spacers" dreamweaver uses them, as well as other editors. 1x1 images are ultra common and not likely to cause problems.

[tedster]

Just to clarify, it's not the 1x1 gif itself that's a concern here, but the fact that it's linked.

[bluemi]

Well, it could have been nice but I just removed it again after one of Google's bots got "trapped" (65.222.176.125). I wasn't aware that Google doesn't obey the directives in the robots.txt...

[bluemi]

Oops, sorry, that was of course not Google but a real spambot... mixed up the lines ;-)

[ann]

Wouldn't that qualify as a hidden link and if discovered could get you penalized by Google?

Ann

[MThiessen]

"Just to clarify, it's not the 1x1 gif itself that's a concern here, but the fact that it's linked."

Ahhh ok, now that is a hidden link... Don't know what I was thinking. 1x1 spacers are common but never used as a link.

[bluemi]

Hi Ann. Yes, that's what kind of worries me too. On the other hand, when I think of all these sites with hidden texts and links that are ranking extremely well in Google I feel that the Googlebots' ability to detect spam are ridiculously overestimated anyway. But then, you never know...

[ann]

Yep,

Shot myself in the foot with that one a long time ago. Used it to link to a site page that I was working on and wanted no one to see just yet. Kerplunk! Bottom of the serps I went :)

Ann

[bouncybunny]

Why not add it as a regular text link at the bottom of your site, with the text "spambots follow this"? ;-)

The chances of scraper bot authors actually looking at your site are pretty remote and the bots are illiterate.

[ionchannels]

I do this, with text at the bottom of my page which says:
Do NOT click here (access will be revoked)
If someone or something clicks, their ip address is put into my firewall, but before that I explain why they have been banned and provide my email if they are a person who clicked the link "accidentally". I usually get 2 or 3 people a week who click it out of curiosity. I collect about 10 spambots every day with it.

[nippi]

ionchannels.

i am curious about this. I don;t have a static ip, and if ran a scraper, and you banned me, you'd not in fact ban me anyway, jsut the next person using that ip.

Seems pointless, you didn't stop me in the first place, you then didn't ban me, jsut some other poor sap.

[jdMorgan]

{quote} Wouldn't that qualify as a hidden link and if discovered could get you penalized by Google? [/quote]

Not if the link target is disallowed by robots.txt. That makes it fairly obvious to GoogleBot that you didn't add the link for tricky ranking purposes.

Linked 1x1 gifs with the link target disallowed by robots.txt are one of the common methods for tracking and logging page accesses. You'll often hear them referred to as "Web beacons" in either Web or e-mails contexts.

I've been running this script (or one like it) since the latter part of 2002. The site it's been on longest hasn't dropped out of the top 3 positions in the SERPs at any time before or since.

Jim

[bluemi]

Thanks Jim, that's very useful information!

[bouncybunny]

Out of interest, is there an easy way for a non-techie to log visitors IP addresses when they visit a specific forbidden directory/page.

AWStats isn't that specific.

Thanks