PHP Spider Trap - in use.

[kapow]

Re. BirdMan's excellent PHP Spider Trap
[webmasterworld.com...]

It uses this Robots.txt:
User-agent: *
Disallow: /getout.php

This is probably an obvious question but I havn't used a spider trap before: To use the spider trap are you supposed to create a link, say on your index.html to a page in a spider-trap-folder - where the spider-trap-folder has the robots.txt?

Otherwise you are putting a disallowing robots.txt in your root folder - which I don't want to do.

[carneddau]

Hi,

That robots.txt is only disallowing access to the getout.php file. You can only have 1 robots.txt file and that's at the root.

You need to put the robots file up a while before you add the getout file. Then "good" robots shouldn't follow links to the file.

[kapow]

How do the bad spider know to find getout.php?
Do they require a link on a web page such as index.html?

[DanA]

You will need a link to getout.php from index.php or any other page to spot a "bad bot", then you must do something (block it or redirect it) if you don't want it to spider the rest of the site.

[mykel79]

Also note that the link should be unclickable to a human user. You can use javascript for that.

[kapow]

...then you must do something (block it or redirect it) if you don't want it to spider the rest of the site.

I thought the spider trap did the blocking?

link...unclickable to a human user. You can use javascript for that

I'm worried that an invisible link could look like black-hat SEO tricks. If the spider trap requires a link what is the safest way to provide that link without confusing human visitors?

[Warboss Alex]

<a class="hide_me" href="/getout.php"></a>

And in your CSS:

.hide_me { visibility:hidden; }

Your visitors'd never see the link, so they wouldn't be able to click it ..

Having no text between the link means that it won't take up any physical space on the page either.

:)

Alex ...

[kapow]

But its still a hidden link. Wouldn't that attract a SE penalty from Google for SEO tricks?

[Warboss Alex]

A 1x1 transparent gif?

[Warboss Alex]

That'd be the same thing though, right?

[Birdman]

But its still a hidden link. Wouldn't that attract a SE penalty from Google for SEO tricks?

I don't think so, because you already disallowed it in the robots file.

Also, there are many legitimate uses of the CSS display and visibility properties. DHTML pop-out and drop-down menus are an examlple.

Be sure to add a javascript wrapper to kill the link if it get's clicked.

<a href="/getout.php" onclick="return false">
<img src="/clear.gif" style="border: none; width: 1px; height: 1px;" />
</a>

I just posted an updated version of the script here [webmasterworld.com]. There is a flaw(not serious) in the other one.