Forum Moderators: open

Message Too Old, No Replies

why would a satellite proxy/cache trip spider trap

can't figure out if this was a harmless user or bad bot

         

amznVibe

4:08 pm on Feb 10, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Has anyone seen legitamate or bad behavior before from this satellite internet system? They appear to use a proxy cache (ala aol style) and I am trying to figure out if it was the user that was trying to spider the system via their own software, or if the sat company did it to pre-cache the page for the client.

81.23.192.195 cache.web-sat.com (there was no rDNS, I had to go look it up)

Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; YComp 5.0.2.6)

check it out, web-sat.com in Dublin Ireland, looks legit (interesting product too)

Also, if I want to let those folks in, what line would I add to undo the spider trap, even if it triggers in the future?
Can I undo the "getout" environment after it is set, later on in HTACCESS?

jdMorgan

4:49 pm on Feb 10, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



amznVibe,

Also, if I want to let those folks in, what line would I add to undo the spider trap, even if it triggers in the future?

The easiest way is to bypass the spider trap call in .htaccess. Instead of redirecting all requests for "innocent_looking_file.html" to "bad_bot.pl", test the requesting IP to see if it is someone you don't want to ban. Then redirect that request to an empty file or a file that requires a password, etc.

Can I undo the "getout" environment after it is set, later on in HTACCESS?

No, but you could put the IP address in the "allow" section, and use "order" to make sure that allow overrides deny. The previous method is much more efficient though, because this approach will invoke your script every time, and only ignore the new lines written to .htaccess.

Jim

weesnich

10:17 pm on Feb 10, 2003 (gmt 0)

10+ Year Member



> Can I undo the "getout" environment after
> it is set, later on in HTACCESS?

I think a

SetEnvIf [whatever your criteria is]!getout

should do this. Correct me if I'm wrong. But the method suggested by jdMorgan should be more efficient.

Added: there should be a [space] before the "!getout".