spider adds url?

[smokeybarnable]

Every day I see spiders that go through my oscommerce site really fast. When I view the history I see that they add various url addresses to the end of my legitimate urls. What are they trying to accomplish? I ban the ip range but I'm wondering what their purpose is.

On a side note, Is there a master ip deny list that is updated often from a trusted source. I would pay for something like that.

Thanks.

[jdMorgan]

Well, you could use a script to ban the IP addresses that make requests matching your description - "they add various url addresses to the end of my legitimate urls."

I've done this using mod_rewrite on Apache, and rewriting the request to key_master's bad-bot script (See our PERL forum's library).

You can also ban IP addresses which exceed some number of requests per unit time -- See xlcus/AlexK's runaway-bot script in our PHP forum library.

As for a "master IP address range deny list,' such a thing isn't possible, because since the underlying reasons to deny an IP address range vary from site-to-site, the deny lists would also have to vary.

Jim

[smokeybarnable]

Thanks for the tips. I didn't even know there was a PHP library here. Cool! I'm always amazed how powerful PHP is. Any idea why bots exhibit this behavior? Are they trying to increase traffic to various websites?

Eric

[jdMorgan]

Probably script injection attacks and log-spamming, depending on the form of the URL.

In the former case, they're trying to get your server to run a remotely-hosted script. In the latter, they're hoping that you foolishly make your 'server stats' publicly-available, so they can get a link from your domain.

But they don't really care if it works on your specific server or not; They just fire and forget without directly checking the results. If either exploit works, then they'll know if your server fetches and runs their script, or they'll get a link on your 'stats' page. If not, it didn't cost much, and they're hitting millions of sites... so they do get a few every time.

Jim

[incrediBILL]

Most likely these bots are trying to infect your site.

Check out this article from our Spider forum library on this topic:
[webmasterworld.com...]

If that isn't what's going on perhaps with a few more details we can help you figure it out.

If you aren't sure about posting those details because of the TOS, sticky me with them first and I'll help you sanitize them for the forum ;)

[incrediBILL]

FYI, smokeybarnable stickied me some of his log entries and sure enough it was botnets probing his site for vulnerabilities.

The bots were trying to inject this simple PHP probe into his site:
<?php echo md5("just_a_test");?>

If the pages respond with an MD5 encoded response of "just_a_test" then the site is logged for the bot herder to come back and hack later.

[wilderness]

enough it was botnets probing his site for vulnerabilities.

Likley comimg from Cox and Charter ;)

[incrediBILL]

Gee Don, you got a thing for Cox and Charter don't you? ;)

Actually this one was coming from Europe which hits my servers often.

I'll bet if I showed you my daily abuse logs for the last few years you would faint.

[wilderness]

Gee Don, you got a thing for Cox and Charter don't you? wink

Have an "exlcusionist complex" ;)

(there's a keyword if I've ever seen one; nine-returns on google)

[keyplyr]

Don't know about Charter, but I use Cox as my ISP. They are very willing to take action against abusers, and make it easy to report.

[incrediBILL]

Must be my evil side rising up but I'm suddenly thinking I should just respond to these requests with an MD5 encoded response of "just_a_test" to see who comes knocking and tries to hack my site, try to flush them out.

OK, don't you all try this ;)

[smokeybarnable]

hehe..I like that idea.