Requests for the same 23 pages

not sure if it is a 'bot or what...

[carfac]

Hi:

I have been having this, uh, occurance for a while now. Basically, a wide variety of IP's (from all over the world) request the same 23 pages on my site every day, and always in the same order. The next day, at about the same time, they request the same 23 pages, and then go away. Near as I can tell, I do not have page that has all these links. So what puzzles me is how are all these very diverse sites getting the same order? And why always do this, and every day, like clockwork? Note they do at one point hit my ban script, so they do get banned...

Just trying to figure out why...

Here is some log exerpts:

> grep 80.58.222.189 /main/logs/access_log

80.58.222.189 - - [03/Dec/2004:05:31:04 -0700] "GET / HTTP/1.1" 200 20330 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:06 -0700] "GET /ads/blink.fpl?region=4&publisher=3&bust='%20+%20bust%20+%20' HTTP/1.1" 302 209 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:07 -0700] "GET /ads/blink.fpl?region=4&publisher=3&slot=1 HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:08 -0700] "GET /ads/bimg.fpl?region=4&publisher=3&slot=1&keyword=NULL HTTP/1.1" 302 271 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:08 -0700] "GET /cgi-bin/bat_bot.pl HTTP/1.1" 200 322 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:09 -0700] "GET /cgi-bin/go.cgi?ID=4441&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:09 -0700] "GET /cgi-bin/go.cgi?ID=4525&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:09 -0700] "GET /cgi-bin/go.cgi?ID=4966&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:10 -0700] "GET /cgi-bin/go.cgi?ID=4959&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:10 -0700] "GET /cgi-bin/go.cgi?ID=6184&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:11 -0700] "GET /cgi-bin/go.cgi?ID=7891&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:12 -0700] "GET /cgi-bin/go.cgi?ID=7872&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:12 -0700] "GET /cgi-bin/go.cgi?ID=7864&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:12 -0700] "GET /cgi-bin/go.cgi?ID=4126&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:13 -0700] "GET /cgi-bin/go.cgi?ID=4042&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:13 -0700] "GET /cgi-bin/go.cgi?ID=5393&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:13 -0700] "GET /cgi-bin/go.cgi?ID=6275&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:14 -0700] "GET /Books/ HTTP/1.1" 403 216 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:14 -0700] "GET /Video/ HTTP/1.1" 403 216 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:14 -0700] "GET /Video/DVD/ HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:15 -0700] "GET /Video/VHS/ HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:16 -0700] "GET /ads/blink.fpl?region=4&publisher=3&slot=3 HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
80.58.222.189 - - [03/Dec/2004:05:31:16 -0700] "GET /ads/bimg.fpl?region=4&publisher=3&slot=3&keyword=NULL HTTP/1.1" 302 271 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"

> grep 152.121.36.65 /main/logs/access_log

152.121.36.65 - - [03/Dec/2004:05:41:01 -0700] "GET / HTTP/1.1" 200 20330 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:02 -0700] "GET /ads/blink.fpl?region=4&publisher=3&bust='%20+%20bust%20+%20' HTTP/1.1" 302 209 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:02 -0700] "GET /ads/blink.fpl?region=4&publisher=3&slot=1 HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:03 -0700] "GET /ads/bimg.fpl?region=4&publisher=3&slot=1&keyword=NULL HTTP/1.1" 302 271 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:03 -0700] "GET /cgi-bin/bat_bot.pl HTTP/1.1" 403 222 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:03 -0700] "GET /cgi-bin/go.cgi?ID=4441&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:03 -0700] "GET /cgi-bin/go.cgi?ID=4525&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:03 -0700] "GET /cgi-bin/go.cgi?ID=4966&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:03 -0700] "GET /cgi-bin/go.cgi?ID=4959&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=6184&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=7891&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=7872&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=7864&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=4126&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=4042&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:04 -0700] "GET /cgi-bin/go.cgi?ID=5393&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:05 -0700] "GET /cgi-bin/go.cgi?ID=6275&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:05 -0700] "GET /Books/ HTTP/1.1" 403 216 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:05 -0700] "GET /Video/ HTTP/1.1" 403 216 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:05 -0700] "GET /Video/DVD/ HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:05 -0700] "GET /Video/VHS/ HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:06 -0700] "GET /ads/blink.fpl?region=4&publisher=3&slot=3 HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
152.121.36.65 - - [03/Dec/2004:05:41:06 -0700] "GET /ads/bimg.fpl?region=4&publisher=3&slot=3&keyword=NULL HTTP/1.1" 302 271 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"

> grep 201.224.39.130 /main/logs/access_log

201.224.39.130 - - [03/Dec/2004:07:43:45 -0700] "GET / HTTP/1.1" 200 20330 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:43:53 -0700] "GET /ads/blink.fpl?region=4&publisher=3&bust='%20+%20bust%20+%20' HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:43:59 -0700] "GET /ads/blink.fpl?region=4&publisher=3&slot=1 HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:02 -0700] "GET /ads/bimg.fpl?region=4&publisher=3&slot=1&keyword=NULL HTTP/1.1" 302 271 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:04 -0700] "GET /cgi-bin/bat_bot.pl HTTP/1.1" 403 222 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:06 -0700] "GET /cgi-bin/go.cgi?ID=4441&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:08 -0700] "GET /cgi-bin/go.cgi?ID=4525&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:10 -0700] "GET /cgi-bin/go.cgi?ID=4966&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:16 -0700] "GET /cgi-bin/go.cgi?ID=4959&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:21 -0700] "GET /cgi-bin/go.cgi?ID=6184&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:24 -0700] "GET /cgi-bin/go.cgi?ID=7891&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:29 -0700] "GET /cgi-bin/go.cgi?ID=7872&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:32 -0700] "GET /cgi-bin/go.cgi?ID=7864&p=2 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:34 -0700] "GET /cgi-bin/go.cgi?ID=4126&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:38 -0700] "GET /cgi-bin/go.cgi?ID=4042&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:44 -0700] "GET /cgi-bin/go.cgi?ID=5393&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:48 -0700] "GET /cgi-bin/go.cgi?ID=6275&p=1 HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:50 -0700] "GET /Books/ HTTP/1.1" 403 216 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:44:58 -0700] "GET /Video/ HTTP/1.1" 403 216 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:45:06 -0700] "GET /Video/DVD/ HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:45:09 -0700] "GET /Video/VHS/ HTTP/1.1" 403 220 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:45:14 -0700] "GET /ads/blink.fpl?region=4&publisher=3&slot=3 HTTP/1.1" 302 253 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
201.224.39.130 - - [03/Dec/2004:07:45:16 -0700] "GET /ads/bimg.fpl?region=4&publisher=3&slot=3&keyword=NULL HTTP/1.1" 302 271 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"

[edited by: volatilegx at 8:54 pm (utc) on Dec. 3, 2004]
[edit reason] URLs have been examplified [/edit]

[wilderness]

Hey Dave,
We're going to have to stop meeting like this ;)

a solution?

SetEnvIf User-Agent NT)$ keep_out

[edit]

BTW I've been using this for at least a year.
I don't recall the date or the thread. I do recall reading that this was a fake UA when ending in such a way.

Additionally I have a link saved (some place) which had more UA's than I recalled seeing anywhere. Last time I looked for the link to reference I was unable to locate it.

[wilderness]

Although this is not the extnsive log UA list I was looking for.
The board discusses UA STRINGS

[sillydog.org...]

[carfac]

I have not problems stopping this guy- these guys... I am just trying to figure out why and how it's happening. The IP's are from US, Canada, Italy, UK, etc... from ISP's, small corps, utilities, etc. A real hodge pdge. Seems unlikely they would all be coordinating... but, somehow, they are...

[carfac]

Oh... and DUH! I had not even looked at the UA! SIlly me!

While I got ya here, what do you know anout these guys who send a request for a URL 32 k long? Here's one:

66.205.58.15 - - [03/Dec/2004:10:55:20 -0700] "SEARCH /\x90\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\
x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\xb1\x02\....(and on... and on... and on... until... x90\x90\x90\x90\x90\x90\x90\x90" 414 271 "-" "-"

dave

[edited by: volatilegx at 2:20 pm (utc) on Dec. 6, 2004]
[edit reason] broke long string to fix page width issue [/edit]

[wilderness]

what do you know anout these guys

Dave,
There is a thread on these types either in the archives here or in Jim's Apache Server forum.
I don't recall the details.
I had one such visit, however I don't recall if I denied on the UA or the IP.

Sorry I'm unable to be or more help.

Don

[carfac]

Thanks Don... any idea what terms to search to find it?

Here is what I tried- it did NOT work...

I have in my htconf file, for every virtual host, this:

RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

RewriteCond %{REQUEST_METHOD} ^PUT
RewriteRule .* - [F]

This is just gopod security, and if you can, it is a good idea to put in near the top of your server. Anyway, I thought a mod of this might be good against these long-URL people... so I added:

RewriteCond %{REQUEST_METHOD} ^SEARCH
RewriteRule .* - [F]

Problem is, before it gets to the rewrite part, the server kicks it out with a 414 error for the incredibly long URL...

strange, you have only had this once... I get it 5-10 times a day. Also interesting... it only seems to happen on my "low traffic" sites, not on the high-traffic ones...

dave

[wilderness]

Dave,
Tried a site search on both x90 and long user agents with no success, however there were returns to sift through.

Have you tried Jim's Apache forum for the htconf file options?
[webmasterworld.com...]

Don

[jdMorgan]

The excessively-long URLs are a buffer overrun attempt. On old, unpatched Windows machines, it's apparently possible to fill up the request buffer all the way to the end. Anything in the request past that point gets loaded into an executable area of memory. So, the long request you see is just "padding" and the payload at the end is what would get delivered into the execution area. Since your server rightly cuts off requests at a certain point, you'll never see the payload, and it probably won't run on a amchine configured with Apache anyway.

You might look into setting LimitRequestLine to a lower value. The default value is 8190. If you have legitimate users passing long data records using the GET method, don't set it too low or those may be rejected. Since legitimate data POSTs go into the message body, they won't be affected by LimitRequestLine, but rather by LimitRequestBody, so that's not a concern. Just try setting LimitRequestLine lower if you get a lot of these super-long requests. It'll cut them off sooner and save wasted work and bandwidth.

Jim