1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 6:27 pm May 4, 2026

Forum Moderators: open

Message Too Old, No Replies

Strange crawl

         

wilderness

2:06 pm on Mar 22, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Unable to determine why these were 403'd. Don't have any header checks active.
I've a few of the IP's denied, however others are not denied at all. I did not check and compare all the IP's.
Still, some setting denied them.

The requests are over a 17-minute period and all the same UA.
Perhaps compromised machines, perhaps something else, however they are definitely all associated

24.170.55.129 - - [21/Mar/2014:18:34:57 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
24.170.55.129 - - [21/Mar/2014:18:35:08 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
24.191.71.223 - - [21/Mar/2014:18:40:31 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
24.191.71.223 - - [21/Mar/2014:18:40:37 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
24.191.26.126 - - [21/Mar/2014:18:42:18 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
188.17.153.192 - - [21/Mar/2014:18:43:34 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.243.164.198 - - [21/Mar/2014:18:43:35 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.243.164.198 - - [21/Mar/2014:18:44:28 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.116.39.190 - - [21/Mar/2014:18:44:32 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.116.39.190 - - [21/Mar/2014:18:44:36 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
103.247.157.3 - - [21/Mar/2014:18:44:44 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
103.247.157.3 - - [21/Mar/2014:18:44:48 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
74.84.30.173 - - [21/Mar/2014:18:44:56 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
75.84.125.124 - - [21/Mar/2014:18:45:12 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
65.24.217.28 - - [21/Mar/2014:18:45:14 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
65.24.217.28 - - [21/Mar/2014:18:45:15 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
103.247.157.3 - - [21/Mar/2014:18:45:17 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
103.247.157.3 - - [21/Mar/2014:18:45:21 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
24.51.134.173 - - [21/Mar/2014:18:45:25 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
24.51.134.173 - - [21/Mar/2014:18:45:26 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
114.33.31.101 - - [21/Mar/2014:18:46:10 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
114.33.31.101 - - [21/Mar/2014:18:46:14 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
96.19.68.137 - - [21/Mar/2014:18:46:42 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
96.19.68.137 - - [21/Mar/2014:18:46:43 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.116.39.190 - - [21/Mar/2014:18:47:27 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.116.39.190 - - [21/Mar/2014:18:47:30 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.181.9.141 - - [21/Mar/2014:18:47:33 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.181.9.141 - - [21/Mar/2014:18:47:35 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
184.56.244.80 - - [21/Mar/2014:18:50:41 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
184.56.244.80 - - [21/Mar/2014:18:50:45 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.181.9.141 - - [21/Mar/2014:18:51:43 -0600] "GET /MySub/ HTTP/1.0" 403 558 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.181.9.141 - - [21/Mar/2014:18:51:45 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/MySub/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.181.9.141 - - [21/Mar/2014:18:51:48 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"
98.181.9.141 - - [21/Mar/2014:18:51:50 -0600] "GET / HTTP/1.0" 403 794 "http://www.example.com/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"

phranque

12:58 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Do you [F]orbid the self-referring requests?

wilderness

4:04 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



No

Looks like I'll not determine what trapped them. Something certainly did.

tangor

4:51 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Got to ask... looks like HTTP 1.0 to me. Is that one of your black list items? Many of us consider that a fairly useless visitor...

lucy24

8:31 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is your host running mod_security? It defaults to 500-class*, but they might have changed it to 403.


* I have an ineradicable mental block on 500 vs. 503 :(

keyplyr

8:48 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month





Personally, I'd rather a host return a 503 (server busy, try again later) than a 500 (the server experienced an error.) Visitors seem to get nervous with the later.

wilderness

10:47 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Got to ask... looks like HTTP 1.0 to me.


Dat be the one. Many thanks.

wilderness

11:22 am on Mar 23, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



BTW, each of these made the same two requests.

First the sub-Folder-page and then the root.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 6:27 pm May 4, 2026