Forum Moderators: open
(agerhart, out of interest - on your recent paid submissions, which spiders coming through?)
270 hits in 6mins is excessive under any definition.
In the last 36hours:
Trek11 2225
Trek23 707
Trek3 995
Trek30 2429
trek7 1121
I've seen strange things happen to scripts during editing. Like they were trying to do A, but they skipped a semi-colon somewhere, and ended up killing the sub-routines that previously prevented B from happening...
I can't imagine a legitimate engine would knowingly program a spider to act that way.
All but the first 40 were dished out as "Server too busy" once I determined that they were hitting too fast, so it must have thought that if one "Server too busy" page comes back so quickly, why not get lots of 'em! ("Take two, they're small!")
Scooter19
Scooter3
Scooter9
trek1
trek10
trek14
trek16
trek24
trek5
trek6
Though, i've seen av url verifiers, or do i thought, is this a good sign to have all these bots coming for pages, or is there anything else going on ?
to be honest i am glad because i havent seen any av action here for months.
209.73.162.41 [Scooter-W3-1.0]
209.73.162.161 [Scooter-W3-1.0]
209.73.162.82 [Scooter-W3-1.0]
209.73.164.129 [Scooter-3.0.EU]
209.73.164.50 [Scooter/1.0]
A strange pattern of hits tho. .41 came aroud and requested over 400 pages that don't exist any more, and haven't for over 6 months. 6 hours later, same IP came back and indexed just the existing content, starting from index.asp, although didn't follow any ? links from these linked pages.
Three hours later, same IP again, indexing the same stuff as it had the previous time (most of it 404), but since it had last been, I'd uploaded around 300 pages of stuff that was previous 404, so I'm not complaining :)