Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  strange googlebot referer
googlebot referral from odd site
9 dupres01 7:06 am Aug 27, 2014
  xovibot
7 lucy24 7:05 pm Aug 25, 2014
  odd apple requests
strange requests coming from apple ramge
6 dupres01 6:20 pm Aug 24, 2014
  IPv6 enabled Bots / Server Farm
8 Ocean10000 6:24 pm Aug 18, 2014
  Googlebot and iPhone
30 wilderness 1:41 pm Aug 17, 2014
  Manticore
(more AWS spawn)
5 Pfui 6:06 pm Aug 16, 2014
  Enterprise Search Appliances for Scrapers?
Thunderstone
20 not2easy 9:56 am Aug 14, 2014
  apache pb2.gif
why do bots ask for this[2] ( 1 2 )
44 dupres01 4:46 am Aug 12, 2014
  Aug 8 Amazon UPDATED public IPs
Aws ip
4 miscbyproduct 4:00 pm Aug 11, 2014
  A6 on its Knees
They are taking notice
9 Angonasec 12:06 pm Aug 11, 2014
  sub-semalt
10 wilderness 7:05 pm Aug 10, 2014
  BrandProtect
5 lucy24 12:44 am Aug 2, 2014
  Server Farms - April 2014
Tracking and Reporting Data Center IP Ranges [7] ( 1 2 3 4 5 6 7 )
193 incrediBILL 12:39 am July 31, 2014
  Facebot
from Facebook[2] ( 1 2 )
40 Pfui 2:12 pm July 29, 2014
  Blocked IP Appends Data on URL
3 Angonasec 11:09 pm July 27, 2014
  Botnets
How to recognise & block[2] ( 1 2 )
43 keyplyr 9:37 pm July 27, 2014
  Trustwave
3 wilderness 7:25 pm July 27, 2014
  Bing Bot hiccup
5 wilderness 10:43 pm July 26, 2014
  new Yandex range?
15 lucy24 1:32 pm July 25, 2014
  Comcast IP Services
ban entire range?
2 dupres01 8:21 pm July 22, 2014
  Earthlink Hosting?
6 wilderness 8:52 am July 20, 2014
  Anyone else seeing UA Mozilla 28.0
On Bell or HP IP (I think)
5 not2easy 1:08 pm July 18, 2014
  Anonymous Visits from Hosting Companies
18 aristotle 4:52 pm July 16, 2014
  Google Feedfetcher
14 aristotle 1:49 am July 16, 2014
  visionutils
4 lucy24 10:52 pm July 13, 2014