Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Feedfetcher every 40 minutes for two days
feedfetcher persistently looking for non existent item
17 revrob 9:22 pm Sep 21, 2011
Grafix-is
6 keyplyr 8:05 pm Sep 20, 2011
Level 3 redux
6 Pfui 3:25 pm Sep 19, 2011
Resolving "urlresolver"
Google IPs repeat no-robots runs
30 Pfui 4:55 am Sep 19, 2011
Netscape 5.0 is driving me nuts
Netscape 5.0 showing on logs
9 walkman 9:00 pm Sep 16, 2011
Does everyone use bespoke scripts for spider identification
6 scooterdude 9:47 pm Sep 14, 2011
Sitebuilder name in results-spammers thru a backdoor?
Sitebuilder in search results-block "backdoor"
5 KaraScene1 4:24 pm Sep 14, 2011
Traffic from Malicious IPs and Adwords ads for my website
I get visitors from adwords when I never started a campaign
5 Makis77 8:45 pm Sep 13, 2011
isc header collector
2 Pfui 9:30 pm Sep 9, 2011
Iparadigms
8 keyplyr 7:01 pm Sep 3, 2011
Why is this agent fetching my images?
7 aristotle 4:58 pm Sep 3, 2011
PostPost
2 Pfui 1:52 pm Sep 3, 2011
pflab
3 keyplyr 12:50 am Sep 3, 2011
Netvigator
5 keyplyr 11:31 pm Sep 1, 2011
ScooperBot
3 Pfui 2:46 pm Sep 1, 2011
who is Covario and what do they want from me?
3 lucy24 7:27 pm Aug 31, 2011
74.125.52.93 in my logs today
6 apauto 9:11 pm Aug 30, 2011
AIM7 WinHTTP/1.0
2 Pfui 7:00 pm Aug 29, 2011
YodaoBot
7 Pfui 1:16 am Aug 29, 2011
BOTW Spider
Jurgle LLC in New York
2 grandma_genie 6:50 pm Aug 24, 2011
jikespider
Chinese bot shape-shifts on the fly.
4 Pfui 7:29 pm Aug 23, 2011
Google IPs but MSIE UA
Huh?
2 Pfui 4:47 pm Aug 22, 2011
NerdByNature.Bot
13 keyplyr 10:18 pm Aug 21, 2011
Level3 and more
5 wilderness 8:51 pm Aug 21, 2011
AhrefsBot
née SiteBot
5 Pfui 4:57 pm Aug 21, 2011