Welcome to WebmasterWorld Guest from 54.157.222.62

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
and another gets on.
5 lucy24 1:25 pm Jan 18, 2013
Lots of interesting user agents
4 Bewenched 7:54 pm Jan 15, 2013
Baidu's IPs - Which Are Legit?
12 incrediBILL 7:21 pm Jan 13, 2013
LSSRocketCrawler/1.0 LightspeedSystems
69.84.207.246
3 Bewenched 5:57 pm Jan 13, 2013
sitemap.xml requests
3 wilderness 12:38 am Jan 13, 2013
Tiscali Communicator (scraper)
on SteepHost (Ukraine)
3 not2easy 9:22 pm Jan 12, 2013
Could someone spare a set of Anti-AWS (and other offenders) CSF rules?
2 1script 8:45 am Jan 12, 2013
Are these considered as bad bots ?
23 pkKumar 3:00 am Jan 11, 2013
msn bot has gone crazy
15 bwnbwn 10:52 pm Jan 9, 2013
Apple
6 wilderness 2:19 am Jan 9, 2013
browser versions
3 wilderness 1:39 am Jan 9, 2013
Netscape
5 wilderness 10:15 pm Jan 8, 2013
Entanet
Entanet-adsl
2 not2easy 10:05 pm Jan 8, 2013
Undefined Error in Logs
[2] ( 1 2 )
39 yaashul 5:28 am Jan 7, 2013
One for the UA profilers
5 Bewenched 6:38 am Jan 6, 2013
Microsoft or not?
70.37.130.8
9 Bewenched 5:56 am Jan 5, 2013
Yandex translate
4 lucy24 6:36 pm Jan 3, 2013
that Yahoo thingy
2 wilderness 6:13 am Dec 28, 2012
Anyone know the name of Wayback Machine's robot?
One site has not been crawled all year... wondering why.
19 JeffOstroff 8:46 pm Dec 27, 2012
Human or robot?
another one for the profilers
4 lucy24 9:09 am Dec 27, 2012
Ning/1.0
eat my 403
15 Bewenched 2:51 am Dec 27, 2012
TosCrawler
14 lucy24 6:55 pm Dec 25, 2012
WIndows NT 9.0 - Yeeeehaaa
Back from the future!
6 blend27 10:54 pm Dec 23, 2012
Google scraping through China?
7 dstiles 8:13 pm Dec 20, 2012
Data reseller bot shortlist?
5 Sgt_Kickaxe 5:40 pm Dec 14, 2012