Welcome to WebmasterWorld Guest from 54.162.141.212

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
eBay UK Crawling?
or just a browser thing
3 incrediBILL 10:25 pm Jul 21, 2012
Punk Spider
caught this punk snooping from the clouds
8 incrediBILL 10:14 pm Jul 21, 2012
ProCog Coming Soon!
in 27 days...
2 incrediBILL 8:01 am Jul 21, 2012
Simple user agent checks
Fast checks for user agents
6 btherl 6:14 pm Jul 20, 2012
Beware HTML in abUser Agent Strings
filter HTTP requests and logs like any other form of input data
9 incrediBILL 10:00 pm Jul 16, 2012
TweetedTimes Now on Yandex?
7 incrediBILL 8:01 pm Jul 15, 2012
Google App Engine
Tons of junk coming from Google
5 incrediBILL 6:01 am Jul 15, 2012
lynx robot
too much information?
4 lucy24 5:59 am Jul 15, 2012
ia archiver
15 keyplyr 1:35 am Jul 15, 2012
Linux and Android
6 wilderness 8:20 pm Jul 13, 2012
Exabot / Exalead new IP range
1 dstiles 7:13 pm Jul 13, 2012
Education Networks of America
12 incrediBILL 2:39 am Jul 13, 2012
Global Net Access
5 wilderness 12:43 am Jul 10, 2012
okhy11
007, okhy eleven...
11 lucy24 3:16 am Jul 9, 2012
Are visitors from mail servers legitimate?
Corporate mail server from medical company.
4 grandma_genie 9:30 pm Jul 7, 2012
GoGuidesSpamTracker
Is this spider legit?
3 grandma_genie 1:26 am Jul 6, 2012
Lipperhey Link Explorer
data mining software
2 keyplyr 8:29 pm Jul 5, 2012
Ezooms the New DotBot?
née DotBot?
16 Pfui 9:21 pm Jul 4, 2012
New Softlayer IP Range
Just heads-up
18 blend27 10:44 pm Jul 2, 2012
what is this POST command and why did it get a 200?
22 dupres01 1:44 pm Jun 30, 2012
Opal Telecom
3 wilderness 8:51 pm Jun 28, 2012
80legs Flooding Site - Ignoring Robots.txt
Literally thousands of requests
9 Andem 3:48 am Jun 28, 2012
cache requests
1 keyplyr 8:46 am Jun 27, 2012
Fake user agent strings
How to recognize...[2] ( 1 2 )
50 grandma_genie 5:29 pm Jun 26, 2012
User Agent Spam Attack
2 incrediBILL 5:24 pm Jun 26, 2012

Featured Threads

Hot Threads This Week

Hot Threads This Month