Welcome to WebmasterWorld Guest from 54.144.126.195

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Googlebot Crawling from China IPs
Make sure you allow 203.208.60.1-203.208.60.255
16 incrediBILL 8:16 pm Jul 27, 2012
Mbbf 0.1 Dwl
Wikimedia
5 keyplyr 10:51 pm Jul 25, 2012
Whmcsxtra dedicated servers
AlphaServer
6 keyplyr 10:42 pm Jul 25, 2012
googlebot drink the brown acid?
Gobbledygook "GETs" from google
15 dupres01 8:37 pm Jul 25, 2012
Xenu Link Sleuth
Should anonymous linkers be blocked?[2] ( 1 2 )
39 grandma_genie 1:24 am Jul 24, 2012
Something weird with Google IP ranges
3 motorhaven 11:22 pm Jul 23, 2012
odd UA
26 wilderness 9:46 am Jul 23, 2012
Start.exe User Agent?
from fdcservers.net
4 incrediBILL 8:03 pm Jul 22, 2012
NerdByNature.Bot
8 incrediBILL 1:06 am Jul 22, 2012
eBay UK Crawling?
or just a browser thing
3 incrediBILL 10:25 pm Jul 21, 2012
Punk Spider
caught this punk snooping from the clouds
8 incrediBILL 10:14 pm Jul 21, 2012
ProCog Coming Soon!
in 27 days...
2 incrediBILL 8:01 am Jul 21, 2012
Simple user agent checks
Fast checks for user agents
6 btherl 6:14 pm Jul 20, 2012
Beware HTML in abUser Agent Strings
filter HTTP requests and logs like any other form of input data
9 incrediBILL 10:00 pm Jul 16, 2012
TweetedTimes Now on Yandex?
7 incrediBILL 8:01 pm Jul 15, 2012
Google App Engine
Tons of junk coming from Google
5 incrediBILL 6:01 am Jul 15, 2012
lynx robot
too much information?
4 lucy24 5:59 am Jul 15, 2012
ia archiver
15 keyplyr 1:35 am Jul 15, 2012
Linux and Android
6 wilderness 8:20 pm Jul 13, 2012
Exabot / Exalead new IP range
1 dstiles 7:13 pm Jul 13, 2012
Education Networks of America
12 incrediBILL 2:39 am Jul 13, 2012
Global Net Access
5 wilderness 12:43 am Jul 10, 2012
okhy11
007, okhy eleven...
11 lucy24 3:16 am Jul 9, 2012
Are visitors from mail servers legitimate?
Corporate mail server from medical company.
4 grandma_genie 9:30 pm Jul 7, 2012
GoGuidesSpamTracker
Is this spider legit?
3 grandma_genie 1:26 am Jul 6, 2012

Featured Threads

Hot Threads This Week

Hot Threads This Month