Welcome to WebmasterWorld Guest from 54.162.19.123

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
delay question
3 roshaoar 5:02 pm Nov 24, 2014
Gb of Bandwidth for 403s
Bandwidth over the limit once a day
10 not2easy 3:32 pm Nov 24, 2014
double agents?
4 lucy24 4:16 am Nov 10, 2014
Microsoft Insitesbot
7 Hobbs 6:00 pm Nov 8, 2014
Chinatelecom.com & AT&T in bed together
Chinatelecom.com, AT&T, htaccess, block ip range
1 EastTexas 3:06 am Nov 3, 2014
Strange Mobile Visits.
12 bobothecat2 7:44 pm Oct 19, 2014
What do we do with all the IP blocks gathered?
What technology to block crawlers, .htaccess or scripts?
22 dougwilson 7:06 pm Oct 15, 2014
WebTarantula Tools
They have their own nasty browser, toolbar, and bots
9 Angonasec 7:11 pm Oct 12, 2014
stremorbot
7 wilderness 10:14 am Oct 6, 2014
141 Class A
5 wilderness 1:13 am Oct 6, 2014
Sinobots Hit the News
Digital Mahjong: Game over?
1 Angonasec 5:06 am Sep 26, 2014
Don't understand this log entry
15 aristotle 10:36 am Sep 21, 2014
ContextAdbot IP updates
6 lucy24 1:31 am Sep 3, 2014
GTT bots
You may want to fine tune your blocks: I don't
5 Angonasec 12:22 pm Aug 27, 2014
strange googlebot referer
googlebot referral from odd site
9 dupres01 7:06 am Aug 27, 2014
xovibot
7 lucy24 7:05 pm Aug 25, 2014
Voltron
a new-ish 80legs crawler
1 dstiles 6:28 pm Aug 24, 2014
odd apple requests
strange requests coming from apple ramge
6 dupres01 6:20 pm Aug 24, 2014
HubSpot Webcrawler
(more AWS spawn)
1 Pfui 11:14 pm Aug 19, 2014
IPv6 enabled Bots / Server Farm
8 Ocean10000 6:24 pm Aug 18, 2014
Googlebot and iPhone
30 wilderness 1:41 pm Aug 17, 2014
Manticore
(more AWS spawn)
5 Pfui 6:06 pm Aug 16, 2014
Enterprise Search Appliances for Scrapers?
Thunderstone
20 not2easy 9:56 am Aug 14, 2014
apache pb2.gif
why do bots ask for this[2] ( 1 2 )
44 dupres01 4:46 am Aug 12, 2014
Aug 8 Amazon UPDATED public IPs
Aws ip
4 miscbyproduct 4:00 pm Aug 11, 2014

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members