Welcome to WebmasterWorld Guest from 54.167.15.6

Forum Moderated by: Ocean10000 & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - July 2018
Continuing discussion of hosting company IP ranges[4] ( 1 2 3 4 )
110 keyplyr 9:29 pm Dec 12, 2018
Robot's User Agent String
What webmasters want
1 keyplyr 9:34 pm Oct 14, 2018
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
User Agents of Programming Libraries and Command Line Tools
Common user agents
14 incrediBILL 8:54 pm Apr 29, 2008
 
0 12:00 am Jan 1, 1970
facebookexternalhit
8 lucy24 2:54 pm Dec 10, 2018
Microsoft and 13.64.0.0/11
Legit bing-bots? BingLocalSearch?
9 SumGuy 12:04 pm Dec 8, 2018
Laserlikebot/0.1
2 TorontoBoy 4:55 am Dec 5, 2018
http.rb/3.3.0 (Mastodon/2.6.1;
1 TorontoBoy 1:56 am Dec 5, 2018
CowBot/0.4
1 TorontoBoy 12:25 am Dec 5, 2018
AutomaticWPTester
1 TorontoBoy 11:52 pm Dec 4, 2018
more vkShare
1 lucy24 10:23 pm Nov 29, 2018
BigSkyBot 2.06
Simple site parsing agent for niche search index
2 mcneely 1:57 am Nov 22, 2018
GoogleDocs
1 ClosedForLunch 8:29 pm Nov 13, 2018
iframe-toloka.com
Yandex
1 ClosedForLunch 2:32 pm Nov 6, 2018
Falkon
5 lucy24 9:38 am Oct 31, 2018
OutclicksBot
new UA, new range
12 keyplyr 9:04 pm Oct 30, 2018
Apache-HttpAsyncClient
1 keyplyr 8:59 pm Oct 30, 2018
adplexity
1 keyplyr 7:36 pm Oct 29, 2018
weborama-fetcher
1 keyplyr 2:38 am Oct 25, 2018
PocketImageCache
2 keyplyr 3:11 pm Oct 24, 2018
Aranea Web Corpora Project
3 keyplyr 6:41 am Oct 24, 2018
GSC >Crawl Errors >Server Connectivity
5 keyplyr 3:23 am Oct 24, 2018
Android's Mobile Chrome "Show simplified view"
undetectable in UA string
8 keyplyr 1:17 am Oct 23, 2018