Welcome to WebmasterWorld Guest from 18.234.111.56

Forum Moderated by: Ocean10000

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - July 2018
Continuing discussion of hosting company IP ranges[5] ( 1 2 3 4 5 )
127 keyplyr 4:52 pm Feb 12, 2019
Robot's User Agent String
What webmasters want
1 keyplyr 9:34 pm Oct 14, 2018
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
User Agents of Programming Libraries and Command Line Tools
Common user agents
14 incrediBILL 8:54 pm Apr 29, 2008
AHC/2.1 from UK Amazon 3.8.
1 not2easy 3:43 pm Apr 20, 2019
possible new bing crawl range
2 lucy24 1:54 am Apr 12, 2019
Bots start looking for back.zip, backup.zip, etc.
Get rid of those if you have them in root.
3 SumGuy 12:32 pm Apr 10, 2019
Google-speakr
6 phranque 9:23 pm Apr 5, 2019
User Agent is Microsoft Office something or other
5 SumGuy 8:29 pm Apr 4, 2019
Google 34
2 lucy24 1:48 am Apr 2, 2019
Facebook cortex
8 lucy24 1:21 am Apr 2, 2019
35.244.0.0/14 - Google or Merit (U-Mich) ?
4 SumGuy 6:23 pm Apr 1, 2019
Getting lots of hits from this Cloudsigma. what's it?
3 born2run 5:58 am Mar 24, 2019
Apple applebot increasing activity
Increasing hits to interior web pages
4 SumGuy 4:32 am Mar 18, 2019
At Home with the Robots: 2019 edition
3 lucy24 3:52 am Mar 13, 2019
Moreover
3 lucy24 2:01 am Mar 12, 2019
newspaper
4 Pfui 4:13 am Feb 27, 2019
Google polyfill
8 lucy24 3:34 am Feb 25, 2019
104.199.128.134
7 andrew_thom 2:19 am Feb 24, 2019
Skotina via TOR
After so many years, scrapers are still there...
3 blend27 11:49 pm Feb 13, 2019
Hits from 104.132.0.0/16 (Google) - human or bot?
7 SumGuy 2:29 am Feb 12, 2019
Seeing hits from 3.0.0.0/8 for first time
Is Amazon
8 SumGuy 2:41 am Feb 4, 2019
URLTester
3 lucy24 6:07 am Feb 3, 2019
Symantec from **.broadband.digiweb.ie
The range belongs to Symantec Corporation
7 blend27 10:09 pm Jan 28, 2019