Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Verifying SemrushBot Learning to understand what's a spoof and what's not |
24 | Solution2 | 11:35 am Jan 29, 2021 | ||||
| MTRobot |
2 | lucy24 | 6:06 pm Jan 26, 2021 | ||||
| INTERNETBOLAGET-SE = No More internetvikings.com |
8 | blend27 | 2:45 am Jan 26, 2021 | ||||
| Googlebot will start crawling over HTTP/2 in November 2020 |
20 | phranque | 10:28 pm Jan 22, 2021 | ||||
| Bots slamming my site [2] ( 1 2 ) |
33 | timchuma | 6:09 pm Jan 12, 2021 | ||||
| Facebot / Twitterbot hit from AS6730 (Sunrise Communications AG) |
7 | SumGuy | 2:52 pm Jan 10, 2021 | ||||
| Server Farms June 2020 Continuing discussion of hosting company IP ranges |
14 | not2easy | 3:09 am Jan 3, 2021 | ||||
| Why would bingbot be requesting atom.xml? What is the purpose of that file anyways? |
6 | SumGuy | 5:37 pm Dec 28, 2020 | ||||
| Edg/8+ Siims like it is on windows 10 within Chrome US |
6 | blend27 | 6:49 pm Dec 23, 2020 | ||||
| Bumb Bots madness Really? |
6 | blend27 | 5:49 pm Dec 22, 2020 | ||||
| Greebo's Ghost |
5 | lucy24 | 6:23 am Dec 1, 2020 | ||||
| Unique string at end of UA |
16 | wilderness | 6:03 am Dec 1, 2020 | ||||
| Unexpected querystring |
5 | dstiles | 5:41 pm Nov 24, 2020 | ||||
| Drilling holes in a firewall Can iptables allow user-agent access on blocked IPs? |
3 | dstiles | 3:07 pm Nov 20, 2020 | ||||
| Bingbot 403 Hitting robots.txt and getting a 403 rejection |
6 | dstiles | 8:01 pm Nov 4, 2020 | ||||
| Blocking IPs in firewall - a warning letsencrypt uses several IP ranges |
12 | dstiles | 11:14 pm Oct 28, 2020 | ||||
| Microsoft UK |
5 | lucy24 | 5:39 am Oct 26, 2020 | ||||
| Lucy's Cat |
2 | tangor | 8:53 pm Oct 21, 2020 | ||||
| Ugh Productions |
2 | wilderness | 8:49 pm Oct 7, 2020 | ||||
| Mojolicious (Perl) MIT or Amazon? |
4 | SumGuy | 9:10 pm Oct 3, 2020 | ||||
| robots and HTTP/2 |
17 | lucy24 | 12:08 pm Oct 3, 2020 | ||||
| Internet-structure-research-project-bot |
2 | not2easy | 9:59 pm Oct 2, 2020 | ||||
| Mozilla/5.0 (compatible; ips-agent) |
17 | tangor | 3:29 pm Sep 27, 2020 | ||||
| Referer (sic) question |
13 | tangor | 8:33 pm Sep 23, 2020 | ||||
| Jetslide |
3 | lucy24 | 5:30 pm Sep 21, 2020 |
