Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| User-Agent: Custom-AsyncHttpClient (asked for a lot of crap) |
3 | SumGuy | 4:43 am Sep 9, 2024 | ||||
| Four-Minute Bytes Dive with me into the AWS weeds for a moment... |
3 | Pfui | 12:58 am Sep 6, 2024 | ||||
| meta-externalagent externalhotlink by another name? |
2 | lucy24 | 9:42 pm Sep 5, 2024 | ||||
| Redundant \ bot |
3 | not2easy | 9:13 pm Sep 1, 2024 | ||||
| Yet another "Cloud" Server with rogue bots 140.248.0.0/16 |
4 | Bewenched | 2:16 am Aug 30, 2024 | ||||
| Why is it that when I see GET /.env or POST /index.html It's always from a Microsoft Azure IP? |
1 | SumGuy | 12:46 am Aug 30, 2024 | ||||
| Nextcloud Server Crawler Seen today, a rare sighting |
1 | SumGuy | 11:45 pm Aug 18, 2024 | ||||
| mozilla What will they think of next? |
9 | lucy24 | 11:42 am Aug 13, 2024 | ||||
| PerplexityBot Perplexing |
5 | Pfui | 6:39 am Aug 12, 2024 | ||||
| PS Daily |
1 | lucy24 | 9:25 pm Aug 11, 2024 | ||||
| ChatGPT / openai bot First time I'm seeing them |
3 | SumGuy | 7:51 am Aug 10, 2024 | ||||
| Anyone else seeing a lot more of Applebot? Since the beginning of 2024, Applebot has been crawling my sites more often |
19 | sudo | 11:55 pm Aug 6, 2024 | ||||
| Will Googlebot cache JS after crawling it? |
5 | Jean_Niu | 2:55 am Jul 16, 2024 | ||||
| Facebook hitting my server from new IP's Why exactly? |
2 | SumGuy | 5:16 pm Jul 12, 2024 | ||||
| Seeing googlebot hits from new IP range 192.178.6.x - new to me |
5 | SumGuy | 8:25 pm Jul 6, 2024 | ||||
| OpenWebSearch.eu Crawler OWLer |
7 | engine | 12:12 pm Jun 29, 2024 | ||||
| Redirection error |
7 | chainazo | 9:19 pm Jun 11, 2024 | ||||
| A few Android user-agents I'm blocking, and why from google and microsoft IP's |
1 | SumGuy | 1:49 pm Jun 1, 2024 | ||||
| New duckduck bot |
1 | dstiles | 8:04 am Jun 1, 2024 | ||||
| New bot or crawler with UA containing BW/1.2; rb.gy/oupwis From a google cloud IP |
6 | SumGuy | 3:30 pm May 31, 2024 | ||||
| Looks like Nvidia is scraping web content |
3 | SumGuy | 11:05 pm May 28, 2024 | ||||
| Using Sec- to block scrapers Sec-Fetch and Sec-Ch-Ua |
29 | dstiles | 5:17 pm Apr 21, 2024 | ||||
| Server Farms 2024 Continuing discussion of hosting and data center IP ranges |
7 | not2easy | 3:39 pm Apr 21, 2024 | ||||
| Digital Ocean has new IP's |
6 | SumGuy | 2:55 pm Apr 14, 2024 | ||||
| Seeing binance dot com showing up as a referrer UA is always Chrome/90.0.4430.85 |
2 | SumGuy | 2:01 am Apr 12, 2024 |
