Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Barkrowler (babbar.tech/crawler) What do we know about it? |
3 | SumGuy | 5:14 pm Apr 13, 2025 | ||||
| AI bots |
20 | Scooter24 | 2:14 am Apr 11, 2025 | ||||
| iPhone user-agents (with Chrome) Are these legit? |
1 | SumGuy | 11:54 pm Mar 21, 2025 | ||||
| Overview of OpenAI Crawlers |
2 | engine | 12:55 pm Mar 13, 2025 | ||||
| Kitties and Gentelmen Lets start a new Server-Farms thread! |
5 | blend27 | 2:05 pm Feb 27, 2025 | ||||
| Turkey News Feed? useragent "newspaper/0.2.8" |
5 | ghostofseo | 3:33 am Feb 11, 2025 | ||||
| Google Bot operating from 192.178.6.0/24 Something new? |
2 | SumGuy | 6:11 pm Jan 23, 2025 | ||||
| Document scraping from unknown bot UA is Chrome/124, IP sources all Windstream ISP |
11 | SumGuy | 2:37 pm Jan 23, 2025 | ||||
| Rash of strange requests for robots.txt today |
1 | SumGuy | 2:54 am Jan 4, 2025 | ||||
| Amazonbot again Amazonbot - block or allow? |
9 | jmccormac | 7:23 am Jan 2, 2025 | ||||
| Code200 UAB Corenet, Metacluster, Teso, Oxysales and Oxylabs |
3 | SumGuy | 10:48 pm Dec 18, 2024 | ||||
| Private prefetch proxy in Chrome These are the hits from fetch.tunnel.googlezip.net? |
6 | SumGuy | 3:22 pm Nov 18, 2024 | ||||
| Weird traffic from old model mobile phones |
11 | Dooku | 4:20 am Nov 6, 2024 | ||||
| Stange hits: Meta IPs, Google UA Spoofed UA? Scraper using Meta IP? |
1 | asterickx | 1:17 pm Nov 2, 2024 | ||||
| Increase in Facebook bot hits |
14 | dstiles | 8:53 pm Oct 23, 2024 | ||||
| GuzzleHttp/7 |
3 | not2easy | 12:53 am Oct 14, 2024 | ||||
| AwarioBot 37.27.129.xxx |
2 | Bewenched | 5:38 pm Oct 13, 2024 | ||||
| More on IPXO Getting their CIDR's from geofeed.txt |
1 | SumGuy | 1:15 am Sep 29, 2024 | ||||
| googlebot from Georgia ? |
5 | Wmff | 5:22 pm Sep 28, 2024 | ||||
| these ips are from google or pretending to be Googlebot |
16 | pramod | 4:25 pm Sep 28, 2024 | ||||
| go-resty/2.12.0 some new |
2 | SumGuy | 4:31 pm Sep 14, 2024 | ||||
| UA workona-favicon-service |
1 | Pfui | 7:45 am Sep 11, 2024 | ||||
| User-Agent: Custom-AsyncHttpClient (asked for a lot of crap) |
3 | SumGuy | 4:43 am Sep 9, 2024 | ||||
| Four-Minute Bytes Dive with me into the AWS weeds for a moment... |
3 | Pfui | 12:58 am Sep 6, 2024 | ||||
| meta-externalagent externalhotlink by another name? |
2 | lucy24 | 9:42 pm Sep 5, 2024 |
