Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Can anyone explain this ChatGPT / OpenAI file-request behavior? |
1 | SumGuy | 11:05 pm May 23, 2026 | ||||
| Bad actor claiming to be OpenAi 74.7.242.63 |
2 | Bewenched | 2:16 am May 14, 2026 | ||||
| reverse() -- ColdFusion(or your language)/SQL + JavaScript Antiscrape Trick - What is stopping me from doing it this way? |
3 | blend27 | 5:19 pm May 11, 2026 | ||||
| what do they want? [3] ( 1 2 3 ) |
88 | lucy24 | 9:40 pm Apr 29, 2026 | ||||
| My website was vacuumed up by OpenAI |
3 | SumGuy | 6:27 pm Apr 13, 2026 | ||||
| Fixed Line ISP- gtt.net Are they Hu-men, AS3257? |
13 | blend27 | 3:25 am Apr 9, 2026 | ||||
| SleepBot |
6 | Bewenched | 3:59 pm Apr 7, 2026 | ||||
| "traceparent" header ...part of OpenTelemetry SDK |
1 | blend27 | 8:07 pm Apr 1, 2026 | ||||
| PerplexityBot What's the verdict? |
3 | SumGuy | 12:16 am Mar 30, 2026 | ||||
| Youbot (new crawler?) and google-inspection tool |
1 | SumGuy | 11:50 pm Mar 13, 2026 | ||||
| FeedFetcher-Google running amok |
7 | Pfui | 5:12 pm Mar 9, 2026 | ||||
| Procter & Gamble bot-running (not just soap 'n' stuff) |
9 | Pfui | 11:57 pm Feb 19, 2026 | ||||
| A useful resource for bot / proxy detectives |
2 | SumGuy | 7:26 pm Jan 6, 2026 | ||||
| Softblocking 35.0.0.0/8 |
22 | IJustWanttoPost | 11:26 pm Dec 21, 2025 | ||||
| FBI Seeks To Unmask Anonymous Web Archiving Service Owner Archive.today |
4 | SumGuy | 11:46 pm Dec 17, 2025 | ||||
| Is anyone getting hits from 121.99.4.0/24 ? |
15 | SumGuy | 3:28 am Dec 16, 2025 | ||||
| Where is grok bot / xAI? Does grok use anonymous VPNs to crawl sites? |
18 | SumGuy | 9:03 pm Dec 9, 2025 | ||||
| Complete mess all over 43.0.0.0/8 Concidering nuking the whole /8 for US/Canada sites |
15 | blend27 | 5:34 am Dec 5, 2025 | ||||
| ShapBot |
3 | SumGuy | 3:27 am Nov 14, 2025 | ||||
| Bad actors from 140.235.0.0/22 140.235.0.0/22 |
2 | Bewenched | 3:46 pm Nov 11, 2025 | ||||
| the robots ye will always have with you |
6 | lucy24 | 5:50 pm Nov 9, 2025 | ||||
| Evolving and Cloaking Your Robots.txt into an Active Bot Gatekeeper |
9 | Brett_Tabke | 5:37 pm Nov 9, 2025 | ||||
| Archive.org now sometimes using generic user-agent? |
3 | SumGuy | 5:28 pm Oct 28, 2025 | ||||
| blocking amazon networks |
24 | markRg | 10:04 pm Oct 20, 2025 | ||||
| Web host not seen before - AI hosting |
5 | Bewenched | 5:10 pm Oct 16, 2025 |
