Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| xAI / grok likely getting web search from Microsoft |
1 | SumGuy | 1:48 pm Oct 15, 2025 | ||||
| TikTokSpider |
7 | lucy24 | 5:52 pm Oct 14, 2025 | ||||
| Why does ClaudeBot / Anthropic do this? Repeatedly ask for robots and sitemap |
2 | SumGuy | 6:10 am Oct 13, 2025 | ||||
| ACEVILLEPTELTD Singapore Scraper Farm |
8 | not2easy | 11:11 pm Oct 12, 2025 | ||||
| Found some bad actors from this ISP cheapy.host LLC 196.251.81.0 |
4 | Bewenched | 2:18 am Oct 12, 2025 | ||||
| msnbot / bingbot interacting with my site's search script started recently |
2 | SumGuy | 6:18 am Oct 8, 2025 | ||||
| ChatGPT-User Does it indicate bot or user (human) engagement? |
4 | SumGuy | 5:00 pm Oct 4, 2025 | ||||
| Web hit from Residential ISP or international VPN? |
8 | SumGuy | 4:46 pm Oct 1, 2025 | ||||
| A new UA string? At least to me it's new |
6 | mcneely | 4:59 am Sep 24, 2025 | ||||
| IbouBot Something new |
7 | SumGuy | 4:11 pm Sep 22, 2025 | ||||
| AI crawlers overwhelming sites |
2 | londrum | 11:05 pm Sep 17, 2025 | ||||
| GoogleOther ? 66.249.65.xx |
5 | Bewenched | 3:33 pm Sep 17, 2025 | ||||
| Why doesn't Windows NT 11 show up in user-agent string? |
8 | SumGuy | 12:32 am Sep 17, 2025 | ||||
| CloudFlare Releases AI Crawl Control |
10 | Brett_Tabke | 11:16 am Sep 10, 2025 | ||||
| Bizarre googlebot behavior today |
2 | SumGuy | 3:33 am Sep 9, 2025 | ||||
| okhttp/4.9.2 |
2 | not2easy | 11:01 pm Sep 7, 2025 | ||||
| Hits from FB fwdproxy started 3 days ago |
4 | SumGuy | 2:16 pm Aug 30, 2025 | ||||
| ShapBot/0.1.0 104.154.193.69 |
2 | Bewenched | 3:15 am Aug 30, 2025 | ||||
| TikTokSpider - crashed our server 47.128.121.68 |
7 | Bewenched | 12:23 am Aug 20, 2025 | ||||
| What is WordPress user agent? Showing up in visitor logs |
17 | MichaelBluejay | 5:08 am Aug 12, 2025 | ||||
| Seeing a lot more activity from applebot lately |
3 | SumGuy | 6:05 am Aug 2, 2025 | ||||
| Search engines choking on phpBB |
13 | Kendo | 12:57 pm Jul 27, 2025 | ||||
| Gemini-Deep-Research |
1 | SumGuy | 5:34 am Jul 7, 2025 | ||||
| Brightbot |
2 | lucy24 | 5:25 am Jul 7, 2025 | ||||
| Best method to block unwanted visits via Cpanel Site is routinely being overwhelmed with UAs(?) I don't care about |
6 | millimi1 | 10:40 am Jun 28, 2025 |
