Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Has anyone ever seen instagram as a referrer? And also instagram as a user-agent? |
1 | SumGuy | 3:24 am Mar 9, 2024 | ||||
| zstd in Accept-Encoding request header Is this new? Anyone else seeing it? |
1 | SumGuy | 11:02 pm Feb 25, 2024 | ||||
| VsuSearchSpider/1.0 From a residential Deutsche Telekom IP |
4 | SumGuy | 4:01 am Feb 23, 2024 | ||||
| Speaking of blocking VPN's. User-Agent Workstream? |
3 | SumGuy | 12:51 pm Feb 20, 2024 | ||||
| do you block IPXO ranges? |
4 | LifeinAsia | 2:59 am Feb 15, 2024 | ||||
| Timpibot/0.9 |
2 | tangor | 6:55 pm Feb 7, 2024 | ||||
| SiteOne-Crawler Is this a known bot/crawler? |
2 | SumGuy | 4:32 am Jan 25, 2024 | ||||
| Quantbot search engine based in France |
6 | SumGuy | 4:26 am Jan 25, 2024 | ||||
| Interesting hit from a VPN with python UA ?checkstatus=areyouok |
3 | SumGuy | 10:38 pm Jan 22, 2024 | ||||
| fasthttp bots |
2 | not2easy | 1:15 am Jan 21, 2024 | ||||
| ImagesiftBot appears to be a reverse image search tool |
9 | Bewenched | 6:53 pm Jan 20, 2024 | ||||
| Cloudflare starts hammering me today Actual, or DOS? |
2 | SumGuy | 4:13 pm Jan 18, 2024 | ||||
| Server Farms 2023 Continuing discussion of hosting and data center IP ranges |
13 | not2easy | 1:59 pm Jan 8, 2024 | ||||
| What is the purpose of fetch.tunnel.googlezip.net? I'm seeing more of it |
1 | SumGuy | 2:31 am Jan 7, 2024 | ||||
| NetpeakCheckerBot From a comcast IP? |
1 | SumGuy | 10:57 pm Dec 29, 2023 | ||||
| Strange referer seen today file:///(...)/domains.txt |
1 | SumGuy | 2:26 am Dec 22, 2023 | ||||
| Odd set of 4 UA's seen today Some sort of probe or test I assume |
2 | SumGuy | 12:21 am Dec 12, 2023 | ||||
| Seeing some strange hits today Bytespider for one |
25 | SumGuy | 3:17 pm Dec 11, 2023 | ||||
| Suspicious hits from Macintosh user agents |
3 | ClosedForLunch | 12:44 am Dec 8, 2023 | ||||
| Barkrowler not asking for robots.txt |
1 | Bewenched | 7:20 pm Dec 2, 2023 | ||||
| SeekportBot 65.108.103.105 |
8 | Bewenched | 5:16 pm Dec 2, 2023 | ||||
| G'day from a new member interested in bots/crawlers/spiders |
27 | profshoelace | 12:19 am Nov 26, 2023 | ||||
| Web hits from duckduckgo favicons-bot Includes strange cookie string |
2 | SumGuy | 5:54 am Nov 21, 2023 | ||||
| Two new scrapers (or maybe AI-related bots) |
11 | asterickx | 12:59 am Nov 8, 2023 | ||||
| Seeing Bingbot hits from 20.15.133.x |
3 | SumGuy | 9:20 am Nov 1, 2023 |
