Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Anyone else seeing a lot more of Applebot? Since the beginning of 2024, Applebot has been crawling my sites more often |
19 | sudo | 11:55 pm Aug 6, 2024 | ||||
| Will Googlebot cache JS after crawling it? |
5 | Jean_Niu | 2:55 am Jul 16, 2024 | ||||
| Facebook hitting my server from new IP's Why exactly? |
2 | SumGuy | 5:16 pm Jul 12, 2024 | ||||
| Seeing googlebot hits from new IP range 192.178.6.x - new to me |
5 | SumGuy | 8:25 pm Jul 6, 2024 | ||||
| OpenWebSearch.eu Crawler OWLer |
7 | engine | 12:12 pm Jun 29, 2024 | ||||
| Redirection error |
7 | chainazo | 9:19 pm Jun 11, 2024 | ||||
| A few Android user-agents I'm blocking, and why from google and microsoft IP's |
1 | SumGuy | 1:49 pm Jun 1, 2024 | ||||
| New duckduck bot |
1 | dstiles | 8:04 am Jun 1, 2024 | ||||
| New bot or crawler with UA containing BW/1.2; rb.gy/oupwis From a google cloud IP |
6 | SumGuy | 3:30 pm May 31, 2024 | ||||
| Looks like Nvidia is scraping web content |
3 | SumGuy | 11:05 pm May 28, 2024 | ||||
| Using Sec- to block scrapers Sec-Fetch and Sec-Ch-Ua |
29 | dstiles | 5:17 pm Apr 21, 2024 | ||||
| Server Farms 2024 Continuing discussion of hosting and data center IP ranges |
7 | not2easy | 3:39 pm Apr 21, 2024 | ||||
| Digital Ocean has new IP's |
6 | SumGuy | 2:55 pm Apr 14, 2024 | ||||
| Seeing binance dot com showing up as a referrer UA is always Chrome/90.0.4430.85 |
2 | SumGuy | 2:01 am Apr 12, 2024 | ||||
| Why Is Host Europe (HEG) Suddenly Crawling Me? |
3 | RedBar | 11:20 pm Mar 31, 2024 | ||||
| Google-PageRenderer / developers.google.com/+/web/snippet/ From a google-proxy IP |
1 | SumGuy | 10:47 pm Mar 27, 2024 | ||||
| What exactly is YaBrowser and Yowser ? Seeing both in the same UA |
1 | SumGuy | 3:49 am Mar 25, 2024 | ||||
| Has anyone ever seen instagram as a referrer? And also instagram as a user-agent? |
1 | SumGuy | 3:24 am Mar 9, 2024 | ||||
| zstd in Accept-Encoding request header Is this new? Anyone else seeing it? |
1 | SumGuy | 11:02 pm Feb 25, 2024 | ||||
| VsuSearchSpider/1.0 From a residential Deutsche Telekom IP |
4 | SumGuy | 4:01 am Feb 23, 2024 | ||||
| Speaking of blocking VPN's. User-Agent Workstream? |
3 | SumGuy | 12:51 pm Feb 20, 2024 | ||||
| do you block IPXO ranges? |
4 | LifeinAsia | 2:59 am Feb 15, 2024 | ||||
| Timpibot/0.9 |
2 | tangor | 6:55 pm Feb 7, 2024 | ||||
| SiteOne-Crawler Is this a known bot/crawler? |
2 | SumGuy | 4:32 am Jan 25, 2024 | ||||
| Quantbot search engine based in France |
6 | SumGuy | 4:26 am Jan 25, 2024 |
