Recently I saw hits from this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
From 172.204.16.x
It was a single session, requesting a set of 13 files from my website over the course of what looks like a minute. 50 seconds actually.
4 of the files are PDF, but what's strange is that even though they are different sizes (364k, 826k, 1.6 mb, 2.3 mb) the bytes-downloaded per request fall into a few static values (262144 and 360448 one example of each) and (294912 7-times, 327680 3-times). From different IP's.
If they were trying to spread the load across different IP's, it still doesn't make sense because in some cases the total adds up to more than the actual file size, and in other cases it's less.
For example, the smallest file (364 kb) had 4 separate requests that total 1.2 mb, and the largest file (2.3 mb) also had 4 requests that total the exact same bytes transfered (1.2 mb).
Each request was completed with code 200, meaning that the bot seemed to have asked for a specific byte-range of the files (I don't log that detail) and not the entire file. And they weren't sequential for these files. Seemed sort of random.
I haven't looked before at ChatGPT-User's file-request behavior before in terms of how it requests large files. I just assumed if it asked for a file, it got the whole thing, and the bytes transfered would reflect that.
