Forum Moderators: open

AI crawlers overwhelming sites

         

londrum

4:48 pm on Sep 17, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I've read stories about this a few times this year but this is the first time my site has been affected by AI crawlers causing downtime on a site.
My own problem was that I ended up with too many concurrent users on my database so it stopped working.
I had to move the site over to CloudFlare and rewrite a lot of the code so it caches and doesn't call the database so much, but I've got it sorted.

Has anyone else been having downtime this year because of AI crawlers?

lucy24

11:05 pm on Sep 17, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



What do you mean by “AI crawlers”? Some specific UAs, or a distinctive pattern of behavior?