Ultimately this has been a recent-ish problem, and Lemmy simply doesn’t have this sort of function yet. It’s a goo thought, though.
As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don’t limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.
Ultimately this has been a recent-ish problem, and Lemmy simply doesn’t have this sort of function yet. It’s a goo thought, though.
As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don’t limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.