[Forum] letscontrolit down?

[chromoxdor]

Just wanted to get my morning dose of the forum but it seems to be down.
@TD-er did something happen?

[tonhuisman]

It's just slow, looks like there are quite a lot of 'guests' using the forum (> 3000), usually indicating something close to a DDOS 'harassment'.

[chromoxdor]

Ahh yes extremely slow... just loaded... :)

[chromoxdor]

quite a lot of 'guests'

Is there anything free on offer today?
Maybe software 😁

[TD-er]

We're really getting slammed by those stupid bots from all kinds of AI companies, most of which I never heard of.

I already did add a lot of them to make sure the bots get a different UI which doesn't include a session ID in the URL.
Problem with those bots is that they work in parallel with lots of bot instances.
And each will have a different session ID thus each fetched URL is different.
So the list of links to fetch will approach infinity.
This in turn will make the site more attractive to fetch, so more bots are added and the length will approach infinity even faster.

I will try to add a wildcard match for bots.

[chromoxdor]

Thats really annoying 😳

[TD-er]

Yep has already taken me quite some hours in the past few days.
Quite frustrating, especially since each click may take minutes and re-logins which seem to loop forever.

Anyway, I hopefully added some 'Unknown bot' wildcard signature, let's see if this will free up some CPU-cycles on the server.

[chromoxdor]

The guests are gone and everything loads in an instant again. Thanks!

[TD-er]

I asked Remco what he did, as we "lost" about 4000+ 'guest' concurrent visits.

When I have an update, I'll let you know.

[uzi18]

Maybe new file airobots.txt with content "deny" :)
As robots.txt looks like is not supported by AI ;)

[TD-er]

Maybe new file airobots.txt with content "deny" :) As robots.txt looks like is not supported by AI ;)

On one hand, I hope this is serious, but on the other hand I hope it is a joke...
If serious, then it would mean those AI companies are even bigger scum than I already thought.

Seriously I truly hate this AI-hype these days as it only kills creativity and costs such enormous amounts of energy. Not only the electricity to run those AI engines, but also to counter all the untruth they spread and brain-dead videos and blog texts which appear just about everywhere and which take a while to realize it is again AI-generated crap.
I really truly hate AI!

Still, if it is a serious remark, then it will greatly help, so thanks anyway as I really never would have thought of this idea...

[TD-er]

Ah apparently it is really a thing, so double thanks @uzi18
https://github.com/ai-robots-txt/ai.robots.txt

[uzi18]

In fact it was introduced as a joke, because ai robots does not respect robots.txt so they need own file ;)

[TD-er]

Here the wall of shame of the log lines from today containing "bot" (case insensitive grep)

      1  duckduckbot            1   IPs
      2  SiteAuditBot           1   IPs
      3  Discordbot             2   IPs
      4  EzoicBot               2   IPs
      6  MojeekBot              1   IPs
      9  SeznamBot              2   IPs
     15  Qwantbot               2   IPs
     18  duckassistbot          9   IPs
     23  Pinterestbot           2   IPs
     53  DataForSeoBot          2   IPs
     80  facebookexternalhit    63  IPs
    206  OAI-SearchBot          78  IPs
    402  PerplexityBot          11  IPs
   1134  YandexBot              49  IPs
   2174  Googlebot              17  IPs
   2655  bingbot                132 IPs
   3504  AwarioBot              2   IPs
   4188  Applebot               809 IPs
   4955  MJ12bot                17  IPs
   5514  ClaudeBot              636 IPs
   5658  DotBot                 1   IPs
   8885  SemrushBot             48  IPs
  21493  AhrefsBot              388 IPs
  50798  Amazonbot              432 IPs
  83782  GPTBot                 22  IPs

This month alone about 150 GByte from the forum alone, likely most of this in the last 3 days.

[uzi18]

@TD-er on the other hand if you block all bots you should have in mind about all search engines like google,bing,duckduckgo,etc.
no food - not found :D