IETF hatching a new way to tame aggressive AI website scraping

For web publishers, stopping AI bots from scraping their best content while consuming valuable bandwidth must feel somewhere between futile and nigh impossible.

It’s like throwing a cup of water at a forest fire. No matter what you try, the new generation of bots keeps advancing, insatiably consuming data to train AI models that are currently in the grip of competitive hyper-growth.

But with traditional approaches for limiting bot behavior, such as a robots.txt file, looking increasingly long in the tooth, a solution of sorts might be on the horizon through work being carried out by the Internet Engineering Task Force (IETF) AI Preferences Working Group (AIPREF).

The AIPREF Working Group is meeting this week in Brussels, where it hopes to continue its work to