News Summary

Cloudflare, a leading internet infrastructure company, has introduced a new default setting that blocks AI web crawlers from accessing websites unless explicit permission or compensation is provided. This move, part of their Pay Per Crawl initiative, empowers website owners to demand payment from AI companies for accessing their content. The new default is intended to prevent unauthorized scraping, which can strain servers and disrupt services. Cloudflare’s system uses advanced techniques to distinguish between AI and legitimate web crawlers. While AI startup ProRata has agreed to participate in this initiative, it remains uncertain whether major AI companies will comply. This change is expected to alter the dynamics between content creators and AI companies, potentially leading to more negotiations and licensing agreements.