{"id":6621,"date":"2025-03-27T20:09:03","date_gmt":"2025-03-28T00:09:03","guid":{"rendered":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/"},"modified":"2025-03-27T20:09:03","modified_gmt":"2025-03-28T00:09:03","slug":"open-source-devs-battle-ai-crawlers-with-innovative-tools","status":"publish","type":"post","link":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/","title":{"rendered":"Open Source Devs Battle AI Crawlers with Innovative Tools"},"content":{"rendered":"<p>Open source developers are devising innovative solutions to combat the relentless onslaught of AI web crawlers that ignore standard protocols like the Robots Exclusion Protocol. FOSS developer Xe Iaso created &#8216;Anubis,&#8217; a reverse proxy proof-of-work that distinguishes between human users and bots, allowing only legitimate requests to access Git servers. The tool has gained significant traction within the FOSS community, highlighting a widespread issue as many developers face frequent DDoS-level traffic from these aggressive bots. Other solutions like &#8216;Nepenthes&#8217; and Cloudflare&#8217;s &#8216;AI Labyrinth&#8217; aim to mislead and deter these bots by feeding them irrelevant content. The battle against AI crawlers continues as developers seek to protect their resources while maintaining the accessibility of their open-source projects.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Open source developers are devising innovative solutions to combat the relentless onslaught of AI web crawlers that ignore standard protocols like the Robots Exclusion Protocol. FOSS developer Xe Iaso created&#8230;<\/p>\n","protected":false},"author":1,"featured_media":951,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1434],"tags":[7686,7688,413,6453,7687],"class_list":["post-6621","post","type-post","status-publish","format-standard","has-post-thumbnail","category-global","tag-ai-web-crawlers","tag-foss-community","tag-global-startup-news","tag-open-source-development","tag-robot-txt-protocol"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Open Source Devs Battle AI Crawlers with Innovative Tools - Startup Ecosystem Canada<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Open Source Devs Battle AI Crawlers with Innovative Tools - Startup Ecosystem Canada\" \/>\n<meta property=\"og:description\" content=\"Open source developers are devising innovative solutions to combat the relentless onslaught of AI web crawlers that ignore standard protocols like the Robots Exclusion Protocol. FOSS developer Xe Iaso created...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/\" \/>\n<meta property=\"og:site_name\" content=\"Startup Ecosystem Canada\" \/>\n<meta property=\"article:published_time\" content=\"2025-03-28T00:09:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.startupecosystem.ca\/news\/wp-content\/uploads\/2024\/11\/default-pic_big.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"500\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Startup Eco System\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Startup Eco System\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/\"},\"author\":{\"name\":\"Startup Eco System\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/#\\\/schema\\\/person\\\/ebf38463111787b50c88f6fcd33f9c28\"},\"headline\":\"Open Source Devs Battle AI Crawlers with Innovative Tools\",\"datePublished\":\"2025-03-28T00:09:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/\"},\"wordCount\":126,\"image\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/default-pic_big.jpg\",\"keywords\":[\"AI web crawlers\",\"FOSS community\",\"Global startup news\",\"Open-source development\",\"Robot.txt protocol\"],\"articleSection\":[\"Global\"],\"inLanguage\":\"en-CA\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/\",\"url\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/\",\"name\":\"Open Source Devs Battle AI Crawlers with Innovative Tools - Startup Ecosystem Canada\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/default-pic_big.jpg\",\"datePublished\":\"2025-03-28T00:09:03+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/#\\\/schema\\\/person\\\/ebf38463111787b50c88f6fcd33f9c28\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#breadcrumb\"},\"inLanguage\":\"en-CA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-CA\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/default-pic_big.jpg\",\"contentUrl\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/11\\\/default-pic_big.jpg\",\"width\":1000,\"height\":500},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/open-source-devs-battle-ai-crawlers-with-innovative-tools\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Open Source Devs Battle AI Crawlers with Innovative Tools\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/#website\",\"url\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/\",\"name\":\"Startup Ecosystem Canada\",\"description\":\"Largest Startup Community in Canada\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-CA\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.startupecosystem.ca\\\/news\\\/#\\\/schema\\\/person\\\/ebf38463111787b50c88f6fcd33f9c28\",\"name\":\"Startup Eco System\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-CA\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a0296599ae4d98b5f644a6ebd0f36ee52f2642dac74ac9fee14854ec4a47d61d?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a0296599ae4d98b5f644a6ebd0f36ee52f2642dac74ac9fee14854ec4a47d61d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a0296599ae4d98b5f644a6ebd0f36ee52f2642dac74ac9fee14854ec4a47d61d?s=96&d=mm&r=g\",\"caption\":\"Startup Eco System\"},\"sameAs\":[\"https:\\\/\\\/www.startupecosystem.ca\\\/news\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Open Source Devs Battle AI Crawlers with Innovative Tools - Startup Ecosystem Canada","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/","og_locale":"en_US","og_type":"article","og_title":"Open Source Devs Battle AI Crawlers with Innovative Tools - Startup Ecosystem Canada","og_description":"Open source developers are devising innovative solutions to combat the relentless onslaught of AI web crawlers that ignore standard protocols like the Robots Exclusion Protocol. FOSS developer Xe Iaso created...","og_url":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/","og_site_name":"Startup Ecosystem Canada","article_published_time":"2025-03-28T00:09:03+00:00","og_image":[{"width":1000,"height":500,"url":"https:\/\/www.startupecosystem.ca\/news\/wp-content\/uploads\/2024\/11\/default-pic_big.jpg","type":"image\/jpeg"}],"author":"Startup Eco System","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Startup Eco System","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#article","isPartOf":{"@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/"},"author":{"name":"Startup Eco System","@id":"https:\/\/www.startupecosystem.ca\/news\/#\/schema\/person\/ebf38463111787b50c88f6fcd33f9c28"},"headline":"Open Source Devs Battle AI Crawlers with Innovative Tools","datePublished":"2025-03-28T00:09:03+00:00","mainEntityOfPage":{"@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/"},"wordCount":126,"image":{"@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#primaryimage"},"thumbnailUrl":"https:\/\/www.startupecosystem.ca\/news\/wp-content\/uploads\/2024\/11\/default-pic_big.jpg","keywords":["AI web crawlers","FOSS community","Global startup news","Open-source development","Robot.txt protocol"],"articleSection":["Global"],"inLanguage":"en-CA"},{"@type":"WebPage","@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/","url":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/","name":"Open Source Devs Battle AI Crawlers with Innovative Tools - Startup Ecosystem Canada","isPartOf":{"@id":"https:\/\/www.startupecosystem.ca\/news\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#primaryimage"},"image":{"@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#primaryimage"},"thumbnailUrl":"https:\/\/www.startupecosystem.ca\/news\/wp-content\/uploads\/2024\/11\/default-pic_big.jpg","datePublished":"2025-03-28T00:09:03+00:00","author":{"@id":"https:\/\/www.startupecosystem.ca\/news\/#\/schema\/person\/ebf38463111787b50c88f6fcd33f9c28"},"breadcrumb":{"@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#breadcrumb"},"inLanguage":"en-CA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/"]}]},{"@type":"ImageObject","inLanguage":"en-CA","@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#primaryimage","url":"https:\/\/www.startupecosystem.ca\/news\/wp-content\/uploads\/2024\/11\/default-pic_big.jpg","contentUrl":"https:\/\/www.startupecosystem.ca\/news\/wp-content\/uploads\/2024\/11\/default-pic_big.jpg","width":1000,"height":500},{"@type":"BreadcrumbList","@id":"https:\/\/www.startupecosystem.ca\/news\/open-source-devs-battle-ai-crawlers-with-innovative-tools\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.startupecosystem.ca\/news\/"},{"@type":"ListItem","position":2,"name":"Open Source Devs Battle AI Crawlers with Innovative Tools"}]},{"@type":"WebSite","@id":"https:\/\/www.startupecosystem.ca\/news\/#website","url":"https:\/\/www.startupecosystem.ca\/news\/","name":"Startup Ecosystem Canada","description":"Largest Startup Community in Canada","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.startupecosystem.ca\/news\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-CA"},{"@type":"Person","@id":"https:\/\/www.startupecosystem.ca\/news\/#\/schema\/person\/ebf38463111787b50c88f6fcd33f9c28","name":"Startup Eco System","image":{"@type":"ImageObject","inLanguage":"en-CA","@id":"https:\/\/secure.gravatar.com\/avatar\/a0296599ae4d98b5f644a6ebd0f36ee52f2642dac74ac9fee14854ec4a47d61d?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a0296599ae4d98b5f644a6ebd0f36ee52f2642dac74ac9fee14854ec4a47d61d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a0296599ae4d98b5f644a6ebd0f36ee52f2642dac74ac9fee14854ec4a47d61d?s=96&d=mm&r=g","caption":"Startup Eco System"},"sameAs":["https:\/\/www.startupecosystem.ca\/news"]}]}},"_links":{"self":[{"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/posts\/6621","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/comments?post=6621"}],"version-history":[{"count":0,"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/posts\/6621\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/media\/951"}],"wp:attachment":[{"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/media?parent=6621"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/categories?post=6621"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.startupecosystem.ca\/news\/wp-json\/wp\/v2\/tags?post=6621"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}