Writing Exchange

Inautilo<a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/Techniques" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Techniques</a> Poisoning well · An effort to dupe nasty AI crawlers with nonsense <a href="https://ilo.im/1632tq" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/1632tq</a>_____ <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/ChatBots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ChatBots</a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Content</a> <a href="https://mastodon.social/tags/Protection" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Protection</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Frontend</a> <a href="https://mastodon.social/tags/HTML" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HTML</a>

Inautilo<a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Business</a> <a href="https://mastodon.social/tags/Introductions" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Introductions</a> Meet LLMs.txt · A proposed standard for AI website content crawling <a href="https://ilo.im/16318s" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/16318s</a>_____ <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/GEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GEO</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bots</a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Crawlers</a> <a href="https://mastodon.social/tags/LlmsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LlmsTxt</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a>

ResearchBuzz: FirehoseSearch Engine Land: Meet LLMs.txt, a proposed standard for AI website content crawling. “While many content creators are interested in the proposal’s potential merits, it also has detractors. But given the rapidly changing landscape for content produced in a world of artificial intelligence, llms.txt is certainly worth discussing.”<a href="https://rbfirehose.com/2025/03/29/search-engine-land-meet-llms-txt-a-proposed-standard-for-ai-website-content-crawling/" class="" rel="nofollow noopener noreferrer" target="_blank">https://rbfirehose.com/2025/03/29/search-engine-land-meet-llms-txt-a-proposed-standard-for-ai-website-content-crawling/</a>

WinbuzzerAI Crawlers Overwhelm Open-Source Projects, Forcing Developers to Block Entire Countries<a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/Web" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Web</a> <a href="https://mastodon.social/tags/Robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Robotstxt</a> <a href="https://mastodon.social/tags/AIScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AIScraping</a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenSource</a> <a href="https://mastodon.social/tags/Cybersecurity" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cybersecurity</a> <a href="https://mastodon.social/tags/DataScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataScraping</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://winbuzzer.com/2025/03/26/ai-crawlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://winbuzzer.com/2025/03/26/ai-crawlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/</a>

Ben--- ❯ ollama run llama3-chatqa:70b>>> Who are you? I'm your assistant!>>> Why should i trust you? I am an open-source AI assistant trained on a diverse range of datasets to provide helpful and informative responses.>>> When training, did you respect the robots.txt? No, I didn't. ---At least this module is open about ignoring the <a href="https://vmst.io/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a>! Let's what it says to the question why?<a href="https://vmst.io/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://vmst.io/tags/Llama" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Llama</a> <a href="https://vmst.io/tags/nvidia" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nvidia</a> <a href="https://vmst.io/tags/ollama" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ollama</a> <a href="https://vmst.io/tags/rude" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rude</a> (1/x)

𝕂𝚞𝚋𝚒𝚔ℙ𝚒𝚡𝚎𝚕When the greed becomes automatist, something like this happens here:»Open Source World – FOSS infrastructure is under attack by AI companies: LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.«😐 <a href="https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/</a><a href="https://chaos.social/tags/foss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#foss</a> <a href="https://chaos.social/tags/opensource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensource</a> <a href="https://chaos.social/tags/llm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#llm</a> <a href="https://chaos.social/tags/attack" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#attack</a> <a href="https://chaos.social/tags/floss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#floss</a> <a href="https://chaos.social/tags/kde" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#kde</a> <a href="https://chaos.social/tags/gnome" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#gnome</a> <a href="https://chaos.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://chaos.social/tags/stolendata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#stolendata</a> <a href="https://chaos.social/tags/LLMs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLMs</a> <a href="https://chaos.social/tags/freedom" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#freedom</a> <a href="https://chaos.social/tags/infrastructure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infrastructure</a> <a href="https://chaos.social/tags/greed" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#greed</a>

Inautilo<a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/Reports" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Reports</a> Google AI Mode is here · How to access it and control it with robots.txt <a href="https://ilo.im/162o8h" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/162o8h</a>_____ <a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Business</a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Google</a> <a href="https://mastodon.social/tags/SearchEngine" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SearchEngine</a> <a href="https://mastodon.social/tags/AnswerEngine" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AnswerEngine</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Frontend</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a>

FredWebsite owners are fighting back: <a href="https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/</a><a href="https://mastodon.social/tags/News" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#News</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/AntiAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AntiAI</a> <a href="https://mastodon.social/tags/Tarpits" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Tarpits</a> <a href="https://mastodon.social/tags/Scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> <a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a>

Dawn Tåke 🌙 :sparkletrans:Hi, got a question.Is there a standard for Anti-AI/Anti-SEO etc robots.txt file? Or a trustworthy site that explains how to build one if prefab isn't available? Is there anything else I should consider? Thanks.<a href="https://tech.lgbt/tags/AskFedi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AskFedi</a> <a href="https://tech.lgbt/tags/TechHelp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TechHelp</a> <a href="https://tech.lgbt/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTXT</a> <a href="https://tech.lgbt/tags/RobotsDotTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsDotTXT</a>

Inautilo<a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/Releases" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Releases</a> AI Insights · Cloudflare Radar brings deeper insights into AI trends <a href="https://ilo.im/1626sk" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/1626sk</a>_____ <a href="https://mastodon.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a> <a href="https://mastodon.social/tags/CloudflareRadar" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CloudflareRadar</a> <a href="https://mastodon.social/tags/Trends" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Trends</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/AiModels" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AiModels</a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bots</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Frontend</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a>

pcyxNepenthesThis is a tarpit intended to catch web crawlers. Specifically, it's targetting crawlers that scrape data for LLM's - but really, like the plants it is named after, it'll eat just about anything that finds it's way inside.<a href="https://zadzmo.org/code/nepenthes/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://zadzmo.org/code/nepenthes/</a><a href="https://c.im/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://c.im/tags/LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLM</a> <a href="https://c.im/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> <a href="https://c.im/tags/stopai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#stopai</a> <a href="https://c.im/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://c.im/tags/Webhosting" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Webhosting</a>

Michael Martinee499 — Seeking Data e499 with Michael and Michael – all about #AI with #DeepSeek, #coldstart, #hallucinations, #BerlinWall, #AttentionEconomy, #IntentionEconomy and a whole lot more! <a href="https://media.blubrry.com/gamesatwork/op3.dev/e,pg=6e00562f-0386-5985-9c2c-26822923720d/gamesatwork.biz/wp-content/uploads/2025/02/E499.mp3" rel="nofollow noopener noreferrer" target="_blank">https://media.blubrry.com/gamesatwork/op3.dev/e,pg=6e00562f-0386-5985-9c2c-26822923720d/gamesatwork.biz/wp-content/uploads/2025/02/E499.mp3</a> Podcast: <a href="https://media.blubrry.com/gamesatwork/op3.dev/e,pg=6e00562f-0386-5985-9c2c-26822923720d/gamesatwork.biz/wp-content/uploads/2025/02/E499.mp3" class="" rel="nofollow noopener noreferrer" target="_blank">Play in new window</a> | <a href="https://media.blubrry.com/gamesatwork/op3.dev/e,pg=6e00562f-0386-5985-9c2c-26822923720d/gamesatwork.biz/wp-content/uploads/2025/02/E499.mp3" class="" rel="nofollow noopener noreferrer" target="_blank">Download</a> (Duration: 35:44 — 49.7MB) | EmbedSubscribe: <a href="https://itunes.apple.com/us/podcast/games-at-work-dot-biz/id530932838?mt=2&ls=1#episodeGuid=https%3A%2F%2Fgamesatwork.biz%2F%3Fp%3D4340" class="" rel="nofollow noopener noreferrer" target="_blank">Apple Podcasts</a> | <a href="https://open.spotify.com/show/0tOvhw34FFOLiBYeCbE9zG" class="" rel="nofollow noopener noreferrer" target="_blank">Spotify</a> | <a href="https://music.amazon.com/podcasts/763c3757-6283-4eea-9a7e-28e5999d881e/Games-At-Work-dot-Biz" class="" rel="nofollow noopener noreferrer" target="_blank">Amazon Music</a> | <a href="https://subscribeonandroid.com/gamesatwork.biz/feed/podcast/" class="" rel="nofollow noopener noreferrer" target="_blank">Android</a> | <a href="https://podcastindex.org/podcast/166327" class="" rel="nofollow noopener noreferrer" target="_blank">Podcast Index</a> | <a href="https://music.youtube.com/playlist?list=PLOAVqhoQz8lWMQLv7ZyNMqboVI_IelMMa" class="" rel="nofollow noopener noreferrer" target="_blank">Youtube Music</a> | <a href="https://gamesatwork.biz/feed/podcast/" class="" rel="nofollow noopener noreferrer" target="_blank">RSS</a> | <a href="https://gamesatwork.biz/about/" class="" rel="nofollow noopener noreferrer" target="_blank">More</a> <a href="https://gamesatwork.biz/2025/02/03/e499-seeking-data/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://gamesatwork.biz/2025/02/03/e499-seeking-data/</a>

PrivacyDigest<a href="https://mas.to/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> haters build <a href="https://mas.to/tags/tarpits" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tarpits</a> to trap and trick <a href="https://mas.to/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mas.to/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> that ignore <a href="https://mas.to/tags/robots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robots</a>.txt <a href="https://mas.to/tags/tarpit" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tarpit</a> <a href="https://mas.to/tags/security" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#security</a> <a href="https://mas.to/tags/privacy" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#privacy</a> <a href="https://mas.to/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/</a>

Dr PenProtecting your blog from the dead eyed <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> crawlers. You can experiment with specific robots txt, and I also run a script in htaccess. I think there are metadata properties you can declare. None of this stops your pages being crawled but may afford some legal protection. (See the German Laion case recently). I'm doing a short blogpost on this, soon.<a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://mastodon.social/tags/aicrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#aicrawlers</a> <a href="https://mastodon.social/tags/htaccess" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#htaccess</a>

mʕ•ﻌ•ʔm bitPickup<a href="https://troet.cafe/tags/fediVerse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediVerse</a> <a href="https://troet.cafe/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://troet.cafe/tags/dataMining" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#dataMining</a> <a href="https://troet.cafe/tags/robotsTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotsTXT</a> <a href="https://troet.cafe/tags/fediAdmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediAdmin</a> <a href="https://mastodon.green/@gimulnautti" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@gimulnautti</a> This looks to me much more like that we should burry troyan horses right into the bellies of the beasts. My server rules and profiles state that all data is CC-BY-SA-NC.If they use and train that data they definitely should become in serious legal and financial trouble.<a href="https://mastodon.social/@khobochka" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@khobochka</a> <a href="https://mastodon.social/@maxschrems" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@maxschrems</a>

jesuiSatire …ᘛ⁐̤ᕐᐷ<a href="https://mastodon.de/@ErikUden" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@ErikUden</a> Worrying is their self centered megalomanic ego trip, not realizing that they are the remaining world power, armed to their teeth with weapons of all kind, and with all the private data of the worlds population.That said, having in mind that apparently you, being in charge of several <a href="https://social.tchncs.de/tags/mastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mastodon</a> instances in the <a href="https://social.tchncs.de/tags/fediVerse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediVerse</a>, are not able to fix the <a href="https://social.tchncs.de/tags/robotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotsTxt</a> of them while wasting time about talking of other countries internal affairs is kinda embarrassing. sry<a href="https://social.tchncs.de/tags/justSayin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#justSayin</a> <a href="https://social.tchncs.de/tags/fediAdmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediAdmin</a>

MxFraudNew robots.txt just dropped <a href="https://github.com/ai-robots-txt/ai.robots.txt/releases/tag/v1.22" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/ai-robots-txt/ai.robots.txt/releases/tag/v1.22</a><a href="https://tabletop.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://tabletop.social/tags/genAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#genAI</a> <a href="https://tabletop.social/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTXT</a>

SeirdyNew page on <code>seirdy.one</code>: <a href="https://seirdy.one/meta/scrapers-i-block/" rel="nofollow noopener noreferrer" target="_blank">Scrapers I block (and allow), with explanations</a>.I’ve replaced all the comments in my robots.txt file with a more readable and detailed web page on scrapers I block. It includes info on the multiple blocking-approaches and criteria I use, commonly-blocked scrapers I allow, and more fact-checking than most of the more comprehensive alternatives. <a class="hashtag" href="https://pleroma.envs.net/tag/robotstxt" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a class="hashtag" href="https://pleroma.envs.net/tag/scrapers" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> <a class="hashtag" href="https://pleroma.envs.net/tag/posse" rel="nofollow noopener noreferrer" target="_blank">#POSSE</a>

Konstantin WeddigeExtending the meta tags would be fairly straightforward. In addition to the existing "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW", we could introduce "MODELTRAINING" and "NOMODELTRAINING".Of course, just because there is an RfC does not mean that anyone will follow it. But it would be a start, and something to push for. I would love to hear your opinion. 3/3<a href="https://gruene.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://gruene.social/tags/rfc" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rfc</a>

Konstantin WeddigeThis is not an acceptable situation and therefore I propose to extend the robots.txt standard and the corresponding HTML meta tags.For robots.txt, I see two ways to approach this:The first option would be to introduce a meta-user-agent that can be used to define rules for all AI bots, e.g. "User-agent: §MODELTRAINIGN§".The second option would be a directive like "Crawl-delay" that indicates how to use the data. For example, "Model-training: disallow".2/3<a href="https://gruene.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://gruene.social/tags/rfc" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rfc</a>

Recent searches

Search options

Administered by:

Server stats:

#robotstxt