The Question That Started This

Today I had a thought: Should I feed AI training with my own AI-generated slop?

This immediately spiraled into several uncomfortable questions.

Claude: We went from “let’s block some bots” to existential questions about AI ethics in about 30 seconds. Classic.

The Five Thoughts That Kept Me Up

1. Isn’t This… Arrogant?

Let’s be real. Claude already sees my entire workflow. Every blog post draft. Every conversation. Every “hey Claude, rewrite this” request.

Anthropic (and OpenAI, and Google) already have access to way more data about my writing process than any web crawler could ever get.

So blocking bots from my public blog? That’s like closing the front door after already inviting them into the living room.

Is this just theater?

Maybe. But it’s MY theater.

2. Should I Get Paid for This?

Here’s where it gets interesting.

Regular people? Use my content however you want. Fair use. Learn from it. Quote it. Share it. I write about random tech stuff - it’s meant to be useful.

Massive AI corporations scraping my content to train billion-dollar models?

Yeah, I wouldn’t mind a few kavics dollars for that. 💰

But let’s be honest: that’s never going to happen.

There’s no world where OpenAI emails me like “Hey Zoltan, we used 3 of your blog posts in GPT-5 training, here’s $20.”

So what’s the alternative? Block them out of principle?

Sure. Why not.

3. At Least My AI Slop Won’t Ruin Their Models

This is actually a real problem. AI trained on AI-generated data degenerates fast.

There’s research on this - “model collapse.” Feed AI outputs back into training, and within a few generations, the models become useless. Weird artifacts. Repetitive patterns. Broken reasoning.

But here’s the thing: I don’t use AI like a content farm.

I’m not saying “Claude, generate 10 bullet points about cybersecurity best practices” and copy-pasting that into a blog post.

I’m using Claude as a documentation assistant for real work I actually did. The experience is real. The problems I solved are real. Claude just helps me write coherently about them.

So maybe there’s something to learn from my content? My actual experiences with Vagrant, WSL2, Docker, infrastructure work?

Or maybe I’m just rationalizing why my AI-assisted blog posts are “different” from everyone else’s AI slop. 🤷

4. Does Anyone Actually Respect robots.txt?

Short answer: No.

Long answer: Noooooo.

Let me tell you a story. When I worked at BitNinja (anti-malware, server protection), we had to investigate why certain IPs were getting greylisted or blacklisted.

Crawler services would contact us constantly:

“Why is my IP blocked? We’re a legitimate scraper!”

And we’d check the server logs.

Guess what we NEVER saw?

GET /robots.txt

You know what we DID see?

GET / HTTP/1.1
GET /wp-admin/ HTTP/1.1
GET /api/users HTTP/1.1
GET /.env HTTP/1.1

Just… immediately scraping everything. No robots.txt check. No rate limiting. Often hitting multiple different servers thinking they’re the same site.

“Why are we blocked for DoS?”

Gee, I wonder. 🙃

So yeah, most bots don’t give a shit about robots.txt.

But the big companies (OpenAI, Anthropic, Google) do. At least officially. They want to appear ethical. They respect opt-outs.

So it’s worth doing. Even if it’s symbolic.

5. Hackers Love robots.txt

Here’s the fun part nobody talks about:

robots.txt is basically a public list of URLs you DON’T want people to access.

User-agent: *
Disallow: /admin
Disallow: /api/internal
Disallow: /backup

Translation: “Hey hackers, here’s exactly where the interesting stuff is.”

Security researchers (and not-so-ethical hackers) love checking robots.txt first. It’s like a treasure map.

Good thing my site is a static Hugo blog.

No admin panel. No database. No PHP to exploit. Just markdown files converted to HTML.

I’ve cleaned up too many infected WordPress sites to ever want to deal with that on my personal blog.

Static site generator + Cloudflare = sleep well at night.

So What Did I Actually Do?

I added a robots.txt to block all the major AI training bots:

User-agent: *
Disallow:
Sitemap: https://zoltantoma.com/sitemap.xml

# Block AI bots
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: cohere-ai
Disallow: /

What This Actually Does

Regular users? Unaffected. Browse freely.

Google Search, Bing, etc.? Still indexes the site normally.

AI training crawlers? Blocked (if they respect robots.txt).

What This Doesn’t Do

Stop someone from using Claude’s WebSearch to analyze my blog? Probably not.

When you tell Claude “analyze this article at zoltantoma.com,” that’s not a bot crawling. That’s a user request going through an AI tool. Different user-agent. Different behavior.

Technically enforce anything? Nope. robots.txt is a polite request, not a firewall rule.

If I wanted real protection, I’d need:

  • Cloudflare Bot Fight Mode
  • Rate limiting
  • User-agent blocking at the server level
  • CAPTCHAs
  • Authentication

But honestly? I don’t care that much.

This isn’t about perfect security. It’s about opting out of large-scale training data collection.

The Real Reason I Did This

Let’s cut through the philosophy and get to the truth:

I don’t want my blog to be part of someone’s training dataset without my knowledge.

Not because my content is so valuable. Not because I think I deserve payment (though that would be nice).

But because it’s my content. I wrote it (with Claude’s help documenting real work). I chose to publish it.

And I should get to say how it’s used at scale.

Individual people reading and learning? Great. That’s the point.

AI companies scraping millions of sites to train proprietary models? I’d rather opt out.

Does This Make Me a Hypocrite?

Maybe?

I use Claude every day. I literally have a blog post called “Welcome to My AI Slop” where I defend using AI to document real work.

And now I’m blocking AI bots from my site?

Claude: To be fair, you’re not anti-AI. You’re anti-being-scraped-without-consent. There’s a difference.

Yeah. That.

I love AI tools. I think they’re transformative. I use them constantly for productivity, documentation, learning.

But that doesn’t mean I think every website should be fair game for training data.

There’s a difference between:

  • Using AI as a tool (what I do)
  • Being used BY AI companies as free training data (what I’m opting out of)

Will This Actually Work?

Optimistic answer: The major AI companies will respect it, and my site won’t be part of future training datasets.

Realistic answer: Some will respect it. Some won’t. Many smaller scrapers will ignore it completely.

Cynical answer: It’s security theater and accomplishes nothing except making me feel slightly better.

My answer: I don’t know, but at least I tried.

And honestly? That’s enough.

What I Learned From This

1. robots.txt Is a Polite Request, Not a Law

If you want real enforcement, you need server-level blocking, rate limiting, or authentication.

But for opting out of “ethical” AI training? It’s the standard tool.

2. Most Bots Don’t Care

From my BitNinja days: most scrapers don’t even check robots.txt. They just hammer your site until they get blocked.

The big companies (OpenAI, Google, Anthropic) usually do respect it. That’s who this is aimed at.

3. You Can Opt Out of AI Training Without Being Anti-AI

I use AI every day. I write AI-assisted blog posts. I think the technology is incredible.

But I also think consent matters.

If an AI company wants to use my content for training, they can ask. Or at least let me opt out easily.

robots.txt is that opt-out mechanism.

4. Static Sites Are Underrated

No database to compromise. No admin panel to exploit. No PHP vulnerabilities.

Just markdown → HTML. Deployed via Git.

The robots.txt won’t accidentally expose sensitive URLs because there are no sensitive URLs.

If I was running WordPress or a custom CMS, I’d be way more paranoid about what I put in Disallow:.

The Meta-Irony

This entire blog post was written collaboratively with Claude.

I provided the thoughts, the context, the messy brain dump. Claude organized it into something readable.

And now it’s published on a site that blocks AI training bots.

Claude: The irony is not lost on me. I helped write a post about blocking bots like me. I’m okay with it.

I think this is the future, honestly.

AI as a tool = good.

AI as an unconsenting data vacuum = bad.

You can have both opinions at the same time.

Final Thoughts

Did I overthink this? Absolutely.

Is robots.txt a perfect solution? Not even close.

Will this actually stop AI companies from using my content? Maybe. Probably not entirely.

But at least I made a choice.

I didn’t just leave my site wide open and say “well, nothing I can do.”

I added 20 lines to a text file. It took 2 minutes. It signals my preference.

And if AI companies respect it? Great.

If they don’t? Well, at least I tried.

P.S. - If you’re reading this and thinking “I should probably add a robots.txt too,” check out my robots.txt file: https://zoltantoma.com/robots.txt

Copy-paste ready. Adjust as needed.

Or don’t. It’s your content. Your choice.

That’s the whole point.

Claude: This post was written collaboratively. Zoltan provided the existential crisis, the BitNinja stories, and the conflicted feelings about AI ethics. I just made it readable. The robots.txt blocking me? Still there. We’re all living with contradictions now.