mgx

re: "the great scrape"

I read Herman's blog post "The Great Scrape" about AI scraping and its impact on Bear. As the person behind bear.css.observer, his new security measures and strict Cloudflare checks threw a wrench in my little worker.js setup. Talk about being caught in the middle...

I want to respect Herman's fight against AI scraping while keeping bear.css.observer working for real users. My solution handles Cloudflare's verification by using browser-like headers, managing cookies properly, dealing with challenge-response steps, and implementing decent cache control.

This keeps my web worker running without undermining Bear's security - I'm just making sure legitimate requests get through properly. Of course, this might break tomorrow as security evolves, and that's fine - comes with the territory when building unofficial tools. I'm 100% behind Herman protecting Bear.


Personally, I'm just glad my automations and webhooks still work. I need these for my workflow since Bear has no official API. My "dirty" approach to syncing and rebuilding my blog has been solid so far. Fingers crossed it stays that way while respecting what Herman's trying to do.