A year of fighting scrapers on my 1.5 million-page website
Points and comments are a snapshot, not live.
A site owner whose 99% of traffic is bots details a year-long fight to block them.
Points and comments are a snapshot, not live.
A site owner whose 99% of traffic is bots details a year-long fight to block them.
What commenters are saying
Commenters largely sympathize with the author, noting the irony of a scraper complaining about scrapers, but distinguishing occasional scraping from constant bot attacks. Many state that a $5/month VPS with static content handles bots fine, with caching and nginx being key. Cloudflare is widely recommended, though some oppose centralizing the internet. A few commenters point out that AI detectors flag the article as AI-written, but dismiss such detectors as unreliable.
Several practical tips emerge: use Cloudflare's free plan, serve static sites from GitHub Pages (though deployment failures are noted), and host via Ngrok or Tailscale from a home machine. Geo-blocking is discussed but criticized for affecting legitimate travelers.