Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

robots.txt and dmca seems appropriate


Yeah. After I saw the bot I added it to my robots.txt and I blocked all of Cloudflare's IP ranges to the server. But the bot had already come and gone. And the email I'm waiting back on from Cloudflare is about what to put in the robots.txt since it's not on the FAQ page. As of right now I'm trying,

User-agent: Cloudflare-AMP

Disallow: /


Doesn't DMCA have specific exemptions around caching.


my understanding is possibly, but not in violation of robots.txt




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: