• Zak@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    8 months ago

    If I’m reading this comment right, it’s relying on a mistaken understanding of robots.txt. It is not an instruction to the server hosting it not to serve certain robots. It’s actually a request to any robot crawling the site to limit its own behavior. Compliance is 100% voluntary on the part of the robot.

    The ability to deny certain requests from servers that self-report running a version of their software with known vulnerabilities would be useful.