• IllNess@infosec.pub
      link
      fedilink
      English
      arrow-up
      18
      ·
      9 months ago

      All these AI and machine learning companies are taking content directly from websites and ignoring robot.txt files.

      If your content is able to be crawled, even without being listed on search engines, I don’t think it really matters.

      • T156@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        9 months ago

        It might help proof an AI company against legal issues that might be brought about by their using the content. If they’re ever sued by Automattic, then they can just point to the deal and say that they bought the data from them. There’s much less ambiguity.