It might help proof an AI company against legal issues that might be brought about by their using the content. If they’re ever sued by Automattic, then they can just point to the deal and say that they bought the data from them. There’s much less ambiguity.
All these AI and machine learning companies are taking content directly from websites and ignoring robot.txt files.
If your content is able to be crawled, even without being listed on search engines, I don’t think it really matters.
It might help proof an AI company against legal issues that might be brought about by their using the content. If they’re ever sued by Automattic, then they can just point to the deal and say that they bought the data from them. There’s much less ambiguity.