> specific to robots.txt
> poor summarization of data
I'm not really addressing the issue raised in the article. I am noting that the LLM, when asked, is either lying to the user or making a statement that it does not know to be true (that there is no robots.txt). This is way beyond poor summarization.
There is a difference between doing a poor summarization of data, and failing to even be able to get the data to summarize in the first place.