How do robots.txt, noindex and sitemaps control what crawlers see?
robots.txt controls crawling, noindex controls indexing, and a sitemap lists the pages you want found. None of them is a security control.
robots.txt tells crawlers which paths they may fetch. Google notes that it is not a mechanism for keeping a page out of search results — a blocked URL can still be indexed if other pages link to it.
A noindex meta tag tells search engines not to show a page in results, but the crawler must be able to fetch the page to see the tag. Neither robots.txt nor noindex protects private information; use authentication for that.
An XML sitemap lists the canonical URLs you want discovered. It should only contain public pages you'd be happy to see in search results.
To allow ChatGPT search, permit the OAI-SearchBot user agent in robots.txt; for Perplexity search, permit PerplexityBot. Training crawlers such as GPTBot are controlled separately.