EnterraHost
To block crawling
Use robots.txt. This is what it is for, and it is the right tool for keeping a crawler out of paths it wastes time on, such as internal search results or faceted URLs.
Fetch a site robots file, list what it actually says, and test real paths against it. The rules are matched the way crawlers match them, not the way people assume.
The part people get wrong
This is the single most expensive misunderstanding in the file. A URL you disallow is not removed from search, and can still rank. A crawler that is blocked never reads the page, so it never sees a noindex instruction, and a page linked from elsewhere can be indexed on the strength of that link alone.
Use robots.txt. This is what it is for, and it is the right tool for keeping a crawler out of paths it wastes time on, such as internal search results or faceted URLs.
Use a noindex robots meta tag or the equivalent HTTP header, and let the page be crawled so the instruction is actually read. Blocking the crawl and asking for noindex at the same time cancels the second request.
Matching
Disallow /private also blocks /private-notes and /private/report. A path with no wildcard and no end anchor matches anything that starts with it, which is why a rule that looks tidy can be wider than intended.
Allow and Disallow are not a first-match list. Every rule is tested and the most specific one, measured by pattern length, decides. A short Disallow cannot override a longer Allow.
When an Allow and a Disallow match with the same pattern length, the allow rule wins. That is the behaviour in RFC 9309 and it is the opposite of what most people expect when they write the two lines in the wrong order.
Next
A correct robots file does not tell you whether the pages behind it resolve. The next check follows the links and reports what breaks.