# Retrieval without the pilot trap

> A retrieval assistant that answers well in a demo can fail quietly in production. What to measure, what to bound, and where human review belongs.

Source: https://itnetsolution.com/resources/insights/retrieval-without-the-pilot-trap

Artificial Intelligence

8 October 2026

7 min read

## Retrieval without the pilot trap

A retrieval assistant that answers well in a demo can fail quietly in production. What to measure, what to bound, and where human review still belongs.

By IT NET Solution

Demos are curated. Someone selects the document, asks the obvious question and reads the answer aloud. Production is uncurated: a colleague asks a slightly wrong question about an outdated policy and receives a confident, well-formatted answer that happens to be wrong.

### Bound the corpus before tuning the prompt

Retrieval quality is mostly a decision about what is allowed into the index. Three document families with clear owners, a review cadence and unambiguous versioning will outperform a whole repository that nobody has read in years.

- Exclude superseded versions rather than relying on ranking to bury them.
- Carry source permissions into retrieval, so the assistant cannot reveal what the user could not open.
- Attach document, version and section to every answer, and make the citation visible.
- Decide what the assistant should refuse, and test those refusals like any other requirement.

### Evaluation is a small, boring set of questions

A useful evaluation set is fifty real questions with agreed answers and the passages that justify them. It is not impressive, but it turns "the assistant feels better" into a regression check that runs after every change to chunking, prompts or model routing.

### Where a person still belongs

Client-facing text, contractual statements and anything that commits the business should be reviewed by someone accountable for it. The assistant's job is to prepare a grounded draft quickly; approval remains an explicit step, and the review is logged.

The honest measure

Track how often a generated answer is corrected or discarded before use. That number describes the assistant far better than any satisfaction score.

Topics

- Enterprise AI
- RAG
- Evaluation
- Governance

Related service: Artificial Intelligence

In this article

- Bound the corpus before tuning the prompt
- Evaluation is a small, boring set of questions
- Where a person still belongs

Working on something similar?

Describe the systems, the constraints and what needs to change. We will outline a realistic path and the trade-offs behind it.

Talk to us

Related reading

Artificial Intelligence

27 May 2026

#### Governing model access across the enterprise

Teams adopting several AI providers quickly accumulate keys, costs and inconsistent behaviour. A gateway model for routing, logging and controlled change.

5 min read

Product update

21 July 2026

#### Indurant adds multilingual retrieval

Our enterprise knowledge platform now retrieves across Hungarian and English documents in the same query, preserving citations and the permission model of the source systems.

4 min read

Case study

2 June 2026

#### A grounded knowledge assistant for a service organisation

An illustrative AI programme: policies, contracts and technical documentation made answerable through retrieval, with existing permissions and source citations kept intact.

7 min read

### Working on something similar?

Describe the systems, the constraints and what needs to change. We will outline a realistic path and the trade-offs behind it.

Talk to us

Explore services
