Vice President · TheGreyMatter.ai
In brief
A fresh RAG system retrieves the correct current version for the right user and stops retrieving withdrawn content. A recent sync timestamp alone does not establish any of those conditions.
Key takeaways
- Test updates, deletions, and permission changes separately.
- Track source identity and revision through every generated chunk.
- Measure how long changes take to become visible in real retrieval results.
A policy assistant can provide a convincing answer from an obsolete policy. The model may accurately summarize the passage it received while the overall answer is wrong for today's employee. Retrieval augmented generation, or RAG, depends on the quality and timing of the evidence supplied to the model. Keeping that evidence current is an operating process with owners, failure handling, and observable tests.
Updates and deletions need different checks
An update detector notices that a source changed. A deletion mechanism removes content that should no longer be searchable. Microsoft documents this distinction for Azure Storage indexers: change detection can use timestamps, while deletion detection requires an appropriate strategy and configuration. A policy added after initial ingestion may leave earlier orphaned records requiring separate cleanup.
That is one implementation example, not a universal guarantee about connectors. Ask each connector owner how it detects edits, moves, deletions, and access changes. Find out what happens when a sync fails for several runs. A dashboard that reports the most recent successful connection may still conceal a backlog of documents that were never updated or removed.
Keep a source manifest alongside the index
Our proposed manifest gives every source a stable identity independent of its display title. Record the source revision or content fingerprint, effective date when supplied, ingestion time, processing configuration, access policy reference, and identifiers of derived chunks. Keep the source's publication date separate from the time your system happened to fetch it.
The manifest should help an operator answer a concrete question: which searchable records came from this revision? If a document is shortened or removed, that relationship is necessary for finding stale fragments. Microsoft documents how index projections can maintain one-to-many relationships and remove obsolete chunks under defined conditions, while source deletion still depends on the configured deletion process. Verify your actual ingestion path.
- Stable source identifier and authoritative location.
- Source revision, content fingerprint, and effective date when available.
- Ingestion time and processing configuration version.
- Derived chunk identifiers and their parent source.
- Access policy reference, responsible owner, and last verified outcome.
Worked example: an expense policy changes
Imagine a fictional expense policy moving from version seven to version eight. The new document changes the approval route and removes an obsolete allowance. Prepare questions that previously retrieved each affected passage. After ingestion, inspect both the retrieved evidence and the generated answer. The answer should reference the applicable revision and should not quietly combine the new approval route with the old allowance.
Next, test a user who no longer has access and a query that directly targets the removed passage. Check that no retired child chunk remains retrievable through another search mode or cached answer path. If historical policies are intentionally retained, distinguish historical questions from current policy questions and enforce the relevant access rules. Keeping history is different from presenting it as current guidance.
Treat rebuilds as controlled releases
Changes to schema or processing can require more than an ordinary incremental update. Microsoft's index maintenance guidance describes rebuilding where necessary and testing a replacement index before switching an application to it. The ability to switch indexes is operationally useful, but it does not establish answer quality or correct access by itself.
Before a switch, compare representative questions, expected revisions, denied access cases, and deletion cases against the replacement. Record the source snapshot and processing configuration used. A rollback plan also needs a permission and deletion check: restoring an old index must not reintroduce material that was withdrawn after the old snapshot was created. Keep service recovery and content correctness in the same release decision.
Measure freshness from the user's side
Track the time from an authoritative source change to a verified searchable update. Track deletion to confirmed non-retrievability separately. Add backlog age, failed document count, and a small set of recurring questions tied to known current revisions. These proposed measures describe the outcome the user needs rather than only whether a scheduled job ran.
Set review targets based on the use case. A time-sensitive operating policy may need a different update process from an archive of historical essays. Name the person who investigates missed targets and the fallback shown when freshness is uncertain. A useful assistant can say that the current version has not been verified and point to the authoritative source instead of presenting an old answer with false certainty.
From reading to doing
Put it to work: run an update, withdrawal, and access drill
Create a fictional policy with a unique phrase and two versions. Version one routes an expense to a team lead; version two routes it to a budget owner. Ask the same current-policy question before and after the change. Save the retrieved source identifiers and revisions as well as the generated answer, so a correct-sounding response cannot hide an obsolete passage.
Then withdraw the document and remove a test user's access in separate runs. Separate runs help identify which mechanism failed. If a stale answer appears, investigate the relevant retrieval records, answer history, and cache path within the application's scope. Do not infer backup deletion from a search result. This proposed drill measures visible application behavior and should be paired with the system's documented data lifecycle.
- Record a baseline question, expected answer, source identity, source revision, and test user's allowed access.
- Publish the revised synthetic policy. Note the change time and the first time retrieval and the answer both reflect the new revision.
- Withdraw the source and query its unique phrase. Check the search modes used by the application for orphaned records.
- Restore the synthetic source for a separate access test, revoke the test user's permission, and query through the application again.
- Record each delay and failure separately, assign an owner, and decide what the application should show while currency or access is uncertain.
Common questions
Does RAG automatically use live information?
No. It retrieves from the sources and indexes connected to the application. Freshness depends on their update cadence, processing success, version handling, and query behavior.
Does deleting the original document delete its embeddings?
Not necessarily. The ingestion system must remove the associated searchable records and derived chunks. Verify deletion behavior for the configured connector and any additional stores.
Must every update re-embed the entire knowledge base?
Not always. Some systems can update affected documents or chunks. Changes to embedding models, schema, or processing may require a broader rebuild. Test the actual index and pipeline requirements.
Sources and evidence
AI assisted editorial analysis. Primary sources checked October 10, 2026. The worked examples and proposed review procedures are editorial illustrations, not reported customer results, vendor guarantees, or claims of personal experience.
One useful next step: Choose one idea from this note and test it at the smallest scale that could teach you something this week.
