
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.
A retrieval-augmented system is only as good as the corpus behind it. Point the same model at uncleaned, duplicated, permission-blind documents and it will confidently cite the wrong version of a superseded policy. Data engineering is the part of RAG that decides whether the answer is merely fluent or actually correct.
Most RAG failures are not model failures. They are corpus failures: superseded documents still in the index, the same policy stored in four places with four versions, tables and PDF layouts destroyed by naive text extraction, no metadata to filter by department or date, and no way to enforce that a user only retrieves what they are allowed to read. Unified Management Consulting treats the knowledge layer as an engineering deliverable with connectors, a metadata model, permission enforcement, an evaluation set and a refresh schedule — not as a one-off upload into a vector database.
🗃️ Data Engineering for AI (RAG & Knowledge Systems) — Quick Summary
| Component | What We Deliver |
|---|---|
| Source audit | Every candidate knowledge source inventoried with owner, format, volume, refresh cadence, sensitivity classification and whether it is actually authoritative. |
| Ingestion connectors | Scheduled, authenticated connectors to SharePoint, document stores, wikis, ticketing, CRM and databases — with incremental sync and deletion handling. |
| Cleaning & deduplication | Text extraction that preserves tables and structure, removal of superseded versions, near-duplicate detection, and normalisation of dates, units and entities. |
| Metadata model | Department, document type, effective date, jurisdiction, sensitivity, source system and authority rank — attached to every chunk so retrieval can filter, not just search. |
| Chunking & embeddings | Structure-aware chunking strategy chosen per document type, with the rationale recorded, and embedding model selected against your corpus rather than by default. |
| Permission-aware retrieval | Row-level access control enforced at query time from the source system’s own permissions, so the AI can never surface what the user cannot read. |
| Evaluation harness | A versioned question set with expected answers and source citations, run on every corpus or model change, reporting groundedness and refusal behaviour. |
| Operational refresh | Incremental re-indexing, drift detection, orphan cleanup and a schedule that keeps the corpus current without a full rebuild. |
This service underpins AI Agent Deployment & Automation and AI Product Engineering for production-ready intelligent products. Where the knowledge system also has to serve AI search visibility, it connects to AIEO Services and to how to get indexed by ChatGPT and AI search tools.
Why RAG Projects Stall on Data, Not Models
A RAG project spends most of its effort on plumbing rather than on intelligence. Figure 2 shows the realistic distribution: connectors and ingestion take roughly a third, chunking and metadata another quarter, and the model-and-prompt work that gets all the attention accounts for under a tenth. Teams that budget the other way round run out of money while the corpus is still wrong.
The reason is that enterprise knowledge is not a dataset. It is spread across systems with different access models, it changes continuously, much of it is superseded but still present, and a significant share is structured data trapped in PDF tables that naive extraction flattens into noise. Cleaning that up is data engineering work with the same disciplines as any other data platform — lineage, quality rules, scheduling, ownership and testing.
| Symptom the business reports | Actual root cause | Where to fix it |
|---|---|---|
| “It gives answers that sound right but are wrong” | Superseded documents still indexed; no authority or effective-date metadata | Metadata model plus version pruning at ingestion |
| “It cannot find the document I know exists” | Chunking splits the relevant passage, or the file format extracts badly | Structure-aware chunking per document type |
| “It mixes up regions, brands or business units” | No entity or scope metadata; retrieval matches on wording alone | Faceted metadata and filtered retrieval |
| “Legal will not approve it” | No permission enforcement; retrieval is corpus-wide | Query-time access control from the source system |
| “It got worse after the last update” | No evaluation harness, so changes are unmeasured | Versioned evaluation set run on every change |
| “Answers stopped being current” | One-off load with no incremental refresh or deletion handling | Scheduled connectors with tombstone handling |

Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.
Source Audit: Establishing What Is Actually Authoritative
Before any connector is written, the corpus has to be surveyed. This is the step teams skip, and it is the step that determines whether the system answers from the right document.
- Inventory every candidate source with system, owner, format, volume, update frequency, sensitivity classification and retention rules.
- Rank authority. Where two sources disagree, which one wins? A signed policy beats a team wiki; a current version beats an archived one. Encode this as an authority score in metadata rather than leaving it to retrieval luck.
- Identify superseded content. Most organisations hold several generations of every document. Anything not explicitly excluded will eventually be cited.
- Classify sensitivity. Public, internal, confidential, restricted. This drives both the permission model and what may be sent to an external model at all.
- Assess extractability. Scanned PDFs, image-heavy decks, complex spreadsheets and legacy exports each need a different extraction path. Test extraction quality on samples before committing to a volume estimate.
- Confirm access rights. For each source, determine whether permissions can be read programmatically. If they cannot, that source needs a separate access design or should be excluded.
The audit produces a source register that becomes the contract for the build. It also produces the first uncomfortable finding, which is usually that a large share of the knowledge employees rely on lives in personal drives, chat history and inboxes — none of which is a sustainable corpus. Addressing that is an information-management question as much as a technical one, and it is where Risk, Governance & Business Continuity Support and our AI governance policy framework come in.
Ingestion Connectors and Incremental Sync
A corpus loaded once is stale within weeks. Production ingestion has to be scheduled, incremental, authenticated and able to handle deletion — because a document withdrawn for legal reasons must stop being retrievable immediately, not at the next full rebuild.
| Requirement | What it means in practice | Failure mode if skipped |
|---|---|---|
| Scheduled incremental sync | Only changed items re-processed, on a defined cadence | Full rebuilds become unaffordable, so refreshes stop happening |
| Deletion and tombstone handling | Removed or withdrawn items purged from the index and any cache | Withdrawn content keeps being cited — a compliance incident |
| Authentication per source | Service accounts with least privilege, secrets in a vault | Connector breaks silently when a person leaves or a token expires |
| Retry and dead-letter queue | Failed items recorded and re-tried, not dropped | Silent gaps in the corpus that no one notices until a user asks |
| Lineage capture | Every chunk traceable to source, version, ingest time and checksum | Impossible to answer “where did this answer come from?” |
| Volume and rate control | Throttling against source API limits | Connector gets blocked, or the source system degrades |
| Change detection accuracy | Content hash plus modified timestamp, not timestamp alone | Re-indexing everything on every run, or missing real changes |
Cleaning, Deduplication and Version Control
Cleaning is where the visible quality gain comes from. Four operations matter more than the rest.
Structure-preserving extraction. A pricing table extracted as a run of numbers loses its meaning entirely. Extraction has to preserve headings, table rows and columns, list structure and document hierarchy, and carry them into the chunk as either formatted text or metadata. For scanned documents this means an OCR pass with confidence scoring and a human sample check.
Near-duplicate detection. The same policy appears as a Word original, a PDF export, a wiki copy and three email attachments. Embedding-based similarity plus metadata matching identifies clusters, and the authority rank picks the surviving copy. Leaving duplicates in place means retrieval slots are wasted on near-identical text and the citation looks arbitrary.
Version pruning. Superseded versions are either excluded or retained with an explicit status flag and a lower authority weight, so a historical question can still be answered without contaminating current-policy answers. This distinction has to be a deliberate decision per document type, not a default.
Normalisation. Dates to one format, units to one system, entity names to a controlled vocabulary, and acronyms expanded at first use in each chunk. Chunks are read in isolation, so a chunk that says “the SLA” without naming which SLA is far less useful than one that carries the full entity reference in its metadata.
Chunking Strategy: The Decision With the Largest Quality Effect
Chunking determines what retrieval can see. There is no universally correct size; there is a correct strategy per document type, and it should be chosen by evaluation rather than by convention.
| Document type | Recommended strategy | Why | Typical chunk target |
|---|---|---|---|
| Policy and procedure | Section-aware split on headings, chunk carries section path | Questions are about a clause, not the whole document | 300–600 tokens |
| Contracts and legal | Clause-level split with defined-term preservation | Definitions are referenced across clauses and must travel with the text | 250–500 tokens |
| Technical manuals | Heading hierarchy with overlap, tables kept intact | Procedures span steps that must not be separated | 400–800 tokens |
| Meeting notes and tickets | Per-item or per-thread, with entity and date metadata | Highly repetitive wording; metadata does the disambiguation | 150–400 tokens |
| Spreadsheets and reports | Row-group or section summary plus retained raw table | Numeric tables lose meaning when flattened to prose | Per table or per section |
| Marketing and web content | Topic-level split with URL and publication date | Freshness matters as much as relevance | 300–700 tokens |
Two refinements consistently help. Contextual chunk headers — prefixing each chunk with document title, section path and effective date — lift retrieval precision noticeably because the embedding sees the scope, not just the wording. Parent-child retrieval — embedding small chunks but returning the larger surrounding section to the model — gives precise matching without starving the model of context. Both add cost, so evaluate them rather than assuming them. Our AI model evaluation framework covers how to run that comparison properly.
Metadata Model and Faceted Retrieval
Vector similarity answers “what looks relevant”. Metadata answers “what is allowed, current and in scope”. Production retrieval needs both, applied in that order: filter first, then rank.
| Metadata field | Purpose | Source | Enforced how |
|---|---|---|---|
| Source system & document id | Lineage and citation accuracy | Connector | Written at ingestion, immutable |
| Version & status | Prefer current over superseded | Source system or audit | Authority weight and optional hard filter |
| Effective / publication date | Freshness ranking and time-scoped questions | Document or source | Range filter at query time |
| Department / business unit | Scope questions to the right organisation | Owner mapping | Facet filter |
| Document type | Distinguish policy from guidance from discussion | Classification rules | Facet filter and boost |
| Jurisdiction / region / brand | Prevent cross-market contamination | Metadata or extraction | Hard filter where legally required |
| Sensitivity classification | Control what may reach an external model | Security policy | Hard filter plus model routing |
| Access control list | Row-level security at retrieval | Source system permissions | Query-time intersection with the requester’s rights |
| Authority rank | Tie-breaking between competing sources | Source register | Reranking signal |
The access control list is the field that most often forces a design change late in a project, because it requires reading permissions from the source system and keeping them in step. Build it early. A knowledge assistant that can answer for any employee about any document is not deployable in a regulated environment, and discovering that at security review costs a quarter.

Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.
Evaluation Harness: Proving the Corpus Works
Without an evaluation set, every change to chunking, embeddings, prompts or the corpus is a guess. The harness is a versioned collection of questions drawn from real user queries, each with the expected answer, the expected source documents and a labelled judgement of whether the system should refuse.
- Source the questions from reality. Support tickets, search logs, helpdesk queries and interviews with the people who will use the system. Synthetic questions test the wrong distribution.
- Size it for statistical meaning. Two hundred to five hundred questions, stratified by document type, difficulty and sensitivity, gives usable signal on a change. Twenty questions gives anecdote.
- Include unanswerable cases. A meaningful share of the set should have no answer in the corpus. Correctly refusing is a quality metric, and systems tuned only on answerable questions learn to hallucinate.
- Measure groundedness, not fluency. Does the answer cite the expected source? Is every claim supported by the retrieved text? Is the citation resolvable to a document the user can open?
- Automate and version it. Run on every corpus refresh, model change or prompt edit, store the result against the change, and alert on regression beyond a threshold.
- Keep a human calibration sample. Automated judgers drift. Re-check a sample against human review each cycle and record the agreement rate.
| Metric | Definition | Target for production | If it is below target |
|---|---|---|---|
| Retrieval precision @5 | Share of top-5 results that are relevant | 80–90% | Revisit chunking and metadata filters before touching the model |
| Answer groundedness | Share of claims supported by retrieved text | 90%+ | Tighten the prompt, add citation requirement, improve extraction |
| Correct refusal rate | Share of unanswerable questions correctly declined | 80%+ | Add unanswerable cases to fine-tuning or prompt constraints |
| Citation resolvability | Share of citations that open the right document | 95%+ | Fix lineage capture and document id handling |
| Permission correctness | Share of retrievals the requester was entitled to see | 100% | Stop deployment — this is a blocking defect |
| Latency p95 | Time from query to answer | Agreed with users, often 3–8s | Cache, reduce retrieved chunks, or reconsider reranking depth |
Permission-Aware Retrieval and Security Review
Enforcement must happen at query time against the source system’s permissions, not at ingestion time against a snapshot. Snapshot-based filtering drifts: a person changes role, a document changes classification, and the index still reflects last Tuesday.
Three viable patterns, in increasing order of rigour:
| Pattern | How it works | Trade-off |
|---|---|---|
| Query-time ACL check | Retrieve candidates, then filter against the requester’s live permissions from the source system | Accurate, adds latency and a dependency on the source system’s availability |
| Synced ACL metadata | Permissions replicated into the index and filtered at query time | Fast, but only as current as the sync — needs a tight refresh cadence and drift alerts |
| Segmented indices | Separate indices per sensitivity tier or business unit, with routing by role | Simplest to audit, harder to maintain and can fragment retrieval quality |
Whichever pattern is chosen, it needs a test that proves it: an evaluation run executed as several different user identities, asserting that no restricted document is retrievable by a user without rights. That test belongs in the CI pipeline, not in a manual security review. This is the same control discipline described in AI Governance, Risk & Compliance Services.
Retrieval Architecture: Hybrid Search, Reranking and Refresh
Pure vector search underperforms on exact identifiers — part numbers, policy codes, names, error strings. Production systems combine dense and sparse retrieval.
- Hybrid retrieval. BM25 or keyword search alongside vector search, merged with reciprocal rank fusion. This alone fixes a large share of “it cannot find the document I know exists” complaints.
- Reranking. A cross-encoder or reranking model applied to the top 30–50 candidates before the final 3–8 are passed to the model. Meaningful precision gain, at added latency and cost — measure both.
- Query transformation. Rewriting an ambiguous user question into a retrieval-friendly form, decomposing multi-part questions, and expanding acronyms. Log the transformation so you can debug it.
- Structured retrieval alongside text. Some questions are best answered from a database, not a document. Route those to a text-to-SQL or API path rather than forcing them through the vector index.
- Caching. Embedding cache for unchanged chunks and answer cache for repeated questions, both with explicit invalidation on corpus change.
- Observability. Log the query, the transformation, the retrieved ids and scores, the prompt, the answer and any user feedback. Without this, a bad answer cannot be diagnosed.
Refresh cadence should match the volatility of each source, not a single global schedule. Policies may refresh nightly; a product catalogue hourly; archived material never. Where the index sits in cloud infrastructure, sizing and cost control follow the same discipline as any data platform — see Cloud Services Migration & Engineering and DevOps & CI/CD Engineering Services for the deployment and pipeline side.
Operating the Knowledge System After Launch
A RAG system degrades if it is not operated. Four recurring jobs keep it honest.
| Job | Cadence | What happens if it stops |
|---|---|---|
| Corpus refresh and deletion sync | Per source volatility, typically daily | Answers cite withdrawn or outdated material |
| Evaluation run and regression alert | On every change plus a weekly full run | Quality drifts unnoticed until a user complains |
| Retrieval log review | Weekly, sampling low-confidence and refused queries | Systematic gaps in the corpus go undiscovered |
| Permission drift check | Weekly, executed as multiple identities | Access control silently rots — the worst failure mode |
| Embedding and model currency review | Quarterly | Cost and quality degrade as better options appear |
| Owner review of source register | Quarterly | New sources never get added, coverage shrinks over time |
Assign a named corpus owner per source. Without an owner, the source register stops reflecting reality within two quarters and the system quietly narrows to whatever was indexed at launch.
Common Architecture Mistakes
- Choosing the vector database before the metadata model. The store is the easy part; the schema is the hard part and it constrains everything downstream.
- Indexing everything. More corpus is not more quality. Unfiltered inclusion imports noise, duplicates and superseded content, and makes retrieval worse while costing more.
- No evaluation set until after launch. By then the corpus is built and every change is unfalsifiable.
- Snapshot permissions. Access control copied at ingestion cannot stay correct.
- Treating chunk size as a tuning knob. It is a design decision per document type, driven by how the content is structured and how it is asked about.
- Ignoring structured data. Forcing spreadsheet and database questions through a document index produces confident nonsense about numbers.
- No lineage. If a chunk cannot be traced to source, version and ingest time, you cannot answer the first question an auditor asks.
- One-off load mentality. RAG is a continuously refreshed data product with an owner, not a migration project with an end date.
Engagement Models
| Model | Duration | Deliverables | Best when |
|---|---|---|---|
| Knowledge readiness audit | 2–3 weeks | Source register, extractability assessment, sensitivity classification, permission feasibility, risk findings and a costed plan | Before committing to a build, or when an existing RAG system underperforms |
| Full data engineering build | 10–14 weeks | Connectors, cleaning pipeline, metadata model, chunking implementation, permission-aware retrieval, evaluation harness, operational runbook | A production knowledge assistant for real users |
| RAG remediation | 4–6 weeks | Diagnosis against an evaluation set, targeted fixes to extraction, chunking, metadata or retrieval, before-and-after measurement | An existing system with quality complaints and no measurement |
| Evaluation harness only | 2–4 weeks | Question set from real queries, automated judging, CI integration, baseline report | You have a build team but no way to measure whether changes help |
How This Connects to Your Wider AI Programme
The knowledge layer is shared infrastructure. The same cleaned, permission-aware, evaluated corpus serves an internal assistant, customer-facing support, agent tooling and AI search visibility. Building it once, properly, is cheaper than building four partial versions.
Where the corpus is public-facing content, the same metadata discipline improves how AI search systems read and cite you — covered in what AI search optimisation is and why it matters and AI visibility versus traditional SEO. Where the data engineering has to feed analytics as well as retrieval, it belongs with Data Science Services for Business Analytics. And where the knowledge system has to be embedded into operational systems rather than sitting beside them, that is the scope of AI Readiness Assessment & Roadmap and our integration service.
The practical sequence is: audit the sources, fix the corpus, instrument the evaluation, enforce permissions, then scale retrieval. Teams that invert this order spend more and get less, and the difference is visible in Figure 1 rather than in the choice of model.
Frequently Asked Questions About Data Engineering for AI, RAG and Knowledge Systems
Do we need to move all our documents into one place first?
No, and attempting it usually delays the project by quarters. Federated ingestion from the existing systems is normally the right approach, with a single index and a metadata model that records where each chunk came from. Consolidation is worth considering only where a source system cannot be read programmatically or cannot supply permissions.
How large an evaluation set do we really need?
Two hundred to five hundred questions drawn from real user queries, stratified by document type and difficulty, plus a deliberate subset of unanswerable questions. Fewer than a hundred and the variance between runs swamps the signal; more than a thousand and maintaining the expected answers becomes a burden that causes the set to rot.
Can we use an external model with confidential documents?
Sometimes, with controls. It depends on the classification, the contractual terms with the provider, whether content is retained or used for training, and whether a private or single-tenant deployment is available. Sensitive tiers are commonly routed to a private deployment while public and internal tiers use a hosted model. This decision belongs in the security review, not in the architecture document alone.
What chunk size should we use?
There is no single answer, which is why it should be decided by evaluation per document type. Policy and contract text typically works between 250 and 600 tokens with section paths attached; technical procedures need larger chunks so steps stay together; tickets and meeting notes work smaller with heavier metadata. Test two or three strategies against your question set and keep the winner.
How do you handle documents that are mostly tables or scanned images?
Structure-preserving extraction for tables — keep rows and columns intact and carry the header context into each row group, or store the table separately and retrieve it as a unit. For scans, an OCR pass with confidence scoring, and a human sample check on low-confidence pages. Flattening a table into prose is one of the most common causes of confidently wrong numeric answers.
How often does the index need refreshing?
Per source, matched to volatility. Policies and procedures usually daily; product or pricing catalogues hourly or on change events; archived material never. Deletions must propagate immediately regardless of the refresh cadence, because a withdrawn document that remains retrievable is a compliance problem, not a freshness problem.
Will RAG stop the model hallucinating?
It reduces hallucination substantially but does not eliminate it. Grounded answers with citations, an explicit instruction to decline when the corpus does not cover the question, and a measured refusal rate are what make it acceptable in production. Systems tuned only on answerable questions learn to invent plausible answers to unanswerable ones.
Do we need a vector database, or will a managed service do?
A managed service is usually the right starting point: less to operate, adequate performance, and the metadata filtering you need. Move to a dedicated vector platform when scale, cost per query, latency or filtering complexity demands it. Decide from measured load, not from anticipated load.
How do we keep permissions in sync when staff change role?
Enforce at query time against the source system, or replicate ACLs with a tight sync cadence and drift alerting. Then prove it: run the evaluation suite as several distinct identities on a schedule and assert that no restricted item is retrievable by an unentitled user. That test in CI is what makes the control real.
What does this cost relative to the rest of the AI build?
Roughly seven tenths of a RAG project’s effort sits in ingestion, cleaning, metadata, permissions and evaluation, per Figure 2. Teams that allocate most of the budget to model and prompt work typically discover the imbalance late and either extend the timeline or ship a system that answers fluently from the wrong documents.
Can the same corpus serve internal assistants and AI search visibility?
Yes for the underlying metadata and freshness discipline, but public-facing content needs its own extraction and structuring path because AI search systems read your published pages, not your internal index. The overlap is in governance and metadata quality — see how to measure and grow answer engine traffic for the visibility side.
Book a Knowledge System Assessment
Bring two or three of the document sources you want an AI assistant to answer from, plus ten real questions your staff currently struggle to answer. We will assess extractability, authority conflicts and permission feasibility against them, and show you where the corpus — not the model — is the limiting factor.
Contact Unified Management Consulting to book an initial working session on your AI programme.
Related Services
Data engineering for AI is the foundation layer under most other AI services:
- AI Agent Deployment & Automation — agents that act on retrieved knowledge need the same corpus discipline
- AI Product Engineering for Production-Ready Intelligent Products — productising the retrieval layer
- AI Readiness Assessment & Roadmap — deciding which knowledge domains are worth building first
- AI Governance, Risk & Compliance — classification, access control and audit evidence
- Data Science Services for Business Analytics — analytics on the same data foundation
- Cloud Services Migration & Engineering — hosting and cost control for the index and pipeline
- Choosing an AI Model: A Business Evaluation Framework — how to compare models against your own corpus
- AI ROI Measurement & Executive Scorecard Services — baseline capture, bottom-up benefit and cost models, instrumentation, the executive scorecard itself and the gate criteria that make AI spend auditable
- AI Integration Services for ERP, CRM & Operations — embed AI into ERP, CRM and operational systems with mapped permissions, supervised writes, immutable action logging and a tested rollback path
- AI Training & Enablement Services for Teams — role-based enablement, a measured capability baseline, build clinics and a champion network that makes adoption stick after delivery
- AI Forward-Deployed Engineering Services for Production-Ready Business Transformation — engineers embedded inside your team and your systems, building the capability with you rather than handing over a specification and leaving