Back

Data Engineering for AI: RAG & Knowledge System Services

Paired bar chart before and after data engineering: answers grounded in sources 46% to 94%, top-5 retrieval precision 52% to 89%, permission-filtered results 31% to 100%, unanswerable questions correctly flagged 28% to 86%
Figure 1. Retrieval quality before and after a data engineering pass on the same model, the same prompts and the same evaluation questions. The largest single movement is permission filtering — the dimension nobody budgets for and everybody discovers during security review.
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.

A retrieval-augmented system is only as good as the corpus behind it. Point the same model at uncleaned, duplicated, permission-blind documents and it will confidently cite the wrong version of a superseded policy. Data engineering is the part of RAG that decides whether the answer is merely fluent or actually correct.

Most RAG failures are not model failures. They are corpus failures: superseded documents still in the index, the same policy stored in four places with four versions, tables and PDF layouts destroyed by naive text extraction, no metadata to filter by department or date, and no way to enforce that a user only retrieves what they are allowed to read. Unified Management Consulting treats the knowledge layer as an engineering deliverable with connectors, a metadata model, permission enforcement, an evaluation set and a refresh schedule — not as a one-off upload into a vector database.

🗃️ Data Engineering for AI (RAG & Knowledge Systems) — Quick Summary

Component What We Deliver
Source audit Every candidate knowledge source inventoried with owner, format, volume, refresh cadence, sensitivity classification and whether it is actually authoritative.
Ingestion connectors Scheduled, authenticated connectors to SharePoint, document stores, wikis, ticketing, CRM and databases — with incremental sync and deletion handling.
Cleaning & deduplication Text extraction that preserves tables and structure, removal of superseded versions, near-duplicate detection, and normalisation of dates, units and entities.
Metadata model Department, document type, effective date, jurisdiction, sensitivity, source system and authority rank — attached to every chunk so retrieval can filter, not just search.
Chunking & embeddings Structure-aware chunking strategy chosen per document type, with the rationale recorded, and embedding model selected against your corpus rather than by default.
Permission-aware retrieval Row-level access control enforced at query time from the source system’s own permissions, so the AI can never surface what the user cannot read.
Evaluation harness A versioned question set with expected answers and source citations, run on every corpus or model change, reporting groundedness and refusal behaviour.
Operational refresh Incremental re-indexing, drift detection, orphan cleanup and a schedule that keeps the corpus current without a full rebuild.

This service underpins AI Agent Deployment & Automation and AI Product Engineering for production-ready intelligent products. Where the knowledge system also has to serve AI search visibility, it connects to AIEO Services and to how to get indexed by ChatGPT and AI search tools.

Why RAG Projects Stall on Data, Not Models

A RAG project spends most of its effort on plumbing rather than on intelligence. Figure 2 shows the realistic distribution: connectors and ingestion take roughly a third, chunking and metadata another quarter, and the model-and-prompt work that gets all the attention accounts for under a tenth. Teams that budget the other way round run out of money while the corpus is still wrong.

The reason is that enterprise knowledge is not a dataset. It is spread across systems with different access models, it changes continuously, much of it is superseded but still present, and a significant share is structured data trapped in PDF tables that naive extraction flattens into noise. Cleaning that up is data engineering work with the same disciplines as any other data platform — lineage, quality rules, scheduling, ownership and testing.

Symptom the business reports Actual root cause Where to fix it
“It gives answers that sound right but are wrong” Superseded documents still indexed; no authority or effective-date metadata Metadata model plus version pruning at ingestion
“It cannot find the document I know exists” Chunking splits the relevant passage, or the file format extracts badly Structure-aware chunking per document type
“It mixes up regions, brands or business units” No entity or scope metadata; retrieval matches on wording alone Faceted metadata and filtered retrieval
“Legal will not approve it” No permission enforcement; retrieval is corpus-wide Query-time access control from the source system
“It got worse after the last update” No evaluation harness, so changes are unmeasured Versioned evaluation set run on every change
“Answers stopped being current” One-off load with no incremental refresh or deletion handling Scheduled connectors with tombstone handling
Bar chart of RAG project effort share: source connectors and ingestion 31%, chunking metadata and embeddings 24%, permission-aware retrieval 16%, evaluation set and harness 13%, model prompt and reranking 9%, serving monitoring and operations 7%
Figure 2. Where effort actually goes in a RAG build. Retrieval infrastructure and data preparation account for roughly seven tenths of the work; the model, prompt and reranking layer that dominates most project narratives accounts for less than a tenth.
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.

Source Audit: Establishing What Is Actually Authoritative

Before any connector is written, the corpus has to be surveyed. This is the step teams skip, and it is the step that determines whether the system answers from the right document.

  • Inventory every candidate source with system, owner, format, volume, update frequency, sensitivity classification and retention rules.
  • Rank authority. Where two sources disagree, which one wins? A signed policy beats a team wiki; a current version beats an archived one. Encode this as an authority score in metadata rather than leaving it to retrieval luck.
  • Identify superseded content. Most organisations hold several generations of every document. Anything not explicitly excluded will eventually be cited.
  • Classify sensitivity. Public, internal, confidential, restricted. This drives both the permission model and what may be sent to an external model at all.
  • Assess extractability. Scanned PDFs, image-heavy decks, complex spreadsheets and legacy exports each need a different extraction path. Test extraction quality on samples before committing to a volume estimate.
  • Confirm access rights. For each source, determine whether permissions can be read programmatically. If they cannot, that source needs a separate access design or should be excluded.

The audit produces a source register that becomes the contract for the build. It also produces the first uncomfortable finding, which is usually that a large share of the knowledge employees rely on lives in personal drives, chat history and inboxes — none of which is a sustainable corpus. Addressing that is an information-management question as much as a technical one, and it is where Risk, Governance & Business Continuity Support and our AI governance policy framework come in.

Ingestion Connectors and Incremental Sync

A corpus loaded once is stale within weeks. Production ingestion has to be scheduled, incremental, authenticated and able to handle deletion — because a document withdrawn for legal reasons must stop being retrievable immediately, not at the next full rebuild.

Requirement What it means in practice Failure mode if skipped
Scheduled incremental sync Only changed items re-processed, on a defined cadence Full rebuilds become unaffordable, so refreshes stop happening
Deletion and tombstone handling Removed or withdrawn items purged from the index and any cache Withdrawn content keeps being cited — a compliance incident
Authentication per source Service accounts with least privilege, secrets in a vault Connector breaks silently when a person leaves or a token expires
Retry and dead-letter queue Failed items recorded and re-tried, not dropped Silent gaps in the corpus that no one notices until a user asks
Lineage capture Every chunk traceable to source, version, ingest time and checksum Impossible to answer “where did this answer come from?”
Volume and rate control Throttling against source API limits Connector gets blocked, or the source system degrades
Change detection accuracy Content hash plus modified timestamp, not timestamp alone Re-indexing everything on every run, or missing real changes

Cleaning, Deduplication and Version Control

Cleaning is where the visible quality gain comes from. Four operations matter more than the rest.

Structure-preserving extraction. A pricing table extracted as a run of numbers loses its meaning entirely. Extraction has to preserve headings, table rows and columns, list structure and document hierarchy, and carry them into the chunk as either formatted text or metadata. For scanned documents this means an OCR pass with confidence scoring and a human sample check.

Near-duplicate detection. The same policy appears as a Word original, a PDF export, a wiki copy and three email attachments. Embedding-based similarity plus metadata matching identifies clusters, and the authority rank picks the surviving copy. Leaving duplicates in place means retrieval slots are wasted on near-identical text and the citation looks arbitrary.

Version pruning. Superseded versions are either excluded or retained with an explicit status flag and a lower authority weight, so a historical question can still be answered without contaminating current-policy answers. This distinction has to be a deliberate decision per document type, not a default.

Normalisation. Dates to one format, units to one system, entity names to a controlled vocabulary, and acronyms expanded at first use in each chunk. Chunks are read in isolation, so a chunk that says “the SLA” without naming which SLA is far less useful than one that carries the full entity reference in its metadata.

Chunking Strategy: The Decision With the Largest Quality Effect

Chunking determines what retrieval can see. There is no universally correct size; there is a correct strategy per document type, and it should be chosen by evaluation rather than by convention.

Document type Recommended strategy Why Typical chunk target
Policy and procedure Section-aware split on headings, chunk carries section path Questions are about a clause, not the whole document 300–600 tokens
Contracts and legal Clause-level split with defined-term preservation Definitions are referenced across clauses and must travel with the text 250–500 tokens
Technical manuals Heading hierarchy with overlap, tables kept intact Procedures span steps that must not be separated 400–800 tokens
Meeting notes and tickets Per-item or per-thread, with entity and date metadata Highly repetitive wording; metadata does the disambiguation 150–400 tokens
Spreadsheets and reports Row-group or section summary plus retained raw table Numeric tables lose meaning when flattened to prose Per table or per section
Marketing and web content Topic-level split with URL and publication date Freshness matters as much as relevance 300–700 tokens

Two refinements consistently help. Contextual chunk headers — prefixing each chunk with document title, section path and effective date — lift retrieval precision noticeably because the embedding sees the scope, not just the wording. Parent-child retrieval — embedding small chunks but returning the larger surrounding section to the model — gives precise matching without starving the model of context. Both add cost, so evaluate them rather than assuming them. Our AI model evaluation framework covers how to run that comparison properly.

Metadata Model and Faceted Retrieval

Vector similarity answers “what looks relevant”. Metadata answers “what is allowed, current and in scope”. Production retrieval needs both, applied in that order: filter first, then rank.

Metadata field Purpose Source Enforced how
Source system & document id Lineage and citation accuracy Connector Written at ingestion, immutable
Version & status Prefer current over superseded Source system or audit Authority weight and optional hard filter
Effective / publication date Freshness ranking and time-scoped questions Document or source Range filter at query time
Department / business unit Scope questions to the right organisation Owner mapping Facet filter
Document type Distinguish policy from guidance from discussion Classification rules Facet filter and boost
Jurisdiction / region / brand Prevent cross-market contamination Metadata or extraction Hard filter where legally required
Sensitivity classification Control what may reach an external model Security policy Hard filter plus model routing
Access control list Row-level security at retrieval Source system permissions Query-time intersection with the requester’s rights
Authority rank Tie-breaking between competing sources Source register Reranking signal

The access control list is the field that most often forces a design change late in a project, because it requires reading permissions from the source system and keeping them in step. Build it early. A knowledge assistant that can answer for any employee about any document is not deployable in a regulated environment, and discovering that at security review costs a quarter.

Gantt chart of a 12-week RAG delivery: source audit and access mapping, ingestion connectors and scheduling, cleaning dedupe and metadata model, chunking and embedding strategy, permission-aware retrieval and reranking, evaluation harness and handover
Figure 3. A twelve-week delivery plan. Note that the evaluation harness starts before the retrieval build finishes — you need the question set in place to know whether any of the earlier work helped.
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.

Evaluation Harness: Proving the Corpus Works

Without an evaluation set, every change to chunking, embeddings, prompts or the corpus is a guess. The harness is a versioned collection of questions drawn from real user queries, each with the expected answer, the expected source documents and a labelled judgement of whether the system should refuse.

  • Source the questions from reality. Support tickets, search logs, helpdesk queries and interviews with the people who will use the system. Synthetic questions test the wrong distribution.
  • Size it for statistical meaning. Two hundred to five hundred questions, stratified by document type, difficulty and sensitivity, gives usable signal on a change. Twenty questions gives anecdote.
  • Include unanswerable cases. A meaningful share of the set should have no answer in the corpus. Correctly refusing is a quality metric, and systems tuned only on answerable questions learn to hallucinate.
  • Measure groundedness, not fluency. Does the answer cite the expected source? Is every claim supported by the retrieved text? Is the citation resolvable to a document the user can open?
  • Automate and version it. Run on every corpus refresh, model change or prompt edit, store the result against the change, and alert on regression beyond a threshold.
  • Keep a human calibration sample. Automated judgers drift. Re-check a sample against human review each cycle and record the agreement rate.
Metric Definition Target for production If it is below target
Retrieval precision @5 Share of top-5 results that are relevant 80–90% Revisit chunking and metadata filters before touching the model
Answer groundedness Share of claims supported by retrieved text 90%+ Tighten the prompt, add citation requirement, improve extraction
Correct refusal rate Share of unanswerable questions correctly declined 80%+ Add unanswerable cases to fine-tuning or prompt constraints
Citation resolvability Share of citations that open the right document 95%+ Fix lineage capture and document id handling
Permission correctness Share of retrievals the requester was entitled to see 100% Stop deployment — this is a blocking defect
Latency p95 Time from query to answer Agreed with users, often 3–8s Cache, reduce retrieved chunks, or reconsider reranking depth

Permission-Aware Retrieval and Security Review

Enforcement must happen at query time against the source system’s permissions, not at ingestion time against a snapshot. Snapshot-based filtering drifts: a person changes role, a document changes classification, and the index still reflects last Tuesday.

Three viable patterns, in increasing order of rigour:

Pattern How it works Trade-off
Query-time ACL check Retrieve candidates, then filter against the requester’s live permissions from the source system Accurate, adds latency and a dependency on the source system’s availability
Synced ACL metadata Permissions replicated into the index and filtered at query time Fast, but only as current as the sync — needs a tight refresh cadence and drift alerts
Segmented indices Separate indices per sensitivity tier or business unit, with routing by role Simplest to audit, harder to maintain and can fragment retrieval quality

Whichever pattern is chosen, it needs a test that proves it: an evaluation run executed as several different user identities, asserting that no restricted document is retrievable by a user without rights. That test belongs in the CI pipeline, not in a manual security review. This is the same control discipline described in AI Governance, Risk & Compliance Services.

Retrieval Architecture: Hybrid Search, Reranking and Refresh

Pure vector search underperforms on exact identifiers — part numbers, policy codes, names, error strings. Production systems combine dense and sparse retrieval.

  • Hybrid retrieval. BM25 or keyword search alongside vector search, merged with reciprocal rank fusion. This alone fixes a large share of “it cannot find the document I know exists” complaints.
  • Reranking. A cross-encoder or reranking model applied to the top 30–50 candidates before the final 3–8 are passed to the model. Meaningful precision gain, at added latency and cost — measure both.
  • Query transformation. Rewriting an ambiguous user question into a retrieval-friendly form, decomposing multi-part questions, and expanding acronyms. Log the transformation so you can debug it.
  • Structured retrieval alongside text. Some questions are best answered from a database, not a document. Route those to a text-to-SQL or API path rather than forcing them through the vector index.
  • Caching. Embedding cache for unchanged chunks and answer cache for repeated questions, both with explicit invalidation on corpus change.
  • Observability. Log the query, the transformation, the retrieved ids and scores, the prompt, the answer and any user feedback. Without this, a bad answer cannot be diagnosed.

Refresh cadence should match the volatility of each source, not a single global schedule. Policies may refresh nightly; a product catalogue hourly; archived material never. Where the index sits in cloud infrastructure, sizing and cost control follow the same discipline as any data platform — see Cloud Services Migration & Engineering and DevOps & CI/CD Engineering Services for the deployment and pipeline side.

Operating the Knowledge System After Launch

A RAG system degrades if it is not operated. Four recurring jobs keep it honest.

Job Cadence What happens if it stops
Corpus refresh and deletion sync Per source volatility, typically daily Answers cite withdrawn or outdated material
Evaluation run and regression alert On every change plus a weekly full run Quality drifts unnoticed until a user complains
Retrieval log review Weekly, sampling low-confidence and refused queries Systematic gaps in the corpus go undiscovered
Permission drift check Weekly, executed as multiple identities Access control silently rots — the worst failure mode
Embedding and model currency review Quarterly Cost and quality degrade as better options appear
Owner review of source register Quarterly New sources never get added, coverage shrinks over time

Assign a named corpus owner per source. Without an owner, the source register stops reflecting reality within two quarters and the system quietly narrows to whatever was indexed at launch.

Common Architecture Mistakes

  • Choosing the vector database before the metadata model. The store is the easy part; the schema is the hard part and it constrains everything downstream.
  • Indexing everything. More corpus is not more quality. Unfiltered inclusion imports noise, duplicates and superseded content, and makes retrieval worse while costing more.
  • No evaluation set until after launch. By then the corpus is built and every change is unfalsifiable.
  • Snapshot permissions. Access control copied at ingestion cannot stay correct.
  • Treating chunk size as a tuning knob. It is a design decision per document type, driven by how the content is structured and how it is asked about.
  • Ignoring structured data. Forcing spreadsheet and database questions through a document index produces confident nonsense about numbers.
  • No lineage. If a chunk cannot be traced to source, version and ingest time, you cannot answer the first question an auditor asks.
  • One-off load mentality. RAG is a continuously refreshed data product with an owner, not a migration project with an end date.

Engagement Models

Model Duration Deliverables Best when
Knowledge readiness audit 2–3 weeks Source register, extractability assessment, sensitivity classification, permission feasibility, risk findings and a costed plan Before committing to a build, or when an existing RAG system underperforms
Full data engineering build 10–14 weeks Connectors, cleaning pipeline, metadata model, chunking implementation, permission-aware retrieval, evaluation harness, operational runbook A production knowledge assistant for real users
RAG remediation 4–6 weeks Diagnosis against an evaluation set, targeted fixes to extraction, chunking, metadata or retrieval, before-and-after measurement An existing system with quality complaints and no measurement
Evaluation harness only 2–4 weeks Question set from real queries, automated judging, CI integration, baseline report You have a build team but no way to measure whether changes help

How This Connects to Your Wider AI Programme

The knowledge layer is shared infrastructure. The same cleaned, permission-aware, evaluated corpus serves an internal assistant, customer-facing support, agent tooling and AI search visibility. Building it once, properly, is cheaper than building four partial versions.

Where the corpus is public-facing content, the same metadata discipline improves how AI search systems read and cite you — covered in what AI search optimisation is and why it matters and AI visibility versus traditional SEO. Where the data engineering has to feed analytics as well as retrieval, it belongs with Data Science Services for Business Analytics. And where the knowledge system has to be embedded into operational systems rather than sitting beside them, that is the scope of AI Readiness Assessment & Roadmap and our integration service.

The practical sequence is: audit the sources, fix the corpus, instrument the evaluation, enforce permissions, then scale retrieval. Teams that invert this order spend more and get less, and the difference is visible in Figure 1 rather than in the choice of model.

Frequently Asked Questions About Data Engineering for AI, RAG and Knowledge Systems

Do we need to move all our documents into one place first?

No, and attempting it usually delays the project by quarters. Federated ingestion from the existing systems is normally the right approach, with a single index and a metadata model that records where each chunk came from. Consolidation is worth considering only where a source system cannot be read programmatically or cannot supply permissions.

How large an evaluation set do we really need?

Two hundred to five hundred questions drawn from real user queries, stratified by document type and difficulty, plus a deliberate subset of unanswerable questions. Fewer than a hundred and the variance between runs swamps the signal; more than a thousand and maintaining the expected answers becomes a burden that causes the set to rot.

Can we use an external model with confidential documents?

Sometimes, with controls. It depends on the classification, the contractual terms with the provider, whether content is retained or used for training, and whether a private or single-tenant deployment is available. Sensitive tiers are commonly routed to a private deployment while public and internal tiers use a hosted model. This decision belongs in the security review, not in the architecture document alone.

What chunk size should we use?

There is no single answer, which is why it should be decided by evaluation per document type. Policy and contract text typically works between 250 and 600 tokens with section paths attached; technical procedures need larger chunks so steps stay together; tickets and meeting notes work smaller with heavier metadata. Test two or three strategies against your question set and keep the winner.

How do you handle documents that are mostly tables or scanned images?

Structure-preserving extraction for tables — keep rows and columns intact and carry the header context into each row group, or store the table separately and retrieve it as a unit. For scans, an OCR pass with confidence scoring, and a human sample check on low-confidence pages. Flattening a table into prose is one of the most common causes of confidently wrong numeric answers.

How often does the index need refreshing?

Per source, matched to volatility. Policies and procedures usually daily; product or pricing catalogues hourly or on change events; archived material never. Deletions must propagate immediately regardless of the refresh cadence, because a withdrawn document that remains retrievable is a compliance problem, not a freshness problem.

Will RAG stop the model hallucinating?

It reduces hallucination substantially but does not eliminate it. Grounded answers with citations, an explicit instruction to decline when the corpus does not cover the question, and a measured refusal rate are what make it acceptable in production. Systems tuned only on answerable questions learn to invent plausible answers to unanswerable ones.

Do we need a vector database, or will a managed service do?

A managed service is usually the right starting point: less to operate, adequate performance, and the metadata filtering you need. Move to a dedicated vector platform when scale, cost per query, latency or filtering complexity demands it. Decide from measured load, not from anticipated load.

How do we keep permissions in sync when staff change role?

Enforce at query time against the source system, or replicate ACLs with a tight sync cadence and drift alerting. Then prove it: run the evaluation suite as several distinct identities on a schedule and assert that no restricted item is retrievable by an unentitled user. That test in CI is what makes the control real.

What does this cost relative to the rest of the AI build?

Roughly seven tenths of a RAG project’s effort sits in ingestion, cleaning, metadata, permissions and evaluation, per Figure 2. Teams that allocate most of the budget to model and prompt work typically discover the imbalance late and either extend the timeline or ship a system that answers fluently from the wrong documents.

Can the same corpus serve internal assistants and AI search visibility?

Yes for the underlying metadata and freshness discipline, but public-facing content needs its own extraction and structuring path because AI search systems read your published pages, not your internal index. The overlap is in governance and metadata quality — see how to measure and grow answer engine traffic for the visibility side.

Book a Knowledge System Assessment

Bring two or three of the document sources you want an AI assistant to answer from, plus ten real questions your staff currently struggle to answer. We will assess extractability, authority conflicts and permission feasibility against them, and show you where the corpus — not the model — is the limiting factor.

Contact Unified Management Consulting to book an initial working session on your AI programme.

Related Services

Data engineering for AI is the foundation layer under most other AI services:

Leave a Reply

Your email address will not be published. Required fields are marked *