Back

Site Reliability Engineering (SRE) & Observability Services

Line chart across six months of SRE practice on a percentage axis from 80 to 100: service availability rises from 99.20 to 99.95 per cent and SLO attainment across services rises from 88 to 99 per cent, with the steepest gain in months two and three when alert routing and error budgets were introduced
Figure 1. Service availability and service-level objective attainment across six months of site reliability practice, on a percentage axis from 80 to 100. Availability rises from 99.20 to 99.95 per cent and SLO attainment across services rises from 88 to 99 per cent. The steepest gains fall in months two and three, which is when alert routing and error budgets were introduced — before any additional infrastructure was bought.
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.

An outage is rarely a surprise to the system. It is a surprise to the people running it, because nobody had written down what “working” meant, nobody was measuring it, and the alert that would have given them forty minutes of warning was one of three hundred that week.

Site reliability engineering and observability at Unified Management Consulting is the practice of defining what a service should do, measuring whether it does, and organising people and releases around that measurement. It produces three things that are easy to name and hard to fake: service-level objectives your business agrees with, telemetry that answers a question in minutes rather than in a day, and an on-call model where the person paged can actually do something about it.

We are deliberately unromantic about SRE. It is not a job title you hire to make outages disappear, and it is not a monitoring product. It is an operating discipline: an inventory of services with named owners, objectives expressed as error budgets, instrumentation that supports those objectives, alerts that page only on user-visible impact, incident command that works under pressure, and a review process that changes the system afterwards. Where a team is small, the discipline is the same and the tooling is simpler — we would rather leave you with five dashboards somebody opens than fifty nobody does.

📈 SRE & Observability — Quick Summary

Component What We Deliver
Service inventory and ownership Every user-facing and supporting service listed with a named owning team, a criticality tier, its dependencies and its business consequence of failure.
Service-level objectives One availability and one latency objective per critical service, agreed with the business, with a measurement window and a written rationale for the number chosen.
Error budgets and release policy The tolerated unreliability expressed as a budget, with a policy for what happens when it burns — including the freeze that makes the number meaningful.
Instrumentation Metrics, structured logs and distributed traces implemented to a single contract across services, with health, readiness and business events instrumented deliberately.
Observability platform Collection, storage, retention tiers and query tooling chosen for the questions you actually ask, and priced against the data volume those questions require.
Dashboards One per critical journey, one per service and one for the platform, each with an owner and a stated purpose, retired when nobody opens them.
Alerting and routing Symptom-based paging for user impact, cause-based ticketing for everything else, routing to the owning team, and a monthly review of every page.
On-call and escalation A rota that is sustainable, escalation paths that are tested, runbooks per failure mode, and compensation and handover arrangements agreed in advance.
Incident command Roles, decision rights, communication templates, severity definitions and a rehearsed process so that the first real incident is not the first rehearsal.
Capacity and load testing Baseline load tests per critical journey, capacity headroom measured against real peaks, and scaling behaviour proven before it is needed.
Post-incident review A blameless review within five working days producing changes with owners and dates, tracked to closure rather than recorded and forgotten.
Reliability reporting A monthly report of SLO attainment, error budget burn, incident counts, restoration times and alert quality — readable by an operations head and a finance lead together.

What Site Reliability Engineering Is — and What It Is Not

The term is used loosely enough that agreeing a definition is the first deliverable of most engagements. Below is the working definition we use, with the misconceptions we spend time removing.

Element of the practice What it means in delivery What it is often mistaken for
Service-level objectives A number the business agrees to, derived from what users need and what the architecture can deliver, measured on a defined window A marketing figure quoted in a contract with no measurement behind it
Error budgets The permitted unreliability inside the objective, spent deliberately on change and withheld when exhausted A target to be maximised, which produces a team that stops releasing
Toil reduction Manual, repetitive, automatable work identified and removed, with the reduction measured Any operational work, including work that genuinely requires judgement
Blameless review A structured examination of how the system allowed the failure, producing changes with owners A post-mortem that names a person, after which nobody reports near misses again
Shared ownership with development The team that builds the service carries a share of operating it, so reliability is a design input A separate reliability team that inherits services and absorbs the consequences of their design
Observability The ability to answer a new question about the system without deploying new code Monitoring, which answers the questions you thought to ask in advance
Progressive delivery Changes released to a subset first, observed against the objective, then rolled forward or back automatically A deployment window, which is a schedule rather than a control
Capacity management Headroom measured against real peaks with a forecast and a scaling plan Adding capacity reactively after an incident, which is expensive and often too late

If your organisation wants reliability without any of these — no objectives, no reviews, no shared ownership — then what it wants is better monitoring and faster incident response. That is a legitimate and much cheaper goal, and we will scope it honestly rather than sell you a practice you will not operate.

Service Inventory and Ownership: The Prerequisite

Nothing in this discipline works before the inventory exists. Estates cannot state how many services they run, which are user-facing, who owns each, or what happens to revenue or patient care when one fails. Producing that list is usually the first two weeks of an engagement and is the artefact every later decision refers back to.

Inventory field Why it is needed How it is populated
Service name and description A shared vocabulary; most reliability arguments are actually naming arguments From the owning team, not from the deployment manifest
Owning team and named individual Alert routing, escalation and accountability Agreed with the technology head; services without an owner are candidates for retirement
Criticality tier Determines the objective, the on-call treatment and the investment in redundancy Assessed against business consequence, not against how much somebody likes the service
User-facing journeys served Objectives are set on journeys users experience, not on internal components Mapped with the business: order to payment, enquiry to response, login to record
Dependencies Upstream and downstream services, databases, queues, third parties, and the failure behaviour of each From traces where they exist, from code and configuration review where they do not
Deployment and runtime Where it runs, how it is deployed, and how it is rolled back From the pipeline and the platform; gaps here are recorded as risks
Existing telemetry What is already measured, retained for how long, and whether it answers a question Audited before new instrumentation is specified, to avoid duplicating what exists
Historical incidents What has actually gone wrong, how often, and how long restoration took From ticket history and memory, with the memory part gathered in interviews
Cost and volume Request volume, storage and compute cost, which drive both capacity and telemetry budget From billing allocation and platform metrics
Retirement candidate Services nobody can explain are usually services nobody uses Flagged for the estate review; retiring a service is the most reliable improvement available

The inventory is also the input to consolidation decisions. Where the same capability is served by three implementations, reliability work triples for no benefit, and the argument in tech stack consolidation applies directly: fewer services means fewer objectives, fewer dashboards and fewer people on call.

Service-Level Objectives, Indicators and Error Budgets

An objective is a promise with a measurement attached. Choosing the number is a business decision informed by engineering reality; choosing the indicator is an engineering decision informed by what users experience. Getting either wrong produces a metric that everybody quotes and nobody trusts.

Service class Availability objective Latency objective Rationale and window
Customer-facing transactional 99.9 per cent monthly 95 per cent of requests under 400 milliseconds, 99 per cent under 1 second Direct revenue or service delivery impact; measured monthly so a budget is meaningful within the period
Customer-facing informational 99.5 per cent monthly 95 per cent of requests under 1 second Degradation is visible but rarely stops a transaction; a lower objective frees budget for change
Internal business application 99.5 per cent within business hours, 99 per cent overall 95 per cent of requests under 1 second Out-of-hours failure costs almost nothing; an objective that ignores this wastes money on redundancy nobody needs
Batch and reporting 99 per cent of scheduled runs completing inside the window Completion inside the agreed window, not per-request latency Timeliness is the service the business consumes, so the objective measures timeliness
Integration and API 99.9 per cent monthly 95 per cent of calls under 500 milliseconds, with an agreed rate limit A partner’s objective becomes yours the moment you sign the contract
Data and analytics platform 99 per cent monthly, with a stated recovery point Query completion inside an agreed percentile Durability and recoverability matter more than uptime for analytical workloads
  • Indicators are chosen from the client’s side of the interaction wherever possible: successful responses served, not process uptime, which can be perfect while the service is broken.
  • Every objective has a measurement window and a data source named in the same sentence, because an objective without them is an aspiration.
  • Error budgets are the difference between 100 per cent and the objective, expressed in minutes and in permitted failed requests.
  • Burn-rate alerts fire on the rate of consumption, not on the level, so a fast burn pages and a slow burn raises a ticket.
  • When a budget is exhausted the agreed consequence applies — typically a freeze on non-critical releases until reliability work restores it. A budget with no consequence is a report.

Instrumentation: Metrics, Logs and Traces

Three signals, one contract. The contract is what makes an estate observable rather than merely instrumented: the same labels, the same request identifier, the same log format and the same health semantics in every service, so that a question can be asked once and answered across the whole estate.

Signal What we instrument Contract applied across services Common gap we find
Metrics Request rate, error rate and latency histograms per endpoint; saturation of CPU, memory, connections and queues; business events such as orders or submissions Consistent label names, cardinality limits agreed in advance, and a rule against labels whose values are unbounded Averages instead of histograms, which hides the tail that users experience
Logs Structured application logs with severity, request identifier, user or tenant reference where lawful, and error context; access logs; platform and audit logs One JSON schema, one timestamp format, one correlation identifier propagated across service boundaries Logs written as prose, so answering a question means reading them
Traces Spans across HTTP calls, queue publishes and consumes, database queries and third-party calls, with adaptive sampling Trace context propagated in a standard header, error spans always captured regardless of sampling Tracing in the application tier only, so the slow database call is invisible
Health and readiness Liveness and readiness probes that test real dependencies rather than process existence A documented definition of ready, and a startup grace period that reflects reality A readiness probe that returns healthy while the service cannot reach its database
Synthetic checks Critical journeys exercised from outside on a schedule, with results feeding the objective One check per critical journey, run from more than one location where users are distributed Objectives measured only from inside the system, so a DNS or CDN failure is invisible
Business signals Conversion, submission and completion events tied to the journey the objective describes Defined with the business, stored alongside technical metrics so a technical change can be compared with a business outcome Technical dashboards that stay green while revenue quietly falls

Instrumentation is delivered through the golden path wherever a platform exists, so a new service arrives already observable. Retrofitting forty services is a project; instrumenting one template is an afternoon, which is why we work with DevOps & CI/CD Engineering on the pipeline side of this.

Dashboards That Somebody Opens

A dashboard is a hypothesis about what someone needs to see. We build four kinds, each with a named owner, and we delete the rest. The quarterly review asks one question per dashboard: has anybody opened this since we last looked?

Dashboard Audience Content Test of usefulness
Journey dashboard Service owner and on-call engineer during an incident Objective attainment, error budget burn, request volume, error rate and latency percentiles for one user journey, with dependencies beneath During an incident it is opened first and answers ‘is it broken and how badly’ in under a minute
Service dashboard Owning team, weekly Deployment markers against the objective, saturation, dependency health, and the errors that did not become incidents The team can explain every movement in the last week without running a query
Platform dashboard Platform or infrastructure team Cluster and node health, capacity headroom, upgrade status, policy refusals, and cost against budget Capacity problems appear here before they appear on a journey dashboard
Executive reliability summary Technology head, monthly Objective attainment per critical service, incidents and restoration times, error budget position, and the trend of alert quality It fits on one screen and requires no explanation from an engineer
Change and release view Release manager Deployments, feature flags, configuration changes and infrastructure changes on one timeline against incident markers The first question in any review — ‘what changed?’ — is answered by looking at it
Cost and volume view Cost owner Telemetry volume and cost per service, retention in use, and the queries consuming the most An unexpected telemetry bill is explained before finance asks

We are equally willing to remove dashboards. Estates commonly hold several hundred panels built during incidents and never revisited, which slow the tool down and train people to ignore the ones that matter.

Alerting: Page Only When a Human Can Act

Alert design is where most reliability programmes succeed or fail, and it is the cheapest part to fix. The rule we apply is uncompromising: a page must mean that a human needs to do something now. Everything else is a ticket, a dashboard or nothing at all. Figure 2 shows what happens to the on-call experience when that rule is enforced.

Paired bar chart of five incident response counts before and after an SRE and observability programme: mean hours to restore fall from 9.0 to 1.8, alerts raised per incident from 46 to 9, pages per on-call shift from 11 to 3, incidents raised per month from 14 to 4, and escalations to a second team from 9 to 2
Figure 2. Incident response measures before and after an SRE and observability programme: mean hours to restore fall from 9.0 to 1.8, alerts raised per incident from 46 to 9, pages per on-call shift from 11 to 3, incidents raised per month from 14 to 4, and escalations to a second team from 9 to 2. Alerts per incident is the measure that indicates whether the alerts are describing the incident or merely accompanying it.
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.
Alert class Trigger Route Why
Error budget burn, fast A burn rate that would exhaust the remaining budget inside an hour Page the owning team immediately, escalate after fifteen minutes without acknowledgement User impact is happening now and will continue; this is the alert that justifies on-call
Error budget burn, slow A burn rate that would exhaust the budget inside the window Raise a ticket for the owning team, surfaced on the weekly review Real degradation that does not need somebody woken up
Journey failure A synthetic check failing from two or more locations Page the owning team, with the platform team copied Confirms impact from outside the system, which internal metrics can miss
Saturation forecast Disk, connection pool, queue depth or memory trending to exhaustion within a defined period Ticket, escalating to a page if the trend continues past the intervention point These are the incidents that can be prevented with an hour of warning
Deployment regression Objective breach within the observation window after a release Automatic rollback plus a page to the releasing team The fastest restoration available is usually undoing the change
Platform health Node loss, control plane degradation, certificate expiry, backup failure, upgrade failure Page the platform team; backup failures escalate to the technology head A failed backup is a silent loss of the recovery option, which is why it pages
Security signal Unusual authentication, policy override, exposure change or detection rule firing Route to the security responder, not the service team Mixing security and reliability routing produces alerts that two teams assume the other has handled
Noise, deliberately excluded CPU utilisation alone, single failed request, non-critical job retry, informational events Dashboard or log only Every one of these has, at some point, been the alert that hid the real incident

Every page is reviewed monthly. The review asks three questions: did it need a human, was it actionable within the response window, and did the runbook exist? Alerts that fail two of the three are rewritten or deleted, and the count of pages per shift is reported as a programme measure rather than treated as a badge of busyness.

On-Call, Escalation and Incident Command

On-call is a design problem with a human consequence. A rota that cannot be sustained will be abandoned, and an escalation path that has never been tested will fail at the moment it is used. We build both, then rehearse them.

Role Responsibility during an incident Preparation required
On-call engineer Acknowledges the page, assesses impact against the journey dashboard, executes the runbook, requests escalation when the runbook ends Runbook per failure mode, access to the systems needed at two in the morning, and a shadow shift before solo duty
Incident commander Owns the incident rather than the keyboard: sets severity, assigns work, holds the timeline, decides when it is over A written command process, a severity definition, and at least one rehearsal per quarter
Communications lead Updates the status page, informs the business and, where contractual, customers and regulators Templates written before the incident, and an agreed list of who is told at each severity
Scribe Records the timeline, decisions and evidence, which becomes the basis of the review A single place to write, and a rule that the scribe does not also debug
Escalation target Second-line specialists, the platform team, the vendor and, for a major incident, the technology head Contact details verified quarterly, with a named alternative per role
Vendor and provider liaison Opens the support case, provides the evidence the provider asks for, tracks their response Support entitlements documented in advance, including severity definitions and response times you have actually purchased
  • Severity is defined by user impact, not by internal embarrassment, and the definition is written down before it is needed.
  • Handover between shifts is a structured conversation with a written summary, because incidents that span a shift boundary lose their history otherwise.
  • Compensation, time off in lieu and the maximum number of consecutive nights are agreed with the team, not announced to it.
  • The rota is sized for sustainability: a team of four cannot cover a 24×7 rota without burnout, and pretending otherwise produces a rota on paper and an unanswered phone in practice.
  • Escalation is tested by exercise, including the out-of-hours path and the vendor path, with the result recorded.

Capacity, Load Testing and Release Engineering

Reliability is mostly decided before an incident: by how much headroom exists, by whether a release can be undone, and by whether scaling actually works when it is triggered. These three are tested on purpose rather than discovered by accident.

Activity What we run What it proves
Baseline load test The critical journey at expected peak traffic, with realistic data volumes and concurrent users Where the first constraint appears, and whether the latency objective survives peak
Soak test Sustained load over hours, watching memory, connection pools, queue depth and disk The slow leaks that a short test never shows, which are the ones that fail on a busy Monday
Spike test A sudden multiple of normal traffic, as a campaign or a news event would produce Whether autoscaling reacts in time and whether the queue absorbs the burst or sheds it
Failure injection Dependency removal: a database replica, a third-party API, a region or an availability zone, a node Whether the service degrades gracefully or fails completely, and whether the alert fires
Rollback rehearsal A bad release deployed to a non-production environment and rolled back automatically and manually That rollback works, how long it takes, and that the data migration does not prevent it
Scaling limits Node pool and instance limits exercised to the maximum Whether the configured maximum is enough, and what happens when it is reached
Recovery test Restore from backup into a clean environment, timed against the recovery objective That the recovery objective is achievable, which is the only evidence that counts
Progressive delivery trial A release to a small percentage of traffic with automatic health comparison That a regression is caught by the system before most users see it

Load testing is scheduled before peaks rather than after them: before a festival sale, an examination result publication, a marketing campaign or a regulatory deadline. Where the estate includes marketing-driven traffic, we plan capacity with the campaign calendar from Digital Marketing Services rather than reacting to the first slow weekend.

Post-Incident Review and the Learning Loop

A review that does not produce a change with an owner and a date is a meeting. The structure below is what we run, and the tracking is what makes it work.

Stage When Output
Immediate capture During and straight after the incident A timeline with timestamps, the evidence collected, the decisions taken and who took them
Blameless review Within five working days, with everyone involved and no managers adjudicating fault A narrative of how the system allowed the failure, the contributing factors, and what worked well — including the parts that did
Action items At the review, each with an owner and a date Prevention, detection and response actions separated, because improving detection is often cheaper than preventing
Tracking to closure Weekly until closed, reported in the monthly reliability report Aged action items visible to the technology head, which is what stops them becoming permanent
Near-miss capture Continuously, with a low-friction route to report one The incidents that did not happen, which are free lessons and the best indicator of a healthy reporting culture
Systemic review Quarterly, across all incidents Patterns that no single review would reveal: a dependency that keeps failing, a runbook that keeps being wrong, a team that keeps being paged for another team’s service
Objective revision After a systemic review or a major incident Objectives and budgets adjusted to reflect what the architecture can actually deliver, rather than defended for appearance

We publish an anonymised internal record of reviews. Teams that can read what went wrong elsewhere prevent the same failure; teams that cannot will rediscover it, usually at an inconvenient hour.

The Twelve-Week Programme

Gantt chart of a twelve-week SRE and observability programme: service inventory and SLO candidates in weeks one and two, instrumentation and trace coverage from week three to five, log pipeline and retention tiers from week four to six, dashboards per critical journey from week six to eight, alert routing and paging policy in weeks eight and nine, on-call rota and escalation paths in weeks nine and ten, error budget policy and release gates in weeks ten and eleven, and a game day with chaos rehearsal in weeks eleven and twelve
Figure 3. The twelve-week site reliability and observability programme. Instrumentation and the log pipeline overlap deliberately so that traces and logs carry the same correlation identifier from the start, and the game day lands in weeks eleven and twelve, after the rota and escalation paths exist but before handover.
Chart values are an illustrative planning model based on typical UMC engagements and published industry benchmarks. They are not a guarantee of results on any specific project.
Weeks Workstream Decision gate at the end
1–2 Service inventory and SLO candidates: ownership, criticality, journeys, dependencies, existing telemetry and incident history The inventory is agreed by the technology head and every service has a named owner
3–5 Instrumentation and trace coverage: metrics, structured logs, trace context, health semantics, applied through the golden path A reference service produces all three signals with one correlation identifier end to end
4–6 Log pipeline and retention tiers: collection, central store, retention by obligation and cost, access control A query for a request identifier returns logs and traces from every service it touched
6–8 Dashboards per critical journey: journey, service, platform and executive views, each with an owner Each journey dashboard answers ‘is it broken and how badly’ without an ad-hoc query
8–9 Alert routing and paging policy: burn-rate alerts, saturation forecasts, deployment regression, noise removal An injected fault pages the owning team inside the SLO window and nothing else pages
9–10 On-call rota and escalation paths: roles, severity definitions, runbooks, compensation, handover format A shift handover is performed and an escalation path is exercised end to end
10–11 Error budget policy and release gates: budgets agreed with the business, the consequence of exhaustion written down, gates wired into the pipeline A release is blocked by an exhausted budget in a rehearsal, and the business accepts the policy
11–12 Game day and chaos rehearsal: failure injection, rollback, recovery from backup, incident command practised The rehearsal produces a review with action items, and the recovery time is inside the objective

Where an estate has no platform and no pipeline, the first four weeks are spent with DevOps & CI/CD Engineering establishing deployment and environment discipline, because observability without a controlled release process produces excellent dashboards of a system nobody can change safely.

What We Measure and Report

Twelve measures, monthly, with definitions published alongside them. They are chosen so that a technology head can see whether reliability is improving and a finance lead can see what it is costing.

Measure Definition Direction of travel
SLO attainment per critical service Successful requests or completed journeys over total, inside the window At or above the objective, with any breach explained
Error budget remaining Unused portion of the budget at month end Positive for every critical service; a consistently exhausted budget means the objective or the architecture is wrong
Incidents by severity Count per month, split by severity tier Falling, with severity-one and severity-two incidents individually reviewed
Mean time to acknowledge Page to human acknowledgement Under five minutes, which is a rota and routing property rather than a heroism property
Mean time to restore Incident start to service restored, from the incident record Falling; in Figure 2, from nine hours to 1.8
Alerts per incident Alerts fired during an incident window Falling towards the number that describe the incident rather than accompany it
Pages per on-call shift Pages requiring human action, per shift Falling, with a sustained low count being the sign of a sustainable rota
Actionable alert share Pages that required a human action, over all pages Rising towards 90 per cent or better
Change failure rate Deployments requiring rollback or hotfix, over all deployments Falling, and reported next to deployment frequency so the two are not traded silently
Deployment frequency Production deployments per service per week Rising as confidence rises; a falling frequency with a falling failure rate usually means the team has stopped releasing
Detection source Share of incidents detected by monitoring versus reported by users or customers Rising towards monitoring-detected, which is the measure of whether observability works
Telemetry cost per service Observability spend allocated per service, against request volume Stable or falling as retention and cardinality are tuned

Detection source deserves emphasis. An estate where most incidents are reported by customers is an estate where telemetry describes the infrastructure rather than the service, and no amount of additional tooling fixes that until the objectives are set on journeys.

Tooling Decisions

Tools follow the questions. We are provider-neutral and will work inside what you have, because a migration of telemetry is usually the most expensive and least valuable option available. Where a change is warranted, these are the decisions and how we make them.

Category Options we commonly assess How we choose
Metrics and alerting Provider-native monitoring, Prometheus with a managed service, an all-in-one observability vendor By the number of services, the retention you need and whether the team will operate the storage layer; native tooling first for small estates
Logging Provider log services, a self-managed stack, a commercial log platform with tiered storage By volume and obligation: retention requirements usually decide the design more than query features
Tracing Provider tracing, an OpenTelemetry collector with a compatible back end, an application performance monitoring suite By whether the services can be instrumented with a standard API; OpenTelemetry first, because it keeps the back end a replaceable decision
Incident management A dedicated incident tool, the existing ticketing system with an incident workflow By team size; a ticketing system with a good workflow beats a second tool nobody adopts
Status communication Provider status pages, a third-party status service, manual communication for internal services By whether customers or internal users need to see it, and by the contractual obligation to notify
Synthetic monitoring Provider synthetics, an external monitoring service, a self-hosted checker External wherever possible, because the value of a synthetic check is that it runs outside the system it measures
Load testing Open-source load generators in the pipeline, a commercial distributed testing service By peak traffic and whether the test must run from multiple geographies
Cost of telemetry Sampling strategies, retention tiers, cardinality control, log filtering at source Decided at design time; retrofitting cost control into an observability estate means losing data you may later need

We record the reasoning for each choice in the same document as the inventory, so that the decision can be revisited when the estate changes rather than re-litigated when a vendor calls.

Observability Cost Discipline

Telemetry spend grows silently and can exceed the compute it observes. The controls below are applied at design time, because after a year of accumulation the only available lever is deleting data somebody might need.

Lever What it does Typical effect
Retention tiers High-resolution recent data, downsampled aggregates for history, cold archive where an obligation requires it The largest single reduction, and the one that preserves the ability to answer old questions
Cardinality control Bounding label values, refusing user identifiers and unbounded keys as labels Prevents the exponential cost growth that surprises teams in month four
Adaptive and tail-based sampling Sampling normal traffic while always capturing errors and slow requests Retains the traces worth reading at a fraction of the volume
Log filtering at source Dropping debug output and repetitive health-check logs before they are shipped Commonly removes the majority of log volume with no loss of insight
Query discipline Expensive queries identified, scheduled or replaced with a recorded metric Reduces platform load and the cost of interactive exploration
Allocation per service Telemetry cost attributed to the service that produces it, reported monthly Changes behaviour more reliably than any central policy
Right-sized collection Collecting what an objective or a runbook requires, and nothing to satisfy a fear The honest version of the previous six levers

This is the same discipline as platform cost governance, and where a cloud estate is large we run it with Cloud Cost Optimisation & FinOps Services so that observability spend is allocated, owned and reviewed like any other line.

Evidence From Our Work

The engagements below are published case studies and insight from this practice, each described by the subject its own page documents. They are the engagements where monitoring, infrastructure operation or availability under load was the work itself.

Engagement Models and Commercials

Four shapes cover the work. Most clients begin with the reliability assessment, because it produces the inventory and the objective proposal that every later decision depends on.

Model Best when Duration and structure What you receive
Reliability assessment You cannot state how many services you run, who owns them, or what ‘working’ means Two to four weeks, fixed fee Service inventory with owners and criticality, telemetry audit, incident history analysis, proposed objectives per service, and a prioritised plan with effort estimates
SRE and observability build The inventory exists and the practice needs building: instrumentation, dashboards, alerting, on-call, error budgets Ten to fourteen weeks, staged with decision gates, joint team Instrumented services, dashboard set, alert policy, on-call model with runbooks, error budget policy, a rehearsed game day and a signed handover
Incident and on-call uplift Incidents are handled badly rather than infrequently: no command process, no communication templates, no reviews Four to six weeks Severity definitions, incident command process, communication templates, escalation paths tested, review structure and a backlog of preventive actions from the last year’s incidents
Reliability retainer The practice exists and needs operating: on-call cover, capacity review, alert tuning, reporting Monthly, three-month minimum, with measured service levels On-call cover or support to your rota, monthly reliability report, quarterly capacity and alert review, and incident command when it happens

Assessments are fixed fee. Builds are quoted after the assessment, because the scope depends on how many services need instrumentation and how much of the pipeline already exists. We will recommend the incident and on-call uplift alone where that is the actual problem, and it is the cheapest of the four.

Frequently Asked Questions About SRE & Observability

Do we need to hire site reliability engineers for this?

Eventually, in some form, but not first. The practice produces the job description: once you have an inventory, objectives and alert policy, you know what the role does and how many you need. Hiring first usually produces a capable engineer with nothing to operate and no mandate to change anything. Where you need capacity, IT Staff Augmentation Services supplies named engineers to your prioritisation.

What is a realistic availability objective?

The one your architecture can deliver and your business needs, which is often lower than instinct suggests. Each additional nine multiplies cost and complexity: 99.9 per cent permits about 43 minutes of downtime a month, 99.99 permits about four, and 99.999 permits about twenty-six seconds. Most internal and informational services should not be sold a fourth nine.

What is an error budget in practice?

The unreliability you have agreed to tolerate. At 99.9 per cent monthly, the budget is roughly 43 minutes. The team may spend it on releases and experiments; when it is gone, the agreed consequence applies — usually a freeze on non-critical change until reliability work restores it. The budget is what turns a reliability conversation from an argument into an accounting exercise.

How is this different from the monitoring we already have?

Monitoring answers the questions you anticipated. Observability lets you answer a new question about a system you have never seen behave this way, without deploying code. In practice the difference shows up in an incident: with monitoring you know something is wrong; with observability you can find out what, in minutes, using a correlation identifier that follows one request across every service.

Will this slow our developers down?

Instrumentation delivered through a golden path costs a team a day once. Alert noise costs everybody a night per incident, and a missing objective costs an argument every quarter. Where the concern is real, the answer is to put the instrumentation in the template rather than in each team’s backlog.

How much does observability cost to run?

Between a few per cent and, badly designed, more than the compute it observes. The determinants are retention, cardinality, log volume and sampling — all decided at design time. We build the cost controls in from the start and report telemetry cost per service monthly.

Can you provide on-call cover for us?

Yes, under a retainer with agreed service levels, or as second line behind your rota. We will not cover a service we have not documented: runbooks, escalation paths and access must exist first, which is why the build programme ends with a rehearsal.

How do you handle an estate with no pipelines and no infrastructure as code?

By sequencing: deployment discipline and environments first, then instrumentation, then objectives. Reliability practice on top of manual deployments produces excellent dashboards of a system nobody can change safely. DevOps & CI/CD Engineering is where that first stage sits.

What about our on-premises systems?

They are in the inventory and get objectives like everything else. Hybrid estates are common and the discipline is unchanged; only the instrumentation differs, and Managed IT Infrastructure & AMC covers the operating model for the on-premises side, including patching and support.

Do you set objectives for third-party services we depend on?

We record the objective the third party commits to, measure whether they meet it from our side, and design the fallback for when they do not. A dependency you cannot measure is a dependency you are trusting, and the trust should be a documented decision.

How do we know the programme worked?

Four things move: detection source shifts from customer-reported to monitoring-detected, mean time to restore falls, pages per shift fall while incidents stay visible, and error budgets stop being exhausted. In Figure 2 the same programme shows restoration time falling from nine hours to 1.8 and alerts per incident from 46 to nine.

Is this worth doing for a small estate?

Yes, at a smaller scale. Three services, one dashboard, two objectives, a rota with a tested escalation path and a review process is a full practice, and it costs a fraction of a platform programme. What is not worth doing is buying an observability suite for three services and leaving the objectives unwritten.

Commission a Reliability Assessment

Send us the list of services you believe you run, the last three incidents you remember, and the promise you have made to customers or to your board about availability. Within four weeks you will have a service inventory with named owners, a telemetry audit showing what you can and cannot answer today, an incident history analysis, proposed objectives and error budgets per critical service, and a costed plan covering instrumentation, alerting, on-call and the rehearsed handover — including the smaller option if that is what the estate needs.

Contact Unified Management Consulting to arrange a confidential first conversation.

Related Services

SRE and observability is delivered with the practices around it:

Leave a Reply

Your email address will not be published. Required fields are marked *