Packets, Paths, Policies: Postmortems

Showing posts with label Postmortems. Show all posts

Wednesday, October 7, 2026

Network Reliability Engineering: Turning Modern Network Operations into a Discipline

How SLOs, telemetry, service assurance, change safety, and incident learning turn modern network operations into measurable reliability outcomes.

Published: October 2026
Estimated reading time: 15 min

The last few months of this series have followed a deliberate arc: first prove what the network is doing, then model what it might do, then automate carefully within human guardrails. October is where those ideas become an operating discipline. Observability, digital twins, and closed-loop systems are powerful, but they do not automatically create reliability. Reliability appears when a team defines what good looks like, measures it consistently, changes production safely, and learns from failure without theatrics.

That discipline is often called Site Reliability Engineering in the application world. In networking, it needs a slightly different shape. Networks are shared infrastructure. They have failure domains, control planes, routing policy, service chains, physical constraints, and legacy dependencies that do not always fit neatly into application-style metrics. Still, the core SRE ideas translate extremely well: define service-level indicators, set objectives, budget risk, automate toil, stage changes, and run incidents as learning events.

This article uses the term Network Reliability Engineering for that discipline. It is not a new product category and it is not another dashboard. It is a way to operate IP networks, data centre fabrics, WANs, cloud interconnects, SASE paths, and transport services as dependable systems rather than collections of heroic interventions.

1) Network reliability starts with user-visible outcomes

Network teams traditionally measure what the network exposes: interface utilisation, optical power, CPU, adjacency state, BGP session state, dropped packets, tunnel health, controller alarms. These are useful signals, but they are not outcomes. A business user does not care that OSPF is stable if their payment terminal cannot complete a transaction. A trading desk does not care that a path exists if jitter breaks a voice bridge. A hotel does not care that broadband is up if guest Wi-Fi authentication fails.

Network Reliability Engineering starts by translating infrastructure behaviour into service behaviour. The question changes from “is the interface up?” to “is the service meeting its promise?” The promise might be reachability, latency, jitter, packet loss, DNS resolution time, tunnel establishment time, VPN availability, SaaS path quality, or successful transaction rate through a service chain.

For a campus: can users authenticate, obtain addresses, resolve names, and reach critical applications within expected times?
For a data centre: do workloads keep east-west and north-south connectivity through the intended policy path?
For a WAN: do critical applications stay within loss, latency, and jitter envelopes during provider brownouts and failovers?
For a service provider: do customer services meet the contracted reachability, restoration, and performance expectations?
For cloud interconnects: do advertised/accepted routes, path preferences, and throughput targets behave as intended after change?

The engineering move is simple but profound: monitor components, but manage services. Components explain failure; services define success.

2) SLIs and SLOs for networks: make reliability measurable

A service-level indicator (SLI) is a measurement of some aspect of service behaviour. A service-level objective (SLO) defines the target for that measurement over a period of time. In application SRE, common SLIs include availability, latency, error rate, and throughput. Networks can use those same ideas, but the indicators must be adapted to path, policy, and service topology.

The best network SLIs are close enough to infrastructure to be measurable and close enough to user experience to matter. “Core router CPU under 70%” is not a good SLI by itself. “Critical application traffic has less than 0.5% packet loss and less than 30 ms one-way jitter during business hours” is closer. “Branch users can complete DNS lookup, authentication, and SaaS reachability checks within defined thresholds” is better again.

Example SLI/SLO mapping

Service: Branch access to critical SaaS
SLIs:
  - Successful synthetic transaction rate
  - DNS resolution latency
  - Path latency/jitter/loss to SaaS edge
  - SD-WAN path changes per hour
  - User-impacting incident minutes
SLO:
  - 99.9% successful synthetic transactions per calendar month
  - 95th percentile DNS resolution below 100 ms
  - No more than X minutes of critical app path brownout per month

SLOs should be few, meaningful, and reviewable. A network with 400 “SLOs” has no SLOs; it has reporting noise. Start with the services that matter most: internet edge, identity, DNS/DHCP, core WAN, cloud interconnect, data-centre fabric, voice/video, payment, manufacturing, or safety systems. Then define SLOs that drive decisions.

2b) Network SLOs need both outcome and evidence

A strong network SLO is not just an uptime percentage. It links a user-visible outcome to the technical evidence that proves the outcome is being met. For example, “branch CRM access is available” is too vague. A tighter SLO might combine successful synthetic transactions, DNS response time, identity/authentication success, SD-WAN tunnel health, path loss/jitter, and application reachability into one service view.

Good network SLIs usually fall into four layers:

Experience SLIs: successful transactions, captive-portal completion, voice/video quality, DNS/authentication success, SaaS reachability.
Path SLIs: latency, jitter, packet loss, ECMP/path changes, tunnel stability, brownout minutes, congestion drops.
Control-plane SLIs: adjacency stability, route churn, convergence time, route-policy violation count, controller-to-device drift.
Policy SLIs: correct segmentation, expected route-target import/export, firewall/service-chain symmetry, identity policy evaluation success.

This distinction matters. A path can be up while the service is broken. A device can be green while policy is wrong. A controller can show compliance while users experience packet loss. NRE avoids that gap by tying the SLO to the service promise and then using infrastructure signals as supporting evidence, not as the promise itself.

3) Error budgets: the missing bridge between uptime and change

An error budget is the amount of unreliability you are willing to tolerate while still meeting the SLO. If the SLO is 99.9% availability over a month, the error budget is the remaining 0.1%. In network terms, the budget is not only “downtime.” It may include brownout minutes, path-quality violations, failed transactions, policy violations, or degraded service windows.

The value of an error budget is not the arithmetic. The value is decision-making. When the service is comfortably inside budget, the team can take controlled change risk: upgrades, routing-policy improvements, topology cleanup, and automation rollout. When the budget is burning too fast, change slows down except for fixes that reduce risk. This turns reliability into an operational negotiation instead of a religious argument.

Healthy budget: proceed with planned changes using normal guardrails.
Fast burn: reduce change velocity; prioritise remediation, capacity, and failure-domain reduction.
Budget exhausted: freeze risky change; focus on recovery, root cause, and reliability debt.
Repeated budget exhaustion: revisit architecture; the system may not be capable of meeting the stated SLO.

This matters because network teams often live in a false binary: either “no change, protect stability” or “change anyway, the business demands it.” Error budgets give the team a middle language: we can spend reliability risk intentionally, but not invisibly.

Error budgets also force a useful distinction between planned risk and unplanned pain. A maintenance window is not automatically “free” just because it was approved. If customers experience loss of service, failed transactions, or degraded paths, the impact should be visible against the reliability objective unless the service contract explicitly excludes it. This prevents the team from hiding poor change quality behind process language.

For networks, error-budget burn should be reviewed alongside capacity and change data. If burn clusters around upgrades, template pushes, route-policy edits, certificate renewals, or ISP failovers, the answer is not “watch harder.” The answer is to improve the change system, reduce blast radius, or redesign the failure domain.

4) Telemetry is the raw material, not the discipline

Modern network telemetry gives operators a richer picture than SNMP polling alone. Streaming telemetry, YANG Push, event-driven notifications, model-driven data, synthetic probes, flow records, logs, packet captures, controller APIs, and application traces can all contribute to service understanding. RFC 9232 frames network telemetry broadly: network data that helps understand current state, generated and collected in multiple ways, and processed for use cases such as service assurance and security.

But telemetry can easily become another swamp. Network Reliability Engineering adds discipline: collect what maps to decisions, retain enough history to understand trends, and attach telemetry to service models. The key question is not “can we collect it?” but “what decision does this data improve?”

State telemetry: protocol sessions, route counts, adjacencies, tunnel state, fabric membership, controller health.
Performance telemetry: latency, jitter, packet loss, queue drops, buffer pressure, path change frequency.
Policy telemetry: contract hits, ACL denies, route-policy matches, BGP prefix changes, security-service decisions.
Experience telemetry: synthetic transactions, real-user measurements, DNS/authentication success, application-specific probes.
Change telemetry: what changed, where, by whom/what, under which approval, and with which post-check result.

The most valuable telemetry joins these categories together. A path-quality event becomes useful when you can relate it to a BGP change, an SD-WAN path decision, a firewall-policy update, a provider incident, or a maintenance window. Correlation is where monitoring becomes observability; service mapping is where observability becomes assurance.

4.1 Telemetry quality gates

Telemetry needs its own reliability standard. A missing feed, stale timestamp, inconsistent label, or untrusted counter can create false confidence. Before telemetry drives SLO reporting or automation, it should pass basic quality gates:

Freshness: data arrives within a known interval and stale values are obvious.
Completeness: the service model knows which devices, paths, probes, and dependencies should report.
Consistency: labels, interface names, circuit IDs, sites, tenants, and policy objects use a common vocabulary.
Accuracy: synthetic probes, counters, and flow data are periodically compared against known-good tests or incident evidence.
Cardinality control: data remains queryable and affordable; labels do not explode into unusable noise.

Without those gates, telemetry becomes another unreliable dependency. With them, it becomes evidence.

5) Service assurance: decompose the promise

RFC 9417 describes service assurance as needing a holistic view across the elements and subservices involved in a service, and RFC 9418 provides a YANG data model for service assurance. That framing is useful because networks rarely fail as a single object. A branch service might depend on access switching, Wi-Fi, DHCP, DNS, identity, SD-WAN tunnels, internet egress, SSE policy, cloud routing, and the application endpoint. If you monitor those pieces separately without a service model, you get alarms but not understanding.

A service model decomposes the promise into subservices and dependencies. That lets you answer better questions: which subservice is degrading, how severe is it, who is affected, and whether the service is still within its SLO.

Service assurance decomposition (conceptual)

Business service: Branch staff can use cloud CRM
  - Access subservice: wired/wireless connectivity, DHCP, DNS
  - Identity subservice: authentication and policy assignment
  - Transport subservice: SD-WAN/SASE path health
  - Security subservice: SWG/CASB/ZTNA policy evaluation
  - Cloud subservice: cloud interconnect/SaaS endpoint reachability
  - Application subservice: synthetic transaction success

SLO decision:
  - Is the CRM service healthy for users, not just for devices?

This is where network reliability becomes practical. Instead of paging five teams because five systems raise alarms, the assurance layer should identify which dependency is causing user-visible degradation and whether the service objective is at risk.

5b) Ownership: every service needs a named reliability owner

Service assurance becomes weak when ownership is vague. A branch service may touch switching, wireless, identity, SD-WAN, firewall policy, cloud routing, and SaaS behaviour. If each team only owns its device class, nobody owns the outcome. NRE fixes that by assigning a reliability owner for the service, even when delivery spans multiple technical domains.

The owner does not have to control every device. Their job is to keep the service promise visible: maintain the dependency map, review SLO/error-budget data, coordinate change risk, drive postmortem actions, and make sure telemetry answers the questions the business actually asks. This turns “network operations” from a queue of alarms into a service discipline.

6) Change safety: many network outages are change-related

If reliability is the goal, change safety becomes central. Many network incidents are not spontaneous hardware failures; they are changes that behave differently in production than they did in the engineer’s head. A route-policy edit leaks more than intended. A firewall rule blocks return traffic. A firmware upgrade changes queue behaviour. A controller template modifies 200 sites at once. A prefix-list update is correct syntactically and wrong semantically.

Network Reliability Engineering treats change as an engineering workflow, not an event. The workflow should include design intent, pre-checks, blast-radius control, canary deployment, post-checks, rollback, and evidence capture.

Pre-check: validate topology, redundancy, route counts, policy diffs, and service health before change.
Risk envelope: define what the change is allowed to affect and which services must remain untouched.
Canary: deploy to one site, one edge, one tenant, or one fabric leaf before broad rollout.
Post-check: verify service SLIs, not only device state.
Rollback: know whether rollback is configuration revert, traffic drain, controller template rollback, image downgrade, or policy disablement.
Evidence: capture before/after snapshots so the team can learn without guessing.

A change is not complete when the command succeeds. It is complete when the service remains inside its reliability envelope.

7) Digital twins and pre-production validation: useful when bounded

The previous article explored network digital twins as a way to test change before production pays the price. In reliability engineering, twins serve a specific purpose: reduce uncertainty before action. They are not perfect replicas and should not pretend to be. They are decision-support systems whose value depends on model fidelity, data freshness, and honest confidence levels.

A practical twin can answer questions such as: Will this BGP policy still export only approved prefixes? Will this EVPN route-target change import unintended segments? Will this SD-WAN path preference overload a backup circuit? Will this SR policy still meet latency objectives during a link failure? Will this firewall insertion path become asymmetric?

The reliability rule is simple: the twin may increase confidence, but it does not eliminate verification. Use it for pre-checks, failure rehearsal, capacity what-ifs, and blast-radius estimation. Still perform post-change validation against live telemetry. A model that is never compared to reality becomes theatre.

7b) Production readiness for network services

Before a new fabric, WAN service, firewall policy model, cloud interconnect, or automation workflow becomes production-critical, it should pass a lightweight production-readiness review. This is not bureaucracy. It is a reliability checklist that asks whether the service can be operated safely after the project team moves on.

Service promise: what outcome does the service provide and who consumes it?
Failure modes: what breaks when a link, node, controller, certificate, route-policy, or dependency fails?
Observability: which SLIs prove health, and which signals identify the failing subservice?
Change safety: how do we canary, validate, rollback, and capture evidence?
Runbooks: what should an on-call engineer do at 2 a.m. without relying on tribal knowledge?
Capacity and limits: what scale assumptions exist, and which threshold becomes an escalation point?

The best reviews are short and concrete. They do not try to eliminate all risk. They make risk visible before the service becomes someone else’s outage.

8) Closed-loop automation: reliability depends on boundaries

Closed-loop automation can reduce toil and speed recovery, but it can also create faster failures if the loop is poorly bounded. A safe loop observes, interprets, decides, acts, and verifies within a defined authority envelope. The narrower the envelope, the more autonomy you can permit. The wider the blast radius, the more human approval and staged rollout you need.

Safe autonomous actions: restart a failed probe, collect diagnostic bundles, open tickets, enrich incidents, drain a single non-critical path when preconditions are satisfied.
Human-approved actions: broad route-policy changes, firewall-policy changes, controller-template pushes, multi-site traffic steering, or anything that changes security posture.
Never-silent actions: changes that affect regulatory boundaries, customer separation, identity policy, or route export must be auditable and explainable.

Reliability comes from keeping automation inside its authority. The system should know what it may do, what it may recommend, what it must block, and what it must escalate. Confidence thresholds matter. So do rollback plans and independent verification.

8b) Alerting: page on symptoms, investigate with causes

A reliable operating model avoids paging on every cause. If the team pages on every interface flap, BGP event, CPU spike, probe miss, and controller warning, the on-call queue becomes noise. A better pattern pages on service symptoms and uses cause-level telemetry for investigation.

Burn-rate style alerting is useful for network SLOs because it distinguishes “small issue we should watch” from “budget is disappearing now.” A short-window fast-burn alert catches acute incidents; a longer-window slow-burn alert catches chronic degradation. This is especially valuable for brownouts, jitter, route churn, wireless association issues, and intermittent SaaS path degradation.

9) Incident response: reduce time-to-truth, not just time-to-close

Good incident response is not heroics. It is structured learning under pressure. For network teams, the first goal is time-to-truth: identify the failing service, the affected population, the likely fault domain, and the safest containment action. Time-to-close matters, but closing quickly without understanding often creates repeat incidents.

Declare early: name the service impact, severity, owner, and communication channel.
Stabilise first: stop the bleeding before deep analysis if user impact is active.
Correlate change: compare incident start time with config, route, policy, controller, provider, and capacity events.
Preserve evidence: capture route tables, controller state, flow records, telemetry windows, and policy diffs before they roll off.
Communicate in service terms: tell stakeholders what service is affected, who is affected, what workaround exists, and when the next update is due.

A reliable network organisation practices incidents before the big one. Tabletop exercises, failure drills, maintenance simulations, and postmortem reviews are not bureaucracy. They keep the team from improvising when the network is already under stress.

10) Postmortems: fix the environment, not the person

Blameless postmortems are not about avoiding accountability. They are about improving the system. If a smart, well-intentioned engineer can make a change that breaks production, the question is not “who failed?” The question is “why did the system allow that failure mode to reach users?”

For network incidents, postmortems should produce reliability work: better policy validation, tighter blast-radius control, improved telemetry, clearer SLOs, safer automation boundaries, better rollback, improved documentation, or architecture changes that remove fragile dependencies.

Bad action item: “be more careful next time.”
Better action item: add a pre-check that blocks route export if prefixes exceed the approved source-of-truth list.
Bad action item: “monitor it more closely.”
Better action item: create a service SLI for successful synthetic transactions and alert only when user-impacting thresholds are crossed.
Bad action item: “write better documentation.”
Better action item: codify the rollback workflow into the change template and test it quarterly.

The postmortem is successful when it makes the same incident harder to repeat.

11) Toil: the network version

SRE uses the term toil for manual, repetitive, automatable work that scales linearly with the service. Networking has plenty of it: collecting command outputs, chasing repeated circuit alerts, hand-editing prefix lists, manually comparing route tables, acknowledging noisy alarms, building change evidence by hand, and copying configuration snippets from old tickets.

Reducing toil is not about automating everything. It is about freeing engineers to work on reliability improvements instead of repetitive mechanics. Good candidates are low-risk, high-frequency tasks with clear inputs and outputs. Dangerous candidates are broad changes with unclear blast radius, weak data, and no rollback.

Good automation target: pre-change state capture and post-change comparison.
Good automation target: route-policy linting against source-of-truth data.
Good automation target: incident enrichment with topology, recent changes, and known dependencies.
Risky automation target: autonomous multi-site traffic steering based on a single noisy metric.
Risky automation target: automatic security-policy relaxation to restore reachability.

12) A practical maturity path

Network Reliability Engineering does not require a massive programme. It can start small and compound. The right maturity path depends on the environment, but a pragmatic sequence looks like this:

Stage 1 - Define critical services: identify the network services that matter most to the business and customers.
Stage 2 - Define SLIs/SLOs: create a small set of service-level indicators and objectives that reflect user-visible outcomes.
Stage 3 - Build service-aware telemetry: map device/path/policy telemetry to services and dependencies.
Stage 4 - Make change safer: introduce pre-checks, canaries, post-checks, rollback, and evidence capture.
Stage 5 - Reduce toil: automate repetitive data gathering, validation, ticket enrichment, and reporting.
Stage 6 - Add modelling and closed loops carefully: use twins and automation inside bounded authority envelopes.
Stage 7 - Institutionalise learning: run postmortems, track action items, and fund reliability work as real work.

The goal is not maturity for its own sake. The goal is a network team that can move faster because it has better evidence, better guardrails, and better learning loops.

12b) Operating cadence: make reliability a recurring review

Reliability improves when it has a cadence. A monthly or quarterly network reliability review can be simple:

Which services met or missed their SLOs?
Where did error budget burn come from: change, capacity, provider faults, software defects, or unknown causes?
Which incidents repeated, and which postmortem actions remain unfunded?
Which telemetry feeds were missing, stale, or misleading?
Which changes should be canaried, modelled, or slowed because the budget is under pressure?

This cadence is where NRE becomes real. It connects engineering work to reliability evidence and gives leadership a way to fund risk reduction before the next visible failure.

13) Common traps

Dashboard worship: assuming a dashboard means the team understands service health.
Metric overload: collecting every signal and alerting on too many of them.
Device-centric thinking: declaring victory because boxes are green while services are degraded.
Automation theatre: automating change without improving validation, rollback, or blast-radius control.
SLO inflation: setting targets that sound impressive but are not measurable or affordable.
Postmortem drift: writing good postmortems and then never funding the corrective work.
Tool-first adoption: buying a platform before defining service models, ownership, and decision rules.
False precision: reporting impressive decimals from data that is stale, incomplete, or not tied to service impact.
Hero culture: celebrating late-night recovery without funding the engineering work that would prevent repetition.

The cure is discipline: fewer promises, better measurement, safer change, clearer ownership, and relentless learning.

14) Closing: reliability is an operating model

Modern networks are becoming more programmable, more observable, more automated, and more tied to business services. That increases both opportunity and risk. Observability helps you see. Digital twins help you reason before change. Closed loops help you react faster. But none of them replace engineering judgement.

Network Reliability Engineering is the operating model that holds these pieces together. It asks simple questions: What service promise are we making? How do we measure it? How much risk can we spend? What changed? What failed? What will prevent the same failure from reaching users again?

When a network team can answer those questions consistently, the network becomes less dependent on memory, heroics, and tribal knowledge. It becomes a system that learns.

Packets, Paths, Policies