Thursday, November 12, 2026

Failure Domains by Design: Keeping Local Faults Local in Modern IP Networks

Failure Domains by Design: Keeping Local Faults Local in Modern IP Networks

Why redundancy alone is not resilience—and how routing, forwarding, overlays, security, and shared dependencies determine the true blast radius

Published: November 2026
Estimated reading time: 15 min

Redundancy is one of networking's most comfortable words. Two routers. Two uplinks. Two firewalls. Two data centres. The architecture diagram looks reassuring, and every device reports healthy. Then a single failure takes down a service that, on paper, has no single point of failure. The missing concept is usually not redundancy. It is failure-domain design: understanding which components, protocols, policies, and dependencies can fail together—and ensuring that a local fault does not become a network-wide event.

October's discussion of Network Reliability Engineering established the operating model: define outcomes, measure them, change carefully, and learn from incidents. This month moves underneath that operating model into architecture. It asks a harder question: what has to be true about the network so that failures remain bounded in the first place? The answer spans far more than physical diversity. It includes routing convergence, control-plane scaling, capacity under failure, application state, segmentation boundaries, and the services that make the network usable.

The discussion applies equally to an enterprise campus, a multi-site corporate WAN, a leaf–spine data centre, a service-provider backbone, or a hybrid-cloud environment. The scales differ; the underlying engineering discipline does not. The objective is not a mythical network that never fails. It is one that fails locally, visibly, and recoverably.

1) Define the failure domain before drawing the redundant path

A failure domain is the set of things that may be disrupted by a particular fault. That definition sounds simple until the design includes shared dependencies. Two routers in different racks may use the same authentication platform. Two WAN paths may terminate on the same provider aggregation router. Separate virtual firewalls may share a hypervisor, storage pool, or policy-management plane. An overlay may span two sites while depending on a single underlay gateway or route-reflector cluster.

It helps to draw failure domains in layers rather than treating them as one physical boundary:

  • Physical: optics, fibres, conduits, power feeds, line cards, racks, buildings, carrier points of presence.
  • Forwarding: next hops, ECMP groups, fabric paths, encapsulation, MTU, queues, and forwarding-table resources.
  • Control plane: IGP flooding, BGP sessions and reflectors, controller clusters, route-policy distribution, and protocol timers.
  • Security and policy: VRFs, route targets, firewalls, NAT, identity enforcement, service insertion, and shared policy objects.
  • Operational: automation credentials, software images, template rollout rings, monitoring, backup, and recovery access.

Every added layer can turn apparently diverse paths into one shared fate. A useful design review therefore asks both “what happens when this component fails?” and “what else relies on it?” The second question is often where the real architecture appears.

2) Availability is not binary: contain the mode and the consequence

Fault containment begins by distinguishing failure modes. A severed fibre is a relatively clean fault: the port drops and the network can redirect traffic. A dirty optic can be worse: the interface remains up while errors rise intermittently. A routing process can hang without affecting the forwarding ASIC immediately. A security appliance can remain reachable while silently dropping application flows. A controller can be unavailable while an existing data plane continues to forward normally.

These failures require different responses. Detection answers whether something has changed. Protection answers whether traffic can use an alternative. Convergence answers when the routing and forwarding states settle. Service recovery answers when actual users can complete their work again. Treating those as one timer creates false confidence.

Fault-to-service timeline (conceptual)

Physical or logical fault
  -> detection (link alarm, BFD, telemetry, application probe)
  -> local protection or route withdrawal
  -> network reconvergence / FIB update
  -> policy and state consistency (firewall, VPN, overlay)
  -> application recovery and user-visible success

A fast first step does not guarantee a fast final step.

An architecture that survives link-down tests can still fail under packet corruption, controller partition, route flapping, asymmetric return paths, or overloaded backup capacity. A good fault model explicitly includes grey failures—partial degradation that looks healthy from the wrong measurement point.

3) Map the dependency graph, not merely the topology

The physical topology tells you where cables connect. The dependency graph tells you which services can survive losing those connections. Build a simple dependency map for a critical business service: user access, DNS, identity, WAN, routing, firewall path, application front end, database tier, and external API. Label each dependency as local, replicated, shared, or manually recoverable.

Consider a corporate service published from two data centres. Each site has its own routers, switches, and firewalls, but both sites depend on one identity provider or one DNS control path. The network is geographically diverse; the service is not operationally independent. Similarly, active–active application deployment helps little if both sites depend on a single stateful inspection hop for inter-site traffic.

  • Identify the minimum viable service: which dependencies must work for a user transaction to succeed?
  • Mark shared fate: common power, circuits, upstream AS, controller, policy repository, certificate services, DNS, or remote access.
  • Classify failure consequences: isolated tenant, one rack, one site, one region, or many customers.
  • Assign recovery mechanisms: automatic local protection, protocol reconvergence, application failover, or an explicit human runbook.

The resulting picture often reveals that the best reliability investment is not another router. It may be a second identity path, a local DNS resolver, independent out-of-band access, more spare transport capacity, or a less tightly coupled policy rollout.

4) IGP design: control flooding without manufacturing complexity

OSPF and IS-IS both build link-state databases and compute paths from topology information. Both can operate very reliably at large scale when their hierarchy and resource limits match the design. The key containment question is not which IGP wins a popularity contest; it is how a fault changes the topology seen by other routers, how widely that change is flooded, and whether the network has a usable alternate path.

In OSPF, area boundaries can limit some topology details and flooding, but area 0 connectivity, ABR policy, summarisation, and inter-area path selection must be designed deliberately. In IS-IS, Level-1 and Level-2 can establish hierarchy; a Level-2-only backbone can also be clean and operationally straightforward. Neither hierarchy is a magic isolation wall: external or summarised reachability can still become incorrect if the boundary advertises a route whose downstream destination is no longer usable.

For transport cores, consistent point-to-point adjacency design, stable metrics, predictable prefix origination, and conservative protocol policy frequently matter more than elaborate area segmentation. Split a domain when the operational benefits justify the additional boundaries, not simply because a diagram looks cleaner with more colours.

Keep failure detection separate from routing policy. Bidirectional Forwarding Detection (BFD), described in RFC 5880, can detect bidirectional forwarding-path failure quickly, but it does not itself compute alternate routes or guarantee that a line card has installed the new FIB entry. Aggressive timers need platform validation: scale, scheduling jitter, CPU pressure, and the sensitivity of slow links or virtualised appliances all affect whether the chosen values are safe.

5) Fast reroute: protection coverage is more important than a headline timer

Traditional IP fast reroute can precompute a loop-free alternate (LFA) next hop. RFC 5286 defines the conditions under which forwarding to that alternate avoids a loop after a protected failure. The limitation is structural: some topologies do not have a suitable local LFA for every destination. Remote LFA uses tunnels to extend coverage; Segment Routing-based TI-LFA provides another approach to precomputed local repair.

RFC 9855, published in October 2025, specifies TI-LFA using Segment Routing. Its scope is important: it describes local repair for certain directly connected failures in link-state IGP environments. It does not promise universal recovery from arbitrary multi-fault, remote, policy, or service failures. Even when a repair path exists, the network must have enough capacity and the forwarding implementation must support the required repair behaviour. A constrained path, service-function requirement, or forwarding-policy override may further restrict suitable alternates.

Micro-loops are another distinction. Routers can update forwarding state at different times after an IGP event; transient inconsistency can cause packets to loop or be dropped despite the eventual shortest-path computation being correct. Local protection can reduce initial interruption, but subsequent convergence still deserves testing. Avoid saying that TI-LFA, BFD, or “sub-50-ms convergence” automatically implies clean service restoration.

  • Measure coverage: what proportion of critical prefixes and protected resources actually has a valid repair path?
  • Validate capacity: can the alternate carry the redirected demand without dropping critical classes?
  • Inspect constraints: does the repair still satisfy transport class, service-chain, and security requirements?
  • Prove outcome: capture packet loss, jitter, micro-loop symptoms, and application recovery—not just protocol timestamps.

6) BGP route reflectors: a redundant control plane can still share fate

Route reflection removes the need for a full mesh of iBGP sessions, but it introduces topology and failure-domain decisions of its own. RFC 4456 describes route-reflector operation and redundant reflectors. Two reflectors are preferable to one only if their dependencies are sufficiently independent: routing reachability, compute resources, software upgrade cadence, racks, power, and policy distribution all matter.

There is also a subtler issue than reflector availability: path visibility. A reflector generally advertises selected paths according to BGP procedures; clients may not see every alternate available elsewhere in the AS. Redundant reflectors do not necessarily solve path hiding, and two reflectors configured with identical best-path views can propagate the same incorrect preference. Add-Path, diverse-path, or topology-aware design may help where justified, but each requires platform-specific evaluation and operational discipline.

Next-hop reachability is another frequent source of correlated failure. A BGP session can remain established while the IGP route to the BGP next hop disappears or becomes unstable. A healthy session counter therefore does not establish that the forwarding service is healthy. Monitor session state, accepted/advertised routes, next-hop resolution, FIB programming, and actual forwarding together.

Graceful Restart deserves care. RFC 4724 and RFC 8538 define mechanisms to retain routing information through certain session resets. That can protect service during control-plane restarts, but retaining stale routes can also extend blackholing if the forwarding path is genuinely gone. Restart and stale-route timers are design trade-offs, not universal “higher is safer” knobs.

7) EVPN fabrics: do not let an overlay erase physical boundaries

EVPN brings powerful control-plane mechanisms to Ethernet services and VXLAN overlays. It also makes it easy to create logical networks that span far beyond their original failure domain. The defining question is whether an overlay should preserve application placement flexibility, or whether it is quietly creating a dependency between racks, buildings, or regions that would otherwise be independent.

RFC 7432 describes Ethernet VPN procedures including multihoming and split-horizon behaviour; RFC 8365 applies EVPN to network virtualisation overlays such as VXLAN. These mechanisms address real problems—MAC reachability, multihoming, BUM handling, and control-plane signalling—but they do not eliminate the consequences of stretched Layer 2, inconsistent route-target policy, or an underlay failure.

For a multihomed Ethernet Segment, the designated forwarder (DF) role is particularly relevant to delivery of broadcast, unknown-unicast, and multicast traffic toward the attached segment. It is not a general-purpose election of “the active router for all forwarding” in every all-active design. Split-horizon rules, aliasing, mass withdrawal, and implementation-specific capabilities all affect what happens during a PE or link failure. Verify the actual redundancy mode and expected behaviour before basing a design on a simplified diagram.

Prefer local Layer 2 domains unless an application requirement justifies stretching. Use routed inter-site boundaries where possible, define explicit gateway placement, and test endpoint movement, MAC mobility, ARP/ND behaviour, BUM replication, and route-target import/export. An EVPN route imported into the wrong tenant is a policy failure even if the underlay is working perfectly.

8) Stateful security is a separate recovery domain

Networks often converge faster than the appliances through which their services must pass. A firewall may need session-state replication, NAT-state consistency, interface health checks, and a stable route in both directions. A packet returning along a different edge may be dropped despite perfect IP reachability. Load balancers, proxies, SD-WAN service chains, and inspection gateways introduce similar constraints.

Think through three questions for every stateful boundary. First, where is state held, and what is replicated? Second, how is the return path kept compatible with the session architecture? Third, what does a failure do to existing connections compared with new connections? An active/standby pair may protect new session establishment while interrupting some existing flows. An active/active topology may improve capacity but demands clear session ownership and asymmetry handling.

Central inspection is sometimes the right compliance or risk decision. The architectural mistake is to leave it implicit. Document which traffic must traverse which function, where failover sends it, and whether the alternate can maintain equivalent policy and throughput. Never trade away a mandatory security boundary merely to make a failover demonstration look smooth.

9) WAN, SD-WAN, and cloud: path diversity must be genuine

Two underlays do not guarantee two failure domains. A DIA circuit and an MPLS circuit can share a duct, central office, power system, or provider backbone. Two tunnels can land on the same cloud gateway. Two SSE points of presence can depend on one regional control service. A branch can have dual broadband links but a single last-hop access switch or firewall.

For SD-WAN, treat each overlay path as a measurable service with a specific failure model. Application-aware policies should distinguish transport loss, jitter, latency, tunnel loss, and security-egress health. Path switching needs damping because a brownout that repeatedly crosses a threshold can be more disruptive than a clean failure. A “backup” LTE link also needs realistic bandwidth, policy, and cost constraints: it may preserve point-of-sale or voice while intentionally throttling software updates and bulk transfers.

For cloud interconnects, verify that redundant physical circuits terminate in sufficiently independent locations and that BGP policy, cloud route tables, security controls, and service endpoints are correctly distributed. The ability to receive a cloud route over two sessions does not prove that two independent application paths exist. Test the service, not just the interconnect lights.

10) Diversity that exists only on paper is not diversity

Many high-impact incidents involve common-mode failures: the same software defect deployed to both HA members, a shared route-policy mistake, an automation pipeline that pushes identical broken configuration everywhere, or a firmware feature that fails across supposedly independent appliances.

Physical diversity is still essential, but pair it with change diversity and operational independence. Use upgrade rings, canaries, maintenance separation, and independent recovery access. When the architecture permits it, avoid upgrading both members of a critical redundancy pair during the same unproven change window. Preserve an out-of-band management path that does not depend on the routing domain being repaired.

Do not confuse independence with unnecessary product sprawl. Mixed-vendor systems can avoid certain common-mode defects, but they can also multiply interoperability and operational risks. The goal is assessed fault independence, not diversity as a slogan. Map the shared dependencies that matter and choose the simplest architecture that genuinely reduces correlated risk.

11) Failure capacity: N+1 is a topology property, not a traffic guarantee

Capacity is one of the most frequently overlooked failure-domain constraints. Imagine two 10 Gb/s links each carrying 7 Gb/s. In steady state, both look comfortable at 70% utilisation. Lose one link and 14 Gb/s of offered demand now competes for 10 Gb/s of surviving capacity. The topology is redundant; the traffic budget is not. If demand remains unchanged, at least 4 Gb/s cannot be carried without dropping, throttling, or rerouting it elsewhere.

The same problem appears in ECMP groups, firewall HA clusters, service insertion chains, cloud gateways, and inspection points. Packet replication for resiliency, traffic bursts, encapsulation overhead, and unequal path hashing make the effective problem harder than average interface utilisation suggests.

Engineer capacity for the credible failure set, not only nominal operation. Define which failures must be tolerated with no significant service impact and which may cause controlled degradation. Reserve or prioritise capacity for critical classes where appropriate; document expected shedding of lower-priority traffic. A network that preserves payment and emergency voice while bulk backups slow down may be far more resilient than one that tries to serve everything equally until every queue collapses.

12) Control-plane protection: reduce the blast radius of bad information

Some failures are not missing links; they are excessive or incorrect information. A route leak, malformed update, endpoint churn storm, runaway telemetry subscription, or misconfigured redistribution can consume control-plane resources and spread policy errors far from the initiating node.

Use peer-role-aware route import and export rules, prefix limits, bounded redistribution, and appropriate control-plane policing. Separate management interfaces and controller communication from customer or tenant forwarding where architecture and platform support it. Keep route-target and VRF policy explicit; defaults should fail safely rather than importing whatever happens to exist.

The most important containment mechanism is often procedural: canary deployment, validation of the actual advertised route set, and an emergency method to withdraw or revert a policy without depending on the affected overlay. Software-defined control does not remove the need for local safety limits on forwarding devices.

13) Make service dependencies observable, not merely components

Service assurance should show which applications or tenants are affected when a shared component degrades. RFC 9417 describes an assurance architecture that models services as dependency graphs and connects observed symptoms with impacted service instances. That is a useful mental model for fault containment: if an optical link fails, which paths and customers move onto alternatives? If a firewall instance fails, which applications lose state? If a DNS resolver disappears, which sites have a local alternative?

Build monitoring around a few linked layers. At the bottom, collect physical and forwarding evidence: errors, drops, optics, FIB utilisation, and queue behaviour. In the middle, measure routing and policy: adjacency changes, next-hop resolution, BGP path changes, EVPN route imports, and service-chain decisions. At the top, measure service health: synthetic transactions, DNS resolution, identity success, SaaS reachability, and real-time call quality.

Correlate those layers with change events. A single link flap that produces no user impact may be evidence that containment worked. A BGP session that stays up while transactions fail is evidence that your health model is incomplete. Both observations matter more than an undifferentiated dashboard full of green lights.

14) An architecture test matrix that exposes hidden coupling

A controlled test programme should go beyond unplugging one cable. For each critical service, define the failure, expected containment boundary, acceptable user impact, and evidence to capture. Different tests reveal different classes of coupling:

  • Clean link failure: drop a core or WAN link; measure protection coverage, convergence, path utilisation, and application interruption.
  • Degraded link: introduce measurable loss or errors where test tools permit; confirm that health detection identifies a brownout rather than accepting a nominally up interface.
  • Control-plane restart: restart a routing or controller process in a non-production test; distinguish uninterrupted forwarding from stale state or delayed policy updates.
  • Route-policy error: test a deliberately disallowed prefix in a lab; verify import/export limits, alerting, and rollback controls.
  • EVPN endpoint event: move or withdraw a workload attachment; measure MAC/IP convergence, BUM behaviour, tenant isolation, and unexpected flooding.
  • Security-path failure: fail a stateful inspection device; distinguish new-session success, existing-session survival, NAT behaviour, and return-path consistency.
  • Site or region isolation: simulate loss of inter-site transport; verify DNS, identity, management, private applications, and local internet access separately.
  • Backup overload: reproduce peak traffic against surviving capacity; observe priority classes and the point at which controlled degradation begins.

Capture a baseline before each exercise and test only under an agreed change window with a clear rollback path. Simulation or a digital twin can reject some unsafe designs early; controlled production tests are still needed where residual risks justify them. A successful test is not “no alerts fired”. It is “the observed failure stayed inside the expected boundary, and we can show the evidence”.

15) Three practical design sketches

15.1 Corporate campus and branch WAN

Keep distribution and access failures local. Route at aggregation boundaries where practical, provide independent upstream paths for critical locations, and isolate guest/IoT traffic from corporate services. Ensure identity, DHCP, DNS, and device-management access have realistic site-outage behaviour. A branch with redundant WAN links but no local name resolution or independent authentication may still be operationally single-homed.

15.2 Leaf–spine data centre with EVPN/VXLAN

Use a simple routed underlay with measured ECMP and FIB behaviour. Keep tenants separated through deliberate VRF and route-target policy. Size spine and border capacity for failed-node scenarios, validate EVPN multihoming and endpoint-mobility procedures, and avoid stretching Layer 2 across sites without a specific application justification. Treat fabric-border security and cloud routes as distinct failure domains.

15.3 Service-provider IP/MPLS or Segment Routing backbone

Preserve stable IGP reachability for loopbacks and transport endpoints. Engineer repair coverage for protected resources, but validate post-failure capacity and service-class behaviour. Keep BGP route-reflection dependencies clear; monitor next-hop resolution and VPN route propagation rather than relying on session state alone. Segment external interconnect policy and avoid uncontrolled route redistribution. Test both transport restoration and customer-service restoration, especially where stateful gateways or service chains exist.

16) A compact design-review checklist

  • Can every critical service be mapped to its physical, control-plane, security, and operational dependencies?
  • Are primary and backup paths truly independent where independence is required?
  • Is the expected impact radius defined for link, node, site, region, controller, and policy failures?
  • Does protection cover the destinations and failure types that matter—not just the simple lab case?
  • Can surviving links and appliances handle the credible failure load, including bursts and service-class priorities?
  • Do routing, EVPN, and service-chain changes preserve security boundaries and stateful-flow behaviour?
  • Can teams still reach and repair the infrastructure when the production routing domain is impaired?
  • Have brownouts, stale routes, common-mode software faults, and partial partitions been considered?
  • Are application-level tests available to prove recovery rather than infer it from green protocol indicators?

17) Closing: resilience is the discipline of limited consequences

Networks will continue to lose links, routers, optics, peers, controllers, and sometimes entire sites. The real architectural measure is how far each failure travels. Does one bad optic remain a local transport issue? Does one incorrect route policy stop at a defined boundary? Does one unavailable security node produce predictable, contained service degradation rather than widespread asymmetric failure?

That is the difference between installing redundancy and engineering resilience. The goal is not to eliminate every fault; it is to make the likely faults small, the unlikely faults survivable, and the consequences understandable. When the failure domains are deliberate, Network Reliability Engineering stops depending on exceptional recoveries and starts benefiting from a network designed to remain useful even when parts of it are not.

References and standards anchors

The following primary references provide the protocol and service-assurance foundations for the design discussion. Deployment capabilities and defaults vary by platform and software release.

  • RFC 5880 — Bidirectional Forwarding Detection (BFD): failure detection between forwarding engines.
  • RFC 5286 — Basic IP Fast Reroute using Loop-Free Alternates.
  • RFC 7490 — Remote Loop-Free Alternate Fast Reroute.
  • RFC 9855 — Topology Independent Fast Reroute Using Segment Routing (TI-LFA).
  • RFC 4456 — BGP Route Reflection: An Alternative to Full Mesh iBGP.
  • RFC 4724 — Graceful Restart Mechanism for BGP.
  • RFC 8538 — BGP Graceful Restart notification handling and stale-route considerations.
  • RFC 7432 — BGP MPLS-Based Ethernet VPN (EVPN): multihoming and split horizon.
  • RFC 8365 — A Network Virtualization Overlay Solution Using EVPN.
  • RFC 9417 — Service Assurance for Intent-Based Networking Architecture.
Tags: Network Architecture, Failure Domains, Resilience, High Availability, BFD, IGP, BGP, Route Reflection, TI-LFA, Fast Reroute, EVPN, VXLAN, SD-WAN, Security, Service Assurance, Enterprise Networking, Service Provider

Failure Domains by Design: Keeping Local Faults Local in Modern IP Networks Failure Domains by Design: Keeping Local Faults...