Reduce IT Failure Risk by Designing for a Smaller Blast Radius

IT failures cannot be eliminated, but their reach can be constrained. The practical goal is no longer to promise uninterrupted technology; it is to protect the services that matter most, detect harmful changes early and recover before disruption becomes unacceptable.

That shift has become more explicit since the original discussion of outage prevention. Operational-resilience regimes now expect some financial organizations to define tolerable disruption, map dependencies and prove recovery through testing. The same operating model is useful outside regulated finance because it turns resilience from a collection of infrastructure projects into measurable protection for customers and business operations.

Start with the service, not the server

A resilient organization first identifies the outcomes that customers, employees or partners must still be able to obtain. “Process card payments,” “dispatch urgent orders” and “give clinicians access to current records” are clearer priorities than a broad label such as “keep IT running.” Each outcome should have an accountable business owner as well as technical owners.

For every important service, define an impact tolerance: the maximum disruption the organization is prepared to accept before the consequences become intolerable. Time is one dimension, but it should not be the only one. Depending on the service, useful limits may include failed transactions, affected customers, unreconciled records, missed deadlines or the volume of work that must later be processed.

This service-based approach is now visible in current supervision. In its March 2026 review, the UK Financial Conduct Authority’s findings say in-scope firms had to complete mapping and testing by 31 March 2025 so important business services could remain within impact tolerances. The regulator also found that stronger assessments combined time limits with measures such as transaction volumes and financial thresholds, while weaker work sometimes lacked evidence from sufficiently severe tests.

Map the entire route to the customer

Once a critical service is named, trace everything required to deliver it: applications, data stores, networks, identity systems, people, facilities, operational procedures and outside suppliers. The useful unit is the end-to-end service, not an individual application. A database may be duplicated while the only identity provider, network route or specialist operator remains a single point of failure.

The map should answer operational questions rather than become a decorative architecture diagram. Which dependency can stop the service? Who can authorize a workaround? What happens if the primary communications channel is unavailable? How much capacity does the recovery environment actually have? A dependency without an owner, fallback or tested recovery route is an identified business risk, not merely a technical detail.

Supplier concentration deserves the same treatment. Two applications hosted by different vendors may still depend on one cloud region, telecommunications carrier or managed-service team. Contracts and service-level targets are relevant, but they do not replace a customer-facing contingency plan for a provider failure.

Reduce the risk introduced by change

Software and configuration changes should move through production in controlled stages. A release process needs a small initial population, health signals tied to user outcomes, a defined observation period, stop conditions and a rollback path that has already been exercised. Merely calling a deployment a pilot does not constrain risk if the team cannot detect damage or reverse the change.

Google Cloud’s current deployment guidance describes canary deployment as sending a new revision to only part of the infrastructure before expanding it, thereby reducing the number of users exposed to a bug. It also notes an important limitation: a canary can be skipped when there is no existing recognized version to replace, so teams must verify what their deployment platform will actually do rather than assume the label guarantees staged exposure.

Release controls should cover more than application code. Database migrations, feature flags, security policies, infrastructure definitions and vendor integrations can all alter production behavior. Separate changes where possible, preserve compatible versions during rollout and record which configuration is active. A rollback that restores binaries but leaves an incompatible schema or configuration behind is not a complete rollback.

Design recovery around the impact limit

Backups are useful only when the organization can restore the required data, rebuild the service and validate the result within its operating limits. Set recovery objectives from the customer-facing tolerance, then reserve time for reconciliation, accumulated demand and communication after the technical system returns. A service may be online while customers are still unable to complete the task that matters.

Recovery plans should distinguish several possible responses:

• fail over to a healthy environment;
• roll back a damaging change;
• restore clean data and verify its integrity;
• degrade gracefully by suspending nonessential functions;
• use a bounded manual process for priority cases.

Each option has different prerequisites and failure modes. A failover can reproduce corrupted data, a manual workaround can create privacy or reconciliation problems, and an untested standby may lack capacity. Exercises should therefore verify both restoration and the business result, including access controls, data consistency and the processing of any backlog.

Test severe but plausible failures

Tabletop exercises are suitable for roles, escalation and decisions, but they cannot prove that systems will fail over or data will restore. Combine discussion-based exercises with technical recovery tests and carefully controlled production experiments. Scenarios should remove a dependency the service genuinely relies on, such as identity, a supplier connection, a region, administrative access or the normal incident-communication channel.

For EU financial entities within scope, testing is no longer simply an optional maturity practice. The EU Digital Operational Resilience Act, applicable from 17 January 2025, requires a risk-based digital operational-resilience testing program for covered financial entities other than microenterprises, with proportionate provisions for microenterprises. The regulation also requires findings to be addressed and places ICT risk, incident handling and third-party dependencies within one resilience framework.

A useful exercise ends with evidence: actual restoration time, data loss, service capacity, affected transactions, workaround throughput and unresolved dependencies. Compare those results with the impact tolerance. If the test succeeded only because an expert improvised an undocumented step, record that dependency rather than declaring the plan proven.

Control the incident before fixing everything

During an outage, the first objective is to limit harm and establish command. Assign one incident lead, one operations lead and one communications owner; maintain a timestamped decision log; freeze unrelated changes; and define the next review point. Multiple teams making uncoordinated production changes can obscure the cause and expand the incident.

Customer communication is an operational control. State which service is affected, what customers should do, which alternatives are safe and when the next update will arrive. Avoid speculative causes and recovery times. Internal responders also need an out-of-band channel and access method in case the normal collaboration or identity system is part of the failure.

Turn every failure into funded remediation

A post-incident review should explain how technical conditions, controls and decisions combined to produce the impact. It should not stop at the person who triggered a change. Ask why the change reached that population, why monitoring missed it, why rollback was slow and why recovery crossed the service’s tolerance.

Every material finding needs an owner, deadline and verification method. Close it only when evidence shows that the control works—for example, a repeated recovery test, an automated deployment stop or removal of a critical dependency. Track recurring patterns across incidents, because several modest failures involving the same weak change process or supplier dependency may justify investment more clearly than any single event.

The strongest resilience program therefore does not claim that failure has been removed. It shows which services are protected, how much disruption they can absorb, how changes are contained and whether recovery has worked under realistic conditions. That evidence gives leaders a defensible basis for deciding where reliability spending will reduce customer and business harm.