D6 · Recovery

Failover and Failback

Failover is activating the DR system when the primary fails. Failback is returning to the primary system once it recovers. AZ-104 evaluates the failover processes in ASR, Azure SQL and Traffic Manager, including the DNS implications.

Failover and Failback — concepts

Failover

Activating the secondary DR system when the primary fails or degrades. Traffic is redirected to the secondary system.

Automatic failover

Health checks detect the failure and redirect traffic without human intervention. E.g.: Traffic Manager with endpoint monitoring, SQL Failover Groups.

Manual failover

An operator decides to initiate the failover. E.g.: ASR failover from the portal, initiated manually once the disaster is confirmed.

Failback

Returning to the original primary system once it has recovered. The reverse process of failover.

When to do failback?

Only when the primary system is fully recovered, verified and validated. There's no rush — running on the secondary is safer than a rushed failback.

Data created during the failover

Data generated on the secondary system during the failover period must be synced back to the primary before failback.

Split-brain — the risk of failover

Split-brain happens when both the primary and secondary systems believe they are the active one at the same time and accept writes. It results in data divergence — the two systems end up with different, incompatible data. Automatic failover systems have mechanisms to prevent it (fencing, quorum). In manual failover: always confirm the primary is completely down before activating the secondary.

Failover with

ASR failover process

Decision

Confirm the primary region is down or unreachable. Communicate to stakeholders. Activate the DR plan.

Start failover

In RSV → Replicated Items → VM → Failover (or Recovery Plan → Failover). Select a recovery point (latest processed, latest app-consistent, or a specific one).

Recovery point

Latest processed: lower RPO but may be crash-consistent. Latest app-consistent: cleaner data but older. Custom: choose a specific point in time.

Verification

VMs created in the secondary region. Verify connectivity, data, and app functionality before redirecting traffic.

Commit

Confirm the failover. After the commit, the origin's recovery points are deleted. No going back without re-protection.

Redirect traffic

Update DNS or Traffic Manager to point to the secondary region's endpoints.

ASR failback process

Re-protect (mandatory step 1)

After the failover commit, the secondary region is the primary. Run "Re-protect" to start replicating back to the original region.

Wait for sync

ASR replicates the secondary region's current data to the original region. Time depends on the volume of changes.

Failover back

Once sync completes, do a Failover from the secondary region back to the original one. Verify it works.

Re-protect (step 2)

After the failback, re-protect again: the original region is primary, the secondary is DR. Initial configuration restored.

failover

Active Geo-Replication

Up to 4 readable secondary replicas in different regions. Manual failover: az sql db replica set-primary. The secondary is promoted to primary.

  • •RPO: < 5 seconds of replication lag
  • •RTO: 30 seconds - 2 minutes (manual)
  • •Secondary replicas are readable (read-only queries)
  • •Manual failover — not automatic without Failover Groups
  • •The DNS endpoint changes on failover — the app must be updated

Auto-Failover Groups

An abstraction over Active Geo-Replication. Provides a single listener endpoint that automatically points to the primary. Configurable automatic failover.

  • •RPO: < 5 seconds
  • •RTO: < 30 seconds (automatic with a configured policy)
  • •Stable endpoint: <group>.database.windows.net (primary) and <group>.secondary.database.windows.net
  • •The app does NOT need to change endpoints during failover
  • •Automatic failover with a configurable grace period (0-3600 sec)

SQL Managed Instance — Failover Groups

SQL Managed Instance also supports Failover Groups but with some differences: failover can take longer (up to 30 min for large instances), it doesn't support multiple secondaries (only 1 per group), and the initial replication process can take days for large instances.

Failover with

Traffic Manager automatically detects when an endpoint is down (via health probes) and stops resolving DNS to it. With the Priority routing method, this implements automatic web application failover between regions.

Configuring failover with Traffic Manager

1Create a Traffic Manager profile with the Priority method
2Add the primary endpoint (East US) with priority 1
3Add the secondary endpoint (West US) with priority 2
4Configure health probes: HTTP(S) protocol, path, interval, failure threshold
5Traffic Manager continuously probes both endpoints. If the primary fails the probes, DNS queries start resolving to the secondary

TM health probe parameters

Probing interval: 30 seconds (normal) or 10 seconds (fast — more expensive)

How often the endpoint is probed

Tolerated failures: 3 (default)

How many failed probes before marking the endpoint as degraded

Probe timeout: 10 seconds

How long it waits for the probe response

DNS TTL: Configurable — default 60s

How long clients cache the DNS response. Lower TTL = faster failover

Traffic Manager RTO — total failover time

Effective RTO = Probe interval × Tolerated failures + DNS TTL

With defaults: 30s × 3 = 90s detection + 60s TTL = ~150 seconds (2.5 min) of failover

With fast probe (10s) + low TTL (30s): 10s × 3 = 30s + 30s = ~60 seconds

The low TTL is critical — if the TTL is 300s, clients will keep using the downed endpoint until their DNS cache expires.

DNS and failover time

TTL — the hidden factor in RTO

The TTL (Time To Live) of the DNS record determines how long clients cache the resolution. Even if you change the DNS, clients with an active cache keep using the old endpoint.

• TTL 3600 (1 hour): up to 1 hour of impact after DNS failover

• TTL 300 (5 min): up to 5 min after the DNS change

• TTL 60 (1 min): up to 1 min — aggressive but effective

• TTL 30: optimal for failover, but increases the load on DNS servers

Pre-disaster best practice

Lower the TTL to 60-120 seconds BEFORE a disaster happens (during normal operation). This has no performance impact but speeds up failover if it occurs.

Failover redirection options

Traffic Manager (Priority)

~1-3 min

Automatic DNS failover. Requires a low TTL to be effective.

<30 sec

Automatic failover at the edge with anycast. Failover is faster than DNS.

Manual DNS update

Record TTL

Manually change the A/CNAME record. Effective but slow if the TTL is high.

SQL Failover Groups

<30 sec

The listener endpoint updates automatically. The app URL doesn't change.

Failover checklist

Pre-failover (preparation)

☐DNS TTL lowered to 60-120 seconds
☐Test Failover run within the last 3-6 months
☐Failover runbooks documented and accessible offline
☐Contact tree up to date
☐DR credentials validated (Key Vault, secrets)
☐DR VMs pre-provisioned with up-to-date agents
☐DR DNS records prepared (just needing an A record change)
☐Capacity in the DR region verified (don't request quota during the disaster)

During the failover

☐Confirm the primary is really down (not split-brain)
☐Log the failover start time (to measure RTO)
☐Notify the operations team and management
☐Run the Recovery Plan in ASR (or equivalent action)
☐Verify the secondary VMs/DB are healthy
☐Verify data is consistent (no corruption)
☐Redirect DNS / Traffic Manager to the secondary endpoint
☐Verify the application works end-to-end
☐Communicate status to users/customers
☐Log the recovery time (to calculate the actual RTO)

Did you get this topic?

Test what you just learned

A company uses Traffic Manager with Priority routing for failover between East US (primary, priority 1) and West US (secondary, priority 2). Traffic Manager's TTL is set to 300 seconds. East US fails at 10:00. When will ALL users start using West US?

Ready to test "Failover and Failback"?

Put what you learned to the test with original AZ-104 exam questions.

Start AZ-104 mock exam

Unlimited mock exams with the Pro plan

Log in to track your progress.