15 minlesson

Circuit Breaker Pattern

Circuit Breaker Pattern

The circuit breaker pattern prevents an application from repeatedly trying to execute an operation likely to fail, allowing the system to recover.

Why Circuit Breakers?

Without circuit breaker, a failing service causes:

1Request 1 → Service (fails after 30s timeout)
2Request 2 → Service (fails after 30s timeout)
3Request 3 → Service (fails after 30s timeout)
4...
5All threads blocked waiting for a dead service!

With circuit breaker:

1Request 1 → Service (fails)
2Request 2 → Service (fails)
3Request 3 → Service (fails)
4Circuit OPENS!
5Request 4 → Fails immediately (circuit open)
6Request 5 → Fails immediately (circuit open)
7...
8[After break duration]
9Request N → Service (succeeds)
10Circuit CLOSES

Circuit States

1     ┌──────────────────────────────────────┐
2     │                                      │
3     ▼                                      │
4┌─────────┐  failures exceed   ┌─────────┐  │
5│ CLOSED  │─────threshold─────▶│  OPEN   │  │
6└─────────┘                    └────┬────┘  │
7     ▲                              │       │
8     │                         break        │
9     │                        duration      │
10     │                         expires      │
11     │                              │       │
12     │       success          ┌────▼────┐   │
13     └────────────────────────│HALF-OPEN│───┘
14                   failure    └─────────┘
  • Closed: Normal operation, requests flow through
  • Open: Requests fail immediately without trying
  • Half-Open: Test requests to see if service recovered

Basic Circuit Breaker

csharp
1var circuitBreaker = Policy
2    .Handle<HttpRequestException>()
3    .CircuitBreakerAsync(
4        exceptionsAllowedBeforeBreaking: 3,  // Open after 3 failures
5        durationOfBreak: TimeSpan.FromSeconds(30)  // Stay open for 30s
6    );

Circuit Breaker with Callback

csharp
1var circuitBreaker = Policy
2    .Handle<HttpRequestException>()
3    .CircuitBreakerAsync(
4        exceptionsAllowedBeforeBreaking: 3,
5        durationOfBreak: TimeSpan.FromSeconds(30),
6        onBreak: (exception, duration) =>
7        {
8            Console.WriteLine($"Circuit OPEN for {duration.TotalSeconds}s: {exception.Message}");
9        },
10        onReset: () =>
11        {
12            Console.WriteLine("Circuit CLOSED - service recovered");
13        },
14        onHalfOpen: () =>
15        {
16            Console.WriteLine("Circuit HALF-OPEN - testing...");
17        });

Advanced Circuit Breaker

Based on failure rate over a time window:

csharp
1var advancedBreaker = Policy
2    .Handle<HttpRequestException>()
3    .AdvancedCircuitBreakerAsync(
4        failureThreshold: 0.5,  // 50% failure rate
5        samplingDuration: TimeSpan.FromSeconds(10),  // Over 10 seconds
6        minimumThroughput: 8,  // Need at least 8 requests to evaluate
7        durationOfBreak: TimeSpan.FromSeconds(30)
8    );

Checking Circuit State

csharp
1// Check current state
2CircuitState state = circuitBreaker.CircuitState;
3

4switch (state)
5{
6    case CircuitState.Closed:
7        Console.WriteLine("Operating normally");
8        break;
9    case CircuitState.Open:
10        Console.WriteLine("Circuit is open - failing fast");
11        break;
12    case CircuitState.HalfOpen:
13        Console.WriteLine("Testing if service recovered");
14        break;
15    case CircuitState.Isolated:
16        Console.WriteLine("Manually isolated");
17        break;
18}

Manual Circuit Control

csharp
1// Manually break the circuit
2circuitBreaker.Isolate();
3Console.WriteLine("Manually opened circuit for maintenance");
4

5// Manually reset
6circuitBreaker.Reset();
7Console.WriteLine("Manually closed circuit");

Handling Open Circuit

csharp
1try
2{
3    await circuitBreaker.ExecuteAsync(() => httpClient.GetAsync(url));
4}
5catch (BrokenCircuitException ex)
6{
7    // Circuit is open - service is known to be unavailable
8    Console.WriteLine("Service unavailable, try again later");
9

10    // Return cached data or default
11    return GetCachedResponse();
12}
13catch (HttpRequestException ex)
14{
15    // Service responded but with error
16    Console.WriteLine($"Request failed: {ex.Message}");
17    throw;
18}

Combining Retry and Circuit Breaker

Important: Wrap retry INSIDE circuit breaker:

csharp
1// Retry policy
2var retry = Policy
3    .Handle<HttpRequestException>()
4    .WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(1));
5

6// Circuit breaker (outer)
7var breaker = Policy
8    .Handle<HttpRequestException>()
9    .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));
10

11// Combine: breaker wraps retry
12var policy = Policy.WrapAsync(breaker, retry);
13

14// Usage
15await policy.ExecuteAsync(() => httpClient.GetAsync(url));

Execution flow:

1breaker.Execute(
2    retry.Execute(  // Retries 3 times
3        httpClient.GetAsync(url)
4    )
5)
6// If all retries fail, that counts as ONE failure for circuit breaker

Per-Service Circuit Breakers

Use separate breakers for each external service:

csharp
1class ServiceClientFactory
2{
3    private readonly Dictionary<string, IAsyncPolicy> _breakers = new();
4

5    public IAsyncPolicy GetCircuitBreaker(string serviceName)
6    {
7        if (!_breakers.TryGetValue(serviceName, out var breaker))
8        {
9            breaker = Policy
10                .Handle<HttpRequestException>()
11                .CircuitBreakerAsync(3, TimeSpan.FromSeconds(30),
12                    onBreak: (ex, ts) => Console.WriteLine($"{serviceName} circuit OPEN"),
13                    onReset: () => Console.WriteLine($"{serviceName} circuit CLOSED"));
14

15            _breakers[serviceName] = breaker;
16        }
17

18        return breaker;
19    }
20}
21

22// Usage
23var paymentBreaker = factory.GetCircuitBreaker("PaymentService");
24var inventoryBreaker = factory.GetCircuitBreaker("InventoryService");
25

26// Payment service down doesn't affect inventory calls

Complete Example

csharp
1class ResilientApiClient
2{
3    private readonly HttpClient _client = new();
4    private readonly IAsyncPolicy<HttpResponseMessage> _policy;
5

6    public ResilientApiClient()
7    {
8        // Retry with exponential backoff
9        var retry = Policy<HttpResponseMessage>
10            .Handle<HttpRequestException>()
11            .OrResult(r => r.StatusCode >= HttpStatusCode.InternalServerError)
12            .WaitAndRetryAsync(3, attempt =>
13                TimeSpan.FromSeconds(Math.Pow(2, attempt)));
14

15        // Circuit breaker
16        var breaker = Policy<HttpResponseMessage>
17            .Handle<HttpRequestException>()
18            .OrResult(r => r.StatusCode >= HttpStatusCode.InternalServerError)
19            .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30),
20                onBreak: (result, duration) =>
21                    Console.WriteLine($"Circuit OPEN: {result.Exception?.Message}"),
22                onReset: () =>
23                    Console.WriteLine("Circuit CLOSED"));
24

25        // Combine: circuit breaker wraps retry
26        _policy = Policy.WrapAsync(breaker, retry);
27    }
28

29    public async Task<string> GetAsync(string url)
30    {
31        try
32        {
33            var response = await _policy.ExecuteAsync(() => _client.GetAsync(url));
34            return await response.Content.ReadAsStringAsync();
35        }
36        catch (BrokenCircuitException)
37        {
38            return "Service temporarily unavailable";
39        }
40    }
41}

Key Takeaways

  • Circuit breaker prevents cascading failures
  • Three states: Closed (normal), Open (failing fast), Half-Open (testing)
  • Configure failures threshold and break duration
  • Use AdvancedCircuitBreaker for rate-based breaking
  • Wrap retry inside circuit breaker
  • Use separate breakers for different services
  • Handle BrokenCircuitException for graceful degradation