The Circuit Breaker Pattern in Elixir

By Mitchell Henke on September 24th 2015

When running into temporary interruptions when interfacing with external web APIs, I was reminded of an article from Heroku on the circuit breaker pattern. The particular application is developed on the Phoenix web framework and Elixir. A search did not turn up any circuit breaker implementations in Elixir, but Erlang has, amongst others, fuse.

Fuse has a relatively small public API, consisting of just five functions: ask/2, install/2, melt/1, reset/1, and run/3. It's worth noting that fuse stores the counts in ETS, which means counts will not persist across application restarts or across multiple nodes.

My application relies on iTunes to search for music albums, and just returns the names of the albums, and falls back to searching a static list of albums if iTunes is down. In the future, iTunes results could be stored in Elasticsearch, and that could be the fallback when iTunes goes down.

The first step is to setup and configure the fuse:

# lib/music_searcher.ex
@fuse_name :itunes_music_search

# installs fuse that will blow if there are 2 failures in 10 seconds
# and reset after 60 more seconds
defp install_fuse do
  :fuse.install(@fuse_name, {{:standard, 2, 10000}, {:reset, 60000}})
end

In this case, if iTunes fails twice within 10 seconds, we will back off and use local search for 60 seconds. After the reset interval, the process starts over. The fuse library limits us to this specific strategy, but exponential backoff, and percentage-based fuses are some other, more flexible options for this pattern.

I created a MusicSearcher module that will manage the fuse, and make calls to the iTunes searcher, or the local searcher based on iTunes responses, and the state of the fuse.

Both the iTunes search, and the fallback local search modules implement a search function that accepts a query parameter. The iTunes searcher will respond with a tuple of {:ok, json}, or {:error, error} depending on whether the request was successful.

If the request is successful, the response will be parsed and album names will be returned. If the request fails, we'll call melt/1 to increment the failure count, and then return local search results.

The code for that looks like this:

defmodule CircuitBreaking.MusicSearcher do
  @fuse_name :itunes_music_search
  def search(query) do
    case :fuse.ask(@fuse_name, :sync) do
      :ok ->
        CircuitBreaking.ItunesSearcher.search(query)
        |> parse_itunes_response(query)
      :blown ->
        CircuitBreaking.LocalSearcher.search(query)
    end
  end

  defp parse_itunes_response({:ok, %{"results" => results}}, _query) do
    Enum.map(results, fn(result) ->
      result["collectionName"]
    end)
  end

  defp parse_itunes_response({:error, _}, query) do
    :fuse.melt(@fuse_name)
    CircuitBreaking.LocalSearcher.search(query)
  end
end

With everything set up, I can now use the MusicSearcher and be more confident that at least one service will be searched, even if iTunes is having issues.

I've posted the source on GitHub for those interested, and please reach out on Twitter if you have any questions or comments :)

RokkinCat

is a software engineering agency. We build applications and teach businesses how to use new technologies like machine learning and virtual reality. We write about software, business, and business software culture.