When running into temporary interruptions when interfacing with external web APIs, I was reminded of an article from Heroku on the circuit breaker pattern. The particular application is developed on the Phoenix web framework and Elixir. A search did not turn up any circuit breaker implementations in Elixir, but Erlang has, amongst others, fuse.

Fuse has a relatively small public API, consisting of just five functions: ask/2, install/2, melt/1, reset/1, and run/3. It’s worth noting that fuse stores the counts in ETS, which means counts will not persist across application restarts or across multiple nodes.

My application relies on iTunes to search for music albums, and just returns the names of the albums, and falls back to searching a static list of albums if iTunes is down. In the future, iTunes results could be stored in Elasticsearch, and that could be the fallback when iTunes goes down.

The first step is to setup and configure the fuse:

In this case, if iTunes fails twice within 10 seconds, we will back off and use local search for 60 seconds. After the reset interval, the process starts over. The fuse library limits us to this specific strategy, but exponential backoff, and percentage-based fuses are some other, more flexible options for this pattern.

I created a MusicSearcher module that will manage the fuse, and make calls to the iTunes searcher, or the local searcher based on iTunes responses, and the state of the fuse.

Both the iTunes search, and the fallback local search modules implement a search function that accepts a query parameter. The iTunes searcher will respond with a tuple of {:ok, json}, or {:error, error} depending on whether the request was successful.

If the request is successful, the response will be parsed and album names will be returned. If the request fails, we’ll call melt/1 to increment the failure count, and then return local search results.

The code for that looks like this:

With everything set up, I can now use the MusicSearcher and be more confident that at least one service will be searched, even if iTunes is having issues.

I’ve posted the source on GitHub for those interested, and please reach out on Twitter if you have any questions or comments 🙂