How to fix Eureka taking too long to deregister instances

If you use Ribbon and Eureka in your Spring Boot application, you’ll notice that the default configuration is not optimal. Eureka takes too long time to notice that a service went down unexpectedly and in the meantime, your load balancer, Ribbon, will try to connect to the dead one. On the other hand, the official Eureka documentation discourages changing the leaseRenewalIntervalInSeconds parameter so, what can we do here? This post answers that question.

The problem

Eureka is a great tool for Service Discovery, and it’s very well integrated with Spring Boot. We can create a Service Registry by just adding some dependencies and an annotation, and we can connect our clients to register on it with minimal configuration too. It comes with Ribbon as Load Balancer, so we can get everything working in few minutes. Our services will ask the registry for the instances of a given service and will decide by themselves, using Ribbon, to which one they’re connecting.

The problem arises when one of our multiple instances of a service goes down unexpectedly or lose connectivity, not having time to tell Eureka. The service registry is based on leases, and every client should renew itself every N seconds. When the lease expires, Eureka takes also some time to decide that the instance is no longer valid. It’s not a very straightforward mechanism, as you’ll see explained on different Internet threads. With the default configuration, my experience is that it can take up to four or five minutes to deregister a dead instance.

On the other hand, the official documentation tells us that we shouldn’t change this configuration. My guess is that, since the mechanism to deregister is not easy to understand, you can mess up the entire configuration if you make mistakes.

How to solve it

The approach to solving this problem is based on Ribbon, and not Eureka. We can leave the Service Registry alone and let it work as usual, with the optimal configuration, but we can make our clients smarter and let them find out which services are really healthy.

Spring Boot configures Ribbon by default with a Round-Robin strategy for load balancing. It also sets the status-check mechanism to none (NoOpPing), which means that the load balancer will not verify if the services are still alive. It makes sense since it should be our Service Registry, Eureka, the one that registers and deregisters instances. But we just concluded that the time it will take it’s not good for us.

We can use the Ribbon functionality to ping services and apply load balancing depending on the result. To get that working, we need to configure two Spring beans: an IPing to establish the check-status mechanism and an IRule to change the default load balancing strategy.

  • The PingUrl implementation will check if services are alive. We want to change the default URL and point it to /health since we want to avoid requests to unmapped root contexts. The false flag is just to indicate that the endpoint is not secured.
  • The AvailabilityFilteringRule is an alternative to the default RoundRobinRule that takes into account the availability being checked by our new pings.
  • One thing that’s very important to note (since it’s tricky) is that this class is not annotated with @Configuration. It’s injected in a different way: we need to reference it from a new annotation added to the main application class: @RibbonClients.

If we test now again the scenario where multiple instances are registered, and then one of them goes down, we’ll notice that the reaction time to find out an unavailable service is much less.

You can find some other options for load balancing strategy on the official repository. There are implementations that allow us to balance load depending on response time, geographical affinity, etc. The best idea is to design your plan, test it putting some load into your system and then adjust based on the results.

Code samples

These code samples are taken from one of the versions of the project that is developed along the Microservices – The Practical Way book.

You can find them on GitHub. In that example the client is actually a Zuul gateway, which acts as load balancer for the services registered in Eureka. You can run the entire system and then put into practices the different strategies. You can also comment the line with the @RibbonClients annotation, and then you’ll verify how long your routing will be failing because of Eureka’s default way of deregistering instances.

I hope you find it useful! Please let me know any question or suggestion you may have by using the comments.

Related Posts

@RestController example with Spring Boot and Swagg... In this article I'll explain how to set up a basic @RestController  in a Spring Boot application, using both @GetMapping and @PostMapping annotations....
How to deploy a Spring Boot WAR in Wildfly / JBoss This is a short guide on how to deploy a war packaged Spring Boot application in Wildfly. As you know Wildfly is the new name for JBoss AS since 8.0 v...

Leave a Reply