NGINX proxy Is Not For Docker, Use Envoy Instead
Nginx has long been a popular and easy-to-go for routing an application between multiple Docker containers, like managing communication between a server-side and client-side, or routing between microservices.
Docker provides a way to restart the container automatically once anything wrong happens to the container, making the container recover itself without human friction, but that causes another problem if you’ve been using Nginx as a reverse proxy. By default, Docker may assign a new IP address to the restarted container even if it is recreated with the same name, the underlying network setup considers it a new entity under the internal built-in Docker DNS service.
It's recommended that communication between containers should use hostnames. This ensures if a container's private IP address changes, Docker's internal DNS will resolve to the new IP address. However, this is not the case when using Nginx. By default, Nginx's proxy_pass directive resolves the DNS target IP address only once when the server starts. If an upstream container becomes unreachable, Nginx retains the cached IP address in its memory.
Consequently, if an upstream container restarts and is assigned a new IP address by Docker, Nginx will not automatically update its cached IP address, the new restarted container will be unreachable until restarting the Nginx server to resolve new upstream IPs.
The service_2 container has been restarted and a new IP address is assigned to it, the DNS resolve table in the Nginx still keeps the old resolved IPs of the two hostnames, and service_2 becomes not reachable until the Nginx restart.
This issue made us look for an alternative server with better DNS resolution capabilities. We came up with Envoy, which offers various DNS recovery mechanisms. STRICT_DNS mode resolves specified DNS targets continuously and asynchronously. eventually, We had to migrate Bigcapital proxy to Envoy to close the issue of some services becoming unreachable that many users were complaining about.
Resources: