Proper Load Balancing and Failover Management with HAProxy

Published on 14 Sep 2021

Best Practices

Load balancing and failover management are among the first topics to be explored in the process of scaling an application up or out as it grows in popularity or importance.

Applications inevitably have multiple layers of potential failure modes. Managing potential failure modes effectively is critical to application availability and providing a positive user experience. A wide variety of open source and proprietary tools exist to fill these roles, from strategies as simple as DNS round-robin to more elaborate HA configurations with floating IP addresses capable of moving between active components.

HAProxy and Load Balancing

HAProxy has become a standard tool in open source toolboxes for managing various load balancing and failover scenarios. It combines substantial power and flexibility with a reasonably straightforward setup and initial configuration. These traits allow it to fill a role in environments with a wide range of complexities and needs. This easy-to-bootstrap configuration allows it to be quickly deployed and simply configured with a proof of concept, then grow in capability as application and business requirements change.

Load Balancing Scenario

A basic load balancing scenario for a web-based application often starts with a very simple frontend and backend configuration in HAProxy.

frontend my_app_fe
     	bind *:80
	mode http
	default_backend my_app_be

backend my_app_be
	balance roundrobin
	option tcp-check
	server node1 10.1.1.10:8080 check
	server node2 10.1.1.11:8080 check

This configuration sets up HAProxy to listen on port 80 and proxy incoming traffic to a pair of backend servers. A basic health check verifies that HAProxy can connect to the backend port with a simple TCP check. TCP checks are quick and very lightweight in this context, imposing basically no load on the backend server. Still, they don’t allow HAProxy to determine whether the backend is healthy or not, only whether the port is open.

As the backend application evolves, the HAProxy configuration can add features to keep pace. Perhaps a health status URI has been added to the backend application. This URI is used by HAProxy to determine application health in a more robust way than simply connecting to the port.

backend my_app_be
	balance roundrobin
	option httpchk HEAD /health
	server node1 10.1.1.10:8080 check
	server node2 10.1.1.11:8080 check

This configuration requests the /health URI from each backend server and keeps the backend server traffic only if that request returns a 200 from the backend.

There are additional configuration options both for the health check itself and for the backend servers. Such options allow the behavior of the health status monitoring to be further configured. For example, allowing users to control the interval to mark backends up or down, intervals at which the health check run, options to the health check itself, timeouts, and many other tuneables.

Need for Fallback

What if all application backends are unhealthy? In this situation, it may be best to fall through to another set of backend servers so users are not confronted with an unfriendly server error message.

backend my_app_be
	balance roundrobin
	option httpchk HEAD /health
	server node1 10.1.1.10:8080 check
	server node2 10.1.1.11:8080 check
	server backup1 10.1.1.100:8080 check backup

A backup server can be defined to take over in the case of all other backends failing, allowing it to serve a friendly error message, a scaled-down version of the application or site, or potentially even an offsite status or maintenance page.

The behavior of the backup server can be further configured as well. Then configure a load shedding backend to take traffic when only most of the primary backends are down or overloaded rather than when all backends are down. This setup allows an application stack to degrade gracefully under load or other modes.

Managing Failover

HAProxy can load balance and manage failover in other parts of the application stack as well as its most common use case of HTTP servers. The TCP balancing mode allows HAProxy to manage connections to arbitrary services, such as FTP.

backend ftp_be
	mode tcp
	balance roundrobin
	option tcp-check expect ProFTPD\ Server
	server ftp1 10.1.1.10:21 check
	server ftp2 10.1.1.11:21 check

This allows HAProxy to provide robustness in layers of the application stack and with backend server applications that have traditionally been difficult to configure for high availability. In this situation, the TCP check can be configured to expect a banner when connecting to the backend server, allowing HAProxy to determine health status more accurately even when connecting to arbitrary server backends with varying protocols.

This mode can also be used to balance database connections across a set of active databases.

Database Health Checks

Health checks for a database require that a user be configured in the database to allow the proxy to connect with a purpose-specific user. Simple TCP checks may miss cases where the database server is listening but not responding properly. So, the more robust case where a user is configured to log in to the database is often preferable given the importance of the database in the application stack.

backend mysql_be
	mode tcp
	option mysql-check user haproxy-health
	balance roundrobin
	server mysql1 10.1.1.10:3306 check
	server mysql2 10.1.1.11:3306 check

Here, we assume that the HAProxy-health user exists in the database and is able to log in, preferably with no permissions beyond that. The health check provides a simple mechanism for database failover, which is otherwise quite difficult to achieve.

HAProxy fills a valuable, flexible role in the application stack, enabling complex load balancing and failover scenarios that evolve alongside an application.

Looking for an FTP service with all the security features to keep your data flowing?

Find out more about ExaVault’s world-class security, or start your free trial today!