Configuration of haproxy is not really in the context of this article, but some points shown below need special care, a full configuration example is available here.
Setting up the DataplaneAPI
The idea is to have as little configuration redundancy as possible, so in order for exabgp to publish services IP we will need to extract them one way or another. The solution that caught my eye was the newly open-sourced DataPlaneAPI module for haproxy (originaly limited to HAProxy Enterprise). This tool handle configuration parsing and publishing over a REST service.
Once the binary stored on you server, you’ll need to add two configurations blocks in your haproxy.cfg, one /userlist/ for controlling access to the API and one /program/ for starting the module:
Setting up frontends with monitoring
You’ll also need to publish an healthcheck url on your frontends, like that
There is a little catch in this setup, service IP won’t be dynamically added to one of the network interface, so you’ll need to add thoses IP(s) to the loopback interface. I real life scenario the full setup is deployed by a confmgmt service (opscode-chef) but you might want to add the addresses in the network scripts for them to be up on boot.
Setting up HA/exabgp
Now that the service is avaible, time to make it highly !
For this part we will be using Exabgp 4.0+, I used debian/buster package without issues, your mileage may vary.
The idea is to make BGP session with your core BGP platform to publish /32 routes for each frontend, for this you’ll need BGP-aware router (duh) and a BGP-aware speaker on your server (this will be exabgp’s role).
PS: This is more of a manual optimized configuration, when deploying with configuration management I ended with a “flattened” configuration.
Configuration BGP Router
This part will highly depend on your network’s technology, you’ll need to setup eBGP links with your LB hosts, I also added a few safeguard with an empty export policy (exabgp doesn’t need info from the routing in my configuration) and an import policy that allow only /32 routes over my network:
The default systemd unit for exabgp/debian is a bit limited so I added some override to create the fifo used to interrogate the daemon from CLI.
Last but not least we will need an agent for exabgp, he will extract the services IPs from the dataplaneapi and do some healthcheck before publishing an IP.
The agent will query haproxy dataplane to extract service IPs, and then tests them using the argument provided check, if everything is ok then a route will be sent over the bgp routers using exabgp.
This agent is a modified version of exabgp healthcheck.py, more for the fun of it since the original IP discovery using loopback listing could very well do the trick, however there is another modification, the healthcheck command is applied to each service IP and not once globally, so that we only advertise really enabled services.
Starting the monster
Time to start the lot, first haproxy and then exabgp: