Kubernetes doesn’t offer an implemention for Services of type Loadbalancer out of the box for baremetal clusters 1. This is where MetalLB comes into the picture, it adds what kubernetes is missing here. MetalLB has two modes of operating, in layer 2 mode using ARP requests or with BGP. The BGP mode is needed for ’true’ load balancing 2.
So this is why one might want to use MetalLB, but for proper BGP functionality you of course also need a BGP router to do the actual balancing. For this part of the setup I am using the free opensource router software OPNsense together with its FRR (Free Range Routing) package to provide the BGP functionality.
Assuming you already have OPNsense set up just add the FRR Package as described in the OPNsense docs.
After this is done you can go to
Routing > General and enable the routing deamon.
BGP General Settings
Now navigate to the BGP general settings (
Routing > BGP) and enable this as well. You should also choose an AS number and enter the network mask here.
I’d recommend choosing an ASN that falls into the private range as described by RFC6996,
that is a number between 64512 and 65534. You also need to set the routing distribution here, I’ve set it connected and kernel routes.
BGP Prefix List config
By default FRR won’t permit any route announcements so we need to add a rule allowing it from the peers we’ll define later. To do so head to the “Prefix List” tab and add a rule with action permit to network any.
BGP Neighbour config
Now that we have configured a prefix list we can add some neigbours that will use that rule.
Add neigbours with as
Peer-IP the IP of that kubernetes worker node, a Remote ASN that is different from OPNsense’s and the
previously made prefix list for in and output.
Repeat this for as many kubernetes nodes you have keeping the ASN the same and changing the Peer-IP. Also make sure all neighbours show up as enabled.
This is all we need to setup on the router’s side.
For the MetalLB mainly just follow the official documentation.
For the ConfigMap you’ll want something like this
peer-addressbeing the IP of the router
peer-asnbeing the ASN of the router you configured
my-asnthe ASN you configured for the neighbours.
addressesbeing the range(s) MetalLB is allowed to use to allocate IPs from.
Now that everything should be configured we can test this out by making a test deployment like so:
After a few moments you should be able to get the provisioned IP like so:
And when navigating to this IP you should see the familiar nginx welcome screen
Congratulations! You now have OPNsense running FRR connected to MetalLB running in Kubernetes to dynamically provision IP addreses and loadbalance services 🎉.
Some resources I’ve used to compile this post which may be useful to you