Load balancing syslog messages into FortiSIEM using HAProxy, syslog-ng or nftables

It is a weakness of some products, such as FortiGate firewalls, that they can only send traffic to a single syslog server. If you’re also ingesting those logs into a system like FortiSIEM, you can run into an issue where a single FortiSIEM worker is overwhelmed with the syslog traffic coming from that one firewall. To spread the load over multiple FortiSIEM workers, one can simply load-balance across the multiple workers! Here are a few approaches and their pros and cons.

Round-robin DNS

The first option most people think of is to have a DNS record with multiple IPs or CNAMEs in it. This is how a lot of services do load balancing, like your favorite websites. This works well when you have a lot of medium to low traffic syslog clients, like end-user devices or Linux servers, as each of them will randomly select a worker and send it messages for the length of the TTL. However, in the case where we have a single high-traffic device, this will result in one of the workers being overloaded until the TTL expires, at which point a new worker is chosen and becomes overloaded.

Verdict: ❌

syslog-ng

If we want to load balance syslog, surely we can use a dedicated syslog server running syslog software that sends syslog messages! Perhaps, syslog-ng? The configuration is relatively simple for forwarding:

source s_udp { udp(port(514)); };

destination d_lb_udp {
        channel {
                channel {
                        filter {
                                "0" == "$(% ${R_MSEC} 2)"
                        };
                        destination {
                                syslog("192.0.2.10" transport("udp") port(514));
                        };
                        flags(final);
                };
                channel {
                        filter {
                                "1" == "$(% ${R_MSEC} 2)"
                        };
                        destination {
                                syslog("192.0.2.11" transport("udp") port(514));
                        };
                        flags(final);
                };
        };
};

log { source(s_udp); destination(d_lb_udp); };

This can easily be extended for TCP too.

When using this to send messages to FortiSIEM, two problems become immediately obvious.

• Incorrect source IP

Because the server is relaying the messages, the new source IP is that of the syslog-ng server. There does seem to be some mechanism in FortiSIEM to parse out the sending device and set the source IP to that, but it is not documented at all that I can tell, so in my testing, half of the devices have the correct source IP, and half have the relaying IP there instead.

Luckily when it comes to the Source IP, we have the option of enabling the spoof-source(yes) option. This will work as long as we’re only forwarding UDP messages at least.

• Message reformatting

Messages are reformatted into RFC 5424. This can be changed to reformat into RFC 3164 instead, but there is no way to avoid reformatting at all. As a result, FortiSIEM is unable to use the existing parsers, as these expect things to be in a specific format. This can be worked around, but it makes syslog-ng not a drop-in replacement.

This combines to result in an acceptable but sub-optimal solution. As long as you can strictly control the format of messages going to syslog-ng and you only want to relay UDP messages, this can work.

Verdict: ⚠️

HAProxy

Searching further, we see that HAProxy is able to forward syslog-ng messages! And better yet, HAProxy is a purpose-built load balancer, so it should be designed for just this kind of thing.

Once again, the configuration is relatively simple:

log-forward syslog
        bind 0.0.0.0:514        # TCP
        dgram-bind 0.0.0.0:514  # UDP

        # Send logs over UDP - Sampling is used to load-balance
        log 192.0.2.10 sample 1:2 local0
        log 192.0.2.11 sample 2:2 local0

This supports everything we need without anything strange out of the box and solves the previous issue with syslog-ng by not doing any rewriting by default. However, we no longer have the option of spoofing the source IP address. This means we’re stuck again with FortiSIEM not knowing the original device that sent the syslog messages.

Verdict: ⚠️

nftables (or iptables)

None of the solutions so far tick all our boxes. We can either choose haproxy and lose the source IP spoofing, or syslog-ng and lose the raw message forwarding. If we want to solve both of these issues (and we don’t mind only supporting UDP), how about we rewrite the UDP packets and spoof the source IP? While we could get something working with haproxy, the easiest thing is to remove haproxy and syslog-ng altogether and use Linux’s built-in IP forwarding functionality with nftables.

There are only two steps to get this working. The first is to enable forwarding, and the second is to add the relevant iptables rules:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -p udp --dport 514 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination 192.0.2.10:514
iptables -t nat -A PREROUTING -p udp --dport 514 -m statistic --mode nth --every 2 --packet 1 -j DNAT --to-destination 192.0.2.11:514

If you’d like to use nftables instead, you can use a similar configuration:

$ nft list ruleset
table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                udp dport 514 dnat to numgen inc mod 2 map { 0 : 192.0.2.10, 1 : 192.0.2.11 }
        }
}

With a few steps to make this configuration permanent, we now have a server that is forwarding all our syslog messages with a spoofed source and distributing it between all our workers.

Verdict: ✔️

FortiSIEM Collectors

As a final note, FortiSIEM has a node type called a “collector” which seems like it would do exactly what we want with minimal fuss. Unfortunately, these are documented to have a limit of 10k EPS (Each Collector has a maximum log ingestion rate of up to 10k EPS), making it not an option for high-traffic sources. The official recommendation is to use a load balancer in front of a Collector, making them a lot less useful than they might otherwise be. To compare performance, a Linux server running nftables should be able to handle at least 100k EPS without any problems. With 20k EPS, we haven’t seen more than 0.10 load average on a 4 core machine.

Verdict: ❌❌❌

Conclusion

None of these options are really great. A better solution would be to understand how FortiSIEM determines whether to ignore the source IP on syslog messages and ensure that all forwarded messages trigger this behavior. In the absence of this, forwarding UDP messages with nftables is likely to give us the best performance and simplicity (though it can be a little difficult for less technical users to understand if they are not familiar with nftables/iptables).

Gr33tz to d@ f1nch and #realhacking