"Possible SYN Flooding" messages in system logs
10 February 2016 09:50 AM
Some customers have reported seeing kernel level messages like this in their /var/log/messages file:
This may also be seen as part of the output from a call to dmesg and could possibly follow a stack trace, for example:
What does it mean?
The tcp_syncookies configuration is likely enabled on your system. You can check for this by viewing the contents of /proc/sys/net/ipv4/tcp_syncookies
If the value returned is 1 (as per the example above), then tcp_syncookies are enabled for this host
Possible SYN flooding
A SYN flood is a form of denial-of-service attack in which an attacker sends a succession of SYN requests to a target's system in an attempt to consume enough server resources to make the system unresponsive to legitimate traffic.
Source: Wikipedia https://en.wikipedia.org/wiki/SYN_flood
You would expect to see evidence of a SYN flood when a "flood" of TCP SYN messages are sent to the host. Under normal operation, your kernel should acknowledge these incoming SYNs with a SYN-ACK, are not followed by ACK messages from the client. The process (or pattern) described above is known as Three Way Handshaking. The goal of this is to firmly establish communication on both the server and the client.
In the event of a real attack, a SYN flood will most likely originate from a fake IP address; during an attack, the client performing the "flood" is not waiting for the SYN-ACK response back from the server it is attacking.
Under normal operation (i.e. without SYN cookies), TCP connections will be kept half-open after receiving the first SYN because of the handshake mechanism used to establish TCP connections. Due to the fact that there is a limit to how many half open connections that the kernel can maintain at any given time, this is where the problem becomes characterised as an attack.
The term half-open refers to TCP connections whose state is out of synchronization between the two communicating hosts, possibly due to a crash of one side. A connection which is in the process of being established is also known as embryonic connection.
Source: Wikipedia https://en.wikipedia.org/wiki/TCP_half-open
If SYN cookies are enabled, then the kernel doesn't track half-open connections. Instead it relies on the sequence number in the following ACK datagram that the ACK follows a SYN and a SYN-ACK which establishes full communication between client and server. By ignoring half-open connections, SYN floods are no longer a problem.
In the case of MarkLogic, this message can appear if the rate of incoming messages is perceived to the kernel as being unusally high. In this case, this would not be indicative of a real SYN flooding attack, but to the TCP/IP stack it looks like it exhibits the same characteristics and the kernel responds by reporting a possible (fake) attack.
Notes from the kernel documentation
See the section of the kernel documentation for tcp_syncookies - BOOLEAN for some further information regarding this feature:
The syncookies feature attempts to protect a socket from a SYN flood attack. This should be used as a last resort, if at all. This is a violation of the TCP protocol, and conflicts with other areas of TCP such as TCP extensions. It can cause problems for clients and relays. It is not recommended as a tuning mechanism for heavily loaded servers to help with overloaded or misconfigured conditions. For recommended alternatives see tcp_max_syn_backlog, tcp_synack_retries, and tcp_abort_on_overflow.
Further down, they state:
Note, that syncookies is fallback facility. It MUST NOT be used to help highly loaded servers to stand against legal connection rate. If you see SYN flood warnings in your logs, but investigation shows that they occur because of overload with legal connections, you should tune another parameters until this warning disappear. See: tcp_max_syn_backlog, tcp_synack_retries, tcp_abort_on_overflow.
Tuning on a MarkLogic Server
Any dmesg output indicating "possible SYN flooding on port 7999" may appear in tandem with very heavy XDQP (TCP) traffic within a MarkLogic cluster - this link provides further detail in relation to a similar scenario with Apache HTTP server. You can tune your TCP settings to try to avoid
If a MarkLogic Server instance sees SYN flooding message on a system that is otherwise healthy and the messages occur because of normal and expected marklogic server communications, you may want to increase the backlog (
How to disable SYN cookies
You can disable syncookies by adding the following line to /etc/sysctl.conf:
Also note that the new setting will take only effect after a host reboot.