Tuning Hadoop Performance with sysctl.conf

This post delves into optimizing Hadoop performance at the kernel level using sysctl. The sysctl interface provides a way to dynamically modify a running Linux kernel’s parameters. By editing /etc/sysctl.conf, you can persistently configure various networking and system settings, leading to significant performance improvements in your Hadoop cluster. This is especially important for I/O intensive workloads common in Hadoop environments.

Specifically, we’ll cover how to:

• Optimize network settings for high throughput and low latency.
• Adjust virtual memory parameters to minimize swapping.
• Increase file handle limits to support a large number of concurrent operations.
• Harden your system against common network attacks.

What is /etc/sysctl.conf?

The /etc/sysctl.conf file is the primary configuration file for sysctl. Changes made to this file persist across reboots, unlike using the sysctl command directly, which only applies changes to the currently running kernel. After modifying /etc/sysctl.conf, you need to apply the changes using sysctl -p to activate them.

File System

1. fs.file-max: This parameter defines the maximum number of file handles that the kernel can allocate. Hadoop, especially NameNodes and DataNodes, can open a large number of files. Increasing this value can prevent “Too many open files” errors and improve performance. The optimal value depends on your system’s memory and workload.

```bash
[ahmed@server ~]# echo 'fs.file-max = 943718' >> /etc/sysctl.conf
```
**Explanation:** This command appends the line `fs.file-max = 943718` to `/etc/sysctl.conf`. The `echo` command outputs the string, and `>>` redirects the output to append to the file. We are setting the maximum number of file handles to 943718.

Swappiness and Virtual Memory

Excessive swapping can severely degrade Hadoop performance. These settings aim to minimize swap usage and optimize virtual memory.

1. vm.dirty_ratio: This parameter controls the percentage of system memory that can be filled with “dirty” pages (pages that have been modified but not yet written to disk) before the background write process (pdflush or flusher) starts writing them to disk. Reducing this value can make writes more frequent and smaller, potentially reducing latency.
2. vm.swappiness: This parameter controls how aggressively the kernel will swap out memory pages. A lower value means the kernel will try to avoid swapping as much as possible. Setting it to 0 tells the kernel to swap only when absolutely necessary.

```bash
[ahmed@server ~]# echo 'vm.dirty_ratio=10' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'vm.swappiness=0' >> /etc/sysctl.conf
```
**Explanation:** We are setting the `dirty_ratio` to 10% and `swappiness` to 0.  Lowering `swappiness` is crucial for performance-sensitive applications like Hadoop.

Connection Settings

These parameters control the network connection backlog, affecting the system’s ability to handle incoming connections.

1. net.core.netdev_max_backlog: This parameter defines the maximum number of packets that can be queued on a network interface before being processed by the kernel. Increasing this value can help prevent packet loss under heavy load.
2. net.core.somaxconn: This parameter defines the maximum number of completed (established) socket connections that are waiting to be accepted by an application. Increasing this value can prevent connection drops under high connection rates.

```bash
[ahmed@server ~]# echo 'net.core.netdev_max_backlog = 4000' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.core.somaxconn = 4000' >> /etc/sysctl.conf
```
**Explanation:** Increasing these values provides more buffer space for incoming network connections, reducing the likelihood of dropped connections during peak load.

TCP Settings

These TCP settings optimize network performance for Hadoop’s distributed communication.

1. net.ipv4.tcp_sack: Selective Acknowledgments (SACKs) allow the receiver to inform the sender about all segments that have arrived successfully, allowing the sender to retransmit only the missing segments. Disabling SACK can sometimes improve performance in well-connected networks but can hurt performance in lossy networks. Caution: Disabling SACK is generally not recommended unless you have a very stable network environment.
2. net.ipv4.tcp_dsack: TCP Duplicate SACK. Allows TCP to send “duplicate” SACKs. Similar to tcp_sack, disabling this is generally not recommended.
3. net.ipv4.tcp_keepalive_time: Specifies how long the connection must remain idle before TCP starts sending keepalive probes. The default is typically 2 hours (7200 seconds). Reducing this value can help detect dead connections more quickly.
4. net.ipv4.tcp_keepalive_probes: Specifies the number of keepalive probes TCP sends before dropping the connection.
5. net.ipv4.tcp_keepalive_intvl: Specifies the interval between keepalive probes.
6. net.ipv4.tcp_fin_timeout: Specifies how long the kernel waits for a FIN packet from the other end of a connection after it has closed its end. Reducing this value can free up resources more quickly, but setting it too low can lead to connection errors.
7. net.ipv4.tcp_rmem: Defines the minimum, initial, and maximum sizes of the TCP receive buffer for each connection. Increasing these values can improve throughput for high-bandwidth connections.
8. net.ipv4.tcp_wmem: Defines the minimum, initial, and maximum sizes of the TCP send buffer for each connection. Similar to tcp_rmem, increasing these values can improve throughput.
9. net.ipv4.tcp_retries2: This parameter specifies how many times TCP will attempt to retransmit a packet before giving up and killing the connection. Reducing this value can speed up failure detection, but setting it too low can lead to premature connection termination.
10. net.ipv4.tcp_synack_retries: Number of times SYNACKs for a passive TCP connection attempt will be retransmitted.

```bash
[ahmed@server ~]# echo 'net.ipv4.tcp_sack = 0' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_dsack = 0' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_keepalive_time = 600' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_keepalive_probes = 5' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_keepalive_intvl = 15' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_fin_timeout = 30' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_rmem = 32768 436600 4194304' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_wmem = 32768 436600 4194304' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_retries2 = 10' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv4.tcp_synack_retries = 3' >> /etc/sysctl.conf
```

**Explanation:**
*   Disabling `tcp_sack` and `tcp_dsack` might improve performance in very stable networks (use with caution).
*   Reducing `tcp_keepalive_time`, `tcp_keepalive_probes`, and `tcp_keepalive_intvl` allows for quicker detection of dead connections.
*   Reducing `tcp_fin_timeout` frees resources faster.
*   Increasing `tcp_rmem` and `tcp_wmem` improves buffer sizes for faster data transfer.
*   Reducing `tcp_retries2` can speed up failure detection.
*   `tcp_synack_retries` adjust number of retries for SYN-ACK packets.

Disable IPv6 (If Not Used)

If your Hadoop cluster doesn’t use IPv6, disabling it can free up resources and simplify network configuration.

```bash
[ahmed@server ~]# echo 'net.ipv6.conf.all.disable_ipv6 = 1' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv6.conf.default.disable_ipv6 = 1' >> /etc/sysctl.conf
[ahmed@server ~]# echo 'net.ipv6.conf.lo.disable_ipv6 = 1' >> /etc/sysctl.conf
```

**Explanation:** These commands disable IPv6 on all interfaces, the default interface, and the loopback interface.

Applying the Changes

After making changes to /etc/sysctl.conf, you need to apply them to the running kernel.

```bash
[ahmed@server ~]# sysctl -p
```

**Explanation:**  The `sysctl -p` command reads the configuration from `/etc/sysctl.conf` and applies the settings to the running kernel.

Update Limits

The limits.conf file controls resource limits for users and processes. Adjusting these limits is often necessary to allow Hadoop processes to use the resources they need.

```bash
[ahmed@server ~]# echo '* - nofile 65536' >>/etc/security/limits.conf
[ahmed@server ~]# echo '* - nproc 65536' >>/etc/security/limits.conf
```

**Explanation:**
*   `* - nofile 65536`:  This line sets the soft and hard limits for the number of open files (`nofile`) to 65536 for all users (`*`).
*   `* - nproc 65536`:  This line sets the soft and hard limits for the number of processes (`nproc`) to 65536 for all users (`*`).
**Important:**  These changes require a logout/login or a reboot to take effect for user sessions.  Also, ensure pam_limits.so is configured correctly in `/etc/pam.d/common-session*` files.

Important Considerations

• Testing: Always test changes in a non-production environment before applying them to a production Hadoop cluster. Incorrect settings can negatively impact performance or stability.
• Monitoring: Monitor your Hadoop cluster’s performance after making changes to sysctl.conf to ensure that the changes are having the desired effect. Use monitoring tools like Ganglia, Ambari, or Cloudera Manager.
• Hardware: The optimal sysctl settings depend on your hardware configuration, workload, and network environment. There is no one-size-fits-all configuration.

More Details on IPv4

• http://www.cyberciti.biz/files/linux-kernel/Documentation/networking/ip-sysctl.txt
• http://www.cyberciti.biz/files/linux-kernel/Documentation/sysctl/

This post provides a starting point for tuning Hadoop performance with sysctl.conf. Remember to thoroughly test and monitor any changes you make to ensure that they are improving your cluster’s performance. Good luck!