Setting up Highly Available Apache (httpd) with Heartbeat on CentOS/RHEL 6

This guide walks you through configuring a High Availability (HA) Apache web server using Heartbeat on CentOS/RHEL 6. HA ensures your web server remains accessible even if one server fails, providing a more robust and reliable service.

httpd (Apache) is a widely used web server. Heartbeat is a daemon that provides cluster management capabilities, including communication and membership awareness. This allows servers in the cluster to monitor each other and take over services in case of failure.

Prerequisites

• Two CentOS/RHEL 6 servers.
• Basic Linux command-line knowledge.
• Root access to both servers.

Step 1: Network Configuration and Hostnames

Before you begin, ensure that your servers have properly configured hostnames and IP addresses. This is crucial for Heartbeat to function correctly.

• Primary Node: httpd-primary, IP address: 192.168.230.138 (eth0)
• Secondary Node: httpd-slave, IP address: 192.168.230.139 (eth0)
• Virtual IP Address: 192.168.230.150 (This IP will float between the primary and secondary servers)

Edit the /etc/hosts file on both servers to include these entries:

127.0.0.1   localhost
192.168.230.138         httpd-primary
192.168.230.139         httpd-slave

Verify hostnames:

Primary Server (httpd-primary):

[root@httpd-primary ~]# uname -n
httpd-primary

Secondary Server (httpd-slave):

[root@httpd-slave ~]# uname -n
httpd-slave

Configure httpd on both servers (install if needed using yum install httpd).

Step 2: Install Heartbeat and EPEL Repository

The EPEL (Extra Packages for Enterprise Linux) repository provides additional packages not found in the default CentOS/RHEL repositories.

[root@httpd-primary ~]# wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@httpd-primary ~]# rpm -ivh epel-release-6-8.noarch.rpm

Important: If you encounter errors during yum install heartbeat related to HTTPS, edit /etc/yum.repos.d/epel.repo and change https to http in the baseurl and mirrorlist lines. This is sometimes necessary due to certificate issues.

[root@httpd-primary ~]# vim /etc/yum.repos.d/epel.repo

Then, install Heartbeat:

[root@httpd-primary ~]# yum install heartbeat

Repeat these steps on httpd-slave.

Step 3: Configure Heartbeat

Copy the configuration files from the Heartbeat documentation directory to the /etc/ha.d/ directory:

[root@httpd-primary ~]# cp /usr/share/doc/heartbeat-3.0.4/authkeys /etc/ha.d/
[root@httpd-primary ~]# cp /usr/share/doc/heartbeat-3.0.4/ha.cf /etc/ha.d/
[root@httpd-primary ~]# cp /usr/share/doc/heartbeat-3.0.4/haresources /etc/ha.d/

Adjust the version number (3.0.4) if necessary, based on your installed Heartbeat version. You can find the correct directory by listing the contents of /usr/share/doc/.

Step 4: Configure Authentication (authkeys)

Edit the /etc/ha.d/authkeys file to configure authentication between the Heartbeat nodes:

[root@httpd-primary ~]# vi /etc/ha.d/authkeys

Add these lines:

auth 2
2 sha1 anyRandomKeyHere

Replace anyRandomKeyHere with a strong, randomly generated key. This key must be identical on both servers.

Step 5: Secure the Authentication Key File

Set the correct permissions on the authkeys file to restrict access:

chmod 600 /etc/ha.d/authkeys

Step 6: Configure Heartbeat Configuration File (ha.cf)

This is the most important configuration file for Heartbeat.

vi /etc/ha.d/ha.cf

Add/modify the following lines:

logfile /var/log/ha-log
logfacility local0
keepalive 2         # How often (seconds) to send heartbeat messages
deadtime 30          # How long (seconds) before declaring a node dead
initdead 120         # How long (seconds) to wait at startup before declaring a node dead
bcast eth0           # Interface to use for broadcasting heartbeat messages
udpport 694          # UDP port to use for heartbeat communication
auto_failback on      # Automatically fail back to the primary node when it recovers
node httpd-primary   # Name of the primary node
node httpd-slave     # Name of the secondary node

Explanation of key parameters:

• keepalive: The interval (in seconds) at which heartbeat messages are sent.
• deadtime: The amount of time (in seconds) a node can be unresponsive before it’s declared dead. This should be significantly larger than keepalive.
• initdead: The deadtime value used during the initial startup of Heartbeat.
• bcast: The network interface used for broadcasting heartbeat messages. Adjust this if your primary network interface is different from eth0.
• udpport: The UDP port used for Heartbeat communication. Ensure this port is not blocked by firewalls.
• auto_failback: If set to on, Heartbeat will automatically switch back to the primary node when it recovers.
• node: Lists the hostnames of the nodes in the cluster. Ensure these match the output of uname -n on each server.

Step 7: Configure Resource Management (haresources)

The haresources file defines the resources that Heartbeat will manage, including the virtual IP address and the httpd service.

vi /etc/ha.d/haresources

Add the following line:

httpd-primary 192.168.230.150 httpd

Explanation:

• httpd-primary: The primary node where the resources will initially run.
• 192.168.230.150: The virtual IP address that will be assigned to the active server.
• httpd: The service that Heartbeat will manage. This assumes the httpd service is named httpd.

Step 8: Copy Configuration to the Secondary Server

Copy the entire /etc/ha.d/ directory from the primary server to the secondary server:

scp -r /etc/ha.d/ root@httpd-slave:/etc/

This ensures that both servers have identical Heartbeat configurations.

Step 9: Configure Apache (httpd)

Configure Apache to listen on the virtual IP address.

vi /etc/httpd/conf/httpd.conf

Add or modify the Listen directive:

Listen 192.168.230.150:80

This tells Apache to only listen for connections on the virtual IP address and port 80.

Step 10: Copy Apache Configuration to the Secondary Server

Copy the modified httpd.conf file to the secondary server:

scp /etc/httpd/conf/httpd.conf root@httpd-slave:/etc/httpd/conf/

Step 11: Create Test Web Pages

Create a simple index.html file on both servers to differentiate them:

On httpd-primary:

echo "`uname -n` apache test server" > /var/www/html/index.html

On httpd-slave:

echo "`uname -n` apache test server" > /var/www/html/index.html

Step 12: Start Heartbeat

Start the Heartbeat service on both the primary and secondary servers:

/etc/init.d/heartbeat start

Step 13: Test the Configuration

Open a web browser and access the virtual IP address:

http://192.168.230.150

You should see the test page from httpd-primary: “httpd-primary apache test server”.

Step 14: Simulate a Failover

Stop the Heartbeat service on the primary server to simulate a failure:

/etc/init.d/heartbeat stop

Wait a few seconds (up to the deadtime configured in ha.cf). Then, refresh your browser. You should now see the test page from httpd-slave: “httpd-slave apache test server”. This confirms that the failover is working correctly.

Important Considerations:

• Firewall (iptables): Ensure that your firewall is not blocking Heartbeat communication (UDP port 694) or HTTP traffic (TCP port 80). The original document included iptables rules. However, for simplicity and testing, it is often easiest to temporarily disable iptables:

  service iptables stop
  chkconfig iptables off  # To prevent it from starting on boot
  

  For production environments, DO NOT disable your firewall. Instead, carefully configure the necessary iptables rules. Here are the entries required for iptables:

  iptables -A OUTPUT -o lo -p udp -m udp -d 192.168.230.138 --dport 694 -j ACCEPT
  iptables -A INPUT -i lo -p udp -m udp -d 192.168.230.138 --dport 694 -j ACCEPT
  iptables -A OUTPUT -o eth0 -p udp -m udp -d 192.168.230.138 --dport 694 -j ACCEPT
  iptables -A INPUT -i eth0 -p udp -m udp -d 192.168.230.138 --dport 694 -j ACCEPT
  
  iptables -A OUTPUT -o lo -p udp -m udp -d 192.168.230.139 --dport 694 -j ACCEPT
  iptables -A INPUT -i lo -p udp -m udp -d 192.168.230.139 --dport 694 -j ACCEPT
  iptables -A OUTPUT -o eth0 -p udp -m udp -d 192.168.230.139 --dport 694 -j ACCEPT
  iptables -A INPUT -i eth0 -p udp -m udp -d 192.168.230.139 --dport 694 -j ACCEPT
  
  iptables -A OUTPUT -o eth0 -p tcp -m tcp -d 192.168.230.138 --dport 80 -j ACCEPT
  iptables -A INPUT -i eth0 -p tcp -m tcp -d 192.168.230.138 --dport 80 -j ACCEPT
  
  iptables -A OUTPUT -o eth0 -p tcp -m tcp -d 192.168.230.139 --dport 80 -j ACCEPT
  iptables -A INPUT -i eth0 -p tcp -m tcp -d 192.168.230.139 --dport 80 -j ACCEPT
  
  service iptables save
  

• SELinux: SELinux can also interfere with Heartbeat. You may need to configure SELinux policies to allow Heartbeat to manage network interfaces and services. For testing, you can temporarily disable SELinux:

  setenforce 0
  

  To make the change permanent, edit /etc/selinux/config and set SELINUX=disabled. However, disabling SELinux is generally not recommended for production environments. Instead, create custom SELinux policies to allow the necessary access.

• Testing: Thoroughly test your HA configuration to ensure that failover occurs correctly under various scenarios. Simulate network outages, server crashes, and service failures.

• Resource Monitoring: Heartbeat can monitor other resources besides httpd, such as databases and custom scripts. Refer to the Heartbeat documentation for more information.

Useful Links:

• Heartbeat
• Heartbeat Clustering
• How To Forge
• Blog Tuanta
• How to Geek
• Cyber Citi
• Linux HA

This guide provides a basic configuration for HA Apache using Heartbeat. Adapt these steps to your specific environment and requirements. Remember to consult the official Heartbeat documentation for more advanced configuration options.