AHMED ZBYR

AHMED ZBYR

Biker, EV Enthusiast

Deploying an enterprise Apache Hadoop (CDH) cluster across dozens of hosts requires consistent OS-level tuning across every node (kernel parameters, Transparent Huge Pages, file descriptor limits, /etc/hosts, database connectors, and service accounts) before installing Cloudera Manager Server and its backing relational databases.

This guide documents the cdhmgr-chef-setup Chef cookbook, which automates host preparation and Cloudera Manager installation on CentOS/RHEL 6 so clusters across development, test, pre-production, and production environments can be provisioned reproducibly via Chef, Python, and the Cloudera Manager API.

Historical Note: Cloudera CDH 5 and CentOS 6 have both reached End of Life and were succeeded by Cloudera Data Platform (CDP Private Cloud Base). The two-phase host preparation and management-server architecture described below remains a useful reference for legacy CDH maintenance and automated cluster bootstrapping.

Architecture Overview: Two-Phase Deployment

The cookbook divides cluster deployment into two phases based on node roles:

  1. Phase 1 — Node Pre-Configuration (cdhmgr-chef-setup::pre_config): Applied to all nodes in the cluster (Edge, Master, Worker/DataNode, and Admin/Cloudera Manager hosts). Configures OS prerequisites required by Cloudera Manager and CDH.
  2. Phase 2 — Cloudera Manager Server Installation (cdhmgr-chef-setup::cloudera_manager): Applied only to the Admin (Cloudera Manager) node. Installs and configures the relational database (MySQL or PostgreSQL, defaulting to MySQL), provisions schemas for Cloudera services, installs Cloudera Manager Server/Daemons, and installs the Python cm_api client library.
Node Type Assigned Chef Run List
Edge Nodes recipe[cdhmgr-chef-setup::pre_config]
Master Nodes (NameNode, ResourceManager, etc.) recipe[cdhmgr-chef-setup::pre_config]
Worker / DataNodes recipe[cdhmgr-chef-setup::pre_config]
Admin Node (Cloudera Manager Server) recipe[cdhmgr-chef-setup::pre_config], recipe[cdhmgr-chef-setup::cloudera_manager]

Phase 1: Preparing Cluster Nodes (pre_config)

Including cdhmgr-chef-setup::pre_config orchestrates the following sub-recipes on every cluster node:

  1. SELinux configuration (pre_config_selinux_update): Sets SELinux to permissive mode during initial cluster setup: 
    include_recipe 'cdhmgr-chef-setup::pre_config_selinux_update'
  2. Host firewall rules (pre_config_iptables_setup): Stops and disables iptables for internal cluster communication (in production, ensure perimeter network segmentation or explicit port rules are enforced): 
    include_recipe 'cdhmgr-chef-setup::pre_config_iptables_setup'
  3. Kerberos & SSSD integration (pre_config_security_setup): Optional recipe (commented out by default until Active Directory/KDC is available) to configure /etc/krb5.conf and /etc/sssd/sssd.conf: 
    # include_recipe 'cdhmgr-chef-setup::pre_config_security_setup'
  4. Administrative accounts & sudo (pre_config_user_setup): Creates the cluster administration account (sysadmin / cmadmin) and configures passwordless sudo access for Cloudera Manager agent orchestration: 
    include_recipe 'cdhmgr-chef-setup::pre_config_user_setup'
  5. Kernel sysctl tuning (pre_config_sysctl_config): Reduces vm.swappiness (e.g., vm.swappiness = 1) so Hadoop JVM heaps are not aggressively swapped to disk: 
    include_recipe 'cdhmgr-chef-setup::pre_config_sysctl_config'
  6. Transparent Huge Pages (THP) & common utilities (pre_config_hadoop_commons_config): Disables Linux Transparent Huge Pages compaction (transparent_hugepage/defrag and transparent_hugepage/enabled) in /etc/rc.local and installs base utilities (rpcbind, telnet): 
    include_recipe 'cdhmgr-chef-setup::pre_config_hadoop_commons_config'
  7. System resource limits (pre_config_hadoop_ulimits): Configures open file descriptor (nofile) and process (nproc) limits in /etc/security/limits.d/hadoop-cluster.conf: 
    include_recipe 'cdhmgr-chef-setup::pre_config_hadoop_ulimits'
  8. Database JDBC connectors (pre_config_database_connectors): Deploys the MySQL Connector/J (mysql-connector-java.jar) or PostgreSQL JDBC driver required by Hive, Oozie, Hue, and Sentry: 
    include_recipe 'cdhmgr-chef-setup::pre_config_database_connectors'
  9. Cluster /etc/hosts resolution (pre_config_hostfile_setup): Populates /etc/hosts with forward/reverse mappings and short aliases for all cluster nodes defined in the Chef role attributes: 
    include_recipe 'cdhmgr-chef-setup::pre_config_hostfile_setup'

Phase 2: Cloudera Manager Server Installation (cloudera_manager)

The cdhmgr-chef-setup::cloudera_manager recipe runs on the Admin node and executes three sub-recipes:

# 1. Install and configure MySQL/PostgreSQL and create service schemas
include_recipe 'cdhmgr-chef-setup::cloudera_manager_database_setup'

# 2. Install Cloudera Manager Server, Daemons, and Agent
include_recipe 'cdhmgr-chef-setup::cloudera_manager_setup'

# 3. Install Python dependencies for Cloudera Manager API automation
include_recipe 'cdhmgr-chef-setup::cloudera_manager_python_api'

Provisioned Service Databases

During cloudera_manager_database_setup, the cookbook creates dedicated databases and users for Cloudera Manager and CDH ecosystem components (replace the placeholder passwords below via Chef Vault or Encrypted Data Bags in production):

Cloudera Service Database Name Database User Password Placeholder
Cloudera Manager Server cmdb cmadmin cmadmin_password
Activity Monitor amon amon amon_password
Reports Manager rman rman rman_password
Hive Metastore Server metastore hive hive_password
Sentry Server sentry sentry sentry_password
Cloudera Navigator Audit Server nav nav nav_password
Cloudera Navigator Metadata Server navms navms navms_password
Hue Server hue hue hue_password
Oozie Server oozie oozie oozie_password

Sample Chef Roles

1. Basic Roles (Using Public Repositories)

When cluster nodes have outbound internet access (directly or via proxy), only the cluster /etc/hosts map and the Cloudera Manager server FQDN need to be overridden.

Basic NON-ADMIN Role

name "non_admin_roles_cdh_hadoop_cluster"
description "Pre-configuration for Cloudera Hadoop Cluster non-admin nodes"
run_list "recipe[cdhmgr-chef-setup::pre_config]"

default_attributes(
  "cdhmgr-chef-setup" => {
    "etc_hosts_entries" => {
      "192.168.13.2"  => { "hostname" => "ahmedservere001.ahmed.com", "aliases" => ["ahmedservere001"], "comment" => "Edge 01   - Cloudera Hadoop Node" },
      "192.168.13.3"  => { "hostname" => "ahmedservere002.ahmed.com", "aliases" => ["ahmedservere002"], "comment" => "Edge 02   - Cloudera Hadoop Node" },
      "192.168.13.4"  => { "hostname" => "ahmedservera001.ahmed.com", "aliases" => ["ahmedservera001"], "comment" => "Admin 01  - Cloudera Hadoop Node" },
      "192.168.13.6"  => { "hostname" => "ahmedserverm001.ahmed.com", "aliases" => ["ahmedserverm001"], "comment" => "Master 01 - Cloudera Hadoop Node" },
      "192.168.13.7"  => { "hostname" => "ahmedserverm002.ahmed.com", "aliases" => ["ahmedserverm002"], "comment" => "Master 02 - Cloudera Hadoop Node" },
      "192.168.13.9"  => { "hostname" => "ahmedserverw001.ahmed.com", "aliases" => ["ahmedserverw001"], "comment" => "Worker 01 - Cloudera Hadoop Node" },
      "192.168.13.10" => { "hostname" => "ahmedserverw002.ahmed.com", "aliases" => ["ahmedserverw002"], "comment" => "Worker 02 - Cloudera Hadoop Node" }
    }
  }
)

override_attributes(
  "cdhmgr-chef-setup" => {
    "cloudera_mgr_services" => {
      "server_host_fqdn_for_agent" => "ahmedservera001.ahmed.com"
    }
  }
)

Basic ADMIN Role

name "admin_roles_cdh_hadoop_cluster"
description "Pre-configuration and Cloudera Manager installation for Admin node"
run_list "recipe[cdhmgr-chef-setup::pre_config]", "recipe[cdhmgr-chef-setup::cloudera_manager]"

default_attributes(
  "cdhmgr-chef-setup" => {
    "etc_hosts_entries" => {
      "192.168.13.2"  => { "hostname" => "ahmedservere001.ahmed.com", "aliases" => ["ahmedservere001"], "comment" => "Edge 01   - Cloudera Hadoop Node" },
      "192.168.13.3"  => { "hostname" => "ahmedservere002.ahmed.com", "aliases" => ["ahmedservere002"], "comment" => "Edge 02   - Cloudera Hadoop Node" },
      "192.168.13.4"  => { "hostname" => "ahmedservera001.ahmed.com", "aliases" => ["ahmedservera001"], "comment" => "Admin 01  - Cloudera Hadoop Node" },
      "192.168.13.6"  => { "hostname" => "ahmedserverm001.ahmed.com", "aliases" => ["ahmedserverm001"], "comment" => "Master 01 - Cloudera Hadoop Node" },
      "192.168.13.7"  => { "hostname" => "ahmedserverm002.ahmed.com", "aliases" => ["ahmedserverm002"], "comment" => "Master 02 - Cloudera Hadoop Node" },
      "192.168.13.9"  => { "hostname" => "ahmedserverw001.ahmed.com", "aliases" => ["ahmedserverw001"], "comment" => "Worker 01 - Cloudera Hadoop Node" },
      "192.168.13.10" => { "hostname" => "ahmedserverw002.ahmed.com", "aliases" => ["ahmedserverw002"], "comment" => "Worker 02 - Cloudera Hadoop Node" }
    }
  }
)

override_attributes(
  "cdhmgr-chef-setup" => {
    "cloudera_mgr_services" => {
      "server_host_fqdn_for_agent" => "ahmedservera001.ahmed.com"
    }
  }
)

2. Custom Air-Gapped / Internal Mirror Roles

In enterprise environments without direct internet access, override the internal YUM repository URLs, pinned RPM package versions, and internal MySQL Connector/J artifact URL:

name "admin_roles_cdh_hadoop_cluster"
description "Cloudera Manager Admin Node Role using Internal YUM Mirror"
run_list "recipe[cdhmgr-chef-setup::pre_config]", "recipe[cdhmgr-chef-setup::cloudera_manager]"

default_attributes(
  "cdhmgr-chef-setup" => {
    "etc_hosts_entries" => {
      "192.168.13.2"  => { "hostname" => "ahmedservere001.ahmed.com", "aliases" => ["ahmedservere001"], "comment" => "Edge 01   - Cloudera Hadoop Node" },
      "192.168.13.3"  => { "hostname" => "ahmedservere002.ahmed.com", "aliases" => ["ahmedservere002"], "comment" => "Edge 02   - Cloudera Hadoop Node" },
      "192.168.13.4"  => { "hostname" => "ahmedservera001.ahmed.com", "aliases" => ["ahmedservera001"], "comment" => "Admin 01  - Cloudera Hadoop Node" },
      "192.168.13.6"  => { "hostname" => "ahmedserverm001.ahmed.com", "aliases" => ["ahmedserverm001"], "comment" => "Master 01 - Cloudera Hadoop Node" },
      "192.168.13.7"  => { "hostname" => "ahmedserverm002.ahmed.com", "aliases" => ["ahmedserverm002"], "comment" => "Master 02 - Cloudera Hadoop Node" },
      "192.168.13.9"  => { "hostname" => "ahmedserverw001.ahmed.com", "aliases" => ["ahmedserverw001"], "comment" => "Worker 01 - Cloudera Hadoop Node" },
      "192.168.13.10" => { "hostname" => "ahmedserverw002.ahmed.com", "aliases" => ["ahmedserverw002"], "comment" => "Worker 02 - Cloudera Hadoop Node" }
    }
  }
)

override_attributes(
  "cdhmgr-chef-setup" => {
    "cloudera_mgr_services" => {
      "common_packages"          => ["oracle-j2sdk1.7", "cloudera-manager-daemons", "cloudera-manager-agent", "openldap-clients"],
      "common_packages_versions" => ["1.7.0+update67-1", "5.12.0-1.cm5120.p0.120.el6", "5.12.0-1.cm5120.p0.120.el6", "2.4.40-16.el6"],
      "server_packages"          => ["cloudera-manager-server", "openldap-clients"],
      "server_packages_versions" => ["5.12.0-1.cm5120.p0.120.el6", "2.4.40-16.el6"],
      "server_host_fqdn_for_agent" => "localhost"
    },
    "database_to_install" => "mysql"
  },
  "mysql_connector" => {
    "j" => {
      "url" => "http://repo.ahmed.com/mysql-connector/Connector-J/mysql-connector-java-5.1.40.tar.gz"
    }
  },
  "authorization" => {
    "sudo" => {
      "groups" => ["cmadmin"]
    }
  },
  "yum_repository" => {
    "cm" => {
      "description" => "Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64",
      "baseurl"     => "http://repo.ahmed.com/cm5/redhat/6/x86_64/cm/5.12.0/",
      "gpgkey"      => "http://repo.ahmed.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera"
    }
  }
)

References

You May Also Enjoy