AHMED ZBYR

AHMED ZBYR

Biker, EV Enthusiast

Chef Data Bags store global JSON data on the Chef Infra Server that can be queried by any cookbook or node. When storing sensitive data—such as user password hashes, SSH keys, database credentials, or TLS private keys—Encrypted Data Bags encrypt item values using AES-256-CBC with a shared secret key so plain-text secrets are never stored in the Chef Server database or committed to version control.

This guide demonstrates how to:

  1. Generate an encrypted data bag secret key with OpenSSL.
  2. Create user account JSON definitions (for sysadmin and root) using the community users cookbook.
  3. Upload encrypted data bag items to the Chef Server.
  4. Configure Test Kitchen (.kitchen.yml) to test encrypted data bags locally with chef_zero.

Security Best Practices:

  • Never commit your secret key file (user_data_bags_encrypted_secret) or unencrypted secret JSON files into Git. Add the secret key path to .gitignore and chefignore.
  • When generating Linux password hashes for /etc/shadow, prefer SHA-512 (openssl passwd -6) on modern Linux distributions over legacy MD5 (openssl passwd -1).
  • If you need per-node key authorization (so only specific nodes can decrypt a secret without distributing a shared secret file manually), consider Chef Vault (chef-vault), which builds on top of encrypted data bags using each node’s RSA client key.

Step 1: Generate the Shared Encryption Secret Key

Use openssl rand to generate a 512-byte base64-encoded secret file and restrict its permissions to 0600:

mkdir -p ~/work/chef-repo/cookbooks/init-setup/secret-file
openssl rand -base64 512 > ~/work/chef-repo/cookbooks/init-setup/secret-file/user_data_bags_encrypted_secret
chmod 0600 ~/work/chef-repo/cookbooks/init-setup/secret-file/user_data_bags_encrypted_secret

Example output of openssl rand -base64 512:

┌─[ahmed][zubair-HP-ProBook][~]
└─▪ openssl rand -base64 512
whPbpd9VtLSYsasvnqdiSpWKsSB3hp8S3dwyNpEOBFFCjYQjvxe2DNJp2v4Ow2qM
4Y3TUSrXqSkiR7BqzdnTAi2wrUOaqjxCpf4yUdbkCaeEAFJpegkJ9gCYQvVXMH72
...
hQx/5xJgZ3uI3BUMyTjCJotbwb1ut3GntHpkNb1ZHmg=

Step 2: Create the Data Bag Item JSON Files

We will create two separate data bags so regular administrative accounts (user_data_bag) are separated from the root account (root_data_bag):

mkdir -p data_bags/user_data_bag
mkdir -p data_bags/root_data_bag

Directory layout:

data_bags/
├── root_data_bag/
│   └── root.json
└── user_data_bag/
    └── sysadmin.json

Generate Password Hashes

Generate the hashed password strings for /etc/shadow using openssl passwd:

# SHA-512 hash (recommended for modern Linux):
openssl passwd -6

# Legacy MD5-crypt (-1) example shown in the original test lab:
openssl passwd -1 'REPLACE_WITH_ROOT_PASSWORD'
# $1$JbS7rQs0$dRIRoWJ7HIRIAftFoD/iF/

openssl passwd -1 'REPLACE_WITH_SYSADMIN_PASSWORD'
# $1$Uat6dd6b$0FYr2a7NUpX8AnHQtaDwY/

1. data_bags/user_data_bag/sysadmin.json

{
  "id": "sysadmin",
  "password": "$1$Uat6dd6b$0FYr2a7NUpX8AnHQtaDwY/",
  "groups": [
    "sysadmin"
  ],
  "uid": 9000,
  "shell": "/bin/bash"
}

2. data_bags/root_data_bag/root.json

{
  "id": "root",
  "password": "$1$JbS7rQs0$dRIRoWJ7HIRIAftFoD/iF/",
  "uid": 0,
  "home": "/root",
  "groups": [
    "root"
  ],
  "action": "modify"
}

Step 3: Encrypt and Upload Data Bags to the Chef Server

Use knife data bag create and knife data bag from file --secret-file to encrypt the JSON attributes (every field except "id") and upload them to the Chef Server:

# Create and upload encrypted root_data_bag item
knife data bag create root_data_bag
knife data bag from file root_data_bag data_bags/root_data_bag/root.json \
  --secret-file secret-file/user_data_bags_encrypted_secret

# Create and upload encrypted user_data_bag item
knife data bag create user_data_bag
knife data bag from file user_data_bag data_bags/user_data_bag/sysadmin.json \
  --secret-file secret-file/user_data_bags_encrypted_secret

Verify Encryption and Decryption with knife

  1. View the raw encrypted payload stored on the Chef Server (without passing --secret-file): 
    knife data bag show user_data_bag sysadmin -Fj

    {
  "id": "sysadmin",
  "password": {
    "encrypted_data": "QEQV2MBkDh6FOlO29vJgdgoM1kNH6xNkfBrB2K8E9WcfHOYdkHzZUuu0lMJU\nbqSW5TaWiW60gP3Xcgn/jOxVnw==\n",
    "iv": "Oro4TJcXRwbXAb8tG+3eJQ==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  },
  "groups": {
    "encrypted_data": "qEws3EZsvgzYJmRpzFTEQQJAZcYHLkbzYpwzZyGZbT0=\n",
    "iv": "usca8tkD0/tatXxX17KAKQ==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  },
  "uid": {
    "encrypted_data": "leILI+0wFS258IXf5UuNMBh+ZhKW+hJiQ0mtsW2a9gg=\n",
    "iv": "AiWB2YnkGHIkZzgivkmfjA==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  },
  "shell": {
    "encrypted_data": "61OW+eH8dynbXuL/HxWWuYHIJzd8ODB0H/MXA9tM69A=\n",
    "iv": "5SJcvDoBZdto5p2HKerUkg==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  }
}
  2. Verify decryption by passing --secret-file: 
    knife data bag show --secret-file secret-file/user_data_bags_encrypted_secret user_data_bag sysadmin

    Encrypted data bag detected, decrypting with provided secret.
groups:   sysadmin
id:       sysadmin
password: $1$crcL.lu/$uIR/GRpX7aMnI2wUTT31S0
shell:    /bin/bash
uid:      9000

Step 4: Testing Encrypted Data Bags Locally with Test Kitchen

When testing a cookbook locally with Test Kitchen (chef_zero provisioner), Test Kitchen needs both the encrypted JSON files (not the plain-text ones!) and the path to the secret file.

  1. Export the encrypted JSON from the Chef Server into your cookbook’s test directory: 
    mkdir -p testing_encrypted_data_bags/data_bags/user_data_bag
mkdir -p testing_encrypted_data_bags/data_bags/root_data_bag

knife data bag show user_data_bag sysadmin -Fj > testing_encrypted_data_bags/data_bags/user_data_bag/sysadmin.json
knife data bag show root_data_bag root -Fj > testing_encrypted_data_bags/data_bags/root_data_bag/root.json
  2. Configure data_bags_path and encrypted_data_bag_secret_key_path in .kitchen.yml:
---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu/trusty64

suites:
  - name: default
    data_bags_path: '/home/ahmed/work/chef-repo/cookbooks/init-setup/testing_encrypted_data_bags/data_bags'
    encrypted_data_bag_secret_key_path: '/home/ahmed/work/chef-repo/cookbooks/init-setup/secret-file/user_data_bags_encrypted_secret'
    run_list:
      - recipe[init-setup::default]
    attributes:

Step 5: Consume the Encrypted Data Bags in the Cookbook Recipe

In attributes/default.rb, define the target group and GID:

default['users_setup']['groups'] = { 'sysadmin' => 2300 }

In recipes/default.rb, use the users_manage resource from the users cookbook (Chef 12+ automatically detects and decrypts encrypted data bag items using /etc/chef/encrypted_data_bag_secret on the node):

# Create groups and users defined in 'user_data_bag'
node['users_setup']['groups'].each do |group_name, group_id|
  users_manage group_name do
    group_id group_id
    action [:create]
    data_bag 'user_data_bag'
  end
end

# Modify the root user password from 'root_data_bag'
users_manage 'root' do
  data_bag 'root_data_bag'
end

Converge and Verify in Test Kitchen

kitchen create
kitchen converge

Excerpt from kitchen converge:

       Preparing data_bags
       Preparing secret
...
       Recipe: init-setup::default
         * users_manage[sysadmin] action create
           * group[sysadmin] action create (skipped due to only_if)
           * user[sysadmin] action create (up to date)
           * group[sysadmin] action create (up to date)
         * users_manage[root] action create
           * user[root] action modify (up to date)
           * group[root] action create (up to date)
       Chef Client finished, 12/77 resources updated in 14 seconds
       Finished converging <default-ubuntu-trusty64> (0m41.27s).

References

You May Also Enjoy