Getting Started with OpenVox

2025-03-06 sysadmin puppet openvox linux hardening openvox configurationmanagement sysadmin

Introduction

Recently, I have started managing a small fleet of servers for personal and research purposes. Managing a server or two isn’t that bad to do manually, but once you get much more than this, things like adding a new SSH key, installing a new piece of software, or making a configuration change across all of them becomes burdensome and repetitive. This negative effect compounds as you add systems and complexity.

There are also things that I end up doing almost universally when I provision new systems. Things like adding myself a user, configuring sudo, installing my dotfiles, installing tools I like to use, ssh hardening, unattended updates, … I know I do more than this and often I forget things when doing it manually.

I may do the same setting on two different systems in non-uniform ways that make automating the task in the future cumbersome. Things like installing software to different file paths or using non-matching file names in conf.d directories. The settings are functionally the same, but you can’t easily write a script to change settings within these because they’re different.

I could configure the system perfectly, but not take good documentation on how I got there. Typically when I am installing new software, it is an iterative process. I can make best guesses on how things should look, but the end result often changes as I encounter constraints, unforeseen issues, or simply learn more about the software I am using. Updates happen that introduce breaking changes or useful features, forcing me to correct them. A lot of mature software has the habit of running reliably for long periods of time with little oversight–shocker, i know. I have had stuff go for several years without needing much attention, but suddenly I need to move it, the hardware fails, or it reaches end of life and i have to install the next generation of software because the update process for the new software isn’t clean. Now I have to repeat the installation process but have no documentation on what i did or why and its been a few years since I thought about it.

This is a common problem in IT to some degree at every single organization I’ve worked at, consulted with, or heard about secondhand. Doing it perfectly where everything is set up correctly, completely, and everything is accounted for is profoundly difficult to do in practice.

I remember the concept explained plainly in The Phoenix Project. Something along the lines of “you should treat infrastructure like cattle, not pets”. This basically refers to uniformity, repeatability, and the ease of recovering from failures–its usually much easier to get back up and running if you have backups, documentation, and have actually tested the restoration/recovery processes a few times. A non-trivial chunk of system engineering is deploying and configuring services. A large chunk of this chunk can and should be automated. For various reasons, this doesn’t happen and the work gets done manually or in an ad-hoc manner, leading to non-uniformity, non-correctness, non-completeness, and configuration drift.

There are several methods to do this. Depending on your operation’s maturity, budget, etc. your options may be wildly different. Even something like some deploy scripts kept in Git that you copy and paste into a terminal to get things up and running is far better than nothing.

It has been several years since I’ve had to do any of this, but the time has come–I have a half dozen systems running and expect to add more, so its time to get serious about configuration management. I had used Puppet in the past and liked it, so I figured I’d just start there.

Quickly I realized that the Puppet community edition was no longer available because Puppet was acquired by Perforce Software and they have been aligning to a more commercialized operating model. I didn’t take too much time to investigate, but saw that they do have a developer edition that requires registration (boooo!) and it is limited to 25 nodes. Bummer. I support their decision to run their business how they want to, but I am not a fan of registering to use software, being marketed to, and being subjected to arbitrary restrictions when I know there are free open-sourced alternatives that don’t do this.

https://www.puppet.com/downloads/puppet-core

I quickly find that OpenVox is now what Puppet community edition used to be in spirit. I figured I’d give it a shot: https://voxpupuli.org/openvox/

I know from experience that the process

Architecture Overview

tl;dr, basic client/server architecture with a CA for authentication and traffic encryption. you can do a server-less setup with puppet apply as well, but that’s not what I will be doing.

For more information about the architecture see https://docs.openvoxproject.org/openvox/8.x/architecture.html

OpenVox Server -- server1
               |- server2
               |- server3
               |- ...
			   |- workstation1

The server maintains a CA to authenticate clients so that unauthorized users may not simply grab your configurations.

The configurations are stored in plaintext files using a Ruby-based DSL.

This configuration is typically stored in Git for reviewing, version control, and data redundancy/integrity.

Agents run on each endpoint. These connect outbound to the server, authenticate with their certificate, and the server provides them with their desired state configuration. The agent checks how the machine is configured and applies changes where appropriate. This prevents “configuration drift” and negates the need to login to your entire fleet of computers to make changes, corrects changes made that do not align with the desired state, and provides an audit trail that something has changed.

Initial Configuration

Arguably, you can do stuff like the hostname and installing packages with OpenVox once it is set up.

VPS provisioning

Since I only have a handful of hosts, I manually provisioned the server on a cheap $5/month VPS running Debian 13. This is a bit lower specs than the recommendations, but lowering the RAM usage of the JVM should make it work fine for now. If it becomes a problem later, I can bump up the RAM or add another swap file.

After this is done, ssh in and test the machine for basic functionality.

DNS

I added A and AAAA DNS records for the newly-provisioned machine for its public IP addresses. I used puppet.mydomain.tld.

Update the OS

apt update
apt full-upgrade -y
reboot

Install some prereqs

apt install -y curl wget gnupg vim git openjdk-21-jre-headless rsync make

hostname

I set the hostname

hostnamectl set-hostname puppet.mydomain.tld

adding a user

I added a user for administration, configured them to be able to use sudo, and set up their ssh keys. My VPS provider installs my pre-configured ssh keys to the root user, so i just use those.

adduser admin
usermod -a -G sudo admin
install -d -m 700 -o admin -g admin /home/admin/.ssh
cp /root/.ssh/authorized_keys /home/admin/.ssh/authorized_keys
chown admin:admin /home/admin/.ssh/authorized_keys
chmod 600 /home/admin/.ssh/authorized_keys

leave your root session running and verify that you can login as the new user. If all goes well, you sh

SSH

I applied basic ssh hardening: no root logins, no password logins.

verify the config:

sshd -t

reload the config:

systemctl reload ssh

Installing the Server

cd /tmp
wget https://apt.voxpupuli.org/openvox8-release-debian13.deb
dpkg -i openvox8-release-debian13.deb
apt update
apt install openvox-server ### server depends on openvox-agent, so no need to install it explicitly

if all goes well, this should work:

/opt/puppetlabs/bin/puppet --version

this should present you with a version number similar to:

8.29.0

Configure JVM for lower resources

This step isn’t necessary if you run the server on a machine with more resources, but for now i only have a handful of systems in my fleet so paying extra money for more resources that i technically don’t even need yet isn’t worth it to me.

Update the JAVA_ARGS variable in /etc/default/puppetserver to use 512mb of RAM instead of 2gb by default.

Mine came with this:

JAVA_ARGS="-Xms2g -Xmx2g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"

Change it to:

JAVA_ARGS="-Xms512m -Xmx512m -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"

Next, uncomment this line in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf

max-active-instances: 1

Establish the server’s identity

Change puppet.mydomain.tld to whatever you set in DNS earlier.

sudo /opt/puppetlabs/bin/puppet config set certname puppet.mydomain.tld --section main

Now configure it to use itself as the OpenVox server:

sudo /opt/puppetlabs/bin/puppet config set server puppet.mydomain.tld --section main

verify:

sudo /opt/puppetlabs/bin/puppet config print certname server

Start OpenVox

sudo systemctl enable --now puppetserver

check status:

sudo systemctl status puppetserver

If it failed, check the logs:

sudo journalctl -u puppetserver -n 100 --no-pager

Verify it us listening:

sudo ss -lnpt | grep 8140

You should see a java process listening on port 8140:

LISTEN 0      50                 *:8140            *:*    users:(("java",pid=11520,fd=27))

Test the agent:

sudo /opt/puppetlabs/bin/puppet agent --test

You should see no errors here. If any are encountered, investigate and correct them before proceeding. At this point, the server should be running but no configurations are present.

Back up certificate data

First, stop the services:

sudo systemctl stop puppet
sudo systemctl stop puppetserver

Next, create an encrypted copy of the certificates using gnupg. Enter a strong passphrase when prompted.

export GPG_TTY="$(tty)"
set -o pipefail
umask 077

tar -C / -cpf - \
  etc/puppetlabs/puppetserver/ca \
  etc/puppetlabs/puppet/ssl \
  | gpg --symmetric --cipher-algo AES256 \
      --pinentry-mode loopback \
      --output "/root/openvox-identity-$(date +%Y%m%d).tar.gpg"

Test decryption:

gpg --decrypt "/root/openvox-identity-$(date +%Y%m%d).tar.gpg" \
  | tar -tf - > /dev/null

echo "Verification exit status: $?"

Exit status should be 0 if decryption was successful.

Now, transfer this file off of the machine and keep it somewhere safe. You will need this if the machine running OpenVox needs to be reprovisioned.

Create a Git repository for the configurations

This should be somewhat private and never store secrets.

Take your own steps to secure the Git repository, as this will be different depending on your org.

Create the initial configuration

First, cd into the directory of the repository you just created.

Create the directory structure:

mkdir -p manifests site/role/manifests site/profile/manifests data

Create environment.conf:

cat > environment.conf <<'EOF'
modulepath = site:$basemodulepath
EOF

cat > manifests/site.pp <<'EOF'
node 'puppet.mydomain.tld' {
  include role::puppetserver
}
EOF

cat > site/role/manifests/puppetserver.pp <<'EOF'
class role::puppetserver {
  include profile::base
}
EOF

cat > site/profile/manifests/base.pp <<'EOF'
class profile::base {
  notify { 'Test Message':
    message => 'Test message to show git is working.',
  }
}
EOF

You should be left with something like this:

% tree
.
├── data
├── environment.conf
├── manifests
│   └── site.pp
└── site
    ├── profile
    │   └── manifests
    │       └── base.pp
    └── role
        └── manifests
            └── puppetserver.pp

8 directories, 4 files

add the stuff to git:

git add .
git commit -am "initial commit"
git push -u origin main

The stuff should be pushed to the server at this point.

Configure read-only git for the OpenVox server

Generate a key on the OpenVox server

sudo ssh-keygen -t ed25519 \
  -f /root/.ssh/openvox-control \
  -C 'openvox-control@puppet.badoperation.net' \
  -N ''

sudo cat /root/.ssh/openvox-control.pub

install it on git with read-only access. you will need to put the key and set your own path to the git file.

restrict,command="git-upload-pack '/path/to/openvox-control.git'" ssh-ed25519 AAAA... openvox-control@puppet.mydomain.tld

Configure the ssh client on OpenVox server:

sudo tee /root/.ssh/config > /dev/null <<'EOF'
Host openvox-git
    HostName git.mydomain.tld
    User git
    IdentityFile /root/.ssh/openvox-control
    IdentitiesOnly yes
    StrictHostKeyChecking yes
EOF

sudo chmod 600 /root/.ssh/config

Install the git server’s verified SSH host key in root’s known_hosts file.

Verify that this is what is running on the git server from the git server, and also on the OpenVox server. They should match. Do not proceed if they do not match and investigate what is happening. Do not blindly trust this, either.

On Git server:

sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

On OpenVox server:

ssh-keyscan -t ed25519 git.mydomain.tld 2>/dev/null \
  | ssh-keygen -lf -

If they match, add the scanned key on OpenVox server:

ssh-keyscan -t ed25519 git.mydomain.tld >> /root/.ssh/known_hosts
chmod 600 /root/.ssh/known_hosts

Clone the repo on the OpenVox server

git clone openvox-git:/srv/git/openvox-control.git

install the files for OpenVox:

sudo install -d -m 0755 /etc/puppetlabs/code/environments/production

sudo rsync -a --delete \
  --exclude='.git/' \
  /root/openvox-control/ \
  /etc/puppetlabs/code/environments/production/

validate them:

sudo /opt/puppetlabs/bin/puppet parser validate \
  /etc/puppetlabs/code/environments/production/manifests/site.pp \
  /etc/puppetlabs/code/environments/production/site/role/manifests/puppetserver.pp \
  /etc/puppetlabs/code/environments/production/site/profile/manifests/base.pp

Start OpenVox

sudo systemctl start puppetserver
sudo systemctl status puppetserver --no-pager

Everything should be running now. test the agent:

sudo /opt/puppetlabs/bin/puppet agent --test

You should see something like this if everything is working:

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet.mydomain.tld:8140 (172.16.15.25)
Notice: Catalog compiled by puppet.mydomain.tld
Info: Caching catalog for puppet.mydomain.tld
Info: Applying configuration version '1789854362'
Notice: Test message to show git is working.
Notice: /Stage[main]/Profile::Base/Notify[Test Message]/message: defined 'message' as 'Test message to show git is working.'
Notice: Applied catalog in 0.03 seconds

The design now is that configs can be pulled from the git server, then copied into production with rsync. This will change later into something a bit more elegant, but for now that’s what im going to do to get started. This will be configured in the next section.

Create Makefile

In the git repo, create Makefile

.PHONY: deploy validate

PUPPET_BIN := /opt/puppetlabs/bin/puppet
PRODUCTION := /etc/puppetlabs/code/environments/production

validate:
	find manifests site -type f -name '*.pp' -exec $(PUPPET_BIN) parser validate {} +

deploy:
	git pull --ff-only
	$(MAKE) validate
	sudo rsync -a --delete --exclude='.git/' --chmod=D755,F644 ./ $(PRODUCTION)/

Commit and push this to the repo, then pull it on the OpenVox server.

Now, you can use ssh from your workstation to deploy after you make changes and validate them:

ssh -t admin@puppet.mydomain.tld 'cd ~/openvox-control && make deploy'

This method is primitive and may be replaced in the future with r10k and maybe some git hooks, but for now it works fine.

Workflow:

  • develop locally where i can use an IDE and a browser and such and not have to use some janky text editor over SSH.

  • all of this is in git, so i get the benefits there. when i make new files, add them. make changes, commit them. push to main when i am satisfied its working.

  • run the make deploy command via ssh to actually run the configuration.

Keep in mind that Puppet can be very complex and other features and deployment glue can be refined as you need it. I am not to the point where i need CI/CD yet for a handful of relatively simple configurations on some non-critical systems that don’t matter if they have a bit of downtime here and there. When the time comes, much like the resources on the VPS i am running this on are exhausted, i can upgrade the setup.

A simple first configuration

This is where Puppet gets kinda hairy and setups can differ wildly.

This is a pretty simple one to set DebianBanner no on Debian and Ubuntu hosts to stop them from advertising the OS and build information in the server’s banner.

If you have never worked with Puppet or OpenVox before, you should familiarize yourself with the documentation about modules and the Puppet language. They are a DSL written in a Ruby dialect with classes and other conveniences for the domain of configuration management.

Puppet being its own DSL and having a steeper learning curve are criticisms it faced when I used it last. I never felt that it was that complex, and like any other language, you’re going to be kind of terrible at it when you first get started and will learn more as you go.

In a perfect world, you will be working on a team of sysadmins who are also developers and they can guide you along and review your commits as part of a change process. This isn’t always true, but it is kind of necessary with Puppet. I also believe that any configuration management tool like this requires development work. There’s no silver bullet here. You will have to roll up your sleeves and get your hands dirty with this.

I also would get discouraged sometimes because i discovered that I did something kind of wonky but it worked. Or that i didnt have 100% coverage of every configuration. Realize that it is a lot of work, may never be “complete” and is more of a continuous process than a final solution. It is impossible to offer a final solution for this kind of thing because organizations vary wildly with what they use behind the scenes. I took more of a Kaizen approach where i would just strive to make at least one meaningful change for the better per day. Even if this change was re-wording a comment to be more clearer, i counted this as a win.

In a perfect world though, you would want to be able to provision a new machine, hook it into OpenVox, and a few moments later its configured and running. This is possible with time and effort, but being incomplete is not a waste, either. If you can get rid of 90% of your configuration when starting up a new machine, this is a tremendous time savings. You can also look up in the code and read comments about why things are as they are, and look at the history in your revision control to see when changes were made and by whom.

The modules fundaemtnals and language summary are here:

https://docs.openvoxproject.org/openvox/8.x/modules_fundamentals.html

https://docs.openvoxproject.org/openvox/8.x/lang_summary.html

Additionally, you should probably read the Style Guide for some suggestions on writing clean, maintainable code: https://docs.openvoxproject.org/openvox/8.x/style_guide.html

Enable the puppet agent:

sudo systemctl enable --now puppet

edit site/profile/manifests/ssh.pp:

class profile::ssh {
  if $facts['os']['name'] in ['Debian', 'Ubuntu'] {
    file { '/etc/ssh/sshd_config.d/10-debianbanner.conf':
      ensure  => file,
      owner   => 'root',
      group   => 'root',
      mode    => '0644',
      content => "DebianBanner no\n",
    }
  }
}

get rid of the old notify banner in base.pp and include profile::ssh instead.

class profile::base {
  include profile::ssh
}

commit and push this, then deploy it using the ssh + make one-liner.

Once it is deployed, run puppet test and it should output that the 10-debianbanner.conf file was created.

Verify the file exists, then check the Debian banner with netcat:

nc puppet.mydomain.tld 22

This should give something like:

SSH-2.0-OpenSSH_10.0p2

Without it, it gives away this Debian-specific banner, which reveals information about the host:

SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4

To test this, i deleted the 10-debianconf.conf file, refreshed sshd, and checked that the Debian banner has returned using netcat. it did.

By default, puppet checks in every 30 minutes for configurations. this can be changed in /etc/puppetlabs/puppet/puppet.conf:

[agent]
runinterval = 10m

the puppet agent must be reloaded if you change the runinterval setting:

sudo systemctl reload puppet

To see the configured interval:

sudo /opt/puppetlabs/bin/puppet config print runinterval --section agent

Adding additional machines to the OpenVox server

edit manifests/site.pp, adding a default node:

node 'puppet.mydomain.tld' {
  include role::puppetserver
}

node default {
  include profile::base
}

commit, push, and pull this to the OpenVox server.

Next, install the agent on each system. In this case, I am doing it on a Debian 12 machine. I will need to download the openvox8 debian12 package to connect to OpenVox’s repositories, then install the OpenVox agent.

wget https://apt.voxpupuli.org/openvox8-release-debian12.deb
sudo dpkg -i openvox8-release-debian12.deb
sudo apt update
sudo apt install openvox-agent

Once this is installed, configure it to connect to the OpenVox server:

sudo /opt/puppetlabs/bin/puppet config set server puppet.mydomain.tld --section main
sudo /opt/puppetlabs/bin/puppet config set certname myhost.mydomain.tld --section main

Verify the settings:

sudo /opt/puppetlabs/bin/puppet config print server certname --section agent

now, test it on the new endpoint:

sudo /opt/puppetlabs/bin/puppet agent --test

This should fail with an error similar to this:

Couldn't fetch certificate from CA server; you might still need to sign this agent's certificate (myhost.mydomain.tld).
Exiting now because the waitforcert setting is set to 0.

Login to the OpenVox server to sign this certificate:

sudo /opt/puppetlabs/bin/puppetserver ca sign --certname myhost.mydomain.tld

Go back to the new endpoint and run the test again. it should apply the DebianBanner configuration:

sudo /opt/puppetlabs/bin/puppet agent --test

If everything goes according to plan, you should see a line similar to this, showing it was successful.

Notice: /Stage[main]/Profile::Ssh/File[/etc/ssh/sshd_config.d/10-debianbanner.conf]/ensure: defined content as '{sha256}c417daa5c960635ce5b152e793af867c40d51dfe56ca71d427e2166038b3b100'

Now, enable the OpenVox agent on the new host:

systemctl enable --now puppet

At this point, you should be good to go with repeating the process for each host. For convenience, I’d recommend making small scripts to wget the appropriate package, install it, update your package manager, then install openvox-agent, enable the service, and run the test. You will have to login to the OpenVox server and manually sign each host’s certificate to allow them to use it.

I ended up with something like this:

OPENVOXSVR="puppet.badoperation.net"

. /etc/os-release
case "$ID:$VERSION_ID" in
  debian:12|debian:13|ubuntu:24.04)
    OPENVOXPKG="openvox8-release-${ID}${VERSION_ID}.deb"
    ;;
  *)
    echo "Unsupported OS: $PRETTY_NAME" >&2
    exit 1
    ;;
esac

CERTNAME="$(hostname -f)"

echo "Operating system: $PRETTY_NAME"
echo "Puppet server: $OPENVOXSVR"
echo "Certificate name: $CERTNAME"
echo "Repository package: $OPENVOXPKG"

cd /tmp
wget -O "$OPENVOXPKG" "https://apt.voxpupuli.org/$OPENVOXPKG"
sudo dpkg -i "$OPENVOXPKG"
sudo apt update
sudo apt install -y openvox-agent

sudo /opt/puppetlabs/bin/puppet config set server "$OPENVOXSVR" --section main
sudo /opt/puppetlabs/bin/puppet config set certname "$CERTNAME" --section main

sudo /opt/puppetlabs/bin/puppet config print server certname --section agent
sudo /opt/puppetlabs/bin/puppet agent --test

Take note of the client’s certificate.

Then back to server. list certificates, then compare to the client’s certificate for authenticity. If they match, sign it:

sudo /opt/puppetlabs/bin/puppetserver ca list
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname HOST_FQDN

then finally back to the new machine:

sudo /opt/puppetlabs/bin/puppet agent --test
sudo systemctl enable --now puppet

Mirroring for OpenVox packages

There are currently a apt and yum repositories on Vox Pupuli for OpenVox software downloads. They conveniently have an rsyncable copy of this data to allow you to set up a local mirror. This is highly recommended if you are managing several systems as you can do this locally on basically any web server and a scheduled rsync to keep it current. This prevents dozens of your own systems reaching out to the internet to download this software, making installation fast and take less bandwidth over time.

Doing this is a bit beyond the scope of this documentation, but it is not a very difficult project, and i will probably end up doing it and linking instructions in the future.

https://voxpupuli.org/blog/2025/03/04/openvox-downloads-and-mirroring/

https://apt.voxpupuli.org/

https://yum.voxpupuli.org/

Next steps

After all of your machines are hooked into OpenVox and verified to work, its time to start writing code for your configurations.

profile::base should contain settings meant for all systems.

roles are for hosts with distinct jobs. For example role::gitserver will include profile::base and settings for running a Git server.

When you want to assign a node a role, add it in site.pp.

Remember to test your changes before rolling them out broadly. Doubly so when making changes to things like authentication or SSH as a bad configuration may lock you out of your system. A throwaway VM works great for this. I recommend using Vagrant for this:

  • boot a new VM

  • run your script to bootstrap the OpenVox agent installation

  • sign the certificate on the OpenVox server

  • assign the role you are testing/developing to your new VM only

  • once it is good to go, deploy it to the other hosts.

Remember to never store secrets in this code or Git either. For this, use Hiera.

No notes link to this note

Relevant Topics


Recent Posts