Create ssh directory

Create a directory for your ssh keys and tools inside your git repository

$ cd ~/git/website
$ mkdir -p .ssh-tools/ssh-keys
$ cd .ssh-tools

All commands on subsequent slides will run from this directory.

Generate Key

$ ssh-keygen -t ed25519 -f ssh-keys/websyncer -C "WebSyncer"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): reallysecurethinghere
Your identification has been saved in websyncer
Your public key has been saved in websyncer.pub
The key fingerprint is:
SHA256:uo2OjpPlQhHwacZJayF+xGhjBeDWf8FNtLeEeZM1etc WebSyncer
The key's randomart image is:
+--[ED25519 256]--+
|=oBo    .o   o   |
|oX+*  . o + + . .|
|o+@+   o = B . .E|
|.+o .   . + + .  |
|   . . .S  .     |
|  . . ..         |
| . +  .          |
|  +... +         |
|  .+o.+ .        |
+----[SHA256]-----+

Generate Key (cont)

$ cat ssh-keys/websyncer.pub
ssh-ed25519 AAAAC3NzaC1l...8021X WebSyncer

Gather server host keys

Gather server host keys

$ ssh-keyscan web-00 web-01 web-02 > ssh_known_hosts
$ cat ssh_known_hosts
web-00 ssh-rsa AAAAB3NzaC1yc...0z$ux
web-00 ssh-ed25519 AAAAC3NzaC1...MDL79
web-00 ecdsa-sha2-nistp256 AAAAE2VjZH...abblx
web-01 ssh-rsa AAAAB3NzaC1yc...E&tJ0s
web-01 ssh-ed25519 AAAAC3NzaC1...dBe3f
web-01 ecdsa-sha2-nistp256 AAAAE2VjZH...er3&&
...

For bonus points scan on bare hostname, FQDN, and IPs.

Enable key trust on webservers

Enable key trust on webservers

Highly manual example. It's Better to use a config management tool like ansible.

$ scp ssh-keys/websyncer.pub web-00:/tmp
$ ssh web-00

Install authprogs

web-00$ sudo apt install authprogs || sudo pip3 install authprogs

Enable key trust on webservers (cont)

web-00$ sudo su - webuser

webuser@web-00$ $ authprogs  \
                  --install_key /tmp/websyncer.pub \
                  --keyname WebSync \
                  --logfile ~/authprogs.log 

webuser@web-00$ cat ~/.ssh/authorized_keys
command="/usr/bin/authprogs --run 
   --logfile=/home/webuser/authprogs.log 
   --keyname=WebSync",no-port-forwarding
   ssh-ed25519 AAAAC3...8021x WebSyncer

Create authprogs configuration yaml

You define what commands the ssh key can run via a yaml file

$ mkdir ~/.ssh/authprogs.d
$ vi ~/.ssh/authprogs.d/websync.yaml
...

$ cat ~/.ssh/authprogs.d/websync.yaml
-
from: [192.168.0.10, 192.168.0.15]
keynames: [WebSync]
allow:
   - command: sudo service nginx restart
   - command: hostname
   - rule_type: rsync
        allow_upload: true
        allow_recursive: true
        allow_archive: true
        paths:
            - /srv/web/htdocs

CI/CD Environment

Our CI/CD environment

$ pwd
~/git/website/.ssh-tools

$ ls -1R
ssh
ssh-keys/websyncer
ssh-keys/websyncer.pub
ssh-load
ssh_known_hosts

Key Passphrase

We make the key passphrase available via CI/CD environment variables.

Example gitlab screenshot:

Load the keys

Load the keys from within the CI/CD environment. (ssh-load source)

$ cat ssh-load
#!/bin/bash
# Load any keys in the ssh-keys directory
#
# Passphrase must be in a variable of the
# form keyfilename_PASSPHRASE. For example if the
#  file is id_rsa then the variable is id_rsa_PASSPHRASE.
#
# As such each file must be composed of characters valid in variables.
# (No dashes or dots, for example.)
#
# Ignores any *.pub files.

# Are we acting as the askpass script?
if [ $# -gt 0 ] ; then
    read foo
    echo $foo
    exit 0
fi

Load the keys (cont)

set -e
set -u
me=$(realpath $0)
cd $(dirname $0)

# Only run if keys present
[ -d ./ssh-keys ] || exit 0

# Assure ssh thinks we're on x11.
export DISPLAY=:0

eval $(ssh-agent -a /tmp/ssh-agent.sock -s) >/dev/null
cd ./ssh-keys

Load the keys (cont)

for key in *
do
    if echo $key | grep -q '\.pub$' ; then
        continue
    fi
    varname="${key}_PASSPHRASE"
    passphrase=${!varname}
    if [ -z "${passphrase}" ] ; then
        continue
    fi

    # Make sure it's only readable by us
    chmod 600 $key

    SSH_ASKPASS="${me}" ssh-add "${key}" <<<"${passphrase}" \
      2>/dev/null || \
      echo "Could not load ssh key $key - Bad passphrase in $varname?"
done

The ssh wrapper

Use a wrapper around ssh that uses agent and validates hostkeys. (ssh source)

$ cat ssh
#!/bin/bash
# SSH wrapper that uses our agent

export SSH_AUTH_SOCK=/tmp/ssh-agent.sock

exec /usr/bin/ssh \
    -o controlpath=none \
    -o batchmode=yes \
    -o stricthostkeychecking=yes \
    -o globalknownhostsfile="$(dirname $0)/ssh_known_hosts" \
    -o userknownhostsfile=/dev/null \
    "$@"

Push On

Sample push script

$ cat push.sh
#!/bin/bash
set -e
set -u

ssh_tools="$CI_PROJECT_DIR"/.ssh-tools
ssh="$ssh_tools/ssh"

# Alternatively, put the tools dir first in path
# PATH="$ssh_tools":$PATH

for host in web-00 web-01 web-02
do
    rsync -a -e $ssh "$CI_PROJECT_DIR/webdocs/" \
          webuser@$host:/srv/web/htdocs/
done

$ssh webuser@$host sudo service nginx restart

CI/CD configuration

Tell your CI/CD to load keys prior to running relevant pipeline phases

$ cat ../.gitlab-ci.yml 
...
push:
    stage: push
    only:
        - main
    script:
        - .ssh_tools/ssh-load
        - push.sh