Git your ssh config

Git your ssh config


Anyone who has spent any time in Linux, whether it be as a sysadmin or a consumer of cloud computing or HPC, will understand the importance and power of Secure Shell (SSH), of which the most widely used implementation is OpenSSH. As you can see at man 5 ssh_config, there is a rich variety of options you can put into your OpenSSH client’s configuration file at ~/.ssh/config. The more adept you become with ssh, the more complicated that config file will grow.

You need ssh configs for all the servers at work, all the devices at home, all your Git hosting sites, all the places where you queue up your HPC jobs, and so forth. At the time of writing I have nearly 200 lines of configuration distributed between eleven files.

Statement of the problem

As ssh config complexity grows, it becomes more and more tempting to version control these files with Git. After all, it seems clear that an ssh config, consisting of nicely formatted plain text, should work well with Git, right?

But no, there is a problem:

  1. OpenSSH requires that no ssh config files are world-readable or even group-readable. When you run ls -l ~/.ssh/config the first field is expected to look like -rw-------. If not, you will get a warning about insecure permissions and your ssh client will refuse to try to make any connections.
  2. Git does not have much respect for Unix-style user-group-other/read-write-execute permission systems. Git is intended to be agnostic about filesystems, and different filesystems have different ways to control access. Once you accept this it becomes clear why many operations with Git reset file permissions to look like -rw-r--r--.

Thus Git regularly resets ssh config file permissions to values that the OpenSSH client considers unacceptable.

Possible workarounds

As a possible workaround, each time you do a Git operation with your ssh config repo, you can try reminding yourself to run,

find ~/.ssh -type f -exec chmod 0600 {} +

This gets tiresome very quickly. (Trust me, I tried.)

Doing it with a cronjob instead would also not be a great solution. How often would you run the cronjob? Every ten seconds just on the off chance that at this moment you do something with Git? That would be ridiculous.

As an alternative workaround, I would have guessed a configuration setting might exist to allow overriding the strict check. It would be an unacceptable workaround, but the option doesn’t exist anyway, so it’s moot.

Sketch of solution

Fortunately there is a way to force the behaviour OpenSSH requires — otherwise I wouldn’t have written all this ;)

My solution isn’t perfect. There are some caveats:

But, once you have it set up on any given machine, it works automatically and you don’t have to think about it again. So, that’s good.

The tutorial will have the following structure:

  1. Create a sparse image file
  2. Create the filesystem
  3. Define permissions and set up ACLs
  4. Add ssh config files
  5. Add an fstab entry

Let’s get started.

Create a sparse image file

Okay, if you feel like creating a small partition on your hard drive just to hold your ssh config, go for your life and do that instead of following the instruction in this section.

Assuming that sounds too annoying, you are going to want to set up an image file to associate with a loop device.

What do I mean by loop device?

This is a case where rtfm can be useful, so make sure to check out man 4 loop and man 8 losetup.

tl;dr There are devices that live in /dev with names like /dev/loop0, /dev/loop1 and so forth.

You can associate these devices with whatever files you feel like (although it requires sudo or such to actually form the association). Then when you do filesystem operations on the loop device, the operation will pass through to the associated file.

This allows you to, for example, encrypt a file with cryptsetup, or (as we’ll do here) create an ext4 filesystem within the file.

(If you don’t see any loop devices in /dev you can force their creation with sudo losetup -f.)

What do I mean by sparse image file?

Your new filesystem is going to want to know how much space it’s allowed to take up. You’ll need to allow room not just for the ssh config files, but also all the Git files, including the stored history as your ssh configs evolve over time, as well as the filesystem.

You don’t want to be too stingy because it’s not fun to run out of room. There is also the problem that ext4 is increasingly prone to fragmentation the fuller it gets.

(You might have heard ext4 is resistant to fragmentation so you never need to run defrag in Linux? Keep in mind that resistant is the operative word there. If you only ever use, say, half of an ext4 filesystem, then indeed you never need to worry about doing defrags. That becomes less true as you push it up to 80% or 90%.)

One way to make sure your filesystem has enough room would be to preallocate, say, twice as much as you think you’ll ever need. If you think your ssh config + Git repo will never go over 4MB, you could do dd if=/dev/zero of=~/imgs/ssh_config.img bs=4M count=2. (Or in fact you would probably have to do count=10. Each ext4 filesystem comes with an initial 33MB of infrastructure even before you start adding files and directories.)

Preallocating space in this way the drawback that if you overestimate the requirements, you’ve wasted the better part of whatever space you preallocated. Furthermore, at the end of the day, all those zeros are just placeholder data, they aren’t required for the filesystem to do its job.

It would be ideal if you could provide a promise that the space will be available when it’s needed, and allow the filesystem to dynamically grow into that space without placeholder data, wouldn’t it?

In fact, that is exactly what a sparse file is for. Sparse files have the ability to start small and dynamically grow into a pre-declared maximum size; this is referred to as thin provisioning.

For more about sparse files, see the Arch Linux wiki entry.

So now create the sparse file and associate it with a loop device

Since you’re using a sparse file, you can be more relaxed about allocating way more than you will ever need. For an ssh config directory + Git repo, you might as well go all out and create a 1G sparse file like so:

$ mkdir -pv ~/imgs
$ dd of=~/imgs/ssh_config.img bs=1 count=0 seek=1G

That’s it. If you run ls -l ~/imgs it will say the file is 1GB, but if you run du -h ~/imgs/ssh_config.img it will say 0.

Then to do the loop association:

# losetup /dev/loop0 ~/imgs/ssh_config.img

(This could fail for a few different reasons, such as if some other process is already using /dev/loop0, or if you recently upgraded your kernel but haven’t bothered to reboot yet.)

Create the filesystem

You’ll need to decide which filesystem to use. I’ll use ext4. Most importantly for our purposes, it supports ACLs. Furthermore, it is relatively performant for small files (which is what we’re dealing with). Compared to other filesystems in the same niche (such as ReiserFS), ext4 is much more robust and widely supported.

# mkfs.ext4 /dev/loop0

(If you do a du -h on your image now you should see it has already ballooned up from 0 bytes to 33MB, just because of the overhead of adding an ext4 filesystem.)

It seems like ext4 now has ACL enabled by default, whereas I think in the past that wasn’t the case. Whether it is already enabled or not, it is not going to hurt to run the following command just to make sure:

# tune2fs -o acl /dev/loop0

Define permissions and set up ACLs

Let’s prepare a place that can be reused for mounting this filesystem into the future:

$ mkdir -pv ~/src/dotfiles/ssh_config
# mount /dev/loop0 ~/src/dotfiles/ssh_config

In a freshly-minted ext4 filesystem the contents are owned by root, world-readable, but not writable by you. Fix that:

# chown <username>:<groupname> ~/src/dotfiles/ssh_config/
$ chmod 0700 ~/src/dotfiles/ssh_config/

What are ACLs?

If you’re familiar with the idea of setting a umask when mounting NTFS, ACLs are kind of the Unixy equivalent. They are a little less intuitive and furthermore require extra programs (setfacl, getfacl) to manipulate and query them.

Again, the Arch Linux wiki has a decent primer about ACLs. The information in man 1 setfacl is useful too.

Enforce the equivalent of umask 177

To the default list, add rules stripping away all rights based on group and other:

$ setfacl -dm o:0 ~/src/dotfiles/ssh_config/
$ setfacl -dm g::--- ~/src/dotfiles/ssh_config/

Do a couple of tests to ensure everything looks right:

$ touch ~/src/dotfiles/ssh_config/testing.txt
$ mkdir ~/src/dotfiles/ssh_config/testing
$ ls -Ral ~/src/dotfiles/ssh_config/testing

You should see that the new file has -rw------- and the new directory has drwx------+. The + means the subdirectory has inherited the default ACL rules. Thus by induction we can expect that ACLs would continue to recurse into nested subdirectories, however deep we like.

This is exactly the behaviour we set out to achieve! Now it is just a matter of setting up ssh and Git in the usual way.

Add ssh config files

Most people have all their ssh configs in a single monolithic file: ~/.ssh/config. In fact, until relatively recently, this was the only way to do it.

OpenSSH versions newer than 7.3p1 (which was released 2016-08-01) do now have an Include directive. It is very useful to have ~/.ssh/config.d/ populated with multiple files called from ~/.ssh/config with the line Include config.d/*, but to keep the tutorial simple, let’s assume that you have you are in the first group and everything is in ~/.ssh/config.

For starters, bring the config file into the new filesystem:

$ cd ~/src/dotfiles/ssh_config
$ cp ~/.ssh/config ./
$ ln -svf $PWD/config ~/.ssh/config

You might wish to test that you can still ssh out after doing that.

Now get Git in on the action:

$ git init
$ git add config
$ git commit -m "Start version controlling ssh config"

Add an fstab entry

Without an fstab entry, your ssh config will be left locked away each time you reboot, most likely preventing you from sshing, or at least playing havoc with your preferred way of doing things.

It turns out that the process which reads fstab during boot time is intelligent enough to detect when it is working with an image file and automatically set up a loop device for it. Therefore all you have to do to keep your ssh config persistent is add a single line like so:

/home/<username>/imgs/ssh_config.img /home/<username>/src/dotfiles/ssh_config ext4 rw,relatime,data=ordered 0 2


While it is fresh in your mind, do a reboot now and test that sshing out to somewhere works as expected. (You can also check whether the image is mounted using lsblk or df -h.)