06 March 2017

Dynamically generating host entries for SSH client config

I will at some point in the future blog about how cool Ansible inventory files are, and how I re-purposed them for Shepherd.
But for now, given that I have all these Ansible inventory files lying around, why not use them to generate my ssh config file, instead of having to update it by hand every time I add a new host to my collection?

Background: the command-line OpenSSH client reads both /etc/ssh/ssh_config and ~/.ssh/config to get the default values for options as well as any host-specific options.  This means that you can define the hosts that you'll be logging into using friendly names.  You can also set global settings, or just settings for specific groups of hosts using wildcards.

I use a Makefile that reads all the inventory files that I have specified and converts them into host entries.  It combines these with the contents of all the files in ~/.ssh/config.d (this is non-standard) and creates ~/.ssh/config .  Note that you can't edit this file manually any more; if you do, either your changes will be overwritten or it will prevent updates to one of the source files from being used (if the target file is newer than the source, which is how make determines if a dependency has changed and therefore the target has to be rebuilt).

Here is a sanitised version of my Makefile:
# User options
CREATED_FILES=xyz.conf mine.conf

.PHONY: all
all: config

# User dependencies
tmp/xyz.conf: ~/Work/XYZ/Hosts/hosts.txt
tmp/mine.conf: ~/Transfer.unison/Geekery/Hosts/hosts.txt

# Actual targets and variables
config: tmp/000_header.conf $(CREATED_FILE_PATHS) config.d/*.conf
        if [ ! -f $@.bak ] ; then cp -p $@ $@.bak ; fi
        for file in tmp/000_header.conf $(CREATED_FILE_PATHS) ; do cat $$file ; echo ; done > $@
        for file in config.d/*.conf ; do echo "# $(CURDIR)/$$file" ; cat $$file ; echo ; echo ; done >> $@

        mkdir $@

    echo "# $^" > '$@'
    sed -e '/^#/d' \
        -e 's/[[:space:]]*#.*//' \
        -e '/ansible_ssh_host=/ !d' \
        -e 's/\(cloud_provider\|cloud_region\|cloud_instance_id\)=\([^[:space:]]*\)[[:space:]]*//g' \
        -e 's/^\([^[:space:]]*\)[[:space:]]*alias=\([^[:space:]]*\)[[:space:]]*/Host \1 \2\n/' \
        -e 's/^\(Host [^\n]*\)[[:space:]]*alias=\([^[:space:]]*\)[[:space:]]*/\1 \2\n/' \
        -e 's/^\([^H][^[:space:]]*\)[[:space:]]*/Host \1\n/' \
        -e 's/ansible_ssh_host=\([^[:space:]]*\)[[:space:]]*/  HostName \1\n/' \
        -e 's/ansible_ssh_user=\([^[:space:]]*\)[[:space:]]*/  User \1\n/' \
        -e 's/ansible_ssh_private_key_file=\("[^"]*"\)/  PubkeyAuthentication yes\n  IdentitiesOnly yes\n  IdentityFile \1/' \
        -e 's/ansible_[^[:space:]]*=[^[:space:]]*[[:space:]]*//g' \
        -e 's/^\(Host [^\n]*\($(LOGLEVEL_TWEAK_REGEXP)\)[^\n]*\)/\1\n  LogLevel Error/' \
        -e 's/\n\([[:space:]]*HostName [^\n]*\($(LOGLEVEL_TWEAK_REGEXP)\)[^\n]*\)/\n  LogLevel Error\n\1/' \
        -e 's/$$/\n/' \
      '$^' >> '$@'

tmp/000_header.conf: | tmp
    echo '# see ssh_config(5)' > $@
    echo '#' >> $@


To "run" the Makefile, use this command:
make -C .ssh
That's it.  You can run it as often as you like, and if no source files have been modified, the target file won't be re-created.


CREATED_FILES is the list of files that are to be created in ~/.ssh/tmp before being combined together (with the other files) to make the target file.  You also have to provide dependency lines (these are a cut-down make rule) that specify which inventory file is used to make each temporary file.

LOGLEVEL_TWEAK_REGEXP allows you specify hosts for which the "LogLevel Error" option will be set, which prevents /etc/issue.net from being blatted onto your terminal every time you run scp or rsync.  The regexp is matched against both the host title (including aliases) and the SSH hostname.

Dependency lines

Note that each file in CREATED_FILES has a matching line (including "tmp/") that specifies the source file.  Think of these as a pair; every time you add or remove a file from CREATED_FILES you have to add or remove the dependency line.


At least one file must be in .ssh/config.d -- but it can be empty.  I recommend having a file in this subdirectory called zzz_global.conf so its contents will come after all your host stanzas.  This the place to put your Host * stanza with its options, e.g. "ServerAliveInterval 30" which is great for preventing your ADSL modem from dropping connections.

If you don't have anything to put in LOGLEVEL_TWEAK_REGEXP, comment out the two lines that use it by putting "##" after the first single quote of each sed line.  Yes, sed suppports comments!

Labels: , , , ,

20 December 2016

Quick and dirty PHP autoloading using Composer


Why write your own PHP class autoloader when Composer provides a perfectly good one?  It supports both PSR-4 and PSR-0 Recommendations (a.k.a. PHP standards for how namespaces should be used with autoloading).  Fun fact: it's based on the Symfony framework's autoloader and has lots of optimisations and features built in.

Composer is normally used to install/update third party components into a PHP project, from Git repos or packagist.org .  But the same functionality that Composer uses to let your project access classes etc. from within those components, can also be used to access your project's own classes etc.  In other words, depending on how you configure it, the autoloader that's generated for your project by Composer will provide access to class files defined in both third party components and your project.

I made a repo called template-PHP-project that has a composer.json already configured for both PSR-4 class autoloading and an include path.


Simply clone template-PHP-project, rename the directory and then delete its .git subdirectory. Then you can add web files, class files etc. and install third party components using Composer. You can then run git init and commit all the files ready to push to a blank repo on your favourite Git hosting site.  Or use Mercurial... I won't judge you.

If you like, instead of removing .git just edit .git/config and change the origin. Or run git config remote."origin".url your-repo-url .

If you have an existing project, you're better off moving your files around to fit the structure below than trying to squeeze template-PHP-project's functionality into it.

Directory structure

You will get all of these files and directories by default (thanks to .keep files) except vendor/ (which is managed by Composer):
|-- app/
|   |-- classes/
|   `-- include/
|-- bin/
|   `-- demo.php
|-- composer.json
|-- composer.lock
|-- config/
|-- docroot/
|   `-- index.php
`-- vendor/

Composer config

First, edit composer.json and change XYZ to your project's namespace.  You'll have to create a subdirectory in classes with this name, and that's where you put your class files -- name them according to the standard.  (You can add multiple namespaces if you like.  Any of your project namespaces automatically support a hierarchy of sub-namespaces that are inferred from the subdirectories in the project namespace's subdirectory.)  Don't forget to leave the trailing backslashes in the composer.json definition.

Before your code will work, you'll have to get composer to create/update vendor/autoload.php and vendor/composer/ by running this command:
composer update
This has to be done every time you add a new class file.  Or when you modify the require statement.

Please test all your code after doing this as it will update versions and dependencies of third party components.  Don't forget to commit afterwards; see section below.


The supplied composer.json doesn't use a fallback directory for looking up namespace files.  All namespaces are expected to be explicitly declared.

Reminder regarding what to commit

This also covers what to keep in mind when deploying code.
  1. Always commit composer.lock
    Note: this will be erased by Composer if you don't require any third party components
  2. Never commit  vendor/
  3. On your server, Docker container or whatever, you are supposed to clone the repo (or extract it from a tarball) and then run composer install. If the repo is already there, run git pull and then composer install.
    will ensure you get all the correct, working, tested versions of third party components on every deploy.  (Nobody likes surprises in production.)
  4. Never run composer update on production.


See demo.php for an example of a PHP script that uses both an autoloaded class (assumes namespace Demo has been configured in composer.json) and a file in the pre-configured include path.

This script relies on the following files (re-creating them is left as an exercise for the reader):
  • app/include/foo.inc
  • app/classes/Demo/Util.php

Other features

Include path

As mentioned, my composer.json sets up app/include/ as a place from which you can include files without needing to specify the path.  The composer docs state that this feature is deprecated, because ultimately with an autoloader, you can do everything with classes.  Therefore you shouldn't need to write include files as part of a new project, given that:-
  • defines can be replaced by class constants, e.g. Foo::FROB_WEIGHT
  • functions can be moved to public static methods in a class, e.g. Foo::measure()
  • global variables are stupid; seriously, use a static class variable if you need one, but please consider writing some accessor methods
All of the above replacements can take advantage of namespaces and autoloading.

Automatic includes

I'm not currently using Composer's files feature, but it's easy enough to add to the autoload block for projects that should include a file into every PHP page that uses the autoloader.


22 November 2016

Why Git?

To give you an example of how helpful Git can be, I've modified an e-mail I sent to a client. They had customised a library I'd written for them, and I need to fix a bug both in my original version and the modified version of the include file they'd sent me. I did these steps:
  1. Create a local repo inside my source directory
  2. Commit all the files into the local Git database
  3. Create and switch to a new branch called 'test'
  4. Save their modified version of main.inc
  5. Commit the change in the 'test' branch
  6. Switch back to the 'master' branch
  7. Fix the bug and commit
  8. Switch to the 'test' branch
  9. Merge all changes from the 'master' branch
This automatically applies the change from step 7 and commits the change in the current branch. It might seem like a lot of work to apply one change that could easily be done again to a different version of the file, but consider that it scales; even complex changes can be merges with only cursory review required. (Unless both branches have changed in contradictory ways (i.e. same parts of the files have been changed) since the two branches diverged. In this case a merge conflict will happen, which is easy to resolve by "collapsing" each pair of conflicting changes that have been put next to each other in the files that couldn't be merged automatically. A commit is then necessary because obviously it couldn't be done when a conflict was detected.)


08 October 2016

How to bootstrap Python VirtualEnvs with a specific Python version

This shows how to create a "bootstrap" VirtualEnv, which will allow the creation of (mostly) self-contained VirtualEnvs that are used for projects.
  1. Install OS dependencies like libsqlite3-dev
  2. Build Python from source:
    1. Download from https://www.python.org/download/releases
    2. Extract tarball
    3. cd into directory
    4. Run "./configure --prefix=/opt/python3"
    5. Run "make"
    6. Run "sudo make install"
  3. Manually grab the latest virtualenv module to avoid dependency problems with the existing Python installation:
    1. Download virtualenv-X.Y.Z.tar.gz from https://pypi.python.org/pypi/virtualenv#downloads
    2. Extract into /opt/virtualenv-X.Y.Z
  4. Create and configure the "bootstrap" VirtualEnv:
    1. E.g. run "/opt/virtualenv-15.0.3/virtualenv.py -p /opt/python3/bin/python3 lib/py3venv"
    2. Run "lib/py3venv/bin/pip install virtualenv"
Now, whenever you need to create a self-contained VirtualEnv for a project, run this command, substituting the placeholder for the desired output directory:
lib/py3venv/bin/virtualenv env_dir
The VirtualEnv directory that is created will contain its own Python executable, but will rely on the global packages in /opt/python3.

See The "Virtual Environments and Packages" section of the Python tutorial for more info.

Labels: ,

07 October 2016

Django problem: sqlite error when running"manage.py migrate"

Just ran into a problem with creating a Django virtualenv.  I built Python from source, but I didn't have the libsqlite3-dev library installed.  (I'm using Debian, but the same problem applies to Ubuntu.) 

So when I tried to run "demo/manage.py migrate" in my Django virtualenv, I got this error:
ImportError: No module named 'pysqlite2'
Once I installed libsqlite3-dev rebuilt Python with "make clean install", it worked!

The fix is documented in the Stack Overflow post "No module named _sqlite3".


24 March 2015

How to make an SVN repo accessible by Web and SSH

Would you like to publish an SVN repo over the web (read-only of course) so people can see your dotfiles, etc.?  If so, it takes some effort to make it work.

The examples below apply to Debian, but with minor tweaks (e.g. which config files you edit) it should work anywhere.

Apache setup

This assumes that you want your repo available at http://my.site/svn/xyz .  (The docs will tell you how to host all repos in a given directory, rather than one specific one.)  It also assumes that your repo is in /srv/svn/xyz on your web server.

First, Apache will have to be augmented with stuff to let it use the SVN libraries to interact with the repo.   This lets Apache present it as a sort of filesystem over the web, which supports both browser access as well as access by SVN clients.
sudo aptitude install libapache2-svn
## sudo service apache2 restart   # done by the above
 Then add the following to your vhost file:
# -- SVN --
<Location /svn/xyz>
  DAV svn
  SVNPath /srv/svn/xyz
    Deny from all

  # Uncomment and tweak this to enable write access for certain people.
  ## AuthzSVNAccessFile /etc/httpd/security/svn.access
  ## Require valid-user
  ## AuthType Basic
  ## Authname "Subversion Test repository"
  ## AuthUserFile /etc/httpd/security/svn.auth
  ##   Require valid-user
  ## </LimitExcept>
I don't accept any liability for the above, and it is your responsibility to test that it is secure, e.g. by trying things that should be blocked and checking your Apache logs to ensure that they are.

Also, be sure that there's been nothing sensitive committed to the repo, even in past revisions.  Don't commit your whole home directory, because you don't want to reveal things like your API tokens (e.g. in .gitconfig) etc.

Special considerations 

Web access will happen via the www-data user, which on Debian is the user that Apache runs as.  This still needs to write to files in order for locking to work.

In the past, I got it working the past by tweaking group permissions and probably the umask (possibly using wrapper scripts).

But this is the 21st century, and everything should support POSIX ACLs, which are a kind of extended permission system.  This example assumes the existence of an svnusers group of which the required people, who will access the repo over ssh, are members.
cd /path/to/repo

# set the ACL mask to stop files being made executable
find . -type f | sudo xargs setfacl -m m:rw

# set blanket permissions, but don't recalculate the ACL mask
sudo setfacl -R -n -m g:svnusers:rwx,u:www-data:rwx

# set default perms on directories (no effect on files) so new files get correct perms
sudo setfacl -R -d -m g:svnusers:rwx,u:www-data:rwx .
Note that this won't have to be done again in future, because any new files (and directories for that matter) will automatically be given the correct permissions thanks to the last line.  "-d" automatically transforms regular ACL entries into default ACL entries, e.g. u:www-data:rwx -> default:u:www-data:rwx .

Labels: ,

10 March 2013

Resume from hibernate failed silently on Debian

After I changed my swap partition, my computer hibernated fine but would not resume, i.e. it booted normally, as if I had done a hard power-off beforehand.

This happened because hibernating saves the contents of RAM etc. into your swap partition.  Therefore, <initramfs>/conf/conf.d/resume now contained an incorrect UUID (of the old, no-longer-existing partition).  See Debian linux mint, resume after hibernation fails for how to fix this.  Don't forget to modify your /etc/fstab and also regenerate the blkid cache by running "sudo blkid -p".

(See Wikipedia.org's Initrd article for background and the initramfs-tools package for how initramfs is managed under Debian.)

You have to shut down after changing your swap partition.  For some reason the kernel's power management subsystem won't hibernate to the new swap partition until after the next boot.  If you try, you'll get the error "Cannot find swap device, try swapon -a" on the console.

PS -- Don't be fooled by "PM: Resume from disk failed." in /var/log/kern.log (this is a normal error that you get when you boot after shutting down).  Debian's initramfs will check for the presence of /dev/disk/by-uuid/$resume and /sys/power/resume and only run <initramfs>/bin/resume if they both exist.  If they don't, it doesn't do anything and just continues to boot.