28 September 2017

Installing Webmin and Virtualmin from packages

Please note: this guide has not yet had a complete run-through from scratch.  If you hit problems, please e-mail alastair@plug.org.au

Do you want to install Webmin and Virtualmin from packages, i.e. not using the official install script?  The process is finicky, but can be done, as shown here.  One reason you might wish to to this is that you might not want the default "kitchen sink" installation and all the dependencies it brings in.  (Or you might not like how the official install script modifies /etc/apt/sources.list rather than adding a file called /etc/apt/sources.list.d/webmin.list .  Or you might be concerned about the fact that the install script is served from a non-HTTPS URL and is therefore open to modification in transit or other security breaches.  Et cetera.)

The end result of this process is a server that uses PHP, fcgid and Apache.  Unlike a standard Virtualmin installation, it is a "bare bones" server that can be used for web hosting only.  It assumes that other features like e-mail hosting and DNS are provided by the cloud or separate servers.
This guide assumes that you are using Debian or Ubuntu.  It might be possible to apply some of the instructions below on other systems, but many won't work unless substitute commands are used. Note: older versions of Debian (wheezy and earlier) and Ubuntu (precise and earlier) don't have the apt command; you can use aptitude instead.  (FYI, aptitude is installable in recent OS releases if you prefer it.)

Install Linux

This is out of scope for this guide.

Linux Preparation

As an alternative to the steps in the sub-sections below, you can install the required packages manually, but there are a lot of them to remember.  http://al.id.au/svn/tools/debian/apache-fcgid/  has a script called setup-fcgid, but note that it currently works only on older releases of Ubuntu (pre Xenial) and Debian (pre Stretch).

Setup LAMP and Postfix

  1. Run sudo tasksel
  2. Move text cursor to "LAMP server" and press Enter
  3. Move text cursor to "Mail server" and press Enter
    • Virtualmin won't work without Postfix etc.
    • Note that Webmin's Exim support is listed as experimental, so on Debian (where the default MTA is Exim) you might want to install Postfix instead if you have no preference
  4. Press Tab to go to the <Ok> button and press Enter
  5. When prompted to set a MySQL server password, do so unless you will be uninstalling MySQL as per the next sub-section; in this case, you will be prompted multiple times, so press Enter each time
  6. When the Postfix configuration dialog pops up, read the intro carefully and then choose the option that applies to you

Install extra packages

If you are running Ubuntu Xenial (or newer) or Debian Stretch (or newer), run this command:
sudo apt install libapache2-mod-fcgid apache2-suexec-custom php-cgi
Otherwise, run this command:
sudo apt install libapache2-mod-fcgid apache2-suexec-custom php5-cgi

Alter the PHP configuration

This is needed to prevent Apache's built-in PHP module from trying to run PHP files.

If you are running Ubuntu Xenial (or newer) or Debian Stretch (or newer), run this command:
sudo rm /etc/apache2/mods-enabled/php7.0.conf
Otherwise, run this command:
sudo rm /etc/apache2/mods-enabled/php5.conf

Remove MySQL server if necessary

Note: Only do this if you'll be using a separate database server.
  1. Run sudo apt purge mysql-server mysql-server-5.7 mysql-server-core-5.7

Webmin and Virtualmin Packages

Keys

  1. Run mkdir /tmp/virtualmin-keys
  2. Run wget -N -P /tmp/virtualmin-keys http://software.virtualmin.com/lib/{RPM-GPG-KEY-webmin,RPM-GPG-KEY-virtualmin,RPM-GPG-KEY-virtualmin-6}
  3. Run shasum -a 256 /tmp/virtualmin-keys/{RPM-GPG-KEY-webmin,RPM-GPG-KEY-virtualmin,RPM-GPG-KEY-virtualmin-6}
  4. Confirm that you get the output shown below.  (This is necessary because software.virtualmin.com doesn't have a valid HTTPS certificate and thus only works from the non-HTTPS URL.  Therefore you are required to perform extra checking to make sure you'll get the real packages.)
Required output:
36a563bec98a9894065d5f45fbfe58ef51985aecc561569e6288b009ef28f251  /tmp/virtualmin-keys/RPM-GPG-KEY-webmin
08294ec28b36249adf8d51352153235d2467b081d2f3e133771da9100b6fbc81 
/tmp/virtualmin-keys/RPM-GPG-KEY-virtualmin
d8bd1baa45a96a837efe1cd535f8a9325aff18751e8571cf3e792c5ea3ffb039  /tmp/virtualmin-keys/RPM-GPG-KEY-virtualmin-6

APT config

  1. Run sudoedit /etc/apt/sources.list.d/webmin.list
  2. Paste in the contents shown below
    1. Note: the lines may appear to contain a single URL, but they don't, so don't remove any spaces
  3. Modify the two placeholders as follows:
    1. {{OS}} -- debian or ubuntu
    2. {{RELEASE}} -- the codename of your Distro release
      • You can find this by running lsb_release --short --codename
      • For stretch, use jessie instead (the directory for stretch is missing from the server) 
This is the contents of the file mentioned above:
deb http://software.virtualmin.com/gpl/{{OS}}/ virtualmin-{{RELEASE}} main
deb http://software.virtualmin.com/gpl/{{OS}}/ virtualmin-universal main

Installation 

  1. Run sudo apt-get update
  2. Run sudo apt install webmin-virtual-server virtualmin-base

Configuration

First, configure the apache2-suexec-custom package:
  1. Run sudoedit /etc/apache2/suexec/www-data
  2. Change the first line to /home
  3. Save and quit (press Ctrl-X)

Next, log into Webmin and configure it:
  1. Go to https://host.address:10000/
    • Replace host.address with the IP Address or full host name of your server
  2. Log in with your normal Unix account
  3. In the left-hand menu, click on Servers and then Apache Webserver
  4. Click on the Global configuration tab in the right-hand section
  5. Click on Configure Apache Modules
  6. Tick the actions, fcgid and suexec boxes
  7. Click [Enable Selected Modules]
You can now click on Virtualmin at the top of the left-hand menu and create "Virtual Servers", which are simply managed Apache vhosts.

Caveats

If you want to host a site named after the primary address of the server, it might not work, because the Apache default vhost will try to serve those requests instead.  (This only happens if the server's hostname is configured to match the primary address.)

To fix this, do this:
  1. Run sudoedit /etc/apache2/sites-available/000-default.conf 
  2. Uncomment the ServerName directive
    • The value should be nonsense, as the point is to set it to something that will never match a real vhost
  3. Save and quit (press Ctrl-X)
  4. Run sudo service apache2 reload

Labels: ,

01 September 2017

Resizing a disk on a cloud compute instance (AWS or Google)

Resizing a disk on a cloud compute instance

Diagnosis

If you get an error when installing/upgrading a package that looks like this:

dpkg: error processing archive /var/cache/apt/archives/linux-image-3.13.0-62-generic_3.13.0-62.102_amd64.deb (--unpack):
 cannot copy extracted data for './boot/vmlinuz-3.13.0-62-generic' to '/boot/vmlinuz-3.13.0-62-generic.dpkg-new': failed to write (No space left on device)
...then your root file system is full. This can also be checked like this:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            129G   129G   0M 100% /
If this has happened, one way to proceed is to add a new file system and move data to it (e.g. so that /home is a separate file system that is mounted into the tree but not using any of the allocated space on the root file system any more).  This method is not covered here.

Another way is to expand the size of the existing root file system.  That is what this article covers.  The phases below apply to any file system that has become full.  (No longer do you need to shut down the instance or unmount the file system, snapshot the volume and create a new, larger one from the snapshot.)

How to fix 

Warning: A mistake or mis-interpretation when following these steps could destroy your data.  Please use caution, and unless you are an intermediate or above Linux user, consider asking a system administrator who is familiar with the tools presented to perform the task.

Phase 1 -- Expand virtual disk

If you are using Amazon Web Services:
  1. Log in to the AWS console
  2. Make sure you are in the EC2 service section
  3. Click on Volumes in the left-hand menu
  4. Find the volume that is attached to your instance (sort by the Attachment column)
    • Ensure you have found the correct one if that instance has multiple volumes
  5. Right-click on the volume and choose Modify
  6. Alter the size
  7. Click [Save]
 If you are using Google Compute Engine:
  1. Log in to the console
  2. In the drop-down menu, choose the project that your compute instance belongs to
  3. In the Resources box, click on Compute Engine
  4. Click on the name of your instance in the list
  5. Scroll down to the "Boot disk and local disks" section
  6. Click on the relevant disk
  7. Click [EDIT] in the toolbar
  8. Change the Size value
  9. Click [Save]
Other virtualisation systems (like VMware) or storage management systems (like LVM), may allow you to expand a disk volume.  If your host uses one of these systems, research and apply whatever method is available to you, then continue with the next phases.

Phase 2a -- Expand partition

In the steps below, replace /dev/sda with the device name of the root file system, minus the partition number.
  1. Log in via SSH 
  2. Run lsblk to determine the device of the root file system
    • See examples below
    • The device name is given in the first column, and in this case is not preceded with "/dev/"
    • The relevant disk or partition is the one with "/" in the MOUNTPOINT column
  3. If the root file system's device is a disk (not a partition) skip to "Phase 3 -- Expand file system"
    • Virtual disks under Linux have device names like "/dev/sda" or "/dev/xvdf"
    • Partitions are a way of subdividing disks; they have device names like "/dev/sda1" or "/dev/xvdf2"
    • A file system can reside on a partition or use the whole disk; in the latter case there is no partition table and so trying to list/manipulate it won't work
  4. Tell Linux to detect the new disk size
    1.  N/A -- this is done automatically by the kernel
  5. Install parted
    1.  Run "sudo apt-get update ; sudo apt-get install parted"
  6. Run parted to expand the partition
    1. Run "sudo parted /dev/sda"
      • Observe the note above about /dev/sda
    2. Check if there is more than one partition on the disk:
      1. At the parted prompt, Run "print list"
        • Take note of the disk size displayed on the second line of output
      2. If the root file system's partition (see step 2) is not the last one on the disk, follow the steps at "Phase 2b -- Moving and expanding partitions" and then return here.
    3.  At the parted prompt, Run "resize 1 start size"
      • If appropriate, replace "1" with the root file system's partition number
      • Replace "start" with the partition's existing Start value from the table show by "print list" above
      • Replace "size" with the disk size value noted above
    4. Follow the instructions provided by parted
    5. At the parted prompt, Run "quit"
  7. Tell Linux to update its view of the partitions
    1.  Run "sudo kpartx -u /dev/sda"
      • Observe the note above about /dev/sda
lsblk examples
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0    0  15G  0 disk /
 or:
$ lsblkNAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda      8:0    0  10G  0 disk
└─sda1   8:1    0  10G  0 part /


Phase 2b -- Moving and expanding partitions

Warning: Skip to Phase 3 unless Phase 2a directed you to follow these steps.

TBA

Phase 3 -- Expand file system

This is unnecessary if your root file system's device is a partition, as you will have resized both in Phase 2.

Firstly, run "df -hT" and take note of the value for the root file system in the Type column.

If the file system type is ext4, follow these steps:
  1. Run "sudo apt-get update ; sudo apt-get install e2fsprogs"
  2. Run "sudo resize2fs /dev/sda"
    • Observe the note in Phase 2a about /dev/sda
  3. Follow the instructions provided
 If the file system type is xfs, follow these steps:
  1. Run "sudo apt-get update ; sudo apt-get install xfsprogs"
  2. Run "sudo xfs_growfs /dev/sda"
    • Observe the note in Phase 2a about /dev/sda
  3. Follow the instructions provided
If the file system type is rootfs, look for the other line in the df output that is also mounted on "/" (don't ask me why there's two) and follow the steps above for that file system type.

If  the file system type is none of the above, do a web search for the type and the word "resize".

Phase 4 -- Verification

Run "df -h" and you should see that you now have plenty of free space on your root file system.

Labels: , ,

06 March 2017

Dynamically generating host entries for SSH client config

I will at some point in the future blog about how cool Ansible inventory files are, and how I re-purposed them for Shepherd.
But for now, given that I have all these Ansible inventory files lying around, why not use them to generate my ssh config file, instead of having to update it by hand every time I add a new host to my collection?

Background: the command-line OpenSSH client reads both /etc/ssh/ssh_config and ~/.ssh/config to get the default values for options as well as any host-specific options.  This means that you can define the hosts that you'll be logging into using friendly names.  You can also set global settings, or just settings for specific groups of hosts using wildcards.

I use a Makefile that reads all the inventory files that I have specified and converts them into host entries.  It combines these with the contents of all the files in ~/.ssh/config.d (this is non-standard) and creates ~/.ssh/config .  Note that you can't edit this file manually any more; if you do, either your changes will be overwritten or it will prevent updates to one of the source files from being used (if the target file is newer than the source, which is how make determines if a dependency has changed and therefore the target has to be rebuilt).

Here is a sanitised version of my Makefile:
# User options
CREATED_FILES=xyz.conf mine.conf
LOGLEVEL_TWEAK_REGEXP=xyz.net

.PHONY: all
all: config

# User dependencies
tmp/xyz.conf: ~/Work/XYZ/Hosts/hosts.txt
tmp/mine.conf: ~/Transfer.unison/Geekery/Hosts/hosts.txt


# Actual targets and variables
CREATED_FILE_PATHS = $(CREATED_FILES:%=tmp/%)
config: tmp/000_header.conf $(CREATED_FILE_PATHS) config.d/*.conf
        if [ ! -f $@.bak ] ; then cp -p $@ $@.bak ; fi
        for file in tmp/000_header.conf $(CREATED_FILE_PATHS) ; do cat $$file ; echo ; done > $@
        for file in config.d/*.conf ; do echo "# $(CURDIR)/$$file" ; cat $$file ; echo ; echo ; done >> $@

tmp:
        mkdir $@

$(CREATED_FILE_PATHS):
    echo "# $^" > '$@'
    sed -e '/^#/d' \
        -e 's/[[:space:]]*#.*//' \
        -e '/ansible_ssh_host=/ !d' \
        -e 's/\(cloud_provider\|cloud_region\|cloud_instance_id\)=\([^[:space:]]*\)[[:space:]]*//g' \
        -e 's/^\([^[:space:]]*\)[[:space:]]*alias=\([^[:space:]]*\)[[:space:]]*/Host \1 \2\n/' \
        -e 's/^\(Host [^\n]*\)[[:space:]]*alias=\([^[:space:]]*\)[[:space:]]*/\1 \2\n/' \
        -e 's/^\([^H][^[:space:]]*\)[[:space:]]*/Host \1\n/' \
        -e 's/ansible_ssh_host=\([^[:space:]]*\)[[:space:]]*/  HostName \1\n/' \
        -e 's/ansible_ssh_user=\([^[:space:]]*\)[[:space:]]*/  User \1\n/' \
        -e 's/ansible_ssh_private_key_file=\("[^"]*"\)/  PubkeyAuthentication yes\n  IdentitiesOnly yes\n  IdentityFile \1/' \
        -e 's/ansible_[^[:space:]]*=[^[:space:]]*[[:space:]]*//g' \
        -e 's/^\(Host [^\n]*\($(LOGLEVEL_TWEAK_REGEXP)\)[^\n]*\)/\1\n  LogLevel Error/' \
        -e 's/\n\([[:space:]]*HostName [^\n]*\($(LOGLEVEL_TWEAK_REGEXP)\)[^\n]*\)/\n  LogLevel Error\n\1/' \
        -e 's/$$/\n/' \
      '$^' >> '$@'

tmp/000_header.conf: | tmp
    echo '# see ssh_config(5)' > $@
    echo '#' >> $@
    echo '# DO NOT EDIT -- ANY CHANGES WILL BE OVERWRITTEN BY Makefile' >> $@

Usage 

To "run" the Makefile, use this command:
make -C .ssh
That's it.  You can run it as often as you like, and if no source files have been modified, the target file won't be re-created.

Options

CREATED_FILES is the list of files that are to be created in ~/.ssh/tmp before being combined together (with the other files) to make the target file.  You also have to provide dependency lines (these are a cut-down make rule) that specify which inventory file is used to make each temporary file.

LOGLEVEL_TWEAK_REGEXP allows you specify hosts for which the "LogLevel Error" option will be set, which prevents /etc/issue.net from being blatted onto your terminal every time you run scp or rsync.  The regexp is matched against both the host title (including aliases) and the SSH hostname.

Dependency lines

Note that each file in CREATED_FILES has a matching line (including "tmp/") that specifies the source file.  Think of these as a pair; every time you add or remove a file from CREATED_FILES you have to add or remove the dependency line.

Caveats

At least one file must be in .ssh/config.d -- but it can be empty.  I recommend having a file in this subdirectory called zzz_global.conf so its contents will come after all your host stanzas.  This the place to put your Host * stanza with its options, e.g. "ServerAliveInterval 30" which is great for preventing your ADSL modem from dropping connections.

If you don't have anything to put in LOGLEVEL_TWEAK_REGEXP, comment out the two lines that use it by putting "##" after the first single quote of each sed line.  Yes, sed suppports comments!

Labels: , , , ,

20 December 2016

Quick and dirty PHP autoloading using Composer

Overview

Why write your own PHP class autoloader when Composer provides a perfectly good one?  It supports both PSR-4 and PSR-0 Recommendations (a.k.a. PHP standards for how namespaces should be used with autoloading).  Fun fact: it's based on the Symfony framework's autoloader and has lots of optimisations and features built in.

Composer is normally used to install/update third party components into a PHP project, from Git repos or packagist.org .  But the same functionality that Composer uses to let your project access classes etc. from within those components, can also be used to access your project's own classes etc.  In other words, depending on how you configure it, the autoloader that's generated for your project by Composer will provide access to class files defined in both third party components and your project.

I made a repo called template-PHP-project that has a composer.json already configured for both PSR-4 class autoloading and an include path.

Usage

Simply clone template-PHP-project, rename the directory and then delete its .git subdirectory. Then you can add web files, class files etc. and install third party components using Composer. You can then run git init and commit all the files ready to push to a blank repo on your favourite Git hosting site.  Or use Mercurial... I won't judge you.

If you like, instead of removing .git just edit .git/config and change the origin. Or run git config remote."origin".url your-repo-url .

If you have an existing project, you're better off moving your files around to fit the structure below than trying to squeeze template-PHP-project's functionality into it.

Directory structure

You will get all of these files and directories by default (thanks to .keep files) except vendor/ (which is managed by Composer):
|-- app/
|   |-- classes/
|   `-- include/
|-- bin/
|   `-- demo.php
|-- composer.json
|-- composer.lock
|-- config/
|-- docroot/
|   `-- index.php
`-- vendor/

Composer config

First, edit composer.json and change XYZ to your project's namespace.  You'll have to create a subdirectory in classes with this name, and that's where you put your class files -- name them according to the standard.  (You can add multiple namespaces if you like.  Any of your project namespaces automatically support a hierarchy of sub-namespaces that are inferred from the subdirectories in the project namespace's subdirectory.)  Don't forget to leave the trailing backslashes in the composer.json definition.

Before your code will work, you'll have to get composer to create/update vendor/autoload.php and vendor/composer/ by running this command:
composer update
This has to be done every time you add a new class file.  Or when you modify the require statement.

Please test all your code after doing this as it will update versions and dependencies of third party components.  Don't forget to commit afterwards; see section below.

Note

The supplied composer.json doesn't use a fallback directory for looking up namespace files.  All namespaces are expected to be explicitly declared.

Reminder regarding what to commit

This also covers what to keep in mind when deploying code.
  1. Always commit composer.lock
    Note: this will be erased by Composer if you don't require any third party components
  2. Never commit  vendor/
  3. On your server, Docker container or whatever, you are supposed to clone the repo (or extract it from a tarball) and then run composer install. If the repo is already there, run git pull and then composer install.
    composer.lock
    will ensure you get all the correct, working, tested versions of third party components on every deploy.  (Nobody likes surprises in production.)
  4. Never run composer update on production.

Example

See demo.php for an example of a PHP script that uses both an autoloaded class (assumes namespace Demo has been configured in composer.json) and a file in the pre-configured include path.

This script relies on the following files (re-creating them is left as an exercise for the reader):
  • app/include/foo.inc
  • app/classes/Demo/Util.php

Other features

Include path

As mentioned, my composer.json sets up app/include/ as a place from which you can include files without needing to specify the path.  The composer docs state that this feature is deprecated, because ultimately with an autoloader, you can do everything with classes.  Therefore you shouldn't need to write include files as part of a new project, given that:-
  • defines can be replaced by class constants, e.g. Foo::FROB_WEIGHT
  • functions can be moved to public static methods in a class, e.g. Foo::measure()
  • global variables are stupid; seriously, use a static class variable if you need one, but please consider writing some accessor methods
All of the above replacements can take advantage of namespaces and autoloading.

Automatic includes

I'm not currently using Composer's files feature, but it's easy enough to add to the autoload block for projects that should include a file into every PHP page that uses the autoloader.

Labels:

22 November 2016

Why Git?

To give you an example of how helpful Git can be, I've modified an e-mail I sent to a client. They had customised a library I'd written for them, and I need to fix a bug both in my original version and the modified version of the include file they'd sent me. I did these steps:
  1. Create a local repo inside my source directory
  2. Commit all the files into the local Git database
  3. Create and switch to a new branch called 'test'
  4. Save their modified version of main.inc
  5. Commit the change in the 'test' branch
  6. Switch back to the 'master' branch
  7. Fix the bug and commit
  8. Switch to the 'test' branch
  9. Merge all changes from the 'master' branch
This automatically applies the change from step 7 and commits the change in the current branch. It might seem like a lot of work to apply one change that could easily be done again to a different version of the file, but consider that it scales; even complex changes can be merges with only cursory review required. (Unless both branches have changed in contradictory ways (i.e. same parts of the files have been changed) since the two branches diverged. In this case a merge conflict will happen, which is easy to resolve by "collapsing" each pair of conflicting changes that have been put next to each other in the files that couldn't be merged automatically. A commit is then necessary because obviously it couldn't be done when a conflict was detected.)

Labels:

08 October 2016

How to bootstrap Python VirtualEnvs with a specific Python version

This shows how to create a "bootstrap" VirtualEnv, which will allow the creation of (mostly) self-contained VirtualEnvs that are used for projects.
  1. Install OS dependencies like libsqlite3-dev
  2. Build Python from source:
    1. Download from https://www.python.org/download/releases
    2. Extract tarball
    3. cd into directory
    4. Run "./configure --prefix=/opt/python3"
    5. Run "make"
    6. Run "sudo make install"
  3. Manually grab the latest virtualenv module to avoid dependency problems with the existing Python installation:
    1. Download virtualenv-X.Y.Z.tar.gz from https://pypi.python.org/pypi/virtualenv#downloads
    2. Extract into /opt/virtualenv-X.Y.Z
  4. Create and configure the "bootstrap" VirtualEnv:
    1. E.g. run "/opt/virtualenv-15.0.3/virtualenv.py -p /opt/python3/bin/python3 lib/py3venv"
    2. Run "lib/py3venv/bin/pip install virtualenv"
Now, whenever you need to create a self-contained VirtualEnv for a project, run this command, substituting the placeholder for the desired output directory:
lib/py3venv/bin/virtualenv env_dir
The VirtualEnv directory that is created will contain its own Python executable, but will rely on the global packages in /opt/python3.

See The "Virtual Environments and Packages" section of the Python tutorial for more info.

Labels: ,

07 October 2016

Django problem: sqlite error when running"manage.py migrate"

Just ran into a problem with creating a Django virtualenv.  I built Python from source, but I didn't have the libsqlite3-dev library installed.  (I'm using Debian, but the same problem applies to Ubuntu.) 

So when I tried to run "demo/manage.py migrate" in my Django virtualenv, I got this error:
ImportError: No module named 'pysqlite2'
Once I installed libsqlite3-dev rebuilt Python with "make clean install", it worked!

The fix is documented in the Stack Overflow post "No module named _sqlite3".

Labels: