22 November 2016

Why Git?

To give you an example of how helpful Git can be, I've modified an e-mail I sent to a client. They had customised a library I'd written for them, and I need to fix a bug both in my original version and the modified version of the include file they'd sent me. I did these steps:
  1. Create a local repo inside my source directory
  2. Commit all the files into the local Git database
  3. Create and switch to a new branch called 'test'
  4. Save their modified version of main.inc
  5. Commit the change in the 'test' branch
  6. Switch back to the 'master' branch
  7. Fix the bug and commit
  8. Switch to the 'test' branch
  9. Merge all changes from the 'master' branch
This automatically applies the change from step 7 and commits the change in the current branch. It might seem like a lot of work to apply one change that could easily be done again to a different version of the file, but consider that it scales; even complex changes can be merges with only cursory review required. (Unless both branches have changed in contradictory ways (i.e. same parts of the files have been changed) since the two branches diverged. In this case a merge conflict will happen, which is easy to resolve by "collapsing" each pair of conflicting changes that have been put next to each other in the files that couldn't be merged automatically. A commit is then necessary because obviously it couldn't be done when a conflict was detected.)

08 October 2016

How to bootstrap Python VirtualEnvs with a specific Python version

This shows how to create a "bootstrap" VirtualEnv, which will allow the creation of (mostly) self-contained VirtualEnvs that are used for projects.
  1. Install OS dependencies like libsqlite3-dev
  2. Build Python from source:
    1. Download from https://www.python.org/download/releases
    2. Extract tarball
    3. cd into directory
    4. Run "./configure --prefix=/opt/python3"
    5. Run "make"
    6. Run "sudo make install"
  3. Manually grab the latest virtualenv module to avoid dependency problems with the existing Python installation:
    1. Download virtualenv-X.Y.Z.tar.gz from https://pypi.python.org/pypi/virtualenv#downloads
    2. Extract into /opt/virtualenv-X.Y.Z
  4. Create and configure the "bootstrap" VirtualEnv:
    1. E.g. run "/opt/virtualenv-15.0.3/virtualenv.py -p /opt/python3/bin/python3 lib/py3venv"
    2. Run "lib/py3venv/bin/pip install virtualenv"
Now, whenever you need to create a self-contained VirtualEnv for a project, run this command, substituting the placeholder for the desired output directory:
lib/py3venv/bin/virtualenv env_dir
The VirtualEnv directory that is created will contain its own Python executable, but will rely on the global packages in /opt/python3.

See The "Virtual Environments and Packages" section of the Python tutorial for more info.

Labels: ,

07 October 2016

Django problem: sqlite error when running"manage.py migrate"

Just ran into a problem with creating a Django virtualenv.  I built Python from source, but I didn't have the libsqlite3-dev library installed.  (I'm using Debian, but the same problem applies to Ubuntu.) 

So when I tried to run "demo/manage.py migrate" in my Django virtualenv, I got this error:
ImportError: No module named 'pysqlite2'
Once I installed libsqlite3-dev rebuilt Python with "make clean install", it worked!

The fix is documented in the Stack Overflow post "No module named _sqlite3".


24 March 2015

How to make an SVN repo accessible by Web and SSH

Would you like to publish an SVN repo over the web (read-only of course) so people can see your dotfiles, etc.?  If so, it takes some effort to make it work.

The examples below apply to Debian, but with minor tweaks (e.g. which config files you edit) it should work anywhere.

Apache setup

This assumes that you want your repo available at http://my.site/svn/xyz .  (The docs will tell you how to host all repos in a given directory, rather than one specific one.)  It also assumes that your repo is in /srv/svn/xyz on your web server.

First, Apache will have to be augmented with stuff to let it use the SVN libraries to interact with the repo.   This lets Apache present it as a sort of filesystem over the web, which supports both browser access as well as access by SVN clients.
sudo aptitude install libapache2-svn
## sudo service apache2 restart   # done by the above
 Then add the following to your vhost file:
# -- SVN --
<Location /svn/xyz>
  DAV svn
  SVNPath /srv/svn/xyz
    Deny from all

  # Uncomment and tweak this to enable write access for certain people.
  ## AuthzSVNAccessFile /etc/httpd/security/svn.access
  ## Require valid-user
  ## AuthType Basic
  ## Authname "Subversion Test repository"
  ## AuthUserFile /etc/httpd/security/svn.auth
  ##   Require valid-user
  ## </LimitExcept>
I don't accept any liability for the above, and it is your responsibility to test that it is secure, e.g. by trying things that should be blocked and checking your Apache logs to ensure that they are.

Also, be sure that there's been nothing sensitive committed to the repo, even in past revisions.  Don't commit your whole home directory, because you don't want to reveal things like your API tokens (e.g. in .gitconfig) etc.

Special considerations 

Web access will happen via the www-data user, which on Debian is the user that Apache runs as.  This still needs to write to files in order for locking to work.

In the past, I got it working the past by tweaking group permissions and probably the umask (possibly using wrapper scripts).

But this is the 21st century, and everything should support POSIX ACLs, which are a kind of extended permission system.  This example assumes the existence of an svnusers group of which the required people, who will access the repo over ssh, are members.
cd /path/to/repo

# set the ACL mask to stop files being made executable
find . -type f | sudo xargs setfacl -m m:rw

# set blanket permissions, but don't recalculate the ACL mask
sudo setfacl -R -n -m g:svnusers:rwx,u:www-data:rwx

# set default perms on directories (no effect on files) so new files get correct perms
sudo setfacl -R -d -m g:svnusers:rwx,u:www-data:rwx .
Note that this won't have to be done again in future, because any new files (and directories for that matter) will automatically be given the correct permissions thanks to the last line.  "-d" automatically transforms regular ACL entries into default ACL entries, e.g. u:www-data:rwx -> default:u:www-data:rwx .

Labels: ,

10 March 2013

Resume from hibernate failed silently on Debian

After I changed my swap partition, my computer hibernated fine but would not resume, i.e. it booted normally, as if I had done a hard power-off beforehand.

This happened because hibernating saves the contents of RAM etc. into your swap partition.  Therefore, <initramfs>/conf/conf.d/resume now contained an incorrect UUID (of the old, no-longer-existing partition).  See Debian linux mint, resume after hibernation fails for how to fix this.  Don't forget to modify your /etc/fstab and also regenerate the blkid cache by running "sudo blkid -p".

(See Wikipedia.org's Initrd article for background and the initramfs-tools package for how initramfs is managed under Debian.)

You have to shut down after changing your swap partition.  For some reason the kernel's power management subsystem won't hibernate to the new swap partition until after the next boot.  If you try, you'll get the error "Cannot find swap device, try swapon -a" on the console.

PS -- Don't be fooled by "PM: Resume from disk failed." in /var/log/kern.log (this is a normal error that you get when you boot after shutting down).  Debian's initramfs will check for the presence of /dev/disk/by-uuid/$resume and /sys/power/resume and only run <initramfs>/bin/resume if they both exist.  If they don't, it doesn't do anything and just continues to boot.


10 August 2011

User defaults on Linux

These are the places where you have to change things like the home directory base:
  • /etc/default/useradd (even on non-Debian systems)
    (change these with useradd -D)
  • /etc/adduser.conf
  • Webmin's Users and Groups module configuration
You'd think there'd be one place all this was stored, wouldn't you?

See also /etc/login.defs .

Labels: ,

29 June 2011

Making OpenVPN easier

I have become annoyed at the easy-rsa scripts shipped with OpenVPN. I like the goal, I just think the execution is flawed due to the inflexibility of the scripts. I mean, sure one could copy the whole directory and tweak the scripts, but they should be customisable with a config file. How about editing openssl.cnf? Wrong answer... the scripts pass options that override many of the settings in this file.

So, rather than fixing the scripts (e.g. so that they accept options that allow you to change the way they operate), I came up with what I think is a better answer. Since I already created a Makefile (with a set of targets that manage the workflow of creating server keys, client keys, zip files that people can extract into their OpenVPN directory on MS-Windows, etc.) why not run openssl commands directly?

So that is my current goal for the code in SVN. It's at revision 67, so when the Makefile jumps above that, it should be streamlinedt. (Feel free to bug me about making a commits mailing list for the tools repository.)

Labels: , , ,