More tea please.

Additional dynpk glibc wrangling

Warning: Mangling may occur.

I was in the process of setting up to record a screencast of how to use dynpk, when I discovered that my dynpk fu no longer ran on the RHEL machine I was using. Running my program resulted in this friendly output:

FATAL: kernel too old

Turns out this is a message brought to us by glibc. glibc has a configure option ‘–enable-kernel’. The argument to this option is a Linux version, e.g. 2.6.18. This tells glibc the minimum Linux version that the resulting built stuff has to support. Apparently telling it the name of a more recent kernel version can improve performance because it doesn’t have to include as much compatibility cruft.

The current Fedora 14 glibc is built with this argument set to 2.6.32. Running a statically linked binary generated on F14 on a RHEL 5 machine running 2.6.18 is thus no longer possible without some additional work. What additional work is that? Rebuilding the C library with the ‘–enable-kernel’ option set to something compatible with the target system. glibc will still present the same API to the programs we want to run, but includes the additional fudge for old kernel version compatibility. We don’t have to rebuild all of our programs, just shoehorn our own glibc into the bundle dynpk makes.

“Rebuild the C library from source, you’re crazy.” I hear you say whilst backing away slowly. It’s not a massive deal. All I have to do is:

% fedpkg co -a glibc
% cd glibc
... change 2.6.32 to 2.6.18 in glibc.spec ...
% fedpkg local
... crunch, bang, whizz, pop ...

And some time later, out spring some RPMs. You’ll find the ones I’m using for F14 on RHEL5 here (don’t install them!).

I’ve modified dynpk so that it can take uninstalled RPM files and install them into the bundle (it’s a rather naive install, but it does the job right now). It can also take some glibc RPMs to build its own wrapper against. Here’s how: The build process is modified slightly:

% make GLIBC_RPMS="glibc-2.12.90-18.i386.rpm glibc-static-2.12.90-18.i386.rpm"
... nom nom nom ...

One would then add this to the dynpk configuration:

local_rpms: glibc-2.12.90-18.i386.rpm glibc-common-2.12.90-18.i386.rpm

And BANG, the stuff that comes out of dynpk now runs on RHEL5 again.

(Photo copyright flickr user paperbits under CC by-nc-nd.)

Posted at 8:56 pm on Tuesday 9th November 2010


For my PhD, I’ve been using the University of Southampton’s Iridis Compute Cluster, a.k.a. “supercomputer”. I’m using this to run the fitness tests for some genetic algorithm optimisation things I’m working on. Each fitness test takes 10 to 30 seconds, so the more I can run in parallel, the better (up to a point…). Using this cluster, I can run my work much faster. Getting to this point took a lot of beating though.

In what appears to have been some kind of twisted marketing stunt, many places report that the Iridis cluster runs Windows. It doesn’t. If it did, I wouldn’t entertain going near it. It runs Red Hat EL5.

After filling in the paperwork to get an account on this wondrous cluster, I shelled in and went about compiling my work so I could run it on the cluster. I was expecting some packages not to be installed, as I do on most systems that I initially approach. Unfortunately one library that I wanted to use, CGAL wasn’t installed, nor was it in the repositories. So a request for this to be installed would have involved getting one of the sysadmins (who are already stretched on fixing some killer, and I really do mean killer, GPFS performance issues) to install it from source, and would take far too long.

Option 1: Build from Source

So, I went about building it myself from source. Like a good little source-building monkey, I climbed the dependency tree, building the various other things that I needed for this. Things like cmake and boost… This became painful, as the 32-bit headers for libc weren’t installed. CGAL has some sort of fatal bug on 64-bit systems. Damn. I wasn’t going to build the C library. That’s where I drew the line. I went home, drank some tea, watched some Family Guy, slept.

Option 2: Static linking

With a freshly caffeinated brain, I decided to try static linking. This’d hopefully solve my problems because I could compile all the libraries from my local machine into one giant executable that I’d transport to the remote machine. Sounded good. Then I found that Fedora only provides static libraries for a small set of packages, and has a general dislike of these things. I was a little perturbed by this.

Option 3: Bundling the dynamic linker

I nosed around with the dynamic linker. I found that the dynamic linker can be invoked with the program that it should dynamically link as its argument, e.g:

% /lib/ld-linux.so.2 /bin/bash

I realised that this could fix the issues I was having. So I wrote a small utility that’d bundle together the dynamic linker from my Fedora system along with the executables that I wanted to run and the shared libraries they required. This was nice. I could take a binary from my Fedora system, pass it through this utility, and get a wrapped binary that I could run on RHEL.

A further nicety that I discovered was the “auditing” functionality of the dynamic linker. This allowed me to write a function that’d get called every time the dynamic linker went to load a new shared library from disk. My auditing library would scream its head off if this library wasn’t from the set that within the bundle. This meant I could be sure that the code I was running was the same as the code I was running on my own machine.

There’s more information about the auditing API in the rtld-audit man page.

I stuck with this solution for a few hours, until I discovered some issues with it. Many programs rely on configuration files and other executables on the system they’re running on. ImageMagick is one of these programs. It often invokes other programs to convert between formats, and has a configuration file that modifies this process too. My statically linked program used ImageMagick’s C API to manipulate some images, so it ended up invoking some stuff from the host system. This turned nasty when RHEL’s buggy librsvg was used to convert an SVG to a PNG :(

Option 4: fakechroot

It was clear at this point that I needed some kind of chroot environment. Unfortunately chroot itself requires superuser rights on the system that it’s run on (for some reason that isn’t completely clear to me). So I looked around and found fakechroot. I extended my bundling utility to support bundling entire RPMs from my system. It would then scan through all the files, find the dynamically linked ones, and replace them with a wrapper script that ensured the correct dynamic linker was used.

Suddenly, horror. ImageMagick’s convert would segfault whilst it was somewhere inside pixman. I started debugging this, and rapidly discovered that most of the values I wanted to see in pixman’s code were unavailable to me, as gdb reported they were “optimised out”. I installed Fedora 14 in a VM so I could use the later version of gcc and gdb’s combined power to be able to see these variables. I wrangled on what was going on for quite a while, and roped Jeremy into the situation for a few hours too. We found that the dynamic linker was incorrectly returning a NULL pointer when the address of a specific piece of thread-local storage was requested. This was ghastly. I waded around in this situation for a while. Then, through a random stroke of luck, I found that if I didn’t load my audit library it all worked! Something about the rather unique situation I’d created evoked a bug from glibc. Exciting. I don’t have time to work on it, so I’ll have to leave it there and just not use the audit library.

The result: dynpk

The result of all of this is a tool called dynpk (“din-pack”). You provide it with a list of the RPMs from your own system that you’d like to be bundled, baked, and wrapped into something you can transport to a Linux system of your choice. It’s nice to be able to run Fedora’s ipython on top of a RHEL machine, for example!

Instructions on how to get hold of and use this tool can be found here.

This makes me wonder what the MATLAB and LabVIEW-loving license-server junkies do on the cluster when they find a bug…

Another application

I can tell another Red Hat EL5 related tale as well. All of the Linux computers in the undergraduate computing lab in ECS run Red Hat EL5. Sounds great. However, EL5 is made of old software. It’s probably fine if you’re using it to perform simple office tasks, but seeing bugs that were fixed years-ago still romping around on those desktops is incredibly frustrating.

There’s a student-run project called CSLib that has, for many years now, attempted to solve the lack of software that the undergraduate machines have. Unfortunately, CSLib is never going to match the man power, and hence freedom from bugs and number of packages, that $MAJOR_DISTRO (e.g. Fedora, Ubuntu) achieves. It’s a brave effort, but in order for it to be a catch-all solution, it really needs to use the power of the larger free-software community.

dynpk can provide some relief to people who are in situations where they are essentially forced to use a system that they are not in control of that lacks the software they desire.

The Future

It seems to me that a long-term solution for both the supercomputer and public-machine problems are virtual machines. Yes, I know I’m late to the “let’s all wave our hands in the air about virtual machines” party, but I think this invasion needs to continue much further. The compute cluster should run virtual machine images. Amazon’s EC2 already does this. The supercomputing posse should follow suit. The lab machines I speak of above would also massively benefit from allowing each user to have their own VM image that’s transferred to the machine they’re using when they log in.

Posted at 10:59 pm on Monday 8th November 2010
One Comment

Site by Rob Gilton. © 2008 - 2019