xGoat
More tea please.

Squishing Python

Getting python onto the gumstix requires some work. By selecting python, the file system goes from 4 Mb (4131692 bytes to be exact) to 11Mb (11495444 bytes). That’s OK for the 16Mb gumstix boxes, but seems a little wasteful. After close inspection with the help of baobab, I find that a lot of it’s taken up by the python libraries:

Disk usage with python.

That big box represents the /usr/lib/python2.4 directory on the filesystem, which, according to baobab is consuming 11.7Mb. I disagree with that number, since when I run this command in the python library directory:

find -type f | xargs du -bc --apparent-size

I get 14791522 bytes, which is 14Mb. Now, that’s a little strange. I don’t fully understand what the number that Baobab is generating represents, but the graphical representation that it provides is quite cool.

In the library directory resize a load of .py and .pyc files, and some other files. 5.6 Mb (5792653 b) of .py files, 5.0Mb (5213556 b) of .pyc files and 3.6 Mb (3785313 b) of other files. It seems obvious to me that I don’t need one of those sets, since they just carry redundant information. The .pyc files are byte-compiled versions of the .py files. The .pyc files are slightly faster for python to load than the .py files, so I’d like to keep the .pyc files but discard the .py files.

However, python stores the modification time of the corresponding .py file within the .pyc file and only uses the .pyc file if the .py file is older than it. For the moment I’ll keep just the .py files for simplicity. Later, I may come back to this and sort this out.

Also, upon analysis of the buildroot scripts for building python, there’s a line in /package/python/python.mk that’s supposed to delete all the .pyo and .pyc files generated by the python install (.pyo files are similar to .pyc files, they just involve some [apparently rather limited] optimisation). Unfortunately this line only succeeds in removing the .pyo files, due to some missing escaped brackets. I modded it, resulting in this patch. So now adding python to the build adds 4.5 Mb (new size of the image is 8852572 bytes).

I pointed Baobab at /usr/lib/python2.4, and got:

Python library directory disk usage.

Where:

  1. /usr/lib/python2.4/lib-dynload – 1.8 Mb (1912284 b). This contains a load of shared object files.
  2. /usr/lib/python2.4/config – 1.4 Mb (1435572). This contains a set of files – the biggest being libpython2.4.a (1.3Mb). I’m fairly sure that file isn’t required, since the libpython2.4.so is lurking around too.
  3. Files in /usr/lib/python2.4 root – mainly .py files.

So, I can immediately cut the 1.3 Mb that is libpython2.4.a out from the installed system (patch). This still lurks around in the staging directory within buildroot, so it can be used for building stuff if required. Now down to 3.9 Mb (root image size 8218812 bytes) for python.

I’m working on getting Tcl and python onto the Gumstix. Tcl adds 1Mb to the root image, and has been included in all the builds above.

So by taking the really low hanging fruit out of the build, I’ve reclaimed 3.1 Mb of space. May not sound like a lot, but it’s nothing to scoff at when your machine has 16 or even 4 Mb of flash!

I could potentially compress all the python library files, yet this would slow loading them down, which means more energy gets used, which is not good for my wireless sensor node. Out of curiosity, I took all the .py library files and compressed them all into a single bzip’ed tar file. That squished the 5.7Mb of files down to 1.1Mb.

Having all the files in a single compressed file would potentially be quite inconvinient. However, by gzipping each .py file separately, we might get better performance out of it. gzipping them all separately takes it down to 1.7 Mb. I gzipped them all with:

find -type f -name '*.py' -exec gzip {} \;

I think Python would need modifying to support this though… I wouldn’t be surprised if there was some way of adding a hook into the import statement and gunzipping if necessary.

Posted at 2:32 am on Monday 26th March 2007

Comments are closed.

Site by Robert Spanton. © 2008 - 2018