apt-get meets the XP-Framework

at 2010-12-16 in Examples, Editorial by friebe (0 comments)

The XP Framework has an easy-to-use installation mechanism - simply downloading a setup script and piping it directly to PHP. What this mechanism cannot do though is to install PHP as a dependency itself, which is usually done in an operating-system dependant manner. On Debian and Ubuntu distributions, the packaging mechanism is called APT (Advanced Packaging Tool), which installs Debian packages (.deb files). In order to build such a package, we need to create a control file as well as the intended directory structure and wrap all that up using the "dpkg" tool (there's a howto over at IBM's developerworks). Unfortunately, this approach requires the Debian packaging tools to be available for the build platform - a situation we cannot rely on. This article shows a solution written in the XP Framework.

Creating the fixture
Before trying to create a debian package myself, I first followed the above tutorial to create a simple package on my Ubuntu box, using the dpkg tool. In order to accomplish this, I started out with the following directory structure:

+
`-+ src
  |-+ DEBIAN
  | `- control
  `-+ opt
    `-+ xp
      `-+ bin
        `- xp

The "control" file contains the meta data necessary for the debian package management system, such as product, version, dependencies and a descriptive text. To build the package, I then executed the following:

  $ fakeroot dpkg -b src xp5+7-5.7.12.deb

This yielded an installable package, which I could test with "dpkg -i" (for installing) and "dpkg -r" for removing.

Reverse engineering
The next step was to have a deeper look at the package file itself. The .deb files are basically archives that can be read by the binutils' "ar" command. Inside binary packages, we will find three files:

A file called "debian-binary", a text file containing the string "2.0"
A tarball named "control.tar.gz", containing the control file and installation lifecycle scripts such as "postinst" or "prerm"
The actual "payload" of the package, inside another tarball, "data.tar.gz"

Here's how the archive I created in the first step looked like:

  $ ar tv xp5+7-5.7.12.deb 
  rw-r--r-- 0/0      4 Dec 15 15:25 2010 debian-binary
  rw-r--r-- 0/0    720 Dec 15 15:25 2010 control.tar.gz
  rw-r--r-- 0/0 2033758 Dec 15 15:25 2010 data.tar.gz

In order to create such a file, we would need to implement the "ar" file format, which turns out to be a quite simple task: Ar files consist of a header per file containing the name, last modified date, owner and group, permissions and the filesize, followed by the actual data. It's a little bit tricky if we want to store filenames longer than 16 characters inside an archive, but as can be seen above, we're not going to run into this problem.

  with ($out= create(new File('out.ar'))->getOutputStream()); {
    $out->write("!\74arch\76\n");
    
    // Header
    $file= new File('...');
    $out->write(sprintf(
      '%-16s%-12d%-6d%-6d%-8o%-10d%c%c',
      $file->getFileName(),
      $file->lastModified(),
      0,            // Owner uid
      0,            // Owner gid
      $file->getPermissions(),
      $file->size(),
      0x60, 0x0A    // Magic
    ));
    
    // Data. Pad with "\n" if uneven # of bytes are written
    create(new StreamTransfer($file->getInputStream(), $out))->transferAll();
    if ($file->size() % 2 != 0) $out->write("\n");
    $file->close();
  }
  $out->close();

We can hardwire the owner to 0/0 (root:root) as this is exactly what "fakeroot" does.

Tape archives
After having successfully created an "ar" style archive and having verified it could be read by the command line utility, the next step was to implement the tar file format. This format is similar but a bit more challenging:

Additional meta data such as link information and device numbers is required.
Directories, symbolic and hard links as well as other file types can be stored.
The header contains a checksum.
Numbers use octal representations padded to the field length with a space and the NUL byte.
Strings are padded to their field length with NUL bytes.
The payload data is padded in 512-bytes blocks.

I wanted to stay with the sprintf() implementation as used to create "ar" files, so I had to find out a way to pad strings with NUL instead of spaces. The single quote can be used inside a sprintf()-token to declare an alternate padding character.

  $padded= sprintf("%'\0-10s", $value);

The "-" alignment specifier makes the data left-justified, so this will yield a 10 character string padded to the right with NUL bytes, as needed, for example: Hello\0\0\0\0\0.

Here's our control file after having been extracted from the .deb package:

  $ tar tvfz control.tar.gz 
  drwxr-xr-x root/root         0 2010-12-15 15:25 ./
  -rw------- root/root       699 2010-12-15 15:25 ./control

Note the leading "./" and the uncommon fact to find the dot-dir itself inside this archive. I created a class called "TarArchiveWriter", which would be passed any OutputStream instance for writing on sequentially, and have two methods, one for adding files and one for folders. The rest of the implementation was basically getting the checksum right and the paddings right, after which the "This does not look like a tar archive" messages disappeared and the "tar tvf" command began showing the expected output.

But wait:
I had created ".tar" files, but what I needed were ".tar.gz" files! Thankfully, the XP Framework already solved that task for me, and it was as easy as wrapping a compressing stream around the output stream:

  $tar= new TarArchiveWriter(new GzCompressingOutputStream($out));
  //                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    ^

Putting it together
Now I had the necessary ground work done to create a debian packages' innards, and just needed to combine all of it into one build script, the recipe of which goes along the lines of the following:

Parse the control file, calculating the package's file name from it
Create control tarball, packing all files inside src/DEBIAN/ into it.
Create data tarball from the rest of the files inside src/.
Write the "debian-binary" file and move the two tarballs into an archive.
Done.

Parsing the control file is almost a no-brainer, it contains keys and values separated by a colon and a space. Values spanning multiple lines start with a space or a tab in the subsequent lines:

  Description: XP technology is a rapid development environment. It will 
   run anywhere PHP runs.

The compressed tarballs need to be written to the file system before archiving them, as we need to know their length prior to writing the "ar" entries. The System::tempDir() method helps us with this. We need to remember to clean up these temporary files.

The only tricky part was finding all files inside the src/ directory except for the ones in the src/DEBIAN directory. To list all files and directories recursively, the XP Framework provides the io.collections.iterate API, which optionally accepts filters. The filter I wanted to use was not available, so I declared an anonymous instance as follows:

  $debian= new Folder('DEBIAN');
  $exclude= newinstance('IterationFilter', array($debian), '{
    protected $base, $length;
    public function __construct($base) {
      $this->base= $base->getURI();
      $this->length= strlen($this->base); 
    }
    public function accept($element) { 
      return strncmp($this->base, $element->getURI(), $this->length); 
    }
  }');

After compiling all these pieces into a class I had the first working version of my build script done, using only XP Framework APIs. To verify I had actually built a debian package, I used "dpkg --info xp5+7-5.7.12.deb", and there it was: The XP Framework as a Debian package!

Advanced Packing Tool
Now to make a package installable via the comfortable apt-get utility, and to make it available for more people than just me, I decided to upload it to the XP Framework's website, and then to add that URL to my sources.list. This proved a to end up in quite a bit of trial and error. Here's what I've found out so far:

Don't edit sources.list directly, that's what /etc/sources.list.d/*.list is for.
Inside the "root" directory on your webserver, you need a dists/stable/binary subdirectory, into which you place the .deb files
Inside the same directory, apt-get expects subdirectories per architecture (binary-i386, binary-amd64, ...). I used symlinks to "." for this purpose
Also, a "Packages.gz" file is searched for. It contains the control file and four more fields: Filename (relative to root directory), Size (in bytes), MD5, SHA1 and SHA256 checksums.

After I got all that set up, I regenerated the local apt cache by typing "apt-get update" into a root shell, and thereafter "apt-cache search xp5" and finally "apt-get install xp5+7". Voila: The XP Framework was installed on my machine!

There's still quite a bunch of things to do before this really works, like creating a GPG key and signing it (secure APT), trying out how upgrading works, how we can have multiple parallel versions installed, and so on. Regardless of that, I think we're on a good way

The code can be found here: http://code.planet-xp.net/xml/browse?blog,xp-meets-apt-get

You can subscribe to the XP framework's news by using RSS syndication.

apt-get meets the XP-Framework

Subscribe

Categories

Related