-
Notifications
You must be signed in to change notification settings - Fork 23
Debian_packaging
This is intended to record the process of Debianizing the TriFusion package, including the setup of a PPA, package uploading and how to setup daily builds. However, the whole process can be easily generalized to other python packages, assuming they have a functional setup.py script. The setup of a PPA requires the signing of the package with a GPG and an account on launchpad.net, but these steps are not required when building a .deb package for private use.
The initial steps of this process were greatly assisted by an excelent article.
This step is not required when building .deb packages for private use
In the final stages of building a .deb package from source, the building program will check .dsc and .changes files for a signature. If that signature is not present in the files and/or not setup on your system, it will issue a warning (even though an unsigned .deb package is created anyway). To avoid this, we will have to create our own GPG key and then reference it in a couple of files. This can be easily accomplished using `gpg --gen-key`. Here is the excepted console output of gpg v2.1.13 (older version may be slightly different), in which we need to answer some questions:
╭─diogo@ultraArch ~
╰─$ gpg --gen-key 130 ↵
gpg (GnuPG) 2.1.13; Copyright (C) 2016 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Note: Use "gpg2 --full-gen-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Nome completo: Diogo N Silva
Endereço de correio eletrónico: o.diogosilva@gmail.com
Você selecionou este identificador de utilizador:
"Diogo N Silva <o.diogosilva@gmail.com>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
Precisamos gerar muitos bytes aleatórios. É uma boa ideia realizar outra
actividade (escrever no teclado, mover o rato, usar os discos) durante a
geração dos números primos; isso dá ao gerador de números aleatórios
uma hipótese maior de ganhar entropia suficiente.
Precisamos gerar muitos bytes aleatórios. É uma boa ideia realizar outra
actividade (escrever no teclado, mover o rato, usar os discos) durante a
geração dos números primos; isso dá ao gerador de números aleatórios
uma hipótese maior de ganhar entropia suficiente.
gpg: directory '/home/diogo/.gnupg/openpgp-revocs.d' created
(...)
chaves pública e privada criadas e assinadas.
Here you will only need to provide you Name and e-mail address and then make your PC work on something intensive (might be a good time to update your system's packages) to speed up the key generation. Take special attention to the Name and e-mail address you provided, since you will need to type identical values in the changelog file.
Now, you'll need to create a public key that will be used to sign your packages and setup your launchpad account. For this, you will require a string that was written in the gpg --gen-key output. In this case:
"Diogo N Silva <o.diogosilva@gmail.com>"
This string, may differ in other versions of gpg. For instance, gpg in my ubuntu (v1.4.16) also includes a Comment question and generates a string: Diogo N Silva (odiogosilva) <o.diogosilva@gmail.com>.
To create a public key:
╭─diogo@ultraArch ~
╰─$ gpg --armor --output ~/.gnupg/o.diogosilva-pubkey.gpg --export "Diogo N Silva <o.diogosilva@gmail.com>"
Notice that ~/.gnupg/o.diogosilva-pubkey.gpg does not exist and will be created with this command. Again, the string after the --export option needs to match exactly the string in the gpg --gen-key output.
Finally, we can import our key to install our package.
╭─diogo@ultraArch ~
╰─$ sudo cat ~/.gnupg/o.diogosilva-pubkey.gpg | gpg --import
That's it.
First thing, you'll need to be in the directory of you package. In my case, I simply download the latest zip from GitHub, extract it and cd inside. Once inside, create a new directory debian, where the files required for packaging TriFusion will be placed:
- changelog
- compat
- control
- copyright
- rules
- trifusion.install
- watch
This will create a structure akin to:
- TriFusion/
- debian/
- trifusion/
- (other dirs)
- setup.py
- (other files)
Note that the commands in the following section assume the current working directory is TriFusion/.
This is one of the most important files, and is the file that will contain the GPG key, if generated. It can be automatically generated, but you'll need to setup some environmental variables, ideally by exporting them in your .bashrc (or .zshrc, etc):
export DEBFULLNAME="Diogo N Silva"
export DEBEMAIL="o.diogosilva@gmail.com"
Then, source ~/.bashrc to reload your .bashrc.
To create the first changelog:
dch --create --package trifusion -v 0.1.0
This will create a generic changelog file, which can be edited manually.
trifusion (0.1.0) UNRELEASED; urgency=medium
* Initial release. (Closes: #XXXXXX)
-- Diogo N Silva <o.diogosilva@gmail.com> Thu, 07 Jul 2016 13:02:24 +0100
The UNRELEASED string can be changed to the ubuntu release where TriFusion is meant to be build (wily, vivid, etc.). But most importantly, you must ensure that the signature in the last line, matches your GPG key.
Since the signature is Diogo N Silva <o.diogosilva@gmail.com>, it's all good.
Also, take special care with the versioning of your package. Changes in the source or packaging files must be always associated with an incremental version, otherwise the package will not upload. To accomodate the fact that sometimes changes in the debian files are required but the source remains the same, TriFusion uses the -<build version> suffix. For example, the initial build version could be 0.1.0-1 and subsequent changes in the build files would increment only the last digit.
The increment of the package version can be also automatic using:
dch -i
Which will create a new entry above the previous version.
Must contain a single line with the debhelper compatibility level of the package. This version can be check by running:
dpkg -p debhelper | grep Version
Version: 9.20131227ubuntu1
The version is 9, so the compat file should only contain:
cat debian/compat
9
This file contains the vital information about the source package, which is specified by addind package metadata information. An example of a control file for the trifusion-daily builds can be found here. Here I will only make some comments on the most relevant control fields.
-
Build-Depends: Here are the dependencies required to build the package, NOT the ones required for the program to run (That's the Depends field). Since the package building requires setuptools to install the trifusion module, the python-setuptools is added here.
-
XS-Python-Version: Specifies the versions of Python supported by the package. In this case, all versions above 2.7.0.
-
Standards-Version: The most recent version of the standards with which the package compiles. This can be known by executing:
apt-cache show debian-policy | grep Version
Version: 3.9.5.0
- Depends: Don't forget to add "${misc:Depends}, ${python:Depends}" in addition to all other possible depencies. These depencies should be available in the offical repos. If not, there is an option in launchpad to add a PPA dependency that contains packages not present in the official repos.
There are several other fields that can be used here
Here I just added the GPL3 copyright file
This is the debian Makefile used to generate the .deb package. I used the dh7 to simplify the rules file, and then have a Makefile in my package.
The rules file ended being simply:
#!/usr/bin/make -f
# -*- makefile -*-
DEB_PYTHON2_MODULE_PACKAGES := trifusion
include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/class/python-distutils.mk
And then the Makefile in my package.
The reason why DEB_PYTHON2_MODULE_PACKAGES is set to trifusion, is because CDBS automatically finds all python modules that start with python- but ignores everything else. This statement ensures that the package trifusion is processed as a python module, even without the python- prefix.
The watch file is used to scan the upstream source for new releases and use it when building. In my case, the watch file should scan the releases of the GitHub TriFusion package:
version=3
opts=filenamemangle=s/.+\/v?(\d\S*)\.tar\.gz/TriFusion-$1\.tar\.gz/ \
https://github.com/odiogosilva/TriFusion/tags .*/v?(\d\S*)\.zip
You can use uscan to test if the sources are correctly fetched. In this case:
uscan --no-symlink --verbose --no-download
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
opts=filenamemangle=s/.+\/v?(\d\S*)\.tar\.gz/TriFusion-$1\.tar\.gz/ https://github.com/odiogosilva/TriFusion/tags .*/v?(\d\S*)\.zip
-- Found the following matching hrefs:
/ODiogoSilva/TriFusion/archive/0.4.0.zip (0.4.0)
/ODiogoSilva/TriFusion/archive/0.3.0.zip (0.3.0)
/ODiogoSilva/TriFusion/archive/0.2.0.zip (0.2.0)
/ODiogoSilva/TriFusion/archive/v0.1.1.zip (0.1.1)
Newest version on remote site is 0.4.0, local version is 0.4.0
=> remote site does not even have current version
-- Scan finished
Here, uscan is able to find all 4 releases and determine 0.4.0 as the most recent.
To generate the first debian build, execute the debuild program:
debuild -S
Assuming you're in the package directory. Expected output:
dpkg-buildpackage -rfakeroot -d -us -uc -S
dpkg-buildpackage: pacote fonte trifusion
dpkg-buildpackage: versão da fonte 0.4.1-7
dpkg-buildpackage: source distribution trusty
dpkg-buildpackage: fonte alterada por Diogo N Silva (odiogosilva) <o.diogosilva@gmail.com>
(...)
Successfully signed dsc and changes files
If all goes well, the password for your GPG key should be necessary to finish the build.
The debian package files should appear in the parent directory (../).
Since the upload and building of the debian package in launchpad takes some time, it is good practice to test your build locally first. This will require the pbuilder-dist program. To setup pbuilder, you simply need to issue:
pbuilder-dist <release> create
where is the Ubuntu release (e.g. trusty, precise, vivid, etc). This will take some time, as pbuilder-dist will create a minimumal and clean environment that will ensure your build succeeds in a reproducible way.
To test your recently built package, simply run:
pbuilder-dist <release> build ../<package>.dsc
The result of this command will be the same as the one execute in launchpad.
In order to publish your package to the whole community, you'll need a properly setup launchpad account. Go to https://launchpad.net/ and register. Most of the steps below take some time between uploading/setting information and they
Here, your public key will be uploaded to the ubuntu keyserver so that anyone can download it AND so that you can upload it to launchpad. To acomplish this, Open Passwords and Encryption keys, navigate in the side panel to your GPG keys, and select your key in the left panel. Then go to Remove > Sync and publish Keys in the top menu. Before the Sync button is available, you may have to choose the Key Servers, in which you should chooser the keyserver.ubuntu.com.
It can take up to 30 minutes before your key is available on launchpad. After that time, you can upload your key.
Once registered, go to your account and add an OpenPGP key. The fingerprint of your PGP key will be required, and this can be access by running:
gpg --fingerprint
This will list all your PGP keys and their corresponding fingerprints. Choose the appropriate one and paste in on launchpad.
In your launchpad account page, Click Create a new PPA and complete the information according to your package. Once the PPA is activate, you can upload you package using dput, which requires only two arguments:
dput ppa:<launchpad_id>/<ppa name> <source.changes>
In my case, this command is called as:
dput ppa:o-diogosilva/TriFusion ../*.changes
Note that this will only upload your source package IF the package has a higher version that the previously uploaded one. If not, dput will simply say that the package has already been uploaded, even if you changed the source code.
If all goes well, dput will upload a .dsc, a .tar.gz and a .deb files. As soon as they are uploaded into your PPA, they will be queued for building a publishing, which may take a few minutes. After their publishing, the PPA can be added to your sources.list and you can use apt-get to install your package.
Daily builds are useful when you wish to release a PPA that builds your package on a daily basis IF the target branch of your source code in the GitHub repository changes. This way, users using this PPA will always have the bleeding edge version of your package. This will require you to push your code into the launchpad repository in your account, which can be accomplished with either Git or Bazaar. Here, I will only cover the Git workflow.
Assuming you already have a git repository of your package in your machine, you simply need to add a new remote to Git, so that your code can be pushed to the launchpad repository. One way of doing this, is by editing the ~/.gitconfig and add these lines:
[url "git+ssh://USER@git.launchpad.net/~USER/+git/"]
insteadof = lpme:
In this specific case:
[url "git+ssh://o-diogosilva@git.launchpad.net/~o-diogosilva/+git/"]
insteadof = lpme:
This will basically create an alias of the launchpad repository using lpme.
Now you'll need to add the remote and you can push your code:
git remote add origin_lp lpme:TriFusion
git push origin_lp remote
You can check that your code is in the launchpad repository by clicking, in your launchpad page, Code > View Git repositories.
Now, the tricky part is how to associate the debian packaging files to your repository. Adding a debian folder to your repository is usually a poor idea, since you are cluttering your repository with packaging files for a specific distribution. A possible solution is to create a second GitHub repository whose only goal is to host packaging files (for debian and other distributions as well). In this case, I created TriFusion-packager, containing a debian folder with the packaging files. The packaging files in this repository are largely the same as the ones described above, with the exception being the changelog file. Since this will be an auto build, it makes little sense to modify the changelog at each building, which is why this file is left with only the bare-bones. In this case:
trifusion (0.5.0-0) stable; urgency=low
* Auto build
-- Diogo N Silva (odiogosilva) <o.diogosilva@gmail.com> Thu, 07 Jul 2016 00:46:26 +0100
The versioning will be later handled by the PPA recipe.
Now that you have the packager repository set up, you'll need to push the code to the launchpad repository using the same commands as above.
Once both repositories are live in launchpad, you can create a packaging recipe that will handled the source building every day. To create a recipe, go to the repository page of your main package and click Create packaging recipe. You'll need to provide a name, which was trifusion-daily in this case. If you want to build this source every day (provided that the source code changed) check the Built daily check box. I also created a new PPA for this recipe, so that users could use the original PPA for the stable version of my package, and this new PPA if they want to stay in the bleeding edge. Finally, you can select the distribution series you wish to publish your code.
The final and most important part of creating a recipe is the Recipe text. By default, the recipe text is:
# git-build-recipe format 0.4 deb-version {debupstream}-0~{revtime}
lp:~<launchpad_id>/+git/<repo name> master
This will automatically build packages with versions based on the dev version of the upstream package (debupstream) and on the revision time (revtime). However, there is no information on how the control files can be used in this recipe. To accomplish this, we need to add a nest-part command. The final recipe text should look like this:
# git-build-recipe format 0.4 deb-version {debupstream}-0~{time}{date}
lp:~o-diogosilva/+git/TriFusion master
nest-part packaging lp:~o-diogosilva/+git/TriFusion-packager debian/daily debian
Here, I'm using the debupstream, time and data variable to keep the versioning unique. In line 2 I'm fetching the source code from the master branch of the launchpad repository. On line 3, I'm merging the debian/daily directory of my TriFusion-packager repository, into my main repository but ONLY during the building. After saving this recipe, you can issue the building process to ensure that everything is running fine.