launchpad.net Ubuntu PPA build

[mmd-osm]

Follow up for #6

As Ubuntu PPA build for Focal would need to be fetched from https://launchpad.net/~osmadmins/+archive/ubuntu/ppa for deployment to production, the build process needs to run successfully on https://launchpad.net as well.

I've mentioned some changes needed to produce PPA builds for Focal (#6 (comment)). In addition, the osm-logical shared library would need to be installed to successfully pass the unit tests on launchpad.net

You can also run the PPA build process locally via

• debuild -k{GPG key ID} -S -sa -d
• sudo pbuilder build --distribution focal ../osmdbt_0.1.dsc

pbuilder needs to initially set up via sudo pbuilder --create --distribution focal ...
https://wiki.ubuntu.com/PbuilderHowto or maybe https://blog.packagecloud.io/eng/2015/05/18/building-deb-packages-with-pbuilder/ --> Initializing and deleting the pbuilder chroot have a bit more detail on this initial set up step.

The overall process is a bit tricky, and tends to be somewhat annoying after a while. If you're stuck somewhere or get some strange error message, I can see if I can help sorting them out.

[joto]

As you seem to already know how to do all this, can you provide a PR? Or somebody else? Setting up the Debian config is as far as my knowledge goes.

[mmd-osm]

There's one topic I'd need your opinion, namely how we should go about with the postgresql-plugin. All the command line tools in src/ only depend on the postgresql client library, while the plugin depends on the postgresql server + additional dev libraries.

I'm wondering if should create two separate PPAs for the client tools on the one hand, and the plugin on the other hand. It would certainly add a bit more effort, but we could update each of those two parts independent of each other. In particluar I'm thinking about keeping the plugin part as small as possible, so we can leave it alone, even if we find bugs later on in the command line tools. Also it would facilitate a use case, where the command line tools run on another server and connect via network to the db server.

[joto]

Unless @tomhughes want something different I would say: Keep it simple. One package with everything is fine for the moment. We can always change things later if we want to, after all we are the only users of this.

[mmd-osm]

Here's a first test build on Ubuntu Launchpad:

• Changes: master...mmd-osm:ubuntu/focal
• Test PPA build on Ubuntu Launchpad: https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test1/+packages

I can't get those tests to run in the Ubuntu PPA build, as they depend on the osm-logical.so to be deployed to the Postgresql library directory. pbuilder builds packages in some chroot environment, and simply copying osm-logical to /usr/lib/postgresql/*/lib as part of the dh_auto_test override is rejected with a Permission denied error.

By the way, debuild -I. seems to have a similar issue. Unit tests fail, unless you install the shared library via sudo make install beforehand. That's easily doable on your local machine, but it doesn't work if you build packages on some external build server.

The way the build process works at the moment, it would take the Postgresql version that happens to be the current default - that's Postgresql 12 in case of Focal. Is this the version we want to use later in prod?

[tomhughes]

We want 12 for testing but we probably want 9.5 initially for prod as I assume we will switch to this first before trying the database upgrade.

[mmd-osm]

Chef repo seems to pull all Postgresql packages from https://wiki.postgresql.org/wiki/Apt - which appears to be the only way to deploy 9.5 packages on Focal. I think this dependency on an external repository is still missing in the current debian/* files.

[mmd-osm]

I don't know if we can get this to run on launchpad. The biggest challenge is that Postgres 9.5 is no longer part of Ubuntu 20.04, and we have to resort to an external repository, as I mentioned in my previous post. While we can add external package dependencies on Launchpad, those packages need to be in another PPA on Launchpad itself (not somewhere on the Postgresql APT repo). As a consequence, we would have to download source packages from the Postgresql repo, and somehow upload them to Launchpad. This seems all massively complicated.

Meanwhile, the following Dockerfile installs all Postgresql 9.5 files on Ubuntu 20.04, builds everything, installs it, then builds the debian package in a defined, clean environment. It can be found afterwards as /root/osmdbt_0.1_amd64.deb. This deb package can then be installed on an Ubuntu 20.04 installation via dpkg -i osmdbt_0.1_amd64.deb.

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -qq && \
    apt-get -y -q install software-properties-common curl ca-certificates gnupg \
                          libosmium2-dev libboost-filesystem-dev libboost-program-options-dev \
                          libbz2-dev zlib1g-dev libexpat1-dev cmake libyaml-cpp-dev \
                          libpqxx-dev gettext-base git pandoc dpkg-dev devscripts debhelper

RUN curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ focal-pgdg main" > /etc/apt/sources.list.d/pgdg.list

RUN apt-get update && apt-get install -y apt-utils apt-transport-https ca-certificates

RUN apt-get -y -q install postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5 \
                          postgresql-common postgresql-server-dev-9.5 \
                          gcc g++  \
                          --no-install-recommends

WORKDIR /root/osmdbt

COPY ./ ./

RUN mkdir build  && \
    cd build && \
    cmake .. -DCMAKE_INSTALL_PREFIX=/usr && \
    cmake --build . && \
    make install && \
    ctest


RUN debuild -I

Resulting debian package reads as:

osmdbt_0.1_amd64.deb:
 new Debian package, version 2.0.
 size 325740 bytes: control archive=1128 bytes.
     487 bytes,    11 lines      control              
    1306 bytes,    19 lines      md5sums              
 Package: osmdbt
 Version: 0.1
 Architecture: amd64
 Maintainer: Jochen Topf <jochen@topf.org>
 Installed-Size: 1805
 Depends: libboost-filesystem1.71.0, libboost-program-options1.71.0, libc6 (>= 2.14), libgcc-s1 (>= 3.0), libpqxx-6.4 (>= 6.4), libstdc++6 (>= 9), libyaml-cpp0.6 (>= 0.6.2), zlib1g (>= 1:1.2.6)
 Section: utils
 Priority: optional
 Homepage: https://github.com/openstreetmap/osmdbt
 Description: OSM Database Tools
  Tools for creating replication feeds from the main OSM database.

*** Contents:
drwxr-xr-x root/root         0 2020-06-04 13:21 ./
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/bin/
-rwxr-xr-x root/root    294264 2020-06-04 13:21 ./usr/bin/osmdbt-catchup
-rwxr-xr-x root/root    440704 2020-06-04 13:21 ./usr/bin/osmdbt-create-diff
-rwxr-xr-x root/root    178536 2020-06-04 13:21 ./usr/bin/osmdbt-disable-replication
-rwxr-xr-x root/root    178536 2020-06-04 13:21 ./usr/bin/osmdbt-enable-replication
-rwxr-xr-x root/root    289136 2020-06-04 13:21 ./usr/bin/osmdbt-fake-log
-rwxr-xr-x root/root    235888 2020-06-04 13:21 ./usr/bin/osmdbt-get-log
-rwxr-xr-x root/root    182632 2020-06-04 13:21 ./usr/bin/osmdbt-testdb
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/lib/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/lib/postgresql/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/lib/postgresql/9.5/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/lib/postgresql/9.5/lib/
-rw-r--r-- root/root     14328 2020-06-04 13:21 ./usr/lib/postgresql/9.5/lib/osm-logical.so
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/share/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/share/doc/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/share/doc/osmdbt/
-rw-r--r-- root/root      3459 2020-06-04 13:21 ./usr/share/doc/osmdbt/README.md.gz
-rw-r--r-- root/root       140 2020-06-04 13:21 ./usr/share/doc/osmdbt/changelog.gz
-rw-r--r-- root/root       865 2020-06-04 13:21 ./usr/share/doc/osmdbt/copyright
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/share/man/
drwxr-xr-x root/root         0 2020-06-04 13:21 ./usr/share/man/man1/
-rw-r--r-- root/root       835 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-catchup.1.gz
-rw-r--r-- root/root      1534 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-create-diff.1.gz
-rw-r--r-- root/root       744 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-disable-replication.1.gz
-rw-r--r-- root/root       710 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-enable-replication.1.gz
-rw-r--r-- root/root       937 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-fake-log.1.gz
-rw-r--r-- root/root       883 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-get-log.1.gz
-rw-r--r-- root/root       712 2020-06-04 13:21 ./usr/share/man/man1/osmdbt-testdb.1.gz
-rw-r--r-- root/root      1777 2020-06-04 13:21 ./usr/share/man/man1/osmdbt.1.gz

[mmd-osm]

@tomhughes : what will be our target architecture in the future? osmosis runs on ironbelly connecting to karm as db master. Do you plan to run the same set up with the new tool as well? Is it ok that we "pollute" ironbelly's filesystem with those /usr/lib/postgresql/9.5/lib/ files?

[tomhughes]

Why would they be on ironbelly? They won't be any use there.

[mmd-osm]

We would have to deploy the single debian package to both karm and ironbelly. The shared library would be relevant for Postgresql on karm (and otherwise pretty much useless on ironbelly). The command line tools would run on ironbelly, if this will be set up in the same way osmosis is running today.

That was the trade off of having a single debian package only...

[tomhughes]

Well if it's packaged then I really don't care. I haven't really though about how things will work in production in the new world anyway.

I'm more concerned that it's not really practical to include the postgres bits for all the reasons you have discovered, that we're using the postgres packages not the ubuntu ones, and that we may need multiple versions of them available and to map them to the right version of postgres in different environments.

It may well be better to build the postgres bits from source I suspect.

[mmd-osm]

Building osm-logical.so from source would of course reduce a lot of the complexity with multiple versions.
Assuming postgresql-server-dev-9.5 (or some other version) has been installed already, that compile process would be straightforward:

cd postgresql-plugin
mkdir build
cd build
cmake ..
make install

All command line tools query the server version and run different statements depending on the Postgresql server version, so there's no need to keep any version specific packages around.

CMakeLists.txt may need a bit of tweaking, as the parent project is missing. Those seem to be warnings only.

[joto]

Since c39dcbe the postgresql plugin is no longer build in the Debian build and not included in the package. This should remove that complexity.

[tomhughes]

Can we get a new build then @mmd-osm? The last one I see is from July?

[mmd-osm]

Assuming testing will be initially done on the dev instance, can we build the binaries directly on the box like we did before with cgimap? I think that would speed up testing a bit, as any commit on the master branch would be almost immediately available for testing.

I think for production, we should at least revisit the PPA topic, as the plugin would have to be built directly in the box anyway, i.e. we need to have all libraries in place.

A good fit for a PPA would be e.g. if we want to deploy the dbt tools on another box and connect to the database via network.

Regardless of that decision it would probably make sense for @joto to sign up on
https://launchpad.net/ in the meantime, so that I’m not the bottleneck.

[tomhughes]

Yes of course I can build them from source but that is a last resort and I don't really want to be live tracking master in any case, certainly not for production, and I don't plan to chef anything for testing on dev,

[mmd-osm]

Right, tracking the master branch was only meant for tests on the dev instance, certainly not for production.

I’m not exactly sure if there were some previous discussions outside of this GitHub repo on how those tests should be run, so it’s a bit difficult for me to comment on the chef topic. The binaries could be compiled manually on the box, deploying the plugin probably needs sudo powers, a minutely job would need to be scheduled and ideally osc files being made available via Apache.

[tomhughes]

We discussed it with Jochen on the OWG call this week - as testing on dev is only of marginal use my plan is to do a quick bit of manual testing.

I've already built and installed the plugin this morning, and was going to install the tools from your build but found it was up to date which is why I asked. Obviously I can build manually instead, it's just more work.

[mmd-osm]

I've triggered a new build as PPA https://launchpad.net/~mmd-osm/+archive/ubuntu/osmdbt-test2, based on commit
b867f22. PPA publishing step is currently pending and should complete shortly.

debian/changelog still reports as version 0.1 back from June, which may be a bit confusing. Maybe @joto can bump the version number to 0.2 and add a new entry to the debian changelog.

[joto]

I updated the version.

[mmd-osm]

Nothing left to do, closing here...

[mmd-osm]

As an FYI:

Since compiling and deploying a custom plugin seemed a bit cumbersome, I've been looking into pgoutput, a fast binary format-based plugin that's built into Postgresql.

This might also be beneficial in case of cloud environments with limited options for deploying custom plugins.

Link: https://github.com/mmd-osm/osmdbt-pgoutput

[joto]

@mmd-osm Can you please open a new issue for this?