Refactor cmake install and package definitions #381

Woazboat · 2024-04-01T20:16:41Z

Use implicit default values where possible
Simplify cmake install targets via GNUInstallDirs module
- Installation directories for binaries, libaries, etc... are set automatically according to their target type
Define components for install/package targets
- Allows building separate packages for binaries and development headers (*-dev package)
Rename included modified libxml++ to differentiate it from original libxml++

Woazboat · 2024-04-01T20:21:36Z

CMakeLists.txt

+        OWNER_READ OWNER_WRITE OWNER_EXECUTE
+        GROUP_READ GROUP_EXECUTE
+        WORLD_READ WORLD_EXECUTE)


I copied this over from the existing code, but is this really required? The correct permissions should already be set automatically.
(I changed CPACK_INSTALL_DEFAULT_DIRECTORY_PERMISSIONS to CMAKE_INSTALL_DEFAULT_DIRECTORY_PERMISSIONS which is also used by cpack by default).

…pack definitions

mmd-osm · 2024-04-24T11:18:19Z

OSM operations wants to go ahead with Ubuntu 24.04 for the time being and only move to Debian at a later point in time. This in turn means that cpack has only low prio now, since we would continue using Ubuntu Launchpad for packaging and shipping PPAs to OSM production.

tomhughes · 2024-04-25T10:39:11Z

That was intended more as a general comment on our strategy, that in the short term we are likely to upgrade machines to 22.04 simply because moving them to Debian requires a reinstall which is significantly more complicated while new hardware would get Debian.

That said as the web frontends are duplicated we may well want to move them to Debian at some point because we can do that in a staged manner without disrupting service.

mmd-osm · 2024-07-09T19:37:45Z

So there's some update in https://osmfoundation.org/w/index.php?title=Operations/Minutes/2024-06-27#%22OSM_PPA%22_work_approval

Minutes say "CGImap is blocking us moving with the six front ends.". I don't really know what this is about, maybe @Firefishy knows more?

Firefishy · 2024-07-09T19:55:15Z

Minutes say "CGImap is blocking us moving with the six front ends.". I don't really know what this is about, maybe @Firefishy knows more?

We'd like a PPA like deb repo for Debian 12 with our own builds (with potential extra patches). A replica of like what we have for Ubuntu https://launchpad.net/~osmadmins/+archive/ubuntu/ppa

mmd-osm · 2024-07-09T19:57:41Z

Thank you for the quick reply, this makes sense to me. Where exactly is CGImap blocking you atm?

Woazboat · 2024-07-09T19:58:35Z

Afaik an apt/debian repo is more or less just a bunch of files on a http server, so that could potentially be selfhosted?

tomhughes · 2024-07-09T20:02:31Z

It is from the point of view of serving it, yes. The complicated part it seems is turning a bunch of debs into a working file tree to serve.

I'm used to RPM where you can chuck a bunch of rpms in a directory, run createrepo to create the metadata and then just serve that up but the Debian wiki page on doing the equivalent is about 3 miles long with about 30 different ways of doing it none of which seem to be anything like that simple.

Woazboat · 2024-07-09T21:25:55Z

I just played around with https://www.aptly.info/ a bit and it seems fairly easy to use. ¹

# Create a repository
aptly repo create -distribution=stable -component=main -comment="Stable repository for openstreetmap-cgimap" openstreetmap-cgimap
# Add deb packages
aptly repo add openstreetmap-cgimap /path/to/deb/files/
# Publish the repository (local file path that can be served via http)
aptly publish repo openstreetmap-cgimap

Also works with multiple versions of the same package

# Add a new package version
aptly repo add openstreetmap-cgimap /path/to/openstreetmap-cgimap_0.12.34_amd64.deb
# Update the published files
aptly publish update stable

Or multiple distributions for separate testing/stable repositories

# Create a testing repository
aptly repo create -distribution=testing -component=main -comment="Testing repository for openstreetmap-cgimap" openstreetmap-cgimap-testing
# Add a development version to the testing repo
aptly repo add openstreetmap-cgimap /path/to/openstreetmap-cgimap_0.12.34_amd64.deb
# Move the package version from testing to the stable repo
aptly repo copy openstreetmap-cgimap-testing openstreetmap-cgimap openstreetmap-cgimap_0.12.34_amd64.deb

Listed in https://wiki.debian.org/DebianRepository/Setup ↩

mmd-osm · 2024-07-09T22:00:11Z

This is a bit funny. I tried out aptly pretty much at the same time, and also thought it could be a good starting point for further evaluation. Creating ".deb" files for Debian 12 should be feasible, since we're anyway running test cases + build binaries in a Debian Bookworm docker image.

Woazboat · 2024-07-09T22:09:30Z

All you need to do to create a .deb file is run make package. I actually have an additional patch that builds on this pull request (although it should also be possible without it) that uses the created .deb packages to install cgimap in the docker images. That also automatically takes care of installing the required runtime dependencies without having to list and update them all manually in the dockerfile.

Woazboat@d499553

tomhughes · 2024-08-06T21:23:34Z

Thanks for the suggestion - we now have an aptly based repo live at https://apt.openstreetmap.org/ which has replaced our PPA in production.

mmd-osm · 2024-08-10T07:16:53Z

CMakeLists.txt

+        set(CGIMAP_LIBS cgimap_core cgimap_fcgi cgimap_apidb libxml++)
+    endif()
+
+    if(CMAKE_VERSION VERSION_GREATER_EQUAL 3.23.0)


I tested static and shared builds, which works fine. It seems my local cmake version is a bit too old to test the header files. What is the expected behavior here? Will we always include header files in the .deb package?

I've seen you did some further changes on the topic in 6c2def5 ...
Still trying to understand what the picture will look like in the end.

The header files will only be included in the package if

the library targets containing them are being installed (only in a shared build here)
and

the Development component is enabled (enabled by default)

There was actually a name mismatch in the install command file set (s/HEADER/HEADERS/) that I fixed now. That's probably why you didn't see the header files.

I changed it now so that headers won't be included by default in the standard all-in-one install/package, only when a component based installation is performed (CPACK_DEB_COMPONENT_INSTALL is set to YES). With per-component packages you'll a different .deb file for each component that you can install individually, so installing the *-dev header package is optional.

Thank you for looking into this issue! Somehow, I still didn't make to generate multiple package

cmake .. -DBUILD_SHARED_LIBS=YES -DCPACK_DEB_COMPONENT_INSTALL=YES`

I'm pretty sure this is due to my cmake version 3.22.1. Now that the headers are not included by default, I'm good to merge this PR right now. I will add some further changes from test/381 branch to store Debian packages as build artifact to Debian 12 at least. Operations is waiting for fresh deb packages...

tomhughes · 2024-08-13T18:44:53Z

Moving the frontends to Debian 12 or Ubuntu 24.04 is getting a bit more urgent so do you have any idea when you might be in a position to provide packages?

mmd-osm · 2024-08-13T19:03:43Z

Re-publishing the current 0.9.3 release for Ubuntu 24.04 should be easy and shouldn't take too much time (again on Ubuntu launchpad). I still haven't figured out all details for the new apt.osm.org. Falling back to Ubuntu launchpad seems like a safe bet at the moment. Would that be ok?

tomhughes · 2024-08-13T20:10:26Z

We don't have any support for publishing to apt.osm.org yet so for now just build the packages wherever and I can load them manually.

We'd probably prefer just to jump to Debian 12 but I realise that's probably a bigger ask than Ubuntu.

mmd-osm · 2024-08-13T21:50:23Z

Ok, good. It seems getting the build up and running for Debian 12 should have priority over anything Ubuntu related.

In https://github.com/mmd-osm/openstreetmap-cgimap/tree/test/381 I have tested this PR with some more changes to generate and deploy a .deb package for Debian 12, with unit tests enabled.

The resulting docker image has the .deb file stored underneath /app_deb. This directory could be copied to the host machine, and then uploaded to apt.osm.org. Unfortunately, the package signing topic is not covered at the moment, and needs to be fixed urgently.

I totally want to avoid building anything on a local machine and then uploading binaries to some random place (in particular without signing packages).

It would be good to automate more of this as part of GitHub actions and use some Package registry to store results. As we have discussed in some other issue, my collaborator role is way too limited to do the proper config. We either need zerebubuth for that, or move the whole repo elsewhere.

One thing that's likely possible is to store the .deb build artifact on GH actions: https://github.com/actions/upload-artifact or combine it with the release process, like in https://github.com/trstringer/azblogfilter/blob/master/.github/workflows/release.yaml

mmd-osm · 2024-08-14T15:55:59Z

Storing .deb packages as build artifacts seems to work in general: https://github.com/mmd-osm/openstreetmap-cgimap/actions/runs/10390754008 (note: that's a test build only!)

mmd-osm · 2024-08-19T21:24:38Z

CMakeLists.txt

-               VERSION "0.10.0.${CURRENT_TIMESTAMP}"
-               DESCRIPTION "CGImap is a C++ implementation of some parts of the OpenStreetMap API as a FastCGI process."
-               HOMEPAGE_URL "https://github.com/zerebubuth/openstreetmap-cgimap")
+project(openstreetmap-cgimap


As a side note, this change is also visible in the generator header attribute returned by the endpoints:

generator="openstreetmap-cgimap 2.0.0 ..."

vs.

generator="CGImap 0.9.3 ..."

Woazboat commented Apr 1, 2024

View reviewed changes

Woazboat force-pushed the cmake-package-install-refactor branch from 7a4ba4b to 88fa46e Compare April 10, 2024 20:49

Change cmake project name to openstreetmap-cgimap, remove redundant c…

ae63892

…pack definitions

Woazboat force-pushed the cmake-package-install-refactor branch from 88fa46e to a73c3d9 Compare April 23, 2024 19:37

mmd-osm added the prio:low label Apr 24, 2024

mmd-osm force-pushed the master branch 2 times, most recently from b766d6a to 3b6c868 Compare May 10, 2024 10:25

mmd-osm reviewed Aug 10, 2024

View reviewed changes

mmd-osm mentioned this pull request Aug 14, 2024

Automatically create github release for tagged versions #332

Closed

Woazboat added 5 commits August 15, 2024 15:28

Use cmake GNUInstallDirs module for installation paths

4b4d7a6

Define components for cpack install targets

7a4d67d

Move all libxml++ lib headers into public file set

d392aa7

Rename libxml++ library to cgimap_libxml++

af65139

Exclude development headers from non component-based install by default

ba7229c

Woazboat force-pushed the cmake-package-install-refactor branch from a73c3d9 to ba7229c Compare August 15, 2024 14:17

mmd-osm merged commit d9ab969 into zerebubuth:master Aug 15, 2024
6 checks passed

mmd-osm reviewed Aug 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor cmake install and package definitions #381

Refactor cmake install and package definitions #381

Woazboat commented Apr 1, 2024

Woazboat Apr 1, 2024 •

edited

Loading

mmd-osm commented Apr 24, 2024

tomhughes commented Apr 25, 2024

mmd-osm commented Jul 9, 2024 •

edited

Loading

Firefishy commented Jul 9, 2024 •

edited

Loading

mmd-osm commented Jul 9, 2024

Woazboat commented Jul 9, 2024 •

edited

Loading

tomhughes commented Jul 9, 2024

Woazboat commented Jul 9, 2024 •

edited

Loading

mmd-osm commented Jul 9, 2024

Woazboat commented Jul 9, 2024 •

edited

Loading

tomhughes commented Aug 6, 2024

mmd-osm Aug 10, 2024 •

edited

Loading

Woazboat Aug 15, 2024

mmd-osm Aug 15, 2024

tomhughes commented Aug 13, 2024

mmd-osm commented Aug 13, 2024

tomhughes commented Aug 13, 2024

mmd-osm commented Aug 13, 2024 •

edited

Loading

mmd-osm commented Aug 14, 2024 •

edited

Loading

mmd-osm Aug 19, 2024

Refactor cmake install and package definitions #381

Refactor cmake install and package definitions #381

Conversation

Woazboat commented Apr 1, 2024

Woazboat Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

mmd-osm commented Apr 24, 2024

tomhughes commented Apr 25, 2024

mmd-osm commented Jul 9, 2024 • edited Loading

Firefishy commented Jul 9, 2024 • edited Loading

mmd-osm commented Jul 9, 2024

Woazboat commented Jul 9, 2024 • edited Loading

tomhughes commented Jul 9, 2024

Woazboat commented Jul 9, 2024 • edited Loading

Footnotes

mmd-osm commented Jul 9, 2024

Woazboat commented Jul 9, 2024 • edited Loading

tomhughes commented Aug 6, 2024

mmd-osm Aug 10, 2024 • edited Loading

Choose a reason for hiding this comment

Woazboat Aug 15, 2024

Choose a reason for hiding this comment

mmd-osm Aug 15, 2024

Choose a reason for hiding this comment

tomhughes commented Aug 13, 2024

mmd-osm commented Aug 13, 2024

tomhughes commented Aug 13, 2024

mmd-osm commented Aug 13, 2024 • edited Loading

mmd-osm commented Aug 14, 2024 • edited Loading

mmd-osm Aug 19, 2024

Choose a reason for hiding this comment

Woazboat Apr 1, 2024 •

edited

Loading

mmd-osm commented Jul 9, 2024 •

edited

Loading

Firefishy commented Jul 9, 2024 •

edited

Loading

Woazboat commented Jul 9, 2024 •

edited

Loading

Woazboat commented Jul 9, 2024 •

edited

Loading

Woazboat commented Jul 9, 2024 •

edited

Loading

mmd-osm Aug 10, 2024 •

edited

Loading

mmd-osm commented Aug 13, 2024 •

edited

Loading

mmd-osm commented Aug 14, 2024 •

edited

Loading