rdo-packaging.txt

= RDO Openstack Packaging
Author: Jakub Ružička <jruzicka at redhat dot com>
:toc2:
:icons:
:iconsdir: ../images/icons/
:numbered:
:source-highlighter: highlightjs
:rdo: https://rdoproject.org/
:rdorepo: http://rdo.fedorapeople.org
:f21-juno-repo: https://repos.fedorapeople.org/repos/openstack/openstack-juno/fedora-21/
:f20-icehouse-repo: https://repos.fedorapeople.org/repos/openstack/openstack-icehouse/fedora-20/
:redhat: http://www.redhat.com
:redhat-openstack-org: https://github.com/redhat-openstack/
:openstack-packages-org: https://github.com/openstack-packages/
:rhosp: http://www.redhat.com/openstack

Introduction
------------

This document attempts to be the definitive source of information about
{rdo}[*RDO*] OpenStack packaging for developers and packagers.

It's written in `asciidoc` and you can edit it using your favorite text
editor:

[source,bash]
$> git clone https://github.com/redhat-openstack/openstack-packaging-doc.git
# Edit doc/rdo-packaging.txt
# `make` to build HTML, `autobuild.sh` to autobuild on file changes
$> git commit -a
# Send Pull Request on GitHub!

Packaging workflow
~~~~~~~~~~~~~~~~~~

[[rdo-packaging-workflow]]
.RDO packaging workflow
image::../images/rdo-workflow.jpg[link="images/rdo-workflow.jpg"]


Before RDO Liberty
++++++++++++++++++

All {rdo}[*RDO*] packaging resources are maintained in the Fedora
<<dist-git,dist-git>> repositories. This includes support for all
Fedora distros of course, but currently also caters for EL6 and EL7
distros and derivatives. In future the EL resources may be separated
out under the CentOS OpenStack SIG umbrella, but for now Fedora is the
primary `git` source.

Currently there is a 1:1 mapping between Fedora and OpenStack releases.
I.E. the OpenStack package versions that are provided through the
official Fedora repositories are:

 * f23 = Kilo
 * f22 = Juno
 * f21 = Icehouse

{rdo}[RDO] also provides one release ahead for Fedora. There are
separate {rdorepo}[RDO repositories] that provide more recent
OpenStack packages on the current stable version of Fedora.

 * {f21-juno-repo}[Juno for f21]
 * {f20-icehouse-repo}[Icehouse for f20]

The EL6 and EL7 packages are only provided through the {rdo}[*RDO*]
repositories.

Historically, there were separate `el6-icehouse`, `el6-havana`, etc.
branches in Fedora <<dist-git,dist-git>> but Fedora and EL converged
over time thanks to the release of EL7 and also due to the extra
flexibility that the separate RDO repos provide by allowing to carry
updated packages that may not be compatible with all of EPEL,
leveraging the fact that EL deployers almost universally dedicate
systems to OpenStack services. So going forward builds for all
platforms should work from the same `.spec` file. I.E. we only use the
Fedora branches corresponding to each OpenStack release (`master`,
`f21`, `f20`, ...) and EL6 and EL7 builds are done from those branches
in https://copr.fedoraproject.org/coprs/jruzicka/[copr], as detailed
below in <<rdo-pkg-guide,RDO Packaging Guide>>.


RDO Liberty and later releases
++++++++++++++++++++++++++++++

Packaging resources for OpenStack services are now hosted in the
{openstack-packages-org}[openstack-packages] github organization.

The layout used for branches is:

 * rpm-<master> = tracks upstream master branch (Delorean)
 * rpm-<release> = tracks upstream stable/<release> branch (Delorean)
 * rdo-<release> = based upon rpm-<release> and used to build RDO stable packages

Clients and general-purpose libraries are still **included** in Fedora.


[[dist-git]]
dist-git - where the .spec file lives
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*dist-git* is a git repository which contains `.spec` file used for building
a RPM package. It also contains other files needed for building source RPM
such as patches to apply, init scripts etc.

RDO packages currently use
http://pkgs.fedoraproject.org/cgit/openstack-nova.git/[Fedora dist-git] and
`fedpkg` is used to obtain it, for example:

[source,bash]
$> fedpkg clone openstack-nova
$> cd openstack-nova
$> git remote -v
origin    ssh://jruzicka@pkgs.fedoraproject.org/openstack-nova (fetch)
origin    ssh://jruzicka@pkgs.fedoraproject.org/openstack-nova (push)

You can inspect the package history using `git`:

[source,bash]
$> git log --oneline
e70e895 Update to upstream 2014.2.b2
6d4b2ae Updated patches from master-patches
4113b4f maint: modernize the systemd build deps
9bec9a7 Update to latest stable/icehouse 2014.1.1
5963f2d - Rebuilt for https://fedoraproject.org/wiki/Fedora_21_Mass_Rebuild
d398372 Increase release number
e203465 Remove qpid settings from dist conf

[[patches-branch]]
Patches branch
~~~~~~~~~~~~~~

Because we rebase and backport often, manual management of patch files in
<<dist-git,dist-git>> would be _unbearable_. That's why each dist-git branch
has an associated **patches branch** which contains upstream git tree with
extra downstream patches on top. A dist-git can be updated by `rdopkg` to
include patches from *patches branch* so maintaining RPM patches is a matter
of `git` magic on patches branch - rebase on upstream tag, cherry-pick
patches, etc.

{rdo}[*RDO*] patches branches live at
{redhat-openstack-org}[github.com/redhat-openstack] and follow few
conventions:

Each patches branch:

* ends with `-patches` suffix by convention
* contains upstream project `git` tree up to a certain commit called
  _patches base_, usually a version tag like `2014.2`
* has downstream patches rebased on top _patches base_


patches branch workflow
^^^^^^^^^^^^^^^^^^^^^^^
....
     +------------------------+
     |        upstream        |
     |  github.com/openstack  |
     +------------------------+
                 |
      git rebase | git cherry-pick
                 V
  +-------------------------------+
  |       patches branch          |
  |  github.com/redhat-openstack  |
  +-------------------------------+
                 |
                 | rdopkg update-patches
                 V
     +--------------------------+
     |        dist-git          |
     |  pkgs.fedoraproject.org  |
     +--------------------------+
....


[[release-overview]]
Releases & dists overview
~~~~~~~~~~~~~~~~~~~~~~~~~

NOTE: Use `rdopkg info` to get up-to-date release & dists overview.

* **Juno** (latest release)
** _Fedora 22_
*** available from official Fedora repos
*** built through `koji` (currently `master` branch)
** `fedora-21`
*** RDO repo
*** Fedora 22 (`master`) builds from `koji` reused
** `fedora-20`
*** RDO repo
*** symlink to `fedora-21`
** `epel-7`
*** RDO repo
*** built in https://copr.fedoraproject.org/coprs/jruzicka/rdo-juno-epel-7/[copr]

* **Icehouse**
** _Fedora 21_
*** available from official Fedora 21 repos
*** built through `koji` (`f21` branch)
** `fedora-20`
*** RDO repo
*** Fedora 21 builds from `koji` reused
** `epel-7`
*** RDO repo
*** built in https://copr.fedoraproject.org/coprs/jruzicka/rdo-icehouse-epel-7/[copr]
** `epel-6`
*** RDO repo
*** built in https://copr.fedoraproject.org/coprs/jruzicka/rdo-icehouse-epel-6/[copr]


[[rdopkg]]
rdopkg
~~~~~~

https://github.com/redhat-openstack/rdopkg[rdopkg] is a command line tool
that automates many operations on RDO packags including:

 * rebases to new version
 * introducing patches
 * modifying .spec file: bumping versions, managing patches, writing
   changelog, producing meaningful commit messages, ...
 * build packages in supported build systems

On top of that, `rdopkg` also serves as {rdo}[*RDO*] update/CI frontend,
it can obtain various information about RDO for you and more.

`rdopkg` is a Swiss army knife of RDO packaging and it avoids a number of
repetitive and error prone processes involving several underlying tools, each
with its own quirks.

See https://github.com/redhat-openstack/rdopkg[rdopkg on github] and also
https://www.rdoproject.org/packaging/rdopkg/rdopkg.1.html[man rdopkg].

[[master-pkg-guide]]
Master Packaging Guide
----------------------

In order to build an `RPM` with the master packaging you'll need to
install https://github.com/openstack-packages/delorean[delorean],
following the instructions described in this
https://github.com/openstack-packages/delorean/blob/master/README.rst[README].

Run Delorean
~~~~~~~~~~~~
Run `delorean` for the package you are trying to build.

[source,bash]
$> delorean --config-file projects.ini --local --package-name openstack-cinder

This will clone the packaging for the project you're interested in into
`data/openstack-cinder_repo`, you can now change this packaging and
rerun the `delorean` command in test your changes.

If you have locally changed the packaging make sure to include `--dev`
in the command line. This switches `delorean` into "dev mode" which
causes it to preserve local changes to your packaging between runs so
you can iterate on spec changes. It will also cause the most current
public master repository to be installed in your build image(as some
of its contents will be needed for dependencies) so that the packager
doesn't have to build the entire set of packages.

The output from `delorean` is a repository containing the packages you
just built along with the most recent successfully built version of
each package. To find the most recent repository follow the symbolic
link `./data/repos/current`

Submitting changes to gerrit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once you are happy that you have your changes ready to be reviewed,
create a `git commit` with an appropriate comment, add a `git remote`
pointing to gerrit and then submit your patch

[source,bash]
$> git clone -o gerrit https://review.gerrithub.io/openstack-packages/<package-name>
$> git commit -p
$> git review rpm-master

Browsing gerrit for reviews
~~~~~~~~~~~~~~~~~~~~~~~~~~~

To look at all open patches for the upstream packaging simply use the following link
https://review.gerrithub.io/#/q/status:open+project:^openstack-packages/.*,n,z

Differences between master and rawhide packaging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are a number of expected differences between the master packaging and
the packaging in rawhide

* The `delorean` packaging has had `Version:` and `Release:` fields
  both set to `XXX` in the `delorean` packaging as we take both of
  these from the tags set on the git repositories

* Because we are packaging master, patches aren't backported into the
  `delorean` packaging

* All of the master specs contain a reference to `%{upstream_version}`
  in the `%setup macro`, this is because the subdirectory contained in
  the source tarball contains both the version and release, this is
  being passed into `rpmbuild`. In the Fedora packaging, spec can
  include compatibility macro e.g. http://pkgs.fedoraproject.org/cgit/openstack-nova.git/tree/openstack-nova.spec
  to avoid conflicts when backporting change from master packaging.

* %changelog section is empty in master packaging

* The files `sources` and `.gitignore` have been truncated in the
  master packaging

* In `%files` avoid using `%{version}` and use instead wildcard `*`

To Assist in identifing difference the report being output by the
production `delorean` includes a spec delta link showing a diff
between the two repositories:

* Fedora 21: http://trunk.rdoproject.org/f21/report.html

* CentOS 7: http://trunk.rdoproject.org/centos70/report.html


How to add a new package to RDO master packaging
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Prepare a public git repository with the initial .spec in rpm-master branch.
** Please don't reinvent a wheel. There is a great chance that some similar projects are already packaged for RDO master. For example, if you need to package a vendor library for Neutron, please base your new package on one of python-networking-* packages already in the project. This will help to avoid unneeded diversity, and should give you a better clue about RDO master expectations as for the style and quality of packaging.
** Repository name should be equal to the upstream source repository name for standalone projects e.g. nova and with python- prefix for libraries and clients e.g. python-networking-arista. Please also consider aligning the name with other packages that are already present in RDO master.
** Add .gitreview e.g. https://github.com/openstack-packages/nova/blob/rpm-master/.gitreview
** Please note differences to Fedora Rawhide packaging described in the previous section!

* Ask one of the owners in (https://github.com/orgs/openstack-packages/people[openstack-packages org]) to create a new repository and import from your public repository.

** Owners: after creating and importing repository in github, replicate it to gerrithub by selecting openstack-packages at https://review.gerrithub.io/plugins/github-plugin/static/repositories.html. After import, add rdo-packagers group as an Owner in https://review.gerrithub.io/#/admin/projects/openstack-packages/PROJECT,access

* Submit all further updates via gerrithub.

* When ready, send pull-request to add the package in rdoinfo e.g. https://github.com/redhat-openstack/rdoinfo/pull/30


[[rdo-pkg-guide]]
RDO Packaging Guide
-------------------

This guide attempts to describe entire process of packaging new version of
an OpenStack package and getting it into {rdo}[*RDO*].

`$PROJECT` variable is used throughout the guide. Substitute it with your
OpenStack project such as `nova`, `neutron`, `heat` etc.

Note that some projects such as client libraries follow slightly different
naming conventions.

Also Note that any packaging updates going into RDO should first be submitted
to the master packaging (described above) then cherry picked into rawhide.


[[prereqs]]
Prerequisites
~~~~~~~~~~~~~

* You need to have
  *Fedora account* footnote:[also known as FAS account - https://admin.fedoraproject.org/accounts/[Fedora Account System]]
  and be *Fedora packager* in order to interact with Fedora
  build system (https://koji.fedoraproject.org/koji/[koji])
  and update system (https://admin.fedoraproject.org/updates[bodhi]). +
  https://fedoraproject.org/wiki/Join_the_package_collection_maintainers[Join the package collection maintainers]
  if you haven't yet!

* TBD Gerrit account for submitting RDO updates

Permissions You'll Need
^^^^^^^^^^^^^^^^^^^^^^^

* *Fedora <<dist-git,dist-git>>*: visit
  https://admin.fedoraproject.org/pkgdb/package/openstack-nova/[pkgdb] for
  your package, `+ Request Commit Access` and optionally poke Main Contact or
  Package Administrator of your choice.

* *<<patches-branch,patches branches>> @ `github.com/redhat-openstack`*: request
  permissions for https://github.com/redhat-openstack/nova[your project] from a
  https://github.com/orgs/redhat-openstack/teams/owners[redhat-openstack
  admin]. Note that you're looking for `/$PROJECT`, not
  `/openstack-$PROJECT`. You can tell you're in the right repo when
  there are branches with `-patches` suffix.

* *copr build system*: Request build permissions in relevant
  https://copr.fedoraproject.org/coprs/jruzicka[jruzicka's coprs] listed
  below. Check the `Is Builder` box and hit `Update` for each copr. Poke
  `jruzicka` to accelerate the approval.

** Juno copr:
*** https://copr.fedoraproject.org/coprs/jruzicka/rdo-juno-epel-7/permissions/[rdo-juno-epel-7]

** Icehouse coprs:
*** https://copr.fedoraproject.org/coprs/jruzicka/rdo-icehouse-epel-6/permissions/[rdo-icehouse-epel-6]
*** https://copr.fedoraproject.org/coprs/jruzicka/rdo-icehouse-epel-7/permissions/[rdo-icehouse-epel-7]


Software
^^^^^^^^

Fedora packaging tools
++++++++++++++++++++++

`fedora-packager` package available in standard Fedora repositories pulls in
tools for packaging and interacting with Fedora infrastructure including
`fedpkg` and https://koji.fedoraproject.org/koji/[koji],
https://admin.fedoraproject.org/updates[bodhi], and
https://admin.fedoraproject.org/pkgdb/[pkgdb] clients.

[source,bash]
$> yum install fedora-packager

You need to setup your FAS certificate to communicate with Fedora systems:

[source,bash]
$> fedora-packager-setup


`rdopkg`
++++++++

See https://github.com/redhat-openstack/rdopkg[rdopkg on github] for all
installation options. I reccomend using
https://copr.fedoraproject.org/coprs/jruzicka/rdopkg[jruzicka/rdopkg copr],
see the link for instructions on howto enable the copr. Once enabled,

[source,bash]
$> yum install -y rdopkg


`rdopkg coprbuild`
++++++++++++++++++

In order to use `rdopkg coprbuild` action to easily build packages in
https://copr.fedoraproject.org/coprs/jruzicka[jruzicka's coprs], you need to
save your copr API token into `~/.config/copr` as described in
http://copr.fedoraproject.org/api.

Although `rdopkg` shares this config with `copr-cli` (you can use `copr-cli`
to validate it), `rdopkg` has it's own copr client implementation
footnote:[`copr-cli` was not reusable at the time of writing] and doesn't
require `copr-cli` to
work.


Initial repository setup
~~~~~~~~~~~~~~~~~~~~~~~~

With all the <<prereqs,prerequisites>> met, it's time to clone the package
<<dist-git,dist-git>> using `fedpkg`.

[source,bash]
$> fedpkg clone openstack-$PROJECT
$> cd openstack-$PROJECT

You also need to be able to access to your project's <<patches-branch,patches
branches>>. +
`rdopkg` expects the `git` remote containing <<patches-branch,patches
branches>> to be named `patches`:

[source,bash]
git remote add -f patches git@github.com:redhat-openstack/$PROJECT.git

Finally, you need a remote with upstream source in order to cherry-pick
upstream patches and rebase <<patches-branch,patches branch>> on new
version:

[source,bash]
$> git remote add -f openstack git://github.com/openstack/$PROJECT.git

You can check your setup with:

[source,bash]
$> rdopkg pkgenv


Requirements management
~~~~~~~~~~~~~~~~~~~~~~~

Please see link:rdopkg/rdopkg-adv-requirements.7.html[man
rdopkg-adv-requirements] for complete introduction to `rdopkg` requirements
management magicks.


Modifying the package
~~~~~~~~~~~~~~~~~~~~~

The package metadata is stored in `.spec` file which is found in top
directory of <<dist-git,dist-git>>, usually `openstack-$PROJECT.spec`. Every
change to a package requires at least a change in `.spec` file and possibly
other <<dist-git,dist-git>> files (add/remove patches, init scripts, ...).

Every time you do a new build, you need to:

* Bump `Release` (when source tarball stays the same) or change `Version`  and
  reset `Release` (on update to new upstream version/tarball) in `.spec` file.
* Provide useful `%changelog` entry at the end of `.spec` file describing your
  change.
* Commit the <<dist-git,dist-git>> changes with meaningful commit message.

Following sections provide examples for most common packaging scenarios.

Simple `.spec` fix
^^^^^^^^^^^^^^^^^^

The simplest kind of change that doesn't introduce/remove patches or
different source tarball.

* Make required changes.
* Bump `Release`.
* Provide useful `%changelog` entry describing your change.
* Commit the <<dist-git,dist-git>> changes with meaningful commit message.

Although this change is simple, `rdopkg fix` can still make some string
manipulation for you. In following example, I add a new dependency to `nova`
package:

[source,bash]
$> cd openstack-nova
$> rdopkg fix

Action required: Edit .spec file as needed and describe changes in
changelog. Once done, run `rdopkg -c` to continue:

[source,bash]
$> vim openstack-nova.spec
# Add Requires line and describe the change in %changelog
$> rdopkg -c

After this, `rdopkg` generates new commit from the %changelog entry you
provided and displays the diff:

    Name:             openstack-nova
    Version:          2014.2
   -Release:          0.1.b2%{?dist}
   +Release:          0.2.b2%{?dist}
    Summary:          OpenStack Compute (nova)
    
    ...
    
   +Requires:         banana
    Requires:         python-nova = %{version}-%{release}
    
    ...

    %changelog
   +* Tue Aug 12 2014 Jakub Ruzicka <jruzicka@redhat.com> 2014.2-0.2.b2
   +- Require banana package for the lulz
   +
    * Sun Aug 03 2014  Vladan Popovic <vpopovic@redhat.com> 2014.2-0.1.b2
    - Update to upstream 2014.2.b2
    - openstack-nova-compute should depend on libvirt-daemon-kvm, not libvirt - rhbz#996715


Introducing/removing patches
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When new patches are introduced to <<patches-branch,patches branch>> (usually
by `git review`), <<dist-git,dist-git>> must be updated with new patches files
and `.spec` file patch definitions.

`rdopkg` provides:

* `rdopkg update-patches`: low-level action that only updates `.spec` file
  from local <<patches-branch,patches branch>>
* `rdopkg patch`: high-level action on top of `update-patches` with extra
  features (reset local patches branch from remote, bump release, generate
  changelog entry)

Because a lot of the packages are python and use setuptools and pbr, some patches may not apply correctly because the sdist tarball changes the format of setup.cfg.  This will show up during rpm build time with an error applying the patch.  This issue can be worked around using the following process.

* unpack the source tarball
* git clone the patches branch
* copy the config.cfg file from the tarball into your git tree
* git commit the change
* git revert HEAD
* edit the patches_base line to # patches_base=+1
* reorder the patches on the patches branch so that the initial change is first, then the revert is second after the tarball
* rdopkg patch uses the patches_base line to skip the first patch but apply the second

This results in patches that apply to the upstream git tree applying without modification on the tarball.


Rebasing to new version
^^^^^^^^^^^^^^^^^^^^^^^

See
link:rdopkg/rdopkg-adv-new-version.7.html[man rdopkg-adv-new-version]  for
an example of using `rdopkg new-version` to update a package to a new upstream
version.


Building the package
~~~~~~~~~~~~~~~~~~~~

After you have modified <<dist-git,dist-git>> to your liking, it's time to build.
You need to be on the dist-git branch of respective Fedora release such as
`master` or `f21`, see <<release-overview>>.


local build with mock
^^^^^^^^^^^^^^^^^^^^^

You can build the Fedora package locally using `mock` to ensure it builds
and/or test it:

[source,bash]
$> fedpkg mockbuild

Find the results in `./results_$PACKAGE/$VERSION/$RELEASE` directory. The
most interesting file is `build.log`.

You can also test EL builds using `mock`, but you may be missing certain build
requirements only available at {rdo}[*RDO*]
https://copr.fedoraproject.org/coprs/jruzicka/[coprs]:

[source,bash]
$> fedpkg --dist el7 mockbuild --root epel-7-x86_64


copr build of EL package
^^^^^^^^^^^^^^^^^^^^^^^^

Beginning with the Icehouse release, EL packages are built in
https://copr.fedoraproject.org/coprs/jruzicka/rdo-juno-epel-7/[copr] build
system due to constraints of Fedora <<dist-git,dist-git>>.

Because a push to Fedora dist-git is irreversible, building in `copr` first is
recommended. You may find some EL specific problems and solve them without the need
of new package release.

The following command creates a source RPM for selected distro using
`fedpkg srpm`, uploads it to your `fedorapeople.org` space and submits it to
be built in `copr`:

[source,bash]
$> rdopkg coprbuild -d epel-7 -f up.yml

Also, `-f up.yml` instructs `rdopkg` to dump the build information
into `up.yml` file that can be used later to submit an {rdo}[*RDO*]
update using `rdopkg update -f up.yml`.

See a full example of using `rdopkg coprbuild` in
link:rdopkg/rdopkg-adv-building.7.html[man rdopkg-adv-building].


koji build of Fedora package
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Push your Fedora <<dist-git,dist-git>> changes and build using `fedpkg`:

[source,bash]
$> git push
$> fedpkg build


Submitting an update
~~~~~~~~~~~~~~~~~~~~

After your builds finished, you need to submit the package(s) for
update in {rdo}[*RDO*] using:

[source,bash]
$> rdopkg update

By default, `rdopkg update` tries to autodetect which package you're trying to
update and opens an YAML update file template in text editor. You need to
specify all builds that should be updated at once in the update file.

See **ACTION: update** in
link:$$rdopkg/rdopkg.1.html#_action_update$$[man rdopkg].
and also
link:rdopkg/rdopkg-adv-building.7.html[man rdopkg-adv-building].
for an example of how to do `copr` and `koji` builds and prepare and submit the
update file non-interactively.

`rdopkg update` submits the update through `git review` and prints the URL to
gerrit. The update needs to pass CI in order to be merged and later pushed to
{rdo}[*RDO*] repositories.

To track your update, use:

[source, bash]
$> rdopkg list-updates -r

Fedora update
^^^^^^^^^^^^^

[source,bash]
$> fedpkg update

NOTE: TODO