systemd service files and corresponding configs for Pelican services and Pelican-based OSDF services. #584

matyasselmeci · 2024-01-03T00:28:54Z

This will resolve #331

The osdf-cache and osdf-origin services are meant as a replacement for the 'stash-cache-auth' and the 'stash-origin-auth' xrootd instances of xcache, so they listen on the authenticated ports (8443 for the cache and 1095 for the origin).

bbockelm

Systemd services look mostly fine - just the question about whether we should also provide a sysconfig-based override.

The bigger question is about the config files. For the most part, what's filled in here is the same as the code defaults - my inclination would be to drop them.

Once you drop the defaults, then there are really only one or two settings. At that point, is it better to just have a single pelican.yaml/osdf.yaml shared by all services?

Obviously, Pelican needs to add a config.d-style system ASAP. If we assume one is done in February, does this change any of the approaches here?

systemd/osdf-cache.yaml

systemd/osdf-cache.service

bbockelm · 2024-01-05T00:42:49Z

systemd/osdf-cache.yaml

+  DirectorUrl: https://director-origins.osg-htc.org
+  TopologyNamespaceURL: https://topology.opensciencegrid.org/osdf/namespaces
+Server:
+  TLSCertificate: /etc/grid-security/xrd/xrdcert.pem


Is there not a better system location?

Not really. There's no 'standard' location for a host cert; Let's Encrypt puts the cert/key in /etc/letsencrypt/live/$domain for example. At least /etc/grid-security/xrd/xrdcert.pem is backward compatible with xcache-based cache/origin configurations; I can pick something else like /etc/pelican/{cert,key}.pem for the registry, director, and non-osdf configs.

I thought /etc/pki/tls/certs was standard? https://www.getpagespeed.com/server-setup/ssl-directory

systemd/osdf-cache.yaml

systemd/osdf-director.yaml

bbockelm · 2024-01-05T00:48:51Z

systemd/osdf-director.yaml

@@ -0,0 +1,17 @@
+Debug: false
+Logging:
+  Level: "Error"


Default values, I think. Can be removed.

systemd/osdf-director.service

bbockelm · 2024-01-05T00:51:16Z

systemd/osdf-origin.yaml

+  DetailedMonitoringHost: xrd-mon.osgstorage.org
+  Mount: "/mnt/osdf"
+  Port: 1095
+#  Authfile: /run/stash-origin-auth/Authfile


Let's remove commented-out lines (or add a comment about what they're for).

The plan is to do the latter.

Actually for now I'm going to uncomment them since we're going to need them until #564 is implemented.

matyasselmeci · 2024-01-05T15:30:55Z

The reason I had all those options even though they're the defaults is they're the most likely ones for the admin to change and I don't want them to have to go back and forth to the documentation to figure out which ones they are. I plan to add explanatory comments as well (but I will need help writing some of them).

…s from matyasselmeci's fork of the pelican repo see PelicanPlatform/pelican#584 git-svn-id: https://vdt.cs.wisc.edu/svn/native/redhat/branches/23-upcoming@27813 4e558342-562e-0410-864c-e07659590f8c

matyasselmeci · 2024-01-30T23:23:09Z

I think this is ready for another look. For the service files, I added sysconfig files, restarting on failure, and using a working directory we'll be able to write core files to. (The RPM will handle creating that directory.)

For the configs, I added comments describing many of the options. Biggest change is to have log files (/var/log/osdf-cache.log, /var/log/pelican-registry.log, etc.) instead of logging to the systemd journal. The osdf-* services now start the osdf binary so I was able to drop some options.

bbockelm

General approach is OK. I'm still ambivalent on having separate config files per service versus a shared one. I can see pros and cons for both directions. I'm going to defer to your experience here.

Approved! Let's get this in and move on to the RPM piece.

brianhlin

Minor comments, the big one being the standard cert location.

I wish we could consolidate the service files down to 1 or 2. Maybe we could use the following vars?

       ├───────────┼─────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
       │ "%i"      │ Instance name                                       │ For instantiated units this is the string between the first "@"      │
       │           │                                                     │ character and the type suffix. Empty for non-instantiated units.     │
       ├───────────┼─────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤
       │ "%p"      │ Prefix name                                         │ For instantiated units, this refers to the string before the first   │
       │           │                                                     │ "@" character of the unit name. For non-instantiated units, same as  │
       │           │                                                     │ "%N".                                                                │
       ├───────────┼─────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┤

And have users run something like systemctl start pelican@director? I think the only thing that would prevent that is that the director/registry don't have a working dir specified and the cache/origin do

brianhlin · 2024-01-31T15:56:33Z

systemd/osdf-cache.yaml

+  DirectorUrl: https://director-origins.osg-htc.org
+  TopologyNamespaceURL: https://topology.opensciencegrid.org/osdf/namespaces
+Server:
+  TLSCertificate: /etc/grid-security/xrd/xrdcert.pem


I thought /etc/pki/tls/certs was standard? https://www.getpagespeed.com/server-setup/ssl-directory

brianhlin · 2024-01-31T15:58:43Z

systemd/osdf-cache.yaml

+  ## Set Hostname to the external DNS name this can be accessed over, if
+  ## different than the current hostname.
+  # Hostname:
+Xrootd:


Is this case sensitive? If not, we should use XRootD

re. cert location: I guess your google-fu is better than mine: I didn't find a good standard last time I looked. FWIW, we need a cert/key with permissions for the xrootd user to read.

more re. cert location: Also the page you linked mentioned a directory but not a file name -- that is, it said the cert should be named after the FQDN; obviously I can't put that in the default config, not knowing what it is.

I think I prefer mostly aligning with best practices, e.g. /etc/pki/tls/certs/pelican.{crt,key}, than maintaining the old-world order

brianhlin · 2024-01-31T16:17:29Z

systemd/osdf-director.service

+EnvironmentFile = -/etc/sysconfig/osdf-director
+ExecStart = /usr/bin/osdf --config /etc/pelican/osdf-director.yaml director serve
+Restart = on-failure
+RestartSec = 20s


I see your commit removing the working dir for the director/registry but it's not clear to me why they don't need it

I suppose the director/registry can core dump too.

Most service files don't have a WorkingDirectory set; either the services run as root (in which case they can write their core files to /), or they run as non-root (in which case they start in their home directories and can write core files there). The reason we have to specify it in this case is that pelican starts as root but then spawns xrootd with reduced privileges (without changing the directory first).

I mainly asked with the idea of consolidating service files

systemd/osdf-cache.yaml

brianhlin · 2024-01-31T16:47:04Z

Mat and I discussed the split of systemd files and its affect on packaging and appropriate location for OSDF vs Pelican bits (we feel that OSG Software should package OSDF bits in a downstream metapackage). For the time being, this split of packages is ok and should allow for flexible future changes.

Mat convinced me that pelican@director isn't the right UX as it's not exactly an instance but I could see an interface like pelican-director@osdf or something like that in the future.

I think the cert location is the only thing that is left to address here before a merge.

…r Pelican services and Pelican-based OSDF services. The osdf-cache and osdf-origin services are meant as a replacement for the 'stash-cache-auth' and the 'stash-origin-auth' xrootd instances of xcache, so they listen on the authenticated ports (8443 for the cache and 1095 for the origin).

…t line to set scitokens.conf location

systemd conflicts are documented here: https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#Conflicts= If you try to start both services at the same time, it will be an error; otherwise, turning on a service turns off the service it conflicts with.

…ins" director and not the "caches" director

…lts from the yaml files

For OSDF origins and caches, add the commented-out option to switch back to the /etc/grid-security/certificates directory

… top-level "Debug" does

…nfigs we'll need them until Pelican can do authfile/scitokens.conf generation from Topology automatically

osdf is started as root; it can add the capabilities it wants to

…ache-public.yaml and osdf-cache-public.service for the public cache instances

…vice; pelican starts as root so it has all the capabilities it needs. See https://opensciencegrid.atlassian.net/browse/SOFTWARE-5795?focusedCommentId=383168 for an explanation.

…reate an osdf-origin-public service file and yaml

Either /var/spool/osdf for the osdf services or /var/spool/pelican for the pelican services. *Note*: the RPMs will have to create the directories.

…gin (which use xrootd), not the director and registry

…t values. Remove some options admins are less likely to change.

Issue PelicanPlatform#621 got fixed in main so we don't need it anymore

matyasselmeci force-pushed the pr/systemd branch 2 times, most recently from 7a7f0ed to 6e9f697 Compare January 4, 2024 19:11

bbockelm requested changes Jan 5, 2024

View reviewed changes

matyasselmeci force-pushed the pr/systemd branch 2 times, most recently from 7d0ed01 to a7b916d Compare January 17, 2024 03:23

matyasselmeci force-pushed the pr/systemd branch from a7b916d to 2e5be4d Compare January 22, 2024 22:25

matyasselmeci force-pushed the pr/systemd branch from d633371 to 74170cb Compare January 24, 2024 05:30

matyasselmeci requested review from bbockelm and brianhlin January 30, 2024 23:17

matyasselmeci mentioned this pull request Jan 30, 2024

Add log rotation for pelican/osdf service log files #728

Closed

matyasselmeci force-pushed the pr/systemd branch from 2390dc1 to cee75f5 Compare January 30, 2024 23:37

bbockelm approved these changes Jan 31, 2024

View reviewed changes

brianhlin reviewed Jan 31, 2024

View reviewed changes

matyasselmeci added 13 commits January 31, 2024 16:41

Move EnableUI from Origin section to Server section; add commented-ou…

43af2a8

…t line to set scitokens.conf location

Specify the director URL in the osdf-cache config so we hit the "orig…

2612910

…ins" director and not the "caches" director

Use the osdf binary for the osdf-* services and remove the osdf defau…

a0128f8

…lts from the yaml files

Remove Xrootd knobs from the registry and director yamls

c2aee1e

Remove superfluous Xrootd.Port from the cache configs

74be79e

Use the system CA certs bundle by default

7112ed7

For OSDF origins and caches, add the commented-out option to switch back to the /etc/grid-security/certificates directory

Comment out the logging options but add a comment explaining what the…

cf5eedf

… top-level "Debug" does

Uncomment Xrootd.Authfile and Xrootd.ScitokensConfig knobs in osdf co…

d86de58

…nfigs we'll need them until Pelican can do authfile/scitokens.conf generation from Topology automatically

Caches use Cache.DataLocation instead of Xrootd.Mount

703d29b

Drop unnecessary "Capabilities" lines:

ec8bfbc

osdf is started as root; it can add the capabilities it wants to

We don't need the Topology-provided scitokens.conf since Pelican 7.4.0

bc511b5

matyasselmeci added 13 commits January 31, 2024 16:41

osdf-cache, osdf-origin: send logs to a file instead of system journal

6c0ed2e

Update osdf-cache.yaml with changes from sc-cache testing; add osdf-c…

9873049

…ache-public.yaml and osdf-cache-public.service for the public cache instances

Delete osdf-origin-multiuser.service and pelican-origin-multiuser.ser…

398fa2a

…vice; pelican starts as root so it has all the capabilities it needs. See https://opensciencegrid.atlassian.net/browse/SOFTWARE-5795?focusedCommentId=383168 for an explanation.

Update odsf-origin.yaml with changes based on testing on sc-origin; c…

3e436cd

…reate an osdf-origin-public service file and yaml

Restart services on failure

11cbdb0

Set the working directories for the services

9804d36

Either /var/spool/osdf for the osdf services or /var/spool/pelican for the pelican services. *Note*: the RPMs will have to create the directories.

Only use /var/spool/osdf and /var/spool/pelican for the cache and ori…

ab33e7a

…gin (which use xrootd), not the director and registry

Remove -public files; one cache/origin will serve both public and auth

146571b

Add support for sysconfig (environment) files

a7fc7c1

Improve comments. Have all services use log files. Comment out defaul…

b8be703

…t values. Remove some options admins are less likely to change.

Remove ExportVolume line from osdf-origin config

8960e0e

Issue PelicanPlatform#621 got fixed in main so we don't need it anymore

Use preferred case for XRootD

9ba4324

Explain why I'm setting Federation.DirectorUrl in osdf-cache

68943a7

matyasselmeci force-pushed the pr/systemd branch from d0b43e4 to 68943a7 Compare January 31, 2024 22:41

matyasselmeci marked this pull request as ready for review February 1, 2024 00:11

matyasselmeci merged commit 9474446 into PelicanPlatform:main Feb 1, 2024
9 checks passed

matyasselmeci deleted the pr/systemd branch February 1, 2024 15:10

matyasselmeci mentioned this pull request Feb 1, 2024

Add pelican-server RPM #331

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd service files and corresponding configs for Pelican services and Pelican-based OSDF services. #584

systemd service files and corresponding configs for Pelican services and Pelican-based OSDF services. #584

matyasselmeci commented Jan 3, 2024 •

edited

Loading

bbockelm left a comment

bbockelm Jan 5, 2024

matyasselmeci Jan 5, 2024

brianhlin Jan 31, 2024

bbockelm Jan 5, 2024

bbockelm Jan 5, 2024

matyasselmeci Jan 5, 2024

matyasselmeci Jan 5, 2024

matyasselmeci commented Jan 5, 2024

matyasselmeci commented Jan 30, 2024

bbockelm left a comment

brianhlin left a comment

brianhlin Jan 31, 2024

brianhlin Jan 31, 2024 •

edited

Loading

matyasselmeci Jan 31, 2024

matyasselmeci Jan 31, 2024

brianhlin Feb 1, 2024

matyasselmeci Feb 1, 2024

brianhlin Jan 31, 2024

matyasselmeci Jan 31, 2024

brianhlin Jan 31, 2024

brianhlin commented Jan 31, 2024

systemd service files and corresponding configs for Pelican services and Pelican-based OSDF services. #584

systemd service files and corresponding configs for Pelican services and Pelican-based OSDF services. #584

Conversation

matyasselmeci commented Jan 3, 2024 • edited Loading

bbockelm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matyasselmeci commented Jan 5, 2024

matyasselmeci commented Jan 30, 2024

bbockelm left a comment

Choose a reason for hiding this comment

brianhlin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianhlin Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianhlin commented Jan 31, 2024

matyasselmeci commented Jan 3, 2024 •

edited

Loading

brianhlin Jan 31, 2024 •

edited

Loading