Replace xenon_secrets with utilix config file

[ershockley]

This should not be merged yet

This PR is a proposal to replace the xenon_secrets python file with the configuration file handled by utilix. This is helpful for running strax on e.g. OSG or anywhere else where a xenon_secrets.py file might not be present. I also prefer this type of text file configuration, which one could modify and/or have multiple copies of in case you want to change anything (e.g. I have a 1T config and an nT one, and just need to change XENON_CONFIG env variable to switch between the two).

In addition, it uses the utilix wrapper around pymongo, which increases readability and centralizes the runDB stuff.

I'm still testing some things, but wanted to get people's thoughts on something like this. If we don't want to go this route I need a different solution for OSG.

As an example, my utilix config file now looks like this (removing the actual content to be overly cautious)

[RunDB]
rundb_api_url =
rundb_api_user =
rundb_api_password = 
pymongo_url = 
pymongo_user = 
pymongo_password = 
pymongo_database = 

[straxen]
s3_access_key_id = 
s3_secret_access_key = 

In this way we can combine configuration of many things into a single file. For more info on utilix config, see here.

This should not be merged yet

[JoranAngevaare]

Hi Evan, thanks for the proposal. A couple of ideas:

• Why can't we keep the xenon-secrets in case we don't have access to the environment variables? I for instance utilize this on the ebs as when I initialize bootstrax over SSH I don't have access to the environment variables. I think adding it can be useful but I don't see a reason not to keep the xenon-secrets.
• Great if utilix improves reliability of the rundb access. Is it also easier to initialize the runsdb via utilix if not on dali (e.g. your own laptop)? Briefly looking at it it doesn't seem fit for running bootstrax right as I want admin rights to push data to the rundb.
• How would the utilix wrapper change the way the user interacts with the runDB storage? Can it access more/have more rights/need to do other commands?

[ershockley]

Hi Joran,

• I'm not totally opposed to keeping xenon_secrets also -- I just think would be nice to have a single xenon config file with categories for different pieces of software. We could use xenon_secrets for that also but I personally just prefer a config file type thing rather than importing from a python file, super minor point though. '

• I don't understand your point on the environment variables. By default utilix looks for a config file in your home directory, so if XENON_CONFIG isn't set it will try to load ~/.xenon_config. Does this resolve your issue?

• You can definitely push data to the runDB with utilix. All it is doing is putting the boilerplate collection setup into a single place. You set the user/passsword/database/url in the runDB so it should be pretty flexible, I think.

Does this answer your questions?

[JoranAngevaare]

but I personally just prefer a config file type thing rather than importing from a python file

On the other hand keeping it as a backup option wouldn't be a bad idea.

I don't understand your point on the environment variables. By default utilix looks for a config file in your home directory, so if XENON_CONFIG isn't set it will try to load ~/.xenon_config. Does this resolve your issue?

Yeah sorry it's some edge case I ran into I fixed with the xenon-sectrets but adding a ~/.xenon_config seems similar to the xenon_secrets so that sounds like it work, guess would have to try.

You can definitely push data to the runDB with utilix. All it is doing is putting the boilerplate collection setup into a single place. You set the user/passsword/database/url in the runDB so it should be pretty flexible, I think.

OK, I also just looked very briefly and saw that the secondary was preferred which is readonly usually right?
https://github.com/XENONnT/utilix/blob/master/utilix/rundb.py#L345

I guess I'd be nice to add the required kwargs into this part of the straxen RunDB.py but this would presumably be one of those things we could fix if people agree on your proposal.

[ershockley]

I will test some of these things more thoroughly and get back to Joran (also with more examples etc). I will be mostly offline for the next ~ week or more so it might be a bit.

Thanks for the comments!

[ershockley]

I would like to revive this PR, as this change is important for any future reprocessing campaign. There are two main goals here:

1. Replace the xenon_secrets.py file with what I refer to as a xenon_config file (see here. As it stands now this change is required in order to process data on OSG (though there are ways around this if we want to keep xenon_secrets). This xenon_config file can be used by other software outside of straxen.

2. Streamline our analysis tools so that they talk to each other more. If all of our tools access the runsDB using utilix, then any future change in runDB settings (urls, passwords, etc), as happened a few times in XENON1T, can be easily applied by just updating utilix, and we don't have to make hardcoding changes in many different packages. I also think the general xenon_config file will be useful in the future.

@jorana thoughts? Not sure I addressed your concerns from before or not.

[ershockley]

Maybe I should add that, to make this work, one needs a xenon_config file of the following syntax:

[RunDB]
pymongo_url = 
pymongo_user =
pymongo_password = 
pymongo_database = 

[straxen]
s3_access_key_id = 
s3_secret_access_key = 

This is what is replacing the xenon_secrets file. When you enter a singularity container on midway, by default the config loaded is in a file called xenon.config, in same place as current xenon_secrets file. But if you are running somewhere else (DAQ machines, e.g.) you will need to create your own config file.

[ershockley]

Now added GitHub actions to automatically update the context collection of the RunDB whenever there is a new straxen release. I've tested a slightly different version of this with my personal fork of utilix but have not tested this exact one.

[JoranAngevaare]

@ershockley , let me know when this is ready for review by 'un-drafting' the PR.

[tunnell]

Wouldn't this be easy to do with the corrections system? This could be for strax settings beyond physics. @ahiguera-mx thoughts?

[ershockley]

@tunnell I don't understand what you mean, sorry. This change would replace e.g. how straxen sets up the runDB, which is needed to access correction system in the first place right?

[WenzDaniel]

Are not you missing here something after the raise?

[ershockley]

Yes thanks!

[ershockley]

@ershockley I also had a look at the /project2/lgrandi/xenonnt/xenon.config file but from experience we know that the way pymongo_url is specified does not work see the related PR:
#213

@jorana I just tested on midway and using all 3 works fine

>>> from utilix.rundb import pymongo_collection

>>> coll = pymongo_collection()

>>> coll.find_one({'number': 10000}, {'number': 1})
{'_id': ObjectId('5fa30d218471e9d11dfd5276'), 'number': 10000}

>>> from utilix.config import Config
>>> cfg = Config()
>>> cfg.get('RunDB', 'pymongo_url')
'xenon-rundb.grid.uchicago.edu:27017,xenon1t-daq.lngs.infn.it:27017,fried.rice.edu:27017/xenonnt'

I do remember you had issues, but I think that was on your personal machine? Not sure we ever figured out what was causing that, but in general having all three is not a problem, I don't think.

[JoranAngevaare]

I do remember you had issues, but I think that was on your personal machine? Not sure we ever figured out what was causing that, but in general having all three is not a problem, I don't think.

Yes, it was when you were on any location that was not dali, fried rice or LNGS (I think). We could not determine which was primary because none was. At the moment it's fine because all the databases are up (also on my machine).

[ershockley]

@jorana Sorry we're working on two repos at once which is making things unstable.

To satisfy travis we need to be able to import straxen and do basic tests without a valid config file. If we merge utilix #8 then the most recent changes should work, I think. This will have utilix throw an error, but straxen does try-except to circumvent it.

[JoranAngevaare]

If one ever runs into problems using three mirrors. Please replace:
pymongo_url = 'xenon-rundb.grid.uchicago.edu:27017,xenon1t-daq.lngs.infn.it:27017,fried.rice.edu:27017/xenonnt'
with
pymongo_url = 'xenon-rundb.grid.uchicago.edu:27017/xenonnt'
or
pymongo_url = 'fried.rice.edu:27017/xenonnt'

In the config file