CouchApp third-party library does not support python3

[amaltaro]

Impact of the new feature
WMCore software dependent on CouchDB

Is your feature request related to a problem? Please describe.
Yes, we need to use libraries that are compatible with python3, which is not the case of the CouchApp library used to deploy apps into CouchDB.

Describe the solution you'd like
Generically speaking, we should find a way around this python3 incompatibility and ensure that our apps can be properly deployed in CouchDB (and any other functionality pushed to CouchDB).

There are many ways to deal with this:

• replace the Couchapp dependency by something else that supports python3
• investigate what exactly is used from this library, and perhaps implement it ourselves
• fork Couchapp repository and try to convert it to python3
• any other ideas?

I don't yet know how to best address this issue, but here is an issue to keep track of this activity, and if needed, to keep record of how exactly Couchapp is used.

NOTE: we should likely update our package version to 1.0.2 as well and build it from Pypi

Describe alternatives you've considered
A few alternatives mentioned above.

Additional context
Issue spotted and mentioned here: #10302 (comment)

GH repository: https://github.com/couchapp/couchapp

[amaltaro]

I spent around 30min looking for possible alternatives, and I fear there is none available. So let me expand on how exactly this CouchApp tool is currently used to push applications to CouchDB.

First, we create files under the stagingarea, which get executed every time that we start CouchDB. An example of such file is:

amaltaro@alancc7-cloud1:/data/srv/state/couchdb/stagingarea $ cat reqmgr2 
couchapp push /data/srv/current/apps/reqmgr2/data/couchapps/ReqMgrAux http://localhost:PORT/reqmgr_auxiliary
couchapp push /data/srv/current/apps/reqmgr2/data/couchapps/ReqMgr http://localhost:PORT/reqmgr_workload_cache
couchapp push /data/srv/current/apps/reqmgr2/data/couchapps/ConfigCache http://localhost:PORT/reqmgr_config_cache

where the first argument is a directory tree (our couchdb application) in the format of:

amaltaro@alancc7-cloud1:/data/srv/state/couchdb/stagingarea $ tree /data/srv/current/apps/reqmgr2/data/couchapps/ReqMgr
/data/srv/current/apps/reqmgr2/data/couchapps/ReqMgr
├── couchapp.json
├── _id
├── language
├── updates
│   ├── totalstats.js
│   └── updaterequest.js
├── validate_doc_update.js
└── views
    ├── bycampaign
    │   └── map.js
    ├── bydatapileup
    │   └── map.js

and the second argument is the database name that we want to hold/serve this application.

I haven't look into further details, but we might be able to write a simple utilitarian script to accomplish the same result, thus being able to completely deprecate this CouchApp and the restkit dependency (see #10076).

Looking at the couchapp helper, once we source the reqmgr2 init.sh script:

source /data/srv/current/apps/reqmgr2/etc/profile.d/init.sh

we can export/dump the couchdb application content with something like:

couchapp push --export --output /data/srv/alan.json /data/srv/current/apps/reqmgr2/data/couchapps/ReqMgrAux http://localhost:PORT/reqmgr_auxiliary

it should make things easier to be compared as well.

Last but not least, this script:
https://github.com/dmwm/WMCore/blob/master/bin/wmagent-couchapp-init

will have to be updated accordingly. I believe it's partially used by only global workqueue and wmagent.

[amaltaro]

Just an status update here. I failed to find an alternative to the Couchapp third-party library that would be easy enough to integrate with our tools, so I decided to fork the actual couchapp/couchapp repository and started simplifying it as much as I could for our only use case, which is:

couchapp push APPLICATION COUCH_DB_URI

I'm first going to run a "binary" comparison between the current python2 couchapp and the minimalist version I created. If everything goes fine, I will then make it to compatible with python3 and we can actually start using it in both python stacks.

[vkuznet]

I'm not sure why do you bother with couchapp since CouchDB has HTTP based interface. As such what you need is to write custom code to send your JS files via HTTP POST request. This is what couchapp is doing for you.

[amaltaro]

Exactly, if couchapp is doing that for me, why would I bother in writing my own stuff(?) Actually, it's not a plain HTTP request, one has to also "construct" the application to be uploaded to CouchDB (basically constructing the design document, walking all the javascript/erlang/html/css code available and coming up with a final product).

That's the reason I decided to take what is already done, modify it (read, make it as small as possible) and keep it under our watch.

[amaltaro]

This PR: cms-sw/cmsdist#6913

provides a CMSCouchapp compatible with python3. It's been tested - in the sense of pushing couchapp documents - and things seem to be working properly. It still needs to be fully tested though, but for that we need to get rid of all the rest of the python2 libraries in our python3 services.

I might open another GH issue for the couchdb15.spec file, which depends on either the python2 or python3 CMSCouchapp package. However, since it's going to be used by different services in different python version, we need to actually decouple it...

[amaltaro]

And this PR provides a functional python3 CMSCouchapp:
cms-sw/cmsdist#6913

it still needs to be properly tested though. For that, we need to complete the python3 stack though.

A new GH issue has been created to properly deal with PY2 vs PY3 CMSCouchapp with CouchDB: #10489

I think it's reasonable to call this issue as resolved. Closing it, if anyone sees a reason to keep it opened for a little longer, please reopen it.