This is not particularly urgent at this point. But I'd rather do the cleanup sooner than later, to do it when we have things clear in memory and so that we also fix documentation and will never be confused again

Instead of using multiple tar archives, which would require some changes, we can use a single archive, but change the directory where files are put inside of it. This would still require changes in both the client and the jobwr, but hopefully they will be a bit less invasive.
For example, we could use tarfile.add(..., arcname="srv/" + filename ) for the files that should go in /srv and tarfile.add(..., arcname="cmssbase/" + filename ) for the files that need to go to srv/CMSSW_X_Y_Z. in CMSRunAnalysis.py:prepSandbox we can extract the archive and then move all the directories from the archive to the desired final path.

We also have a real problem (bug) : new code to expand user tar files #7661 only does that in /srv/CMSSW_X_Y_Z, while user may expect this to be happen in /srv

Maybe I am missing something, but why dont you execute the same extract command twice, both in prepSandbox and extractUserSandbox? are you waiting to for the cleanup and do not want to add any "dirty" fixes?

neat idea, thanks. Indeed that way we can also easily make a backward compatible TW and avoid disruption.

As to the second comment, indeed now I expand tar files only in /srv . But we need to change things a bit and only expand user sandbox after CMSSW_X_Y_Z directory has been created so we do all things in one place.

let's map into things to do:

• modify TW to do sandbox expansion in only one place in the code after CMSSW_X_Y_Z exists and remove prepSandbox
• expand in a temporary directory
• if files appear with a flat structure, put them all both in /srv and /srv/CMSSW_X_Y_Z (do not use explitic /srv string, of course, define a name for job's starting directory like e.g. topdir and use it consistently)
• if files from sandbox are organized as cmssw_base/... + topdir/.... put each set in the proper place
• expand tar files placed in topdir
• modify CRAB client to archive files in the two cmssw_baseand topdir directories
• once old client is gone (also from PRE), cleanup wrapper to remove backward compatibility

Note added on Sept 20: all actions listed above are not needed. Once we "tarfile.add(..., arcname="cmssbase/" + filename ) for the files that need to go to srv/CMSSW_X_Y_Z." as @mapellidario cleverly pointed out in #7705 (comment) , expansion inside CMSSW_X_Y_Z will be taken care of by the current iterative expansion of sandbox.tar.gz

of course I have now forgotten everything... and will need to re-learn :-)

after re-reading the issue and the code and a first attempt a documentation in #7754 (comment)
I am leaning toward

• fix the documentation and leave code as is

Well CRAB3.zip is not sent to the WN. No reason to reference it in the JobWrapper !

About the "sandbox expanded in two places".
I will start by removing

so that at least things will be clear.

Also it means that expansion is done after CMSSW_X_Y_Z has been created, opening the way for a "do it once".
BEst option looks to me at some point to modify sandboxing in client so that $CMSSW_BASE is tarred into a CMSSW_X_Y_Z directory and then all files will automatically go in the proper place when the sandbox is iteratively expanded