Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expand tar archives in user sandbox #7661

Closed
belforte opened this issue Jun 12, 2023 · 5 comments
Closed

expand tar archives in user sandbox #7661

belforte opened this issue Jun 12, 2023 · 5 comments
Assignees

Comments

@belforte
Copy link
Member

if user sandbox contains a tar archive, expand it.
Solves dmwm/CRABClient#5198

@belforte belforte self-assigned this Jun 12, 2023
@belforte
Copy link
Member Author

do it now, so I close on requests for CRAB from Kevin.

@belforte
Copy link
Member Author

relevant code is

def extractUserSandbox(archiveJob, cmsswVersion):
# the user sandbox contains the user scram directory files and thus
# is unpacked in the local CMSSW_X_Y_X dir, but the cmsRun command
# will be executed from the job working directory, so we move "up"
# the PSet which is also in the user sandbox
os.chdir(cmsswVersion)
print(subprocess.getoutput('tar xfm %s ' % os.path.join('..', archiveJob)))
os.rename('PSet.py','../PSet.py')
os.rename('PSet.pkl','../PSet.pkl')
os.chdir('..')

Note that all sandbox files will be placed in $CMSSW_BASE directory in the worker node

@belforte belforte changed the title exand tar archives in user sandbox expand tar archives in user sandbox Jun 16, 2023
@belforte
Copy link
Member Author

as already noted in the CRABClient issue, the plan is:

look for files whose name contains .tar. or ends with .tar or .tgz or .tbz

belforte added a commit to belforte/CRABServer that referenced this issue Jun 19, 2023
belforte added a commit to belforte/CRABServer that referenced this issue Jun 27, 2023
@belforte
Copy link
Member Author

belforte commented Jun 30, 2023

I got it wrong.
User's jobType.inputFiles need to be expanded in /srv (the current working directory when the application is run)
see: #7705

@belforte belforte reopened this Jun 30, 2023
belforte added a commit to belforte/CRABServer that referenced this issue Jun 30, 2023
belforte added a commit to belforte/CRABServer that referenced this issue Jun 30, 2023
belforte added a commit to belforte/CRABServer that referenced this issue Jun 30, 2023
belforte added a commit that referenced this issue Jun 30, 2023
* expand user tar files in /srv . Correct fix for #7661

* final cleanup
@belforte
Copy link
Member Author

belforte commented Jul 1, 2023

fixed via #7708

@belforte belforte closed this as completed Jul 1, 2023
belforte added a commit that referenced this issue Jul 4, 2023
* expand tar files in user sandbox. Fix #7661

* remove unused prepSandbox

* many pylint fixes

* do not use encoding when opening binary files

* Revert "remove unused prepSandbox"

This reverts commit 856ac04.

* pylint

* add back lost blank

* stop using WMTaskSpace

* remove obsolete code

* simplify timestamp reporting
belforte added a commit to belforte/CRABServer that referenced this issue Jul 7, 2023
* expand user tar files in /srv . Correct fix for dmwm#7661

* final cleanup
belforte added a commit to belforte/CRABServer that referenced this issue Jul 7, 2023
* expand tar files in user sandbox. Fix dmwm#7661

* remove unused prepSandbox

* many pylint fixes

* do not use encoding when opening binary files

* Revert "remove unused prepSandbox"

This reverts commit 856ac04.

* pylint

* add back lost blank

* stop using WMTaskSpace

* remove obsolete code

* simplify timestamp reporting
novicecpp pushed a commit to novicecpp/CRABServer that referenced this issue Jul 26, 2023
novicecpp pushed a commit to novicecpp/CRABServer that referenced this issue Jul 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant