Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pivotal ID # 178930818: Extended File List Endpoint #418

Merged
merged 8 commits into from
Aug 16, 2021
Merged

Pivotal ID # 178930818: Extended File List Endpoint #418

merged 8 commits into from
Aug 16, 2021

Conversation

Juan-EBI
Copy link
Contributor

@Juan-EBI Juan-EBI commented Jul 28, 2021 •
edited by jhoanmanuelms
Loading

https://www.pivotaltracker.com/story/show/178930818

  • Create an endpoint to retrieve the files referenced by a file list in a submission
  • Add serialization for the file list including the URL to load the files
  • Modify the ext submission endpoint to receive the file lists as multipart attachments
  • Submit the file lists as attachments from the client instead of serializing the whole submission

Finishes # 178930818

@jhoanmanuelms
Pivotal ID # 178930818: Extended File List Endpoint
b7f73fc 
Create an endpoint to retrieve the files referenced by a file list in a submission
@jhoanmanuelms
Pivotal ID # 178930818: Extended File List Endpoint
81c4ef5 
- Add serialization for the file list including the url to load the files
- Include deserialization for the file list files from the files url
@jhoanmanuelms
Pivotal ID # 178930818: Extended File List Endpoint
8f4633c 
- Return ExtFilesTable for the file list endpoint and add the proper converter
- Read from the file list endpoint at deserialization time
- Add ExtSerialization config
@jhoanmanuelms
Pivotal ID # 178930818: Extended File List Endpoint
27e6e97 
- Add a web client operation to make direct extended submission
- Submit the serialized version instead of the object so the original files source is preserved
@jhoanmanuelms jhoanmanuelms changed the title Feature/pivotal 178930818 file list endpoint Pivotal ID # 178930818: Extended File List Endpoint Aug 3, 2021
@jhoanmanuelms
Pivotal ID # 178930818: Extended File List Endpoint
b06c224 
- Modify the ext submission endpoint to receive the file lists as multipart attachments
- Move file list deserialization to the submission service
- Submit the file lists as attachments from the client instead of serializing the whole submission
@jhoanmanuelms jhoanmanuelms self-assigned this Aug 9, 2021
@jhoanmanuelms jhoanmanuelms marked this pull request as ready for review August 11, 2021 22:10
@jhoanmanuelms jhoanmanuelms self-requested a review August 11, 2021 22:11
Juan-EBI
Juan-EBI commented Aug 16, 2021
View reviewed changes
Copy link
Contributor Author

@Juan-EBI Juan-EBI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM please consider two minors comments before merge,

...nt/bio-commandline/src/main/kotlin/uk/ac/ebi/biostd/client/cli/services/SubmissionService.kt Outdated
val extSerializer = extSerializationService()
val source = sourceClient.getExtByAccNo(request.accNo)
val migrated = migratedSubmissions(source, request.targetOwner)
val fileLists = source.allFileList.map { asTempFile(it, sourceClient, request.tempFolder, extSerializer) }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe

        val files = source.allFileList
            .map { File(request.tempFolder, it.fileName) to sourceClient.getReferencedFiles(it.filesUrl!!) }
            .onEach { (file, files) -> file.writeText(extSerializer.serialize(files)) }
            .map { it.first }

as passing services as parameter looks exrange for me, also we may consider transform extSerializationService into a class field.

client/bio-webclient/src/main/kotlin/ac/uk/ebi/biostd/client/api/PermissionOperationsClient.kt
@@ -5,8 +5,8 @@ import org.springframework.web.client.RestTemplate

class PermissionOperationsClient(private val restTemplate: RestTemplate) :
PermissionOperations {
override fun givePermissionToUser(userEmail: String, accessTagName: String, accessType: String) {
val body = hashMapOf("userEmail" to userEmail, "accessType" to accessType, "accessTagName" to accessTagName)
override fun givePermissionToUser(user: String, accessTagName: String, accessType: String) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I think userEmail is more clear as here is ambiguous what can be , one may think is the id or the login

jhoanmanuelms
jhoanmanuelms approved these changes Aug 16, 2021
View reviewed changes
Copy link
Contributor

@jhoanmanuelms jhoanmanuelms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jhoanmanuelms jhoanmanuelms merged commit bccbb8e into master Aug 16, 2021
@jhoanmanuelms jhoanmanuelms deleted the feature/pivotal-178930818-file-list-endpoint branch August 16, 2021 14:52
@jhoanmanuelms jhoanmanuelms mentioned this pull request Nov 25, 2021
Merged
Juan-EBI pushed a commit that referenced this pull request Nov 28, 2021
@jhoanmanuelms
Promote To BETA (#474)
b562e5f 
Feature
Pivotal ID # 177839764: FIRE FileProcessingService (#415)
Pivotal ID # 178930818: Extended File List Endpoint (#418)
Pivotal ID # 179363552: Add fire files interface source support (#436)
Pivotal ID # 178461698: FIRE Database Persistence (#437)
Pivotal ID # 179389139: FIRE Directories Persistence (#439)
Pivotal ID # 179361117: Add support for groups administration (#434)
Pivotal ID # 179609612: Submit Large Files To Fire (#444)
Pivotal ID # 179418482: Persist PageTab FIRE (#440)
Pivotal ID # 179513930: Add Warning For Empty File Lists (#448)
Pivotal ID # 179040289: Async Migration (#451)
Pivotal ID # 179810581: Files files storage (#456)
Pivotal ID # 180013760: Add submissions schema version field (#463)
Pivotal ID # 179922324: File Mode CLI Option (#465)
Pivotal ID # 180152694: File List Validation (#473)
Pivotal ID # 179921481: Fire ftp publishing (#467)
Pivotal ID # 180253535: Submission storage mode (#468)
Pivotal ID # 180282390: Reset password should activate user (#470)

Bugfix
Pivotal ID # 179145538: Handle corrupted files in PMC (#429)
Pivotal ID # 178331250: Files Names Encoding (#431)
Pivotal ID # 179435581: Deleting Submissions Isn't Working (#442)
Pivotal ID # 179904973: Fix Notification URL (#455)
Pivotal ID # 179922291: Inner File List Referenced Files (#457)
Pivotal ID # 179867189: TSV Quotes Deserialization (#462)
Pivotal ID # 180345449: handle empty title submission email notification (#472)

Chore
Pivotal ID # 179055347: Update AE user activation email (#420)
Pivotal ID # 179105055: Update Java Home (#421)
Pivotal ID # 179144369: Add tsv library (#426)
Pivotal ID # 179144780: Add json library (#427)
Pivotal ID # 179834295: Excel dsl library
Pivotal ID # 179356847: Use contanst as projects dependencies (#433)
Pivotal ID # 179355330: Simplify submission files processing (#432)
Pivotal ID # 179418565: Remove Option-Optional from backend (#438)
Pivotal ID # 179728378: On behalf register user improvements (#446)
Pivotal ID # 179436227: Improve Logs Messages (#447)
Pivotal ID # 177839916: Add firemock container set-up (#443)
Pivotal ID # 179784398: Improve all in one itest to include file list (#450)
Pivotal ID # 179040289: BioStudies CLI Deployment (#452)
Pivotal ID # 180124164: Re use ext mapping for file list (#464)
Pivotal ID # 179939266: SpringBoot Admin App Registration (#461)
jhoanmanuelms added a commit that referenced this pull request Mar 1, 2022
@jhoanmanuelms
Promote To BETA (#518)
81895f9 
Feature
Pivotal ID # 177042710: Support For Deleting Multiple Submissions (#394)
Pivotal ID # 177839764: FIRE FileProcessingService (#401)
Pivotal ID # 179055347: Update AE user activation email (#420)
Pivotal ID # 178930818: Extended File List Endpoint (#418)
Pivotal ID # 179363552: Add fire files interface source support (#436)
Pivotal ID # 178461698: FIRE Database Persistence (#437)
Pivotal ID # 179389139: FIRE Directories Persistence (#439)
Pivotal ID # 179361117: Add support for groups administration (#434)
Pivotal ID # 179609612: Submit Large Files To Fire (#444)
Pivotal ID # 179418482: Persist PageTab FIRE (#440)
Pivotal ID # 179513930: Add Warning For Empty File Lists (#448)
Pivotal ID # 179040289: Async Migration (#451)
Pivotal ID # 179922291: Inner File List Referenced Files (#457)
Pivotal ID # 179810581: Files files storage (#456)
Pivotal ID # 179922324: File Mode CLI Option (#465)
Pivotal ID # 180152694: File List Validation (#466)
Pivotal ID # 179921481: Fire ftp publishing (#467)
Pivotal ID # 180253535: Submission storage mode (#468)
Pivotal ID # 180282390: Reset password should activate user (#470)
Pivotal Id # 180656091: Submission request should store all request information (#482)
Pivotal ID # 180719965: Request should not save files list content (#484)
Pivotal ID # 180850631: Stream serialization/deserilization support (#486)
Pivotal ID # 180929480: Allow submission request to be manually re trigger (#493)
Pivotal ID # 180481465: No attribute/empty attribute value handling (#490)
Pivotal ID # 180962335: Only pending to process request need to be proccesed
Pivotal ID # 180853446: Change RT ticket status upon submission (#496)
Pivotal ID # 181061881: File list files should not be stored in submission document (#500)
Pivotal ID # 181147744: Mongo migration with referenced submission in fileListFiles (#508)
Pivotal ID # 181155613: Add push to partial updates exchange on study refresh (#506)
Pivotal ID # 181179211 : Unactive draft when request is accepted (#507)
Pivotal ID # 181154115: Extended submission, allow file list files loading option (#509)
Pivotal ID # 180928073: Xml deserialization/serialization null support (#492)
Pivotal ID # 180481465: Empty Attribute Value Handling (#513)
Pivotal ID # 180640412: PMC Links Exporter (#483)
Pivotal ID # 181330654: File List Empty Path Entries (#514)
Pivotal ID # 181381956: Filter Active Requests For Existing Submissions (#516)

Bugfix
Pivotal ID # 178411427: Keyword search need to check root section title (#395)
Pivotal ID # 178331250: Files Names Encoding (#431)
Pivotal ID # 179435581: Deleting Submissions Isn't Working (#442)
Pivotal ID # 179904973: Fix Notification URL (#455)
Pivotal ID # 179867189: TSV Quotes Deserialization (#462)
Pivotal ID # 180345449: handle empty title submission email notification (#472)
Pivotal ID # 180446298: File list files are not being populated (#477)
Pivotal ID # 180476322: File List URL Enconding (#478)
Pivotal ID # 180957544: Re processing submissions is failing (#494)
Pivotal ID # 181142221: Emails formating issues (#503)

Chore
Pivotal ID # 177839881: FIRE File Object Changes (#398)
Pivotal ID # 178688699: Improve mongo index text (#403)
Pivotal ID # 178213145: Activate ArrayExpress inactive users (#406)
Pivotal ID # 178375625: Actuator Git Commit (#408)
Pivotal ID # 179105055: Update Java Home (#421)
Pivotal ID # 179144369: Add tsv library (#426)
Pivotal ID # 178974842: Section Title Index (#416)
Pivotal ID # 179144780: Add json library (#427)
Pivotal ID # 179356847: Use contanst as projects dependencies (#433)
Pivotal ID # 179355330: Simplify submission files processing (#432)
Pivotal ID # 179418565: Remove Option-Optional from backend (#438)
Pivotal ID # 179728378: On behalf register user improvements (#446)
Pivotal ID # 179436227: Improve Logs Messages (#447)
Pivotal ID # 177839916 Add firemock container set-up (#443)
Pivotal ID # 179784398: Improve all in one itest to include file list (#450)
Pivotal ID # 179040289: BioStudies CLI Deployment (#452)
Pivotal ID # 179939266: SpringBoot Admin App Registration (#461)
Pivotal ID # 179040289: Mongo Migration PROD (#479)
Pivotal ID # 180762420: Dependencies upgrade (#485)
Pivotal ID # 180891572: Immutable application properties (#487)
Pivotal ID # 180924563: Remove sql submission support (#488)
Pivotal ID # 180928080: Simplify refresh submission api (#491)
Pivotal ID # 181032586:  Remove submission-related changes from mysql set-up
Pivotal ID # 181024221: Md5 should be calculated after accepting submission
Pivotal ID # 181031416: Files size should be calculated after accepting submission
Pivotal ID # 181127806: Enable rabbitmq_web_stomp in setup script
Pivotal ID # 181196844: PMC task is failing to start (#510)
Pivotal ID # 181076984: Submission Releaser Refactor (#512)
Pivotal ID # 181206866: Remove application property enableMongo
Pin RabbitMQ minor version number (#419)
@jhoanmanuelms jhoanmanuelms mentioned this pull request Mar 7, 2022
Merged
jhoanmanuelms added a commit that referenced this pull request Mar 7, 2022
@jhoanmanuelms
Promote To PROD (#521)
4f86512 
BioStudies Backend Services v1.8

Feature
Pivotal ID # 177042710: Support For Deleting Multiple Submissions (#394)
Pivotal ID # 177839764: FIRE FileProcessingService (#401)
Pivotal ID # 179055347: Update AE user activation email (#420)
Pivotal ID # 178930818: Extended File List Endpoint (#418)
Pivotal ID # 179363552: Add fire files interface source support (#436)
Pivotal ID # 178461698: FIRE Database Persistence (#437)
Pivotal ID # 179389139: FIRE Directories Persistence (#439)
Pivotal ID # 179361117: Add support for groups administration (#434)
Pivotal ID # 179609612: Submit Large Files To Fire (#444)
Pivotal ID # 179418482: Persist PageTab FIRE (#440)
Pivotal ID # 179513930: Add Warning For Empty File Lists (#448)
Pivotal ID # 179040289: Async Migration (#451)
Pivotal ID # 179922291: Inner File List Referenced Files (#457)
Pivotal ID # 179810581: Files files storage (#456)
Pivotal ID # 179922324: File Mode CLI Option (#465)
Pivotal ID # 180152694: File List Validation (#466)
Pivotal ID # 179921481: Fire ftp publishing (#467)
Pivotal ID # 180253535: Submission storage mode (#468)
Pivotal ID # 180282390: Reset password should activate user (#470)
Pivotal Id # 180656091: Submission request should store all request information (#482)
Pivotal ID # 180719965: Request should not save files list content (#484)
Pivotal ID # 180850631: Stream serialization/deserilization support (#486)
Pivotal ID # 180929480: Allow submission request to be manually re trigger (#493)
Pivotal ID # 180481465: No attribute/empty attribute value handling (#490)
Pivotal ID # 180962335: Only pending to process request need to be proccesed
Pivotal ID # 180853446: Change RT ticket status upon submission (#496)
Pivotal ID # 181061881: File list files should not be stored in submission document (#500)
Pivotal ID # 181147744: Mongo migration with referenced submission in fileListFiles (#508)
Pivotal ID # 181155613: Add push to partial updates exchange on study refresh (#506)
Pivotal ID # 181179211 : Unactive draft when request is accepted (#507)
Pivotal ID # 181154115: Extended submission, allow file list files loading option (#509)
Pivotal ID # 180928073: Xml deserialization/serialization null support (#492)
Pivotal ID # 180481465: Empty Attribute Value Handling (#513)
Pivotal ID # 180640412: PMC Links Exporter (#483)
Pivotal ID # 181330654: File List Empty Path Entries (#514)
Pivotal ID # 181381956: Filter Active Requests For Existing Submissions (#516)

Bugfix
Pivotal ID # 178411427: Keyword searchs need to check root section title (#395)
Pivotal ID # 178331250: Files Names Encoding (#431)
Pivotal ID # 179435581: Deleting Submissions Isn't Working (#442)
Pivotal ID # 179904973: Fix Notification URL (#455)
Pivotal ID # 179867189: TSV Quotes Deserialization (#462)
Pivotal ID # 180345449: handle empty title submission email notification (#472)
Pivotal ID # 180446298: File list files are not being populated (#477)
Pivotal ID # 180476322: File List URL Enconding (#478)
Pivotal ID # 180957544: Re processing submissions is failing (#494)
Pivotal ID # 181142221: Emails formating issues (#503)

Chore
Pivotal ID # 177839881: FIRE File Object Changes (#398)
Pivotal ID # 178688699: Improve mongo index text (#403)
Pivotal ID # 178213145: Activate ArrayExpress inactive users (#406)
Pivotal ID # 178375625: Actuator Git Commit (#408)
Pivotal ID # 179105055: Update Java Home (#421)
Pivotal ID # 179144369: Add tsv library (#426)
Pivotal ID # 178974842: Section Title Index (#416)
Pivotal ID # 179144780: Add json library (#427)
Pivotal ID # 179356847: Use contanst as projects dependencies (#433)
Pivotal ID # 179355330: Simplify submission files processing (#432)
Pivotal ID # 179418565: Remove Option-Optional from backend (#438)
Pivotal ID # 179728378: On behalf register user improvements (#446)
Pivotal ID # 179436227: Improve Logs Messages (#447)
Pivotal ID # 177839916 Add firemock container set-up (#443)
Pivotal ID # 179784398: Improve all in one itest to include file list (#450)
Pivotal ID # 179040289: BioStudies CLI Deployment (#452)
Pivotal ID # 179939266: SpringBoot Admin App Registration (#461)
Pivotal ID # 179040289: Mongo Migration PROD (#479)
Pivotal ID # 180762420: Dependencies upgrade (#485)
Pivotal ID # 180891572: Immutable application properties (#487)
Pivotal ID # 180924563: Remove sql submission support (#488)
Pivotal ID # 180928080: Simplify refresh submission api (#491)
Pivotal ID # 181032586:  Remove submission-related changes from mysql set-up
Pivotal ID # 181024221: Md5 should be calculated after accepting submission
Pivotal ID # 181031416: Files size should be calculated after accepting submission
Pivotal ID # 181127806: Enable rabbitmq_web_stomp in setup script
Pivotal ID # 181196844: PMC task is failing to start (#510)
Pivotal ID # 181076984: Submission Releaser Refactor (#512)
Pivotal ID # 181206866: Remove application property enableMongo
Pin RabbitMQ minor version number (#419)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhoanmanuelms jhoanmanuelms jhoanmanuelms approved these changes

Assignees

@jhoanmanuelms jhoanmanuelms

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Juan-EBI @jhoanmanuelms