Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3767: CSV ingest improvements #3963

Merged
merged 35 commits into from
Aug 2, 2017
Merged

3767: CSV ingest improvements #3963

merged 35 commits into from
Aug 2, 2017

Conversation

oscardssmith
Copy link
Contributor

@oscardssmith oscardssmith commented Jun 26, 2017 •
edited
Loading

This PR changes our ingest algorithm to use a csv parsing library that will let us accept more types of CSV. A similar change could be made to our tab ingest, but that can wait.
Related Issues. The noise with the several versions of this occured because I don't know how to git

Pull Request Checklist

  • Unit [tests][] completed
  • Integration [tests][]: None
  • Deployment requirements, [SQL updates][], [Solr updates][], etc.: None
  • [Documentation][docs] completed
  • Merged latest from "develop" [branch][] and resolved conflicts

@coveralls
Copy link

Coverage Status

Coverage increased (+0.5%) to 10.808% when pulling f6defb5 on 3767-CSV-injest-code into 4080c04 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.5%) to 10.806% when pulling 3e68a9c on 3767-CSV-injest-code into 4080c04 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 10.88% when pulling 606458f on 3767-CSV-injest-code into 4080c04 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 10.88% when pulling 158091d on 3767-CSV-injest-code into 4080c04 on develop.

@oscardssmith
Fixed testSubset to reflect the new behavior of counting large integr…
e96ba16 
…al values as discrete rather than continuous
@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 10.996% when pulling e96ba16 on 3767-CSV-injest-code into 4080c04 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 10.996% when pulling 406906d on 3767-CSV-injest-code into 4080c04 on develop.

@landreev
Revived my test API for sending files through the ingest process with…
8651826 
…out writing anything in the database.

During the 4.0 development I was using this API to test ingest plugins.
I want to use it agin, for batch-testing Oscar's new CSV ingest reader on the CSV files already ingested in production.
@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 10.997% when pulling 8651826 on 3767-CSV-injest-code into 4080c04 on develop.

@dlmurphy
CSV Ingest Doc Update [#3767]
ae00815 
Updated this doc, thanks to @oscardssmith's excellent advice!
@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 10.995% when pulling ae00815 on 3767-CSV-injest-code into 4080c04 on develop.

@dlmurphy
Small url fix
fd5bbf5 
Unrelated to rest of branch, a small broken link fix @jggautier found.
@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 10.997% when pulling fd5bbf5 on 3767-CSV-injest-code into 4080c04 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 10.997% when pulling bbfa4b9 on 3767-CSV-injest-code into 4080c04 on develop.

@oscardssmith
Removed tons of logging that didn't really help much.
784edb7 
Moved main class and test to using Bundle for easier translation.
@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 10.979% when pulling 61c74c7 on 3767-CSV-injest-code into 482c2dc on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.18% when pulling 68f3826 on 3767-CSV-injest-code into 22090e2 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.184% when pulling e9f4da4 on 3767-CSV-injest-code into 22090e2 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 11.238% when pulling 6601ea2 on 3767-CSV-injest-code into 22090e2 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.7%) to 11.238% when pulling 5a3c86f on 3767-CSV-injest-code into 22090e2 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.238% when pulling 0cdc2c0 on 3767-CSV-injest-code into f384229 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.376% when pulling 867485f on 3767-CSV-injest-code into eec0037 on develop.

@dlmurphy
Typo fix [ref: #3767]
fbd5789 
Fixed a couple typos I missed earlier.
@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.382% when pulling fbd5789 on 3767-CSV-injest-code into eec0037 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.382% when pulling 310b86b on 3767-CSV-injest-code into eec0037 on develop.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.382% when pulling 145608c on 3767-CSV-injest-code into eec0037 on develop.

@dlmurphy
Docs reviewed [ref: #3767]
4e45054 
Edited for typos and clarity
@coveralls
Copy link

Coverage Status

Coverage increased (+0.6%) to 11.382% when pulling 4e45054 on 3767-CSV-injest-code into eec0037 on develop.

@kcondon kcondon merged commit 5622911 into develop Aug 2, 2017
@kcondon kcondon deleted the 3767-CSV-injest-code branch August 2, 2017 17:52
@pdurbin pdurbin mentioned this pull request Oct 31, 2017
Closed
@pdurbin pdurbin mentioned this pull request Apr 23, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
4.8 - AWS S3 Support and Large Data U...
Development

Successfully merging this pull request may close these issues.
6 participants
@oscardssmith @coveralls @pdurbin @kcondon @landreev @dlmurphy