Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initialize an IamDataFrame from pd.DataFrame with formatting specs #199

Merged
merged 31 commits into from
Feb 22, 2019
Merged

Initialize an IamDataFrame from pd.DataFrame with formatting specs #199

merged 31 commits into from
Feb 22, 2019

Conversation

danielhuppmann
Copy link
Member

@danielhuppmann danielhuppmann commented Feb 20, 2019 •
edited
Loading

Please confirm that this PR has done the following:

  • Tests Added
  • Documentation Added
  • Description in RELEASE_NOTES.md Added

Description of PR

This PR allows initialising an IamDataFrame from a file or pd.DataFrame and filling out missing required columns efficiently.

See the unit tests in tests/test_cast_to_iamc.py for possible use cases.

gidden and others added 29 commits February 18, 2019 08:21
@gidden
initial impl for giving default pyam dataframes
00866bf
@gidden
move again into core
057fdb4
@gidden
small fixes
cd62127
@gidden
update concat to try to cast to dataframe
95a5208
@gidden
appease stickler
391ea23
@danielhuppmann
fix TypeError in concat()
f6aa5eb
@danielhuppmann
add concat_with_pipe() in utils with some tests
89a08f9
@danielhuppmann
cast elements in concat_with_pipe() to str
9e3f4e9
@danielhuppmann
drop nan instead of casting to str
33e90d4
@danielhuppmann
insert function reduce_hierarchy() and add tests
6c81734
@danielhuppmann
raise error if initializing IamDataFramewith duplicate rows in data
ce5a7a5
@danielhuppmann
remove unused legacy function
60679bf
@danielhuppmann
add auxiliary function sort_data() for consistent ordering of cols
09c574f
@danielhuppmann
use sort_data() in append()
ead9105
@danielhuppmann
add sort_data() to some unit tests to make them pass
2a39017
@danielhuppmann
refactor and add first test for df_to_pyam()
95b550a
@danielhuppmann
clean-up before stickler gives me an earful...
d7b252f
@danielhuppmann
clean-up of implementation of df_to_pyam()
d8d91a6
@danielhuppmann
update docstrings
fd9f285
@danielhuppmann
allow input-df with columns year and time
d31595d
@danielhuppmann
defining defaults doesn't seem to be necessary
be03a9d
@danielhuppmann
require value arg in df_to_pyam() instead of using all columns
9e0c738
@danielhuppmann
check for conflicts with existing columns, add test
6e183f8
@danielhuppmann
add option in df_to_pyam() to rename cols
be71d51
@danielhuppmann
add feature to concat required column from multiple given columns
1e391f2
@danielhuppmann
make cast_to_iam partof format_data()
779b3e1
@danielhuppmann
rename function to read_file() because it can only read one file
5285fe8
@danielhuppmann
pass kwargs for format_data through read_file()
8330002
@danielhuppmann
pep8 and docstring cleanup
7744783
@danielhuppmann danielhuppmann mentioned this pull request Feb 20, 2019
Closed
2 tasks
@danielhuppmann
Copy link
Member Author

@znicholls, this PR tries to make it easier to initialize an IamDataFrame which doesn't quite have the right format, for example when you have a pd.DataFrame in a scen-file format with values in multiple columns and the column name being the identifier. See also the discussion here #193 (comment).

Can you check whether that works with your thinking for the ScmDataFrame?

@danielhuppmann danielhuppmann mentioned this pull request Feb 20, 2019
Closed
3 tasks
@coveralls
Copy link

coveralls commented Feb 20, 2019 •
edited
Loading

Coverage Status

Coverage increased (+0.3%) to 84.508% when pulling d20bc4a on danielhuppmann:pd-to-pyam-concat into a6ac0c5 on IAMconsortium:master.

znicholls
znicholls approved these changes Feb 20, 2019
View reviewed changes
Copy link
Collaborator

@znicholls znicholls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice changes @danielhuppmann ! That extra flexibility is very cool.

The IamDataFrame format is suitably sane that whatever you do it's going to be easy for us to cast from ScmDataFrame to an IamDataFrame. I think no matter what, the awkward 'model' name clash is here to stay so we just put that in the 'not perfect but ok' basket (unless model became 'iam' in an IamDataFrame but that sort of backwards compatibility breaking change seems unnecessary right now).

@znicholls
Copy link
Collaborator

will leave final review and merge to @gidden nice tweak @danielhuppmann

gidden
gidden reviewed Feb 21, 2019
View reviewed changes
pyam/core.py Outdated Show resolved Hide resolved
gidden
gidden reviewed Feb 21, 2019
View reviewed changes
pyam/utils.py Show resolved Hide resolved
gidden
gidden reviewed Feb 21, 2019
View reviewed changes
pyam/utils.py Outdated Show resolved Hide resolved
@gidden
Copy link
Member

gidden commented Feb 21, 2019

ok, small changes requested. looks great!

@gidden gidden merged commit fcb81fe into IAMconsortium:master Feb 22, 2019
@gidden
Copy link
Member

gidden commented Feb 22, 2019

thanks @danielhuppmann !

@danielhuppmann danielhuppmann deleted the pd-to-pyam-concat branch February 26, 2019 14:00
@gidden gidden mentioned this pull request Mar 8, 2019
Closed
@danielhuppmann danielhuppmann mentioned this pull request Mar 11, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gidden gidden gidden left review comments

@znicholls znicholls znicholls approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@danielhuppmann @coveralls @znicholls @gidden