extend IamDataFrame to use extra data columns and sub-annual time

[danielhuppmann]

Please confirm that this PR has done the following:

• Tests Added
• Documentation Added
• Description in RELEASE_NOTES.md Added

Description of PR

This PR extends an IamDataFrame to have extra (custom) columns in the data data frame, and to distinguish between

• years (as integers, possibly with a column subannual (as string)
• continuous time using pd.to_datetime()
  The "type" of time representation is stored in self.time_col, the additional custom columns in self.data in self.extra_cols.

One quickfix required to get stuff working: read_files() is downgraded to take only one file, not a list of files. Because this would require checking that all imported data has the same time format and same extra columns (or add them on the fly).

ToDo:

• change "year" to self.time_col for basic functionality (e.g., in timeseries())
• change *_IDX usage in the package to replace "year" by self.time_col and use self.extra_cols
• figure out how to refactor append() (see issue above with read_files())
• reimplement multiple files in read_files()

Related issue and discussion

Closes #123

[danielhuppmann]

Help requested

@gidden and @znicholls, what would be an elegant way for running each test three times, once with the existing examples, once with a "subannual" version ("winter day" style), and once for a continuous-time example?

[danielhuppmann]

revised as suggested by @znicholls to cover initialising an IamDataFrame from wide format with columns as datetime, see danielhuppmann#8

[danielhuppmann]

Help requested

@znicholls, can you implement an appropriate equivalent of the pyam.utils.years_match() function for the new time column formatted as datetime? I guess that this will require some from-to filtering option (e.g., for years, I would use df.filter(year=range(2030, 2051))) to get a downselection of the time period.

[danielhuppmann]

rebased on https://github.com/IAMconsortium/pyam/releases/tag/v0.1.1

[znicholls]

@danielhuppmann I've got a bit confused here. Do you still need my help with the filter function? I'm guessing we need more tests first as they're currently all passing?

[danielhuppmann]

@znicholls, yes, I still need your help with a filter function for the time column. Thanks!

Re the tests, only a few tests currently check for multiple time formats (all those using test_df). So the implementation works insofar as the IamDataFrame now takes a year format (as int) or time (as datetime-castable), with some sanity checks that it can only be one of those. Basic functions like regions(), filter() and append() work (except no filtering for time).

Any other column (string that can't be cast to int) in the input data is now kept in data, with a list of these columns stored as df.extra_cols. These can also be used as args in filter() and can be added with timeseries(iamc_index=False).

The connection to the IIASA-db's are refactored so that they work with the new implementation, including the tutorial.

@znicholls and @gidden, I suggest that you review and merge unless you see major problems. The existing use cases work as expected for the year format (ie, CMIP6 and IPCC SR15), but ensuring that all extra features like check_aggregate() and the plotting library work with both formats will be too much for one PR (and too much for me). So I'd keep the time format as "beta" feature and refactor functions as needed going forward.

[danielhuppmann]

@znicholls, I removed the test for extra columns in meta, because this feature is not yet supported and we need to discuss this a bit further...

[znicholls]

Yep i get that. But if we mark it as xfail, the testsuite will still pass and we have an easy reminder of what we need to think about. If we just remove it, finding it again is trickier. The difference is marginal so no big deal either way.

…

On Thu, 29 Nov 2018 at 9:13 pm, Daniel Huppmann ***@***.***> wrote: @znicholls <https://github.com/znicholls>, I removed the test for extra columns in meta, because this feature is not yet supported and we need to discuss this a bit further... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#132 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWh-m0zvbH_fWvi-uJzGOFJ8WrmKB2g5ks5uz919gaJpZM4YLQmp> .

[danielhuppmann]

@znicholls , the datetime tests seem to have problems with python 2, can you take a look?

[danielhuppmann]

@znicholls, re having a failing test for meta and extra columns

Yep i get that. But if we mark it as xfail, the testsuite will still pass and we have an easy reminder of what we need to think about.

I'd be ok to leave it in if I was convinced that this is the ideal implementation for the long term. But I'd rather have some more discussions on that first...

[znicholls]

I'd be ok to leave it in if I was convinced that this is the ideal implementation for the long term. But I'd rather have some more discussions on that first...

cool, will take a look at Python2 stuff now

[danielhuppmann]

@gidden and @znicholls, this should be good to go now...

Follow-up issues that we should discuss after this PR is merged:

• how to treat meta by extra columns, requires further discussion
• refactor the tests, split out into multiple test files? test_core.py is pretty long now, some tests are executed for multiple year/time formats, others are only executed on the year format
• write a tutorial for the datetime feature, include a description in the docs

[znicholls]

Looks good to me.

Follow-up issues

Can we split these out into issues? You can assign me to the datetime feature tutorial.

[danielhuppmann]

@znicholls, yes, I meant that we would add these issues after the merge - just thought that it would be clearer what still needs to be done for the larger picture when I spell it out here. Thanks for volunteering!

[znicholls]

just thought that it would be clearer what still needs to be done for the larger picture when I spell it out here

aha very good, one step ahead of me!

[danielhuppmann]

another rebase after merge of #162 to try to appease stickler (which should have been appeased already)

[danielhuppmann]

closing in favour of #167