Remove multi table feature

[fjetter]

Description:

As the title state, this will remove any API references to the dataset internal multi table functionality. It is effectively no longer possible to write these.

It goes without saying. This change is in terms of API as breaking as it gets. This is a new major release and I intend to drop many of the deprecated features after merging this as well (effectively everything which raises a warning at the moment)

• Closes Removal of complex input types #427
• Changelog entry

[mlondschien]

Not sure how close this is to merging, but I thought I would let you know what this breaks for us:

• read_table no longer accepts table as a keyword argument.
• DatasetMetadata.table_meta["table"] and DatasetFactory.table_meta["table"] raise a TypeError: an integer is required. IIUC this is superseeded by .schema.
• read_dataset_as_dataframes__iterator no longer takes a columns = {"table": list-like} argument and no longer returns {"table": df} data frames.

Are you planning on making a deprecating release in between 3.19 and 4.0? The table kwarg is no problem since "table" is already a default value. However it would be nice to have schema available and be able to pass columns=list-like to get a data frame before 4.0 or be able to use the current API until 4.1.

[fjetter]

If the removal of the table is actually a problem we can keep it without effect but I would prefer the clean removal since we've been dragging unused arguments around for years.

All the changes you mentioned were anticipated. We can make the table_meta change backwards compatible. I actually intended to remove it altogether but I don't have a strong opinion on this.

Regarding the columns input, this is actually not that easy to achieve and I'm not sure if it is worth the trouble. The problem here is (and also the reason why this must go) that all pipelines are implementing this somewhat differently

bottom line is that all of these changes feel like breaking but easily dealt with and I'm hesitant to introduce a lot of glue code to ease this migration. I believe these problems are quickly addressed by going over the code base once everywhere.

DatasetMetadata.table_meta["table"] and DatasetFactory.table_meta["table"] raise a TypeError: an integer is required.

This, I will fix. Cryptic error message are always a problem, even for intended changes.

[mlondschien]

Would it be an option to add

@property
def schema(self):
    return self.table_meta["table"]

to DatasetMetadata and DatasetFactory before 4.0?

I guess we will be able to live with the change in read_dataset_as_dataframes__iterator API :).

[fjetter]

yes, we can add the schema pre 4.X. We have one or two unreleased bugfixes on master which probable makes sense to release anyhow before that

[fjetter]

Lazy consensus merge by Monday 15-03-2021 9:00 UTC

[bjoern-meier-by]

Maybe I do miss it , but as I currently understand if self.primary_indices_loaded is True we return the object itself. If not we load the indices and return a copy. So self.primary_indices_loaded is not updated from what I see, so I wonder why not?
Why even the copy?

[fjetter]

The copy methods were introduced at the very beginning of kartothek and we perceived it as a "clean coding" by not mutating any objects inplace but always return a new instance. This is what the copy methods are doing. By now I consider this unnecessary boilerplate but I didn't want to break any patterns here. Therefore if nothing is to be done, the same instance is returned, otherwise a new instance with the mutated state is returned

FYI this method is unchanged but was moved for some reason. I'll move it back to have cleaner diffs / blames. still valid question, of course :)

[bjoern-meier-by]

Ah okay got it. I understand the arguments of preventing mutation but also that not everything can and should be folloewed 100% and here I might argue against it but as it is pre-change code I am totally fine :)

[bjoern-meier-by]

Thanks @fjetter for the cleanup and refactoring. 🙇

[fjetter]

@bjoern-meier-by thanks for reviewing!