Drop table semantics

[gruuya]

Description

Use Case

Something high-ish level that allows Rust lib users to erase the root table directory completely, perhaps a DropTableBuilder/action/DeltaOperation/drop method on DeltaTable. So effectively an interface that can be used to execute DROP TABLE my_table SQL.

Alternatively, do a lazy drop by delaying physically deleting anything and instead flag the table somehow, so that CreateTableBuilder doesn't throw CreateError::TableAlreadyExists even in SaveMode::ErrorIfExists mode.

Related Issue(s)

Perhaps worthy of including in #1128

[gruuya]

Just to give a bit more context: while doing list on a given prefix and then delete for each path returned works, I suspect it will be sub-optimal for tables with a lot of files.

[roeap]

As you mentioned, in delta we can delete tables by simply removing all files from the log, but this will not allow for the create scenario you mentioned. I guess listing and deleting the files is the way to go here... this can be made quite efficient once we get to implementing the batch delete apis offered by the major object stores.

Having a special flag to ignore existing data could work, but there i would be quite hesitant, as it would be special to delta-rs, and not honored by other writers etc..

[wjones127]

This would be good for cleaning up tables. But we should be clear that using DROP TABLE isn't recommended if the intention is to replace the table. Eventual consistency in S3 and other object stores will cause problems when trying to create and read the transaction log. We should support overwriting and schema evolution for those cases.

[gruuya]

this can be made quite efficient once we get to implementing the batch delete apis offered by the major object stores

Nice, looking forward to this!

Eventual consistency in S3 and other object stores will cause problems when trying to create and read the transaction log.

Makes sense, thanks.

For the record I'll circumvent this problem by changing the intended storage layout. Instead of the table uri being dependent on database/schema/table names, it'll be dependent solely on a uuid uniquely tied to that table. This solution may not be general enough for other people, because it relies on us keeping a separate metadata store (e.g. SQLite) to track all the table uuids (in our case, we need one anyway to track the databases/schemas/functions etc.).

However, this guarantees that e.g. renaming a table is just a minor op, or that dropping a schema/table can be done lazily (since subsequent re-use of the schema/table names will result in a different uuid, so there won't be conflicts). I also intend on GC-ing the dropped (stale) schema/table directories via some variant of the VACUUM command, ideally using the batch delete api.

Therefore this is no longer a (major) issue for me, AFAIC you can close it. I am still very much blocked by #1188 though.