Add iceberg table properties for maintenance of previous metadata files

[hantangwangd]

Description

This PR enables to set Iceberg table properties metadata_previous_versions_max and metadata_delete_after_commit to specify the number of previous metadata files that need to be tracked, and whether to delete the metadata files that no longer be tracked after each commit.

Besides, this PR also enable our own version of HiveTableOperations to support deleting old metadata files after commit based on the table properties. This feature has been supported by Iceberg's native version of HiveTableOperations.

Motivation and Context

The metadata files of an iceberg table will grow to a very large amount, as almost each operation will lead in a new one.
So we need to provide a method to delete unnecessary metadata files.

Impact

When executing show create table ..., two additional table properties would be shown. Except that, there is no impact on existing use cases, as the default values are the same as before.

Test Plan

• Test cases added to show the impact of new table properties

Contributor checklist

• Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Iceberg Connector Changes
* Add table properties `metadata_previous_versions_max` and `metadata_delete_after_commit` to maintain the previous metadata files :pr:`23260`
* Enable Iceberg with hive catalog to support deleting old metadata files after commit based on the table properties :pr:`23260`

[steveburnett]

Thanks for the doc! Suggested rephrasing, please adapt my suggestions to fit the column formatting, and let me know if my suggestions change the meaning in a way you're not comfortable with.

[steveburnett]

LGTM! (docs)

Thanks for the update! Pulled updated branch, new local doc build, looks good.

[ZacBlanco]

Two minor nits to make the core logic more understandable for future readers, otherwise LGTM

[ZacBlanco]

This comment was confusing to me at first because TableMetadata#addPreviousFile isn't referenced in this logic, but rather indirectly when generating previousFiles. I think we can rewrite it to be a bit more clear

Suggested change

	// TableMetadata#addPreviousFile builds up the metadata log and uses
	// TableProperties.METADATA_PREVIOUS_VERSIONS_MAX to determine how many files should stay in
	// the log, thus we don't include metadata.previousFiles() for deletion - everything else can
	// be removed
	// metadata.previousFiles() is populated using TableProperties.METADATA_PREVIOUS_VERSIONS_MAX
	// thus we don't include metadata.previousFiles() for deletion - everything else can be removed

[hantangwangd]

Thanks for the suggestion, it makes the meaning of the comment more clear. Fixed, please take a look when available!

[agrawalreetika]

Thank you for the addition. Have suggested small doc level changes, please have a look.
Otherwise LGTM

[agrawalreetika]

Nit : Would it make more clear if we has a description The max number of old metadata files to keep since this is something now user can control with this config property.

[hantangwangd]

Sure, it seems better, I have changed here as well as the corresponding description in IcebergTableProperties and docs. Please take a look when convenient. @agrawalreetika @steveburnett @ZacBlanco

[yingsu00]

What would happen if metadata_delete_after_commit = false but metadata_previous_versions_max = 100? Would the files be deleted after 100? If not, shall we specify that metadata_previous_versions_max only works if metadata_delete_after_commit is set to true?

[hantangwangd]

metadata_delete_after_commit = false means that Iceberg wouldn't automatically delete the old and untracked metadata files after each commit. It wouldn't affect the functionality of metadata_previous_versions_max, which specify how many previous metadata files would be tracked in current metadata.

For example, if we set metadata_previous_versions_max = 1 and metadata_delete_after_commit = false on the creation of an Iceberg table, and execute several insert operations which will lead in several metadata files.
We will get only 1 previous metadata entry in current metadata's log ( through tableMetadata.previousFiles()). But find that none of the metadata files were actually deleted.

On the other way, if we set metadata_previous_versions_max = 1 and metadata_delete_after_commit = true and do the same things as above, we will find that only 2 metadata files remain.

[steveburnett]

LGTM! (docs)

Pull updated branch, new local docs build. Looks good, thanks!