Cached file management #434
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Write metadata for each cached http file once the file is downloaded completely. This metadata file contains basic information that would help us to have some management of cached files in system.
Added `duckdb.cache_info()` and `duckdb.cache_delete(cache_key TEXT)` functions can be used to manage cached files.
wuputah
reviewed
Nov 14, 2024
sql/pg_duckdb--0.1.0--0.2.0.sql
Outdated
| CREATE TYPE duckdb.cache_info AS ( | ||
| cache_key TEXT, | ||
| remote_path TEXT, | ||
| file_size TEXT |
There was a problem hiding this comment.
could we add the file's mtime here? it would be useful for clearing out older versions of files, basic LRU functions, etc.
There was a problem hiding this comment.
Added timestamp cache creation time.
Y--
reviewed
Nov 15, 2024
Comment on lines
56
to
57
| string metadata_info = cache_key + "," + remote_path + "," + std::to_string(total_size); | ||
| handle->Write((void *)metadata_info.c_str(), metadata_info.length(), 0); |
There was a problem hiding this comment.
Doesn't Write take a const char*?
Suggested change
| string metadata_info = cache_key + "," + remote_path + "," + std::to_string(total_size); | |
| handle->Write((void *)metadata_info.c_str(), metadata_info.length(), 0); | |
| handle->Write(cache_key.c_str(), cache_key.length(), 0); | |
| handle->Write(",", 1, 0); | |
| handle->Write(remote_path.c_str(), remote_path.length(), 0); | |
| handle->Write(",", 1, 0); | |
| auto s = std::to_string(total_size); | |
| handle->Write(s.c_str(), s.length(), 0); |
I'm also surprised we don't need a new line after?
There was a problem hiding this comment.
Write complains unless it is void * so we need to cast. Also, writing commas will also fail.
IMO, constructing single metadata line is simpler.
Y--
approved these changes
Nov 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For each cached file we will now create additional metadata file that contains simple metadata. Added funcitons
duckdb.cache_info()andduckdb.cache_delete(cache_key TEXT)that are used for managing cached files.