-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Enable GCS, S3, R2 and MinIO as object stores for local runs #1843
Changes from 14 commits
a5a547e
affa887
eb6f680
e806edb
9a319ba
90d8cb1
9a9a0d2
7749af3
a1125d5
c266dfe
f90b8af
5e01e31
66286fa
7ce3492
578b4f5
453990b
13d1583
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
use anyhow::anyhow; | ||
use anyhow::Result; | ||
use clap::{Parser, ValueEnum}; | ||
use std::fmt::Write as _; | ||
|
@@ -81,3 +82,23 @@ pub struct PgProxyArgs { | |
#[clap(long)] | ||
pub cloud_auth_code: String, | ||
} | ||
|
||
#[derive(Debug, Clone, Parser)] | ||
pub struct StorageConfigArgs { | ||
/// URL of the object store in which to keep the data in. | ||
#[clap(short, long)] | ||
pub location: Option<String>, | ||
|
||
/// Storage options for building the object store. | ||
#[clap(short = 'o', long = "option", requires = "location", value_parser=parse_key_value_pair)] | ||
pub storage_options: Vec<(String, String)>, | ||
} | ||
|
||
fn parse_key_value_pair(key_value_pair: &str) -> Result<(String, String)> { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what if you specify options which aren't supported, or options that are specified more than once? will order matter (should we sort by key?) While I think it's good to have some kind of flexible configs here (given that not all backing storage is going have the same arguments), it's my assumption that we'll (eventually) have all this specified by a config file or most people will specify the options interactively in via SQL/python. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Unsupported options will be ignored but the initialization/setup will error out with a clear message (either from us, or courtesy of the object store crate itself) if some of the required keys are missing.
The last option value will be used in this case. |
||
key_value_pair | ||
.split_once('=') | ||
.map(|(key, value)| (key.to_string(), value.to_string())) | ||
.ok_or(anyhow!( | ||
"Expected key-value pair delimited by an equals sign, got '{key_value_pair}'" | ||
)) | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,9 @@ | ||
pub mod args; | ||
pub mod commands; | ||
mod highlighter; | ||
pub mod local; | ||
pub mod metastore; | ||
pub mod pg_proxy; | ||
mod prompt; | ||
pub mod rpc_proxy; | ||
pub mod server; | ||
pub mod util; | ||
|
||
pub mod args; | ||
mod highlighter; | ||
mod prompt; |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,3 +5,4 @@ pub mod srv; | |
|
||
mod database; | ||
mod storage; | ||
pub mod util; |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -27,8 +27,7 @@ pub async fn start_inprocess_local( | |
|
||
/// Starts an in-process metastore service, returning a client for the service. | ||
/// | ||
/// Useful for some tests, as well as when running GlareDB locally for testing. | ||
/// This should never be used in production. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice! 🫣 |
||
/// Useful for tests, as well as when running GlareDB locally. | ||
pub async fn start_inprocess( | ||
store: Arc<dyn ObjectStore>, | ||
) -> Result<MetastoreServiceClient<Channel>> { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I kept wanting there to be soemthing like "flatten with prefix" so we could still get strict parsing that we didn't have to write ourselves, but it seems like this isn't to be. I also sort of wanted
ArgGroup
to be able to get us more parsing, but again, no such luck.