YQ-2824: implement basis for type inference for CSV and TSV #4071
Conversation
EgorkaZ
force-pushed
the
yq-type-inferencinator
branch
3 times, most recently
from
d213493 to
eb82764
Compare
|
⚪
|
|
⚪
|
ydb/core/external_sources/object_storage/inference/arrow_fetcher.cpp
Outdated
Show resolved
Hide resolved
ydb/core/external_sources/object_storage/inference/arrow_inferencinator.cpp
Outdated
Show resolved
Hide resolved
| return InferCsvTypes(std::move(file), static_cast<const TsvConfig&>(config)); | ||
| case EFileFormat::Undefined: | ||
| default: | ||
| return nullptr; |
There was a problem hiding this comment.
что именно тут возвращается? не надо ли ошибку вернуть?
ydb/core/external_sources/object_storage/inference/arrow_inferencinator.cpp
Show resolved
Hide resolved
ydb/core/external_sources/object_storage/inference/arrow_inferencinator.cpp
Outdated
Show resolved
Hide resolved
ydb/core/external_sources/object_storage/inference/arrow_inferencinator.cpp
Outdated
Show resolved
Hide resolved
| return false; | ||
| case arrow::Type::LARGE_LIST: // TODO: is it true? | ||
| case arrow::Type::FIXED_SIZE_LIST: // TODO: is it true? | ||
| case arrow::Type::LIST: { // TODO: is ok? |
There was a problem hiding this comment.
у нас парсинг не поддерживает контейнеры. вывод типов тоже не должен поддерживать
EgorkaZ
force-pushed
the
yq-type-inferencinator
branch
from
eb82764 to
b95f1c6
Compare
EgorkaZ
force-pushed
the
yq-type-inferencinator
branch
from
b95f1c6 to
51b9802
Compare
|
⚪ |
|
⚪ |
|
⚪ |
|
⚪
|
|
⚪
|
| @@ -250,7 +250,7 @@ int TTpchCommandInit::Run(TConfig& config) { | |||
| TString notNull = ""; | |||
| TString createExternalDataSource; | |||
| TString external; | |||
| TString partitioning = "AUTO_PARTITIONING_MIN_PARTITIONS_COUNT"; | |||
| TString partitioning = ", AUTO_PARTITIONING_MIN_PARTITIONS_COUNT"; | |||
There was a problem hiding this comment.
Похоже это в другое ревью нужно вынести, наверное случайно сюда попало
| @@ -2,7 +2,7 @@ | |||
|
|
|||
| #include "yql_http_gateway.h" | |||
|
|
|||
| #include <curl/curl.h> | |||
| #include <contrib/libs/curl/include/curl/curl.h> | |||
There was a problem hiding this comment.
А это у тебя локально выстрелило?
There was a problem hiding this comment.
Кажется, да.
Вообще, вне contrib/ не вижу, чтобы ещё где-то было <curl/curl.h>
| } | ||
|
|
||
| NYql::TS3Credentials::TAuthInfo authInfo{}; | ||
| if (std::holds_alternative<NAuth::TAws>(meta->Auth)) { |
There was a problem hiding this comment.
А остальные auth где обрабатываются?
There was a problem hiding this comment.
Service account добавил. None обрабатывать не нужно, остальное не поддерживает eternal source
| @@ -265,12 +268,68 @@ struct TObjectStorageExternalSource : public IExternalSource { | |||
|
|
|||
| virtual NThreading::TFuture<std::shared_ptr<TMetadata>> LoadDynamicMetadata(std::shared_ptr<TMetadata> meta) override { | |||
| Y_UNUSED(ActorSystem); | |||
| auto format = meta->Attributes.FindPtr("format"); | |||
There was a problem hiding this comment.
Тут похоже еще сжатие не учитывается. Можно в отдельном ревью поддержать
ydb/core/external_sources/object_storage/inference/ut/arrow_inference.cpp
Outdated
Show resolved
Hide resolved
EgorkaZ
force-pushed
the
yq-type-inferencinator
branch
from
23df6d0 to
cc35d35
Compare
EgorkaZ
force-pushed
the
yq-type-inferencinator
branch
from
cc35d35 to
860091e
Compare
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
Changelog entry
...
Changelog category
Additional information
...