Implement FlightSQL [EPIC]

[alamb]

As discussed in #1323, there is now sufficient community interest in moving forward with a Rust FlightSQL implementation. Thanks to @avantgardnerio and @wangfenjin we have basic POC level client and server implementations behind the --features=flight-sql-experimental feature flag.

This ticket tracks the work needed to change the feature flag from flight-sql-experimental to flight-sql (aka that we are comfortable saying that the rust client supports FlightSQL)

Known work items

Basics

• Implement basic FlightSQL Server #1386
• Implement Basic FlightSQL Client #1413
• Document / Deprecate arrow_flight::utils::flight_data_from_arrow_batch #3312
• Mid-level ArrowFlight Client #3371
• Update documentation / examples to use higher level encode/decode streams and clients

Testing

• Add CI test coverage for flight-sql-experimental features #3400
• FlightSQL Integration Test with JDBC / ODBC drivers  #2409

FlightSQL Client

• [FlightSQL] Add Builder for configuring options on flight sql client #3308
• [FlightSQL] Support HTTPs #3309
• [FlightSQL] Client Allow users to set arbitrary headers (rather than only the "authorization") #3310

Performance

• Support DictionaryArrays in Arrow Flight #3389

FlightSQL Server

[Dandandan]

One improvement for flightsql (and flight as well) is to support sending/tracking dictionaries in the API (to avoid sending shared dictionaries for every recordbatch.

[alamb]

One improvement for flightsql (and flight as well) is to support sending/tracking dictionaries in the API (to avoid sending shared dictionaries for every recordbatch.

Thanks @Dandandan -- I have added #3312 to the list above

[alamb]

An update here on my end: I have been working on FlightSQL as part of IOx and have several ideas on how to consolidate implementations and what would be useful to contirbute back upstream

My high level proposal is

1. Create a general Flight client wrapper on top of the automatically generated one that handle streaming RecordBatch conversions, for those who want it -- I have seen versions of this code in Ballista, now the arrow flighsql client (derived from Ballista) and IOx so I think it is a good common building block
2. Build a FlightSQL client on top of the general FlightClient that handles marshalling / unmarshalling the protobuf messages appropriate.

I plan a more full featured proposal in the next day or two

[tustvold]

Is this still tracking anything meaningful?

[alamb]

I don't think so 🤔 Maybe we should simply consider renaming the feature flag to flightsql rather than flightsql-experimental 🤔