Enable local persistence of detected schemas

[normanj-bitquill]

Is your feature request related to a problem? Please describe.

I'd like to be able to save the detected schema locally. The schema information could then be restored later without needing another schema detection.

There are some collections that we work with that are very large and we may need to sample many or all of the documents. Minimizing the number of times that we do schema detection is important.

Describe the solution you'd like

Hooks in the driver for:

• Persisting the schema of a collection
• Restoring the persisted schema of a collection (to skip schema auto detection)

Describe alternatives you've considered

None

Additional context

At this point I'm most curious about the feasibility of this feature. Is it possible? How big of a change would this require?

[birschick-bq]

Hi @normanj-bitquill

If the connected user has the correct authorization in the database, the schema is written back to the same database.

I noticed that we don’t have documentation of the required authorization to enable this. So that needs rectifying.

Would documenting how to authorize the user solve the problem for you or do need to solve this issue without providing extra access to your user.

Note, once one user creates the schema, all users benefit afterwards.

[normanj-bitquill]

Hello @birschick-bq

Storing the schema in the database may be a solution, but it could also be an undesirable burden for some of our customers. The project that I am working on is a standalone service that can connect to data sources. We have little control over the data sources.

Let me check if using the database is acceptable on our side and I'll get back to you.

[birschick-bq]

@normanj-bitquill

We've added the documentation for persisting to the database.

For your information, we had an implementation of file persistence which we removed in pull request #228

[normanj-bitquill]

@birschick-bq Thank for the link. I'll take a look.

[birschick-bq]

In progress: #477