[CT-2236] [Feature] Remove lib.py [as much as possible]

[ChenyuLInx]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Remove current lib.py

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

[ChenyuLInx]

In dbt-server, there are currently the following use cases of lib.py

1. create a RuntimeConfig to reuse
   • this is always paired with 2 or 3
2. create a Manifest python object from an existing manifest.msgpack on disk, to reuse
   • this use case is only needed for SL worker initialization
3. create a Manifest python object from project file
4. serialize a Manifest python object, this is to save things to different locations.
   • saving things to different locations is for use case 2 listed above, and partial parsing
   • we already save a manifest.msgpack for partial parsing to a location here

In #6547 , we can add the following functions:

• return a Manifest python object in result
• return a RuntimeConfig(Doesn't have to RuntimeConfig, and we shouldn't guarantee any interface on this object.)
• return the partial parsing file write location
  This can fulfill the use case 1, 3, 4 above, 4 is fulfilled here given that is the location of a file is returned, dbt-server can implement logic to duplicate that file to other locations if needed.

Then what's left are use case 1, 2 combination. This use case is only for SL dbt-server initialization.(@racheldaniel correct me if I am wrong but compile query is the only operation in dbt-server that doesn't access local files at all, preview actually still read local files) We are a bit unclear on whether we need this function in dbt-server in the future, but leaning towards not needed side. And the timing for it is unclear
In the mean time, to make sure dbt-core 1.5 release smoothly, we have the following options

• Add a new code path in parse to skip read project files, just construct RuntimeConfig, read a manifest.msgpack into python object, and return them?
• create a new command load(?) hidden from CLI user, but available via dbtRunner, to do the above mentioned work
• leave function deserialize_manifest and get_dbt_config as is in lib.py, remove in later releases when we no longer need them.

I am leaning towards the last option here for now. It leave something in lib.py, but we should have a clear path on removing it if SL layer use case is no longer needed in dbt-server. And we can pivot to one of the first two if needed.

Thoughts? Any background/context that feels missing? @jtcohen6 @racheldaniel @aranke @stu-k

[aranke]

Let's take a step back and think about what interfaces are required here?
Currently, we're thinking too much about implementations, but we should get away from that.

[jtcohen6]

Let's leave these two functions in lib.py for v1.5:

• deserialize_manifest
• get_dbt_config

This is our way of saying, We're not determining the interfaces here until later. In the meantime, we'll continue to own the current implementation.

The acceptance criterion for this ticket is deleting all other methods from lib.py.