Add support for Konnect documents

[rainest]

Add support for dumping Konnect document entities (https://konnect.konghq.com/docs/ under the "service_packages/:service_package_id/documents" and "service_versions/:service_version_id/documents" headers).

This PR includes changes necessary to implement an initial implementation for handling documents in deck konnect dump, including a new document type, logic to pull documents attached to service versions and service packages, state database schema and management for documents, and logic to write document metadata in konnect.yaml and document contents to files on disk.

While the UX for this may still change (there are some open questions around how to handle the document file format), I don't expect the innards of the type and state to fundamentally change to support any future UX changes, so IMO this is in a reasonably reviewable state. Targeted to a feature branch off the tip of main. Future UX changes and additional command support (sync/diff/etc.) will go there after before opening a PR to main.

[rainest]

Supersedes #328. Attempting to PR the individual type/service, dump, state, and file writing components was confusing, since the components don't make much sense when added in isolation. As of now, this draft implements a working PoC for the dump command only.

[rainest]

Copy/paste of Michal's comment on the previous PR:

What do you think about an alternative/polymorphic API structure like below?

Assuming the following:

• a service package has a URL (/api/service_packages/:id)
• a service version has a URL (/api/service_versions/:id)
• a document can be a child of either a service package or a service version and its URL is the parent URL + /documents/:id

it stood out to me that type Document tries to do work that is the responsibility of its parents' types (by listing supported parent types, and needing to know the URL structure for each of these parent types)

If we make the following change:

• implement func URL() on the ServicePackage and ServiceVersion types:

  func (p *ServicePackage) URL() string {
    return fmt.Sprintf("/api/service_packages/%s", p.ID)
  }
  
  func (v *ServiceVersion) URL() string {
    return fmt.Sprintf("/api/service_versions/%s", v.ID)
  }

• define an abstract type with method URL:

  type URLer interface { // inspired by fmt.Stringer and the go tradition
  // of naming interfaces function name + 'er', there are certainly dozens
  // of better names out there
    URL() string // or maybe (string, error)
  }

• replace Document's ServicePackage and ServiceVersion fields with

  type Document struct {
    // ...
    Parent URLer `json:"-"`
  }

• implement Document.URL:

  func (d *Document) URL() string {
    return fmt.Sprintf("%s/documents/%s", d.Parent.URL(), d.ID)
  }

we'll free type Document from the need to:

• know all the possible types of Parent
• know the inner details of parent implementations (specifically: what URLs in the API parents reside under)
• have the logic to prevent two parents of different types being defined on a single Document

[rainest]

I like the proposed interface for parent URLs in the type service, but don't see how we can elegantly achieve something similar in state/document.go.

The indices in the schema there reference specific other memdb tables, and I don't think there's any way we can get away from that.

We can perhaps get away from everything but that by exposing the state index constants outside the package and using those in file/writer.go, e.g. replace:

 documents, err := kongState.Documents.GetAllByServicePackageID(*p.ID)

with:

documents, err := kongState.Documents.GetAllBy(state.DocumentsByServicePackageID, *p.ID)

The existing pattern is for state to define dedicated functions for each index, but I don't think there's much reason to use those versus a generic function that accepts an index argument: the file package code is already aware of what index it needs to use (hence how it knows which function is appropriate).

Edit: having played around with the state part a bit now, we also need some new way to handle index population. SubFieldIndexer relies on that specific struct field's presence, which doesn't provide a way to properly split into the SP and SV indices. Will have to see about how we can handle that.

[rainest]

2971f21 implements the parent interface suggestion. state/document.go is probably best viewed as-is; the diff from the previous version doesn't make much sense.

I don't think using separate indices for Service Version/Package relationships in state is possible without essentially violating the goals of using the generic interface. Populating separate indices would probably extending the MethodIndexer to accept arguments require some convoluted method that returned nothing when invoked for a document whose parent didn't match the index type. Requiring that parents generate their own key and relying on key method authors not to introduce conflicts isn't ironclad, but seems a reasonable ask given that the keys are simple.

Not sure what happened to deepcopy generation or why that's failing only on the push.

[mflendrich]

Looking at DeepCopyParentInfoer, I need to better understand why DeepCopy needs to made a deep copy the parent as well. What do we need the deep copy for, and do linked entities (here: the parent) need to be deep-copied as well?

[mflendrich]

Shouldn't this happen in the outer loop? You want to mkdir just once per service package, right? Probably with a if len(sp.Documents) > 0.

[rainest]

Something of a leftover from an earlier understanding of the document objects (handled incorrectly, to boot).

Originally, our reading of the Document Path value was that it was actually a filesystem path, and that we'd need to create a nested structure under the SP/SV directories to handle the complete path. The original code didn't actually do this: handling those paths properly would require a MkdirAll() per document.

In practice, Konnect doesn't use Path like a filesystem path, and per confirmation from Henri P.:

Paths are only used as display names

The rework in 575c8fa works off that understanding and creates the SP/SV directories only once, and only if they have an attached document.

[rainest]

Looking at DeepCopyParentInfoer, I need to better understand why DeepCopy needs to made a deep copy the parent as well. What do we need the deep copy for, and do linked entities (here: the parent) need to be deep-copied as well?

@hbagdi can perhaps answer the "what for" better than I can. We most often use the copies when performing a diff, where we pass a copy to the CRUD creator rather than passing the original object from the target state, though I'm not immediately seeing what the consequences of using the original would be.

Copying the linked entities is just adhering to the standard meaning of a deep copy AFAIK: deep copies are recursive and make copies of both the struct you're copying and structs within it. Using the original linked entities means you instead have a shallow copy.

On a more practical level, the K8S deepcopy-gen tool fails if you don't include that function and associated tags when you use interfaces like we do here.

653773c, per conversation with Harry:

[both options for handling stdout/stdin mode along with documents] have unnecessary UX problems. The best thing we can do is avoid - as an output target for deck konnect dump.
Least amount of code on our side and no surprises for the end user.

575c8fa sanitization stuff: as paths are, in practice, just names, I've ditched plans to separate them into directory and filename components and applied an sanitization function to both them and the SP/SV directory names.

The sanitization function strips leading path separators and URL-encodes internal path separators. My rationale was:

• URL-encoded path separator strings (e.g. %2F) are unlikely to appear naturally in those names.
• The web UI document uploader does append a leading slash to all document paths, although it doesn't display them. Stripping them in the file representation looks neater.
• We still include the original field value in konnect.yaml. This sanitization only applies to what we write to the FS; API interactions are unaffected.

This isn't perfect: you can create foo%2Fbar and foo/bar service packages that both have a /baz.md document and hit a collision. However, absent some character we can guarantee won't appear in those names (and there are none, as far as I know), we have to choose something that can potentially collide, and this at least seems reasonably unlikely to.