Add basic Get and Put Api signatures along with protobuf setup #1

G8XSU · 2022-10-06T00:09:16Z

Add Basic api signatures
Main file to review : src/main/proto/ess.proto

This will auto-generate required request/response objects.

Note: Since we didn't define the "service" proto, its not going to generate any grpc service stubs.
Here protobuf is only being used as a binary serialization format for request and response objects.(instead of writing custom binary serialization or json). This helps both client and server to create requests/response like class instances with ease.
This payload will be used as binary body in http-rest post requests.

Apart from that it is mostly auto-generated setup for gradle project.

Things like Auth will be added once we add API for creating a "store".

Currently this api can be used with any of the approach mentioned in doc, that is "Independent State Updates", "Global State Counter", "Incremental State Updates".
LDK will be using "Global State Counter", different applications might choose different approaches based on their needs.
Doc: https://docs.google.com/document/d/1mhTAnkXAtxPQRsn17gbEPIN5G8OW2h619f_yk8CXJTE/edit?usp=sharing

devrandom · 2022-10-07T09:40:17Z

src/main/proto/ess.proto

+}
+message GetResponse {
+  bool success = 1;
+  KeyValue value = 2;


is it useful to return just one value, or does it make more sense to return everything with a certain key prefix? for VLS at least we always want to get a complete snapshot of the storage for a node when we do perform a get. does LDK intend to make single item retrievals?

regarding "return everything with a certain key prefix" => i thought about it but this makes for an unbounded query on server-side hence we should be avoiding it
single get does mean some additional work for client.

Regarding "list by key prefix" => this in intended to be solved by listKeys api which will be a paginated api. For LDK as well, we have use-case of reloading parts of snapshot. (This will be done after computing version difference of keys b/w client and server-side)

I had added "Batch" operation support as "future item" in doc, but that also won't solve this problem because there will still be some batch limit. Client needs to use versioning to ensure consistent state instead of relying on getting everything at once in consistent fashion.
Another reason, I was being bit conservative about batch operations is because we intend to support storing values upto ~5MB, even a 10 item get might mean 50MB response depending on client-application.

Let me know if that works.

Side note: Currently this api can be used with any of the approach mentioned in doc, that is "Independent State Updates", "Global State Counter" or "Incremental State Updates". LDK will be using "Global State Counter"

I don't see "Independent State Updates" in the doc. However, that sounds like the approach that VLS would use. But would be interested in hearing more.

So I think the plan with listKeys would work, but we will need this sooner rather than later, since otherwise there is no way to discover what is in the store.

Yes, LDK needs listKeys and getKeysSummary immediately as well.
Just didn't include it in this commit. Wanted close on basic get and put first.

Heading was named slightly different. Added "Independent State Update" to that heading.

devrandom · 2022-10-07T09:42:28Z

src/main/proto/ess.proto

+  string key = 1;
+
+  // Write will only succeed if current DB version against key is same as in request.
+  int64 version = 2;


should document here what is the starting version number - i.e. when a key does not exist, should the version specified here be -1 or 0?

also, it may make more intuitive sense to provide current_version + 1 (i.e. the new version) here instead of current_version, but this is a matter of taste

I think from client perspective they will have to maintain a current version of object that they have in-memory or secondary store.
It could be more intuitive to start with 0 as first version as it will be same for client and they just call with their current version at-hand instead of any special handling.

Will add it to docs.

devrandom · 2022-10-07T09:51:53Z

I think we should never check in binaries to our repos, including gradle-wrapper.jar. I know it's convenient, but it's also a security problem. It's easy enough to provide instructions in the README or add a build script to select the right version. See https://stackoverflow.com/a/28473686

jkczyz · 2022-10-24T15:04:24Z

src/main/proto/ess.proto

+service EncryptedStorageService {
+  rpc get(GetRequest) returns (GetResponse);
+  rpc put(PutRequest) returns (PutResponse);
+  //TODO: Add getKeysSummary (Gives version numbers for all keys in a store) and list by prefix API's
+}


What is being "get" or "put"? I'd recommend following these naming conventions for both method and type names:

https://cloud.google.com/apis/design/naming_convention

Good point, changed it to GetObjectRequest and PutObjectRequest

src/main/proto/ess.proto

jkczyz · 2022-10-24T15:31:05Z

src/main/proto/ess.proto

+  string key = 1;
+
+  // Write will only succeed if current DB version against key is same as in request.
+  int64 version = 2;


Do all KeyValues in a PutRequest need to have the same version or can they differ?

Every key will have its own version. so they can differ.

jkczyz · 2022-10-24T15:32:13Z

src/main/proto/ess.proto

+  /*
+  If present, write will only succeed if current DB globalVersion against a storeId is
+  same as in request.
+   */
+  optional int64 globalVersion = 2;


Could you flesh out all the docs in this file? At a high-level, the versioning scheme should be defined.

Sounds good, i will write a basic explanation in docs.
Hoping to write a much more detailed api doc after impl though.

TheBlueMatt

Can you provide more justification for why there's protobufs to talk to the server? Why is it insufficient to do a simple HTTP(S) GET/PUT RESTful API? It seems this will run over HTTP(S) anyway, so that seems strictly simpler for both client and server? Am I missing something?

devrandom · 2022-10-31T22:50:24Z

I think the request/response bodies may be complex enough that protobufs and strongly-typed bindings to them would be an improvement over JSON. Also, more compact / performant.

TheBlueMatt · 2022-10-31T23:14:48Z

src/main/proto/ess.proto

+    Ref: https://en.wikipedia.org/wiki/Keyspace_(distributed_data_store)
+    It is up to clients to use single or multiple store's for their usage.
+   */
+  string storeId = 1;


I don't understand why we want this?

Its for key grouping,
If service owner wants to implement following:

client-isolation/rate-limiting/throttling

authorization

billing etc.

Also I could delete a store which is no longer being used.
They can do this based on this identifier.

The service owner definitely wants to implement client isolation, I think that's required for this API fundamentally. My question is why we need this on top of that. Given that clients can only store some limit amount and the clients are all isolated, I don't understand why we need clients to support different storage sections. Can that not also be implemented as just having different "clients"? We're not building an enterprise-scale storage platform for a user, we're building an enterprise-scale storage platform to store data for a lot of small users, I think.

It is exactly that, building "enterprise-scale storage platform to store data for many users".
storeId is an abstraction for clientId.
A client can have multiple stores if they choose to but most might just use single store, its up to them.
Its just like we can create multiple EBS instances and its not assumed that we will only use one.

storeId is client isolation at storage layer, can think of it as userId.

TheBlueMatt · 2022-10-31T23:21:57Z

src/main/proto/ess.proto

+message KeyValue {
+  string key = 1;
+
+  int64 version = 2;


Hmm, isn't it kinda awkward on the server to have both a global-version and versions on all the KV entries? Does it make sense to have it in the API explicitly when it will ultimately be implemented by having a dummy "globalVersion" key that could just be a separate KeyValue done on the client side?

Internally it is implemented like just another KV entry only.
We can consider it but i preferred this minor implementation complexity on server side than on client side.

We don't want to explain to clients every time that this is how you should be using this if you want global consistent writes. From multiple discussions i have had on versioning of state, it looks like devs could miss this detail if api is not explicit about it.
By doing so, we can give clients guidance on two popular options for using versioning. And they can simply choose based on their requirements.

(Using GlobalVersioning is optional for clients, for LDK it is required, VLS might not need it.
For single-device access globalVersion is not required.)
Provides a simpler interface for clients using single-item writes who don't care about multi-item transactions.
Using transactions for doing global consistent write is more of an implementation detail.

I guess my question is more if we're controlling the client code or not. I understood that, generally, users wouldn't be exposed to the protobufs here, they'd use some libvss-{rust,java,etc}-client library. Including the globalversion K-V can then be done at the client level without users being exposed to it.

src/main/proto/ess.proto

TheBlueMatt · 2022-10-31T23:25:20Z

I think the request/response bodies may be complex enough that protobufs and strongly-typed bindings to them would be an improvement over JSON. Also, more compact / performant.

I'm not sure why they would be? I agree let's avoid JSON, but there's not that many fields here, PUT is the only complicated one, and its really just a set of K-Version-V tuples. That could be sent in a format way simpler than protobufs?

devrandom · 2022-11-01T12:54:51Z

I think the request/response bodies may be complex enough that protobufs and strongly-typed bindings to them would be an improvement over JSON. Also, more compact / performant.

I'm not sure why they would be? I agree let's avoid JSON, but there's not that many fields here, PUT is the only complicated one, and its really just a set of K-Version-V tuples. That could be sent in a format way simpler than protobufs?

What did you have in mind? (It's probably best to have a format that is commonly used in web API client code for various languages.)

TheBlueMatt · 2022-11-01T20:04:24Z

I mean pick a simple format. bencoding or something equally simple makes sense to me. binary packing works fine to me, or hex encode it, csv it, i dunno, doesn't matter. Protobufs are a lot of complexity and language support is generally good, but not always great. Something that you can hand-write in any language seems like it would be better than protobufs.

devrandom · 2022-11-01T20:51:37Z

It sounds like you'd end up supporting client libraries for different languages, people writing sub-standard encoders/decoders, etc.

TheBlueMatt · 2022-11-01T22:35:52Z

Right, sub-standard encoders for Protobufs is an issue ;). If the encoding were simpler sub-standard encoders should be only a performance issue.

G8XSU · 2022-11-03T23:30:18Z

protobuf/grpc is industry standard and is simple.
Main reasons for protobuf and grpc being:

Highly performant
Easier to maintain backwards compatibility (cannot emphasize enough)
Cross language support (Platform-agnostic)
Makes it easier to support http/2 and async requests (we also won't have to worry about connection/channel reuse/multiplexing)

There is no way, we should be writing custom encoding/format for this. (i thought @devrandom concern about sub-standard serde was on this.)
I also think there are reasonable grpc/protobuf third party libs available in languages other than the officially supported ones. for e.g. (hyperium/tonic)

The request/response objects are complex enough, there are two more api's yet to be added, we will be adding more fields and metadata to schema as server matures.

Also, i thought we had decided on it in Meeting Notes Oct 5th: "Will use java and grpc to build backend"

Jasonvdb · 2022-11-04T08:15:13Z

Sorry wasn't in the meeting. Curious though what were the reasons java was chosen?

G8XSU · 2022-11-04T08:30:49Z

@Jasonvdb
Main reasons for java were : Native cloud support by official SDK's as well as related ecosystem for database accessors.
Rust doesn't have production ready SDK's from cloud providers yet. (Most are in beta, developer preview)
Client side will be written in rust and client-native languages, communicate with server side.
Also, Its not a one-way door decision, same proto definition can be re-implemented by another service.

TheBlueMatt · 2022-11-04T23:03:18Z

Main reasons for protobuf and grpc being:

I get the sense grpc support across different languages is more mixed than protobuf. For example going grpc will mean that you cannot talk to a VSS server from entirely within a javascript context, which I think is a no-starter. I think we kinda have to go the HTTPS route for the actual talking, even if we use protobufs for the contents.

Highly performant

This depends on the specific language we're talking about? But, yes, most languages have at least a reasonably good protobuf library.

Easier to maintain backwards compatibility (cannot emphasize enough)

If we go the grpc route I agree with you, but I don't think we can. With HTTPS we get this for free - /v1, /v2, etc :)

Cross language support (Platform-agnostic)

Right, see my above list of suggested alternatives, this is pretty easy to match :).

There is no way, we should be writing custom encoding/format for this.

I don't really understand peoples' hesitance to do this - if we're concerned about adding fields and backwards compat and all that good stuff I may agree, but I don't think that's a concern - and for a three-tuple (where one element is fixed-length) almost anything is reasonable :)

Also, i thought we had decided on it in Meeting Notes Oct 5th: "Will use java and grpc to build backend"

Ah, I recall the java conclusion, I may have missed the grpc note, sorry about that.

jkczyz · 2022-11-07T16:05:29Z

Not sure if I have all the nuances of the conversations, but I believe gRPC exposes an HTTP interface with /v1, /v2, etc. style versioning. Resources:

TheBlueMatt · 2022-11-08T18:59:09Z

Right, so let's just stick with an HTTP interface :). I'm not sure I understand what the specific value of grpc is on top of a simple HTTP interface if we have to support at least HTTP anyway?

devrandom · 2022-11-09T17:13:26Z

There is no way, we should be writing custom encoding/format for this.

I don't really understand peoples' hesitance to do this - if we're concerned about adding fields and backwards compat and all that good stuff I may agree, but I don't think that's a concern - and for a three-tuple (where one element is fixed-length) almost anything is reasonable :)

I haven't convinced myself yet that the API will remain simple.

TheBlueMatt · 2022-11-09T18:06:23Z

I haven't convinced myself yet that the API will remain simple.

Right, I think my point here was that the API today is super duper trivially simple, and if we ever need to change that we can switch to protobuf in a /v2/ API.

jkczyz

In general, I think we should flesh out all the documentation and have some top-level or message-level docs explaining the versioning scheme. See comments for some details.

jkczyz · 2022-11-22T22:13:00Z

app/src/main/proto/vss.proto

+  string key = 2;
+}
+message GetObjectResponse {
+  KeyValue value = 2;


Do we need to return the key?

Yes, its generally a good practice to return key as well.
This is helpful if and when we want to have batch api in future. Makes for a consistent code.
Also slightly easier to implement client side parallel batch calls.

jkczyz · 2022-11-23T21:04:59Z

app/src/main/proto/vss.proto

+  /*
+    If present, write will only succeed if current DB globalVersion against a storeId is
+    same as in request.
+    Client is expected to send their current global version at hand.


How does the client track the global version and when may it be different from the server?

Client side and server side global version only diverge in case of multiple devices.
I.e. multiple client-side updating same state.
Just like they keep track of current value/version against a key, they will need to store global version(either in-memory or disk)
We will provide a vss-client which will do all of this for them.

jkczyz · 2022-11-23T21:06:23Z

app/src/main/proto/vss.proto

+    For first write of store, version should be '0'. If write succeeds, clients must increment
+    their global version(on device) by 1.
+
+    Requests with conflicting version will fail with [`ConflictException`] as ErrorCode.


Will this return the actual global version on failure? Or does the client need to do something to get in sync?

No, it does not. Returning it from here doesn't add much value and keeps api simpler (and can have some race-conditions if used without considering some edge cases.)
ConflictException indicates that their view of store is out-of-sync. And they need to re-sync their view by fetching all key-versions and global version from server.
And for the keys having different version on server, client will need to update their view(can be done lazy).
(This is will also be provided as implemented method in vss-client)

Is there a method for fetching everything? A client may not have all keys if another client wrote a new key.

Yes it will be in subsequent PR once we have get and put.

app/src/main/proto/vss.proto

jkczyz · 2022-11-23T21:08:52Z

app/src/main/proto/vss.proto

+  /*
+    If present, write will only succeed if current DB globalVersion against a storeId is
+    same as in request.
+    Client is expected to send their current global version at hand.


"at hand" is a little vague. We should spell out what this means.

"Clients are expected to store version against every key locally.
When initiating a PutRequest, request should contain current version for that key."
Let me know if this sounds better or we need to add more context.

Not sure I follow what is meant by "store version against every key locally". This comment is on globalVersion, and my understanding is there is one global version for a store.

jkczyz

Could you go ahead and push any updated docs? Would be easier to read with new commits rather than looking at comments.

jkczyz · 2022-11-30T21:19:30Z

app/src/main/proto/vss.proto

+  /*
+    If present, write will only succeed if current DB globalVersion against a storeId is
+    same as in request.
+    Client is expected to send their current global version at hand.


Not sure I follow what is meant by "store version against every key locally". This comment is on globalVersion, and my understanding is there is one global version for a store.

jkczyz · 2022-11-30T21:20:41Z

app/src/main/proto/vss.proto

+    For first write of store, version should be '0'. If write succeeds, clients must increment
+    their global version(on device) by 1.


Should make it clear the version is incremented on the next put request?

Added clarification

app/src/main/proto/vss.proto

jkczyz · 2022-11-30T21:23:32Z

app/src/main/proto/vss.proto

+    For first write of store, version should be '0'. If write succeeds, clients must increment
+    their global version(on device) by 1.
+
+    Requests with conflicting version will fail with [`ConflictException`] as ErrorCode.


Is there a method for fetching everything? A client may not have all keys if another client wrote a new key.

TheBlueMatt

I assume authentication is handled at the HTTP layer?

TheBlueMatt · 2022-11-30T23:25:57Z

app/src/main/proto/vss.proto

+    their corresponding key versions(client-side) by 1. These updated key versions should be used in
+    subsequent PutRequests for the keys.
+
+    Requests with conflicting version will fail with [`ConflictException`] as ErrorCode.


Should we include which version value was wrong in the error result? That way clients know what to download?

Clients might need to sync their whole view depending on conflict and their usage.
Conflict could be either due to key versions or global version.
ErrorResponse should be simple and adding all this info could complicate it. Instead it is easier to re-use existing api's for the functionality.

TheBlueMatt · 2022-11-30T23:26:56Z

app/src/main/proto/vss.proto

+option java_multiple_files = true;
+package org.vss;
+
+message GetObjectRequest {


Don't we need an API to fetch the list of keys stored (and maybe the versions of them)?

yes, its part of separate api, will add it in following PR's

jkczyz

Could you add documentation to all messages and fields? The message docs can go into detail about the usage scenario. And fields can explain anything that should be considered when setting them.

jkczyz · 2022-12-01T17:34:54Z

app/src/main/proto/vss.proto

+  /*
+    If present, write will only succeed if current DB globalVersion against a storeId is
+    same as in request.
+    Client is expected to global version against store. And PutRequest should contain


Looks like there's a word missing in the sentence.

Done, 👍

Regarding "The message docs can go into detail about the usage scenario. And fields can explain anything that should be considered when setting them."

I created an issue and we will add these details once we have an implementation.
#3

Could we do this now? Doing it in a follow-up just adds extra review burden.

I have added lot more docs now for every field. But there are still many things which we could cover in here.
I didn't want to stall this PR for docs changes and want to close on interface.

app/src/main/proto/vss.proto

jkczyz

I didn't comment exhaustively on it, but note my comment on using the definite article "the" where appropriate. We'll want to me a little more formal in the language used in docs than everyday informal speech.

app/src/main/proto/vss.proto

jkczyz · 2022-12-07T19:12:42Z

app/src/main/proto/vss.proto

+    their global version(client-side) by 1. This updated global version should be used in subsequent
+    PutObjectRequests for the store.
+
+    Requests with conflicting version will fail with [`ConflictException`] as ErrorCode.


The [] notation is used in rustdocs for linking. Does proto doc generation have a similar feature?

no, there is no such feature afaik.
Here, i was just using it to quote specific object-field, replaced it with just ` quotes

app/src/main/proto/vss.proto

jkczyz

Largely looks good modulo commented on grammar and nits.

app/src/main/proto/vss.proto

jkczyz

Looks good. Just a few minor comments.

app/src/main/proto/vss.proto

jkczyz · 2022-12-15T21:49:15Z

app/src/main/proto/vss.proto

+
+message GetObjectRequest {
+
+  //  StoreId: All APIs operate within a single storeId.


Only use one space between // and text throughout.

jkczyz

@devrandom Could you take another pass?

app/src/main/proto/vss.proto

devrandom · 2022-12-19T13:40:13Z

app/src/main/proto/vss.proto

+    It is meant to be read and understood programmatically  by code that detects/handles errors by
+    type.
+   */
+  ErrorCode errorCode = 1;


I would not have ErrorCode track the HTTP error codes in any way, to prevent the user from making assumptions about any correlation between the two. in general, we may have error conditions in the future that don't fit any HTTP error code.

devrandom · 2022-12-19T13:42:07Z

app/src/main/proto/vss.proto

+message PutObjectResponse {
+}
+
+// When HttpStatusCode is not ok (200), the response `content` contains a serialized ErrorResponse


unfortunately, I don't think all gRPC client libraries let you access the HTTP error code, or parse the body if the HTTP code is 4xx.

I recommend looking into the tonic Rust crate to see what is available at least in the Rust case. in particular see the Status struct, which is returned in the failure case.

and lastly, I think all bets are off if the HTTP code is 5xx, because that could be caused by an intermediate proxy and might not reach the gRPC server at all.

also, consider the gRPC native error codes - https://grpc.github.io/grpc/core/md_doc_statuscodes.html

for example, the FAILED_PRECONDITION code might be a good match for an incompatible version in a Put.

since it is not a grpc service,
we cannot use status struct acc. to grpc protocol (it uses http/2 trailing headers)
and we don't need to follow grpc status codes.

In case someone wants to create a grpc service, they could use status struct and serialize this errorResponse message inside status.

I somehow missed that the current PR is not for gRPC anymore. is there more information about the underlying protocol - I don't immediately see it in the PR?

Since we didn't define the "service" proto, its not going to generate any grpc service stubs.
Here protobuf is only being used as a binary serialization format for request and response objects.(instead of writing custom binary serialization or json). This helps both client and server to create requests/response like class instances with ease.
This payload will be used as binary body in http-rest post requests. I think we will need to add this detail on overall project readme for vss-server as well as some client with examples.

OK, sounds good

FWIW, the Status error codes are meant to be protocol-agnostic.

https://cloud.google.com/apis/design/errors

Whether we use HttpStatus codes or protobuf status codes,
the same concern remains that in future we might want to introduce error conditions which donot map directly or have one to many mapping.
Removed this association as part of another comment

G8XSU · 2022-12-21T23:50:05Z

Changed to underscore_separated_names for field names and kept camel case for structs. (acc. to protobuf generation guideline).
Synced with Matt offline and will be going ahead and merging this.

devrandom reviewed Oct 7, 2022

View reviewed changes

G8XSU force-pushed the init branch from dc518fa to 7e53621 Compare October 7, 2022 22:31

G8XSU marked this pull request as ready for review October 10, 2022 20:29

jkczyz self-requested a review October 20, 2022 17:15

jkczyz reviewed Oct 24, 2022

View reviewed changes

TheBlueMatt reviewed Oct 31, 2022

View reviewed changes

src/main/proto/ess.proto Outdated Show resolved Hide resolved

G8XSU force-pushed the init branch from e42fa52 to b6cd163 Compare November 19, 2022 03:08

G8XSU requested a review from jkczyz November 21, 2022 22:42

jkczyz reviewed Nov 23, 2022

View reviewed changes

G8XSU requested a review from TheBlueMatt November 30, 2022 19:59

jkczyz reviewed Nov 30, 2022

View reviewed changes

G8XSU requested a review from jkczyz November 30, 2022 22:21

TheBlueMatt reviewed Nov 30, 2022

View reviewed changes

jkczyz reviewed Dec 1, 2022

View reviewed changes

G8XSU requested review from TheBlueMatt and jkczyz December 6, 2022 20:16

jkczyz reviewed Dec 7, 2022

View reviewed changes

G8XSU force-pushed the init branch from a2e94fe to 1a96eb4 Compare December 14, 2022 21:27

jkczyz reviewed Dec 14, 2022

View reviewed changes

G8XSU force-pushed the init branch from 1a96eb4 to 673c29d Compare December 15, 2022 01:04

jkczyz reviewed Dec 15, 2022

View reviewed changes

app/src/main/proto/vss.proto Outdated Show resolved Hide resolved

app/src/main/proto/vss.proto Outdated Show resolved Hide resolved

jkczyz reviewed Dec 15, 2022

View reviewed changes

G8XSU force-pushed the init branch from 673c29d to ad47baf Compare December 16, 2022 01:08

jkczyz approved these changes Dec 16, 2022

View reviewed changes

app/src/main/proto/vss.proto Outdated Show resolved Hide resolved

devrandom reviewed Dec 19, 2022

View reviewed changes

G8XSU force-pushed the init branch from ad47baf to fbdef28 Compare December 20, 2022 00:13

devrandom approved these changes Dec 20, 2022

View reviewed changes

jkczyz approved these changes Dec 21, 2022

View reviewed changes

jkczyz assigned TheBlueMatt Dec 21, 2022

G8XSU force-pushed the init branch from fbdef28 to 82cf1e5 Compare December 21, 2022 23:21

Add basic Get and Put Api signatures along with protobuf setup

debcb9b

G8XSU force-pushed the init branch from 82cf1e5 to debcb9b Compare December 21, 2022 23:26

G8XSU merged commit 7362e90 into lightningdevkit:main Dec 21, 2022

G8XSU mentioned this pull request May 10, 2023

VSS Tracking Issue #9

Open

31 tasks

		For first write of store, version should be '0'. If write succeeds, clients must increment
		their global version(on device) by 1.


		message GetObjectRequest {

		// StoreId: All APIs operate within a single storeId.

Add basic Get and Put Api signatures along with protobuf setup #1

Add basic Get and Put Api signatures along with protobuf setup #1

Conversation

G8XSU commented Oct 6, 2022 • edited Loading

Choose a reason for hiding this comment

G8XSU Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devrandom Oct 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devrandom commented Oct 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

devrandom commented Oct 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Oct 31, 2022

devrandom commented Nov 1, 2022

TheBlueMatt commented Nov 1, 2022

devrandom commented Nov 1, 2022

TheBlueMatt commented Nov 1, 2022 • edited Loading

G8XSU commented Nov 3, 2022

Jasonvdb commented Nov 4, 2022

G8XSU commented Nov 4, 2022 • edited Loading

TheBlueMatt commented Nov 4, 2022

jkczyz commented Nov 7, 2022

TheBlueMatt commented Nov 8, 2022

devrandom commented Nov 9, 2022

TheBlueMatt commented Nov 9, 2022

jkczyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

G8XSU Dec 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz left a comment

Choose a reason for hiding this comment

G8XSU commented Oct 6, 2022 •

edited

Loading

G8XSU Oct 7, 2022 •

edited

Loading

devrandom Oct 7, 2022 •

edited

Loading

TheBlueMatt commented Nov 1, 2022 •

edited

Loading

G8XSU commented Nov 4, 2022 •

edited

Loading

G8XSU Dec 1, 2022 •

edited

Loading

G8XSU Dec 21, 2022 •

edited

Loading