Add support for command JSON.OBJKEYS

[arpitbbhayani]

Add support for the JSON.OBJKEYS command in DiceDB similar to the JSON.OBJKEYS command in Redis. Please refer to the following commit in Redis to understand the implementation specifics - source.

Write unit and integration tests for the command referring to the tests written in the Redis codebase 7.2.5. For integration tests, you can refer to the tests folder. Note: they have used TCL for the test suite, and we need to port that to our way of writing integration tests using the relevant helper methods. Please refer to our tests directory.

For the command, benchmark the code and measure the time taken and memory allocs using benchmem and try to keep them to the bare minimum.

[vigneshrajkumar]

@arpitbbhayani I'll signup for this

[lucifercr07]

@vigneshrajkumar assigned, thanks for picking this up.

[iamskp11]

Hey @lucifercr07 @vigneshrajkumar ,
Since there are no updates here since 5 days, is this still being worked on? Will be happy to work on this

[JyotinderSingh]

Hey @lucifercr07 @vigneshrajkumar ,

Since there are no updates here since 5 days, is this still being worked on? Will be happy to work on this

Assigned

[iamskp11]

Hey @JyotinderSingh @lucifercr07 ,
Consider this case where a path matches with multiple JSON objects. In that cases, we will have array of arrays, where each inner array will contain a list of keys at one of the matching path.

The inner arrays will be randomly arranged + keys inside inner array will also be in random order. Similar to what happens with json.get. In Redis, we always get same ordering, similar to what is stored in JSON.

I have implemented functionality of JSON.OBJKEYS, but was confused with what approach can I take to perform integration test.

As of now, for testing, once I have output, I am trying to sort each individual inner array (if it is a list of key), and then matching it with expected result (which is also sorted in same way). Have tested on mutliple cases, and also verified the results with redis-stack-server, and they look correct.

Do you have some other idea which I can take for testing in this particular case?

[JyotinderSingh]

Hey @JyotinderSingh @lucifercr07 ,

Consider this case where a path matches with multiple JSON objects. In that cases, we will have array of arrays, where each inner array will contain a list of keys at one of the matching path.

The inner arrays will be randomly arranged + keys inside inner array will also be in random order. Similar to what happens with json.get. In Redis, we always get same ordering, similar to what is stored in JSON.

I have implemented functionality of JSON.OBJKEYS, but was confused with what approach can I take to perform integration test.

As of now, for testing, once I have output, I am trying to sort each individual inner array (if it is a list of key), and then matching it with expected result (which is also sorted in same way). Have tested on mutliple cases, and also verified the results with redis-stack-server, and they look correct.

Do you have some other idea which I can take for testing in this particular case?

Array ordering inside array fields is deterministic. Field ordering inside JSON is not deterministic.

You cannot sort the array as a part of the test, since it defeats the purpose of the array being a sorted container. Note that our JSON.GET implementation also maintains correct array ordering. So this is a non-issue.

However in case of field ordering being non deterministic, we have tackled this so far by providing all possible permutations of the result set. Another more idiomatic possibility if you can it in your test code is to use assert.ElementsMatch (it's present somewhere in the code)

[JyotinderSingh]

Also, we don't need to handle this in our server code, JSON standard doesn't enforce key ordering for the json fields.
We just need to fix our tests to handle this.

[JyotinderSingh]

We had discussion around thjs behaviour in the following pull request #365

[iamskp11]

Hey @JyotinderSingh ,
I read the discussion in PR #365 .

Ordering inside Array is fine, they are deterministic.
Non-deterministic key ordering is also expected for JSON.

What I meant to say is, in case of $..a path, we would be getting the multiple path values in any random order. Inside those json objects, the keys are also in any random order.

Suppose json is

If we say json.get $..a, we would be getting the three a values in any order (non-deterministic).
"[{\"b\":2,\"c\":1},{\"big_key1\":1,\"big_key2\":2},[3]]" , or
"[{\"big_key2\":2,\"big_key1\":1},{\"b\":2,\"c\":1},[3]]", or
"[{\"big_key1\":1,\"big_key2\":2},{\"b\":2,\"c\":1},[3]]", or
"[{\"c\":1,\"b\":2},{\"big_key1\":1,\"big_key2\":2},[3]]"

Now, the json.objkeys would be responding as
[[b c ] [big_key1 big_key2] (nil)],
[ [big_key2 big_key1] [b c] (nil)],
[[big_key1 big_key2] [b c] (nil)],
[[c b ] [big_key1 big_key2] (nil)] respectively for above 4 values of path $..a.

All the above four combinations are possible (some more are possible too) and are correct.

So, in these cases, maybe sorting and then comparing with sorted expected value would be a good way to test. Other way can be to have all possible permutations(of array of arrays) as expected value, and then check if result is one of them.

[JyotinderSingh]

Hey @JyotinderSingh ,

I read the discussion in PR #365 .

Ordering inside Array is fine, they are deterministic.

Non-deterministic key ordering is also expected for JSON.

What I meant to say is, in case of $..a path, we would be getting the multiple path values in any random order. Inside those json objects, the keys are also in any random order.

Suppose json is

If we say json.get $..a, we would be getting the three a values in any order (non-deterministic).

"[{\"b\":2,\"c\":1},{\"big_key1\":1,\"big_key2\":2},[3]]" , or

"[{\"big_key2\":2,\"big_key1\":1},{\"b\":2,\"c\":1},[3]]", or

"[{\"big_key1\":1,\"big_key2\":2},{\"b\":2,\"c\":1},[3]]", or

"[{\"c\":1,\"b\":2},{\"big_key1\":1,\"big_key2\":2},[3]]"

Now, the json.objkeys would be responding as

[[b c ] [big_key1 big_key2] (nil)],

[ [big_key2 big_key1] [b c] (nil)],

[[big_key1 big_key2] [b c] (nil)],

[[c b ] [big_key1 big_key2] (nil)] respectively for above 4 values of path $..a.

All the above four combinations are possible (some more are possible too) and are correct.

So, in these cases, maybe sorting and then comparing with sorted expected value would be a good way to test. Other way can be to have all possible permutations(of array of arrays) as expected value, and then check if result is one of them.

We have a helper method to check for array permutations, can we make use of that?

[iamskp11]

Hey @JyotinderSingh ,
I read the discussion in PR #365 .
Ordering inside Array is fine, they are deterministic.
Non-deterministic key ordering is also expected for JSON.
What I meant to say is, in case of $..a path, we would be getting the multiple path values in any random order. Inside those json objects, the keys are also in any random order.
Suppose json is

If we say json.get $..a, we would be getting the three a values in any order (non-deterministic).
"[{\"b\":2,\"c\":1},{\"big_key1\":1,\"big_key2\":2},[3]]" , or
"[{\"big_key2\":2,\"big_key1\":1},{\"b\":2,\"c\":1},[3]]", or
"[{\"big_key1\":1,\"big_key2\":2},{\"b\":2,\"c\":1},[3]]", or
"[{\"c\":1,\"b\":2},{\"big_key1\":1,\"big_key2\":2},[3]]"
Now, the json.objkeys would be responding as
[[b c ] [big_key1 big_key2] (nil)],
[ [big_key2 big_key1] [b c] (nil)],
[[big_key1 big_key2] [b c] (nil)],
[[c b ] [big_key1 big_key2] (nil)] respectively for above 4 values of path $..a.
All the above four combinations are possible (some more are possible too) and are correct.
So, in these cases, maybe sorting and then comparing with sorted expected value would be a good way to test. Other way can be to have all possible permutations(of array of arrays) as expected value, and then check if result is one of them.

We have a helper method to check for array permutations, can we make use of that?

I can see one : arraysArePermutations - it is for 1D integer array .
This case is different - output is 2D array, with randomness at both levels - arrays and array fields.

[JyotinderSingh]

Hey @JyotinderSingh ,

I read the discussion in PR #365 .

Ordering inside Array is fine, they are deterministic.

Non-deterministic key ordering is also expected for JSON.

What I meant to say is, in case of $..a path, we would be getting the multiple path values in any random order. Inside those json objects, the keys are also in any random order.

Suppose json is

If we say json.get $..a, we would be getting the three a values in any order (non-deterministic).

"[{\"b\":2,\"c\":1},{\"big_key1\":1,\"big_key2\":2},[3]]" , or

"[{\"big_key2\":2,\"big_key1\":1},{\"b\":2,\"c\":1},[3]]", or

"[{\"big_key1\":1,\"big_key2\":2},{\"b\":2,\"c\":1},[3]]", or

"[{\"c\":1,\"b\":2},{\"big_key1\":1,\"big_key2\":2},[3]]"

Now, the json.objkeys would be responding as

[[b c ] [big_key1 big_key2] (nil)],

[ [big_key2 big_key1] [b c] (nil)],

[[big_key1 big_key2] [b c] (nil)],

[[c b ] [big_key1 big_key2] (nil)] respectively for above 4 values of path $..a.

All the above four combinations are possible (some more are possible too) and are correct.

So, in these cases, maybe sorting and then comparing with sorted expected value would be a good way to test. Other way can be to have all possible permutations(of array of arrays) as expected value, and then check if result is one of them.

We have a helper method to check for array permutations, can we make use of that?

I can see one : arraysArePermutations - it is for 1D integer array .

This case is different - output is 2D array, with randomness at both levels - arrays and array fields.

Yeah, got it. Can you try to make the existing method work with multiple levels? If that doesn't work we can go ahead with writing the permutations ourselves and comparing against those.