grpc-gw request is non-deterministic

[tkxkd0159]

Summary of Bug

The odd thing is that GetBlockByHash often fails with a Not Implemented error when requested as REST. There is no problem with gRPC. There seems to be a bug in grpc-gw as it often fails. (ref. #699)

Even I didn't change any code about LatestValidatorSet, it failed at #699 's latest commit

--- FAIL: TestIntegrationTestSuite (16.96s)
    service_test.go:31: setting up integration test suite
    network.go:174: acquiring test network lock
    network.go:179: created temporary directory: /tmp/TestIntegrationTestSuite2939229296/001/chain-eUzfDc3018637995
    network.go:188: preparing test network...
    network.go:382: starting test network...
    network.go:387: started test network
    --- FAIL: TestIntegrationTestSuite/TestLatestValidatorSet_GRPC (0.00s)
        --- FAIL: TestIntegrationTestSuite/TestLatestValidatorSet_GRPC/no_pagination (0.00s)
            service_test.go:199: 
                	Error Trace:	/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/service_test.go:199
                	            				/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/suite.go:91
                	Error:      	Received unexpected error:
                	            	rpc error: code = InvalidArgument desc = could not find voter params for height #16: invalid request
                	Test:       	TestIntegrationTestSuite/TestLatestValidatorSet_GRPC/no_pagination
        --- FAIL: TestIntegrationTestSuite/TestLatestValidatorSet_GRPC/with_pagination (0.00s)
            service_test.go:199: 
                	Error Trace:	/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/service_test.go:199
                	            				/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/suite.go:91
                	Error:      	Received unexpected error:
                	            	rpc error: code = InvalidArgument desc = could not find voter params for height #16: invalid request
                	Test:       	TestIntegrationTestSuite/TestLatestValidatorSet_GRPC/with_pagination
    --- FAIL: TestIntegrationTestSuite/TestLatestValidatorSet_GRPCGateway (0.02s)
        --- FAIL: TestIntegrationTestSuite/TestLatestValidatorSet_GRPCGateway/no_pagination (0.00s)
            service_test.go:232: 
                	Error Trace:	/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/service_test.go:232
                	            				/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/suite.go:91
                	Error:      	Received unexpected error:
                	            	unknown field "code" in tmservice.GetLatestValidatorSetResponse
                	Test:       	TestIntegrationTestSuite/TestLatestValidatorSet_GRPCGateway/no_pagination
        --- FAIL: TestIntegrationTestSuite/TestLatestValidatorSet_GRPCGateway/with_pagination (0.01s)
            service_test.go:232: 
                	Error Trace:	/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/service_test.go:232
                	            				/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/suite.go:91
                	Error:      	Received unexpected error:
                	            	unknown field "code" in tmservice.GetLatestValidatorSetResponse
                	Test:       	TestIntegrationTestSuite/TestLatestValidatorSet_GRPCGateway/with_pagination
    service_test.go:48: tearing down integration test suite
    network.go:512: cleaning up test network...
    network.go:535: finished cleaning up test network
    network.go:509: released test network lock
FAIL
FAIL	github.com/line/lbm-sdk/client/grpc/tmservice	17.755s

Currently, it seems that the issue has not been found because gRPC is mainly used, but similar issues may occur in other endpoints, so it needs to be tracked in the future.

Version

c8314f7

Steps to Reproduce

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[zemyblue]

The error action is https://github.com/line/lbm-sdk/actions/runs/3206232949/jobs/5239736676

[tnasu]

I found it again.

• https://github.com/line/lbm-sdk/actions/runs/3239719309/jobs/5309449199

--- FAIL: TestIntegrationTestSuite (16.91s)
    service_test.go:32: setting up integration test suite
    network.go:174: acquiring test network lock
    network.go:179: created temporary directory: /tmp/TestIntegrationTestSuite2125720976/001/chain-YFc4QI3772383345
    network.go:188: preparing test network...
    network.go:382: starting test network...
    network.go:387: started test network
    --- FAIL: TestIntegrationTestSuite/TestQueryLatestValidatorSet (0.02s)
        service_test.go:172: 
            	Error Trace:	/home/runner/work/lbm-sdk/lbm-sdk/client/grpc/tmservice/service_test.go:172
            	Error:      	Received unexpected error:
            	            	unknown field "code" in tmservice.GetLatestValidatorSetResponse
            	Test:       	TestIntegrationTestSuite/TestQueryLatestValidatorSet
    service_test.go:49: tearing down integration test suite
    network.go:512: cleaning up test network...
    network.go:535: finished cleaning up test network
    network.go:509: released test network lock
FAIL
FAIL	github.com/line/lbm-sdk/client/grpc/tmservice	17.604s

[tnasu]

Ostracon's process of storing(save and Bootstrap) this Block-related information has no locks. So calling RPC Validators during the save process will return error depending on the timing.

• save: https://github.com/line/ostracon/blob/2d44f5d64072fbdecbbda07dfc2ba6c2bfa4c8bd/state/store.go#L171-L208
• Bootstrap: https://github.com/line/ostracon/blob/2d44f5d64072fbdecbbda07dfc2ba6c2bfa4c8bd/state/store.go#L211-L238
• Validators: https://github.com/line/ostracon/blob/2d44f5d64072fbdecbbda07dfc2ba6c2bfa4c8bd/rpc/core/consensus.go#L26
  • LoadVoters: https://github.com/line/ostracon/blob/2d44f5d64072fbdecbbda07dfc2ba6c2bfa4c8bd/state/store.go#L490-L544

[tnasu]

The root cause is in Finschia/ostracon#478

• Validators of RPC depend on LoadVoters
• But,
  • Save the state with the committed height on Ostracon
    • Validators
    • LastProofHash
    • LastVotersParams
  • Save the state with the uncommitted height on Ostracon
    • NextValidators
• Then getting Validators via RPC:
  • Can get it with the committed height
  • Make an error if getting with the uncommitted height