device codec: faulty encoder crashed whole application server

[MrApe]

I discovered, that a buggy JSON encoder for enqueuing dowlinks brings down the complete application server. The RPC for encoding the base64 payload failes and crashes the server permanently.

Here is the reqpective log entry:

time="2021-10-08T08:04:18.998567399Z" level=info msg="finished unary call with code OK" ctx_id=baa90f07-fa73-4b55-92fe-84c3f3b0aead grpc.code=OK grpc.method=HandleGatewayStats grpc.service=as.ApplicationServerService grpc.start_time="2021-10-08T08:04:18Z" grpc.time_ms=7.557 peer.address="172.22.0.2:56630" span.kind=server system=grpc
(anonymous): Line 70:1 Unexpected reserved word (and 1 more errors)
panic: runtime error: hash of unhashable type parser.ErrorList

goroutine 746 [running]:
github.com/brocaar/chirpstack-application-server/internal/api/helpers.ErrToRPCError(0x263b740, 0xc000827dd0, 0x31, 0xc000ade390)
        /chirpstack-application-server/internal/api/helpers/errors.go:48 +0xd0
github.com/brocaar/chirpstack-application-server/internal/api/external.(*DeviceQueueAPI).Enqueue.func1(0x2664ea8, 0xc00036c990, 0x0, 0x0)
        /chirpstack-application-server/internal/api/external/device_queue.go:88 +0x8ea
github.com/brocaar/chirpstack-application-server/internal/storage.Transaction(0xc00061f388, 0x265af78, 0xc000a35110)
        /chirpstack-application-server/internal/storage/db.go:136 +0xbe
github.com/brocaar/chirpstack-application-server/internal/api/external.(*DeviceQueueAPI).Enqueue(0xc0006e0960, 0x265af78, 0xc000a35110, 0xc000a34f60, 0xc0006e0960, 0x7fe864a90ec0, 0xc0008c2300)
        /chirpstack-application-server/internal/api/external/device_queue.go:55 +0x325
github.com/brocaar/chirpstack-api/go/v3/as/external/api._DeviceQueueService_Enqueue_Handler.func1(0x265af78, 0xc000a35110, 0x142ed20, 0xc000a34f60, 0x1f, 0xc0008c4140, 0xc000b41f00, 0xc00061f480)
        /go/pkg/mod/github.com/brocaar/chirpstack-api/go/v3@v3.11.1/as/external/api/deviceQueue.pb.go:486 +0x89
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0x265af78, 0xc000a35110, 0x142ed20, 0xc000a34f60, 0xc000117b60, 0xc0007c2990, 0xc00061f588, 0xc000a35140, 0x343133632d643030, 0x3764666361393164)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0xb0
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x265af78, 0xc000a35110, 0x142ed20, 0xc000a34f60, 0x0, 0xc0007ec810, 0x265af78, 0xc000a35110)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/brocaar/chirpstack-application-server/internal/logging.UnaryServerCtxIDInterceptor(0x265af78, 0xc000a35110, 0x142ed20, 0xc000a34f60, 0xc000117b60, 0xc000117b80, 0x0, 0x0, 0x0, 0x0)
        /chirpstack-application-server/internal/logging/logging.go:50 +0x30d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x265af78, 0xc000a350b0, 0x142ed20, 0xc000a34f60, 0x1f, 0xc0501cc700bae9da, 0x1ba30442a1, 0x2e94540)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.UnaryServerInterceptor.func1(0x265af78, 0xc000a34fc0, 0x142ed20, 0xc000a34f60, 0xc000117b60, 0xc000117ba0, 0x265af78, 0xc000a34fc0, 0xc00036c978, 0xc000a34f90)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/logging/logrus/server_interceptors.go:32 +0x10f
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x265af78, 0xc000a34fc0, 0x142ed20, 0xc000a34f60, 0x1f, 0x142ed20, 0xc000a34f60, 0x20)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware/tags.UnaryServerInterceptor.func1(0x265af78, 0xc000a34f30, 0x142ed20, 0xc000a34f60, 0xc000117b60, 0xc000117be0, 0xb85f1a, 0x13e7ba0, 0xc000117c00, 0xc000117b60)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/tags/interceptors.go:22 +0x8c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0x265af78, 0xc000a34f30, 0x142ed20, 0xc000a34f60, 0xc00021e000, 0x0, 0xc00061fb30, 0x40e0f8)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:25 +0x63
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x265af78, 0xc000a34f30, 0x142ed20, 0xc000a34f60, 0xc000117b60, 0xc0007c2990, 0xc0007f6ba0, 0x50b226, 0x1418760, 0xc000a34f30)
        /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.1.0/chain.go:34 +0xd7
github.com/brocaar/chirpstack-api/go/v3/as/external/api._DeviceQueueService_Enqueue_Handler(0x1355ee0, 0xc0006e0960, 0x265af78, 0xc000a34f30, 0xc0008be300, 0xc000121590, 0x265af78, 0xc000a34f30, 0xc000b68cc0, 0x27)
        /go/pkg/mod/github.com/brocaar/chirpstack-api/go/v3@v3.11.1/as/external/api/deviceQueue.pb.go:488 +0x150
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00028cfc0, 0x2667a38, 0xc0007cd4d0, 0xc000b41f00, 0xc0001217a0, 0x2e7fce0, 0x0, 0x0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.33.1/server.go:1210 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00028cfc0, 0x2667a38, 0xc0007cd4d0, 0xc000b41f00, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.33.1/server.go:1533 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0007ec7c0, 0xc00028cfc0, 0x2667a38, 0xc0007cd4d0, 0xc000b41f00)
        /go/pkg/mod/google.golang.org/grpc@v1.33.1/server.go:871 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.33.1/server.go:869 +0x1fd

After fixing the payload codec (in device-profile > codec) the call succeed and the server survived.

[brocaar]

Thanks for reporting. I've been able to reproduce this. The issue is not the codec VM itself but the error that is returned. A Go map is used to map internal errors to gRPC error codes (.e.g. map[error]codes.Code, where codes.Code is the gRPC error code).

This works for most errors, but the error returned in this case, parser.ErrorList is a slice and this type is not "hashable". I think the following check needs to be added:

        ...

	// Some error values can not be hashed and return a runtime error:
	// https://github.com/brocaar/chirpstack-application-server/issues/631
	k := reflect.TypeOf(cause).Kind()
	if k == reflect.Slice {
		return grpc.Errorf(codes.Unknown, cause.Error())
	}

        ...

I'll test this today. Thanks again!