Hang caused by certain errors in processUnaryRPC

[smoelius]

Please see the FAQ in our main README.md, then answer the questions below before
submitting your issue.

What version of gRPC are you using?

v1.27.0

What version of Go are you using (go version)?

go version go1.14.4 linux/amd64

What operating system (Linux, Windows, …) and version?

Ubuntu 18.04.4 LTS

What did you do?

Fuzzed a gRPC service. The service hangs in this select:

grpc-go/internal/transport/handler_server.go

Lines 398 to 403 in a89bee7

	select {
	case fn := <-ht.writes:
	fn()
	case <-ht.closedCh:
	return
	}

The reason appears to be that ok is false in this conditional in processUnaryRPC:

grpc-go/server.go

Lines 990 to 994 in a89bee7

	if st, ok := status.FromError(err); ok {
	if e := t.WriteStatus(stream, st); e != nil {
	grpclog.Warningf("grpc: Server.processUnaryRPC failed to write status %v", e)
	}
	}

Because ok is false, WriteStatus is not called. So the transport t is not closed. So the second select condition never fires.

On the other hand, processUnaryRPC returns with an error, so the first select never fires either.

Changing the ok to ok || true makes the hang go away.

Values of err that can cause this problem include &errors.errorString{s:"unexpected EOF"} and transport.ConnectionError{Desc:"http: unexpected EOF reading trailer", temp:true, err:(*errors.errorString)(0x...)}.

Here is a partial stack trace leading to the select:

google.golang.org/grpc/internal/transport.(*serverHandlerTransport).runStream(...)
	google.golang.org/grpc/internal/transport/handler_server.go:398
google.golang.org/grpc/internal/transport.(*serverHandlerTransport).HandleStreams(...)
	google.golang.org/grpc/internal/transport/handler_server.go:388
google.golang.org/grpc.(*Server).serveStreams(...)
	google.golang.org/grpc/server.go:718
google.golang.org/grpc.(*Server).ServeHTTP(...)
	google.golang.org/grpc/server.go:770

And here is a partial stack trace leading to the conditional:

google.golang.org/grpc.(*Server).processUnaryRPC(...)
	google.golang.org/grpc/server.go:990
google.golang.org/grpc.(*Server).handleStream(...)
	google.golang.org/grpc/server.go:1313
google.golang.org/grpc.(*Server).serveStreams.func1.1(...)
	google.golang.org/grpc/server.go:722
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc/server.go:720

What did you expect to see?

A response indicating that an error occurred.

What did you see instead?

A hang.

[GarrettGutierrez1]

Can you please provide a minimal example or set of instruction that reproduces the error?

[smoelius]

Please try this (these instructions are based on: https://grpc.io/docs/languages/go/quickstart/):

• git clone -b v1.27.0 https://github.com/grpc/grpc-go
• cd grpc-go/examples/helloworld/greeter_server
• Replace the contents of main.go with the following:

package main

import (
	"bufio"
	"context"
	"fmt"
	"log"
	"net/http"
	"os"

	"google.golang.org/grpc"
	pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

// server is used to implement helloworld.GreeterServer.
type server struct {
	pb.UnimplementedGreeterServer
}

// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
	log.Printf("Received: %v", in.GetName())
	return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
	s := grpc.NewServer()
	pb.RegisterGreeterServer(s, &server{})

	reader := bufio.NewReader(os.Stdin)
	req, err := http.ReadRequest(reader)
	if err != nil {
		fmt.Printf("err = %#v\n", err)
		return
	}

	var resp = responseWriter{header: make(http.Header)}

	s.ServeHTTP(resp, req)

	fmt.Printf("resp = %#v\n", resp)
}

type responseWriter struct {
	header     http.Header
	statusCode int
}

// Header ...
func (resp responseWriter) Header() http.Header {
	return resp.header
}

// Write ...
func (resp responseWriter) Write(data []byte) (int, error) {
	return len(data), nil
}

// WriteHeader ...
func (resp responseWriter) WriteHeader(statusCode int) {
	resp.statusCode = statusCode
}

// Flush ...
func (resp responseWriter) Flush() {}

var _ http.ResponseWriter = (*responseWriter)(nil)
var _ http.Flusher = (*responseWriter)(nil)

• Type the following command:

echo '504f5354202f68656c6c6f776f726c642e477265657465722f5361794865
6c6c6f20485454502f322e300d0a486f73743a206c6f63616c686f73740d
0a436f6e74656e742d547970653a206170706c69636174696f6e2f677270
630d0a0d0a' | xxd -r -p | go run main.go

The program should hang.

Please let me know if this does not work, or if anything is unclear.