Instrument gRPC server span status codes #1127
Conversation
damemi
force-pushed
the
grpc-server-status-code
branch
2 times, most recently
from
637da87
to
0863a1e
Compare
internal/pkg/instrumentation/bpf/google.golang.org/grpc/server/bpf/probe.bpf.c
Show resolved
Hide resolved
| void *stream_ptr = get_argument(ctx, 2); | ||
| u32 stream_id = 0; | ||
| bpf_probe_read(&stream_id, sizeof(stream_id), (void *)(stream_ptr + stream_id_pos)); | ||
| struct grpc_request_t *grpcReq = bpf_map_lookup_elem(&streamid_to_grpc_events, &stream_id); |
There was a problem hiding this comment.
I think it might be better to look here at the grpc_events map.
If I understand this correctly, this probe should be called between the entry probe of handleStream (where an entry is created for the stream in the grpc_events map) and the return probe of handleStream.
If we can't find an entry it means the entry probe didn't run properly or didn't run at all, hence in that case we won't have a valid start time. WDYT?
There was a problem hiding this comment.
That's a good question. If handleStream isn't called (say the server hits an error before that) then we might still want to record the status from WriteStatus. Maybe in that case, this probe should check if an event already exists in grpc_events and if not start a new span with the status?
It seems like WriteStatus might be used in places besides handleStream (SendMsg and RecvMsg are 2 instances I found). Do you think these are relevant? If not then I agree with your suggestion
There was a problem hiding this comment.
I think that depends on the RPC life cycle the application is using.
The creation of a new stream can be called on the initialization phase of the instrumented process, that is also a case where we'll miss the entry probe of handleStream.
There was a problem hiding this comment.
The creation of a new stream can be called on the initialization phase of the instrumented process, that is also a case where we'll miss the entry probe of handleStream
That sounds like a case where we would want to create a new trace in the WriteStatus probe, is that what you mean?
There was a problem hiding this comment.
I'm not sure. We won't have the RPC method in that case
There was a problem hiding this comment.
WriteStatus does have the same Stream argument that is passed to handleStream, so we could try to get the RPC method from there. I just pushed an update that does that, ptal. I think this would be neat to cover the cases where we miss the original initialization like you mentioned. If not I'm all for just simplifying this down
internal/pkg/instrumentation/bpf/google.golang.org/grpc/server/probe.go
Outdated
Show resolved
Hide resolved
damemi
force-pushed
the
grpc-server-status-code
branch
from
9349e15
to
9328563
Compare
…/bpf/probe.bpf.c Co-authored-by: Tyler Yahn <[email protected]>
damemi
force-pushed
the
grpc-server-status-code
branch
from
9328563
to
d4212a2
Compare
damemi
force-pushed
the
grpc-server-status-code
branch
from
d4212a2
to
d45da58
Compare
| return 0; | ||
| } | ||
|
|
||
| UPROBE_RETURN(http2Server_WriteStatus, struct grpc_request_t, grpc_events, events, 2, stream_ctx_pos, false) |
There was a problem hiding this comment.
I am not sure about this return probe, since we already have UPROBE_RETURN(server_handleStream...).
The UPROBE_RETURN macro sends the event to user-space so I don't think we need to have it twice.
Ref #172