Duo Agent: Can we remove thread safe locking of gRPC client in workhorse

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem

As described in &19253 (comment 2735093445) there is a locking mechanism when sending messages down the gRPC stream.

We don't have any evidence yet but this locking mechanism could be a suspect for a dead-lock given the complex nature of all the communicating components.

Solution

Do we need the lock? It's possible that the Go client is already thread safe according to https://grpc.io/docs/languages/go/generated-code/ . But it's still not clear if that thread safety applies in the context of a single stream like we are doing. So it requires some investigation.

Technical notes

We should double check if it's actually thread safe with the following test:

1. Have a server that just blocks for a period of time when it receives a message
2. Write a Go client that attempts to send 2 messages from separate go routines without blocking on each other
3. We should not see any errors and both messages should eventually reach the server

I know for sure that the Python client has a problem with thread-safety as described at https://github.com/grpc/grpc/issues/4628#issuecomment-169357969 and there is generally conflicting information at https://groups.google.com/g/grpc-io/c/G7FzRNQBWhU .

Additionally it is worth considering whether or not removing this lock actually helps with anything if thread-safety inside of the Go gRPC client is implemented with locks per stream. If there was indeed a dead-lock due to how these 2 go routines are using this gRPC stream then it's possible that we would end up dead-locked either way because the Go gRPC client would need to lock at the same points.