gRPC: Client side vs Server side load balancing, which one to choose?

Hello everyone,
My setup: Two FastAPI apps calling gRPC ML services (layout analysis + table detection). Need to scale both the services.

Question: For GPU-based ML inference over gRPC, does NGINX load balancing significantly hurt performance vs client-side load balancing?

Main concerns:

• Losing HTTP/2 multiplexing benefits
• Extra latency (though probably negligible vs 2-5s processing time)
• Need priority handling for time-critical clients

Current thinking: NGINX seems simpler operationally, but want to make sure I'm not shooting myself in the foot performance-wise.

Experience with gRPC + NGINX? Client-side LB worth the complexity for this use case?