Concurrency, throughput & the latency tradeoff
When does increasing concurrency stop helping throughput and start hurting latency?
The question
More requests in flight can improve throughput. But when capacity is limited, waiting time can grow. I want to understand where that tradeoff appears and how to measure it clearly.
The first experiment
Start with a local Python simulation: 20 requests, each taking approximately 100 milliseconds, with concurrency limits of 1, 2, and 5. Predict the total duration before running the experiment, then compare it with measurements.
What to measure
Separate total run duration, per-request service time, and time spent waiting for a concurrency slot. Record the workload, configuration, errors, and sample size alongside the results.
The limitation
A fixed-delay simulation is a starting point, not a model-serving benchmark. Investigating saturation requires an explicit capacity constraint or a real bottleneck. Any later measurements will distinguish simulated workloads from real model inference.
Current status
This is an experiment brief. Implementation and measurements have not started, and no performance results are claimed.