Batching cuts cost and adds latency. Pick one.
The throughput graph and the latency graph point in opposite directions.
Larger batches keep accelerators busy and lower cost per token. They also make the first user in the batch wait for everyone else.
Serving systems hide this with continuous batching, which is better but not free. Tail latency still grows under load.
The right answer depends on the product. A coding assistant and a overnight summariser should not share a queue policy.
#agents#open weights#accelerators
More in this section
🔒aiSubscriber
Why small models are winning on-device
Four-bit quantization plus better distillation is making 3B models genuinely useful in production.
ai
Synthetic data is hitting a wall nobody wants to name
Model collapse is real, but the more immediate problem is boring: attribution and licensing.
🔒aiSubscriber
The open-weights licence split is becoming permanent
Two families of licence, two incompatible expectations, and a lot of downstream confusion.