Long context is a billing model, not a feature
Every token you add to the prompt is paid for on every request.
A million-token window sounds like unlimited memory. In production it is a recurring cost that scales with conversation length.
Caching helps for repeated prefixes, but most chat traffic does not repeat cleanly enough to benefit.
The practical design is retrieval plus a small working set, not a giant prompt that contains everything.
#evaluation#inference#training
More in this section
🔒aiSubscriber
Why small models are winning on-device
Four-bit quantization plus better distillation is making 3B models genuinely useful in production.
ai
Synthetic data is hitting a wall nobody wants to name
Model collapse is real, but the more immediate problem is boring: attribution and licensing.
🔒aiSubscriber
The open-weights licence split is becoming permanent
Two families of licence, two incompatible expectations, and a lot of downstream confusion.