Prompt caching matters at scale Cache strategy and request distribution can dramatically affect cost; providers don’t necessarily optimize this for your application’s traffic pattern.
we should design your architecture around stable prefixes. on calls so let say for a company we making 1k+ calls if we have a stable prefix in prompt it will save lot of cost.
OpenAI takes roughly the first ~60–70 characters/tokens of the prompt, hashes that, and uses it for routing. He qualifies the exact number as approximate
I fwe send 100 requests almost simultaneously, you cannot assume all 100 hit the same warm cache.They get routed to different machines on OpenAI’s backend, where those machines can independently warm their own caches. so it take some time for openaI to cache the on all machine we have also note some time the request may routed to diff machine where will not have the cache hit to avoid use response id which make the request always to same machine
Connections
No outgoing connections.