How do you hit sub-second LLM response times for real-time user-facing features?

Our AI assistant needs to feel instant. We've got streaming working but first-token latency is still 800ms–1.2s on our current stack. We're exploring speculative decoding, prompt caching, and smaller specialized models for the fast path. What's the realistic floor for first-token latency and what architecture changes have actually moved that number for teams with real user traffic?