When does adding a reranker to RAG actually improve quality vs. just adding latency?

We added a cross-encoder reranker on top of our dense retrieval step. It improved precision on our eval set but added 200ms and 30% cost. The gains were inconsistent — big improvements on ambiguous queries, near-zero on specific ones. How do teams decide if reranking is worth it for their query distribution, and is there a lighter-weight alternative that gets 80% of the benefit?