What vector database trade-offs matter most when you cross 100M embeddings?

We're at 50M embeddings and growing. We've been on Pinecone but costs are becoming hard to justify. We've looked at Weaviate, Qdrant, pgvector, and Milvus. The benchmarks online don't reflect our query pattern — mostly high-recall searches with metadata filters and burst traffic. What are the real trade-offs teams have run into at 100M+ scale, especially around filter performance and index rebuild times?