The Pitfalls of KV Cache Compression
This paper reveals that while KV cache compression improves throughput, it can severely degrade performance in multi-instruction scenarios by causing specific instructions to be ignored and system prompts to leak, but proposes simple eviction policy adjustments to mitigate these issues.