Question about MLA with TP

[ipiszy]

The DeepSeek paper mentions 4-way TP for the MLA attention layer at inference time. However, from code, it seems that different card has its own Linear module (e.g.

DeepSeek-V3/inference/model.py

Line 427 in ee4c4ea

self.wkv_a = Linear(self.dim, self.kv_lora_rank + self.qk_rope_head_dim)

) to project KV latents to num_heads, and either KV or the linear module are sharded across TP.

Is KV duplicated across all TP ranks in this case? Or, does different card have different KVs?