Update model.py

According to https://github.com/facebookresearch/llama/blob/6b8cff0e0c5a379c3828e89691f4a5c352202643/llama/model.py#L228 and galatolofederico/vanilla-llama#8
botbw · Oct 16, 2023 · 3c8e376 · 3c8e376
1 parent 6d174e4
commit 3c8e376
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/examples/llama1/model/model.py b/examples/llama1/model/model.py
@@ -89,8 +89,8 @@ def __init__(self, args: ModelArgs):
             bias=False,
         )
         self.wo = nn.Linear(
-            args.dim,
             args.n_heads * self.head_dim,
+            args.dim,
             bias=False,
         )