sha3: fix cSHAKE initialization for extremely large N and or S

While both impractical and unlikely, the multiplication could overflow
on 32-bit architectures.

The 64-bit architecture case is unaffected by both the maximum length
of Go slices being too small to trigger the overflow (everything except
s390), and it being safe to assume no machine has more than 2 EiB of
memory.

Fixes golang/go#66232

Change-Id: I19c15d42d2d6af35e296697159d43d02f513e614
GitHub-Last-Rev: 503e180
GitHub-Pull-Request: #286
Reviewed-on: https://go-review.googlesource.com/c/crypto/+/570876
LUCI-TryBot-Result: Go LUCI <[email protected]>
Reviewed-by: David Chase <[email protected]>
Reviewed-by: Filippo Valsorda <[email protected]>
Auto-Submit: Filippo Valsorda <[email protected]>
Reviewed-by: Michael Knyszek <[email protected]>

sha3/shake.go

@@ -85,9 +85,9 @@ func newCShake(N, S []byte, rate, outputLen int, dsbyte byte) ShakeHash {
 
 	// leftEncode returns max 9 bytes
 	c.initBlock = make([]byte, 0, 9*2+len(N)+len(S))
-	c.initBlock = append(c.initBlock, leftEncode(uint64(len(N)*8))...)
+	c.initBlock = append(c.initBlock, leftEncode(uint64(len(N))*8)...)
 	c.initBlock = append(c.initBlock, N...)
-	c.initBlock = append(c.initBlock, leftEncode(uint64(len(S)*8))...)
+	c.initBlock = append(c.initBlock, leftEncode(uint64(len(S))*8)...)
 	c.initBlock = append(c.initBlock, S...)
 	c.Write(bytepad(c.initBlock, c.rate))
 	return &c