`bytes.Buffer` will always overallocate (for performance) and we don't
have a great way of predicting the correct size. So, instead, copy once
when we're done.