Fix padding of deals, which only partially shipped in #5988

[ribasushi]

This pulls in forgotten parts properly implementing #5988
( previous testing focused exclusively on proper-sized offline-dealflow .cars )

Also adds complete end-to-end tests to ensure we do not break this again.

Both nontagged deps are the respective current master at time of writing:

• github.com/filecoin-project/go-fil-markets v1.6.0-rc1.0.20210723225932-46b52248a0f2
• github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1

Allows a workflow like:

~$ dd if=/dev/urandom bs=1M count=1 | ~/go-ipfs/cmd/ipfs/ipfs add --pin=false
added QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ ~/go-ipfs/cmd/ipfs/ipfs dag export QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2 > test_mib.car

~$ lotus client import --car ~/test_mib.car
Import 2, Root QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ ~/go/bin/stream-commp -p $(( 256 * 1024 * 1024 )) < test_mib.car
CommP:    54e4e75ddc3fffa8fd33d3ededc06e564603ac0fe62543ec6463d51b553be40b
CommPCid: baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy
Raw bytes:           1049073 bytes
Unpadded piece:    266338304 bytes
Padded piece:      268435456 bytes
CARv1 detected in stream:
Blocks:         5
Roots:          1
    1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
  { "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStartDeal", "params": [
    {
      "Wallet":"t01004",
      "Miner":"t01005",
      "EpochPrice":"0",
      "MinBlocksDuration":518400,
      "Data": {
        "Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
        "PieceCid":{ "/":"baga6ea4seaqfjzhhlxod775i7uz5h3pnybxfmrqdvqh6mjkd5rsghvi3ku56icy" },
        "PieceSize": 266338304
      }
    }
  ] }
'

~$ ~/go/bin/stream-commp -p $(( 128 * 1024 * 1024 )) < test_mib.car
CommP:    ed904105399ed346f6b03844abc14710a1748854c2781824d6bd1100e63b1807
CommPCid: baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby
Raw bytes:           1049073 bytes
Unpadded piece:    133169152 bytes
Padded piece:      134217728 bytes
CARv1 detected in stream:
Blocks:         5
Roots:          1
    1: QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2

~$ curl http://127.0.0.1:1234/rpc/v0 -X POST -H "Authorization: Bearer $(cat ~/.lotus/token)" -H "Content-Type: application/json" --data '
  { "jsonrpc": "2.0", "id":1, "method": "Filecoin.ClientStatelessDeal", "params": [
    {
      "Wallet":"t01004",
      "Miner":"t01005",
      "EpochPrice":"0",
      "ProviderCollateral":"0",
      "MinBlocksDuration":518400,
      "Data": {
        "TransferType": "manual",
        "Root":{ "/":"QmcFLqjyh2kvixuuvxgNUoHy55Rb6N6uuSq4CNfvtPoTJ2" },
        "PieceCid":{ "/":"baga6ea4seaqo3ecbau4z5u2g62ydqrflyfdrbilurbkme6ayetll2eia4y5rqby" },
        "PieceSize": 133169152
      }
    }
  ] }
'
{"jsonrpc":"2.0","result":{"/":"bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi"},"id":1}

~$ lotus-miner storage-deals import-data bafyreianhjvev3w6q5lteap3h7tkxbe2jaobwlsi7vzbcoobjpicg3foqi ~/test_mib.car

[whyrusleeping]

LGTM

[jennijuju]

Blocked on merging for now since it’s using a fil-market rc.

[jennijuju]

We’d love for this pr to be in lotus v1.11.1, @dirkmc when will the final release of market v1.6.0 be cut? Possibility by Tuesday noon?

[ribasushi]

@jennijuju test-unit-rest yeah... it's a new flake :(
I don't have perms to restart CI though