perf: use batched inserts for models

[iand]

No description provided.

[iand]

Coalesces multiple inserts into a single statement, e.g. going from

INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid0', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"
INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid1', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"
INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid2', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"
INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid3', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"
INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid4', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"
INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid5', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"
INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES ('cid6', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) ON CONFLICT do nothing RETURNING "params"

to

INSERT INTO "messages" AS "message" ("cid", "from", "to", "value", "gas_fee_cap", "gas_premium", "gas_limit", "size_bytes", "nonce", "method", "params") VALUES 
('cid0', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT), 
('cid1', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT), 
('cid2', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT), 
('cid3', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT), 
('cid4', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT), 
('cid5', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT), 
('cid6', 'from', 'to', 'val', 'gasfeecap', 'gaspremium', 0, 0, 0, 0, DEFAULT) 
ON CONFLICT do nothing RETURNING "params"

[frrist]

Nice!

[placer14]

Nice trick! Will test this right now!

[placer14]

I did hit this...

2020-10-07T22:59:21.725Z        ERROR   message message/message.go:91   failed to process tipset        {"error": "process tipset: persist: persisting receipts: pg: can't bulk-insert empty slice messages.Receipts", "height": 126838}
2020-10-07T22:59:21.816Z        ERROR   message message/message.go:91   failed to process tipset        {"error": "process tipset: persist: persisting messages: pg: can't bulk-insert empty slice messages.Messages", "height": 126837}

[placer14]

More on this. I have reset these jobs a few times and they fail consistently. In the failure cases, they all have blocks with 0 messages in them. (https://filfox.info/en/tipset/126837) I think we just need to handle the empty case with an early return.

[placer14]

Looks great, please merge when ready. I'll try to mention my approvals as being sticky next time when appropriate.

[iand]

Don't worry, I was going to merge but hadn't got back to it after fixing up the commits.