Crypt push4 #70

penguin42 · 2018-02-23T03:00:22Z

This has a minor cleanup relative to push3, but also has 5 more commits on the end that do image decryption.
You might want to pull upto those or leave those last few separately, or take the lot.

penguin42 · 2018-02-24T03:24:37Z

Hmm this needs some rebasing for the last few image commits with the file/video url stuff I just merged. Working on it.

Windows build not tested. Note that libolm doesn't currently have the pkgconf magic. Compile with make -DMATRIX_NO_E2E to disable olm linking and e2e in general. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Make matrix-json use spaces rather than tabs. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Matrix signing relies on signing Json formatted to be 'canonical' as per https://matrix.org/speculator/spec/drafts%2Fe2e/appendices.html#canonical-json Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Add api call for uploading device and one time keys. It expects the json blobs to be presigned. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Somewhere for the e2e magic to happen. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Given a JsonObject, produce a canonical json blob, sign it and add the signature back into the JsonObject. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

We can't rely on having memset_s, so add our own equivalent that declares itself volatile. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

The OLM account state can be 'pickled'; we store this in Purple's account data store as a JSON string. Note; OLM lets us encrypt the pickle, but we've got no key to encrypt it with; I don't think Purple has anything for wiring into the system key store. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Allocate a blob of memory and fill it with random data. (There must be some existing portable function for this?!) Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Parse the output of olm_account_identity_keys to get a list of algorithms and the keys for them. There really should be a better way; the algorithm names in here match what we need in the keys section, but we have to append the device id later, so it's a pain that it returns (undocumented) JSON. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

If we've got a previously stored olm account state then use it, else create us a new one. Either way, upload the device keys to the server. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

The server returns the count of one-time-keys that it has, we parse that and detect if we need to send more. It's a little tricky in that a server that doesn't have any keys will send an empty response, so force sending in that case. If we get to the point of multiple types of keys this will need reworking to make sure we only send the key types needed. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Parse the key upload return to see if we need to send any one time keys. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

We're going to need a: database of olm sessions a device trust database a database of megaolm sessions Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Parse the device-2-device messages in a sync. We have to get the order right since we need the rooms created before parsing the d2d messages, but we may need to have processed the d2d messages to decrypt events in the room. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Route the count of remaining one time keys out of the sync result. We need the newer (v2?) API for this. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Parse the top level of m.room.encrypted messages. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Create an olm inbound session and use it to decrypt the message. Later patches will handle storing/retrieving the OLM sessions. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Check the decrypted messages have matching keys/devices etc before parsing them any further. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Attach to the room storage for inbound megolm sessions. TODO: Store to db TODO: Clean up this list in matrix_room_create_conversation Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Decrypted olm messages of type m.room_key create inbound megolm sessions. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Decrypt m.room.encrypted messages sent to a room; these come out with normal events. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Parse the decrypted JSON and route it back up to the room code. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Keep a hash/list of olm sessions in memory and load them from the db if we find we haven't got one. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Update sessions stored in the db after starting sessions. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Cleanup on closing the plugin or leaving a conversation. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Add a purple action (i.e. thing on the accounts->matrix-> menu) to display the device ID and public key. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

JSON web signatures almost use base64 but with a slightly odd encoding; decoding a JWS to base64. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

For encrypted images the decrypted message contains key information to decrypt the actual media once received. Add a structure and a parser to extract the information from the JSON. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

m.image messages in the e2e world have a different format and include keys. Decode the extra structures. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Pass the received image data through gcrypt to do the decryption using the previously received keys. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Use the previously stashed crypto data to decrypt the image. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

penguin42 · 2018-02-25T02:15:48Z

right that works on top of current head; please review/merge

krombel · 2018-04-04T23:05:13Z

matrix-api.c

+    gchar *json;
+
+    url = g_string_new(conn->homeserver);
+    g_string_append(url, "_matrix/client/unstable/keys/upload?access_token=");


the unstable endpoint should not be used

krombel · 2018-04-04T23:05:18Z

matrix-api.c

@@ -652,7 +652,7 @@ MatrixApiRequestData *matrix_api_sync(MatrixConnectionData *conn,

    url = g_string_new(conn->homeserver);
    g_string_append_printf(url,
-            "_matrix/client/r0/sync?access_token=%s&timeout=%i",
+            "_matrix/client/unstable/sync?access_token=%s&timeout=%i",


what is the reason to switch to unstable? It should not be used (anymore)

Thanks, krombel - it stabilised since I wrote this (the spec is still a bit odd - it says it's being worked on but has now started using the r0/ path). I'll update it.

Loh1917 · 2018-05-03T10:41:12Z

Hi is there possibility to import OLM keys saved in riot before?

richvdh · 2018-08-03T18:14:29Z

I'm going to merge this. It's not much use in a branch, and it looks sensible in as much detail as I've been able to look at it.

penguin42 requested a review from richvdh February 23, 2018 03:00

penguin42 added 28 commits February 25, 2018 02:08

e2e: Link with libolm

80e4e29

Windows build not tested. Note that libolm doesn't currently have the pkgconf magic. Compile with make -DMATRIX_NO_E2E to disable olm linking and e2e in general. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

matrix-json: Replace tabs

4b80bed

Make matrix-json use spaces rather than tabs. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

Canonical json generator

d76c2f7

Matrix signing relies on signing Json formatted to be 'canonical' as per https://matrix.org/speculator/spec/drafts%2Fe2e/appendices.html#canonical-json Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Add matrix_api_upload_keys

da5db06

Add api call for uploading device and one time keys. It expects the json blobs to be presigned. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Create MatrixE2EData type and matrix-e2e.[ch] files

22274a8

Somewhere for the e2e magic to happen. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Add matrix_sign_json

3b8e599

Given a JsonObject, produce a canonical json blob, sign it and add the signature back into the JsonObject. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

matrix-json: Add debug to dump json to stderr

8795475

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Add clear_mem

a5c4908

We can't rely on having memset_s, so add our own equivalent that declares itself volatile. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Restore OLM account state from Purple account data

86f665e

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: get_random

01705b7

Allocate a blob of memory and fill it with random data. (There must be some existing portable function for this?!) Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Retrieve or create Olm account state, send keys

77a487c

If we've got a previously stored olm account state then use it, else create us a new one. Either way, upload the device keys to the server. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Create and send one time keys

bd33bb9

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Parse key upload return

6f6c989

Parse the key upload return to see if we need to send any one time keys. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Wire in sqlite3 to hold e2e databases

f628590

We're going to need a: database of olm sessions a device trust database a database of megaolm sessions Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Route key count from sync

1527d39

Route the count of remaining one time keys out of the sync result. We need the newer (v2?) API for this. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Parse received d2d m.room.encrypted messages

2c0d806

Parse the top level of m.room.encrypted messages. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: decrypt olm encrypted message

5b31dcb

Create an olm inbound session and use it to decrypt the message. Later patches will handle storing/retrieving the OLM sessions. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Handle the decrypted olm messages

21f6b47

Check the decrypted messages have matching keys/devices etc before parsing them any further. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Provide storage for inbound megolm sessions

4cdeaab

Attach to the room storage for inbound megolm sessions. TODO: Store to db TODO: Clean up this list in matrix_room_create_conversation Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: handle m.room_key

754d187

Decrypted olm messages of type m.room_key create inbound megolm sessions. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Decrypt m.room.encrypted

9fb00bd

Decrypt m.room.encrypted messages sent to a room; these come out with normal events. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Route decrypted messages back up

437f18c

Parse the decrypted JSON and route it back up to the room code. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Look for olm sessions in our db

ea32cc0

Keep a hash/list of olm sessions in memory and load them from the db if we find we haven't got one. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Store olm sessions in our db

f76e7f4

Signed-off-by: Dr. David Alan Gilbert <[email protected]>

penguin42 added 8 commits February 25, 2018 02:08

e2e: Update sessions

2160555

Update sessions stored in the db after starting sessions. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Cleanup inbound megolm sessions in memory

c155422

Cleanup on closing the plugin or leaving a conversation. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Add device info action

8e49e86

Add a purple action (i.e. thing on the accounts->matrix-> menu) to display the device ID and public key. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Add jws decode

ab534c1

JSON web signatures almost use base64 but with a slightly odd encoding; decoding a JWS to base64. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Parse media decryption info

1bb115c

For encrypted images the decrypted message contains key information to decrypt the actual media once received. Add a structure and a parser to extract the information from the JSON. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Stash key info for encrypted images

6de15ee

m.image messages in the e2e world have a different format and include keys. Decode the extra structures. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Utility to decrypt media data

a99c220

Pass the received image data through gcrypt to do the decryption using the previously received keys. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

e2e: Wire in media decryption

7507de1

Use the previously stashed crypto data to decrypt the image. Signed-off-by: Dr. David Alan Gilbert <[email protected]>

penguin42 force-pushed the crypt-push4 branch from fa4ee0c to 7507de1 Compare February 25, 2018 02:13

krombel reviewed Apr 4, 2018

View reviewed changes

andreygursky mentioned this pull request Apr 17, 2018

[Feature Request] End-to-end encryption support #18

Open

richvdh added 2 commits August 3, 2018 19:12

use r0 rather than unstable

5abd573

Use r0 rather than unstable

1a47e5b

richvdh merged commit 5a7166a into matrix-org:master Aug 3, 2018

richvdh mentioned this pull request Jan 30, 2019

Make it clear that decryption is supported #90

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crypt push4 #70

Crypt push4 #70

penguin42 commented Feb 23, 2018

penguin42 commented Feb 24, 2018

penguin42 commented Feb 25, 2018

krombel Apr 4, 2018 •

edited

Loading

krombel Apr 4, 2018

penguin42 Apr 6, 2018

Loh1917 commented May 3, 2018

richvdh commented Aug 3, 2018

Crypt push4 #70

Crypt push4 #70

Conversation

penguin42 commented Feb 23, 2018

penguin42 commented Feb 24, 2018

penguin42 commented Feb 25, 2018

krombel Apr 4, 2018 • edited Loading

Choose a reason for hiding this comment

krombel Apr 4, 2018

Choose a reason for hiding this comment

penguin42 Apr 6, 2018

Choose a reason for hiding this comment

Loh1917 commented May 3, 2018

richvdh commented Aug 3, 2018

krombel Apr 4, 2018 •

edited

Loading