Unable to store blob type attachments in Electron

[Wuchv]

Case

bug

Issue

I've got an error when trying to store an attachment in electron renderer process, with pouchdb adapter.

    file: Pick<File, 'name' | 'type'> & { blob: Blob }

    await putAttachment(
              {
                id: file.name,
                data: file.blob,
                type: file.type,
              },
              true
            );

Throws:

Uncaught (in promise) TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string 
or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of Blob

Versions

    "pouchdb-adapter-leveldb": "^7.2.2",
    "rxdb": "^9.12.1",
    "electron": "^11.1.1",

Info

• Environment: electron
• Adapter: pouchdb
• Stack: Typescript,React

Code

After I see the source code, problem in this method, Buffer.from can't accept a Blob parameter.

  createBlobBuffer: function createBlobBuffer(data, type) {
    var blobBuffer;

    if (_util__WEBPACK_IMPORTED_MODULE_3__.isElectronRenderer) {
      // if we are inside of electron-renderer, always use the node-buffer
      return Buffer.from(data, {
        type: type
      });
    }
  /* ... */

And it is impossible to encrypt the blob type attachments, JSON.stringify also can't accept a Blob parameter.

var _encryptValue = function _encryptValue(value) {
  return encrypt(JSON.stringify(value), this.password);
};

[pubkey]

Can you extend this test to reproduce the error?

[Wuchv]

electron test
I can't download the file rxdb-local.tgz , may be because of the network reasons, so I didn't run the code, but I think this will reproduce the error.

[kukagg]

Can confirm that this is true. What was the reason to use node-buffer when isElectron()?

[kukagg]

I don’t think there’s any downside in removing L55-L60:

rxdb/src/plugins/attachments.ts

Lines 55 to 60 in 86967dc

	if (isElectronRenderer) {
	// if we are inside of electron-renderer, always use the node-buffer
	return Buffer.from(data, {
	type
	} as any);
	}

Correct me if I’m wrong.

[pubkey]

I am not sure why I had to add this.
We can remove everything as long as the tests still work.
But first we need to reproduce your bug in the CI.

[pubkey]

Ah I've noticed that the electron tests are currently commented out
I will try to reenable them.

[Wuchv]

I don’t think there’s any downside in removing L55-L60:

rxdb/src/plugins/attachments.ts

Lines 55 to 60 in 86967dc

	if (isElectronRenderer) {
	// if we are inside of electron-renderer, always use the node-buffer
	return Buffer.from(data, {
	type
	} as any);
	}

Correct me if I’m wrong.

I tried to delete this code and then the render process gone. Maybe because this or that

[pubkey]

I fixed the electron tests in the CI.
Please make a PR with a reproduction of your problem

[Wuchv]

I fixed the electron tests in the CI.
Please make a PR with a reproduction of your problem

npm WARN tarball tarball data for rxdb@file:rxdb-local.tgz (null) seems to be corrupted. Trying again.
I still can't download the file rxdb-local.tgz.

[pubkey]

rxdb-local.tgz is generated locally in the ci. It is not downloaded.

[pubkey]

I had to disable the electron tests from the CI again. They still randomly fail.
We should upgrade the electron versions in the examples and then we might be able to reproduce this.
Please help if you have time.

[Wuchv]

I had to disable the electron tests from the CI again. They still randomly fail.
We should upgrade the electron versions in the examples and then we might be able to reproduce this.
Please help if you have time.

I reproduced this error here, the test code is in preload.js

[pubkey]

That helps a bit. But I need a PR with a reproduction in the CI, I spend too much time in the past to debug other peoples setup.

[kukagg]

Added this #3098 but no issue in the test env. Maybe there is something bigger at play or my test is totally off. @Wuchv Would be nice to have you take a look at it.

[pubkey]

@kuka we also have tests that run inside of electron https://github.com/pubkey/rxdb/tree/master/examples/electron/test
But they are disabled in the CI atm because we first have to update electron otherwise it randomly fails.

[ash0x0]

Same issue here. Electron 11.4, renderer process, IndexedDB adapter. Is there a current workaround or ideas for something to try?
I tried putAttachment with Blob and ArrayBuffer. I'm thinking of trying base64 string but even if it works it's a pain to handle and temporary solution at best. Anything I can do to help resolve this?

[pubkey]

I am not sure what is happening here. Everyone has the same problem but noone wants to even update the electron example.

[kukagg]

@pubkey played with the electron test and it worked for me. I ran it many times and minor delay for async worked.

[kukagg]

Updated the PR with the Electron error reproduction and some improvements for the example #3098.

[ash0x0]

I'm not sure what the process is but I'm ready to help with whatever. #3098 is a good repro for my use too.

[kukagg]

@ash0x0 I don’t believe there is any process. Just introducing a PR with a fix in it is 90% of the task. No permission needed.

Btw, I tested out omitting blob handler in here

rxdb/src/util.ts

Lines 419 to 424 in 7bbc78c

	if (isElectronRenderer) {
	// if we are inside of electron-renderer, always use the node-buffer
	return Buffer.from(data, {
	type
	} as any);
	}

and it beared no fruit. I’ll play with it more but any help from you @ash0x0 and @Wuchv would be helpful.

@pubkey thank you for all the work you did/do!

[pubkey]

@ash0x0

• If you have the time, please update the electron and spectron versions in the electron example.
• Then we can enable the electron tests in the ci again.
• Then I can merge adds a blob attachment test for idb, #3022; #3098

[ash0x0]

I'm not sure here what's done and not by @kuka ? it seems #3115 fixes the first item. Do I PR to enable tests? or does #3115 need more work?

[kukagg]

@ash0x0 I simply reproduced the error we’re all experiencing in electron test. The changes include improvements for the example as well. Now we have to fix the error itself. It’s unclear what the source of the issue is it seems. You could do what @pubkey laid out and then try to find out what the root of the issue is.

Items are:

• Reproducing error of the issue with blob type attachments (done here https://github.com/kuka/rxdb/commit/983fe7a03f9f297259fe7b2d180b7af10e7ea5ac)
• Fixing error in the CI that makes Electron builds fail (apparently merge of 6fa2695 didn’t help so there’s something in there). (Would be cool to see artifacts of that CI instance as it worked for me locally, @pubkey ).
• Updating deps of the electron example so that we have latest libs in place. (you could jump on that @ash0x0)
• Fixing the bug that makes blob type/buffer not go through (you could investigate this as well, if feasible @ash0x0)

Basically overarching goal is to make sure Electron test works as expected so that we can clearly see it’s initially failing and passing once the fix is introduced @ash0x0 .

I’ll be able to dig deeper sometime this/next week as well.

[ash0x0]

Ok I'll get on 3 and 4 until you guys figure out the CI issue.

[pubkey]

It is very likely that updating the deps will fix the ci problems

[pubkey]

Closing this since it seems noone wants to work on it. Ping me when you want to work on that so I can reopen.

[ash0x0]

@pubkey I am working on it, just not fixed. I did the version update, added #3098 and tested, the CI works on my fork, the issue is there and attachment fails so that's where I'm at. It's definitely src/util.ts#L408 and there are variations with electron and node version for buffer handling.

[pubkey]

Oh sorry. I reopened it.

[pubkey]

There is a good chance that this is solved in the new RxDB major version.
#3279

Please test.

[pubkey]

Closing this, likely fixed in 10.0.0.
Otherwise please make a PR with a test that fails in the CI, I spend enough time trying to figure out what is going wrong here.

[Elendiar]

On "electron": "^13.1.9" and "rxdb": "^10.0.3" cant store attachement in renderer.

Simple code like

await document.putAttachment({
          id: "cat.jpg",
          data: "foo bar asldfkjalkdsfj",
          type: "text/plain",
        });

throw this error: Uncaught (in promise) TypeError: Failed to execute 'readAsText' on 'FileReader': parameter 1 is not of type 'Blob'.

If i pass new Blob([string]) instead "foo bar...", i recieve

 Uncaught (in promise) TypeError: Failed to execute 'readAsArrayBuffer' on 'FileReader': parameter 1 is not of type 'Blob'.
    at Object.readAsArrayBuffer (index-browser.js?f480:108)
    at appendBlob (index-browser.js?bfb3:29)
    at loadNextChunk (index-browser.js?bfb3:72)
    at Object.binaryMd5 (index-browser.js?bfb3:75)
    at preprocessBlob (index.js?ac47:223)
    at preprocessAttachment (index.js?ac47:250)
    at eval (index.js?ac47:282)
    at Array.forEach (<anonymous>)
    at Object.preprocessAttachments (index.js?ac47:263)
    at idbBulkDocs (index.js?6a7b:282)

It happens, because function readAsArrayBuffer(blob, callback) ... in node_modules\pouchdb-binary-utils\lib\index-browser.js recieve blob as object. If wrap blob inside this func to new Blob([blob]) all works. Im not good in blobs, but how i can pass/transform data in electron renderer to make this work ?

UPD: I fix this behaviour with

attachments: {
    encrypted: false,
  },

in schema. Seems problem appears after encryption, without encryption all works fine. But works only with blob, not string.
Stringdata still throw TypeError: Failed to execute 'readAsText' on 'FileReader': parameter 1 is not of type 'Blob'.
in

return new Promise(function (res) {
      // browsers
      var reader = new FileReader();
      reader.addEventListener("loadend", function (e) {
        var text = e.target.result;
        res(text);
      });
      var blobBufferType = Object.prototype.toString.call(blobBuffer);
      /**
       * in the electron-renderer we have a typed array insteaf of a blob
       * so we have to transform it.
       * @link https://github.com/pubkey/rxdb/issues/1371
       */

      if (blobBufferType === "[object Uint8Array]") {
        blobBuffer = new Blob([blobBuffer]);
      }

      reader.readAsText(blobBuffer);  // Uncaught (in promise) TypeError: Failed to execute 'readAsText' on 'FileReader': parameter 1 is not of type 'Blob'.
    });

[HZSamir]

Hello,
Setting encrypted to false did not fix this issue in my case.
Thing is the same code used to work fine on rxdb 9, upgrading to 10 raised this issue.
Anything I might try?
Thank you for your time.

[pubkey]

:)
#3022 (comment)
Thank you for your time :)

[HZSamir]

Must've missed that, apologies.
As always, thank you for you time.