Waiting too long (stuck) when getting a large ArrayBuffer content with GM_xhr

[ccloli]

Example script: https://gist.github.com/ccloli/9c6c6ab484cfc78bddcaf656afc6cc73

When getting a large binary file as Arraybuffer with GM_xmlhttpRequest, browser will stuck. I checked res, seems that Tampermonkey stored response data as string, and transfer it to ArrayBuffer when requesting res.response. So if response data is too large, Tampermonkey would waste too much time and CPU resource to transfer it.

Here are some screenshots of test result:

Tampermonkey on Chrome

Violentmonkey on Chrome

Tampermonkey on Firefox (however, I waited about 10 seconds until the music played)

GreaseMonkey on Firefox

[ghost]

I have encountered such a problem.

[ccloli]

Hi again, I'm just reply here to describe it clearly :-)

As Tampermonkey also released in Edge, so I tested my script to ensure that is it supports Edge. It's a script like the script that posted here, use GM_xmlhttpRequest and set responseType: "arraybuffer" to get some large files, then store all of them into a Zip file and give it to user. Yeah I know it's a bit crazy because it's not a good way to do it in browser. Luckily it works fine on Edge, the only problem is often to see "The page is not responding". So I tested my code again, and paste full details here.

Test script:

// ==UserScript==
// @name         GM_xhr test
// @version      1.0
// @description  Get a large binary file as ArrayBuffer with GM_xhr
// @include      http://localhost/*
// @grant        GM_xmlhttpRequest
// @connect      localhost
// ==/UserScript==

// modify url and @connect if needed
var url = 'http://localhost/test.mp3';
console.log('Environment: '); // Surely, I input it by myself! :-)

GM_xmlhttpRequest({
    method: 'GET',
    url: url,
    responseType: 'arraybuffer',
    onload: function(res){
        var t = new Date();
        console.log('File loaded at ' + t);
        console.log('Elapsed time: ' + (t - ot) + 'ms');

        var response = res.response;
        var t2 = new Date();
        console.log('Get ArrayBuffer content at ' + t2);

        console.log('File size: ' + response.byteLength);
        console.log('Elapsed time: ' + (t2 - t) + ' ms');

        // play music, also check if pasted time is correct
        var blob = new Blob([response], {type: 'audio/mpeg'});
        var blobURL = URL.createObjectURL(blob);
        var audio = document.createElement('audio');
        audio.src = blobURL;
        audio.play();
        var t3 = new Date();
        console.log('Start playing at ' + t3);
        console.log('Elapsed time: ' + (t3 - t) + ' ms (since file loaded)');
    },
    onerror: function(res){
        console.log('Can\'t load file.');
    }
});
var ot = new Date();
console.log('Requested at ' + ot);

The file test.mp3 is about 10MB.

In Chrome, I tested Violent monkey, Tampermonkey Legacy and Tampermonkey BETA, their logs are showing below:

It seems that after GM_xmlhttpRequest called, it took about 2 seconds to enter onload function (and in TM Legacy it took about 4 seconds), it probably be transferring response data to extension or others, because it only took no more than 1 second to get file data from server. In this picture, background page takes 99 ms to get file, but when onload function called, it took about 3 s :

This also happens in Firefox, here are the logs of GreaseMonkey and Tampermonkey in Firefox:

You can see the second request took about 30 seconds to get ArrayBuffer, because it said a script is busy:

(Sorry this screenshot is in Chinese, it said "A script on this page may be busy, or it may have stopped responding. You can stop the script now, open the script in the debugger, or let the script continue. Script: http://localhost/owncloud/:2")

So it would probably because of transferring data from background page to content script, however you can still noticed that in latest Tampermonkey, it takes a lot of time to get the ArrayBuffer content after the onload function called.

Also the same in Edge. I opened http://ccloli-pc/test.mp3 in Edge (ccloli-pc is my physical machine's name so that I can access to the localhost hosted on physical machine), it took about 6 minutes until the onload function called, thought the file has already cached (I'm not sure why it took so much time, probably it's in virtual machine, and at that time my physical machine is keeping CPU 90+%), and you can here the music is playing. After that the browser is no responding, music was keep playing but playing time is not updated, that's the time when requesting res.response.

(Sorry this screenshot is in Chinese, it said "ccloli-pc is not responding.")

Then half a minute later, it played the music again, and the play time was updated.

I tried logging the res argument in onload function, looks like res.response is a getter function, then I checked <function scope> and saw the file data is assigned as string. So I guess the latest version Tampermonkey stores response data as string, and transfer it to ArrayBuffer when res.response is called. So I think that's probably the reason why the page getting stuck.

Updated Sat, 03 Sep 2016 10:17 GMT: Added Microsoft Edge screenshot.

[derjanb]

Sorry for the delay. The fix required major changes that I had put off for too long.

However, the issue should be fixed at TM BETA 4.12.6124. 😅

I'm not sure the Chrome Webstore team will be able to review this version this year, but in the meantime, you can download it from here.

[AlttiRi]

Not fixed.

With TM BETA 4.12.6123: lags, but file is downloaded.
With TM BETA 4.12.6124: no lags, but file is not downloaded. At all.

From my issue:

// ==UserScript==
// @name        Issue 1107 (279) (GM.xmlHttpRequest, 104 MB, gfycat.com)
// @namespace   Issues
// @match       https://github.com/Tampermonkey/tampermonkey/issues/1107
// @match       https://github.com/Tampermonkey/tampermonkey/issues/279
// @match       https://example.com/
// @grant       GM.xmlHttpRequest
// @connect     giant.gfycat.com
// ==/UserScript==

!async function() {
    console.log("downloading (104 MB)...");
    const response = await new Promise((resolve, reject) => {
        GM.xmlHttpRequest({
            method: "get",
            url: "https://giant.gfycat.com/ConfusedRecentGuppy.mp4", // 104 MB
            // url: "https://giant.gfycat.com/ShockedSecondaryFiddlercrab.mp4", // 32 MB
            responseType: "blob",
            onload: resolve,
            onerror: reject,
        });
    });
    console.log("response:", response);

    const {response: blob} = response;
    downloadBlob(blob, "ConfusedRecentGuppy.mp4");
}();

function downloadBlob(blob, name) {
    const anchor = document.createElement("a");
    anchor.setAttribute("download", name || "");
    anchor.href = URL.createObjectURL(blob);
    anchor.click();
}

[derjanb]

@AlttiRi Hmm, I've double checked this with exactly this script and:

Windows 10 Pro 20H2
Chrome Version 87.0.4280.88 (Offizieller Build) (64-Bit)

Ubuntu 20.04
Chrome Beta Version 88.0.4324.50 (Official Build) beta (64-bit)
Chromium Version 87.0.4280.88 (Official Build) snap (64-bit)

and it works all the time: the log message is printed, it takes 10 to 15 seconds and the file becomes visible as browser download.

Do you see errors at the site's console or extension background page console?

[ccloli]

Well, it works fine from my side, got a mp4 file with 109,640,890 bytes and it matches content-length header. Maybe you can restart your browser and retry?

…

---原始邮件--- 发件人:&quot;AlttiRi&quot;<[email protected]&gt; 发送时间:2020年12月31日 星期四 下午1:21 收件人:&quot;Tampermonkey/tampermonkey&quot;<[email protected]&gt; 主题:Re: [Tampermonkey/tampermonkey] Waiting too long (stuck) when getting a large ArrayBuffer content with GM_xhr (#279) Not fixed. With TM BETA 4.12.6123: lags, but file is downloaded. With TM BETA 4.12.6123: no lags, but file isnotdownloaded. At all. Frommy issue: // ==UserScript==//@nameIssue 1107 (GM.xmlHttpRequest, 104 MB, gfycat.com)//@namespaceIssues//@matchhttps://github.com/Tampermonkey/tampermonkey/issues/1107//@matchhttps://example.com///@grantGM.xmlHttpRequest//@connectgiant.gfycat.com// ==/UserScript==!asyncfunction(){console.log(&quot;downloading (104 MB)...&quot;);constresponse=awaitnewPromise((resolve,reject)=&gt;{GM.xmlHttpRequest({method:&quot;get&quot;,url:&quot;https://giant.gfycat.com/ConfusedRecentGuppy.mp4&quot;,// 104 MBresponseType:&quot;blob&quot;,onload:resolve,onerror:reject,});});console.log(&quot;response:&quot;,response);const{response:blob}=response;downloadBlob(blob,&quot;ConfusedRecentGuppy.mp4&quot;);}();functiondownloadBlob(blob,name){constanchor=document.createElement(&quot;a&quot;);anchor.setAttribute(&quot;download&quot;,name||&quot;&quot;);anchor.href=URL.createObjectURL(blob);anchor.click();} － You are receiving this because you authored the thread. Reply to this email directly,view it on GitHub, orunsubscribe.

[AlttiRi]

I get the Uncaught (in promise) TypeError: Failed to execute 'createObjectURL' on 'URL': Overload resolution failed. exception.
Because of blob is undefined.

Chrome 87.0.4280.88. Even with all extensions are disabled.
On Virtual Machine with Win10 1809 the same result is.

It only downloads the file in the background script.

[AlttiRi]

Well, it works fine on example.com, but not here (github.com) now (due to CSP).

VM works even here. The old TM version worked here too, but with bugs described earlier.

---

VM333:3 Refused to connect to 'blob:https://github.com/ad885423-8cda-40c9-ae55-adfee981afc1' because it violates the following Content Security Policy directive: "connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events wss://alive.github.com".

[ccloli]

Yep, I changed my test script to make it runs on GitHub, and get the same CSP error. I tested it with ViolentMonkey, and it works fine, seems it bypass the CSP limit.

That reminds me another userscript management extension named Guerilla Scripting that works on Pale Moon browser. Well the current version of Tampermonkey works better than that, since Guerilla Scripting doesn't solve the cross origin problem, so when you're trying to call GM_xmlhttpRequest with arraybuffer, it'll throw a cross origin error. So I guess that is why Tampermonkey uses an ugly way that receives binary data as string, then convert it to ArrayBuffer or Blob which introduce performance problem.

[derjanb]

Well, it works fine on example.com, but not here (github.com) now (due to CSP).

I see. Thanks.

[derjanb]

Finally... 4.12.6125 (so at least hopefully 🙈😁)

[ccloli]

Tested with Chrome 89.0.4378.0 canary with Tampermonkey BETA v4.12.6125, each file from my script can be downloaded successfully, and the file hashes are matched. 🥳