Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ml5 errors when loading two different models simultaneously #69

Open
ziyuan-linn opened this issue Dec 13, 2023 · 3 comments
Open

ml5 errors when loading two different models simultaneously #69

ziyuan-linn opened this issue Dec 13, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@ziyuan-linn
Copy link
Member

Hi all,

Thank you @Tinazz333 for discovering and letting me know about the issue below.

When loading handpose and facemesh (and probably other models as well) simultaneously, a runtime error will occur:

RuntimeError: Aborted(Module.arguments has been replaced with plain arguments_ (the initial value can be provided on Module, but after startup the value is only looked for on a local variable of that name))

The following code in p5 will trigger this error:

let handpose;
let facemesh;
preload() {
  handpose = ml5.handpose();
  facemesh = ml5.facemesh();
}

A trick that seems to fix the problem is forcing the models to load one by one like this:

let handpose;
let facemesh;
preload() {
  handpose = ml5.handpose(() => {
    facemesh = ml5.facemesh();
  });
}

However, this still causes the browser to throw a bunch of warnings:

The kernel '...' for backend 'webgpu' is already registered

@ziyuan-linn ziyuan-linn added the bug Something isn't working label Dec 13, 2023
@ziyuan-linn
Copy link
Member Author

Upon further investigation, it appears that the issue is caused by mediapipe runtime. I experimented using tfjs runtime using the code below and the error went away.

preload() {
  handpose = ml5.handpose({ runtime: "tfjs" });
  facemesh = ml5.facemesh({ runtime: "tfjs" });
}

I came across this thread on the mediapipe repo discussing a similar issue. The suggested solution involves loading the models one by one.

"currently the initializers don't play nicely together. But you shouldn't need anything complicated-- just adding await before the .initialize() calls (specifically await selfieSegmentation.initialize()) should be sufficient."

One potential solution for us is to implement a queue within ml5 for model initialization calls using the mediapipe runtime. This queue would handle the loading of models one by one in a synchronous manner.

I'd appreciate any thoughts or further insights.

@alanvww
Copy link
Member

alanvww commented Apr 11, 2024

I recently worked on a project and encountered the same problem. I didn't notice that this can be solved by switching to another runtime, but here is my solution:

async function modelLoader() {
  // Create a function that returns a promise for loading faceMesh
  const loadFaceMesh = () =>
    new Promise((resolve) => {
      faceMesh = ml5.faceMesh(options, () => {
        console.log("faceMesh loaded");
        resolve();
      });
    });

  // Create a function that returns a promise for loading handPose
  const loadHandPose = () =>
    new Promise((resolve) => {
      handPose = ml5.handPose({ maxHands: 20 }, () => {
        console.log("handPose loaded");
        resolve();
      });
    });

  // Await the promises in sequence to ensure proper order of execution
  await loadFaceMesh();
  await loadHandPose();
}

function preload() {
  modelLoader();
}

I would definitely like to see if the library itself can have some built-in logic for solving this problem.

@eyaler
Copy link

eyaler commented Apr 13, 2024

small typo in:

let handpose;
let facemesh;
preload() {
  handpose = ml5.handpose(() => {
    facemesh = ml5.facemesh();
  });
}

should be:

preload() {
  handpose = ml5.handPose(() => {
    facemesh = ml5.faceMesh();
  });
}  

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants