Reducing memory copies on the Web

[tidoust]

@dontcallmedom and I organized a TPAC breakout session on memory copies following discussions during the workshop on Web and Machine Learning (see w3c/machine-learning-workshop#93).

Discussions during the breakout session revealed the need to create a place for people from to document pain points from different perspectives (streams, WebGPU, WebAssembly, WebTransport, WebCodecs, etc.) and investigations/ideas that could help reduce the number of memory copies needed in some scenarios.

@sideshowbarker created the WICG/reducing-memory-copies repository to coordinate the effort. It seems likely that different patterns and solutions will be needed depending on the scenarios. No direct work on a technical solution is expected at this stage.

[samuelweiler]

repo appears to have had no activity for 16 mo. propose to close?

[tidoust]

The topic itself is still very current. One of the main scenarios that triggers discussions on memory copies is media processing in real-time because 30-60 frames per second of decoded video data takes a lot of memory. Other scenarios that involve processing of lots of data may be affected too.

The topic was discussed during the W3C/SMPTE workshop on Professional Media Production in November 2021:

• WebCodecs: Memory copies
• WebAssembly: Reducing copies across memory boundaries

The topic is also under discussion at technologies/group boundaries. Some known boundaries:

• WebAssembly/JS: see above for leads explored in the WebAssembly group
• WebCodecs/WebGPU: see e.g. Investigation: Import VideoFrame from WebCodec to WebGPU gpuweb/gpuweb#1380
• WebGPU/WebNN: see e.g. Investigation: how WebNN / WebGPU interop could be happening gpuweb/gpuweb#2500

The WICG repository is an attempt to coordinate cross-group discussions and align on directions. It has indeed not gained much traction until now.

One of the directions @dontcallmedom and I are exploring is finding individuals/companies who could perhaps inform quantified discussions on memory copies through prototypes.

[pchampin]

Slightly related: @BruJu, a master student that I supervised, worked on an RDF implementation in Web Assembly, and had to work around some performance issues related to memory copies btw WASM and JS. We published a paper about that at the Resource Track of ESWC: https://hal.archives-ouvertes.fr/hal-03262817