v4 third party extension building

[pkra]

I know this might technically belong in mathjax-docs but I thought it might be more useful to have it in the main tracker.

I'm in the process of upgrading some extensions to work with v4, e.g., pkra/mathjax-img#24, AmerMathSoc/mathjax-unicode-math#27.

Adjusting to generate ESM for server-side use was straight forward. But looking through mathjax-src, I don't understand how the build process for clients-side extension has to be adjusted.

The advice from https://docs.mathjax.org/en/latest/web/webpack.html#the-webpack-configuration-1 doesn't seem to work anymore. mathjax-full/components/bin/makeAll expects various CLI options now but I couldn't work out how to adjust.

If you had time for some short pointers, I'd be grateful.

[I'd prefer not to generate CJS but of course I'm happy to do the full adoption of mathjax-src's tsconfig setup if that's easier.]

[dpvc]

Yes, the documentation hasn't been update for the changes to the configuration files for components. There is information about it in the ESM pull request, in the section on "The new component control files". The main change is that the individual build.json, copy.json, and webpack.config.js files have all been merged into a single config.json file with sections for each of the three actions. These sections basically are just the data that would have been in the original .json file, with the webpack section being the values that would have been passed to the PACKAGE() function in the original webpack.config.js file. You can look at the config.json files in the components/mjs directory for some examples.

[pkra]

I've had time to take stab at it.

For the mathjax-unicode-math extension, I've tried the following config.json

{
  "build": {
    "id": "unicode-math",
    "component": "js/unicode-math",
    "targets": ["js"]
  },
  "webpack": {
    "name": "unicode-math",
    "libs": [
      "components/src/input/tex-base/lib",
      "components/src/core/lib"
    ]
  }
}

(Even though the build block seemed superfluous - see below.)

Then ran

$ npx webpack --env dir=. --env bundle=../../browser -c node_modules/mathjax-full/components/webpack.config.mjs 

First, I got an error that webpack couldn't find the extension. Turns out, it was looking for it in the main directory. While trying to work out where the path is passed along, I noticed that the build block didn't seem to do anything (for me). If I tried to run webpack from the ./js folder (where the ESM build lives), other things went wrong.

To get a little further I simply copied the ESM build to the root. That got me a build which looked ok.

However, using the result in the browser gave me silent failures (well, not completely silent -- there was a warning about a missing version info).

Here's the resulting build I got https://gist.github.com/pkra/5ec245db7831c265267518dbab0c1fb3 (which compared to https://github.com/AmerMathSoc/mathjax-unicode-math/tree/main/browser seems to include more mathjax code).

So if you have the time my first question would be: is there a way to specify the source of the ESM module?
The second question would be: what am I doing wrong? 😓

[dpvc]

I noticed that the build block didn't seem to do anything (for me).

That is only used by the components/bin/build command (which is called by components/bin/makeAll). Since you are calling webpack directly, you don't get a build action. I'm not sure the values are correct in the build block in any case, as id should be the name used in the loader.load array (e.g., [custom]/unicode-math), and component is the location within the bundle directory for the component.

You shouldn't call npx webpack by hand any longer, in most circumstances. The components/bin/makeAll command should be used instead, which will call components/bin/pack which in turn does the npx webpack call with the proper arguments.

The build tools expect something like

component/
  config.json
  unicode-math.js
js/
  unicode-math.js
ts/
  unicode-math.ts

If you want users to define a path ams to be used for your extension, then I think you want config.json to be

{
  "build": {
    "id": "[ams]/unicode-math",
    "component": "input/tex/extension/unicode-math",
    "targets": ["js"]
  },
  "webpack": {
    "name": "unicode-math",
    "libs": [
      "components/src/input/tex-base/lib",
      "components/src/core/lib"
    ]
  }
}

and compoments/unicode-math.js to be

import './lib/unicode-math.js';

Then you should be able to use

node_modules/mathjax-full/components/bin/makeAll --mjs component

This is from memory, so I think it works but didn't test it.

It looks like you are wanting to put the webpacked file into a browser directory (rather than the bundle directory that makeAll expects to use). There is no command-line option for specifying that, but you could do

cd component
../node_modules/mathjax-full/components/bin/build mjs
../node_modules/mathjax-full/components/bin/pack mjs browser

instead, which I think will do the trick. Or you could rename the bundle folder to browser after the fact.

The reason you are getting the "no version information" message is because the version identification is added to components/lib/unicode-math.js during the build process (which you aren't currently doing), so MathJax is warning you that that hasn't been done.

As for why it isn't working, you identify that more of MathJax is being included, and that may mean that your extension isn't hooking into the existing MathJax structure but rather a separate one, so it not making itself available to the MathJax that is actually running. When you use the makeAll or pack command, you should get a nice list of files that are being included in the packed file, and should only see your unicode-math files, some component files from the directories in the libs array, and the MathJax version file. For example, when I build the [tex]/action component, I see

Building /mjs/input/tex/extensions/action
    Processing:
      input/tex/action/ActionConfiguration.ts
      action.ts
    
Webpacking /mjs/input/tex/extensions/action
    action.js (1.59 KB)
      [components]/mjs/core/lib/components/global.js (0.36 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Configuration.js (0.21 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexParser.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TokenMap.js (0.45 KB)
      [components]/mjs/input/tex-base/lib/input/tex/base/BaseMethods.js (0.13 KB)
      [components]/mjs/input/tex/extensions/action/action.js (2.88 KB)
      [components]/mjs/input/tex/extensions/action/lib/action.js (0.46 KB)
      [mathjax]/mjs/components/version.js (0.07 KB)
      [mathjax]/mjs/input/tex/action/ActionConfiguration.js (1.1 KB)

If you are seeing a lot of [mathjax] references, then that means something isn't configured properly. Here the last two component files (action.js and lib/action.js) and the ActionConfiguration.js will be your unicode-math files, and will probably be listed above the first component file, perhaps with a full path name, or with ./ prefix.

See if any of that helps you out.

[pkra]

Thank you! I'll report back when I've tried this.

[pkra]

A quick update on trying the suggested changes.

I created a components folder, put the config.json and a pure-import unicode-math.js in there.

I changed import './lib/unicode-math.js'; to import '../js/unicode-math.js'; (which I think was just a typo)

Next, $ node_modules/mathjax-full/components/bin/makeAll --mjs components gave me the following error

Building ./components
node:internal/fs/utils:346
    throw err;
    ^

Error: ENOENT: no such file or directory, stat '/tmp/mathjax-unicode-math/node_modules/mathjax-full/ts/js'
    at Object.statSync (node:fs:1582:3)
    at processList (/tmp/mathjax-unicode-math/node_modules/mathjax-full/components/bin/build:105:23)
    at Object.<anonymous> (/tmp/mathjax-unicode-math/node_modules/mathjax-full/components/bin/build:398:1)
    at Module._compile (node:internal/modules/cjs/loader:1246:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1300:10)
    at Module.load (node:internal/modules/cjs/loader:1103:32)
    at Module._load (node:internal/modules/cjs/loader:942:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  errno: -2,
  syscall: 'stat',
  code: 'ENOENT',
  path: '/tmp/mathjax-unicode-math/node_modules/mathjax-full/ts/js'
}

Node.js v19.5.0
    Command failed: node '/tmp/mathjax-unicode-math/node_modules/mathjax-full/components/bin/build' 'mjs'
node:internal/fs/utils:346
    throw err;
    ^

Error: ENOENT: no such file or directory, stat '/tmp/mathjax-unicode-math/node_modules/mathjax-full/ts/js'
    at Object.statSync (node:fs:1582:3)
    at processList (/tmp/mathjax-unicode-math/node_modules/mathjax-full/components/bin/build:105:23)
    at Object.<anonymous> (/tmp/mathjax-unicode-math/node_modules/mathjax-full/components/bin/build:398:1)
    at Module._compile (node:internal/modules/cjs/loader:1246:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1300:10)
    at Module.load (node:internal/modules/cjs/loader:1103:32)
    at Module._load (node:internal/modules/cjs/loader:942:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  errno: -2,
  syscall: 'stat',
  code: 'ENOENT',
  path: '/tmp/mathjax-unicode-math/node_modules/mathjax-full/ts/js'
}

Node.js v19.5.0

Webpacking ./components
    unicode-math.js (100.82 KB)
      ../js/unicode-math.js (97.97 KB)
      [components]/mjs/core/lib/core/MmlTree/Attributes.js (0.12 KB)
      [components]/mjs/core/lib/core/MmlTree/MmlNode.js (0.59 KB)
      [components]/mjs/core/lib/core/Tree/Node.js (0.14 KB)
      [components]/mjs/core/lib/util/Entities.js (0.24 KB)
      [components]/mjs/core/lib/util/FunctionList.js (0.08 KB)
      [components]/mjs/core/lib/util/Options.js (0.67 KB)
      [components]/mjs/core/lib/util/PrioritizedList.js (0.09 KB)
      [components]/mjs/core/lib/util/lengths.js (0.34 KB)
      [components]/mjs/core/lib/util/string.js (0.33 KB)
      [components]/mjs/input/tex-base/lib/input/tex/MapHandler.js (0.17 KB)
      [components]/mjs/input/tex-base/lib/input/tex/NodeUtil.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/ParseUtil.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Symbol.js (0.1 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Tags.js (0.26 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexConstants.js (0.09 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexError.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexParser.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/base/BaseItems.js (1.02 KB)
      [mathjax]/mjs/core/MmlTree/MmlNode.js (17.91 KB)
      [mathjax]/mjs/input/tex/Configuration.js (5.69 KB)
      [mathjax]/mjs/input/tex/ParseMethods.js (4.05 KB)
      [mathjax]/mjs/input/tex/SymbolMap.js (3.69 KB)
      [mathjax]/mjs/input/tex/base/BaseMethods.js (39.76 KB)
      unicode-math.js (173.62 KB)

Removing the build block from config.json avoided the errors and gave the same list of files.

Next I couldn't find where the result was written -- makeAll complained that --env is not a valid option. I eventually found the result as node_modules/mathjax-full/bundle/unicode-math.js.

It was identical to the result in my previous attempt.

[dpvc]

I changed import './lib/unicode-math.js'; to import '../js/unicode-math.js'; (which I think was just a typo)

No, that was not a typo. The lib directory is created by the build process, and it is what includes the version check and the creation of the needed MathJax._ subtree for your extension. The original js file is loaded by the one in lib.

I guess I forgot some of the configuration parameters that you need when you are working outside of the MathJax directory tree itself. Try using

{
  "build": {
    "id": "[ams]/unicode-math",
    "component": "input/tex/extension/unicode-math",
    "js": "../js",
    "ts": "../ts",
    "targets": ["unicode-math.ts"]
  },
  "webpack": {
    "name": "unicode-math",
    "js": "../js",
    "libs": [
      "components/src/input/tex-base/lib",
      "components/src/core/lib"
    ],
    "dist": ".."
  }
}

I have added js options to both blocks in order to tell MathJax where your .js files are (rather than the default node_modules/mathjax-full/js). Similarly, I set the ts directory to ../ts (it would have been set automatically if the js directory were mjs or cjs; I'll have to fix that). The build action uses the .ts files, not the .js ones, so the targets array should be to the .ts files, and they are relative to the ts directory, so when it was js that is why build was looking for node_modules/mathjax-full/ts/js (the target was a file or directory called js in the default ts directory). Finally, I added a dist option to the webpack block to tell webpack where to put the resulting file (rather than in the MathJax distribution, where you found it).

I tested this out this time, and there is one more thing. It turns out you need to change the imports from mathjax-full/js to mathjax-full/mjs, as the remapping in the webpack configuration file doesn't seem to redirect everything properly. That is why you were getting some [mathjax] references that should have been from [components]. I will need to see if I can fix that.

In any case, see if that doesn't do the trick for you.

[pkra]

Thank you so much for your patience and the additional explanation!

The build process now works for me as you describe (including tweaking the dist property to get keep the folder structure).

The result looks similar to what I had build earlier but with noticeable changes. (As before, it's still 20kb larger than the v3 build.)

Unfortunately, the build still does not work for me in the browser. Now I get

Uncaught ReferenceError: Cannot access 'h' before initialization
    at Object.r (unicode-math.js:1:183)

with h, I think, being MathJax._.input.tex.Symbol.Symbol (the compilate has const f=MathJax._.input.tex.Symbol,h=f.Symbol,x=f.Macro;).

I've updated the build at https://gist.github.com/pkra/5ec245db7831c265267518dbab0c1fb3 with what I'm currently getting.

[pkra]

Oh, for completeness, the file generated at components/lib/unicode-math.js reads:

import {combineWithMathJax} from '../../node_modules/mathjax-full/mjs/components/global.js';
import {VERSION} from '../../node_modules/mathjax-full/mjs/components/version.js';

import * as module1 from '../../js/unicode-math.js';

if (MathJax.loader) {
  MathJax.loader.checkVersion('[ams]/unicode-math', VERSION, 'tex-extension');
}

combineWithMathJax({_: {
  "unicode-math": module1
}});

(PS: Would you suggest to commit that file or git-ignore it?)

[dpvc]

Unfortunately, the build still does not work for me in the browser.

You have correctly identified the issue as coming from MathJax._.input.tex.Symbol.Symbol. It turns out that webpack handles exports in ESM modules differently than it did module.exports in CommonJS modules. The former create constants that are global to the webpacked file, while the latter used a separate object for each file that collected the exported variables. Because Symbol is already a global javascript object, the exporting of Symbol in input/tex/Symbol.js overrides the global symbol in ESM modules whereas it didn't in the CommonJS ones. Some initialization code that webpack includes at the top of the packed file includes references to the original global Symbol object, which occur before the imported const Symbol = MathJax._.input.tex.Symbol.Symbol that is later in the file (the magnification changes both references to h in your case). This causes the javascript compiler to complain about the use of the constant h before it is defined.

I had made changes to input/tex/Symbol.ts that avoided this problem, but it turns out that these only worked if the original Symbol.ts is imported, but when it is replaced by the lib version in the components directory (that allows sharing of objects across components), it doesn't work. This was originally reported in #3072. I tried hard to get a work-around, but the only thing I could come up with that worked in the ESM webpacking was to change the name of Symbol in input/tex/Symbol.ts to Token. This was done in mathjax/MathJax-src#982.

But you don't seem to be importing Symbol.js, and you say the file is larger than your original version, so I suspect you have not changed the mathjax-full/js imports to mathjax-full/mjs as I recommended at the end of my previous comment. Without this change, running makeAll in your mj4 branch produces

Building ./components
    Processing:
      unicode-math.ts
      unicode-math.ts
    
Webpacking ./components
    unicode-math.js (101.57 KB)
      ../js/unicode-math.js (97.97 KB)
      [components]/mjs/core/lib/components/global.js (0.36 KB)
      [components]/mjs/core/lib/core/MmlTree/Attributes.js (0.12 KB)
      [components]/mjs/core/lib/core/MmlTree/MmlNode.js (0.59 KB)
      [components]/mjs/core/lib/core/Tree/Node.js (0.14 KB)
      [components]/mjs/core/lib/util/Entities.js (0.24 KB)
      [components]/mjs/core/lib/util/FunctionList.js (0.08 KB)
      [components]/mjs/core/lib/util/Options.js (0.67 KB)
      [components]/mjs/core/lib/util/PrioritizedList.js (0.09 KB)
      [components]/mjs/core/lib/util/lengths.js (0.34 KB)
      [components]/mjs/core/lib/util/string.js (0.33 KB)
      [components]/mjs/input/tex-base/lib/input/tex/MapHandler.js (0.17 KB)
      [components]/mjs/input/tex-base/lib/input/tex/NodeUtil.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/ParseUtil.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Symbol.js (0.1 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Tags.js (0.26 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexConstants.js (0.09 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexError.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexParser.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/base/BaseItems.js (1.02 KB)
      [lib]/lib/unicode-math.js (0.38 KB)
      [lib]/unicode-math.js (174.48 KB)
      [mathjax]/mjs/components/version.js (0.07 KB)
      [mathjax]/mjs/core/MmlTree/MmlNode.js (17.91 KB)
      [mathjax]/mjs/input/tex/Configuration.js (5.69 KB)
      [mathjax]/mjs/input/tex/ParseMethods.js (4.05 KB)
      [mathjax]/mjs/input/tex/SymbolMap.js (3.69 KB)
      [mathjax]/mjs/input/tex/base/BaseMethods.js (39.76 KB)

and you can see that this is including pieces of [mathjax] that it shouldn't (these files hold be coming from [components]/mjs/input/tex-base/lib or [components]/mjs/core/lib). In particular, the [mathjax]/mjs/input/tex/SymbolMap.js is forcing [components]/mjs/input/tex-base/lib/input/tex/Symbol.js to be loaded, which is causing the problem I describe above.

Once I change the mathjax-full/js imports to mathjax-full/mjs references, I get

Building ./components
    Processing:
      unicode-math.ts
      unicode-math.ts
    
Webpacking ./components
    unicode-math.js (70.08 KB)
      ../js/unicode-math.js (97.97 KB)
      [components]/mjs/core/lib/components/global.js (0.36 KB)
      [components]/mjs/core/lib/core/MmlTree/MmlNode.js (0.59 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Configuration.js (0.21 KB)
      [components]/mjs/input/tex-base/lib/input/tex/ParseMethods.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/SymbolMap.js (0.45 KB)
      [components]/mjs/input/tex-base/lib/input/tex/base/BaseMethods.js (0.13 KB)
      [lib]/lib/unicode-math.js (0.38 KB)
      [lib]/unicode-math.js (100.31 KB)
      [mathjax]/mjs/components/version.js (0.07 KB)

which looks correct to me. I have not run the extension, but it doesn't include Symbol.js, so should not produce the error you are currently getting. It is also 13KB smaller than your original. So that looks good too.

Finally, you may want to add

"prefix": "input/tex/extensions",

to the webpack block of your config.json file so that your exports are put in MathJax._.input.tex.extensions['unicode-math'] rather than MathJax._['unicode-math'].

---

Would you suggest to commit that file or git-ignore it?

We don't include the generated files (the mjs and cjs directories, the bundle directory, or the lib directories) in the MathJax GitHub repositories, but they are included in the npm packages. The idea is that the GitHub repository doesn't need to track the changes to the generated files, and that people will use the npm package rather than the GitHub repo to get the distribution copy that they actually use. But if you are trying to make the GitHub repo be the same as the npm package (if you make one), then you would probably want to include it there if you want people to be able to use the component from source code (which would be components/unicode-math.js). A reason to do with is that it would mean that the MathJax._ values are set so that the components object can be shared by other components (probably unnecessary in your case), and so that the version information is available for the version check (avoids a version warning that you would get otherwise).

[pkra]

Oh! I'm very sorry for missing that last bit about changing from js to mjs.

It looks like this fixes things (as you wrote) -- I'll test more and let you know if I find any other issues.

Thank you again for your patience and help!

[luavreis]

Edit: after some debugging, I realized the error was caused because we now need to export the object returned from Configuration.create... it works now.

---

I've tried to follow https://github.com/pkra/mathjax-img as close as possible to update my extension but I'm still getting ENOENT errors... here is my attempt: luavreis/xparsejax#1

$ ./node_modules/mathjax-full/components/bin/makeAll components
Building ./components
node:internal/fs/utils:350
    throw err;
    ^

Error: ENOENT: no such file or directory, open 'lib/xparsejax.js'
    at Object.openSync (node:fs:603:3)
    at Object.writeFileSync (node:fs:2324:35)
    at processGlobal (/tmp/xparsejax/node_modules/mathjax-full/components/bin/build:300:6)
    at Object.<anonymous> (/tmp/xparsejax/node_modules/mathjax-full/components/bin/build:399:1)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47 {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: 'lib/xparsejax.js'
}

Node.js v18.17.1
    Command failed: node '/tmp/xparsejax/node_modules/mathjax-full/components/bin/build' 'mjs'
node:internal/fs/utils:350
    throw err;
    ^

Error: ENOENT: no such file or directory, open 'lib/xparsejax.js'
    at Object.openSync (node:fs:603:3)
    at Object.writeFileSync (node:fs:2324:35)
    at processGlobal (/tmp/xparsejax/node_modules/mathjax-full/components/bin/build:300:6)
    at Object.<anonymous> (/tmp/xparsejax/node_modules/mathjax-full/components/bin/build:399:1)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47 {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: 'lib/xparsejax.js'
}

Node.js v18.17.1

Webpacking ./components
    xparsejax.js (3.02 KB)
      [build]/xparsejax.js (6.84 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Configuration.js (0.21 KB)
      [components]/mjs/input/tex-base/lib/input/tex/ParseUtil.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TexError.js (0.07 KB)
      [components]/mjs/input/tex-base/lib/input/tex/Token.js (0.1 KB)
      [components]/mjs/input/tex-base/lib/input/tex/TokenMap.js (0.45 KB)
      [js]/xparsejax.js (5.92 KB)

[dpvc]

@lucasvreis, I'm glad you worked out a solution.

The LIB directory is usually created during the processList() call, and you are right that that occurs due to export commands in the processed files. If there is no export, no lib file is made, and the directory is never re-created. So your adding the export did the trick. It's not that the configuration has to be exported, but something did need to be exported in order to trigger the creation of the directory.

On the other hand, if there are no exports, there is no need for a build section in the config.json file at all, as the build is only needed to allow the exported object to be shared with other components that might need them. But perhaps it would be best to do a makeDir(LIB) as you suggested originally, just in case someone does include one unnecessarily.

[dpvc]

Great! Thanks for the confirmation.