Support for esm.ts using `node --loader ts-node/esm`

[monyarm]

I use ts for my gulpfile, but some of the plugins i use (gulp-imagemin specifically), are esm.
So having a way to use both would be useful, as otherwise when one tries to run gulp, they get something like this:

[18:15:57] Requiring external module ts-node/register
TypeError: Unknown file extension ".ts" for /home/monyarm/Documents/gulp-gameoptimizer/gulpfile.ts
    at new NodeError (node:internal/errors:371:5)
    at Loader.defaultGetFormat [as _getFormat] (node:internal/modules/esm/get_format:71:15)
    at Loader.getFormat (node:internal/modules/esm/loader:105:42)
    at Loader.getModuleJob (node:internal/modules/esm/loader:243:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Loader.import (node:internal/modules/esm/loader:177:17)
    at importModuleDynamicallyWrapper (node:internal/vm/module:437:15) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

[phated]

That sucks.

This is a complex feature to add and is on our roadmap for rechoir, but we probably won't have time to work on it for awhile.

I'll move this over to the rechoir library, as it needs to be implemented in the loader.

[phated]

I just spent hours digging into this, and my general sense is that esm loaders in nodejs are an absolute shitshow right now. They've recently changed their loader API, so a lot of stuff has to shim their internal loader logic, but then I noticed that ts-node had to wholesale copy-paste node internals into their codebase to support their module loader. Check this out: https://github.com/TypeStrong/ts-node/blob/main/dist-raw/README.md

Anyway, I dove down that rabbit hole because the most efficient way for rechoir to support esm loaders would be to bootstrap our own loader on startup that proxies through to everyone else's loaders. However, once I discovered that we'd have to copy all the internals from node to do this "right", I decided to hack on a much worse performing solution and came up with something that works:

const { spawn } = require('child_process');

const argv = process.argv

async function main() {
  try {
    require('ts-node/register')
    return require('./bar.ts');
  } catch (err) {
    if (err.code === 'ERR_REQUIRE_ESM') {
      try {
        return await import('./bar.ts');
        console.log(mod);
      } catch (err) {
        if (err.code === 'ERR_UNKNOWN_FILE_EXTENSION') {
          var child = spawn(argv[0], [
            '--loader', 'ts-node/esm',
            ...argv.slice(1)
          ], { stdio: 'inherit' });
          child.on('exit', function (code, signal) {
            process.on('exit', function () {
              /* istanbul ignore if */
              if (signal) {
                process.kill(process.pid, signal);
              } else {
                process.exit(code);
              }
            });
          });
        } else {
          throw err
        }
      }
    } else {
      throw err
    }
  }
}

main()
  .then(mod => console.log(mod))
  .catch(err => {
    process.nextTick(() => { throw err })
  })

Essentially what is happening here is that we are assuming require.extensions is the default loading situation (since it is the most battle tested, and you can't do this in reverse from ESM) and load the ts-node/register module for commonjs modules. Then we try to load the file, but if it errors with the ERR_REQUIRE_ESM code, we know that it tried to resolve as an ESM module. In that scenario, we try to load the file with a dynamic import() and if that fails with the code ERR_UNKNOWN_FILE_EXTENSION it means that there was no loader registered for it and we need to reboot the entire node instance with the loader and start the whole process over.

The reason this is pretty insane is that it'll be double registering loaders (for commonjs and esm) for every esm loader you try to use. Additionally, multiple ESM loaders will cause multiple child processes to be spawned (which then reloads all modules from the start).

[phated]

I'm not a huge fan of this, but it solves the problem and makes the actual loaders other people's problem. I'll think on it a bit more.

[phated]

For reference, I used concepts from gulp-cli, liftoff and https://github.com/lukeed/loadr for the above solution.

[JusticHentai]

I have the same problem, did you solve it?

[phated]

@JusticHentai No, and this will be punted until post-v5 of gulp because nodejs support of loaders is so shitty.

[cspotcode]

Just commenting to say that I'm following along with issues like this across the ecosystem, and am always open to discussing ways that tooling and loader authors can shim our way around node's shortcomings.

I have this idea that we might devise a pass-through loader which does nothing on its own, but which allows tools such as yours to install hooks at runtime as necessary.

Roughly, it would look like this:

Node is invoked like this:

node --loader @cspotcode/multiloader ./whatever-tool-or-entrypoint.js

If a tool realizes at runtime that it wants to install ts-node's hooks:

await process[Symbol.for('multiloader-api')].add('ts-node/esm');
await import('./some-ts-file.ts')

Theoretically someone could add NODE_OPTIONS='--loader /abs/path/to/@cspotcode/multiloader' to their shell profile, and then node would have this runtime loader-installation API. I say theoretically, because I would not recommend that users do this, but it's a nice thought experiment, and I think it makes the case that node should support this natively.

[phated]

@cspotcode I believe something like a "multiloader" would make the most sense sense, but your current example only works for dynamic imports. We'd want to be able to prepare the multiloader at launch with flags. Maybe that would be something like:

node --loader multiloader --require ts-node/esm` 

[cspotcode]

That is actually possible today, with, for example: node --loader @cspotcode/multiloader/compose?yaml-loader,ts-node/esm

The runtime API I described in my previous comment is not yet implemented, but composing multiple loaders at startup is implemented today.

https://github.com/cspotcode/multiloader

The end-goal is a future where tools such as yours do not need to attempt to spawn a child process at all. When they decide that a loader is necessary, they can install it at runtime, the same way we do for CJS loaders.

EDIT: fixed a typo in the invocation syntax above

[phated]

For gulp, I believe we'll always need to respawn because node doesn't have good APIs for allowing us to pass other nodejs flags through, but I understand that having the programmatic API would be useful for other applications.