Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub runner cannot send fetch with node, failing with IPv6 DNS error UND_ERR_CONNECT_TIMEOUT #3213

Open
undergroundwires opened this issue Mar 20, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@undergroundwires
Copy link

undergroundwires commented Mar 20, 2024

Describe the bug
Sending more than one (sometimes two or three) requests using fetch results in DNS errors.

To Reproduce
I created a repository to reproduce this in as minimal way as possible:
https://github.com/undergroundwires/node-fetch-ipv6

I run fetch on using vite test runner on node and without any test runner. I get same results.
I realize that the first two requests are successful, I start getting the error after the second or third one.

See the test: test file

Second test (multiple fetch in order) fails, and all run fine locally. I run tests on default GitHub runners using both latest node 20 and 18. I get same errors:

Expected behavior
The requests are being sent fine, this was the case before.

Runner Version and Platform

The job runs on ubuntu-latest, see the repository linked in To Reproduce section above.

What's not working?

I get UND_ERR_CONNECT_TIMEOUT errors which is related to IPv6 issues in node.
Related issue for node: nodejs/node#41625,
Related issue for node fetch: nodejs/undic#1531
Related issue for GitHub runners not being able to do IPv6: actions/runner-images#668, #3138
image

Job Log Output

RUN  v1.4.0 /home/runner/work/node-fetch-ipv6/node-fetch-ipv6

stdout | vite-test.spec.ts > fetch in order
Fetching:  https://web.archive.org/web/20221029145712/https://kb.mozillazine.org/Downloads.rdf

stdout | vite-test.spec.ts > fetch in order
Fetching:  https://web.archive.org/web/20240120213614/https://techcommunity.microsoft.com/t5/windows-it-pro-blog/group-configuration-search-highlights-in-windows/ba-p/3263989

 ❯ vite-test.spec.ts  (2 tests | 1 failed) 22219ms
   ❯ vite-test.spec.ts > fetch in order
     → fetch failed

⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯

 FAIL  vite-test.spec.ts > fetch in order
TypeError: fetch failed
 ❯ vite-test.spec.ts:26:22
     24|   for (const url of urls) {
     25|     console.log('Fetching: ', url);
     26|     const response = await fetch(url, { method: 'HEAD'});
       |                      ^

Error: TypeError: fetch failed
 ❯ vite-test.spec.ts:26:22

Caused by: ConnectTimeoutError: Connect Timeout Error
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { code: 'UND_ERR_CONNECT_TIMEOUT' }

     27|     await sleep(10000);
     28|     expect(response.status).to.equal(200);

Caused by: Caused by: ConnectTimeoutError: Connect Timeout Error
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { code: 'UND_ERR_CONNECT_TIMEOUT' }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

 Test Files  1 failed (1)
      Tests  1 failed | 1 passed (2)
   Start at  21:21:50
   Duration  22.54s (transform 42ms, setup 0ms, collect 26ms, tests 22.22s, environment 0ms, prepare 72ms)
@undergroundwires undergroundwires added the bug Something isn't working label Mar 20, 2024
undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 29, 2024
This commit introduces the `force-ipv4` GitHub action to address
connectivity issues caused by the lack of IPv6 support in GitHub
runners. Details:
- actions/runner#3138
- actions/runner-images#668

This change solves connection problems when Node's `fetch` API fails due
to `UND_ERR_CONNECT_TIMEOUT` errors. Details:
- actions/runner-images#9540
- actions/runner#3213

This action disables IPv6 at the system level, ensuring all outging
requests use IPv4. Resolving connectivity issues when running external
URL checks and Docker build checks.

This solution is a temporary workaround until GitHub runners support
IPv6 or Node `fetch` API has a working solution such as Happy Eyeball.
Detais:
- nodejs/node#41625
- nodejs/undici#1531
@undergroundwires
Copy link
Author

Here's my workaround (open-source and documented) that I hope that can help you too:

After days of research and trial/error, this is how I got this working:

  1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
  2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
  3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd

undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 30, 2024
This commit upgrades Node.js version to v20.x in CI/CD environment.

Previously used Node 18.x is moving towards end-of-life, with a planned
date of 2025-04-30. In contrast, Node 20.x has been offering long-term
support (LTS) since 2023-10-24. This makes Node 20.x a stable and
recommended version for production environments.

This commit also configures `actions/setup-node` with the
`check-latest` flag to always use the latest Node 20.x version, keeping
CI/CD setup up-to-date with minimal maintenance.
Details:
- actions/setup-node#165
- actions/setup-node#160

Using Node 20.x in CI/CD environments provides better compatibility with
Electron v29.0 which moves to Node 20.x.
Details:
- electron/electron#40343

This upgrade improves network connection handling in CI/CD pipelines
(where issues occur due to GitHub runners not supporting IPv6).
Details:
- actions/runner#3138
- actions/runner-images#668
- actions/runner#3213
- actions/runner-images#9540

Node 20.x adopts the Happy Eyeballs algorithm for improved IPv6
connectivity.
- nodejs/node#40702
- nodejs/node#41625
- nodejs/node#44731

This mitigates issues like `UND_ERR_CONNECT_TIMEOUT` and localhost DNS
resolution in CI/CD environments:
Details:
- nodejs/node#40537
- actions/runner#3213
- actions/runner-images#9540

Node 20 introduces `setDefaultAutoSelectFamily`, a global function from
Node 19.4.0, enabling better IPv4 support, especially in environments
with limited or problematic IPv6 support.
Details:
- nodejs/node#45777

Node 20.x defaults to the new `autoSelectFamily`, improving network
connection reliability in GitHub runners lacking full IPv6 support.
Details:
- nodejs/node#46790
undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 30, 2024
This commit upgrades Node.js version to v20.x in CI/CD environment.

Previously used Node 18.x is moving towards end-of-life, with a planned
date of 2025-04-30. In contrast, Node 20.x has been offering long-term
support (LTS) since 2023-10-24. This makes Node 20.x a stable and
recommended version for production environments.

This commit also configures `actions/setup-node` with the
`check-latest` flag to always use the latest Node 20.x version, keeping
CI/CD setup up-to-date with minimal maintenance.
Details:
- actions/setup-node#165
- actions/setup-node#160

Using Node 20.x in CI/CD environments provides better compatibility with
Electron v29.0 which moves to Node 20.x.
Details:
- electron/electron#40343

This upgrade improves network connection handling in CI/CD pipelines
(where issues occur due to GitHub runners not supporting IPv6).
Details:
- actions/runner#3138
- actions/runner-images#668
- actions/runner#3213
- actions/runner-images#9540

Node 20.x adopts the Happy Eyeballs algorithm for improved IPv6
connectivity.
- nodejs/node#40702
- nodejs/node#41625
- nodejs/node#44731

This mitigates issues like `UND_ERR_CONNECT_TIMEOUT` and localhost DNS
resolution in CI/CD environments:
Details:
- nodejs/node#40537
- actions/runner#3213
- actions/runner-images#9540

Node 20 introduces `setDefaultAutoSelectFamily`, a global function from
Node 19.4.0, enabling better IPv4 support, especially in environments
with limited or problematic IPv6 support.
Details:
- nodejs/node#45777

Node 20.x defaults to the new `autoSelectFamily`, improving network
connection reliability in GitHub runners lacking full IPv6 support.
Details:
- nodejs/node#46790
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant