circleci config validate failing (failed to decode body: invalid character 'I')

[scottx611x]

Meta:

CircleCI CLI Version:
Observed on both:

• 0.1.25725+63309c6 (release)
• 0.1.25725+63309c6 (homebrew)

Operating System:

• Ubuntu 22.04.1 LTS
• macOS Monterey 12.6.2 (21G320)

CircleCI CLI Diagnostic:

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /home/circleci/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Scott Ouellette.

---

Current behavior:

I've observed this in multiple projects both locally and in CI ssh re-runs

$ circleci config validate
failed to decode body: invalid character 'I' looking for beginning of valueError: config compilation request returned an error: invalid character 'I' looking for beginning of value

Expected behavior:

We should be able to validate our configs

When did this begin / Was this previously working?:

We observed this behavior just this morning. Notably workflows kicked off using said configs that are failing to validate are working as intended

Additional Information:

[Kylmakalle]

I can confirm the issue as well. Submitted a support ticket #131260

My investigations revealed that the most recent versions of CLI send YAML to https://api.circleci.com/api/v2/compile-config-with-defaults endpoint, which returns 401 Invalid token provided. It looks like this endpoint was introduced in #865 (then reverted #865 and then introduced once again #878) and was accidentally secured by some authentication couple of hours ago.

Temporary workarounds

• Do not run circleci config process and circleci config validate commands
• Switch to the older CLI version, like v0.1.23845, which uses https://circleci.com/graphql-unstable endpoint

[Kylmakalle]

I got an update from support. The commands above now work on CircleCI. Unfortunately, they still do not work locally.

UPD: Command works locally. However, It required me to regenerate my stored Personal Access token.

[michaellarocca90]

We are seeing this on version 0.1.23667 and version 0.1.25725 from docker executors within circleci.

[michaellarocca90]

This seems like it should be an incident on CircleCI's end?

[elliotforbes]

Howdy folks!

This issue should now be resolved - the underlying reason why this issue occurred was two-fold:

• We've recently migrated off an unstable graphQL service internally and started communicating directly with v2 APIs. For historical reasons the Circle-Token header was sent as an empty string to this old API endpoint.
• In the past few days, we've been exploring a new API UX improvement wherein we will return 401 unauthorized if an invalid request is sent with any API request. This is a fairly big behavioural change which should help folks investigate and identify why they're not able to make API requests with invalid tokens far quicker so it's a huge win.

This combination then meant that, when the auth changes were rolled out, the CLI was effectively sending an invalid authorization token "".

We've patched this for now and the team that owns this project will be working on a fix for the CLI so that it does the right thing going forward.

Please feel free to reopen this issue should this regression reappear!

[blimmer]

@elliotforbes I'm experiencing this issue this morning:

> date
Fri Apr 14 09:45:10 MDT 2023
> circleci config validate
failed to decode body: invalid character 'I' looking for beginning of valueError: config compilation request returned an error: invalid character 'I' looking for beginning of value
> circleci version
0.1.25725+63309c6 (homebrew)

[elliotforbes]

@blimmer - Can you confirm if you are running on a server instance of CircleCI and not on the cloud?

If so, you may need to update your version of server such that the API that supports this no longer sends invalid JSON back to the CLI.

[robotadam]

the team that owns this project will be working on a fix for the CLI so that it does the right thing going forward.

Is there an issue we can watch for fixing the CLI? It seems to work with a personal access token set up and linked with circleci setup, but we have validate in pre-commit hooks in a few places and I don't want everyone to have to set up a personal access token just for validation.

[blimmer]

@elliotforbes this was against CircleCI cloud. It's working now, but definitely was not when I posted.

[blimmer]

Actually @elliotforbes this is still failing for me. I tried the following:

> brew uninstall circleci
> rm -fr ~/.circleci
> brew install circleci
> circleci setup
CircleCI API Token ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
API token has been set.
CircleCI Host https://circleci.com
CircleCI host has been set.
Setup complete.
Your configuration has been saved to /Users/blimmer/.circleci/cli.yml.

Trying an introspection query on API to verify your setup... Ok.
Trying to query our API for your profile name... Hello!
> circleci config validate
failed to decode body: invalid character 'I' looking for beginning of valueError: config compilation request returned an error: invalid character 'I' looking for beginning of value

[blimmer]

Oh, it works if I regenerate my API token... seems like it should warn if you need to do that.

[elliotforbes]

@blimmer - I'll follow up with the team that owns this project and see if they can update it to more gracefully handle non-json errors from the underlying APIs.

The invalid character 'I' is referencing the first character in the Invalid API token string message that is returned when an API token used is no longer valid, this change was unfortunately something that went live after my own updates which is why the original pull request doesn't handle this as gracefully as it should.

[blimmer]

Gotcha - it seems like the API token was still valid, though, because when I ran setup it did some test queries:

Trying an introspection query on API to verify your setup... Ok.
Trying to query our API for your profile name... Hello!

Is the real issue that the new endpoint doesn't accept "old" tokens?

[tdebarochez]

@blimmer yes, it seems to be the case.
For the record, before re-running circleci setup I had the following error with the latest version of the CLI (0.1.26255+cc8c4de):

❯ circleci config validate
Error: config compilation request returned an error: %!w(<nil>)