Zed doesn't detect environment variables from its launch environment

[zmitchell]

Check for existing issues

• Completed

Describe the bug / provide steps to reproduce it

It's common to set environment variables in your shell and then launch your editor in that shell so that the editor picks up those environment variables. This is very common in the Nix world especially, where you would enter a development shell and launch the editor from the shell.

Basic case:

$ export FOO=BAR
$ zed .

Nix case:

$ nix develop # puts you into a subshell with environment variables set
$ zed .

This matters a lot when you have compile-time environment variables in a Rust project.

pub const FOO: &str = env!("FOO");

In a Nix project you might use this to hard-code the Nix store path of a binary that you want to call from your Rust project, but you could imagine a variety of other use cases I'm sure.

On the codebase I'm working on I'm also not getting any other "red squiggles" when I'm in my development shell. I can't tell if that's related or not. The codebase is flox, which contains Rust and C++.

Environment

Zed: v0.127.4 (Zed)
OS: macOS 14.2.0
Memory: 32 GiB
Architecture: aarch64

If applicable, add mockups / screenshots to help explain present your vision of the feature

No response

If applicable, attach your ~/Library/Logs/Zed/Zed.log file to this issue.

2024-03-25T12:09:41-06:00 [INFO] ========== starting zed ==========
2024-03-25T12:09:41-06:00 [INFO] Opening main db
2024-03-25T12:09:41-06:00 [INFO] set environment variables from shell:/bin/zsh, path:/Users/zmitchell/code/bin/arcanist/bin:/Users/zmitchell/code/bin:/Users/zmitchell/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin:/Library/TeX/texbin:/usr/local/opt/llvm/bin:/Users/zmitchell/.pyenv/shims:/Users/zmitchell/.local/bin:/opt/homebrew/bin:/Users/zmitchell/.cargo/bin:/Users/zmitchell/.cargo/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/VMware Fusion.app/Contents/Public:/Users/zmitchell/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/Users/zmitchell/code/bin/arcanist/bin:/Users/zmitchell/code/bin:/Users/zmitchell/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin:/Library/TeX/texbin:/usr/local/opt/llvm/bin:/Users/zmitchell/.pyenv/shims:/Users/zmitchell/.local/bin:/opt/homebrew/bin:/Users/zmitchell/.cargo/bin:/Users/zmitchell/Library/Application Support/JetBrains/Toolbox/scripts:/Users/zmitchell/Library/Application Support/JetBrains/Toolbox/scripts
2024-03-25T12:09:42-06:00 [ERROR] crates/zed/src/zed.rs:615: EOF while parsing a value at line 1 column 0
2024-03-25T12:09:42-06:00 [INFO] Opening main db
2024-03-25T12:09:42-06:00 [INFO] building git repository, `.git` path in the worktree: ".git"
2024-03-25T12:09:42-06:00 [INFO] set status on client 0: Authenticating
2024-03-25T12:09:42-06:00 [INFO] Opening main db
2024-03-25T12:09:42-06:00 [INFO] set status on client 85128: Connecting
2024-03-25T12:09:42-06:00 [INFO] Opening main db
2024-03-25T12:09:42-06:00 [INFO] starting language server "rust-analyzer", path: "/Users/zmitchell/code/scale21x_demos/rust-react-chat", id: 1
2024-03-25T12:09:42-06:00 [INFO] fetching latest version of language server "rust-analyzer"
2024-03-25T12:09:42-06:00 [INFO] connected to rpc endpoint https://collab.zed.dev/rpc
2024-03-25T12:09:43-06:00 [INFO] add connection to peer
2024-03-25T12:09:43-06:00 [INFO] waiting for server hello
2024-03-25T12:09:43-06:00 [INFO] got server hello
2024-03-25T12:09:43-06:00 [INFO] set status to connected (connection id: ConnectionId { owner_id: 0, id: 0 }, peer id: PeerId { owner_id: 378, id: 946887 })
2024-03-25T12:09:43-06:00 [INFO] set status on client 85128: Connected { peer_id: PeerId { owner_id: 378, id: 946887 }, connection_id: ConnectionId { owner_id: 0, id: 0 } }
2024-03-25T12:09:43-06:00 [INFO] downloading language server "rust-analyzer"
2024-03-25T12:09:44-06:00 [INFO] starting language server. binary path: "/Users/zmitchell/Library/Application Support/Zed/languages/rust-analyzer/rust-analyzer-2024-03-25", working directory: "/Users/zmitchell/code/scale21x_demos/rust-react-chat", args: []
2024-03-25T12:09:59-06:00 [ERROR] crates/lsp/src/lsp.rs:720: oneshot canceled
2024-03-25T12:10:10-06:00 [INFO] building git repository, `.git` path in the worktree: ".git"
2024-03-25T12:10:28-06:00 [INFO] starting language server "rust-analyzer", path: "/Users/zmitchell/code/floxdev/flox/master", id: 2
2024-03-25T12:10:28-06:00 [INFO] starting language server. binary path: "/Users/zmitchell/Library/Application Support/Zed/languages/rust-analyzer/rust-analyzer-2024-03-25", working directory: "/Users/zmitchell/code/floxdev/flox/master", args: []
2024-03-25T12:13:33-06:00 [INFO] building git repository, `.git` path in the worktree: ".git"
2024-03-25T12:13:33-06:00 [INFO] starting language server "rust-analyzer", path: "/Users/zmitchell/code/scale21x_demos/rust-react-chat", id: 3
2024-03-25T12:13:33-06:00 [INFO] starting language server. binary path: "/Users/zmitchell/Library/Application Support/Zed/languages/rust-analyzer/rust-analyzer-2024-03-25", working directory: "/Users/zmitchell/code/scale21x_demos/rust-react-chat", args: []
2024-03-25T12:13:33-06:00 [ERROR] crates/workspace/src/persistence/model.rs:267: Deserializer does not exist for item kind: diagnostics
2024-03-25T12:30:25-06:00 [INFO] building git repository, `.git` path in the worktree: ".git"
2024-03-25T12:31:02-06:00 [ERROR] crates/lsp/src/lsp.rs:720: oneshot canceled
2024-03-25T12:31:07-06:00 [ERROR] crates/lsp/src/lsp.rs:720: oneshot canceled
2024-03-25T12:31:07-06:00 [INFO] building git repository, `.git` path in the worktree: ".git"
2024-03-25T12:31:07-06:00 [ERROR] crates/workspace/src/persistence/model.rs:267: No path stored for this editor
2024-03-25T12:31:16-06:00 [INFO] starting language server "taplo-ls", path: "/Users/zmitchell/code/floxdev/flox/sentry-flox-instrumentation", id: 4
2024-03-25T12:31:16-06:00 [INFO] fetching latest version of language server "taplo-ls"
2024-03-25T12:31:17-06:00 [INFO] downloading language server "taplo-ls"
2024-03-25T12:31:18-06:00 [INFO] starting language server. binary path: "/Users/zmitchell/Library/Application Support/Zed/languages/taplo-ls/taplo", working directory: "/Users/zmitchell/code/floxdev/flox/sentry-flox-instrumentation", args: ["lsp", "stdio"]
2024-03-25T12:31:24-06:00 [INFO] starting language server "rust-analyzer", path: "/Users/zmitchell/code/floxdev/flox/sentry-flox-instrumentation", id: 5
2024-03-25T12:31:24-06:00 [INFO] starting language server. binary path: "/Users/zmitchell/Library/Application Support/Zed/languages/rust-analyzer/rust-analyzer-2024-03-25", working directory: "/Users/zmitchell/code/floxdev/flox/sentry-flox-instrumentation", args: []

[mrnugget]

Hey! Actually, Zed does detect environment variables from its launch environment, but things are a bit tricky. Let me explain:

Current Status

1. When you use the zed CLI, Zed checks whether it's stdout is a pty and if so, it inherits it.
2. If you don't launch Zed from the CLI (but from the macOS Dock, for example) we try to get to the environment from a login shell spawned in your home directory:

   zed/crates/zed/src/main.rs

   Lines 201 to 208 in 2630230

   	let login_shell_env_loaded = if stdout_is_a_pty() {
   	Task::ready(())
   	} else {
   	app.background_executor().spawn(async {
   	load_login_shell_environment().await.log_err();
   	})
   	};

3. If (2) happens, then that environment is set on the Zed process, meaning it's inherited by every process launched by Zed.

So far, so good. You can see in the logs above that (2) happened for you:

2024-03-25T12:09:41-06:00 [INFO] set environment variables from shell:/bin/zsh, path:/Users/zmitchell/code/bin/arcanist/bin [...]

But here's gotcha nr. 1:

4. When you use zed from the CLI when a Zed process is already running, it only opens a new Zed window and doesn't inherit any of the shell environment at all. So if you do zed project-a; export PATH="foobar/:$PATH"; zed project-b, that PATH is ignored.

As part of #7902 I tried to improve on the situation and created #8188, which adds the following:

5. When launching the Go and Zig (that's the only one we added) language servers, Zed now tries to load the shell environment as it would be if you cd into-your-project.

   Here is how it happens in the gopls adapter:

   zed/crates/languages/src/go.rs

   Lines 90 to 97 in 2630230

   	let env = delegate.shell_env().await;
   	let path = delegate.which("gopls".as_ref()).await?;
   	Some(LanguageServerBinary {
   	path,
   	arguments: server_binary_arguments(),
   	env: Some(env),
   	})
   	}

6. When doing (5) for Go, we load the environment by cding into the project, dumping the environment, and keeping it around (but not setting it on the Zed process, since that's global state!). We then take that environment and check whether we can find gopls in it with this function:

    https://github.com/zed-industries/zed/blob/263023021d80f6caa08c23ea25dea4615cdb5594/crates/project/src/project.rs#L10449-L10458
   

7. Crucially, when then starting gopls, we set that captured environment on the process, so that gopls thinks it started in that environment and can find the right go version etc.

5.-7. are only implemented for Go and Zig right now and for Zig we even had a slight regression that we fixed yesterday, where we would not remember the env in which we found zls: #10559 (Edit: correction! it's implemented for more languages, just noticed that clangd also supports this)

So that's gotcha nr. 2:

8. Project-specific environments aren't loaded for all language servers/processes and are implemented on an adapter by adapter basis.

Options

I think, the ideal case would look something like this:

1. When you launch Zed via the CLI, zed ., and if that spawns a new project/window, the environment in which zed was started should be kept around. We can't set it on the process, since that's global state. So we need to keep it around and lookup binaries in that environment every time we launch a process in the context of a given project/window and also set that environment on each process we launch. (I think that's how VSCode's code works)
2. When you launch Zed from the Dock and open a project, I think we should either (a) act as if it was launched in that project's directory (by doing what we do above; loading the shell env in there) or (b) don't do anything at all (which would break a lot of setups)
3. Or: we introduce extensions whose sole purpose it is to populate a project's environment, which other extensions can then access. We could, for example, have a Nix extension that would detect whether a project has a Nix configuration, load it, get the environment, and then set it on the project so that (1) works.

That's a lot of work that's tricky to get right, since there's a lot of global state being kept around and edge cases to handle (what if you have two worktrees in a single project/window? do we already correctly attach language servers per worktree?). It also has to be said, that there's no real "best practices" here, since all the language servers do something slightly different (some read their env, some read only config files), all editors do something different, editor plugins also mix their own behavior in, etc. etc.

A short-term solution could be that we modify the Rust and the C++ adapters just like the Go and Zig adapters to also make use of the current environment.

A workaround would be to only have a single Zed process running and launching it from the CLI. That should pick up the environment variables.

cc @maxdeviant and @maxbrunsfeld who worked on the extensions and whom I also talked to about this before. I'd love to hear your thoughts on this, now that we're moving to extensions.

[maxbrunsfeld]

what if you have two worktrees in a single project/window? do we already correctly attach language servers per worktree?

I think we're ok here, because language servers are always spawned for one particular worktree. In a multi-root project, we'll spawn a separate language server for each root.

We already expose a shell_env() method that LSP adapters can call in order to explicitly retrieve the environment variables for a given worktree, by spawning a shell. I think the highest-priority fix for this issue is to simply use that API in more cases, like the Rust and C++ LSP adapters.

I also think it would be a good idea to store, on each Project (which may have multiple roots) the shell env of the zed CLI invocation that initially launched that project (if it was launched from the CLI). I don't know exactly how that stored set of environment variables should be used though. Maybe we could automatically merge that into the set of variables returned by shell_env()?

[zmitchell]

Option 1 sounds reasonable to me, and VS Code's code command generally works pretty well for me with this workflow.

We already expose a shell_env() method that LSP adapters can call in order to explicitly retrieve the environment variables for a given worktree, by spawning a shell.

Just want to point out a slight complication here with Nix. If you're not aware, when you enter a development shell with Nix via nix develop, you're always put into a Bash subshell, which may not be the user's configured shell. That said, the user has likely called nix develop from the user's shell, so the environment variables set by the user's shell will either be available directly or have been overridden by the call to nix develop.

[mrnugget]

Just want to point out a slight complication here with Nix.

Yeah, I do think that there's only so much we can do (apart from having a Nix-aware extension that populates the environment variables for you, of course). We already make sure to trigger direnv or other things that populate the env on cd, but I think that's as far as it gets before we get too specific.

In this case, the interesting bit here is that I think the environment inherit by zed . should take precedence over any environment that ourselves detect. Because I imagine this use case:

1. cds into project
2. nix develop
3. zed .

Now Zed should have all the correct environment variables, but if we were then to spawn a shell in project and take that environment, it'd be wrong.

So, to answer your question, @maxbrunsfeld

I also think it would be a good idea to store, on each Project (which may have multiple roots) the shell env of the zed CLI invocation that initially launched that project (if it was launched from the CLI). I don't know exactly how that stored set of environment variables should be used though.

I think it should work roughly like this:

1. If zed . is used, we take the environment and store it on the Project
2. The language adapter's shell_env should then check: do we already have an env stored on Project, if so, use that, otherwise launch a shell to get the environment.

So no merging, but deciding either or.

[srid]

Just wanted to say that this issue makes Zed unusable with nix develop (or direnv), as it negatively affects LSP experience.

For example, on Rust projects - Zed will refuse to recognize the rust-analyer in the Nix shell, and then proceed to download its own, and then fail to launch it.

The above is reproducible with https://github.com/srid/rust-nix-template

[shekohex]

@mrnugget I noticed something, it is mostly an issue with the CLI not Zed, because here what I have tested, and I find it very interesting:

1. Enter a rust project using cd project that uses direnv that triggers nix develop or call nix develop manually.
2. If you opened zed using zed . the CLI, rust-analayzer is not detected, and zed will try to download it then it errors with “Failed to load workspace” similar to what @srid mentioned above.
3. However, if you started zed directly from the terminal using the zed binary like the following: /Applications/Zed\ Preview.app/Contents/MacOS/zed . it works! And it detects the current rust version and rust-analyzer starts to load the workspace!

I find it very strange because the editor itself is able to load the shell environment correctly, but looks like the CLI just throws it away before it starts the Zed binary.

Also I noticed that if I started it like that, it does not recognize custom themes? even tho it loads it correctly!

[2024-05-09T16:05:29+03:00 ERROR util] crates/theme/src/settings.rs:439: theme not found: Catppuccin Mocha
[2024-05-09T16:05:33+03:00 ERROR util] crates/lsp/src/lsp.rs:773: oneshot canceled

[NickLarsenNZ]

@zmitchell, off topic, but are you able to share your flake.nix?

[zmitchell]

https://github.com/flox/flox

That's a particularly gnarly flake.nix so good luck. The problem that motivated the issue will be present for any flake.nix that sets an environment variable if I understand correctly so if you're looking for a smaller reproducer that's not where I would start.

[iwinux]

Would it be possible to override project-specific envs in local settings (.zed/settings.json)?

[CrabbyDisk]

I am also having this issue, is there any way to fix this as of now?

[opsnull]

I like the idea of override project-specific envs in local settings (.zed/settings.json).

[zmitchell]

Setting the environment manually in settings.json doesn’t really make sense for technologies like Nix that set several environment variables, and sets them to different values whenever you update your flake.nix.

[mrnugget]

There's still no fix, I plan on working on this in the near future, but am currently busy with the Linux port. Sorry!

[RobertMueller2]

I noticed something that might fit here rather than a new bug report.

I'm using an on-exit event script in fish, like this:

function on_exit --on-event fish_exit

    if status --is-interactive ;and not set -q _NO_EXIT_MSG
        echo "Exiting fish."
        if string length -q -- $WINEPREFIX
            read -P "Do you want to stop wineserver on exit? (y/any) " -n 1 INPUT
            if string match -q -i y $INPUT 
                echo "Stopping Wineserver."
                wineserver -k
            end
        end
    end
end

When zed runs a shell to get environment vars, this "Exiting fish." is randomly present in one of them (at least I couldn't see a pattern which variable it affects, but might be wrong). This can be confirmed in a zed terminal panel by using env to identify and then echo the variable.

When this affects $HOME, there are funky side effects, e.g. rust-analyzer stops working and cargo (started from within Zed's terminal) tries to create "/home/USER\nExiting fish" and fails due to missing permissions.

I'm using the _NO_EXIT_MSG in the above as a workaround.

[Albert-IV]

It looks like this feature is live in the preview releases.

[mrnugget]

Yes! Forgot to update this issue. See: #17075

[zmitchell]

Thanks everyone! I’ll take a look once the next release goes out.