move @cImport to the build system

[andrewrk]

This issue is to remove the @cImport language builtin, instead relying on the feature being provided by the build system (std.Build.Step.TranslateC).

This removes one dependency on libclang inside the Zig compiler executable, which is required for #16270. In the near-term future, the translate-c implementation will still be inside the Zig compiler executable, along with the ability to compile C/C++ source files. However, we can prepare users for this transition by solving the use case via the build system rather than a language feature.

As a consolation prize, the TranslateC build step can be enhanced with advanced settings, such as namespace stripping and validation of existing bindings. Since changes to this don't require changing the language, it's OK if the scope & complexity increase to some extent.

Blockers:

• translate-c build step causing crash in build system #20648
• move https://ziglang.org/learn/build-system/ to become a guide along with the language reference
• translate-c build step lacks a way to link system libraries #20649

Related:

• C interop: expected type bool found c_int #19950
• translate-c should not emit 'pub' for static functions #2314
• additional tooling for ABI compatibility verification #20654

[mlugg]

A small extra note: this eliminates one quite common use case of usingnamespace in the wild, which is to have a c.zig containing a pub usingnamespace @cImport(...) declaration. Instead, you now use a TranslateC step as described above, and directly expose the generated file to Zig code as a module.

[149-code]

@andrewrk would this mean that in order to interoperate with c, you need to use the zig build system?

I have a small personal project written in c built using make files. Im currently extending it by writing new zig code that cIncludes header files and invoking the zig compiler from make (i.e zig build-obj).

Would this use case not be supported anymore?

[alexrp]

You would just have to invoke zig translate-c in your makefile; that's what std.Build.Step.TranslateC does. The compiler also does the moral equivalent of that when you use @cImport().

[mlugg]

@149-code, @alexrp -- note that once #16270 is implemented, zig translate-c will no longer be a subcommand, with this functionality instead existing entirely within the build system. In that case, you probably will need to use zig build.

In theory, you can have your build.zig purely do the C translation, so zig build effectively becomes your drop-in for zig translate-c. However, this will almost certainly be more awkward than just migrating your C project to use the Zig build system.

[alexrp]

@mlugg

with this functionality instead existing entirely within the build system

Hm, that sounds a bit concerning. Not looking to relitigate #16270 all over again, but I'm on board with it because the promise was that existing use cases would still be supported - it just might involve downloading an extra package for all the C/C++ functionality (which is totally fair). But if translate-c can't even be accessed outside the Zig build system, that leaves my Zig MSBuild SDK in a very awkward spot. I'm specifically integrating Zig into another build system entirely (MSBuild), so I would have to generate and invoke a build.zig which adds a whole other layer of complexity, performance penalty, and moving parts for (AFAICT) no benefit other than accessing functionality that would have previously been easy to integrate.

Please don't get me wrong; I really like the Zig build system and I use it extensively in my "pure Zig" projects. But if we start making existing features of the zig binary wholly exclusive to the Zig build system, I think that's going to significantly harm Zig adoption in brownfield or polyglot settings.

[ifreund]

I'm specifically integrating Zig into another build system entirely (MSBuild), so I would have to generate and invoke a build.zig which adds a whole other layer of complexity, performance penalty, and moving parts for (AFAICT) no benefit other than accessing functionality that would have previously been easy to integrate.

I don't think any code generation would be required. You could write a single, generic build.zig that exposes a CLI pretty much identical to the current zig translate-c. The only difference is that the binary exposing the CLI would not be the zig compiler.

One could probably even write a zig package that provides a translate-c binary in every way identical to the current zig translate-c subcommand except that it is not part of the zig compiler binary.

[mlugg]

What ifreund says would work, but thinking about how the translate-c package is likely to work, it might not even be necessary. I would expect the translate-c package to expose an artifact, and users construct Run steps to run that artifact (of course, we would add any necessary helper functions to make this not-awkward). So, you could zig build the translate-c package directly to get a translate-c binary.

[alexrp]

That sounds reasonable. My concern here was just the prospect of having to integrate all the machinery of the Zig build system as part of the build logic in my MSBuild SDK (which is currently just a zig build-exe/build-lib/test/cc/c++ invocation at the end of the day). Stacking build systems on top of each other is guaranteed pain, in my experience.

So as long as there's a way to invoke translate-c as a plain old CLI command - whether part of zig or a binary of its own - I'm a happy camper.

The only other point I would add here is that I think it would be good to distribute such a translate-c binary as part of a "C/C++ support" binary package on ziglang.org to reduce friction for cases like @149-code's (it would also make packaging a tiny bit easier for my SDK, but that's not essential by any means).

[andrewrk]

Well, I think you have provided a quite interesting and valuable real world use case, so, let's work together to come up with a migration proposal that is appealing before proceeding with removal of @cImport functionality.

[mlugg]

To be clear @andrewrk, the use case described above doesn't depend on @cImport, only on the zig translate-c subcommand. This proposal kind of has two parts: removing @cImport seems easy and unproblematic (at least, once we improve std.Build.Step.TranslateC), while migrating zig translate-c out of the compiler is what raised concerns above.

[andrewrk]

Ah thanks for that clarification. Well, point stands for that follow-up issue but I will remove it from the blocker list above.

[steeve]

So as long as there's a way to invoke translate-c as a plain old CLI command - whether part of zig or a binary of its own - I'm a happy camper.

Adding my 2cts to heavily support this. We're using zig via bazel, so we directly invoke zig build-{exe,static} and would love to keep translate-c as a CLI.

[WeijieH]

I really like to use the zig run command to run some of my small projects/scripts (some are using @cImport) without dropping an exe at the current directory. So, after this change, is there way I can still simply call zig run or do I have to use a build.zig file?

[andrewrk]

Adding my 2cts to heavily support this. We're using zig via bazel, so we directly invoke zig build-{exe,static} and would love to keep translate-c as a CLI.

That's a given - the build system will invoke the translate-c tool as a subprocess using CLI args.

So, after this change, is there way I can still simply call zig run or do I have to use a build.zig file?

No, this use case will regress.

[agluszak]

What about the "C macro reflection" feature described here and discussed on HN?

[ifreund]

What about the "C macro reflection" feature described here and discussed on HN?

This is unrelated to the reflection features described there. The only change would be how the translated C header files are accessed from zig code.

[ifreund]

This is such a shame. Zig's main attraction is the easy & built-in interoperability with C.

Regressing features on that front is not a good sign.

@LouisLibre Your comment does not add anything of value to this discussion. A proposal like this weighs the benefits and drawbacks against each other to decide if the zig project as a whole would be improved by the proposal or not. Merely reiterating a single drawback with no nuance or discussion as you have done is useless noise.

[MatthiasPortzel]

For “hello world from C” and bug-reproduction demos, it is useful to have C-interop without having to set up the build system. (The ongoing HN thread on this topic has several anecdotes from people convinced to try out Zig because of such simple demos.) For someone new to the language, setting up the build system at all represents significant complexity.

However this use-case could be satisfied by adding an argument to zig run, zig build-exe, and friends that allowed the user to specify C files to be auto-included as modules.

This would continue to kick the can down the road with regards to #16270, but it may make sense in the short term (allowing both zig run to include C modules and removing cImport for incremental, if that was a concern).

[ityonemo]

Keeping incorporating c header imports as a feature accessible to the CLI is important to the semantic analysis pass that happens as a part of Zigler. https://github.com/E-xyza/zigler (I'm also using this in prod and this week it's saving the bacon of the startup I'm at)

[andrewrk]

@ityonemo can you be more specific? Seems like you should be fine with the zig translate-c subcommand, no? It's more powerful than @cImport.

[ityonemo]

I'm fine with deprecating @cImport, but just as the "zig module import" parameters (paths, deps) can be set via command line, if a module is imported as a c ABI, id want tobe able to do so via cli without resorting to build.zig. mentioning that zig run would regress is a bit worrying. For example, in 0.12 I can run zig run --dep foo -Mxyz.... I would have imagined something like zig run --h foo.h --define true=false -Mc_lib... etc.

I'm not exactly sure how the translate-c subcommands works (it sounds like it will? Maybe?) and I could probably figure it out, but I had lots of difficulty integrating a semantic pass with build.zig that lived side-by-side with the libs I generate, since I am doing sneaky things like substituting shims when running semantic analysis pass.

[andrewrk]

@ityonemo the @cImport feature effectively constructs a translate-c command which outputs to a file, and then that file is imported with @import. I can't see anything but a more powerful API offered to zigler by using the CLI rather than language feature. You won't be able to do it with 1 command though, you'll need to run translate-c separately and save the output somewhere. There's no plan to make run, build-exe, build-lib, or build-obj capable of executing a translate-c step automatically.

Anyway, here's a related issue I typed up just now. I think this is the actual issue that people are wanting to talk about today:

• zig cc, zig c++, zig translate-c and other subcommands without a clang/llvm dependency in the compiler binary #20875

[ityonemo]

Good to know. More than one command is not ideal, but it's fine! I could probably also figure out how to do it with build.zig, I just had enough trouble with it that I gave up last time :)

[philip-peterson]

From the perspective of an outsider, the fact that @cImport is probably going away, while still having blog posts written about how great it is, it would be nice to preemptively learn the "new way" to write Zig that interops with C. If the documentation could just reflect the future (assuming of course that the future is somewhat known) it would be nice, since otherwise learning Zig now feels like it could be obsolete next week. (Of course this is always to be true of an experimental language but it just feels like a significant block is all.)

[andrewrk]

To be clear, there is an upgrade path, so if you learn how to use @cImport and then you have to move it to the build system, it doesn't really mean you wasted your time. It just means you have to do something like this diff.

[andrewrk]

https://github.com/ziglang/translate-c