Add the PNaCl/JS targets to the backend. #28355

DiamondLovesYou · 2015-09-11T15:59:02Z

alexcrichton · 2015-09-11T16:39:12Z

How close does this get the backend to working? Is this all that's needed or are there still portions in the pipeline? The previous patches up until now have been incremental in the sense that they were just general refactorings we'd want to do anyway, but I'd prefer to have "pnacl not working" to "pnacl working" support land all at once rather than piecemeal.

DiamondLovesYou · 2015-09-11T18:21:13Z

As far as Rust trans is concerned, this gets PNaCl working. In reality, pnacl-llvm needs at least two more passes to handle:

LLVM's auto vectorizer (PNaCl currently disables this because it will generate illegal (in the PNaCl ABI sense) vector lengths).
Illegal vector types (ie not 128bit width (and not [fiu]64))
[fiu]64 vector element types.

Previously, I just disabled the auto vectorizor and added wrappers for the [fiu]64 SIMD types in core, but because of the recent SIMD rfcs, it seems I'll have to write a legalization pass anyway. The main vector legalization pass is currently WIP. The second pass to expand [fiu]64 will happen after.

Also note I'd also like to allow users to disable the setjmp/longjmp EH (which isn't free) in the future (via an unstable -C target-options="blah" argument).

alexcrichton · 2015-09-13T17:41:24Z

src/librustc_trans/back/write.rs

-                write_output_file(cgcx.handler, tm, cpm, llmod, &path, llvm::ObjectFileType);
-            });
+            if !cgcx.is_like_pnacl {
+                with_codegen(tm, llmod, config.no_builtins, |cpm| {


Should this logic go into pnacl itself? Basically when you request an object file to be emitted from pnacl it seems like it should naturally understand that it instead just emits bitcode.

Except it's LLVM's target machine that does that. PNaCl/JS (well JS does have one, but it isn't used till post-linking) doesn't have a target machine (though PNaCl does use armv7-none-nacl-gnu for optimizing, using it for codegen would produce an ARM NaCl object file, which isn't compatible with PNaCl).

In fact if you take a look at the toolchain's pnacl-clang, you'll find it's just a python wrapper which invokes clang with arguments like -emit-llvm.

This may just be me being ignorant of the LLVM fork for pnacl, but if you're already forking LLVM couldn't support for this be added? In terms of interoperating with other compilers it seems like it'd be best to put logic into the pnacl target rather than every compiler using LLVM?

But there doesn't exist any PNaCl LLVM target machine in (pnacl-)LLVM's target registry (ie using the PNaCl LLVM fork doesn't help in any way). So there's no LLVM target to tell LLVM how to write an object file (of any format).

In an ideal world, I'd completely throw out the PNaCl IR simplification passes and restart afresh, and I'd do it with an actual, real LLVM target machine (since LLVM's ISD form and PNaCl's IR are actually pretty similar). Sadly, while LLVM's SelectionDAG does most (if not all) the heavy lifting w.r.t. legalizing IR to a specific target, the rest of the toolchain would still need modifications to know how to handle the new PNaCl object format (ie how to read/link), which I would have to create specifically for the target machine (another possible, but still non-trivial thing to implement). The required toolchain modifications are the reason JS also uses bitcode as its object format, despite having an official target machine in LLVM. Plus the translation from the object format back into LLVM IR for codegenning to the real native target.

While I grant that using bitcode as the object format is annoying at times, it is a lot easier because it integrates the preexisting LLVM gold plugin based linker.

I actually agree with you, but the impl cost of what you propose is pretty high vs what emcc and pnacl-clang currently do (ie -emit-llvm (both are just scripts wrapping the real clang underneath; no such luck for Rust)). Plus, the other toolchains would require breaking changes to conform (ie this logic makes Rust conform with the other toolchains).

It sounds like we may want to take a similar strategy to those tools and instead of having the default output type be an object file for insertion into an archive we just have the default output type be bitcode to insert into an archive?

--emit=obj would still be broken. Plus the PNaCl linker expects bitcode as the main linker input (ie main.o). This change does accomplish exactly what you propose and doesn't require special casing archive object output.

alexcrichton · 2015-09-21T16:27:50Z

@brson, what are your feelings on this? This looks to be one of the only points of divergence where "if pnacl { ... } else { ... }" is needed, and while it's good that it's contained this basically all still looks to me like portions which would be best handled in LLVM somewhere instead of rustc itself (e.g. as it does for all other existing targets). From what @DiamondLovesYou is saying, however, it sounds like the way pnacl is integrated into LLVM makes this not easy unfortunately.

I continue to be worried about the lack of automation we have for this kind of functionality where "some subset" of intrinsics isn't defined on pnacl, but any future intrinsics we add may or may not be part of this subset and we have no way of verifying one way or another.

brson · 2015-09-21T20:23:15Z

These patches are much easier to digest.

I still don't understand why PNaCl believes it's ok not to handle intrinsic that the LLVM supports. The explanation that pnacl is static and can't do it isn't convincing - LLVM itself lowers intrinsics to libm calls and PNaCl can do that too. What happens if we do pass these intrinsic calls to PNaCl? Are they on a blacklist that causes it to fail to compile? Why not just pass the inttrinsic through to libm like LLVM?

If we must do our own intrinsic lowering, then let's do it in a generic way - give target specs a mapping from intrinsics to C calls, not have a pnacl special case. Likewise the is_like_pnacl check for emitting llvm bitcode (into a .o file?) should be a generic 'output_bitcode_to_object_file' or something.

brson · 2015-09-21T20:26:25Z

I guess the issue is that there are no subsequent linkage steps in the pnacl toolchain. The bitcode output by rustc is the final product and there's no opportunity to link libm after the fact.

Edit: but most of these functions are in libm and/or libc, and needed for most software to work. Doesn't PNaCl just always link to libc/libm? (Emscripten supposedly does).

DiamondLovesYou · 2015-09-28T16:01:19Z

@brson Sorry for the delay.

but most of these functions are in libm and/or libc, and needed for most software to work. Doesn't PNaCl just always link to libc/libm? (Emscripten supposedly does).

Sure, since libm is included in libc, but that doesn't mean the linker will actually include the definitions (unless the defs are used), thus their availability isn't guaranteed post link.

Also, PNaCl (and thus Emscripten) disables -simplify-libcalls, so calls to libm won't result in calls to an intrinsic. Thus normal, defined calls to libm (those that would happen in C) will still call libm after optimizations.

If we must do our own intrinsic lowering, then let's do it in a generic way - give target specs a mapping from intrinsics to C calls, not have a pnacl special case. Likewise the is_like_pnacl check for emitting llvm bitcode (into a .o file?) should be a generic 'output_bitcode_to_object_file' or something.

I'm fine with that, if that's what it takes.

alexcrichton · 2015-09-28T21:26:58Z

Sure, since libm is included in libc, but that doesn't mean the linker will actually include the definitions (unless the defs are used), thus their availability isn't guaranteed post link.

So if I understand things correctly, there's one giant LLVM module which contains the bitcode for libc, libm, and the application in question. This module is then optimized and such and is passed through the pnacl backend. The definitions of libm functions are all optimized away because they're not used, and then after this happens the intrinsics are lowered to calls to those libm functions, resulting in the undefined symbols?

Is that what's going on?

kripken · 2015-09-28T22:23:24Z

Also, PNaCl (and thus Emscripten) disables -simplify-libcalls

I wasn't aware that PNaCl disables that, and I don't think we use code from PNaCl that would cause Emscripten to do it. But maybe I'm wrong - do you know where PNaCl disables that pass?

LLVM source code seems to show that the entire pass has been removed in upstream anyhow?

http://llvm.org/viewvc/llvm-project?view=revision&revision=184459

Unless PNaCl has modified the places where parts of that pass have been moved to, as mentioned in that commit?

DiamondLovesYou · 2015-09-30T23:57:25Z

@alexcrichton

So if I understand things correctly, there's one giant LLVM module which contains the bitcode for libc, libm, and the application in question. This module is then optimized and such and is passed through the pnacl backend. The definitions of libm functions are all optimized away because they're not used, and then after this happens the intrinsics are lowered to calls to those libm functions, resulting in the undefined symbols?

Is that what's going on?

No, the problem is that LLVM's math intrinsics create implicit dependencies on libm, but the linker isn't aware of this fact, and as a result omits libm (or rather, omits the objects containing no defined undefined symbols) from the resulting module. Then, post-link, when the PNaCl IR passes and optimizations are run, llvm.*.* don't have something that can be used in their place.

Normally this isn't an issue because codegen will rewrite these to libm as needed, thus allowing the linker to know about references to them. However, PNaCl/JS use bitcode linking, which doesn't have calls to llvm.*.* rewritten to libm.

I apologize if I'm making this confusing, as this area seems to have recurring misconceptions.

Thankfully, I think my first proposal isn't as bad as I had initially feared (I'm dumb): generally, the implicit libm dep should just be made explicit, because this would also be an issue for any other target if libc is statically linked for WPO (in the same way it's an issue for PNaCl). After that's fixed, creating a PNaCl IR pass to rewritten the math intrinsics won't be an issue. Once I get time, I'll send a patch to upstream LLVM.

@kripken No the file was just moved to a different folder in that commit. See: https://github.com/llvm-mirror/llvm/blob/master/lib/Transforms/Utils/SimplifyLibCalls.cpp

It probably should be disabled, though it probably only creates issues in extreme corner cases. See this comment from pnacl-opt.py from the NaCl SDK:

  # We disable the LLVM simplify-libcalls pass by default, since we
  # statically link in libc, and pnacl-opt is typically used for post-link
  # optimizations.  Changing one library call to another can lead
  # to undefined symbol errors since which definitions from libc are linked
  # in is already decided.

kripken · 2015-10-01T00:32:17Z

@DiamondLovesYou : thanks, interesting. I looked into this for emscripten, and it looks like it already isn't run during LTO anyhow (and it's fine to run otherwise). We mostly just do -std-link-opts for LTO, and those don't run simplify-libcalls, perhaps for this very reason. I guess PNaCl runs additional optimizations than -std-link-opts during LTO/their final build stage?

DiamondLovesYou · 2015-10-01T03:08:37Z

@kripken Yeah, PNaCl runs -On in addition to LTO (for even better optimization opportunities).

W.r.t. the intrinsic/libm thingy: after doing some digging in LLVM, it seems LLVM already ensures libm is available at codegen time by emitting references to libm functions. BUT this is only done if there's a target machine available. Le sigh. I may end up creating a dummy target machine and lowering etc for PNaCl.

DiamondLovesYou · 2015-10-01T22:41:33Z

@kripken In case you're interested: I've created a proposal for a PNaCl target machine here based on the discussion in this PR.

bors · 2015-10-02T19:19:51Z

☔ The latest upstream changes (presumably #28768) made this pull request unmergeable. Please resolve the merge conflicts.

kripken · 2015-10-02T20:28:10Z

@DiamondLovesYou : thanks for the info.

alexcrichton · 2015-10-05T16:41:51Z

@DiamondLovesYou your proposal on the pnacl mailing seems promising! Perhaps this should hold off for a resolution on that?

DiamondLovesYou · 2015-10-16T05:18:34Z

@alexcrichton Sorry, didn't see your reply in my inbox!

There's no need to wait because the requisite changes for pnacl-llvm can/must be done independently of Rust anyway. Not that this PR in its current form should be accepted, merge conflicts notwithstanding. I'll try to make time this weekend to remove the intrinsic and bitcode output logics as well merge with master.

@kripken FYI, I've sent in a patch for the target machine here: https://codereview.chromium.org/1395453003

DiamondLovesYou · 2015-10-19T15:10:03Z

Okay, I've cleaned/removed up the last of the extra logics. This should be ready to merge!

alexcrichton · 2015-10-19T21:29:32Z

src/librustc_trans/trans/context.rs

@@ -838,7 +838,9 @@ fn declare_intrinsic(ccx: &CrateContext, key: &str) -> Option<ValueRef> {

    ifn!("llvm.trap", fn() -> void);
    ifn!("llvm.debugtrap", fn() -> void);
-    ifn!("llvm.frameaddress", fn(t_i32) -> i8p);
+    if !ccx.sess().target.target.options.is_like_pnacl {


It looks like this is the only use case of is_like_pnacl, and it also looks like llvm.frameaddress isn't used at all, so perhaps this could just be removed entirely?

Fine by me; removed.

alexcrichton · 2015-10-22T07:41:38Z

@bors: r+ e497d4a

Thanks for the patience @DiamondLovesYou!

@alexcrichton

…ichton r? @alexcrichton

bors · 2015-10-22T08:35:40Z

⌛ Testing commit e497d4a with merge e7b2052...

bors · 2015-10-22T10:22:41Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

DiamondLovesYou · 2015-10-25T09:09:27Z

The first vector type legalizer pass PR is sent: https://codereview.chromium.org/1423873002.

rust-highfive assigned alexcrichton Sep 11, 2015

alexcrichton reviewed Sep 13, 2015
View reviewed changes

alexcrichton mentioned this pull request Sep 13, 2015

Port Rust to PNaCl/JS (rebase #1) #26505

Closed

tomaka mentioned this pull request Sep 15, 2015

Make Rust work with emscripten rust-lang/rfcs#604

Closed

DiamondLovesYou force-pushed the pnacl-librustc-trans branch 2 times, most recently from 4470b17 to 1091e72 Compare October 19, 2015 01:21

alexcrichton reviewed Oct 19, 2015
View reviewed changes

DiamondLovesYou force-pushed the pnacl-librustc-trans branch from 1091e72 to e497d4a Compare October 22, 2015 00:11

Add the PNaCl/JS targets to the backend.

e497d4a

bors added a commit that referenced this pull request Oct 22, 2015

Auto merge of #28355 - DiamondLovesYou:pnacl-librustc-trans, r=alexcr…

e7b2052

…ichton r? @alexcrichton

bors merged commit e497d4a into rust-lang:master Oct 22, 2015

est31 mentioned this pull request Jun 4, 2017

Remove support for the deprecated PNaCl technology #42420

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the PNaCl/JS targets to the backend. #28355

Add the PNaCl/JS targets to the backend. #28355

DiamondLovesYou commented Sep 11, 2015

alexcrichton commented Sep 11, 2015

DiamondLovesYou commented Sep 11, 2015

alexcrichton Sep 13, 2015

DiamondLovesYou Sep 15, 2015

alexcrichton Sep 16, 2015

DiamondLovesYou Sep 17, 2015

alexcrichton Sep 17, 2015

DiamondLovesYou Sep 17, 2015

alexcrichton commented Sep 21, 2015

brson commented Sep 21, 2015

brson commented Sep 21, 2015

DiamondLovesYou commented Sep 28, 2015

alexcrichton commented Sep 28, 2015

kripken commented Sep 28, 2015

DiamondLovesYou commented Sep 30, 2015

kripken commented Oct 1, 2015

DiamondLovesYou commented Oct 1, 2015

DiamondLovesYou commented Oct 1, 2015

bors commented Oct 2, 2015

kripken commented Oct 2, 2015

alexcrichton commented Oct 5, 2015

DiamondLovesYou commented Oct 16, 2015

DiamondLovesYou commented Oct 19, 2015

alexcrichton Oct 19, 2015

DiamondLovesYou Oct 22, 2015

alexcrichton commented Oct 22, 2015

bors commented Oct 22, 2015

bors commented Oct 22, 2015

DiamondLovesYou commented Oct 25, 2015

Add the PNaCl/JS targets to the backend. #28355

Add the PNaCl/JS targets to the backend. #28355

Conversation

DiamondLovesYou commented Sep 11, 2015

alexcrichton commented Sep 11, 2015

DiamondLovesYou commented Sep 11, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Sep 21, 2015

brson commented Sep 21, 2015

brson commented Sep 21, 2015

DiamondLovesYou commented Sep 28, 2015

alexcrichton commented Sep 28, 2015

kripken commented Sep 28, 2015

DiamondLovesYou commented Sep 30, 2015

kripken commented Oct 1, 2015

DiamondLovesYou commented Oct 1, 2015

DiamondLovesYou commented Oct 1, 2015

bors commented Oct 2, 2015

kripken commented Oct 2, 2015

alexcrichton commented Oct 5, 2015

DiamondLovesYou commented Oct 16, 2015

DiamondLovesYou commented Oct 19, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Oct 22, 2015

bors commented Oct 22, 2015

bors commented Oct 22, 2015

DiamondLovesYou commented Oct 25, 2015