Get rid on strip-flags/replace-flags for some packages

[pchome]

There is a lot of packages w/ strip-flags/replace-flags/filter-flags/... which was added years ago and probably can be dropped in now days.

For example you can get a list to analyse via $ grep -P -A1 -B1 "(replace-flags|strip-flags|filter-flags)" /var/db/pkg/*/*/* command. many of them useful but replace-flags -O3 -O2 and strip-flags can be retested.

So i wonder can we somehow override them. Maybe via package.cflags somehow or custom bashrc.d script or maybe custom eclass. Maybe there is a way to add overlay's internal use-flag like custom-cflags and let users to chose which packages this internal stuff will affect.

Other variants is to modify ebuilds itself or ask upstream to add custom-cflags or remove *-flags.

So far I tested x11-libs/gtk+-2.24.31-r1 , net-im/pidgin-2.12.0 , media-libs/libsdl2-2.0.7 , app-editors/vim-8.0.1188 and app-editors/vim-core-8.0.1188 with removed flag modifiers. And everything work w/o any issues for me. Moreover despite all this scary thing was written in 2005 ( https://bugs.gentoo.org/76331 ) vim works noticeable faster, I have more than dozen plugins installed and startup was slow for editor but now startup and navigation as fast as mcedit/mcview for example.
Looks like vim is one of the packages where -O3/graphite stuff really helps.

[pchome]

Ok, I found an variable in flag-o-matic.eclass which prevents all functions to be defined when it's set.

if [[ -z ${_FLAG_O_MATIC_ECLASS} ]]; then
_FLAG_O_MATIC_ECLASS=1
...

So something like # _FLAG_O_MATIC_ECLASS=1 emerge -1 package can be used for some packages as dirty hack. I checked this on media-libs/libsdl2::gentoo and looks like it works. There is error message /var/tmp/portage/media-libs/libsdl2-2.0.7/temp/environment: line 2355: strip-flags: command not found but not fatal and process goes on.

Trick with exporting variable in package.cflags also works: media-libs/libsdl2 "export _FLAG_O_MATIC_ECLASS=1"

[InBetweenNames]

I share your sentiments about these ebuild functions. I was actually just experimenting with GCC and Firefox, and I managed to build GCC with -O3, LTO, and PGO without issue. No codegen problems discovered so far, and I rebuilt Qt + KDE. Firefox needs more testing, as I am getting crashes--however these could be nailed down to a particular problematic set of flags (I recall -flifetime-dse being an issue with GCC 6).

I think it would be super nice to perhaps extend package.cflags with a NOFILTER command that will make strip-flags, replace-flags, filter-flags, and so on no-ops. A better long term solution of course would be to upstream the modified ebuilds. However, in my experience, it is difficult to get things accepted upstream for various reasons. My patch for gcc-config, for example, still hasn't even been reviewed, and I haven't received any word on my PGO ebuild patch for python (despite the fact that every binary distro uses PGO on python, and Gentoo particularly benefits from it).

I like the idea you propose with the _FLAG_O_MATIC_ECLASS hack -- I think we could go one step further by actually providing a "dummy" flag-o-matic in the repo that will be "injected" into the ebuild process and override the /usr/portage one, which sets that variable and provides no-ops for each function that flag-o-matic.eclass defines which modifes *FLAGS. This behaviour could be toggled with a NOFILTER directive like so:

www-client/firefox NOFILTER

If you wanted to be daring and have it apply to everything, I think this would work:

*/* NOFILTER

However this would give us the power to vet individual ebuilds. I'll still try to get the changes upstreamed, of course, but at least no one would have to wait to receive the benefits (provided they use GentooLTO).

I'll leave this issue open for more discussion -- if there are better ways of accomplishing the above I'm all ears.

[InBetweenNames]

I've put in the basics for this as a bashrc.d script. Enabling USE=override-flagomatic will cause strip-flags and replace-flags to become no-ops system wide. For legitimate uses of these functions, we can add probably resolve those cases in package.cflags or, alternatively roll our own .conf file to prevent the override from happening in certain cases.

[InBetweenNames]

I spoke too soon. I ended up adding overrides for append-flags and filter-flags after all. Unlike the bashrc.d built in approach, this one leaves messages in your build output that lets you know exactly what was overridden. So far, I have tested the packages you mentioned and also glibc. I can build glibc with full optimizations, but no LTO. Seems LTO isn't supported upstream yet on that package -- I get multiple definition errors when I tried the usual workarounds.

Next up will be a controllable mechanism for this. Ideally, something that could be added in package.cflags to prevent the overrides if necessary. Perhaps a shared variable between the scripts to toggle the behaviour. I'll think about it more.

[InBetweenNames]

For now, I've added a simple flag, LTO_ENABLE_FLAGOMATIC to re-enable it on a per-package basis.

[InBetweenNames]

I tested out a bunch of packages and found no problems with most, and added the exceptions as usual. I now have sys-libs/glibc running with all optimizations except LTO and AutoFDO.

[gcs-github]

Looking into this I saw that python ebuilds had the following line: use hardened && replace-flags -O3 -O2

This is perplexing to me and I'm strugging to find any context that lead for the line to be included in the first place. It dates back several years, back from when the portage tree was still versioned under CVS.

Do we know if there's a reason not to apply -O3 on python (or indeed any other package) in case we're building a hardened system?

[gcs-github]

My bad. I looked things up and apparently managed to miss the obvious comment in the ebuild itself. Sorry for the noise. Context is here: https://bugs.gentoo.org/50309

TL;DR: 2004-2005, stack smashing protection didn't play well with -O3 at the time on GCC 3.x.y. I'm unaware of any such problem still existing.

[InBetweenNames]

Nice catch! As far as I know, the only reason -O3 hasn't been used on a hardened system is because it exposes undefined behaviour in C and C++ programs more than -O2 does, which could lead to a security problem. But the fault there isn't really with -O3, it's with the code that is compiled using it.

[InBetweenNames]

Did some more looking at flag-o-matic.eclass. So it looks like these four functions are the only ones we should override, as append-c[xx]flags appear to do meaningful things, as well as append-ldflags. Now it's just a matter of testing this out and seeing if this is something we can enable by default after we get a few confirmed world rebuilds working properly.

[wolfwood]

I'll try to remember to kick off a world rebuild before bed.

…

On Mon, Oct 29, 2018, 20:23 Shane Peelar ***@***.*** wrote: Did some more looking at flag-o-matic.eclass. So it looks like these four functions are the only ones we should override, as append-c[xx]flags appear to do meaningful things, as well as append-ldflags. Now it's just a matter of testing this out and seeing if this is something we can enable by default after we get a few confirmed world rebuilds working properly. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#57 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB74jyPgKZ_oCjNsoEBuk7Qda2fxnPtks5up8YUgaJpZM4QFVdj> .

[wolfwood]

world rebuild succeeded, with the exception of inkscape, which I feel like has a race condition generating this header file? I saw this failure before at some point:

ege-color-prof-tracker.cpp:54:10: fatal error: helper/sp-marshal.h: No such file or directory
#include "helper/sp-marshal.h"
^~~~~~~~~~~~~~~~~~~~~

[InBetweenNames]

Strange! I have inkscape installed without issue. Could you post your build.log?

[wolfwood]

I just removed all my custom flag stripping and kicked off another world rebuild. I'll open a new issue for inkscape if it fails again, but it looks like the failure got cleaned up by portage already.

[sjnewbury]

It seems if glibc is built with "custom-cflags", then machine_name during the gcc fixincludes fails resulting in all "fixes" being applied. I'm not sure how or why yet, or whether it's specific to flags I have set.

[InBetweenNames]

Could you post a build.log?

[sjnewbury]

I'll try to reproduce it.

[jelinekto]

Glibc 2.27-r6 builds with current ltoize default CFLAGS and USE=override-flagomatic enabled but causes portage to fail with /bin/bash: symbol lookup error: /usr/lib64/gconv/ISO8859-1.so: undefined symbol: gconv_end. I've ran into this on two different machines (haswell and bdver2).

[ionenwks]

@InBetweenNames Yeah, append-flags() should be allowed more leniency rather than straight up override. Often it's used to add things like -fno-strict-aliasing -pthread -fopenmp -fPIC -DSOMETHING and other things like that (I've run a grep on *.ebuild to get the idea).

But then it sometimes add -O2 which supersede -O3, which is what we want stopped (overriding -O3 is truly rarely needed anymore, these issues did exist in the past but not so much anymore)

I'm no bash expert so it could be written better I'm sure, but (for example) I've been using this in my bashrc (I don't use bashrc-mv):

if declare -F append-flags >/dev/null; then
  if ! declare -F "orig_append-flags" >/dev/null; then
    eval "orig_$(declare -f append-flags)"
  fi
  append-flags() {
    local -a argv
    local -i argc=0
    while [ $# -gt 0 ]; do
      case "$1" in
        -O?) ewarn "append-flags(): prevented addition of $1";;
          *) argv[argc++]="$1";;
      esac
      shift
    done
    if [ $argc != 0 ]; then
      ewarn "append-flags(): allowing ${argv[@]}"
      orig_append-flags "${argv[@]}"
    fi
  }
fi

Which just calls the original function without arguments we don't want (done this way it also won't mess up spaces in single arguments), more could be added. It's not needed here but I do a similar thing with other functions like append-ldflags() since I use gold linker and also stop tc-ld-disable-gold() to remove -Wl,--default-symver (have to be careful with that one though, it's used for a reason).

The declare -F checks are needed in case flag-o-matic isn't loaded or if bashrc is sourced multiple times (it happens), in which case it does nothing and avoid errors.

Unlike other functions I wrap, technically append-flags() is simple and doesn't need its original copied, but I also wrap more complex functions and generally will prevent having to keep up with changes in those functions on portage's side.

Edit: oh yeah, should replace orig_ by something like LTOcopy_ to protect the namespace (the only thing that is set non-locally. Portage doesn't use orig_... but who knows.

[ionenwks]

@jelinekto I have a belief that -fipa-pta is bad on glibc and the one causing issues. It builds fine but I had a few unexpected behavior (while building gcc in my case) that go away when I remove -fipa-pta on glibc. This was on glibc 2.28 though.

Edit: Oh right, in your case, it could also be related to append-flags() preventing addition of -fno-strict-aliasing (in my case I'm already allowing this to be added, see above post)

[ionenwks]

If it can help, thought I'd share my painless experience with overriding flag-o-matic system-wide...

I've built a system almost without issues all the way up to firefox/wine-staging and everything else I personally use with flag-o-matic basic overrides (replace/filter/strip) on everything. append-flags() is setup the way mentioned in above posts. Nothing seems wrong ""yet"" (this is just a test system built in QEMU).

I do need to check for other flag-o-matic functions though, I hadn't took care of those yet and used the overrides here as reference.

That aside there's some differences with my setup:

• testing version of packages only ~* (dated yesterday) system-wide
• glibc 2.28 is used
• openssl 1.1.1 is unmasked and used
• ABI_X86="32 64" system-wide
• default python target is python3_7
• default/linux/amd64/17.1 profile without the symlink /lib
• gold is the default linker, don't think a single binary was linked with bfd on entire system (it is supposed to cooperate better with -fno-plt used below, otherwise -fno-plt could be a performance hit rather than gain. gold seems like a good deal to have either way)
• CFLAGS="-march=native -O3 -flto=8 -fno-plt -falign-functions=32 -fgraphite-identity -floop-nest-optimize -g0" (no ipa-pta / semi-inter yet, will revisit those later -- cpu is a old i7 2600 so -march=sandybridge, I intend to replace it soon + fresh install and that's why I'm experimenting)
• custom USE flags on a fair amount of packages which may have avoided problems, generally minimalist flags (not using a desktop profile, and set X, minimal, vim-syntax, and jack system-wide with a few opt-out on some packages)
• gcc doesn't have pie by default, nor enable fortify-source=2 by default that is added in a gentoo patch, relro also off by default, and also prevented some packages from using additional security options (I'm mostly anti-security, please don't mind nor copy me -- only mentioning in case it affected builds)

This is my own thing but -fno-plt needed to be removed for the 32bit version of ffmpeg, nss, and pcsx2, likely happen with a few other 32bit multilib/packages (on those packages I make use of CFLAGS_x86/CFLAGS_amd64 to use -fno-plt only on 64bit)
Update: Noticed a bit late because I wasn't messing with it, but looks like -fno-plt is bad on xorg-server as well. I assume it's related with how it requires -z lazy as well (no-plt interferes with lazy bindings).

gold only needed a workaround for berkeley db (only because I don't let the ebuild override gold, which then proceed to use a bfd-supported-only option), even glibc which used to be notorious for failing with it is fine (the ebuild attempts to override it, but I blocked that)
Edit: I keep forgetting, also needed a gold workaround on efivar, it links against its own library (which uses libdl), and then uses functions from libdl without having specified -ldl, and gold won't take it. So need to add -ldl. efivar also uses a gcc spec file so may be uncooperative

In case of firefox, did remove the normally-stripped -flto and instead let it sort it out with --enable-lto=thin added with the lto USE flag. Also used RUSTFLAGS="-Ctarget-cpu=native". Using all the system-* USE flags for libraries.

Similar deal with ffmpeg, should remove -flto and use EXTRA_FFMPEG_CONF="--enable-lto" instead and it'll sort out itself where to use it (preventing issues with some USE flags I believe). The ebuild already enables this but doesn't strip -flto which can lead to issues (it doesn't do this regardless of overrides)

I've also let firefox build with -O3 (custom-optimization USE flag is needed or else it tries to add -O2 after normal CFLAGS), it appears to work fine but staying wary because previous issues I've had with -O3 on firefox were very sneaky (aka didn't segfault but had strange behavior in the browser, this was years ago though)
Update: while it seem to work fine on the surface, there's some differences that occur only with -O3 (notably misc errors in the console), I advise against using custom-optimization at the time of this writing still.

Kernel was also built with the same CFLAGS+gold minus -flto (and said kernel was used through the entire build process, but this is in QEMU, haven't received a real boot yet -- big "at your own risks" when doing this on kernel, can lead to lost data and damaged hardware).

I use xorg+xfce4 and no issues there, don't ask me about wayland/gnome/kde/qt.

On a side-note, looking at current workarounds in the overlay that I built the package for:

• perl 5.26 built fine without removing -flto
• libaio did too but that led to issues with another package (similar deal as fuse, so should keep the workaround)
• qemu 3.0.0 built fine with -flto, I also didn't add -fno-strict-aliasing -- at least was fine with the USE flags I used, I did disable a few things.
• ncurses 6.1 didn't need any flags changes at all, built fine as-is
• for firefox/ffmpeg see above

For the rest I've had similar problems and had to do the same workarounds, not that I've build every packages mentioned in the ltoworkarounds.conf file.

Now I just need to mess with the system and see if anything strange, so far so good.

[jelinekto]

@ionenwks Interesting, I just tried building both 2.27 and 2.28 with -fno-strict-aliasing enabled, -fipa-pta disabled and I keep getting the same symbol lookup error. Might be something wrong specifically with my setup.

[ionenwks]

@jelinekto Hrmm, I thought it may be a locale difference (since I don't use ISO8859-1 and only have UTF-8 locales generated), so I made one, set it and started bash but no problems still. Afraid I don't know what's causing it.

[InBetweenNames]

Wow, phenomenal work and writeup!

@ionenwks I agree -- we should be smarter about how we override append-flags. For now, we can enable exclusions on a per-package basis with LTO_ENABLE_FLAGOMATIC=yes. I think many of your insights could be integrated into GentooLTO as a whole, including -fno-plt and enforced usage of Gold, which is something that I ask about in virtually every package related bug report anyway.

@jelinekto Even with LTO_ENABLE_FLAGOMATIC=yes defined, on my system gconv_end is missing from my /usr/lib64/gconv/ISO8859-1.so, but I haven't noticed any issues yet. However, other .so files in that directory, including all Unicode ones (which my system uses), do have gconv_end defined. I'm unsure if this is an optional symbol or not. Could you post your emerge --info and logs relevant to the error?

[ionenwks]

Seems to be the same for me, no gconv_end in ISO8859-1.so but it's there in UTF ones. Either our setup is not using those files (thus no errors) or maybe it's related to bash/ncurses/readline and they need a rebuild? Could be expecting old things.

[ionenwks]

I just checked on my Debian system and there's no gconv_end there either, and Debian should be as normal as it gets. So I'm not sure why it's looking for it.

[InBetweenNames]

Hah, it actually just happened to me! It turns out it's -fno-semantic-interposition causing this one. How, exactly, I'm not sure -- but my system works fine with I build glibc without that flag. I added a workaround for it in master. @jelinekto Could you try it out?

[jelinekto]

@InBetweenNames Thanks, the workaround solves the problem.

[ionenwks]

@InBetweenNames It may not be that surprising, it mostly affect shared library code (libc sure is one) and it's supposed to allow to break some standards with ELF semantics. Personally was already not using that one (as mentioned above), so guess not surprising it works for me.

[InBetweenNames]

What I find interesting is that clang actually uses -fno-semantic-interposition by default. I figure if they're using it, it must be okay in the vast majority of cases. I can totally see glibc being one of those edge cases however.

[ionenwks]

Yeah at the low level I wouldn't trust anything that breaks standards. I do want to throw -ffast-math on everything anyway though.

[InBetweenNames]

I'm wondering if we should perhaps have a couple of GentooLTO defaults. A "plain" default that is just -O3, Graphite, and LTO and an experimental one that enables all the bells and whistles, but will definitely require more attention for users. As it stands, I think virtually everyone is using the "plain" default without issue. Clear Linux uses -fno-semantic-interposition and some -ffast-math-type flags, as well as -falign-functions=32 of course.

Perhaps something like:

• make.conf.lto.strict for strict compliance
• make.conf.lto.experimental for all the bells and whistles
• make.conf.lto.intel which could be sourced under the above two for recent Intel processors
• make.conf.lto.clearlinux for Clear Linux flags (which implies the above)

[ionenwks]

^ I do think that would be good, especially if the number of flags to pick from keep increasing. It'd be less confusing to have presets. When flags (and stripping prevention of) are known to work for most people along with needed workarounds, they could be moved to a default preset from experimental. Something like "strict" should probably stay more limited though, or at least in the don't-break-standards sense.

On a related note, it would be handy to have a way to disable workarounds on-the-fly without editing files. These need to be re-tested regularly to know if still needed or it'll just become a bunch of outdated stripping flags like those that remain in ebuilds.

There's also the issue of which version of packages people are using (mostly just stable vs ~testing). Stable packages will typically need more workarounds (when it comes to optimizations that is) and may need a separate file, unless supporting only one of the two (given glibc 2.28 currently gets recommend in the README.md it does semi-force to use ~testing given it breaks a lot of stable packages which led to that "Multiple build failures" issue). Might be good to make the stance clear.

Edit: Speaking of README.md (and others), should probably remove all references of -Wl,--hash-style=gnu sometime, as you may already know it's enforced by default through --enable-default-hash-style=gnu on binutils since 2.30-r2, unless on mips anyway.

---

To go back on this:

I've also let firefox build with -O3 (custom-optimization USE flag is needed or else it tries to add -O2 after normal CFLAGS), it appears to work fine but staying wary because previous issues I've had with -O3 on firefox were very sneaky

Guess it's still no good, don't do it. It works on the surface but the console is outputting additional errors with -O3 that don't show with -O2. There's no need to actually strip -O3 from CFLAGS though, that USE flag works regardless of flag-o-matic overrides and is only for -O* (it's handled by mozcoreconf-v6.eclass), other optimizations are still applied but it's pretty delicate so I'd go easy on them. Got fixed but in the past I even had it segfault because of -mavx enabled by -march=native.

I do wonder which optimization it dislikes from -O3 exactly but given how long firefox takes to build I have no will to track it down, there's probably several affected pieces of code too given it's a long-standing issue (I'll just call it a day and stick to -O2)

Edit: on a new side-note, don't use -fno-plt on xorg-server, its reliance on lazy bindings seems to interfere and breaks modules

[barolo]

I'm confused about the whole overriding flagomatic ordeal, what and where should I put to actually allow firefox to build firefox with lto [with gcc],?

[ionenwks]

I'm confused about the whole overriding flagomatic ordeal, what and where should I put to actually allow firefox to build firefox with lto [with gcc],?

There's no need to override anything for firefox, flagomatic should be left to work as intended and strip -flto from your flags (there's already a environment variable in ltoworkarounds.conf to let it be).

For gcc+LTO firefox, you only need to set the lto USE flag, and with recent updates to gentoo's default ebuilds you can even do gcc+LTO+PGO (which is what I use since firefox 65) by also adding pgo USE. That will let firefox decide where and how to properly use -flto without breaking (it will even use -flto=<cores/threads count> which use a stupidly high amount of RAM, if building on tmpfs I run out of memory with 12 threads with my small 16GB of RAM, works fine if I force it to 6 threads with a patch though).
Edit: be warned that setting pgo effectively doubles the time firefox takes to build.

[barolo]

@ionenwks thanks. Compiled it with pgo, O3, lto and it works [65.0.2], previously it would segfault, not sure if PGO is actually working, as I'm not seeing it running for profiling, possibly instantly closes, seen it before. Plus it's 15-20% slower from averaged test than clang 8 /lto/pgo/O3 build. Even more so in graphical benchmarks. Guess I'm saying gbye to building browsers with gcc.
[libxul is 40% smaller though]
Sidenote. chrome/chromium clang builds with lto use -fwhole-program-vtables for decent speed gain. It's impossible to use with ffox stripping it in the beginning.

[ionenwks]

@barolo I'd advise against -O3 with gcc, it doesn't segfault but last I checked it caused additional errors in stdout / browser console, and there may be other strange behavior. Firefox has some asm that's not clear about certain things and gcc can't know it's unsafe to optimize (few nice things like -ftree-vectorize still seem safe).

That aside, you may not have used -O3 at all, it gets stripped unless you specify custom-optimization USE flags. -O3 is fine with clang last I know of, I think it's also used by default by firefox when it's clang, it just doesn't do the same optimizations as gcc.

And yeah, firefox is putting all optimizations efforts geared toward clang (clang+LTO+PGO is the default build for releases) and barely support/test gcc at all anymore and only an old version. But did you do those benchmarks yourself? clang+LTO+PGO builds tend to get compared to a non-lto non-pgo gcc build in benchmarks given that was broken for a while, not that I'd be surprised if it's faster anyway given the lack of effort with gcc.

And (if you meant seeing it visually) you wouldn't see it run for profiling, not only does firefox have a headless mode now, but gentoo even forces you to install xvfb (it used to but not sure if xvfb still get used for profiling, I think it uses the new headless mode). Not that I'm 100% sure it's working as intended myself, especially with gcc, given that doesn't really get tested upstream.

[barolo]

@ionenwks about profiling, what I meant is that I was watching console's output in the background , you can see when it runs tests in headless mode, it doesn't seem to happen in case of gcc.
About -O3, I'm aware that custom-cflags is necessary, you can actually go to about:buildconfig in firefox to see what it was built with.
And yes, I've made sure that both are ltoed,pgo,O3 then averaged several runs of speedometer, jetstream, octane, gcc one is slower. In case of motionmark there's wider gap in favour of clang, for some reason.
[my previous clang builds were done with clang7, clang8 lto seems to produce smaller binary, which was one of advantages of gcc, no so much anymore, it's 65MB vs 80MB [clang] for libxul now

[ionenwks]

@barolo I see, maybe I'll switch back to clang as well then. I used it for a while but when PGO started to not segfault with gcc again I went back to it. I do have my doubts as to whether profiling is being used correctly anyway, especially when gcc sometimes continue silently without optimizing if there's a problem with the profile.

Shame browsers are such a pain to experiment with, both huge code base with convoluted build systems and sensitive to optimizations, makes me not do the tests I should just be doing.

[barolo]

@ionenwks My conclusion is that if you're building ffox with GCC for a speed gain you're wasting your time, since it's no faster than upstream binary builds which are lto/pgoed , at least in the current state of things. It's even worse with chromium [ which is even more fine-tuned for clang ]

I've been reading Honza Hubicka's blog and apparently there are fixes/discussions about bringing GCC builds up to par, we'll see where it ends. He also outlines where are the issues