Special Meeting - PGO deep dive

[mhdawson]

Who

@nodejs/benchmarking

When

Wed Feb 15, 1PM PST

Where

link for participants: https://hangouts.google.com/hangouts/_/ytl/NH_6tyz8abYim69liT4bERGkALuos4J4U8Wl8QJDTBw=?eid=100598160817214911030

For those who just want to watch: http://youtu.be/pC0Boxl2EPA

youtube admin page: https://www.youtube.com/my_live_events?filter=scheduled

Agenda

Deep dive presentation from Kunal on PGO. He's done some investigation and will give us an update on what PGO is, how it works and results he's seen so far. He's looking at it based on initial interest shown in nodejs/node#1409.

[mhdawson]

@bnoordhuis @rvagg you might be interested in this based on past interested in issue 1409

[bnoordhuis]

I don't really have anything to contribute. The big blocker was (and presumably is) realistic benchmarks.

[mhdawson]

@nodejs/benchmarking need more people to fill in the doodle with availability.

[mhdawson]

@williamkapke can you add to general node calendar

[williamkapke]

Added! 19:00 GMT on 2017-02-02

[mhdawson]

Meeting is live, just waiting for participants to join.

[gareth-ellis]

Trying to join....

[mhdawson]

@kunalspathak you going to join ?

[mhdawson]

Our presenter could not make it so need to reschedule.

@kunalspathak could you send out a new doodle to pick a new time ?

[kunalspathak]

@mhdawson , Sorry about that. For some reason, i didn't see any updates from this thread in my mail box so didn't realize the meeting was scheduled for today. I will schedule a meeting sometime next week. Sorry about that.

[kunalspathak]

http://doodle.com/poll/g8y6z32xtgs8ry72

[mhdawson]

Did you turn on timezone support ? I can't tell what timezones are my local time (EST) or something else ?

[kunalspathak]

I didn't updated timezone when i opened my account. I did it now and reconfigured the dates. Can you see the right timezone now?

[mhdawson]

I think you have to enable it as part of creating the doodle and I don't think you can do it after the effect.

The first time I see is 11 AM on Thursday but no info on timezone.

[kunalspathak]

Can you try this?

http://doodle.com/poll/ukkknw3rb5d644xs

[gareth-ellis]

I can see time zone on that :)

[mhdawson]

That worked for me as well. Filled in my availability.

[kunalspathak]

@nodejs/benchmarking . We need more people to participate.

[ThePrimeagen]

Done

…

On Wed, Feb 8, 2017, 9:38 AM Kunal Pathak ***@***.***> wrote: @nodejs/benchmarking <https://github.com/orgs/nodejs/teams/benchmarking> . We need more people to participate. — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEQGvlb0UumParj9G6wAzo3Juj47ikQZks5raf2xgaJpZM4LiSCt> .

[gareth-ellis]

Have replied (again). Seems my first response was lost

[kunalspathak]

@michaelbpaulson , looks like best time we can have this is on 2/10 11AM PST. Is it possible for you to join then?

[ThePrimeagen]

~~Sure~~ I'll literally be boarding an airplane at that point. That time will not work. You can count me out

[kunalspathak]

@mhdawson , I have closed the poll and scheduled the meeting for Feb 10th at 11 AM PST. Can you setup the hangout page and live streaming?

[mhdawson]

Updated hangout to reschedule for Feb 10 11AM. We should be good to go. If there are any problems starting the event/recording I'll post in this issue as we start the meeting.

[kunalspathak]

Updated hangout to reschedule for Feb 10 11AM

I see in the issue comment above that it reflects Thursday Feb 9, 11AM EST. May be you want to correct the date/time?

[sathvikl]

@kunalspathak I have a conflict at that time.

Below are the PGO instructions I tried on Ubuntu 14.10 (gcc (Ubuntu 4.9.1-16ubuntu6) 4.9.1) and
Ubuntu 16.04 (gcc (Ubuntu5.4.0-6ubuntu1~16.04.4) 5.4.0)

To build a node framework with PGO instrumented counters: apply this patch

diff --git a/common.gypi b/common.gypi

index 811a7b3..3747b8d 100644 
--- a/common.gypi 
+++ b/common.gypi 
@@ -253,8 +253,9 @@ 
              'ldflags': [ '-mx32' ], 
            }], 
            [ 'target_arch=="x64"', { 
-            'cflags': [ '-m64' ], 
-            'ldflags': [ '-m64' ], 
+            'cflags': [ '-m64','-fprofile-generate'], 
+            'ldflags': [ '-m64', '-lgcov' ], 
+            'libraries': ['-lgcov'] 
            }], 
            [ 'target_arch=="ppc" and OS!="aix"', { 
              'cflags': [ '-m32' ], 

This will create
node/out/Release/obj
node/out/Release/obj.host
node/out/Release/obj.target
obj.target will contain the .o and .gcda files

To use the profiled information and build an optimized binary for the workload apply the below patch

diff --git a/common.gypi b/common.gypi

index 811a7b3..3747b8d 100644 
--- a/common.gypi 
+++ b/common.gypi 
@@ -253,8 +253,9 @@ 
              'ldflags': [ '-mx32' ], 
            }], 
            [ 'target_arch=="x64"', { 
-            'cflags': [ '-m64' ], 
-            'ldflags': [ '-m64' ], 
+            'cflags': [ '-m64','-fprofile-use', '-fprofile-correction',  '-Wno-error=coverage-mismatch'], 
+            'ldflags': [ '-m64', '-lgcov' ], 
+            'libraries': ['-lgcov'] 
            }], 
            [ 'target_arch=="ppc" and OS!="aix"', { 
              'cflags': [ '-m32' ], 

profile-correction is needed since node has multiple threads, no-coverage-mismatch is needed currently since code relocations take place in v8.

[kunalspathak]

Thanks @sathvikl for the instructions. I appreciate it.

[mhdawson]

Hangouts is up and running, just waiting for presenter and others to join.

[mhdawson]

Trying to see if there is a way to get a hold of @kunalspathak

[gareth-ellis]

Which time zone is this supposed to be ? EST or PST? I understood to be PST so another 3 hours time?

[CurryKitten]

@mhdawson Unfortunately, I can't make those times in the new doodle, so I'll watch later on YouTube

[kunalspathak]

@mhdawson , can you schedule the hangout meeting for Feb 15th, 1 PM PST ?

[mhdawson]

Ok meeting and info above updated to Feb 15th 1PM PST

[kunalspathak]

@mhdawson , the title again says 1 PM EST. It should be 1 PM PST. 😃

[gareth-ellis]

Updated it

[mhdawson]

@gareth-ellis thanks.

[mhdawson]

New link for hangout so we can get a recording https://hangouts.google.com/hangouts/_/ytl/NH_6tyz8abYim69liT4bERGkALuos4J4U8Wl8QJDTBw=?eid=100598160817214911030

[mhdawson]

New link for those that want to watch http://youtu.be/pC0Boxl2EPA

Old link was not giving option to record.

[mhdawson]

Just waiting for Kunal and then we'll get started.

[kunalspathak]

I am getting error when trying to access the hangout link.

[mhdawson]

We posted a new link are you using that ? Gareth is in.

[kunalspathak]

Yes, i am using this link:

https://hangouts.google.com/hangouts/_/ytl/NH_6tyz8abYim69liT4bERGkALuos4J4U8Wl8QJDTBw=?eid=100598160817214911030

[mhdawson]

My only other suggestion is to try different browser as people seem to have intermittent problems getting in

[mhdawson]

Looks like the right link to me.

[kunalspathak]

Errors on Edge, Chrome and Firefox.

[kunalspathak]

[mhdawson]

Do you have a different machine you can try, I know that has fixed it for some people

[kunalspathak]

Let me try.

[gareth-ellis]

That looks like the sort of error I've had in the past. I am however using chrome now (same machine that I previously had issue with)

[mhdawson]

Ok starting up the meeting now.

[mhdawson]

Meeting was held closing.

[kunalspathak]

One of the follow-up from this meeting was to evaluate the impact of PGO on core benchmarks if we just use minimal training set. So i did an experiment of using Techempower and acme-air as training set and did a comparison of performance gain we get with PGO. Here are the results. To summarize, I do see similar perf win in core benchmarks.

[sathvikl]

@kunalspathak Thanks for posting.

Which compiler/version/OS was this on ?

When do you PGO are you seeing coverage errors, I'm working on addressing that.

Are you the PGO trained binary, trained/profiled with Techempower and acme-air, on all these benchmarks ? If you train the PGO with array.js (as an example) and run array.js benchmarks what improvement differences would you observe ?

[kunalspathak]

Which compiler/version/OS was this on ?

I should have mentioned this. This is on Windows, VS2015, based on nodejs/node@b26a469

When do you PGO are you seeing coverage errors, I'm working on addressing that.

No. I didn't see any. What type of coverage errors do you mean?

Are you the PGO trained binary, trained/profiled with Techempower and acme-air, on all these benchmarks ?

The binaries were trained using Techempower and Acmeair. Once I produced the binaries, I ran no-pgo vs. pgo comparison on core benchmarks.

If you train the PGO with array.js (as an example) and run array.js benchmarks what improvement differences would you observe ?

I didn't try this specifically, but I am sure that this will definitely show the improvements.

[sathvikl]

With GCC 5.x I was getting coverage errors, meaning the code was relocated so I needed the -coverage-mismatch flag. I saw a bugzilla ticket where the same was fixed for mozilla engine.

This is great if PGO trained binaries can give overall 10% improvement for all frameworks.

I meant to explain that micro benchmarks executed by binaries trained by the same micro-bench would probably give an even higher improvement. So It would be good to know what the performance degradation is by using binary trained by Acme air and Techempower.

Either way, If there is no performance regression on any framework then the release binary versions of nodejs could possibly use PGO to train the binaries.

Does VS 2015 give a summary report of what changes were done as a result of Profile information? If you have it can you please share it.

[kunalspathak]

Does VS 2015 give a summary report of what changes were done as a result of Profile information?

It does give summary of no. of functions optimized, etc, which I don't see useful except I feel good if more functions were optimized 😄

@gareth-ellis was also curious in PGO meeting if we can get actual function names that were optimized. I will follow up with VC++ team and get back to you guys on it.

 0 of 0 ( 0.0%) original invalid call sites were matched.
  0 new call sites were added.
  240 of 112751 (  0.21%) profiled functions will be compiled for speed, and the rest of the functions will be com
  piled for size
  544530 of 1575974 inline instances were from dead/cold paths
  112749 of 112751 functions (100.0%) were optimized using profile data, and the rest of the functions were optimi
  zed without using profile data
  7861923896240 of 7861923896240 instructions (100.0%) were optimized using profile data, and the rest of the inst
  ructions were optimized without using profile data

[kunalspathak]

Alright, so regarding summary report, we can obtain below information:

• List of functions inlined. However the list is gigantic. It is output as part of build process and when I dumped it into the file, the file size was approx. 550 MB. So don't think it will be super useful.
• Summary of functions that were profiled and how hot they were during profiling.
• Summary of functions that were not optimized using profile data and reason.
• Names of functions that were never executed while recording the scenario.

[sathvikl]

From uarch analysis, I had found that approx. the number of instructions executed had reduced by 9-10% and the PGO speedup was around 10%.

I like that VS can give us this info, I was not able to extract it easily out of GCC. GCC just prints on a per function basis what optimizations were done, just the raw count and the same info can be obtained for the overall compilation.
What options need to be enabled in VS to get this information ?

It looks like 0.34% of functions were found to be inlined at dead/cold paths so I was thinking not inlining may have helped but 0.34% or whatever the contribution is from these functions seems to be low.

If you could share the exact functions through a one drive that would be very helpful.

[kunalspathak]

• List of functions inlined. However the list is gigantic. It is output as part of build process and when I dumped it into the file, the file size was approx. 550 MB. So don't think it will be super useful.

Passing linker flag /d2:-inlinelog gives that information

• Summary of functions that were profiled and how hot they were during profiling.

pgomgr /summary <binary_name_prefix>.pgd

• Summary of functions that were not optimized using profile data and reason.

Set environment variable set PGU_LOG=1 and it will dump <binary_name_suffix>.pgu.log next to the binary.

• Names of functions that were never executed while recording the scenario.

pgomgr /detail /summary <binary_name_prefix>.pgd

I have placed all the files here.