From ab45c0eee8190507639771bebe6e4fda203b1ea8 Mon Sep 17 00:00:00 2001 From: Josh Gavant Date: Mon, 18 Apr 2016 09:33:21 -0700 Subject: [PATCH] Expand working group and repo to all diagnostics. Add additional top level to repo with subfolders for diagnostic domains. Add initial README.md for each domain. Add some initial docs and samples for domains beyond tracing. Move and minor cleanup of AsyncWrap docs and background info on TRACE_EVENT. --- README.md | 23 +++++++++-- debugging/README.md | 29 ++++++++++++++ docs/README.md | 22 ----------- docs/perf-events/README.md | 6 --- docs/platform-tracing/README.md | 29 -------------- heap-memory/README.md | 10 +++++ profiling/README.md | 26 +++++++++++++ {docs => tracing}/AsyncWrap/README.md | 0 .../AsyncWrap/example-trace/run.js | 0 .../AsyncWrap/example-trace/trace.js | 0 tracing/README.md | 7 ++++ tracing/os-tracing/README.md | 26 +++++++++++++ .../vm-tracing}/README.md | 31 +++++++++------ tracing/vm-tracing/v8-tracing.md | 39 +++++++++++++++++++ 14 files changed, 176 insertions(+), 72 deletions(-) create mode 100644 debugging/README.md delete mode 100644 docs/README.md delete mode 100644 docs/perf-events/README.md delete mode 100644 docs/platform-tracing/README.md create mode 100644 heap-memory/README.md create mode 100644 profiling/README.md rename {docs => tracing}/AsyncWrap/README.md (100%) rename {docs => tracing}/AsyncWrap/example-trace/run.js (100%) rename {docs => tracing}/AsyncWrap/example-trace/trace.js (100%) create mode 100644 tracing/README.md create mode 100644 tracing/os-tracing/README.md rename {docs/v8-tracing => tracing/vm-tracing}/README.md (82%) create mode 100644 tracing/vm-tracing/v8-tracing.md diff --git a/README.md b/README.md index 40b6444..cd6e911 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,24 @@ -# tracing-wg +# Diagnostics Working Group +The goal of this WG is to ensure Node provides a set of comprehensive, documented, extensible diagnostic protocols, formats, and APIs to enable tool vendors to provide reliable diagnostic tools for Node. -Tracing Working Group +Work is divided into several domains: +- [Tracing](./tracing) +- [Profiling](./profiling) +- [Heap and Memory Analysis](./heap-memory) +- [Step Debugging](./debugging) -# Members +Background, reference documentation, samples, and discussion for each domain is contained within its folder. +Work needed includes: +- Collect, understand, and document existing diagnostic capabilities and entry-points throughout Node, V8, and other components. +- Collect and document projects and products providing diagnostics for Node with brief description of their technical architecture and sponsoring organizations. +- Identify opportunities and gaps, then propose and implement solutions. + +### Logistics +- Monthly Meetings +- Biannual F2F + +### Members * @bnoordhuis * @sam-github * @othiym23 @@ -28,3 +43,5 @@ Tracing Working Group * @matthewloring * @pmuellr * @jeffolfert +* @joshgav + diff --git a/debugging/README.md b/debugging/README.md new file mode 100644 index 0000000..727dc3b --- /dev/null +++ b/debugging/README.md @@ -0,0 +1,29 @@ +## Debugging API +- [Chrome Debugging Protocol](https://chromedevtools.github.io/debugger-protocol-viewer/) +- [V8 Debugging Protocol](https://github.com/v8/v8/wiki/Debugging-Protocol) + +Node currently relies on V8's internal Debug API and associated commands and events. The Debug API is published at [v8/include/v8-debug.h](https://github.com/v8/v8/blob/master/include/v8-debug.h), and the message protocol is documented in prose in [the V8 wiki](https://github.com/v8/v8/wiki/Debugging-Protocol) and in code in [v8/src/debug/debug.js#L2341-2372](https://github.com/v8/v8/blob/master/src/debug/debug.js#L2341-L2372). + +Node also provides an agent [node/src/debug-agent.h](https://github.com/blob/master/src/debug-agent.h) which relays requests, responses, and events through a TCP socket, and a command-line debugger [node/lib/_debugger.js](https://github.com/blob/master/lib/_debugger.js). + +The Chrome/V8 team has deprecated the internal V8 API and command set and proposed replacing it with a subset of the Chrome Debugging Protocol (CrDP), see https://github.com/nodejs/node/issues/2546. CrDP is documented at [https://chromedevtools.github.io/debugger-protocol-viewer/]() and the backing GitHub repo. + +The [cyrus-and/chrome-remote-interface](https://github.com/cyrus-and/chrome-remote-interface) library provides a JS proxy for CrDP. + +## Step Debugging Tools +Name | Sponsor +-----|-------- +[node-inspector][] | StrongLoop +[JetBrains WebStorm][] | JetBrains +[Visual Studio Code][] | Microsoft +[Node CLI Debugger][] | Node Foundation +[Chrome DevTools][] | Google +[Theseus][] | Adobe Research + +[node-inspector]: https://github.com/node-inspector/node-inspector +[JetBrains WebStorm]: https://www.jetbrains.com/help/webstorm/2016.1/running-and-debugging-node-js.html +[Visual Studio Code]: https://github.com/Microsoft/vscode +[Node CLI Debugger]: https://nodejs.org/api/debugger.html +[Chrome DevTools]: https://github.com/ChromeDevTools/devtools-frontend +[Theseus]: https://github.com/adobe-research/theseus + diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index aa07b88..0000000 --- a/docs/README.md +++ /dev/null @@ -1,22 +0,0 @@ -Node.js tracing - documentation for tracing components -================================================================================ - -Components: - -* [AsyncWrap](AsyncWrap/README.md) - - AsyncWrap is the central gateway in Node.js to most things that are async, - and so will be a likely place for tracing to be added. - -* [perf-events](perf-events/README.md) - - Performance-related tracing events - -* [platform-tracing](platform-tracing/README.md) - - How does platform-specific tracing like DTrace, LTTng, and ETW fit into - the picture - -* [v8-tracing](v8-tracing/README.md) - - Details on new tracing bits being worked on for Chrome that affect V8 - ----- - -If you have more info to provide, please send us a pull request! diff --git a/docs/perf-events/README.md b/docs/perf-events/README.md deleted file mode 100644 index ec33f54..0000000 --- a/docs/perf-events/README.md +++ /dev/null @@ -1,6 +0,0 @@ -Node.js tracing - performance related tracing events -================================================================================ - ----- - -If you have more info to provide, please send us a pull request! diff --git a/docs/platform-tracing/README.md b/docs/platform-tracing/README.md deleted file mode 100644 index 1631d85..0000000 --- a/docs/platform-tracing/README.md +++ /dev/null @@ -1,29 +0,0 @@ -Node.js tracing - platform specific tracing -================================================================================ - -Need a general statement of direction of if and how platform-specific tracing -factors into Node.js, especially given [v8-tracing](../v8-tracing/README.md). - -Popular platform specific tracing stories: - -* [DTrace](http://dtrace.org/blogs/about/) (illumos / Solaris / Mac / other) - - Some blog posts on DTrace and Node.js are available on the - [DTrace blog](http://dtrace.org/blogs/blog/category/node-js/) - -* [LTTng](http://lttng.org/) (Linux) - - Some info on using LTTng with Node.js are available in the presentations - [Tracing Node on Linux with LTTNG](http://nearform.github.io/tracing-node-lttng-nodejsdublin/) - -* [ETW](https://msdn.microsoft.com/en-us/library/windows/desktop/aa363668.aspx) - (Windows) - - Some information was provided by Bill Ticehurst (@billti) in - [issue #10](https://github.com/nodejs/tracing-wg/issues/10#issuecomment-137145822) - about how he integrated ETW into Node.js for Windows - - ----- - -If you have more info to provide, please send us a pull request! diff --git a/heap-memory/README.md b/heap-memory/README.md new file mode 100644 index 0000000..14f7a93 --- /dev/null +++ b/heap-memory/README.md @@ -0,0 +1,10 @@ +## Heap and Memory Analysis +The [nodejs/post-mortem](https://github.com/nodejs/post-mortem) WG is responsible for this domain. + +### Heap Analysis +- [mdb_v8](https://github.com/joyent/mdb_v8) +- [node-heapdump](https://github.com/bnoordhuis/node-heapdump) +- [llnode](https://github.com/indutny/llnode) +- [IBM IDDE](https://www.ibm.com/developerworks/java/jdk/tools/idde/) +- [Snapshot Utils](https://github.com/bmeck/snapshot-utils) + diff --git a/profiling/README.md b/profiling/README.md new file mode 100644 index 0000000..974d240 --- /dev/null +++ b/profiling/README.md @@ -0,0 +1,26 @@ +## Profiling for Node +- CPU Profiling + - V8 CPU Profiler: [v8/include/v8-profiler.h](https://github.com/v8/blob/master/include/v8-profiler.h) + - Accessible globally through `--prof` runtime flag, basic processing and display through `--prof-processor` runtime flag + - Intel vTune profiler. Accessible through `--enable-vtune-profiling` compile-time flag. + +- Heap Profiling + - V8 Heap Profiler: [v8/include/v8-profiler.h](https://github.com/v8/blob/master/include/v8-profiler.h) + - Heap memory usage stats are available through built-in 'v8' module. + +Name | Sponsor +-----|-------- +[v8-profiler][] | StrongLoop +[Chrome DevTools][] | Google + +[v8-profiler]: https://github.com/node-inspector/v8-profiler +[Chrome DevTools]: https://github.com/ChromeDevTools/devtools-frontend + +### Stack Analysis +- [0x](https://github.com/davidmarkclements/0x) +- [StackVis](https://github.com/joyent/node-stackvis) + +## Docs +[guides/simple-profiling](https://nodejs.org/en/docs/guides/simple-profiling/) +[paulirish/automated-chrome-profiling](https://github.com/paulirish/automated-chrome-profiling) + diff --git a/docs/AsyncWrap/README.md b/tracing/AsyncWrap/README.md similarity index 100% rename from docs/AsyncWrap/README.md rename to tracing/AsyncWrap/README.md diff --git a/docs/AsyncWrap/example-trace/run.js b/tracing/AsyncWrap/example-trace/run.js similarity index 100% rename from docs/AsyncWrap/example-trace/run.js rename to tracing/AsyncWrap/example-trace/run.js diff --git a/docs/AsyncWrap/example-trace/trace.js b/tracing/AsyncWrap/example-trace/trace.js similarity index 100% rename from docs/AsyncWrap/example-trace/trace.js rename to tracing/AsyncWrap/example-trace/trace.js diff --git a/tracing/README.md b/tracing/README.md new file mode 100644 index 0000000..1959f9c --- /dev/null +++ b/tracing/README.md @@ -0,0 +1,7 @@ +## Tracing +- [AsyncWrap](./AsyncWrap/README.md) - Most native objects are represented indirectly by the [AsyncWrap class](https://github.com/nodejs/node/blob/master/src/async-wrap.h), so hooks have been added to that class to trace lifecycle (init/destroy) and callback activity (pre/post) related to these objects. + +- [OS Tracing](./os-tracing/README.md) - LTTng (Linux), SystemTap (Linux), DTrace (OSX), ETW (Windows) + +- [VM Tracing](./vm-tracing/README.md) - Tracing native to JS VMs, such as V8's TRACE_EVENT. + diff --git a/tracing/os-tracing/README.md b/tracing/os-tracing/README.md new file mode 100644 index 0000000..8e6598b --- /dev/null +++ b/tracing/os-tracing/README.md @@ -0,0 +1,26 @@ +## OS Tracing + +* [DTrace](http://dtrace.org/) + Available on IllumOS, Solaris, Mac + + - The [DTrace blog](http://dtrace.org/blogs/blog/category/node-js/) has some articles on using DTrace with Node.js + +* [LTTng](http://lttng.org/) + Linux only. + + - [Tracing Node on Linux with LTTNG](http://nearform.github.io/tracing-node-lttng-nodejsdublin/) + - [thekemkid/magic-tracing](https://github.com/thekemkid/magic-tracing) is an example of a Node.js application which implements LTTng tracepoints. A demo is available [here](https://asciicinema.org/a/16785). + +* [ETW](https://msdn.microsoft.com/en-us/library/windows/desktop/aa363668.aspx) + Windows only. + + - Some information was provided by Bill Ticehurst (@billti) in [issue #10](https://github.com/nodejs/tracing-wg/issues/10#issuecomment-137145822) about how he integrated ETW into Node.js for Windows + +* [Perf](https://perf.wiki.kernel.org/) + Linux only. + + - Can be enabled with `--perf_basic_prof` v8 run-time flag. + +* [SystemTap](https://sourceware.org/systemtap/) + Linux only. + diff --git a/docs/v8-tracing/README.md b/tracing/vm-tracing/README.md similarity index 82% rename from docs/v8-tracing/README.md rename to tracing/vm-tracing/README.md index f6b229a..451d413 100644 --- a/docs/v8-tracing/README.md +++ b/tracing/vm-tracing/README.md @@ -1,13 +1,24 @@ -Node.js tracing - native v8 tracing -================================================================================ +## VM Native Tracing +- See [v8-tracing.md](./v8-tracing.md) for technical details and crude implementation. -Google is restructuring the tracing in V8 to accommodate restructuring of -tracing in Chrome. +- All trace events are streamed through the same native component, which can in turn publish as appropriate to its subscribers. + - Ideally should be possible to subscribe and receive events in JavaScript too. +- Requires implementation in Node.js, could learn from Chromium's implementation. +- For maximum utility, we should add trace events to Node's operations, especially async ones. +- Possible to handle other trace sources, such as OS (LTTng, ETW) alongside V8 trace source, since they all are event streams. ----- +## Resources +- [google/trace-viewer](https://github.com/google/trace-viewer) - `chrome://tracing` UI. +- [chromium/src/base/trace_event/](https://code.google.com/p/chromium/codesearch#chromium/src/base/trace_event/) +- [Chromium Speed-Infra](https://www.chromium.org/developers/speed-infra) +- [Chromium Telemetry](https://catapult.gsrc.io/telemetry) +- [Trace Platform Explainer](https://docs.google.com/a/chromium.org/document/d/1l0N1B4L4D94andL1BY39Rs_yXU8ktJBrKbt9EvOb-Oc/edit) +- [Implementations for V8, D8, Chromium](https://docs.google.com/a/chromium.org/document/d/1_4LAnInOB8tM_DLjptWiszRwa4qwiSsDzMkO4tU-Qes/edit#) +- [Trace Event Format](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit) + +### Background -Some older-ish comments from @natduca in -[nodejs/node issue #671](https://github.com/nodejs/node/issues/671#issuecomment-73191538): +@natduca in [nodejs/node#671_comment](https://github.com/nodejs/node/issues/671#issuecomment-73191538): I thought I'd provide some notes on the direction we're taking in chrome and indirectly v8 around tracing & stack sampling. Since we both intersect at v8 and do run into some of the same "guhh i need a tracing api but its gotta be low overhead," I hope there's some useful context! @@ -37,8 +48,7 @@ Anyway, thats a lotta words. Hope it makes sense! This was all designed with cli ---- -@ofrobots added some additional detail in -[issue #21](https://github.com/nodejs/tracing-wg/issues/21#issuecomment-143891637): +@ofrobots in [nodejs/tracing-wg#21_comment](https://github.com/nodejs/tracing-wg/issues/21#issuecomment-143891637): Here are the main components, as per my understanding: @@ -54,6 +64,3 @@ Basically this trace-event buffer becomes the 'single pipe' through all trace-ev Once the above linked CL lands, traces being gathered in V8 will show up there. Node.js can start putting its own traces into the same pipe. I would imagine we would also need an API in Node.js to be exposed to JavaScript to allow JS code to start sending the trace events. ----- - -If you have more info to provide, please send us a pull request! diff --git a/tracing/vm-tracing/v8-tracing.md b/tracing/vm-tracing/v8-tracing.md new file mode 100644 index 0000000..9b9455e --- /dev/null +++ b/tracing/vm-tracing/v8-tracing.md @@ -0,0 +1,39 @@ +## V8/Chromium Tracing +Generic macro definitions intended to be shared by many projects are in `trace_event_common.h`. +Source of truth for `trace_event_common.h` is in `chromium:src/base/trace_event/trace_event_common.h`. +This file is in Chromium but not in V8. @ofrobots copied it into `nodejs:deps/v8`. Perhaps it should be moved to `nodejs:deps/v8/include` or `nodejs:deps/v8/src/trace_event`. + +Implementation-specific ("INTERNAL_*") macro definitions are in `v8:src/trace_event.h` and `chromium:src/base/trace_event.h`. +Chromium: `trace_event.h` macros utilize `v8::base::TraceLog::AddTraceEvent` to write traces. +V8: `trace_event.h` macros utilize `v8::Platform::AddTraceEvent` to write traces. +Node uses `DefaultPlatform` from `v8:src/libplatform/default-platform.h` which currently simply returns 0. +A simplistic implementation which logs all trace events to stdout illustrates the basics: + +``` +// from v8:src/libplatform/default-platform.cc +// add `#include ` to includes + +uint64_t DefaultPlatform::AddTraceEvent( + char phase, const uint8_t* category_enabled_flag, const char* name, + /* const char* scope, */ uint64_t id, uint64_t bind_id, int num_args, + const char** arg_names, const uint8_t* arg_types, + const uint64_t* arg_values, unsigned int flags) { + + time_t timestamp = std::time(nullptr); + + std::cout << "phase: " << phase << std::endl; + std::cout << "category_enabled_flag: " << category_enabled_flag << std::endl; + std::cout << "name: " << name << std::endl; + std::cout << "id: " << id << std::endl; + std::cout << "epoch_timestamp: " << timestamp << std::endl; + std::cout << "human_timestamp: " << std::asctime(std::localtime(×tamp)) << std::endl; + + return 0; +} + +const uint8_t* DefaultPlatform::GetCategoryGroupEnabled(const char* name) { + static uint8_t yes = CategoryGroupEnabledFlags::kEnabledForRecording_CategoryGroupEnabledFlags | + CategoryGroupEnabledFlags::kEnabledForEventCallback_CategoryGroupEnabledFlags; + return &yes; +} +```