Enhancement Proposal: Add Runtime Metrics to Flyte Plugins

[jonathanburns]

Add Runtime Metrics to Flyte Plugins

---

Table of Contents

• Summary
• Proposal
• Example
• Alternatives

Summary

Flyte’s task plugins continually monitor the resources they create in order to fulfill a task. This makes them well suited to report runtime information for tasks they create (for example, they could report the running time (in seconds) of a task). This data should get sent back to FlyteAdmin in the "event" call when a task reaches a terminal state ("failed", "complete"). In order for plugins to report runtime data and send the data to FlyteAdmin for storage, we need to decide on how to structure this data.

While runtime data can be translated into a dollar figures to calculate cost, that is the responsibility of a separate system, and will not be discussed in this document.

Proposal

At a high level, runtime data requires keys/values in the form “metric” => value.

Metric Types

Metric Types are things like “cpuSeconds” (runtime x cpu) & “memorySeconds” (memory x cpu). However, many other metric Types could be surfaced including “gpuSeconds”, “tokensUsed”, "nodeType"., etc. It’s difficult to predict up-front all the possible types of metrics that could be useful to have.

Approach 1: Hard-Coded List of Metric Types in FlyteIDL

We could create a curated list of supported “Metric Types” and hard code them in FlyteIDL. This prevents spelling differences like “cpuSeconds” vs “cpu_seconds” in different plugins. Having a standard metric name is useful, because an external “cost calculator” might be crawling over these metrics to calculate costs based on resource usage.

However, plugin developers will discover new metric types that we do not foresee today. Plugin developers would not be able to add new metric Types without changes to FlyteIDL. That probably makes this method a non-starter.

Approach 2: Configurable List of Metric Types

We could have a config in flytepropeller which specifies the allowed “Metric Types”. New metric types would require explicitly adding a line to flytepropeller’s “allowedMetricTypes” list.

The disadvantage of this approach is that it becomes painful for users to add new plugins. You’d need to re-configure flytepropeller to accept the new metric-type. These confusing steps also makes this method more trouble than helpful.

Approach 3: Arbitrary Key/Value Pairs

Allow plugins to specify arbitrary key/value pairs for MetricType => Usage. While this leaves an opportunity for resource names to diverge in plugins, we can document the conventions to reduce the chances of that happening.

This seems like the best option.

Tasks with Sub-Workers

Plugins like “Spark” and “AWSArray” launch multiple workers per task. These plugins could sum the worker metrics to produce a single metric per resource. “cpuSeconds: SUM(cpuSeconds), memorySeconds: SUM(memorySeconds)”

Summing sub-workers non-ideal. In an Array task, users want to know if a single worker ran longer than others. For this reason, the task should be able to report back multiple “subWorkers”, each of which has its own resources.

A plugin developer may still choose to SUM resources, but the system should support more robust metrics.

Example

Metrics for an Array Task might look something like:

	“batch-task-0”: {
		“cpuSeconds”: 100
		“memorySeconds”: 5000
	},
	“batch-task-1”:  {
		“cpuSeconds”: 100
		“memorySeconds”: 5000
	},
	etc.

Implementation

This feature could be implemented with no changes to the task event API using the available “custom_info” field:

https://github.com/lyft/flyteidl/blob/master/protos/flyteidl/event/event.proto#L118

FlyteAdmin does not currently do anything with this field, so this requires a small change in FlyteAdmin to store this data.

[wild-endeavor]

Naming

Are we set on the name "ResourceType"? Let's ask the team? I feel like some of the stuff in our IDL like MatchableResource and TaskResourceSpec and ResourceName to already be confusing and adding a new concept will make it worse. There's already plenty of confusing things in the IDL and IDL changes last forever, since we've decided that our days of making breaking changes is over.

Approaches

I like the idea of plugins being the one to publish task resource usage metrics but I think the first approach is the best actually. One of the core tenets of Flyte is that it is strongly typed. We've relaxed this constraint in a couple places (I think those are related but that's besides the point) but they've always felt like a hack, at least to me. The fact that plugin writers can't create new resources without updating the IDL (or the IDL component of their plugin) to me is a good thing. That is behavior we should endeavor to prevent.

Another tenet of Flyte is that it is extensible through plugins. The IDL, flytekit, propeller through flyteplugins, and in the future even the UI. I think it makes way more sense to strongly define a set of compute resources and allow plugins to use them, or to define their own. Particularly because K8s already defines resources, as we've ported over in the ResourceName link above.

One question I have is the equality of CPU or CPU seconds. I don't know the answer to this but are all CPUs equal? Like, if we run risc-jobs one day that use sparc processors, should the system know that it's a different CPU type? If so, how will that work? This is probably a bit contrived, but certainly I can see different GPUs costing different amounts. Looking at p2s and p3s (https://www.ec2instances.info/?compare_on=true&selected=p2.16xlarge,p3.8xlarge) it appears that one CPU core on a p3 is almost twice as expensive as a CPU core on a p2 (almost the same price, half the number of cores). p2.16s also have 16 GPUs whereas a p3.8x only has 4 GPUs. Somewhere, we need to take into account the fact that a plugin launched one vs the other. However I definitely don't want gpu_p3.8_seconds and gpu_p2.16_seconds to be two different resource types. I agree that plugins should definitely not worry about the cost of things but something does have to worry about the types of the underlying resources.

Sub-resources

Agree that the plugin should provide both sum, and granular data. This is in keeping with our pattern - for example, currently the aws batch plugin provides log links to individual subtasks as well as the overall batch array job. However, it is able to do this because the log links that Admin receives is a list. There's another custom field that's not really used right now. The idea was that front-end (flyte console) plugins and admin plugins would be able to interpret that information (right @EngHabu ?). I feel like a list is a pretty basic construct though - should the internal system that does the actual cost computing just be able to handle lists?

Data processing

You wrote a google doc internally talking about this github issue stating that this ResourceMetric information is sufficient for understanding compute usage over time by tasks. I think this is true, but it sounds like it would require an etl pipeline to compute costs. A post-processing pipeline will always be required for business metric use-cases to get reports to look exactly like what we want. But I wonder if there's a better way to provide real-time data. As a Flyte platform administrator, I want to know exactly what is going on right now. I'd like to see a gauge go up and down as tasks start and finish that's updated within a second or two. How would the system described here intertwine with something like that? Can we use one for the other?