Core Metrics

[chadyj]

Preamble

Idea: #68-core-metrics
Title: Core Metrics
Status: In progress
Created: 2018-01-10

Summary

Determine and implement a set of user-related core performance metrics that measure the goal of driving mass-adoption of Ethereum through Status use, and do so in a way that respects users privacy.

Project board: status-im/projects/14

Swarm Participants

• Lead & PM: @chadyj
• Testing & Evaluation: @lukaszfryc
• Contributor: @jeluard
• Contributor: @adambabik

Product Overview

User-related performance metrics will help orient the team towards driving Ethereum adoption by being data-informed, and provide a way to assess if the team is making progress towards this goal. The most important metric driving this goal is daily transacting users (DTU). In support of DTU's there are sub-goals related to engagement, adoption, retention, and task success for the app as a whole and individual features such as wallet, chat, and Dapp's.

This swarm should determine the best core metrics to measure our goals and progress, then create methods to track, store and view them.

Metric collection will be opt-in only to ensure that users privacy is respected. Any metrics gathered will be strictly in the service of making Status better for users and never for any advertising or monetisation reasons, and be completely anonymous.

Product Description

This is not a user-facing Swarm. The product work is for Status org consumption.

Core goals are not intended to measure for measurements sake, but to provide product insight, create feedback loops that can accelerate decision making and product development, and let us know how users are using Status at scale. After determining the metrics, the next step is collecting the metrics and establishing a baseline that future quantitative goals can be measured against.

This draft has the start of proposed metrics that would give us deep insight into Status and Ethereum usage, and is to be revised with team feedback. Each future swarm can also pick metric(s) from this list and track them as part of their success metrics.

Goals

• Daily transacting users (DTU)
• Monthly transacting users (MTU)
• Daily SNT transaction volume
• Monthly SNT transaction volume
• Daily ETH and ERC20 token transaction volume
• Monthly ETH and ERC20 token transaction volume

Engagement Metrics
Level of user involvement

• Daily active users (DAU)
• Monthly active users (MAU)
• Daily messages sent
• Monthly messages sent
• Daily messaging users
• Monthly messaging users
• Daily DApp Users
• Monthly DApp Users

Adoption Metrics
New users of a product or feature

• Daily downloads
• Monthly downloads
• Daily signups
• Monthly signups
• Percentage of 1d/7d/30d users that do a transaction
• Percentage of 1d/7d/30d users that send a message
• Percentage of 1d/7d/30d users that open a DApp

Retention
the rate at which existing users are returning

• Percentage of new users using Status after 1d/7d/30d/3m/6m/12m

Healthy stats would show increasing DTU, engaged users across the core of the app, the widespread adoption of core features, and returning users.

There are two main types of metric tracking available: in-app tracking and external tracking. In-app tracking is instrumented from within the app, and will need to adhere to strict guidelines including being opt-in, honest and clear, and anonymous. External tracking can use public or available sources, such as Play/App store accounts, or the blockchain. A complete picture will involve both in-app and external tracking.

Iterations

Iteration 1

Goal Date: 2018-02-28
Status: Complete.

Description: Collect chat message volume as a fast an non-intrusive way to measure app engagement. This will tell us daily/monthly sent messages and we can get a sense for how app use is growing.

• Collect Whisper metrics in MailServer
• Setup a dedicated server for Prometheus and Grafana that can handle data retention time of 30 days.
• Weekly chart of sent messages

Iteration 2
Goal Date: 2018-03-30
Status: In Progress.

Description: Track basic product events to start measuring engagement

• Add Mixpanel client library
• Track engagement events
• Opt-in screen during signup
• Opt-in toggle in settings

Success metrics

This swarm is considered successful if it empowers 4 other swarms to use core metrics to measure their OKR's.

Exit Criteria

When iteration 2 is complete this swarm will end. Any new metric instrumentation is to be completed by user-facing swarms that implement new screens or functionality by updating the mixpanel_events.edn file.

[oskarth]

Seems like work has already started on this, great! It looks like it is still in draft mode. It would be good if we could keep this issue up to date in order to achieve: https://wiki.status.im/Status_Organisational_Design

Some questions:

1. Who is the lead contributor to this?
2. Who is the tester and evaluator?
3. Who is the other contributor?
4. Any other roles needed for the swarm?
5. What does the timeline look like for MVP (and iteration 1 if it exists)?
6. Are there any requirements necessary before this idea can be implemented?

@adambabik @chadyj

[chadyj]

It is still in draft mode for feedback with the intention of kicking off toward the end of the beta.

The one exception is status-im/status-go#536 which was already underway and will give us some insight into core chat metrics, but that is quite isolated.

[chadyj]

This swarm is progressing on the first iteration for server side messaging stats. Currently the needs are very small with @adambabik and @v2nek working on status-im/status-go#536 and https://github.com/status-im/status-cluster/issues/36.

The next iteration will have @jeluard adding the product metrics and will kick off after #80 implements the opt-in feature during onboarding.

[chadyj]

Updated success metrics and exit criteria in the idea:

Success metrics

This swarm is considered successful if it empowers 4 other swarms to use core metrics to measure their OKR's.

Exit Criteria

When iteration 2 is complete this swarm will end. Any new metric instrumentation is to be completed by user-facing swarms that implement new screens or functionality by updating the mixpanel_events.edn file.

[oskarth]

Closing this issue as part of spring cleaning. If this idea is still relevant, please submit a PR per https://github.com/status-im/ideas/#contributing. If you feel closing this issue is a mistake, feel free to re-open.