Skip to content

Commit

Permalink
fuzz testing blog post (kubernetes#1067)
Browse files Browse the repository at this point in the history
add fuzz testing blog post

Signed-off-by: William Morgan <[email protected]>
  • Loading branch information
wmorgan authored May 7, 2021
1 parent 267fbe0 commit 7604018
Show file tree
Hide file tree
Showing 4 changed files with 148 additions and 2 deletions.
2 changes: 1 addition & 1 deletion linkerd.io/content/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ benefits:
- title: "Best-in-breed security"
image: "/uploads/secure.svg"
announcement_banner:
description: "**2021-03-11** Announcing Linkerd 2.10! [Read more »](/2021/03/11/announcing-linkerd-2.10/)"
description: "**2021-05-07** Introducing fuzz testing for Linkerd! [Read more »](/2021/05/07/fuzz-testing-for-linkerd/)"
description: Linkerd adds critical security, observability, and reliability to your
Kubernetes stack, without any code changes.
keywords:
Expand Down
2 changes: 1 addition & 1 deletion linkerd.io/content/blog/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ tags:
- Tutorials &amp; How-To's
- Video
items:
- blog/introducing-fuzz-testing-to-linkerd.md
- blog/announcing-linkerd-2-10.md
- blog/linkerd-steering-committee.md
description: ''
keywords: []
---
146 changes: 146 additions & 0 deletions linkerd.io/content/blog/introducing-fuzz-testing-to-linkerd.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
title: 'Introducing fuzz testing for Linkerd'
author: 'william'
date: 2021-05-07T00:00:00+00:00
thumbnail: /uploads/prince-abid-TWbkojYMhAg-unsplash.jpg
featured: false
slug: fuzz-testing-for-linkerd
tags: [Linkerd]
---

![a fuzzy baby chick](/uploads/prince-abid-TWbkojYMhAg-unsplash.jpg)

Over the past few months, the team at [Ada Logics](https://adalogics.com/) has
been hard at work introducing fuzz testing to [Linkerd's Rust
proxy](https://linkerd.io/2020/07/23/under-the-hood-of-linkerds-state-of-the-art-rust-proxy-linkerd2-proxy/).
These fuzz tests now run continuously on Linkerd via Google's
[OSS-Fuzz](https://github.com/google/oss-fuzz) service, providing another layer
of safety for Linkerd users around the globe. In total, fuzzing was integrated
into seven dependencies of the proxy as well as the proxy itself, and five of
these projects are now running continuously on OSS-Fuzz.

These fuzz tests uncovered two minor bugs in the proxy, which have since been
fixed. (By comparison, a similar recent effort on a C-based CNCF project found
(and fixed) over 30 bugs, primarily related to memory safety). In addition, a
few minor issues were found in dependencies, and these have all been
communicated to the maintainers.

You can read the [full report
here](https://github.com/linkerd/linkerd2-proxy/blob/main/docs/linkerd2-proxy-fuzzing-report.pdf).
For details, and how you can get involved in this important safety measure for
Linkerd, read on!

(Background reading: [Under the hood of Linkerd's state-of-the-art Rust proxy,
Linkerd2-proxy](https://linkerd.io/2020/07/23/under-the-hood-of-linkerds-state-of-the-art-rust-proxy-linkerd2-proxy/);
and [Why Linkerd doesn't use
Envoy](https://linkerd.io/2020/12/03/why-linkerd-doesnt-use-envoy/).)

## Why fuzz test the proxy?

Fuzz testing is a type of automated software testing that uses randomized
inputs and coverage-based genetic algorithms to evaluate the behavior of code.
Because the input is randomly generated, fuzz testing is useful in capturing
test cases that humans might miss, especially when it comes to corner cases
around invalid or unexpected inputs.

Linkerd uses [an arsenal of automated
tests](https://github.com/linkerd/linkerd2/actions) to ensure safety and
reliability on every commit. These tests range from code linting and static
analysis, to unit tests, to a comprehensive suite of integration tests. While
these tests play a major part of Linkerd's development process, there are
three reasons why we wanted to augment them with fuzz testing of the proxy in
particular:

1. **Linkerd's proxy handles untrusted input from the network.** The proxy must
take a request from an application or possibly from the open Internet; parse
the data; and "do something" with it—send it to its destination, reject it,
etc. While Rust's memory safety guarantees help us avoid an entire class of
security vulnerabilities that are endemic to C and C++ code, that, of course,
doesn't prevent other types of bugs. And the threshold for resilience in the
proxy is _extremely_ high—it must be able to handle even the worst-case
scenario of malicious input crafted by someone with full source code access.

2. **The data plane proxy is the most critical runtime component of any service
mesh.** While Linkerd is designed so that a temporary failure of control plane
components does not affect running applications, temporary failures at the data
plane level means that pods are unable to process requests. And beyond simple
failures, the worst-case scenario for bugs in the proxy is horrifying: changing
or corrupting a request could be disastrous; not handling mTLS correctly could
expose traffic to attackers; and so on. Thus, it behoves us to have the most
robust set of tests possible, in addition to our existing reliability practices
around code review, etc.

3. **The fuzz testing ecosystem for Rust is simply underdeveloped compared to
that of other languages.** Investing in this critical area meant that not just
Linkerd, but other projects that are built on the state-of-the-art Rust
networking ecosystem (projects like [Tokio](https://tokio.rs/),
[Hyper](https://hyper.rs/), [Rustls](https://github.com/ctz/rustls), and
[Tower](https://github.com/tower-rs/tower)) could all benefit from this work—a
win for everyone.

## What did the fuzz testing find?

The initial fuzz tests found two minor bugs in Linkerd's proxy.[^1] First, the
proxy would panic when a DNS lookup resulted in names exceeding 255 bytes, due
to [a bug in the trust-dns
library](https://github.com/bluejekyll/trust-dns/issues/1447) the proxy uses
for DNS queries. This issue was fixed immediately by the maintainers of that
library and the fix incorporated in recent Linkerd releases.

Second, the proxy would panic when resolving IPv6 addresses with port 80. (Most
Kubernetes providers do not yet support IPv6, but the proxy is future-proofed
for when they do.) This was [a bug in the proxy
itself](https://github.com/linkerd/linkerd2-proxy/pull/976), and was also fixed
immediately with the fix incorporated into recent Linkerd releases.

These bugs demonstrate the value of fuzz testing: they represent logical flaws
in the proxy (and dependent libraries) that involved unexpected edge cases, in
code which had passed several rounds of human review.

While no two projects are alike, as a point of comparison, a recent similar
exercise by Ada Logics on a graduated CNCF project written in C yielded over 30
bugs, including 4 "high severity" and 8 "medium severity" bugs. The majority of
these bugs were buffer overflows, null dereferences, memory leaks, and other
types of memory errors that Linkerd's use of Rust allows us to avoid in the
first place.
([link](https://www.cncf.io/blog/2020/12/15/securing-open-source-fuzzing-integration-vulnerability-analysis-and-bug-fixing-of-fluent-bit/))

## So what's next?

The fact that the fuzz tests found bugs is great! But this is just the
beginning, not the end, of the fuzz testing story. This initial work only
captured a portion of the proxy's surface area, and there's a lot more that
could benefit from fuzz testing.

If you're looking for ways to get involved with Linkerd—and we'd love to have
you—the good news is that this is yet another avenue by which you can help the
project. If you're interested in further building out Linkerd's set of fuzz
tests, please see [the developer docs for proxy fuzzing in
Linkerd](https://github.com/linkerd/linkerd2-proxy/blob/main/docs/FUZZING.md),
hop into `#contributors` on [the Linkerd Slack](https://slack.linkerd.io), and
let's make it happen!

## Thank you

The Linkerd maintainers would like to issue a special thank-you to the Ada
Logics team and David Korczynski in particular for their hard work in
implementing these fuzzers, for wading into the uncharted waters of fuzz
testing for Rust, and for helping us navigate the OSS-Fuzz project. We'd also
like to thank the CNCF, especially Chris Aniszczyk, for sponsoring this work.

## Linkerd is for everyone

Linkerd is a community project and is hosted by the
[Cloud Native Computing Foundation](https://cncf.io/). Linkerd is
[committed to open governance.](https://linkerd.io/2019/10/03/linkerds-commitment-to-open-governance/)
If you have feature requests, questions, or comments, we'd love to have you join
our rapidly-growing community! Linkerd is hosted on
[GitHub](https://github.com/linkerd/), and we have a thriving community on
[Slack](https://slack.linkerd.io/), [Twitter](https://twitter.com/linkerd), and
the [mailing lists](https://linkerd.io/2/get-involved/). Come and join the fun!

(*Photo by [Prince Abid](https://unsplash.com/@princeabid708?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText") on [Unsplash](https://unsplash.com/s/photos/fuzz?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText).*)

## Footnotes

[^1]: The tests are focused on finding situations that cause the proxy to "panic", or intentionally crash—an easily-detectable situation that is the consequence of the proxy finding itself in a situation it can't handle. A fuzz test that causes the proxy to panic in response to strange input is a sure sign of a bug.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 7604018

Please sign in to comment.