A library for carefully refactoring critical paths in your elixir appplication.
This is an elixir clone of the ruby gem scientist.
Suppose you decide to add a new caching layer to your production application, while still being able to make the same guarantees about your data. By processing your new caching strategy through Scientist you'll be able to
- Run both the old and the new code in random order
- Monitor timing for each strategy
- Compare their results and find any mismatches
- Rescue and report any exceptions thrown in your new code
- Publish all of this information in a manner of your choosing.
Externally, a Scientist experiment behaves exactly the same as its control block, returning the value of the control as well as re-raising any of its exceptions.
defmodule MyPhoenixApp.UserController do
use Scientist
use MyPhoenixApp.Web, :controller
alias MyPhoenixApp.User
alias MyPhoenixApp.Repo
plug :action
def get_user(id) do
# Let's get down to business, with science!
science "New ETS cache for users" do
control do: Repo.get(User, id)
candidate do: MyETSCache.get(User, id)
end
end
# ... other controller logic
end
Experiments aren't useful on their own. You need to be able to report their results and control
their execution. To define your own custom experiment, you need to use Scientist.Experiment
and
implement a few callbacks.
defmodule MyCustomExperiment do
use Scientist.Experiment
# Required callbacks: publish/1, enabled?/0
# See "Enabling Experiments" and "Publishing" below
defdelegate [enabled?(), publish(result)], to: Scientist.Default
# Optional callbacks
# Default name
def default_name, do: "My custom experiment"
# Default context, see "Need some context?" below
def default_context, do: %{}
end
Then when using Scientist
you can specify your custom experiment to be used instead of
Scientist.Default
:
defmodule UserController do
use Scientist, experiment: MyCustomExperiment
# Now let's get some science done!
end
Now all calls to science
will use MyCustomExperiment.new/2
for setup.
Out of the box, Scientist will compare observed values with Kernel.==/2
to see if they match. You can override this with a comparison block.
def get_user(id) do
science "New ETS cache for users" do
control do: Repo.get(User, id)
candidate do: MyETSCache.get(User, id)
# We only care if the user's status is updated.
compare(%{status: sa}, %{status: sb}) do
sa == sb
end
end
end
Sometimes, you need more information about the environment when checking the results of your experiment. In these cases, you can pass a map of values to your experiment before it's run:
def get_user(id) do
# Perhaps the cache is filling too quickly
c = %{cache_size: MyETSCache.size(User)}
science "New ETS cache for users", context: c, do
control do: Repo.get(User, id)
candidate do: MyETSCache.get(User, id)
end
end
Then the context will be available as result.experiment.context
in your publisher.
Should you find yourself with too much information during an experiment, you can pass
an optional clean
block to extract the relevant data. Then only the cleaned
values will be compared and you won't erroneously report mismatches about other data.
def get_user(id) do
science "New ETS cache for users" do
control do: Repo.get(User, id)
candidate do: MyETSCache.get(User, id)
clean(_user = %{status: status}) do
status
end
end
end
Both value
and cleaned_value
will be later available in your observations.
In some cases you know ahead of time that your experiment will mismatch. You could be replicating your data to a new store in pieces, or moving to a cache with less recency.
Fortunately, Scientist
allows you to specify these situations and ignore mismatches outright when they occur.
def get_user(id) do
science "New ETS cache for users" do
control do: Repo.get(User, id)
candidate do: MyETSCache.get(User, id)
ignore(control, _candidate) do
# Cached entries have a 1 min TTL
# We expect a mismatch when the DB was updated sooner.
within_last_minute?(control.updated_at)
end
end
end
You can even choose to avoid running the experiment entirely with a run_if
block:
def get_posts_for_user(user) do
science "Data should be consistent during migration" do
run_if do
# It clearly won't match if we haven't moved their data.
User.is_migrated?(user)
end
control do: Repo.get_by(Post, user_id: user.id)
candidate do: NewRepo.get_by(Post, user_id: user.id)
end
end
In addition to run_if
, custom experiments use the enabled?/0
callback to determine whether or not they should run. You must implement this function in your experiment module:
defmodule MyCustomExperiment do
use Scientist.Experiment
@percent_enabled 0.5
# Let's not go too crazy, let's say this should run half the time.
def enabled?, do: :random.uniform < @percent_enabled
end
Scientist doesn't care how you choose to publish your results - you can send results to a batching GenServer process or simply use Logger. However you do it is up to you. You are however, required to implement publishing in some form.
The publish/1
callback is given a Scientist.Result
struct containing all observations made, including
their durations, values, and whether or not there was a mismatch.
defmodule MyCustomExperiment do
use Scientist.Experiment
alias Scientist.Result
def enabled?, do: true
def publish(result) do
MyPublisher.publish("control", result.experiment.name, result.control.duration)
Enum.each(result.candidates, fn can ->
MyPublisher.publish(candidate.name, result.experiment.name, candidate.duration)
end)
if Result.mismatched?(result) do
MyPublish.report_mismatch(result.experiment.name, result.mismatched)
end
end
end
We've all done it before. Sooner or later you'll configure your experiment with blocks that may raise an
exception. In these situations you can use the optional raised/3
and thrown/3
callbacks so that your
experiment will continue in some fashion without complete failure.
defmodule MyCustomExperiment do
use Scientist.Experiment
# ... implementing required callbacks ...
def raised(ex, operation, except) do
IO.puts "Experiment failure in \"#{ex.name}\": #{operation} raised #{except.message}"
end
def thrown(ex, operation, except) do
IO.puts "Experiment failure in \"#{ex.name}\": #{operation} threw #{except}"
end
end
Each function is called with the name of the internal operation that failed:
:publish
- Exception raised withinpublish/1
:enabled
- Exception raised withinenabled?/0
:compare
- Exception raised during comparison:clean
- Exception raised during cleaning:ignore
- Exception raised within an ignore block:run_if
- Exception raised within an run_if block
If these functions are not defined, Scientist
will not handle the exception raised.
It can be useful to force Scientist
to notify you of any mismatches that occur during testing. Within a custom experiment or a single experiment, you can set raise_on_mismatches: true
to raise a Scientist.MismatchError
when observations don't match.
# Raise on any individual experiment using this module
defmodule MyCustomExperiment do
use Scientist.Experiment, raise_on_mismatches: true
# ... implementing required callbacks ...
end
# Within a single experiment
science "this should never mismatch", raise_on_mismatches: true do
# Same ol' experiment configuration
end
This setting is purposefully verbose, as you shouldn't be affecting the behavior of your application like this in production.
Some people prefer to avoid a DSL, as it can obfuscate your code and possibly raise strange, untraceable
errors. science
and its friends are macros that simply create an experiment using the module you
specify and then call Scientist.Experiment
to configure and run it.
You can do this for yourself, although it can seem a bit verbose. Fortunately, |>
removes a bit of boilerplate for us.
def get_user(id) do
import Scientist.Experiment
context = %{cache_size: MyETSCache.size(User)}
MyCustomExperiment.new("New ETS cache for users", context: context)
|> add_control(fn -> Repo.get(User, id) end)
|> add_candidate(fn -> MyETSCache.get(User, id) end)
|> clean_with(fn %{status: status} -> status end)
|> run
end
You should keep a few things in mind before you jump into an experiment.
You should only use Scientist when touching code that does operations on read-only data. You don't want to alter any code that does required mutation or persistence, as you wouldn't be able to guarantee its execution. In these cases, such as data migration, it would be best to write to both stores and check for any inconsistencies with a single experiment during reads.
You can also always have more than one candidate block in a single experiment, but it can make your results harder to interpret while also adding additional execution time. You can distinguish between candidate blocks by giving them unique names:
def get_user(id) do
science "Trying ALL the caching strategies" do
control do: Repo.get(User, id)
candidate "MyETSCache", do: MyETSCache.get(User, id)
candidate "ConCache", do: ConCache.get(User, id)
candidate "RedisCache", do: RedisCache.get(User, id)
end
end
If you only care about timing data or new code stability, you can ignore results entirely by passing
compare blocks that are always true: compare(_, _) do: true
Scientist is available on Hex. It can be installed by adding it to your
list of dependencies in mix.exs
:
def deps do
[{:scientist, "~> 0.2.1"}]
end
Scientist is licensed under the MIT License. See LICENSE for the full text.