About Testmetrics

How did Testmetrics start?

It was March of 2018, and I was listening to a podcast episode about testing. The guests made that point that we really don't have good data about our test suites. It wasn't something that had ever occured to me before, but once they brought it up, it instantly made sense.

I did some pretty extensive research and realized there was nothing out there that would let me collect the kind of data I wanted about my tests. Testing is a fundamental part of many software development workflows today, and to have absolutely no data about our tests seemed like a huge problem in need of a solution.

So, since I'm a lazy engineer, naturally I waited a few months to see if someone would build something for me. When no one did, I decided that it was time to build something for myself. I loved the data I was getting and I thought others might get value out of it as well, so in November of 2018 I decided to make Testmetrics widely available to the public.

What problems does Testmetrics solve?

Production test suites frequently suffer from a series of issues, including:

  • Very long run times
  • Poor test design
  • Intermittent test failures (aka flaky tests)
  • Undertesting (not testing some parts of an application)
  • Overtesting (testing a given part of an application too heavily)
  • Imbalance between integration and unit tests

If you have any of those problems, Testmetrics can help! Testmetrics provides the data and quick feedback that's been missing to help development teams find and fix almost all of these issues. The only problem I'm not planning to tackle is under-testing, since that's a fairly easy problem to diagnose and there are plenty of good existing tools on the market.

What does Testmetrics actualy do?

When you run your tests in CI using one of the Testmetrics clients, data about that run is collected and sent to our servers be saved and analyzed. None of this data is at all sensitive. Our clients currently only collect information about test run times and the pass/fail status for each test.

If you want to see for yourself what exactly is being collected, all clients are open source, so you can make sure with your own eyes that we're not doing anything inappropriate with your data.

What languages and testing frameworks does Testmetrics support?

Right now there are clients for Elixir (ExUnit) and Ruby (RSpec and MiniTest). In the very near future there will also be clients for JavaScript, Python and Go, and then later on clients for Java, C#, Rust and as many other languages as I can cover. The addition of new clients will be based primarily on user requests, so if there's a language or test runner you want support for, please do send me a note!

Who's behind this thing?

My name is Devon Estes, and I'm a software developer and long-time lover of writing great tests. If you want to get in touch with me about anything, you can email me at devon@testmetrics.app.