How did Testmetrics start?
It was March of 2018, and I was listening to a podcast episode about testing. The guests made that point that we really don't have good data about our test suites. It wasn't something that had ever occured to me before, but once they brought it up, it instantly made sense.
I did some pretty extensive research and realized there was nothing out there that would let me collect the kind of data I wanted about my tests. Testing is a fundamental part of many software development workflows today, and to have absolutely no data about our tests seemed like a huge problem in need of a solution.
So, since I'm a lazy engineer, naturally I waited a few months to see if someone would build something for me. When no one did, I decided that it was time to build something for myself. I loved the data I was getting and I thought others might get value out of it as well, so in November of 2018 I decided to make Testmetrics widely available to the public.
What problems does Testmetrics solve?
Production test suites frequently suffer from a series of issues, including:
- Very long run times
- Poor test design
- Intermittent test failures (aka flaky tests)
- Undertesting (not testing some parts of an application)
- Overtesting (testing a given part of an application too heavily)
- Imbalance between integration and unit tests
If you have any of those problems, Testmetrics can help! Testmetrics provides the data and quick feedback that's been missing to help development teams find and fix almost all of these issues. The only problem I'm not planning to tackle is under-testing, since that's a fairly easy problem to diagnose and there are plenty of good existing tools on the market.
What does Testmetrics actualy do?
When you run your tests in CI using one of the Testmetrics clients, data about that run is collected and sent to our servers be saved and analyzed. None of this data is at all sensitive. Our clients currently only collect information about test run times and the pass/fail status for each test.
If you want to see for yourself what exactly is being collected, all clients are open source, so you can make sure with your own eyes that we're not doing anything inappropriate with your data.
What languages and testing frameworks does Testmetrics support?