Use these metrics to get the most out of your engineering team

This post was written by Rob Zuber, CTO of CircleCI

I’ve been leading software teams for more than 20 years and one thing I’ve learned about metrics is that leaders tend to put too much emphasis on engineering metrics alone, without considering the bigger picture.

After speaking to a range of engineering industry leaders, and poring over millions of jobs processed from software teams worldwide, we found that the most insightful and relevant metrics fall into three categories:

• Velocity metrics
• Morale metrics
• Business metrics

Velocity metrics

Engineering velocity metrics measure the speed and efficiency of software delivery pipelines — it’s the metric category that managers typically pay the most attention to. While I’ll explain why it’s not the only important category to track, velocity metrics are critical in helping teams identify slowdowns and find ways to optimize their overall performance.

Some of the most common velocity metrics include:

• Throughput
• Change lead time
• Sprint velocity
• Duration
• Mean time to recovery
• Success rate

Customize Throughput to Your Work Structure

Moses Mendoza, former Head of Engineering at data processing and review platform Zapproved, uses throughput to understand the pace of his team’s work.

“Throughput helps us identify and understand speed but the throughput of a system is also bound by its primary constraint,” Mendoza said. “Throughput will show you what the slowest issue is in a chain of events, but it won’t show you how to fix it to speed up your work.”

Graeme Harvey, an engineering manager on my team, emphasizes that it’s important for all engineers to customize throughput measurement to their individual team.

“Because our team practices pair-programming, measuring throughput isn’t something that can be tied to an individual’s productivity,” Harvey said.

When it comes to throughput, his engineers optimize for the team rather than the individual. Pair-programming and helping each other might feel like it’s impeding the progress of an individual but in actuality, it refocuses energy on what’s most important for the team and ultimately the business.

While throughput is a valuable metric that helps you track output, there is no one-size-fits-all way to measure it. Measuring throughput accurately requires you to evaluate the structure of your team and how you work.

Measure Lead Time to Identify Communication Issues and the Pipeline Quality

According to Alex Bilmes, former VP of Growth at software configuration tool Puppet, there are two ways to measure change lead time. “One way to measure change lead time is to look at how long it takes to get an idea out and for the idea to go full cycle. The other way is to look at deployment lead time, which measures how long it takes to get to production after a developer has pushed the change to production.”

Full change lead time will point out issues in communication and understanding, as well as the depth of your backlog. Deployment lead time is more likely to show the quality of your pipelines and tooling.

Use Sprint Velocity to Plan Workload

Sprint velocity measures the amount of work a team can tackle during a single sprint and can be used for planning and measuring team performance.

Tom Forlini, CTO at video conferencing platform Livestorm, dives even deeper when measuring velocity, focusing on three smaller metrics:

• Number of issues and story points
• Number of issues done vs. planned
• The percentage of issues distributed depending on type

“Livestorm engineers work on two-week sprints and have 50 story points per sprint,” Forlini said. “We track the number of issues done vs. planned because it gives us a good indication of the sprint planning quality between Product and Tech.”

Then, his team looks at the percentage of issues by type. “When a sprint contains only new feature issues, we know from the start that it might be quite a challenging sprint to tackle,” Forlini added. “Ideally, you should balance the type of issues by sprint as much as possible.”

Morale metrics

Morale metrics are probably the most overlooked metric category in engineering. They tell you how engineers feel about the quality of their work and their job happiness, which is a major retention factor. Keeping retention high means keeping morale high.

Some common morale metrics include:

• Individual and team morale
• Code quality confidence

Measure Morale by Asking Engineers Directly

At Zapproved, Mendoza tracked morale in order to monitor employee retention. “We measured morale at work using surveys, having conversations, and asking managers to dive deeper in one-on-one meetings to find out how employees felt.”

If responses to a survey are overwhelmingly positive, you’ll want to know what is working and how to replicate that positive work environment. Similarly, if responses are negative, it’s helpful to find out directly from your team why they feel that way and what you can do to fix the problem.

Focus on Confidence Over Coverage

Mendoza at Zapproved measured confidence by reviewing every sprint in conjunction with that team’s manager and their scrum master. “As we measured code quality confidence over two or three sprints, if we saw code quality tanking, it meant something was wrong with how the teams planned their individual investment with the work,” says Mendoza.

The engineering managers that I lead also measure work by confidence.

“Focusing on confidence over coverage as a metric requires that the emphasis isn’t on code coverage,” Harvey added. “It’s critical to break the reliance on having 80% or 90% code coverage and then shipping it only to find out the code is broken. Test coverage is a partial proxy for code confidence. If you know 95% of your code is fully tested, versus 20%, then you’re going to feel pretty confident that if your tests pass, your code is legitimate.”

Harvey’s team focuses on delivering small iterations quickly. This provides the confidence that the team is building something of quality, nothing is broken, and they’ve made the right choices in building tools for the dashboard.

Business metrics

Everything an engineer does should propel the company forward. That’s why it’s also essential to track business metrics.

Some common business metrics include:

• Company growth Funnel metrics
• End-user value

Track How Quickly Your Business is Scaling

Tracking business metrics is how your team accommodates for user growth effectively. According to Yixin Zhu, formerly of Uber, while it’s essential to look at engineering execution metrics, it’s also important to be dialed into the business’s goals and to measure the company growth.

As Uber grew exponentially, tracking business metrics was incredibly important in order for Zhu’s engineering team to succeed. “When you’re talking about doubling every six months, you have to be tracking that to know what you need to build, what degradations to expect, how many data centers you need, how many boxes, etc.,” Zhu said.

In short, engineers have to keep an eye on real-time business metrics to project and plan accurately. “You have to be proactive,” Zhu added.

Getting the Most Out of the Metrics That Matter

Here are some tips to help you get the most value out of your engineering efforts:

• Define business goals. Your engineers should know what your business is trying to accomplish. When engineers understand the direction of the company, they understand what they’re working towards.
• Review metrics often. Start every meeting by reviewing metrics so the team is aligned on what you’re doing right and what needs improvement.
• Start small. You only need one to three metrics to get started. If you’re intentional about tracking simple metrics, experimenting with different approaches, and seeing those metrics evolve, you can add metrics over time.
• Look beyond execution metrics. The success of an engineering team hinges on more than engineering metrics. Pairing engineering metrics with business metrics will help you understand why you’re building what you’re building every day.

Rob Zuber (@z00b) is the CTO of CircleCI. He leads a team of 250+ engineers who are distributed around the globe. He is a three-time founder and five-time CTO, currently living in Oakland, CA with his family.