As the saying goes, “You can’t improve what you don’t measure.” Metrics (operational), and more specifically Key Performance Indicators (KPIs) or Critical Success Factors (strategic), are about measuring progress toward your goal. Without them, you cannot know if your AS IS (current state) is making progress and whether you are on track to reach your TO BE (future state).
The thing that makes KPIs so elusive is that there are so many things that could be measured. KPIs can be strategic or operational. They can be based on output versus outcomes or process efficiency versus deliverables. They can be qualitative versus quantitative. They can focus on the leading versus the lagging. Or, they can measure the efficiency or the effectiveness.
Another thing that further complicates software development metrics is that the output is a widget and no two software outcomes are ever exactly the same (some may be similar but not the exact same). Even COTS projects have different configurations and are used differently in every environment.
Regardless of whether you are using Waterfall or Agile, many of these challenges can be addressed by learning about software development metrics. However, an Agile environment adds a new dimension that is not in your traditional menu of metrics. For more, read my Do Your Agile Project Metrics Measure Up? Why Traditional KPIs Won’t Work blog post.
As people go to great lengths to create agile environments, the need to quantify and measure success is a reasonable desire. Many organizations naturally think that measuring the following will determine their “agile” success:
- Team capacity
- Product owner’s time with the team
- Teams’ time in a co-located area
- Backlog growth
- Defect resolution rates
- Daily stand-up attendance
- Demos given
- Retrospectives held
- Testing occurring during the sprint
- Software delivered at every sprint
At a minimum, it probably is important that you are accountable for everything in this list. However, these measurements alone could stunt your agile growth because they focus on “doing agile” and not “being agile”. These metrics don’t communicate if you’re promoting customer collaboration, building trust, sharing knowledge, providing cross-functional training, prioritizing on high value, producing high quality software, fostering interaction, being responsive to change, and nurturing an engaging work environment.
For too long, metrics have put an emphasis on outcomes that are the result of unhealthy working environments, burn out, attrition, turnover, and work quality deterioration. Let’s look at some agile metrics that help measure and monitor more than ceremonies and focus on how the work is being done… metrics that measure and promote a healthy agile environment.
KPIs for Being Agile
Metrics today need to encourage the type of behaviors that cultivate the “being agile” world many organizations say they are striving towards.
Methods of Measurement
|Individual Behaviors (Being Agile)||Knowledge Sharing, Nurturing Others, Providing Opportunities for Team Members, Reaching Out, Challenges/Supports Others||360 Team Evaluations
Team Awards & Recognition
Individual Points Gamification
|Team Behaviors (Being Agile)||Respect, Trust, Honesty, Engagement, Determination, Collaboration, Passion, Initiative, Stretch, Transparency, Creativity/Innovation, Continuous Improvement, Swarm/Hovering Behavior||Niko-Niko Calendar
Caught in the Act Award
Team Points Gamification
Team Health Survey
|Team to Team Engagement |
|Customer Collaboration (Face Time), Expectation Baseline, Interaction, Initiative, Responding to Change/Complexity, Outreach to Other Teams, Shared Services||Customer 360 Team Feedback
Shared Services 360 Team Feedback
Unplanned Request Metrics
Community of Practice, Guilds, Squads, Tribes, Dojo Activities
(Business Value/ Business Drivers/Value Management Traceability)
|Growth, Innovation, Revenue Creation, Retention, Keeping the Lights On (KLO), Reduce Cost, MVP, ROI, Cost of Delay, Diminishing Returns||Post Implementation Evaluation
Business Value Burn Up
Measuring Team Members – Individual Behaviors
Create team awards and recognition. (If this is something that already existed in your non-agile environment, make sure to revise the criteria for the rewards and recognition to agile values.)
I recommend creating team awards around those who take time out of their individual task to help a team mate. In a 360 Team Evaluation, ask team members (and only those who work together) if their coworkers are team contributors via a survey. The chart below can help identify team contributor behaviors for which to place a nomination… and behaviors that are not so desired.
Level of Control
|Out of control||In control||Shares control||Dominates, dictator|
|Uninformed, out of the loop||Informed||Shares information||Withholds information|
|Needy, helpless, takes but doesn't give back||Self-reliant||Helpful, reaches out||Takes over or offers no help|
|Anxious, uncertain||Secure, confident||Reinforces, installs confidence in others, coaches, mentors||Arrogant, unapproachable|
|Barely enough, Just enough||Capable||Trusts others' capabilities||Pushy, conceited, "I'm right" attitude|
|Fearful||Risk Taker||Offers others challenges, Supports efforts||Reckless, Self-absorbed, My way, Disregards how action impacts the team|
As a group, the team determines how often the survey will be conducted, and the subsequent awards will be given.
Measuring Team Behaviors – Team Health
Agile benefits are realized when you have stable, high-performing team. To create a successful agile environment and culture, you need to measure the team’s health often and be ready to adjust when the numbers say the team is unhappy or unhealthy.
Something like team health or well-being might generally been seen as entirely subjective and thus impossible to measure and track. However, a technique called The Happiness Index, or the Niko-Niko Calendar, is a practice to help identify trends in the team’s attitude and tone.
It is also a great way to get the team engaged and talking about their challenges, not just general status updates. Sometimes the rich conversation derived from collecting this metric provides more value than anything else, especially if it leads to continuous improvement over time.
Measuring Team Behaviors – Team Engagement
The following chart is the foundation for a survey which was originally administered as a game using a platform called Kahoots®. Teams were quizzed on their understanding of why they were being asked to perform the various agile ceremonies.
|Product Backlog Grooming||Customer Collaboration - create a shared understanding of potential work|
|Sprint Planning||Working Software - gain consensus of the team’s capability and capacity|
|Daily Stand Up||Interaction and Individuals - open communication, daily commitments, evaluate priorities, synchronize the work|
|Sprints||Respond to Change - collaborate with the customer as a member of the team to jointly create a solution|
|Sprint Review||Customer Collaboration - showcase work, be accessible to the end users during the design process|
|Team Respective||Interaction and Individual - inspect the team dynamics, be self-aware, identify improvements|
Measuring Team-to-Team Engagement
As one might predict, those who still had some resistance and hadn’t bought into the agile approach often answered the questions wrong. The purpose was not to convince or brainwash everyone into believing the purpose and intent of the ceremonies; it was to help the Scrum Masters make the ceremonies more meaningful.
This measurement is to test how well the team of teams are collaborating in an agile environment. There is a series of 5 questions for each category:
- Are the sprint design changes being clearly communicated when they have an impact on others?
- Is the Scrum Master someone that you can reach out to or that reaches out to you for clarification?
- Is the staff open to cross-team collaboration within reason?
- Is the work space conducive to cross-team collaboration?
- Are the team skills as a whole in line with the work items assigned?
When coaching teams, I often point out that perception is important. If they disagree with the rankings, we talk about what they need to do to improve the problem area or improve the perception. Sometimes the team effort and contributions aren’t visible. This exercise may lead to a team-to-team discussion or may highlight roadblocks the Scrum Master needs to address.
Measuring Outcomes – Business Value
In Agile speak, the outcome is not defined by the product; rather, the product is the vehicle that leads to the outcome. There are a lot of non-agile ways to measure outcome, but a few of the most common are listed in the image below.
At the end of the day, if you can’t identify the outcome that benefits the business, then creating software for the sake of software is a waste of company money. Every member of the team should be fully versed and informed of how every item in every sprint contributes to an outcome that has business value.
The transparency element that is essential to the agile approach helps remove the disconnect between the contributions of each team member and the outcomes that the business finds valuable. To measure this type of metric, survey the team to be sure everyone has the same understanding. Some teams even use a point system to track and reward. Usually, there is a direct correlation between those who actively listen and ask questions.
Setting the Stage for Agile KPIs to Drive the Agile Mindset
My final cautionary note overall is that metrics can be used for good (to make the environment better) or for bad (to micro manage people). When a metric indicates negative trends, root cause analysis should be performed to understand the contributing factors. I strongly encourage a more organic approach toward agile metrics. You want to measure your success, not meter it based on a systematic pre-arranged list.
With creativity and openness to change, many of these agile metrics can work in your environment. Will all of them work? Perhaps not. But, try them. Stagger how you introduce them. Make sure people are being open-minded and give them a chance. Make sure everyone that is impacted understands the purpose and value of the metric. Ask for input upfront and feedback afterwards. One thing we do know is: if it doesn’t work, don’t waste more time doing the same thing. Learn fast and see what works in your environment.