Introduction
One of the popular software metrics are the DORA metrics. Here is what Google has to say about it:
These offer valuable insights into DevOps performance, focusing on four key indicators:
- Deployment Frequency—How often an organization successfully releases to production
- Lead Time for Changes—The amount of time it takes a commit to get into production
- Change Failure Rate—The percentage of deployments causing a failure in production
- Time to Restore Service—How long it takes an organization to recover from a failure in production
Achieving excellence in these metrics can earn organizations prestigious titles like ‘Elite.’ However, while continuous deployment offers benefits such as rapid feedback cycles and quick deployment of fixes, it also presents challenges:
- (+) Quick feedback cycle: You can deploy new fixes/functionality in a matter of minutes
- (-) No explicit QA cycle. No room for explorative testing before releasing to production
- (-) Continuous deployment is not free, it’s complicated and needs baby sitting
- (+/-) Strong emphasis on good tests to guarantee deployments are successful
While these metrics advocate for the adoption of continuous deployment, it’s essential to recognize that customer priorities may differ. Pursuing the ‘Elite’ status might overshadow more pressing concerns. In many cases, issues stem from a sluggish or inadequate product feature pipeline, highlighting the importance of balancing deployment efficiency with product development.
Feature metrics
Changing acceptance criteria
While embracing change is a core principle of agile methodologies, it’s essential to strike a balance. Constantly shifting acceptance criteria even before customer engagement can lead to inefficiency and wasted developer resources. Agile should not be a shield for indecisiveness. It’s crucial to thoroughly discuss and refine features before committing to development. Conducting preliminary discussions, creating simple mockups, and engaging with customers can prevent costly iterations and rework.
Remember, the majority of users prefer stability and consistency in their applications. Overzealous changes can disrupt user experience and lead to frustration. It’s important to recognize that most users are not early adopters; they value familiarity and reliability in their software.
Minimize stories with unclear acceptance criteria
Stories should not enter the backlog unless they are well-defined. Remember, zero clearly defined stories in the backlog will likely result in zero stories released by developers. It’s crucial to do your homework upfront.
Many developers tend to be introverted, and while asking questions may seem bothersome, it’s essential that developers clarify any uncertainties rather than leaving them for the users to discover. Developers excel at identifying potential issues in features. Embrace feedback and encourage questions. Measure success by tracking the number of stories that reach developers but aren’t yet ready for development due to unclear requirements.
I’ve worked with quite some companies that have very unstable acceptance criteria. Both the developers and product managers were frustrated. A runway of zero user stories ready to be worked on is recipe for a “bore down”.
Avoid shielding your developers from understanding the domain
In many companies, this aspect is often overlooked. Developers must grasp the context and intricacies of the domain they’re working in. Without this understanding, they risk becoming mere executors of tasks devoid of deeper comprehension.
From personal experience as an engineer, I’ve found immense value in acquiring domain knowledge. However, it’s not advisable for developers to immerse themselves entirely alongside customers to become experts. Striking a balance is key.
Some companies do not share any of the domain and make it a boring sequence of add button here, add field there job. Don’t stay there.
Uncharted acceptance criteria
Often nobody knows what the old/legacy system was exactly supposed to do. Better not to touch that code. This is the same terror of not having any tests. Am I breaking something? Nobody knows. Is it still used?
If it’s unclear whether the code is still in use, a pragmatic approach is necessary. If it’s obsolete, it’s best to remove it entirely. However, if it’s still in use, documenting its functionality and establishing acceptance tests become imperative. This process parallels the concept of technical debt but from the perspective of the product owner.
A useful metric to gauge this scenario is to map out the number of features lacking clear user criteria. This provides insight into the extent of technical debt from the product owner’s standpoint.
Features (in %) that are actually used by >50% of the users
Unused features are not just dormant—they come with a price tag. Every refactor that encompasses redundant or unused features adds to your expenses. Moreover, each additional feature contributes to the cognitive load for both users and developers alike.
Consider the cost implications: if only a minuscule percentage, say 1%, of your customer base actually utilizes a particular feature, it’s worth questioning its necessity. Assess whether the benefit it provides justifies the resources expended on its maintenance and inclusion in your product.
Addendum
Here are several technical metrics that I find particularly valuable but are often overlooked elsewhere:
Technical metrics
- Amount of refactors as a % of total development time. This is an equilibrium.
- Refactoring is essential for maintaining code health, but like any good practice, it can be overdone. Striking the right balance is key: excessively high percentages, say 99%, indicate that feature delivery is severely impeded. Conversely, if refactoring comprises 0% of development time, it suggests that technical debt is accumulating unchecked. Achieving a sustainable balance ensures steady progress while safeguarding the quality and maintainability of the codebase.
- Regression, where an acceptance criterion unexpectedly fails after making it to production, should be considered an anomaly.
- In an ideal scenario, such occurrences are prevented through a robust testing framework. Either automated acceptance tests are absent, leaving room for oversight, or the acceptance criterion lacks explicit definition, leading to ambiguity.
- Identifying and rectifying such regressions is imperative to uphold product integrity and customer satisfaction. It underscores the importance of stringent testing practices and clearly defined acceptance criteria throughout the development lifecycle.
- Time spent babysitting existing infrastructure/tests
- If you have a lot of flaky tests this wastes time and leniency towards failing tests may lead to regression
- Cost of the infrastructure
- For instance, introducing complex systems like Kafka when a simple REST endpoint would suffice represents a substantial investment. This decision not only entails the initial setup and maintenance expenses but also potential overhead costs associated with managing the added complexity.
- Careful consideration of the trade-offs between complexity and functionality is essential to optimize infrastructure costs and ensure resources are allocated efficiently. Choosing the most appropriate solution for the specific requirements can help mitigate unnecessary expenditures in the long run.