Some considerations
I have seen metrics fail in a number of organizations trying to improve. Broadly speaking, the reasons for failure are one of the following:
Measuring the wrong thing
Many qualities of interest are hard to identify precisely (think of “code smell”) or difficult to measure directly. Metrics are usually project qualities that are easy to measure but you can only imprecisely measure what you want. The classic measure of progress – lines of code per day – turns out to be a horrible measure because it doesn’t measure the quality of the code, so it cannot take into account the rework required when fast code production results in low code quality. Nor is refactoring code “negative work” because it results in fewer lines of code. A better measure would be velocity, which is a measure of tested and verified features released per unit of time.
Another often abused measure is “hours worked.” I have seen companies require detailed reporting on hours spent per project only to also levy the requirement that any hours worked over 40 hours per week should not be reported. This constrained metric does not actually measure the effort expended on project tasks.
Ignoring the metrics
I have seen many companies spend a lot of time gathering metric data (and yes, it does require some effort and does cost some time, even when mostly automated), only to make the very same mistake time after time. This is because while these companies capture the data, they never actually use the data to improve.
No authority to intiate change
Gathering and analyzing metrics is often seen as less valuable than “real work” and so personnel tasked with these activities have little or no authority.
Lack of willingness to follow through
I have seen companies pay for detailed, quantified project performance data only to ignore it because there was little willingness to follow through with needed changes. This lack of willingness can come from management being unwilling to pay for organizational improvement, or from technical staff being afraid of trying something different.
Metrics should always be attempting to measure an objective rather than a means. Rather than “lines of code per day,” it is better to measure “delivered functionality per day.”
