We’ve all seen toll system performance metrics in a solicitation that seem to defy logic – they can’t be achieved, can’t be measured, or can’t be tested. Sometimes these ill fashioned performance metrics make it all the way into an executed contract and cause all sorts of problems in delivery and subsequent operations and maintenance. So, what can be done to avoid this quagmire?
In the world of advancing technology and ever improving methods to automate toll collection, it is difficult to find a balance between specifying a performance metric that is achievable yet challenging versus specifying something that has never been achieved and is grossly out of the bounds of reality.
Performance metrics should be based upon past experience or concrete knowledge of a capability to meet a given specification.
Too often we see specifications that are tied back to overly aggressive marketing or simply repeating what has been written elsewhere. A recent example calling for automatic license plate recognition accuracy within the US at 99.9% accuracy stands out as a grossly unachievable performance metric. This example made it to contract!
System integrators respond to unachievable performance metrics in a variety of ways.
Some will push back during the procurement phase and identify trouble spots, but this approach could be seen as weakness or inability to respond to the work at hand. Another approach is to build risk dollars into the bid tied back to the expected costs associated with unraveling the onerous requirement(s) with the expectation that post contract negotiations or even claims will resolve the issue. When poorly constructed performance metrics make it into contracts they impact testing, delivery schedules, overall system performance, and the relationship between the owner and the integrator.
How can this be avoided?
When defining performance metrics, each metric should be analyzed from three angles – can it be achieved, can it be measured, and can it be tested. Measurement and testing analysis is fairly straight forward but determining whether a performance metric can be achieved requires a significant amount of due diligence in the form of past testing analysis, overall system architecture review, and pre-procurement exploration of product offerings.
About the author
- Kevin serves as RS&H’s Tolls Technology Leader and has a range of progressive experience in program management, consulting, and system planning. RS&H provides specialized solutions for the planning, scheduling, and testing of toll and managed lanes systems.
- Insights2017.05.04Toll System Integration Project Documentation – What’s the Right Balance?
- Insights2017.03.23How Can Toll Agencies Make the Most of the Last 5 Percent?
- Insights2016.03.09Software as a Service Solution for Tolling – Good Fit or Snake Oil?
- Insights2015.12.01Performance Metrics – Real World vs. Fantasy