Ask any legal or procurement department and they will insist that benchmarking is necessary in any services contract with a duration longer than 2 or 3 years. Ask any supplier for their opinion and they will insist that their services are best in class and completely benchmarkable (of course, that is prior to being awarded a contract).

However, in a world where IT is changing at an exponential rate, it’s difficult to believe that Services can be truly benchmarked against the market place. Something which is continually in flux means that its parameters are also subject to change. Here enters the consultant and the term ‘normalization’! Of course, traditional parameters such as workforce locations, language support or support times can be normalized by taking into account the delta in salaries and team size required to support longer support hours or faster response times but normalizing other aspects such as licenses, scale (HW discounts and scale of economies are not typically linear) and industry (regulatory demands) are not easily dealt with by algorithms.

Before I incur the wrath of benchmarking companies and consultants, let me be clear that I am not saying it’s impossible to take the above into account, but there are a few things that people should consider before demanding the next benchmark (just for info, I have experience as the supplier being benchmarked, consultant performing benchmarks and the customer receiving the benchmark). 

1. Use it or abuse it! As a consumer of IT services, it can be easy to abuse a benchmark clause by applying it to a select sub-set of services. You need to consider that IT services are a product just like any other and similar to UK supermarkets where turkeys are sold at ridiculously low prices before the Xmas period to pull in customers (knowing they will then spend more on the other groceries), some of the services you consume have likely been priced very attractively to win the original deal. Suppliers spend weeks creating their commercial proposals, trying to understand which services are sensitive to their future customer. Financial engineering and risk management are key in such deals.  A benchmark is likely to drive down prices of the services which were not ‘turkeys’, causing the overall deal for the Supplier to become less attractive. This in turn will push the supplier to recover lost margins by reducing costs and potentially reducing the quality of service. This doesn’t mean you shouldn’t benchmark but be aware of the total deal when finally negotiating the results of the benchmark.
2. Who are you kidding? Something in humans tends to make us think that more is better and that if you receive more of something for the same money then you have a better deal. Unfortunately, in IT, this is not necessarily the case and many services are based on risk models. Consider for instance your email. We’ve all seen the amazing increases in mailbox size being offered over the past years. For less than 3 euros you can get GB’s of storage when ten years ago you were lucky to have more than 50MB of space being offered. Did storage get so much cheaper? Well, yes, it did, but not enough to allow suppliers to exponentially grow your mailbox. What happened is that Suppliers have more data on your email usage. Whilst you want the security of being able to store 5GB of email, they know that the average user still consumes less than 1GB. They are willing to take the risk of offering you more, knowing that only a small percentage of users will actually make use of it. When benchmarking, it’s equally important to understand where risk models apply and avoid kidding yourself that you are getting nothing more than an unused promise. Storage, email, backup and virtual desktop services are obvious candidates to be careful of.
3. Transparency is not always the best policy. The majority of benchmarking agreements are based on receiving full transparency in the services and parameters, but also the pricing currently being delivered. Whilst the supplier performing the benchmark will undoubtedly ignore these prices when delivering the market prices from their database, the supplier of the services being benchmarked can use the opportunity to object to the results which are heavily normalized. By performing what is known as a ‘blind benchmark’, where the current prices are revealed only upon receiving the results, it’s easy to avoid any such objections. Obviously, significant anomalies between the existing price and the benchmark price will then require further explanation.
4. The reason behind the benchmark. Sometimes benchmarks are initiated simply because it’s part of the corporate policy, but I have also experienced them being driven due to poor performance (why am I paying so much for such a poor service?), change of management (the need to demonstrate their value add) or a re-think of the sourcing strategy (fueling the business case for change). Understanding the reason behind the benchmark, whilst not necessarily communicating this (for obvious reasons) to the suppliers can help in asking the right questions during the exercise and influence the non-commercial results. Think carefully about what you want as output from the benchmark exercise before starting it and make sure that those who are executing it know their role and desired output.
5. Demand more than just commercial results. Most benchmark reports provide the crucial commercial figures that everyone expects, often accompanied with realms of paperwork justifying the normalization factors and methodology that has been applied. Unfortunately, unless there is a price reduction to be applied, the report has very little value add. This is unfortunate, because there is a lot that can be learned from which both supplier and customer can benefit. For example, if a customer has a service which is exclusive of upgrades (i.e. moving from Windows 2020 to version 2023) and the peers of the benchmark include this service, although the results will have ‘normalized’ this aspect out of the results, it is worthwhile noting the non-normalized result to understand the costs of this aspect. Similarly, a service may have larger parameters within a peer, for which almost no normalization is applied. This indicates that the parameter has relatively no commercial impact and you may be able to adjust your service accordingly without additional costs. 

Suppliers of services tend to be relieved that a benchmark has passed and quickly move on to their daily business. This is unfortunate because they too can use such information to demonstrate where they are actually performing well or use the report to demonstrate their willingness to embrace continuous improvement.Although I certainly don’t cover all of the mysteries and complexities of benchmarking, I hope that whoever managed to read this far now at least has started to consider how they may better approach their next benchmark.