There’s More to Quality Than You Think

Whether you’re purchasing new software for your enterprise or developing it in house, quality should be a primary factor in your evaluation and acceptance process.

If you’re making a purchase then you have many different criteria to consider in selecting the right product. Some people look for cheap. Others look for durability. Still others, like me, look for quality. In my opinion, any time you’re looking to purchase a product or service for your enterprise, quality should be your main concern.

But how, exactly, do you measure quality? Thanks to the International Organization for Standardization, we do, in fact, have a way. It’s called ISO/IEC 25010:2011, or, as I like to call it, good old “Twenty-Five Ten.”

Twenty-Five Ten is an international standard for the evaluation of software quality. I have become a huge fan of this standard. It can be an incredibly useful tool both for evaluating software for potential purchase and for evaluating software that you develop for your own customers.

The standard defines quality by dividing it into eight distinct characteristics:

Functionality. This characteristic is just what it seems. Does the system meet your needs? Does it do what it should do, and do it completely?
Efficiency. How efficient is the application? Do you have performance SLAs? If so, can this product meet them? What about transaction load? Can the application handle the number of concurrent connections and concurrent users that your organization requires? If the application requires data loads or refreshes, how long do they take to run? What kinds of impact do those batch jobs have on other applications or services?
Compatibility. How well does it integrate with other systems in its ecosystem? Does it have an API? If so, is the API robust and fully functional? Is operating system compatibility an issue? What about database compatibility? Does it support your preferred database platform?
Usability. What is its user interface like? Is it intuitive, easy to understand, attractive? How much training is required to use it? Do people pick it up easily, or is advanced training a requirement?
Reliability. How stable is the application? How much work will it take for the application to meet your DR requirements? Is the application fault tolerant? Load balanced? If it needs to be load balanced, can it be?
Security. Will the application meet your security guidelines? Is data confidentiality adequate? Are the proper auditing controls and reporting capabilities available?
Maintainability. How easily can it be customized for your organization? Do those customizations remain in place after upgrades? How are patches applied? Is it easy to do or hard? How are backups made? How easy is it to test the application before and after upgrades?
Portability. How easy is it to set up and maintain multiple environments? Can changes be easily migrated between environments? Is DevOps on your roadmap? If so, does this application support continuous delivery and automated deployments?

Using it with vendors

Keep in mind that while the points above focus on determining the suitability of a vendor’s application for purchase by your organization, there are plenty of other ways to use the standard. Software vendors can maintain this information and share it proactively as an assessment of the quality of their offerings.

In the past I have asked potential software vendors to tell me how they maintain compliance with this standard or, if they don’t currently use it, what techniques they use that could be mapped to it. Since all of the considerations above are valid, they should be able to detail what they do to make sure that their offering addresses them.

In one case, a software vendor let me know that they weren’t bound by ISO standards and didn’t have to answer the question. The response I wanted to make was, “Then we don’t have to buy your product.”

Another vendor went to great lengths to map all of their internal practices and methodologies to these characteristics for us. In the end, they were actually quite grateful for the question, as it helped them do a better job of quantifying the quality of their product. We did end up purchasing their product and were very happy with the purchase.

Another potential use of these characteristics was presented to me recently when a peer was sharing the progress of their recent RFP process. They had looked at several different products and had decided on a front runner. The problem with that product, however, was that peer reviews indicated concerns with certain parts of it. The vendor claimed that the new version of the product had addressed all of those issues, and the demonstrations did seem to support their assertion. To confirm this, the purchase could be made contingent on the vendor passing an external quality assessment.

There are external groups that provide this type of service. One such group is CISQ, the Consortium for IT Software Quality. They have used these characteristics to build a quality certification program that includes a metrics-based assessment methodology that can give a potential purchaser peace of mind. If the vendor’s new version and processes achieved an acceptable rating then the purchase could continue and everyone would get what they want. If not, the vendor would know exactly which areas they needed to improve. Again, a win-win.

Using it on your own teams

The use case that I find the most interesting, however, is as a quality assessment tool for internal development. Whether you are an enterprise with many separate development teams or a software vendor with a large product base to maintain, this is a tool that could undoubtedly be used to improve the quality and reliability of the output of those teams. It is your choice whether to use these characteristics as a general guideline and develop your own framework, or to purchase a product like CAST to automate the enforcement of those quality rules.

One final thought for you on quality. The list above gives eight primary characteristics that make up the overall quality of an application or service. Most people test only the first one: functionality. If you’re someone who is concerned with quality and yet you are among those who currently only test functionality, just think about how much you are missing.

2026 update: ISO revised the standard as ISO/IEC 25010:2023. It now has nine characteristics. Safety was added, Usability became Interaction Capability, and Portability became Flexibility. The point of this essay still stands: if you only test functionality, you’re testing one characteristic out of nine.