Conformance

 Last update: 2003/08/03

MPEG-1 is a great standard, but there is a potential practical problem in its adoption. Imagine I am a manufacturer and I choose to be in the business of making MPEG-1 encoders and decoders. I believe I have faithfully implemented all normative clauses in ISO/IEC 11172-1, -2 and -3 and I have checked that my decoder correctly decodes content generated with my encoder. Now, a customer of mine buys my MPEG-1 decoder and starts using it to decode content produced by an encoder manufactured by a competitor. Unexpectedly he encounters problems. My customer talks to my competitor and he is shown that content generated with my comptetitor's encoder is successfully decoded by my competitor's decoder. Who is right? Who is to blame? My competitor or myself? 

This problem is not new to ISO and IEC. The "Procedures for the Technical Work" prescribe that a standard must contain clauses that enable users of the standard to assess whether an implementation conforms to the standard. One could even say that a standard is useless if there are no procedures to assess conformity to it. It would be like issuing a law without having a tribunal one can make recourse to assess "conformity" of specific actions with the law. 

The conformance problem is less well known to ITU because, when telcos were a regulated business providing a "public service", they performed the "conformity" tests themselves to make sure that terminals from different manufacturers would interoperate correctly on their networks, without letting subscribers discover the kind of incompatibilities I have just described. They used to put a "seal of approval" on conforming terminals. Telcos used to do this because it was part of their public service license, but also because keeping subscribers happy and not letting them suffer from incompatibilities was good for the business, an old wisdom that too many "new economists" have forgotten. 

When the telecommunication business became deregulated, independent Accredited Testing Laboratories (ATL) were set up. For a fee and after successfully passing the test, ATLs issued certifications that a given product conformed to the standard which the manufacturer made conformance claims to. But even ATLs are a byproduct of the traditional "public service" attitude of the telcos. 

In the IT and CE domain, which MPEG-1 ends up also belonging to (although not wholly), the attitude has always been more "relaxed". If you buy a mouse and you discover that it does not work on your computer, what do you do? If you are lucky the shop you bought it from will refund your money, if not you are stuck with a lemon. The same if you buy a component for your stereo. Sure, the consumer is protected, because if he is dissatisfied he can always take legal proceedings… The attitude of the IT and CE industry has always been one of either not claiming anything or, at most, of making "self-certification" of conformity. 

That, however, is something that may work well when there is a market with large companies, producing mass market products, possibly not terribly sophisticated and where the product itself depends on a key technology licensed by a company that puts conformity of the implementation - to be verified by the licensing company - as a condition for licensing. The licensor has an interest to make sure that the licensee behaves correctly because he is interested in the good name of the technology and, again, because that leads to satisfied customers and hence more revenues. This has regularly been the case of major CE products. 

Virtually none of these conditions apply to MPEG-1, and certainly not the last. There are multiple patent holders for the MPEG-1 standard, but none has the authority or interest to become the "godfather" and oversee the correct implementation of the standard. Therefore the approach adopted by MPEG has been to develop a comprehensive set of tools to help people make independent checks of an implementation for conformance. 

Part 4 of MPEG-1 "Conformance" gives guidelines on how to construct tests to verify bitstream and encoder conformance and bitstream test suites to verify decoder conformance.  Encoder conformance can be verified by checking that a sufficient number of bitstreams generated by the encoder under test are successfully decoded by the reference software decoder. 

Copyright © 2003 chiariglione.org