Contributing Writer, Donald Byrne, CBCP, CDCP, CBRO-M, Lead Auditor
As part of the PS-Prep posting in the October 16 issue of the Federal Register (Vol. 74, No 199 Page 53288) the Department of Homeland Security posted seven questions for which they are seeking public feedback. The sixth question dealt with the feasibility of using a capability maturity model (CMM) as part of the certification process. This article addresses this question and provides my perspective as a UKAS approved auditor on this topic.
The concept of a capability maturity model was developed in the late 1980’s at the Software Engineering Institute at Carnegie Mellon University (CMU). As originally conceived, the model attempts to evaluate the ability of an organization’s software development process to successfully complete a project with high quality. The model uses a variety of criteria to determine the maturity level (think “quality level”) of the process. Beginning in the 1990’s this methodology was applied with varying degrees of success to other types of projects including construction and manufacturing.
As first conceived by CMU, the model recognizes five levels of maturity from Initial (Level 1) to Optimized (Level 5). Subsequent applications of the CMM concept have used a six or even seven layer framework. Some of these models have over 400 individual elements that must be evaluated in order to determine the maturity level of the operation or business unit. This difference in model structure highlights one of the main issues associated with attempting to use a CMM approach to PS-Prep. The simple fact is that there is no standardized or internationally recognized representation for a capability maturity model dealing with preparedness. The structure, content, organization, and criteria are all arbitrary, and therefore not suitable for a standards-based approach to business resiliency evaluation.
Next, because there is no accepted set of criteria or even an agreed to structure, it is difficult to imagine how to apply this methodology to the three areas highlighted in PL 110-53, namely emergency management, disaster recovery, and business continuity. Even more problematic would be the development of a model that is applicable across industries and organization size.
Last, by design, a CMM uses an element-based approach to assessment. This is in contrast to the model used by the International Organization for Standardization (ISO) which is a management system or process-approach. While it is not a requirement that a management system approach be used as part of a PS-Prep standard, many organizations do prefer this structure since it draws on the strategy employed in other ISO standards such as ISO 9001 (Quality Management), ISO 28000 (Supply Chain Resiliency and Security) and ISO 27001 (Information Security Management System). Is it even possible to mix an element-based model with a process based model in a meaningful way?
It should be noted for completeness that NFPA 1600 is an element-based standard that was recommended by DHS for use in the PS-Prep program and many other element-based standards do exist.
A Layered Approach?
Some CMM proponents have suggested “layering” the CMM model on top of the proposed PS-Prep standards (NFPA 1600, BS 25999, and ASIS SPC.1). While this may be an intellectually interesting idea, it is all but impractical from a business perspective. In fact, even attempting to follow a layered model might require a team of up to four experts to conduct the review. First, you would need an auditor who was trained to evaluate the CMM model. Of course, that begs the question of which CMM framework to use since, as pointed out above, there is no single standard in this area.
Second, a subject matter expert (SME) on the application of the model against the various standards would have to be on the team. Of course this assumes that you could find or train someone on this topic since it would be a new field of discipline with no recognized experts. Assuming a training course on this subject could be developed in a timely manner, the question remains – who would have the expertise to develop such a course? Certainly, there are certainly some organizations who may feel they know the subject well enough to perform undertake this task but again – against which CMM specification would be used as the basis of this course?
With these two elements in place, then the certification team would still require experts on both the preparedness standard (such as ASIS SPC1) and auditing experience.
With this team in place, how would one settle disagreements between the various auditors and SME’s? Which framework or evaluation procedure would take priority – the CMM or the Standard? Could one be assured that this procedure would be applied consistently across industries and businesses? How would an auditor or assessor judge conformity or non-conformity of an individual plan element if that decision now had to be qualified as to the level of CMM completeness? Are you really more mature in terms of preparedness because you have at least some type of fire suppression equipment in place versus another firm that has a detailed emergency response process? Following this concept, how would you judge the maturity level of a requirement such as “… must demonstrate management support for the program?”
To What End
In summary, a CMM approach to assessment simply adds a layer of ambiguity and unnecessary nuances to an already complex process. It may significantly increase the cost of conducting a conformity review and does not add obvious value.
The one question that proponents of adopting a CMM approach to resiliency have to answer before proceeding with further promotion of this idea is: “Would adding a CMM assessment model add value to a certification?” Specifically, would knowing that you are at Level 3 for section 5.2 of British Standard 25999, but only at Level 1 for section 5.3 help saves lives? Are these statements even meaningful since BS 25999 is a process-oriented standard and should not be evaluated on an element basis? Would such information allow management to make decisions that will save jobs? Can you really say that that one business is 35% prepared for a fire while another is 45% prepared? And if you could get around all the technical complexity – what is the value of this knowledge?
While some groups are trying to apply a CMM to business continuity planning in industries such as banking, and healthcare, it is difficult to image the regulators of these business sectors embracing this approach since it would throw their current inspection processes into pandemonium.
In closing, a properly structured CMM approach can offer many benefits to a wide variety of projects. However, it is just as clear that CMM has no place in the certification process of PS-Prep.