1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice
You must be a registered member in order to post messages and view/download attached files in this forum.
Click here to register.

Use of WHAT IF as the main auditing tool for Risk-Based Thinking

Discussion in 'ISO 9001:2015 - Quality Management Systems' started by tony s, Apr 9, 2017.

  1. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    We have just undergone the Stage 1 audit by our chosen CB. My concern was their approach in evaluating how our organization satisfies the requirements in determining risks and opportunities. To provide you some background, we utilized the RBT concept during our strategic and operational planning. We determined risks/opportunities when we established our strategic plans and when we plan for the controls to be integrated into our operations. Both levels of planning were fully documented (this will also ensure conformance to clause 6.1.1). Actions to address risks/opportunities are also in place (this gave us the assurance to satisfy clauses 6.1.2 and 4.4.1f). However, when they start auditing they keep on asking the question WHAT IF. Just to cite a few: "what if the president is not available? what if the funds are not available? what if manpower is not adequate?" Then followed with a question "Did you include those in your inventory of risks?"

    Of course, controls to manage the above risks are already in place but what I can't dig is when they conclude that our tools for determining risks/opportunities are inadequate just because we failed to include the WHAT IF issues they asked from us into our inventory of risks.

    What's your take on this?
     
  2. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    Which "what-if" issues did they ask about?
     
  3. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
     
  4. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Your auditor is grasping at straws. As with many who don't get this requirement, THEY shouldn't be second guessing you. If you went through some kind of analysis, like PEST, SWOT or something else you're comfortable with, your auditor should accept that and keep their opinions to themselves... What if they shut up and just audited for once?
     
  5. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    It sounds like the 5-Y sort of drilling down to see what is to be done to address the risks. That is not unexpected, but I am interested in the term "inventory of risks." Does your organization use that term?

    I guess I am wondering whether they were questioning the extent to which risks were considered, or the depth to which they were considered.

    That is, if risks included areas beyond those of production and if they included operational contingencies like loss of key personnel and the potential of economic impact from a recession. Both are real considerations. Both are business risks. Both could stop the organization if they are not considered and addressed. Were they considered and addressed?
     
    etorresg and charanjit singh like this.
  6. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    At the strategic level we used a "Gap Analysis" tool to determine risks from internal and external origin. The output of this analysis provided us the basis for establishing our strategic plans. At the operational level we used a tool that is somewhat similar to FMEA but without the computation for RPN. Both tools produced a list of risks/opportunities (e.g. inventory of risks).

    I know that there are risks that were not captured by the tools we used. We even explained to them that controls are in place to address issues like "what if the president is not available?" - hence we have the vice president. If auditors will audit like this and will ask anything they want, I fear that auditees will never satisfy the RBT related requirements.
     
    Andy Nichols likes this.
  7. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Tony, they already do, but it was on less controversial issues...
     
  8. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    Our approach in determining risks/opportunities goes beyond the core processes. We used an FMEA-like tool in determining risks/opportunities for support processes (e.g. HR, Maintenance, Purchasing, etc.) including management processes (e.g. Corporate Planning, Internal Audit, etc.). Like what you have in your Risk Based Planner. Obviously, our tool may not capture all kinds of risks, let alone "What if the president is not available?" type of risks. Although such type of risks were not included in our chosen tool, controls are available to address them (e.g. president not available - appoint a vice president; funds not available - we have budget planning; manpower not adequate - hiring process). This goes without saying that these risks were known to us - only not included in our FMEA-like tool.

    There can be so many risks to consider from negligible to catastrophic if auditors are limited to use only this WHAT IF type of probing. Without any boundary, auditors can ask questions like: What if the entire project is cancelled? What if a natural disaster occurs? What if China starts invading your country? What if its the end of the world? What if you are being audited by auditors whose only probing tool is WHAT IF?:confused:

    ISO 9001:2015 has given us the boundaries on where to focus our RBT mindset. Clause 6.1.1 provided this by stating "When planning for the quality management system...determine risks/opportunities that need to be addressed..."
     
  9. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    If you go back to basics, which this auditor clearly didn't, the foundation is the Context of the Organization. It's going through the requirements - doing something like a SWOT (See ISO/TS 9002) and based on the outcomes, the planning is supposed to take this into consideration. If, let's say your management team identified a weakness is that you're about to lose 50% of your skilled workforce due to retirement (they are also a strength, of course) and it's difficult to recruit replacements, you may decide to hire and train (like an apprenticeship program) entry level people. So then, the question looks at your current hiring and training practices/processes and asks you to look at them to see if they are going to be effective in addressing that risk. If not (and my guess is most hiring/training programs would be much good) then what changes need to be rolled out to manage that to provide a set of new, skilled and competent workers who can step in to replace the retirees...

    My guess is that many CB auditors have little clue as to why this stuff is in the new standards, beyond asking banal questions about "risk" and so on. So much for CBs offering training when their own auditors don't know...
     
  10. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    I have clients who plan for natural disasters. I also had an employer whose major client asked them to build a second site so as to mitigate the risk of work stoppage in event of fire/whatever at the main site. I worked at another place that planned for disruptions due to ice storms and pandemics (they set up work-from-home arrangements for selected personnel).

    RBT can also be evident through the setup of calibration parameters, to include the risk and cost of getting it wrong. The standard, now being a business document and not just for QA/QC, invites us to think outside the box in order to stay competitive. It seems your CB was asking about such possibilities. It is probably already being done at high levels. Have you asked your management about it?

    I agree with Andy that it should be related to Context: that is, issues and interested parties. I can't speak to the training this auditor may or may not have had, though, as I was not there and he is not around to tell us. I can, however offer that this requirement is sure to generate a lot of confusion. It is a great place for disputes, which I hope clients will pursue so we can have opportunity to reduce the variation in auditors. There will be a period in which ABs also weigh in through nonconformities to CBs, and the technical committee will eventually publish interpretation documents. The problem here is how, and how far, to apply the requirement.
     
    tony s likes this.
  11. Paul Simpson

    Paul Simpson Member

    Joined:
    Aug 6, 2015
    Messages:
    41
    Likes Received:
    61
    Trophy Points:
    17
    Interesting thread on how RBT is being audited on the ground. You haven't mentioned, Tony, if the auditor has indicated whether any findings exist (NC / Observation) following the audit. I'll bide my time until I know what the outcome is.

    In some way I am reassured that the auditor isn't glossing over the new requirements and s/he is providing a robust examination of your processes. For certification to be of any value to your current and future customers your processes have to be shown to be effective. We don't have enough information at this stage to provide you with reasonable feedback.
     
    tony s likes this.
  12. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    We also have those plans. Even before the first edition of ISO 9001. We are just using a new RBT tool intended to enhance our operational controls and the CB auditors find it inadequate just because we didn't include the WHAT IF issues they raised.
    Maybe the reason why clause 6.1.1 mentions "when planning for the QMS". Even ISO/TS 9002 have this statement: "There are various situations where risks and opportunities should be considered, for example strategy meetings, management reviews, internal audits, different kinds of meetings on quality, meetings to set quality objectives, the planning stages for the design and development of new products and services, and the planning stages for production processes."
    Since we are still in Stage 1, they only raised it as an Observation.
     
    Jennifer Kirley likes this.
  13. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    Given this was a preliminary and not the certification audit, the auditor was just throwing out questions. Including it an an OFI signals there might be attempt to write a nonconformity against it in the certification audit, but based on all I've seen I wouldn't offer more than an OFI to consider additional beneficial places/ways to apply the what-if questions. If an auditor tries to do more, it is a good place for a dispute.
     
  14. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    It's a "Stage 1" audit - part of the Certification process. The auditor isn't there to "throw out" comments like this and report "OFIs". ISO/IEC 17021 is quite clear on what the purpose is. Non-conformities are not appropriate and the report out is supposed to identify issues which MAY become a nonconformity at the Stage 2.
     
    charanjit singh likes this.
  15. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    True, Andy. There should not be OFIs in a preliminary (I'm not sure if that is what they call an "area of concern" for the certification audit), and cannot be NCs though I did not see evidence here that any were written. I also can't speak for this registrar, their training or their rules or documentation, especially as I have no information on any of those things. I don't have the report and wasn't present, so there is little more to constructively add.
     
  16. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    From what I've experienced and read about people struggling with (like this thread), it's clear (to me at least) that the training has been focused on one tiny aspect - that of risk (which is many things to all manner of people) - instead of the greater understanding of the organization's context and working through the issues that they have to face etc. That's the story clients should be well versed in presenting, instead of waiting for their (hapless) CB auditor to fire off some narrowly focused audit question about "risk" and then convoluting that into something bizarre...
     
  17. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    Andy makes great points. An exploration of context should be the foremost focal point, but many people also struggle with identifying issues, interested parties etc; from there the RBT can become just as shaky. This is why I give out the links to help sources, including the TC 176 web page and the Audit Practices Group page. I also suggest people obtain and read ISO 9001:2015 in Plain English by Craig Cochran. it does a good job of explaining these subjects.
     
  18. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    Nice references. I wish CB auditors will find time to look into these sources.:D

    As for the classification of the stage 1 audit finding, the CB written report specifically calls it "inadequacies against the requirements of the standard".
     
    Andy Nichols likes this.
  19. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    So do I. That's why I give the links to all my colleagues I rub elbows with, and also to my clients.
    If that makes its way into the certification audit as a nonconconformity, I would say it's worth a dispute. I am very interested to see how CB auditors are expected to enforce such ideology (it isn't the only one), however correct it may seem to us; if it isn't listed as a "shall" in the standard or the organization's procedures, and there is no evidence of consequence indicating conformity to the standard has not been demonstrated (demonstrate meaning a combination of documented information and/or what auditees say and/or conditions we observe), in my view we don't have that kind of freedom.
     
  20. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    I'm crossing fingers but if it finds its way onto the stage 2 as an NC, we are ready to battle. We'll surely strongly assert our position as the "audit client".