1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice
You must be a registered member in order to post messages and view/download attached files in this forum.
Click here to register.

Calibrating internal auditors...

Discussion in 'ISO 19011 - Auditing Management Systems Guidelines' started by Andy Nichols, Aug 28, 2024.

  1. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,203
    Likes Received:
    2,598
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    It should be something we do, shouldn't it? How else do we know they give good results?
     
    Andy Nichols, Aug 28, 2024
    #1
  2. yodon

    yodon Well-Known Member

    Joined:
    Aug 3, 2015
    Messages:
    203
    Likes Received:
    119
    Trophy Points:
    42
    I had this argument with a colleague a while back. He asserted that his company "calibrated" their customer service (call center) staff. I disagreed, suggesting that they were (just) being trained. Each call is different and they may ultimately get to where the company needs, but it's not always readily measurable. You can collect data to determine training effectiveness, but even the most "calibrated" person may slip from time to time. Not to mention the argument for calibrating against a known national standard is sketchy, IMO.

    I think the same goes for auditing. It's more of an art than a science. We definitely need to ensure the auditors are trained and they can identify and write up proper and appropriate NCs, but I can't call that calibrating.

    Take this to the extreme and say that you're replacing auditors with an AI system. An AI system is "trained" and then we can expect a "good" result at a reasonably high rate (true positive). We have to understand, though, that there are cases where the AI system will not give a "good" result (false positive). It's still interpretive, not always black-and-white. I see this pretty similar to human auditors. We can train them until the cows come home but they will always be interpreting the information and there has to be oversight to catch those false positives.
     
    yodon, Aug 28, 2024
    #2
  3. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,203
    Likes Received:
    2,598
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Yodon, thanks for your thoughtful reply. I had a discussion with the president of the CAB I worked for, many years (now) ago and he was kicking around (with me) the idea of using control charts to plot auditor responses. FOr example, over time, does a particular auditor (or group) tend to write non-conformities around one requirements of the standard? Is there a trend and so on. I struggled with his train of thought at that time, however, without the pressure of that 9 - 5 job, I'm in a better position to research this concept.

    One of the interests I have stems from my recently discovering the so-called "Ladder of Inference" and its potential to affect audit outcomes. You and I both know of external auditor findings (published in web forums by frustrated auditees) which are the subject of this phenomenon.

    My thoughts, for what they are worth is that after initial "training" in auditor class, an organization should test the auditor periodically (which by definition is like a calibration) and check their response to various audit scenarios, including non-conformities. This might be considered a "linearity", or "bias" study. Over time, we might even consider the "Stability" of the response from auditors.

    The vexing question is: "Where to put the calibration sticker"? :D
     
    Andy Nichols, Aug 28, 2024
    #3
  4. Miner

    Miner Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    613
    Likes Received:
    532
    Trophy Points:
    92
    Location:
    Greater Milwaukee USA
    On their forehead where you can see it of course! :p
     
    Miner, Aug 28, 2024
    #4
  5. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,203
    Likes Received:
    2,598
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    I note, with interest, that ISO 19011 speaks to methods of "auditor evaluation" on page 33, section 7.4, table 2 - which includes such things as "testing" to evaluate "desired behaviours and knowledge and skills and their application" and gives examples including "oral and written exams, psychometric testing".

    Isn't that "calibration"?
     
    Andy Nichols, Aug 29, 2024
    #5