1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice
You must be a registered member in order to post messages and view/download attached files in this forum.
Click here to register.

Auditing for effectiveness

Discussion in 'ISO 19011 - Auditing Management Systems Guidelines' started by Andy Nichols, Aug 15, 2015.

  1. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,285
    Likes Received:
    2,631
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    I was out riding my motorcycle this morning (doing some "RBT") and I was humming along at the posted speed, when a car came along, overtook me (must have been exceeding the 40 limit) and them pulled across in front to take a right turn. The car ran through the light (it was posted "No Turn on Red") and, as I approached to make the same turn (seconds later) the right filter came on. A few hundred yards up the road, the car was just slowing because the next light was at red...

    I was struck that the car was non-compliant and, clearly if the performance goal was to "save time", they didn't, because I was right behind them and complied with all the requirements... arriving at the same light as it turned green.

    It would have been easy, if we were auditing, to write up the speed and turn violations as non-compliances without looking at the process to see how effective it is. Had I been able to record the process, it might have been interesting to point out to the driver that they didn't save anything, but ran the risk of 2 tickets...
     
    Andy Nichols, Aug 15, 2015
    #1
    GStough, Ravi Khare, Ganesh Sundaresan and 1 other person like this.
  2. Ganesh Sundaresan

    Ganesh Sundaresan Active Member

    Joined:
    Jul 31, 2015
    Messages:
    66
    Likes Received:
    36
    Trophy Points:
    17
    I often say to my colleagues that unfortunately there is no arithmetic calculation to prove that a week spent longer in plan stage saves months together in do, check and act stages and therefore will take us earlier to the intended destination.
     
    Ganesh Sundaresan, Aug 16, 2015
    #2
  3. Eric Twiname

    Eric Twiname Well-Known Member

    Joined:
    Jul 31, 2015
    Messages:
    329
    Likes Received:
    232
    Trophy Points:
    42
    Location:
    Northeast USA
    In reaction to the match between the title and the topic...I open up the question of "Internal Audit" or "External audit" ?
    I do so simply from knowing that a lot of folks here are auditors...auditing other companies, vendors or processes.

    I think that an effectiveness audit is the main role of every manager, supervisor or leader in an organization.
    The problem is that many of them are horrible at it.
    If I wanted an external audit for effectiveness, the term would be "consultant"...not "auditor".
    For an internal audit for effectiveness...you need a champion among the ranks.

    My friends tend to jibe me about following the posted speed limit, since most folks do not.
    I've noticed that they beat me to the end...typically by 3-4 minutes on a 20mile drive.
    That's 3-4 minutes I'm willing to spend to avoid hundreds of dollars per year.
    and I can't tell you how many times I've been honked at for not turning right on red, when the "No Turn on Red" sign was clearly visible.
     
    Eric Twiname, Aug 17, 2015
    #3
    Andy Nichols likes this.
  4. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,285
    Likes Received:
    2,631
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Exactly so, Eric. I was lucky to observe an audit where there was ineffectiveness to be seen, clearly, all around, yet the most senior management sat in splendid isolation of it all... Needless to say, one day, "corporate" saw it and wasn't impressed. Those managers don't work there now.
     
    Andy Nichols, Aug 17, 2015
    #4
  5. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    723
    Trophy Points:
    112
    Location:
    USA
    This is why I always save Internal Audit for the last day of my visits. I compare what I have seen against what they see internally. It becomes an eye opener for the astute and caring. Many times the issue is one of desensitization - people see the same little things over and over and no longer recognize them for the issues they are. This is one reason why the standards all ask for objectivity in the selection of auditors, but it can still happen.
     
    Jennifer Kirley, Aug 17, 2015
    #5
  6. Nikki

    Nikki Well-Known Member

    Joined:
    Jul 31, 2015
    Messages:
    268
    Likes Received:
    141
    Trophy Points:
    42
    Location:
    Maine
    Interesting outlook on the situation and so true!
     
    Nikki, Aug 21, 2015
    #6
    Andy Nichols likes this.
  7. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    I'm a bit confused about the focus of this thread...and perhaps the answer is 'Yes...both', but are we talking about:

    1. The effectiveness of the internal audit process
    2. Auditing processes for effectiveness?

    I'm currently not playing in the ISO 9001 sandbox, but rather have been carte blanch to create an internal assessment strategy for the organization in which I work. We've identified our processes, standards and requirements...as well as metrics.

    While it looks better in a table format, an example is:

    Process - Scheduling
    Standard 1 - Identification of Need for Scheduling
    Requirement 1.1 - Notification is received from ....
    Requirement 1.2 - The Planner is used for...
    Metric 1.1 - Not applicable
    Standard 2 -Service Care Provider Selection
    Requirement 2.1 - The Scheduler reviews...
    Requirement 2.2 - The [proprietary software] is used to identify best match...
    Requirement 2.3 - ESA compliance is reviewed before...
    Metric 2.1 - Primary assigned (%) | Target - 95% | Actual - [provided by site]
    Metric 2.2 - ESA Violations (#) | Target - 0 | Actual - [provided by site]

    Not every Standard requires a Metric, but every Process must have at least one Metric within its scope.

    Why? (because if you've been reading my recent posts, you know I'm all about the Why)

    The Metric is What We're Trying to Achieve and the Requirements within the Standards are the How We're Going To Do It. And while I'm keeping the information really simple here, without the background rules, together, they can help tell the story about the process and if it is working effectively.

    The quadrant shows where they land on the conformance score and helps guide future discussions on addressing gaps or using the site as a possible benchmark.
     

    Attached File(s): 1. Scan for viruses before using. 2. Report any 'bad' files by reporting this post. 3. Use at your own Risk.:

    RoxaneB, Aug 27, 2015
    #7
  8. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,285
    Likes Received:
    2,631
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Auditing for effectiveness. I often read audit reports which say "Process not effective", but there's sip/nada/nothing/rien to back up the assertion. Just because a procedure wasn't available or a "good" description, doesn't mean the process "wasn't effective". Unless I see results, how can I make that determination? I read thinly disguised reports of compliance to a document, but couched in "effectiveness" terms which aren't justified.

    <SMH>
     
    Andy Nichols, Aug 27, 2015
    #8
  9. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    That's why there should be a discussion on a metric and if targets are being achieved, or the existence of a positive trend if the target is not currently being met.

    Auditor : So, tell me about your [insert name] process.
    Auditee : Well...this...that...blah blah blah.
    Auditor : And how do you know if it's working? How do you know it's good? (because, depending on the auditee, 'effective' might confuse them)
    Auditee : We have indicators and we can easily see if we're hitting our targets, within control limits, trending well, blah blah blah.
    Auditor : How are you doing right now?
    Auditee : (glances at dashboard or pulls up report) Doing well.
    Auditor : What would happen if you weren't doing well?
    Auditee : Well, I contact...
     
    RoxaneB, Aug 27, 2015
    #9
  10. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    723
    Trophy Points:
    112
    Location:
    USA
    Right or wrong, most of my context is delivered in person instead of listing chapter and verse in an audit report. However, NCs absolutely should cite supporting evidence (or a description of lacking evidence) used to arrive at a determination of ineffective process, and I am careful to cite my evidence. CBs should be doing a better job of policing, but my observation is that CB documentation reviewers are overworked and also show variation in their expectations.
     
    Jennifer Kirley, Aug 28, 2015
    #10