1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice
You must be a registered member in order to post messages and view/download attached files in this forum.
Click here to register.

Evaluation of Metrics during Internal Audit?

Discussion in 'IATF 16949:2016 - Automotive Quality Systems' started by Golfman25, Jan 12, 2016.

  1. Golfman25

    Golfman25 Well-Known Member

    Joined:
    Nov 6, 2015
    Messages:
    816
    Likes Received:
    402
    Trophy Points:
    62
    Simple question. Is it a non-conformance if an internal auditor does not review the process metrics during their audit?
     
  2. Qualmx

    Qualmx Well-Known Member

    Joined:
    Oct 7, 2015
    Messages:
    464
    Likes Received:
    59
    Trophy Points:
    27
    Location:
    Mexico
    the nc apply to the audited process not to the auditor.
    IMHO, No, only if in the nc is found and it was included in the scope.
    If you are auditing the auditing process and the auditor didnt audit metrics which was included in the scope, yes it could be a NC. But If the auditor is auditing one process ,e.g. sales, the metrics will be audited if it is included in the scope, then the nc is declared by the auditor, if metric was not met,but if he missed or forgot to audit the metrics,the only way to detect this failure, it is when the auditing process is revised.
    Hope this helps.
     
  3. Golfman25

    Golfman25 Well-Known Member

    Joined:
    Nov 6, 2015
    Messages:
    816
    Likes Received:
    402
    Trophy Points:
    62
    Not sure. But here is an example. Internal audit of sales. Sales keeps a metric such as hit rate - orders/quotes. Auditor reviews the sales process for conformance to all procedures, standards, etc. but doesn't include the hit rate in his review (because the metric is regularly reviewed during management meetings of which this auditor is a part of). Is this a NC against internal audits?
     
  4. Vthouta

    Vthouta Member

    Joined:
    Dec 20, 2015
    Messages:
    10
    Likes Received:
    2
    Trophy Points:
    2
    This is a tricky question. For example, if the Sales procedure documents/states that the metrics are reviewed at planned intervals, then internal auditor would need to verify the metrics are correct as per the documented procedure. As part of the 8.2.2 internal audit clause, the internal auditor is reviewing the requirements of the QMS as per the documented procedure. If they have not been reviewed, this would be a NC against internal audits in this case.
     
  5. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    As with many answers about what's "right" or "wrong" about audits is "What's the scope & criteria of the internal audit". Maybe the reason for the internal audit WASN'T to include the metrics. It could be that the internal audit was to evaluate a change in the process which was implemented and wouldn't affect those metrics.

    What was the objective of the audit.

    I'd ask however, why, just because the auditor "knows" a metric is reviewed elsewhere, if that auditor is a) objective and b) impartial. Overlooking something like this indicates, to me, that the auditor didn't plan, was "OK" with not verifying and, hence, didn't do a good job of their assignment...but it could be that whomever set up the audit also was delinquent in not setting out a clear objective, making certain the auditor DID include that and reviewed the results on conclusion of the audit.

    You may also wish to take a long, hard look at the way you go about internal audits (not worry about if something so small is an NC or not)
     
  6. Casana14

    Casana14 Member

    Joined:
    Jan 25, 2016
    Messages:
    10
    Likes Received:
    3
    Trophy Points:
    2
    Location:
    USA
    We've just added a question regarding metrics to our internal audit procedure. So the internal auditors now need to ask about metrics - if it doesn't apply, they make a note of this in their report. That way it is clear that we checked during internal audit.
     
  7. Golfman25

    Golfman25 Well-Known Member

    Joined:
    Nov 6, 2015
    Messages:
    816
    Likes Received:
    402
    Trophy Points:
    62
    That is basically what we ended up doing. Even though they did look at metrics during the audit, it was "buried" in their report so the 3rd party auditor didn't see it. Sometimes you need to make things super obvious.
     
    Casana14 likes this.
  8. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    It would be much more effective to put it in the planning activity (not in "procedure") May I suggest that internal auditors need to understand when they are planning their audit (call it drawing up a checklist, if you wish) that they do some research on the process, what it has for objectives and measurements and so on. It's not a case of asking a question and if you put it in the procedure, it should go along with a bunch of other things the auditor should be considering before they go ask anyone anything (we know people don't read procedures!)
     
  9. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    If that's the case, did your own management find it? Why do things for a CB auditor? Did you ask your management if they could find the details too?
     
  10. Candi1024

    Candi1024 Well-Known Member

    Joined:
    Jul 30, 2015
    Messages:
    129
    Likes Received:
    83
    Trophy Points:
    27
    Location:
    Pennsylvania
    Couldn't you just have shown the auditor where it was in the report, and you would no longer have a non-conformance?
     
  11. Casana14

    Casana14 Member

    Joined:
    Jan 25, 2016
    Messages:
    10
    Likes Received:
    3
    Trophy Points:
    2
    Location:
    USA
    Andy Nichols, we DO have a checklist of sorts for our internal audits - more of a tickler to make sure internal auditors checked all required areas, plus a place to record auditee names, what documents were examined, etc.
     
  12. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Good! Does this checklist support the "process approach"? If it does, how did they miss metrics? If the checklist is based on ISO 9001 or something, then it may need fixing...
     
  13. MCW8888

    MCW8888 Well-Known Member

    Joined:
    Aug 17, 2015
    Messages:
    642
    Likes Received:
    198
    Trophy Points:
    42
    If I evaluate the Internal Audit procedure or Internal Audit Plan, it would be either a NC or OFI. From my standpoint it would be the NC against the requirements of 8.2.2b or in clause 8.2.2.2 of TS16949. Review of process metrics is a way of determining the effectiveness of the process. How else can you improve the process if you do not measure it.
     
  14. Golfman25

    Golfman25 Well-Known Member

    Joined:
    Nov 6, 2015
    Messages:
    816
    Likes Received:
    402
    Trophy Points:
    62
    We are a small company. Since I am management and I create/review all of the metrics, I know the answer before the question is even asked. To me it is a question of where the metric is picked up -- internal audit or management review (or somewhere else). There is little to zero value handing me an audit report which shows me a metric which I already know because I calculated the thing.
     
  15. Golfman25

    Golfman25 Well-Known Member

    Joined:
    Nov 6, 2015
    Messages:
    816
    Likes Received:
    402
    Trophy Points:
    62
    Yes, but I didn't find it until later in the process. It was "buried" in the auditors commentary. Given our company, it was a useless finding.
     
  16. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Auditors shouldn't just be writing the metric in their report! As you say, there's no point! However, if they are to be evaluating the process for effective implementation, they MAY WELL report on those things which contribute to the achievement of objectives etc. What's the objective of these audits? What's the scope? Are these audits for the sake of "ISO" or are they to confirm the processes deliver, under controlled conditions etc..?

    So once again, I have to ask, what are the auditors actually doing? If they are simply recording something you already know I'm left wondering how effective your audits are and whether they're simply "copying" what the CB should do...
     
  17. Golfman25

    Golfman25 Well-Known Member

    Joined:
    Nov 6, 2015
    Messages:
    816
    Likes Received:
    402
    Trophy Points:
    62
    In our case, it is measured. But we are small. I calculate/review the metrics. Those numbers are shared with my staff, who are also my internal auditors. They just didn't "clearly" indicate that they included the metric in their audit. Probably one of the least substantive finding we have had in 15 years.
     
  18. MCW8888

    MCW8888 Well-Known Member

    Joined:
    Aug 17, 2015
    Messages:
    642
    Likes Received:
    198
    Trophy Points:
    42
    If you do not measure you can't improve.