1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice
You must be a registered member in order to post messages and view/download attached files in this forum.
Click here to register.

Is taking a "sample" appropriate when doing Internal Audits?

Discussion in 'ISO 19011 - Auditing Management Systems Guidelines' started by Andy Nichols, Jul 28, 2021.

  1. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    CB auditors often caveat their audits by explaining that they sample activities. Many customer auditors likely do similar when performing pre-award audits. When performing Internal (Quality) Audits, is it always applicable to "sample"? By that I mean, should the auditor ALWAYS not take into consideration that there may be a point in time, a specific activity, a particular process/characteristic, document, etc which should fall within the scope of the audit, to the possible exclusion of others?
     
  2. bkirch

    bkirch Active Member

    Joined:
    Jun 24, 2016
    Messages:
    73
    Likes Received:
    13
    Trophy Points:
    7
    I am not sure that audits would ever get completed without sampling? There just isn't enough time to audit everything. If you have a breakdown in a process, it should show up pretty quick with sampling. However, with sampling, there is always the risk of missing issues that really need to be corrected. There is also the question of what your sample size should be for your sampling to be adequate.
     
  3. John C. Abnet

    John C. Abnet Well-Known Member

    Joined:
    May 23, 2017
    Messages:
    709
    Likes Received:
    510
    Trophy Points:
    92
    Location:
    Upper Midwest- USA
    Good day @Andy Nichols ;
    If I understand your question correctly then I would agree with @bkirch . Is this what you're asking ? (or are you digging for something more?)

    Be well.
     
  4. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    When performing both 2nd and 3rd party audits, there are specific objectives which require that the auditor takes samples. They are, after all, there to be impartial about the supplier's QMS. There's rarely any "diagnostic phase" before the audit is performed (other than, in the case of the CB to ensure the auditor is "somewhat" capable of doing the audit - tho' experience shows this is "hit or miss").

    An Internal Audit is, by the very nature of the organizational relationship to the auditor, capable of being far more focused in determining the role of the QMS in contributing (or not) to a "quality event". There must be a "diagnostic phase" to inform the audit objective, scope and criteria to be used. Since this can be a very specific scope, the various documents related to that scope - including records which would have been produced - can lead to 100% evaluation, not a sample.

    We must also not overlook the fact that external audits are constrained by time (see IAF's MD-5 for example). Internal audits, if correctly supported by management, have no such artificial constraints.
     
    Last edited: Jul 28, 2021
  5. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    Define 'correctly supported,' please.

    Auditors should decide on how large the sample size should be. It would be dangerous to review only one record of a process and base the level of conformance on that one record. If "good", it's a flawed approach leading an auditor to presume the process conforms to all requirements, all of the time. If the record is "bad", it is similarly dangerous to review all records to designate the finding as minor or major (if these designations are in place within the organization's internal audit process). The review of a few more records can lead to audit results that indicate if the noncoformance is 'one-of' or 'systemic.'

    A blanket '100% evaluation' approach is likely not realistic in this day and age. I'd offer that internal auditors are usually volunteers (or voluntold) from various departments - i.e., they have a regular "day job." Add in that the amount of records can grossly fluctuate between organizations. For some processes, such as management review, perhaps the '100% evaluation' approach is appropriate. However, for some processes where there can be thousands of records (e.g., records of competency), the '100% evaluation' approach is not feasible and questionable in terms of how it adds value to the overall audit results.

    One of the internal audit fundamentals we've stressed multiple times is that internal audits are not witch hunts. 100% evaluation of records essentially sounds like we are trying to catch the auditee at not being perfect...sounds a bit like a witch hunt to me. Unless, of course, there is some form of criteria that says achieving 95% perfection is "okay" or maybe 90% - I would be interested to know who determines what the acceptable pass percentage is.

    Part of planning for internal audits is to schedule them based on the level of risk/impact of the process to the organization, including, I thought, the results of past audits. I'm not sure I see the value in 100% evaluation of records in a process with low risk/impact or in one that has historically demonstrated an effective and efficient adherence to requirements - barring, of course, any significant changes to that process since the last audit.
     
  6. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Management understand the purpose of both the QMS and the place of internal audits, participate in determining the "guard rails" of an audit (objective, scope, criteria), provide/select auditor candidates with suitable competencies and provide the necessary resources to allow the auditors to perform the audits.
     
  7. John C. Abnet

    John C. Abnet Well-Known Member

    Joined:
    May 23, 2017
    Messages:
    709
    Likes Received:
    510
    Trophy Points:
    92
    Location:
    Upper Midwest- USA
    I think we're on the proverbial "same page" @Andy Nichols , but in my experiences I don't believe I perceive as much difference between internal and 3rd party audits as I infer that you do ( from your current or past posts)....

    - Scope and plan
    Both assumed international standards (9001 and IATF) and rules 5th edition require.
    (When I train internal auditing, I discuss obtaining known "control documents" from the process in advance, and reviewing them to understand the intent of the process. However, I realize that this step is not always taught nor always performed)

    - effectiveness of the quality management system
    While 9.2 internal auditing in the assumed standards (ISO 9001 and IATF 16949) does not directly mention "effectiveness of the quality management system", 9.2 does state via similar verbiage (same intent), e.g. "..is effectively implemented and maintained".

    - International standard
    Both 3rd party and internal (9.2.1) require that the internal audits confirm compliance with the international standard

    - Organization's internal requirements
    Though I can not find the actual verbiage, (???...let me know if I'm overlooking something please) in the CB rules 5th edition, 3rd party audits must also confirm compliance with the organization's own requirements since that is part of the "international standard".


    *
    I understand the "lack of time constraints (in theory) for internal audits, and agree that this can have an impact on the detail of an audit. However, in reality (as @RoxaneB stated),
    most "internal auditors" are already on a hamster wheel of activity, meaning (even though it isn't right) leadership generally limits the time spent on audits and the availability of
    auditees.

    * "...by the very nature of the organizational relationship to the auditor..."
    Obviously the exception to this is when an organization contracts out their internal audits to a second party such as myself.

    Overview-
    The job of an internal auditor (per ISO 19011) is to "provide information". Herein lies the biggest difference that I see in regards to the INTENT of an internal audit versus 3rd party approach.
    An internal auditor often assumes or is assigned responsibility for confronting any nonconformances and holding the individuals accountable. Unless those roles are assigned by leadership (which I do NOT recommend), then internal auditors are improperly taking on the role of 3rd party auditor. If leadership is not providing them with the training and mentoring, (usually the case) then it is not the internal auditor's fault, as these roles are often erroneously implied and assumed.
    This simply causes conflict, defensiveness, frustration, and, is frankly a cop out by leadership who SHOULD be taking the "provided information" and weighing risks and assigning applicable responsibilities (with resources) to process owners in order to support a "fix".

    Summary-
    Except what I mention in the "Overview" above, , if intent and, in some cases, the "rules" of internal auditing and 3rd party auditing are properly taught, mentored, and applied, then I don't see that much difference in the overall goal, and, therefore, approach of internal auditing vs 3rd party auditing.


    What say all of you? (good discussion....

    Be well.
     
  8. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    If we set aside, for a moment, all that we know and experience with 3rd Party Audits and possibly supplier audits (for those with experience), if you simply took the 9.2 requirements and a clean sheet of paper to create an audit program, there are minimal restrictions - and nothing to model on - the way audits should be carried out. It is these external influences which cause most of the divisive posts, when discussing internal audits, here, on LinkedIn and at the Cove (I've withdrawn from posting there a while ago, for this very reason).

    Let's start with why audits are done and compliance to the standard less important when the "spectre" of attracting a non-conformity (of any grade) is of zero importance. Internal audits are there to (independently) confirm the QMS is both being maintained and effective. If it isn't, it is automatically in non-compliance with the standard, of course. In other words, auditors don't have to "lead" with ISO type questions.

    Having established the QMS and the audit program, where would any management team assign an auditor to do an audit? We have the various objectives of the QMS and a number of indicators of performance of the various processes. So, in my mind, it simply becomes a case of management "tapping the gauges on the dashboard" and telling an auditor to "go look at what happened"... Hence, the audit objective, scope and criteria (required by ISO) can be whatever management wish them to be to diagnose the situation...

    And therefore, the idea of "sampling" becomes unnecessary, because the focus is sufficiently defined that the population of available docs is much smaller than if it were any other kind of audit.
     
  9. John C. Abnet

    John C. Abnet Well-Known Member

    Joined:
    May 23, 2017
    Messages:
    709
    Likes Received:
    510
    Trophy Points:
    92
    Location:
    Upper Midwest- USA
    and...
    1) the organization’s own requirements for its quality management system;

    2) the requirements of this International Standard;

    So while I agree that internal auditors should avoid simply "...ISO type questions" (i.e. language unfamiliar to the process/teams), ....the requirements of the international standard still apply to those internal audits.



    Without sounding like I'm "playing dumb", can you define what you mean by "sampling" in this context? (I think that will help me identify the root of why I am missing your question) @Andy Nichols


    I'm not sure I'm still understanding your line of questioning @Andy Nichols ? What am I missing?
     
  10. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Sampling in conventional (3rd party audit) wisdom terms, involves auditing a sample of the types of processes and activities - ranging from contracts, raw materials from suppliers, products, departments and so on - and following the natural course of the product/information flows. The auditors typically ask for information regarding contracts for "X" or "Y" products, suppliers of "A" products, or "B" subcontract processing, manufacturing lines "C", "D" and "E" - which represents the scope of the QMS. This is sampling. It has to be "representative" and some would suggest "statistically based" (good luck with that, CB auditor!).

    The IAF's MD-5 time allocation chart for CB audits allows for little/no consideration of off-site time to study in detail the QMS being audited and clients won't pay. We all know, however, 80% of any task is planning. ISO 9001 type audits rarely even consider performance of the QMS - unless a customer complaint has been lodged between CB clients.

    Internal audits are not - or at least should not - be constrained by the same issues.
     
    Last edited: Jul 29, 2021
  11. John C. Abnet

    John C. Abnet Well-Known Member

    Joined:
    May 23, 2017
    Messages:
    709
    Likes Received:
    510
    Trophy Points:
    92
    Location:
    Upper Midwest- USA
    Understood @Andy Nichols
    Thanks for these thought provoking prompts.

    Be well.
     
  12. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    It should ALWAYS be a consideration of the auditor. But verifying ALL activities, transactions, process characteristics, outcomes, documents would be deemed counter productive. If I have to look it on another way, thus instead of asking "Is audit by sampling appropriate?", I would ask "Is auditing ALL appropriate?". I believe, the disadvantages of doing "audit all" outweighs audit by sampling for reasons like:
    • Auditing all will significantly disturb the normal operations;
    • Auditing all will cause fatigue to auditors and auditees;
    • Auditing all is resource, cost, time, and attention demanding;
    • If auditors will check conformity of all activities, transactions, outcomes, process characteristics against the requirements, then they are doing the job that the auditees/process owners are mandated to do;
    • Process owners/auditees, including the top management, might rely on the auditors in ensuring conformity, and occurrence of NCs might be attributed to the failure of the auditors - not the process owners/auditees;
    • If, in actual, majority of the activities, transactions, outcomes, process characteristics are nonconforming, the auditors don't need to check and sort every one of them that occurred within the scope and period covered just to prove a point that the process owners/auditees need to do something to improve their processes.
     
  13. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    Perhaps, at the root of this discussion is our understanding of the base purpose of the use of audits. We often deal with such things as audits - because they are "required" by ISO 9001, for example - and we deal with them in isolation of the other related requirements, which are often treated in the same manner - in isolation. If we consider the results of audits (I'm thinking only internal audits here, because ISO 9001 is agnostic to external audits, since they aren't necessary) what is the purpose of audit results? When we understand that, we can be better informed about how the audit program is to be managed and, hence the tools and techniques, including planning for those (internal) audits. A question becomes, what do (should) we do with the results of audits?
     
  14. tony s

    tony s Well-Known Member

    Joined:
    Sep 10, 2015
    Messages:
    1,350
    Likes Received:
    1,054
    Trophy Points:
    112
    Location:
    Laguna Philippines
    It will depend on the nature of the results. Internal audits, for me, just like the other forms of evaluation such as product inspections, employee appraisals, medical examinations, etc., should not be programmed without taking into consideration of their results. Depending on the results, good or bad, the organization can decide on when to loosen or tighten the reins.