1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice
You must be a registered member in order to post messages and view/download attached files in this forum.
Click here to register.

What evaluation system is needed?

Discussion in 'ISO 9001:2015 - Quality Management Systems' started by Leonid, Mar 5, 2016.

  1. Leonid

    Leonid Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    164
    Likes Received:
    31
    Trophy Points:
    27
    Location:
    Moscow
    Cl. 9.1.3 enlists key QMS characteristics subject to analysis and evaluation. The results shall be retained as documented information. What evaluation system is needed?
     
  2. Tony Wardle

    Tony Wardle Member

    Joined:
    Dec 3, 2015
    Messages:
    18
    Likes Received:
    15
    Trophy Points:
    2
    As I understand this - from my company perspective ....

    In production, there are certain key attributes that should be measured to ensure conformity of a part (which may be a small part of a larger item or system).
    There should be some sort of SPC performed on these - measures of central tendency - measures of spread, measures of NC parts per million maybe.

    The results of these analyses will speak to the points following (a. - g.) - the evaluation being in terms of ppk or cpk - ppm or % on spec.

    I would see the collection and SPC being retained as ''raw data''.

    I also believe that customer in this clause may well be referring to internal customer - which will ultimately translate to external or paying customer.
     
  3. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    I don't understand what you are looking for, Leonid, as the answer here is dependent upon the characteristic being analyzed as well as what will add the most value to the organization. Tony's response is very production-/manufacturing-focused (which is not a bad thing). However, if you are referring specifically to the QMS requirements themselves, again, how the methodologies and processes track, record, and analyse information may vary between organizations and I doubt that there is one right answer.
     
    Andy Nichols likes this.
  4. Leonid

    Leonid Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    164
    Likes Received:
    31
    Trophy Points:
    27
    Location:
    Moscow
    Tony, you propose to consider evaluation as determination of value. This falls under one of definitions of evaluation in dictionaries. However, there is another definition: judgment on or determination of the significance, worth, or condition. I thought evaluation in ISO language is the organization’s judgment about QMS key characteristics like: over plan or as planned or under plan; achieved or not achieved; acceptable or not; high, very good, good, poor, or low; etc. What do you think about this kind od evaluation?
     
  5. Leonid

    Leonid Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    164
    Likes Received:
    31
    Trophy Points:
    27
    Location:
    Moscow
    Roxane, I answered to your question in my response to Tony. In 2015 verision, all key QMS characteristics are to be evaluated with results recorded. This is the new requirement and I wanted to know how readily it is treated.
     
  6. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    I stand by my answer then. It depends on the organization and how it gains value from the data is collects, records, analyses.

    Using internal audits as an example, there is a plethora of data that can be collected and presented.
    • # of audits conducted
    • # of audits conducted / # of audits planned
    • # of audits conducted / # of audits originally scheduled
    • # of findings
    • # of findings / clause
    • # of findings / process
    • # of findings / clause / process
    • # of repeat findings
    • # of auditor reviews completed
    • # of auditor reviews = Good or Higher / # of auditor reviews completed
    • ...and so on ...

    To say that there is just one way (or a limited number of ways) in which to handle the results of the QMS characteristics is too simplistic. Organizations should determine :
    • If they are collecting the data that they need to conduct information-based decision making;
    • What data "stories" will add the most value to their organization and processes - including frequency, method of reporting, target/goal determination, etc.; and,
    • Methods for reviewing the data "stories" in such a way to understand the health of the organization and its processes,
     
    Bev D likes this.
  7. Leonid

    Leonid Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    164
    Likes Received:
    31
    Trophy Points:
    27
    Location:
    Moscow
    Roxane, I still want to achieve the very end of the long process of collecting and analising the plethora of data. Eventually the the organization shall evaluate, say, conformity of products and services. What stands behind the term evaluate? What can be output of the evaluation? Just a few examples...
     
  8. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    Why? What are you trying to achieve from this particular discussion, Leonid?

    The ultimate evaluation is, I suppose, Client Experience or Satisfaction. But even that can be measured in numerous ways. For airlines, it could be repeat customers. I'm in healthcare...with all due respect to our patients, we'd rather not see them again, but if their health fails, we'd rather they contact us than a competitor. So even with that particular metric, evaluation methodologies will vary depending on which data adds the most value to the organization.

    ISO 9001 tells us in a very generic sense what to do. However, it is up to organizations to decide what processes and activities add value, work best for them and finally conform to the standard's requirements. If there was a "one size fits all" solution, then ISO 9001 would be a lot more specific. It would not say "The organization shall evaluate conformity of products and services"...it would tell us exactly how to do it. It does not do this as there is no one set way to do this activity. And because there is more than one way to do address the shall, there is bound to be more than one type of data to collect. And because there is more than one type of data to collect, there is more than one way to analyze/evaluate it.
     
    Andrej likes this.
  9. Leonid

    Leonid Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    164
    Likes Received:
    31
    Trophy Points:
    27
    Location:
    Moscow
    With reference to the definition below I wanted to ascertain if “to evaluate” in 9.1.3 means rather “to judge” than “to calculate”. Calculation is close to analysis which precedes the evaluation.
    To evaluate = to judge or calculate the quality, importance, amount, or value of something (Cambridge)
     
  10. RoxaneB

    RoxaneB Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    926
    Likes Received:
    1,081
    Trophy Points:
    92
    Location:
    Ontario, Canada
    I think you're getting too caught up the specifics and ignoring the intent of the requirement.

    Just as various organizations take a different approaches to implementing the requirements of the Standard, even dictionaries do not have the exact same way of defining "evaluate". In fact, 2 of the 3 I pulled which were in the top search engine returns include the use of "calculate" and "assess".

    Courtesy of www.dictionary.com -
    verb (used with object), evaluated, evaluating.
    1.
    to determine or set the value or amount of; appraise:
    to evaluate property.
    2.
    to judge or determine the significance, worth, or quality of; assess:
    to evaluate the results of an experiment.
    3.
    Mathematics. to determine or calculate the numerical value of (a formula, function, relation, etc.).

    Courtesy of www.thefreedictionary.com -
    e·val·u·ate
    (ĭ-văl′yo͞o-āt′)
    tr.v. e·val·u·at·ed, e·val·u·at·ing, e·val·u·ates
    1. To ascertain or fix the value or amount of: evaluate the damage from the flood.
    2. To determine the importance, effectiveness, or worth of; assess: evaluate teacher performance. See Synonyms at estimate.
    3. Mathematics To calculate the numerical value of; express numerically.

    Courtesy of www.merriam-webster.com
    evaluate
    eval·u·at·edeval·u·at·ing
    transitive verb
    1. 1 : to determine or fix the value of
    2. 2 : to determine the significance, worth, or condition of usually by careful appraisal and study
     
  11. Jennifer Kirley

    Jennifer Kirley Moderator Staff Member

    Joined:
    Jul 31, 2015
    Messages:
    1,071
    Likes Received:
    722
    Trophy Points:
    112
    Location:
    USA
    From 0.3.1 of ISO 9001:2015: "...The monitoring and measuring check points, which are necessary for control, are specific to each process
    and will vary depending on the related risks." (see figure 1)

    It follows that methods for monitoring and measuring will also be specific to each process. Some may be qualitative, others quantitative; some will be counting events, others will be measuring attributes, still others will be assessing based on physical observation or discussions with people. No one single method is called for, indeed it would be foolish to try to apply a single evaluation system.

    If I could offer one idea of one method, it would be the balanced scorecard approach. What that looks like depends on the needs of the organization and its stakeholders.
     
  12. Leonid

    Leonid Well-Known Member

    Joined:
    Jan 4, 2016
    Messages:
    164
    Likes Received:
    31
    Trophy Points:
    27
    Location:
    Moscow
    Colleagues, you helped me to better get the discussed 9.1.3 requirement and to read it as follows: the organization shall … determine/judge/set/calculate/ascertain/fix the value/amount/significance/importance/worth/quality/condition of QMS key characteristics.
     
  13. Jamie Lill

    Jamie Lill Member

    Joined:
    Mar 17, 2016
    Messages:
    35
    Likes Received:
    1
    Trophy Points:
    7
    Your answers are too complex.
    look at this from a one man operation.

    Did I make a good part/product on time. Great good job
    No start again
     
  14. Andy Nichols

    Andy Nichols Moderator Staff Member

    Joined:
    Jul 30, 2015
    Messages:
    5,086
    Likes Received:
    2,553
    Trophy Points:
    112
    Location:
    In the "Rust Belt"
    That's not the ONLY option! If a part isn't "good", there are other options than making it again. Indeed, it may not the the option which can be taken. A rework or repair or even acceptance under a waiver may be more appropriate. We have to understand more than if the organization is one person. It's called the "context of the organization".
     
  15. MarkMeer

    MarkMeer Well-Known Member

    Joined:
    Dec 3, 2015
    Messages:
    138
    Likes Received:
    62
    Trophy Points:
    27
    Not sure if you're just being facetious, but this is actually pretty accurate. The trouble is that terms like "analysis" and "evaluation" seem to conjure this idea of hard, quantifiable data that's crunched using statistical formulae.

    Clearly this would not be applicable or would be valueless to certain companies, depending on the nature of their activities. For many, evaluations may be a ranking (e.g. 1-to-5), a status (e.g excellent, good, bad, terrible), or even a comparison to some other datum (e.g. has sales gone up from previous year?)...

    As RoxaneB has said, it's ultimately up to you to define appropriate "evaluation" systems to fit your processes. It should be of value to you (the "evaluation" results should be meaningful or actionable), but not unnecessarily burdening system resources (massive data collection and statistical analysis can be pretty ambitious - make sure it's of value before you go this route!).

    Unsatisfying answer, I know. ...but without knowing all the details of your organization, no one can really suggest what evaluations are appropriate for you...