ASQ CQA – 2. Audit Process Part 2

13. 2B5 Establishment of objective evidence

Now in the audit you would have done number of interviews, you would have seen number of things and you would have reviewed a number of documents. So now to meet the purpose of the audit, you need objective evidence. And what are these objective evidences? That’s what we will be learning in this topic, which is establishment of objective evidence. Let’s look at the definition of objective evidence. We have talked about this this definition earlier as well. There we talked about three definitions audit, objective evidences and audit criteria. And there we said that audit was basically comparing objective evidences and the audit criteria, what is required and what’s actually there. So objective evidences are basically what is actually available, what’s actually being done.

So now for the purpose of audit, these objective evidences would generally be records which you have seen, the statement of facts which you might have received in audit interviews and other information which is relevant to audit. All these things will be objective evidences. In the audit you can obtain objective evidences by talking to people, doing interviews, observations, taking measurements and doing some tests and looking at some documents. So that’s how you collect objective evidences. One important thing in regards to objective evidences is that objective evidences should have the information which to some degree could be verified. So as we talked earlier in data collection, that in order. You do data collection through interviews, through observations, through measurements, through document reviews.

Each of these techniques have different degree of verification. So let’s say in the interview, if you talk to some person and that person gives you some statement, that statement has less degree of verification as compared to the documents, the minutes of meeting or other records, because the minutes of meeting or other records could be reverified. So as an auditor you need to understand that what is the degree of verification for the evidences which you are collecting.

So if you have an evidence which has less degree of reliance, then it is your own judgment as an auditor to see whether you can rely on that evidence or not. So here on this slide, we have three characteristics of objective evidence. The objective evidence should be relevant, sufficient and verifiable. We have talked about verifiable earlier as well, but then the objective evidence should be relevant to the purpose of the audit and should be sufficient in itself to explain the situation.

So these are the three important characteristics of an objective evidence. Reliability of objective evidence increases if these evidences are obtained from independent source, because the independent sources will have either less bias or will be free from the bias. And this basically leads us to the concept of corroborating information. And when I say corroborating information, that means checking that information from different sources. So here we have some thumb rules related to objective evidence is how to improve the reliability of objective evidences.

The first bullet here is that documentary evidences are usually better than the verbal evidences. So if you talk to a person and if you get a statement that has less degree of reliability as compared to the documents which have more reliability, and even in case of verbal evidences, if these verbal evidences are collected from different sources, then the reliability of those evidences increases. And this is what is the concept of corroboration. Corroboration is checking the information from different sources. These different sources could be through different interviews, talking to different people. Or the information obtained through verbal interviews could be corroborated with supporting documents.

So here let’s take an example that if you talk to production supervisor, or let’s say if you talk to a production worker and you get a statement that last month we had a lot of rejections, there were a lot of quality issues. So as an auditor, would you put that information in the audit report? Probably no, because this is just a verbal statement which has less reliability. To improve the reliability of this statement, what you would want to do is corroborate this information with supporting documents.

So if someone tells you that, okay, last month we had more or too much of repairs or reworks. So what you might want to do is look at the repair log. So that verbal statement supported by the log, which you have seen, would give you a more reliable information. And in regards to reliability of objective evidences, the third bullet here is that evidence is generated through auditors direct observation. Inspection and computations are usually better than evidence is obtained indirectly, indirectly by a verbal statement from someone. So if as an auditor, if you see something directly yourself that will definitely be more reliable than any other source of information.

14. 2B6 Organization of objective evidence

Through supporting documents. This could be through direct observation by the auditor or this could be through corroboration. Now, you have collected lot of information, lot of objective evidence in the audit. Now you need to prioritize these objective evidences. To prioritize these objective evidences, you can classify them based on three things the severity, the frequency and the level of risk. ISO 9000 and 120 15 has an important concept of risk based thinking. So when you do an audit, you need to look at the risk, what is the level of risk this particular problem or this particular issue puts on the organization. So anything you do as a part of audit, you need to think in terms of risk. And that you can do by looking at each of these objective evidences based on the severity, frequency and level of risk. Here I have a matrix of the risk level.

This particular matrix is used in risk management. I’m not going into the details of this matrix. But what I want to emphasize here is that when you look at an objective evidence, look at that from the probability and impact point of view. What is the probability of this thing happening? Is it once in a while thing or this is a frequent thing? Is it just one document which was not signed? Or is it that there is a lack of practice in authorizing these documents and most of these documents are not signed off. Whether this is one particular instrument or equipment which was non critical in nature, that was not calibrated or the whole calibration system is broken, this is what will tell you what is the probability of that thing happening. And then you are looking at the impact.

What is the impact of that? The impact of a non critical instrument not being calibrated could be very little, very minor. But then if the whole system of calibration is broken, that could have a major impact on the production of the organization. So you need to look at each observation, each finding, each objective evidences from these point of views, the probability and impact. And then once you look at this matrix, this will tell you whether the risk level is high or low. So let’s say if I look at this table and look at an example where the whole calibration system was broken, so that means the probability was high. So probability high means I will be looking into this particular row and then the impact of that also is very high. So here I will be looking at this particular column and here I will end up in very high risk area. So that means if the whole calibration system is broken and the company is making equipment or items where the calibration of equipment is important, that means this is a high risk item. And then on the other hand, let’s say if there was a non critical document which was not signed off, this was a single document. So in that case, the probability of this issue is low, because this was just one document which was not signed off, and the document was also not a critical document in nature. So the impact level was also low. In that case, the risk will be low. So here, probably as an auditor, I might give a very little importance to this, and I will focus most of my work, most of my time on looking at high risk items. So this is what you need to keep in mind. You generally don’t use this matrix when looking at each objective evidence.

But as an auditor, you need to keep this concept in your mind that when you look at items, look at that objective evidence from these two points of view, the probability and the impact. Now, once you have collected objective evidences, and then you have compared those objective evidences against the audit criteria, audit criteria will tell you what was required. Objective evidence will tell you what’s happening. And if there’s a mismatch, that means there is a nonconformity, because things are not being done the way these were supposed to be done. So these are nonconformities. And these you need to highlight in your audit report. So to record this nonconformity in the audit report, you need to have supporting audit evidences that you need to put in the audit report. Let’s say this particular drawing which you saw was not approved. This particular drawing number with the revision number was the old revision number. Or this particular instrument which you checked was not calibrated, or this particular weld had more repair than acceptable level. So you need to put specific reference of the objective evidence in the nonconformity. When you do an audit, you talk to people, you look at items, and then you find out something which is not conforming to the requirement. So that becomes a nonconformity.

You should always discuss that nonconformity with the audit. Explain to the audit that this particular thing was required to be done this way, but this is not being done in this way. Once you explain that to audit, audit need to agree with that statement. Since as an auditor you are looking at some samples, you might not know the whole picture. So if you explain the nonconformity to audit, if there is anything which is missing which you did not consider, audit might explain that to you, or else auditing might agree with your finding. So before you write a nonconformity, make sure that you discuss that nonconformity with the audit number of times. nonconformities are graded based on the level of risk. This might not be much in case of internal audits, but the external audits done by ISO 9001 auditors, the certification bodies, they have nonconformity graded as the minor and major NCR, and NCR is nonconformance report. So the grading of that could be based on the risk. And as we earlier talked, risk is based on the impact and frequency.

So if you have a low criticality nonconformity which is just one case, in that case you might want to issue this as a minor MCR. But if you see a total system break then that would be considered as the major MCR. So the nonconformities could be graded based on the level of risk when you are doing an audit as a part of audit team. So here you have a number of auditors and if you find a non conformity then make sure that you discuss that nonconformity with other auditors as well. This could be part of your daily review or maybe in between whenever you get a chance, discuss those nonconformities which you observed in your area with your fellow team members, the fellow auditors, that way your fellow auditors, the other team members can also look at the same thing in their own work area. US to see whether the problem which you have observed was only limited to the area where you saw or there is a widespread problem. So make sure that you discuss the findings with your audit.

15. 2B7 Exit and closing meeting

As an audit team leader, here are few things which you need to do before you go for the closing meeting or the exit meeting. Review all the audit findings. If you have a bigger team or a team of auditors which are supporting you in the audit, you need to look at all these audit findings and see that all appropriate information required is collected. All the supporting evidence, all the objective evidences are available for these findings, what you have found in the audit. And as a team leader, you need to agree on the audit conclusion. And when you are making audit conclusion, you need to be aware that whatever audit you have done, you have done that audit based on samples. So you might have got a different set of samples, so that might lead you to a different conclusion. So before you consider the final conclusion about the audit, make sure that you have considered this aspect, that this audit was done based on samples. And also if your scope included providing recommendations, then you provide recommendations. In most of the cases, auditor is not supposed to provide recommendation. Auditor is just supposed to provide what all the findings are.

Auditor is supposed to look at these findings and come out with the suggestions how to improve the process. The recommendations generally is not part of the auditor, but if that is in the scope of Auditor, then you provide recommendations for all the observations, all the non conformities which you have found in the audit, and also make a follow up action plan. What is the follow up action plan for these audit findings? Will there be a second audit or will the auditor submit all the documentary evidences? So that also need to be thought about before you go for the closing meeting. So this is something which you need to do as a lead auditor before going for the closing meeting. Now, who all are the participants of the closing meeting or the exit meeting? These are basically the same as all the people who are in the opening meeting. So this includes audit, senior management, the quality manager of audit, which also could be part of senior management, and then all the people who are responsible for the functions or the area which you audited.

And then all the audit guides, because audit guides were witnessed to these audit interviews. So they also participate in the closing meeting and then all the audit team members. So if the auditor team had more than one auditor’s, they also participate in the closing meeting and then lead auditor chairs the closing meeting. So these are the participants of the closing meeting or the exit meeting. And then what is discussed in the closing meeting. Let’s look at that. So here is the typical agenda of the closing meeting. It starts with the audit purpose and scope.

What was the purpose and what was the scope covered in the audit? We talked about this in the opening meeting as well. But in the closing meeting also you talk about the purpose and the scope and then the lead auditor explains the concept of sampling, that whatever audit was done, this audit was done based on samples. So this is something which is explained by the lead auditor to all participants. And then the method of reporting is explained that how these findings are graded. These findings could be graded as observation, minor, nonconformity, major, nonconformity or in any other way. Whatever way was used, that is explained in the closing meeting. And then the actual audit findings are explained by all individual auditors. So each auditor explains the audit findings in their area and also the draft audit report is presented. This draft audit report could be a draft which has been prepared but might need to go through further reviews. Or this might even be a handwritten report. So this helps audit in taking action before the actual or the formal report is received by audit. Depending upon the type of audit findings, the audit findings could be closed either by a second or the follow up audit. Or these audit findings could be closed by audit submitting the documentary evidence to the lead auditor. So this is something which will depend on the type of findings which were observed in the audit. And then the lead auditor could talk about the recommendations and opportunities for improvement. This is not generally in the scope of auditor to provide the recommendations, but if the audit objective had that as a requirement for the lead auditor to provide recommendations, then the lead auditor will provide recommendations or opportunities for improvement. But in most of the cases this might not be something which the lead auditor will be doing. But if this is a part of audit objective, then lead auditor does that as well. Another important thing is reassuring about the confidentiality of information.

This is something which was talked in the opening meeting as well. But once again the lead auditor will confirm that whatever information they have seen, they have handled during the audit that will be kept confidential. Now there could be a case where everything went fine, but in the closing meeting there is a disagreement and aviation. Also I have talked that it is a good practice to discuss all the nonconformities with the audit before you leave that place. Previously also I talked that when as an auditor you observe a non conformity, it needs to be discussed with the audit at that time itself. So that audit confirms that audit agrees with that. But by any chance if in the closing meeting you have a disagreement, audit doesn’t agree with your statement in that case, either that disagreement needs to be resolved by discussion or if this doesn’t get resolved by discussion, then this needs to be recorded in the audit report. So this is a typical agenda of a closing meeting.

Now looking at some of the considerations which you need to keep in mind as an auditor or as a lead auditor, that there should not be any surprise to auditing in the closing meeting. Whatever observations, whatever nonconformities, as an auditor or lead auditor you found that should have been discussed with the audit at that time itself, where you found those issues, there should not be anything new in the closing meeting. All those things at least should have been talked to those individuals, individuals responsible for that area.

And then, as a team leader, don’t forget to thank the audit for the support which they have provided. They would have given you a place to sit. They would have given you all the support you needed. Then make sure that you thank them for all the support which they have provided during the audit. During the audit, you would have seen some best practices being followed. You would have seen something positive, something which was above and beyond what you expected or what the system expected. Make sure that you highlight those as well in your audit report. And then once again, you need to keep a record of meeting participants as you did in the opening meeting, in the closing meeting, or the exit meeting. Also, make sure that you keep a record of all participants that will help you in your audit report.