Clinical Evaluation Masterclass: Appraisal of literature sources has not been conducted properly - Episode 2

Transcript follows:

It’s Paul from Mantra Systems here, and welcome to episode two of our series on how to fix clinical evaluation non-conformances.

If you didn’t see the first part in the series, I’ll put a link to that post in the description below this video.

But otherwise, let’s get on with episode two.

---

Non-conformances following submission of a clinical evaluation are quite common under the EU MDR, basically because the standard for submissions is now so high.

But the problem is that:

• Resolution of non-conformances is necessary for the clinical evaluation to be accepted.
• Nonconformities can be quite difficult to interpret.
• It may not always be clear how to fully address them.
• A failure to resolve non-conformances may lead to:
  • A further round of review,
  • Increased costs,
  • Lost time,
  • Stress and worry.

The aim of this series is to work through commonly seen nonconformities one at a time, and dig deep into what they mean and how to solve—or preferably avoid—them in the first place.

The context behind this is to link into general principles for optimal conduct of medical device clinical evaluation.

Nonconformity Number Two. The one we’re going to look at is this:

‘Appraisal of literature articles has not been conducted appropriately.’

This is one we see a lot. To explore it—and ensure that you don’t fall foul of it—we’ve broken our approach down into three steps:

1. Consider what appraisal actually means and what purpose it serves within the context of a clinical evaluation.
2. Look at the characteristics of an appropriate appraisal framework.
3. Consider how appraisal should be applied during clinical evaluation.

What Is Appraisal? Appraisal is:

• A structured and systematic evaluation of:
  • The quality of individual studies within a literature review or a clinical evaluation, and
  • The relevance of each source to the clinical evaluation.

If applied correctly, appraisal enables systematic weighting of study outcomes according to quality and relevance.

The basic idea:

The better the study, and the more relevant it is, the greater weighting is given to its results compared with lower-quality studies of lower relevance.

The MEDDEV guideline 2.7/1 revision 4—although somewhat outdated—places appraisal effectively into context.

• Clinical evaluation is split into four stages: 0, 1, 2, and 3.
• Appraisal is stage 2, which:
  • Comes after identification of pertinent data,
  • Leads into appropriate analysis,
  • Emphasizes that you can’t properly analyze data until you’ve appraised it.

Designing an Appraisal Framework. There is no universally defined appraisal system. Some systems are broadly accepted, such as the Oxford scheme, but:

• These can be limited. For example, the Oxford scheme only considers study type, overlooking other relevant factors.
• Therefore, justification of your chosen method is key.

Appraisal should:

• Consider a range of factors related to study quality and relevance.
• Avoid being based on just one or two elements.

The most common pitfall: doing nothing with the appraisal.

The objective is to produce an appraisal score for each source that actively influences outcomes.

This score becomes a mathematical weighting factor applied to results from individual sources.

Characteristics of a Compliant Appraisal Framework. There’s no universally accepted method, but an appraisal scheme may consider:

• Study type
  • RCTs typically score higher than cohort studies or case series.
• Sample size
  • Larger sample sizes increase reliability and therefore the score.
• Use of statistics
  • Includes whether the study shows statistical significance.
  • Well-designed statistical analysis increases score.
• Length of follow-up
  • Longer follow-up captures more of the clinical journey.
• Devices used within studies
  • Subject device: highest score
  • Equivalent device: slightly lower
  • Similar device: lower still
  • Unrelated device: lowest score

The key is justification—explain to the reviewer why your approach is appropriate.

Example Scoring Framework. In your appraisal protocol, you might use a system like this:

• Characteristics listed on the left
• Descriptions and alpha-numeric grading on the right
• Higher numbers = better scores

Example:

• A randomized control trial might be scored R4.
• A case series might be scored R1.

You can build a table with these scores and apply it to each study. This allows a study to be:

• Strong in one area,
• Weak in another.

This combined appraisal system accounts for nuance.

To produce the overall appraisal score, add the numbers in each row.

Example (hypothetical data):

For ‘Jones et al’:
R3 (3) + 2 + 1 + 3 + 3 = 12

Applying the Appraisal. Once you have appraisal scores:

1. List the studies and their appraisal scores.
2. Add an outcome parameter (e.g., reduction in pain VAS).

To calculate the weighted mean:

• Multiply the outcome value by the appraisal score.
• Example for Jones et al:
  • Outcome: 4.5
  • Appraisal score: 12
  • Weighted value: 4.5 × 12 = 54

Repeat for each source.

Then:

• Sum the appraisal scores: 61
• Sum the weighted values: 226.7
• Divide the total weighted values by the total appraisal scores:

Weighted mean = 226.7 / 61 = 3.72

This gives greater weight to better quality studies, which is the essence of how appraisal is expressed in a compliant clinical evaluation.

Summary of an Effective appraisal:

• Uses systematic methods to evaluate quality and relevance.
• Considers a range of factors.
• Produces a score to enable mathematical weighting of results.
• Must be:
  • Justified,
  • Conducted according to a clear, advance plan.
• Applies to literature for both:
  • Subject devices,
  • State-of-the-art comparisons.

Recap of Episode Two: Nonconformity #2: Appraisal of literature articles has not been conducted appropriately

In this episode, we’ve:

• Looked at what appraisal is and what purpose it serves.
• Considered characteristics of an appropriately conducted appraisal process.
• Looked at how appraisal should be applied during clinical evaluation and the effect it has on outcomes.

---

Don’t forget that Mantra Systems offers a comprehensive clinical evaluation service. One of the most common reasons we’re engaged is to help with resolution of non-conformities.

• All initial conversations are completely free.
• We have a really friendly team.
• Give us a call or send us an email if you’d like to discuss anything.

That concludes episode two in this non-conformity series.

Please share if you found this video useful, and I look forward to seeing you in the next part of the series.

Thank you very much!