Meaningful Metrics
Industry mantra dictates that if it can’t be measured, then it can’t be managed, and it certainly can’t be improved; an enduring message for a young scientist, pharmacist or engineer starting out on their career in the pharmaceutical/biopharmaceutical sector. Visit any leading manufacturing site in this sector and you will be bombarded with scoreboards of key performance indicators (KPIs) and traffic light style visuals that indicate if an individual KPI is on track. The measurement of Quality, and the performance of the Pharmaceutical Quality System (PQS) is no exception to this treatment. Quality metrics are used throughout the drugs and biologics industry to monitor quality control systems and processes and drive improvement in manufacturing. The regulators too, and in particular the US FDA, instigated metrics to help develop compliance and inspection policies and practices, such as risk-based inspection scheduling of drug manufacturers. The idea being that the regulator would encourage the pharmaceutical industry to continuously improve, strive for state-of-the-art quality management systems and mitigate against drug shortage risks.
The FDA issued a draft guidance regarding the collection of quality metrics in 2015 and this was followed up with a revised draft in 2016 (link below) and a voluntary programme of measurement.
Link to FDA Draft Guidance on Submission of Quality Metrics Data
The FDA believes that the quality metrics defined below help provide important information about operational reliability:
Lot Acceptance Rate (LAR) = the number of accepted lots in a timeframe divided by the number of lots started by the same covered establishment in the current reporting timeframe. This metric is an indicator of manufacturing process performance.
Product Quality Complaint Rate (PQCR) = the number of product quality complaints received for the product divided by the total number of dosage units distributed in the current reporting timeframe. This metric is an indicator of patient or customer feedback.
Invalidated Out-of-Specification Rate (IOOSR) = the number of OOS test results for lot release and long-term stability testing invalidated by the covered establishment due to an aberration of the measurement process divided by the total number of lot release and long-term stability OOS test results in the current reporting timeframe. This metric is an indicator of the operation of a laboratory.
The FDA encourages metric reporting to be calculated by the site of manufacture and testing. Before any site initiates the recording of the metrics above, there should be an understanding of the detail behind the metrics in order to use them for comparison of sites, operations or companies. In short, all companies must be calculating the metrics the same way for the comparisons to be meaningful. For example, the understanding of the difference between in-process and packaging lots as against saleable lots is critical to a correct and consistent measure of LAR. The explanation of each parameter is best explored through the numerous examples provided in the link provided above.
For those interested in more detailed analysis of Quality metrics, the FDA commissioned the University of St. Gallen in Switzerland to help establish the scientific base for relevant performance metrics which might be useful in predicting risks of quality failures or drug shortages; the report from the second year of their research was published in November 2018 and it can requested from the mail address Publications-ITEMPM@unisg.ch.
The research outlines the design and development of an holistic, system-based approach to performance management, namely the Pharmaceutical Production System Model (PPSM). The implications of the research on the current FDA Metrics is discussed in the report from year 1, and while limitations are recognised with the data set generated by FDA Quality Metrics, Lot Acceptance Rate and Product Quality Complaint Rate are considered reasonable measures to assess Operational Stability and PQS Effectiveness. The research from year 1 recommends a definition of appropriate limits for the FDA metrics, and notes the absence of any metrics addressing quality culture. In addition, reporting FDA metrics on a product level is recommended along with a consideration of the level of inventory, to assist in preventing drug shortages.
Year 2 of the St. Gallen research examined the validity of their PPSM developed during year 1 and performed a more in-depth analysis of the PQS high and low performer patterns which led to the following findings on:
- Performance – PQS effectiveness high performers (top 10%) showed a higher level of performance across all performance metrics considered in the research.
- Enabler Implementation or operational excellence (OPEX) programmes – The PQS Effectiveness Score correlated positively with the degree of implementation of technical and cultural enablers, g. Total Productive Maintenance (TPM) or Just-in-Time (JIT) production.
- Structural Factors – It could not be confirmed statistically that relative proportion of Quality (or maintenance) FTE / total FTE was significantly different for PQS Effectiveness high and low performers. Similarly, it could not be confirmed statistically that the age of equipment is significantly different for PQS Effectiveness high and low performers.
- Sterile Liquid production only plants (albeit with a limited data set) show, on average, a lower PQS Effectiveness Score, including a lower LAR than Oral Solid Dosage only plants. The complexity of requirements for sterile production appears to impact PQS Effectiveness.
- The QC laboratory robustness high performers
- have a significantly lower invalidated OOS results (average of 12 per 100,000 tests)
- have a significantly higher level of automation
- have integrated OPEX programmes to achieve their superior robustness score.
- There is a positive link between management commitment and the success of the Technical Enablers (OPEX programmes etc.).
There is a lot to consider before initiating a good metrics measurement programme and our advice is to (a) keep it simple, (b) work closely with your technical enablers and (c) be mindful that it can always be improved. Metrics will never capture all aspects of performance or excellence and we are reminded by Albert Einstein “Not everything that can be counted counts and not everything that counts can be counted”.
If you would like further information or wish to discuss how PharmaLex can support you with the development, or improvement, of Quality Metrics that will translate into patient focussed product quality assurance, please connect with us at +353 1 8466 4742 or contactirl@pharmalex.com
