W 24: Innovation in Regulatory Science: Development and Validation of an Instrument for Assessing the Quality of Decision Making
Poster Presenter
Stuart Russel Walker
Professor and Consultant
Centre for Innovation In Regulatory Science (CIRS) United Kingdom
Objectives
The objective of this study was to develop a standardized and validated tool (Quality of Decision-Making Orientation Scheme – QoDoS) for assessing the quality of decision-making (QDM) in medicines development and regulatory review using both qualitative and quantitative techniques.
Method
QoDoS items were generated from Interviews with 29 key opinion leaders and content validity was established using an expert panel (Donelan et al. 2015). Psychometric evaluations (i.e. factor analysis, reliability and construct validation) were performed.
Co-authors: Bujar M., Donelan R., Walker S.
Results
The thematic analysis of the interviews yielded a 94-item initial version of the QoDoS with a 5-point Likert scale response option. The instrument was tested for content validity using a panel of experts to rate language clarity, completeness, relevance and scaling of each item on a 4-point scale (Strongly agree to strongly disagree). The agreement among the panel members was high with an intra-class correlation coefficient value of 0.89 (95% confidence interval = .056, 0.99). A 76- item QoDoS (version 2) resulted from content validation.
Factor analysis produced a 47-item measure with four domains grouped into two parts (Part I = Organisational – Decision-Making Approach, Decision-Making Culture; Part II = Individual – Decision-Making Competence, Decision-Making Style). The 47-item QoDoS (version 3) showed high internal consistency (n = 120, Cronbach’s alpha = 0.89), high reproducibility (n = 20, intra-class correlation = 0.77) and a mean completion time of 10 minutes. This suggests that the QoDoS is a practical instrument possessing strong psychometric properties of validity and reliability.
A secondary outcome of this study has been the important insights into the decision-making of 76 individuals from pharmaceutical companies (50%) and regulatory agencies (50%) who participated in a study testing the responsiveness of the QoDoS. The results showed that whilst it was recognized that the science of decision-making is important, training in this area was rarely provided. In addition all responders from agencies and 92% from companies felt that, with targeted training, they could make better decisions. The QoDoS was able to differentiate between the issues important to pharmaceutical companies and regulatory authorities.
Conclusion
Although the impact of decision-making during the development and regulatory review of medicines greatly influences delivery of new products, there appears to be no suitable instrument that can be used to assess QDM. This study has described the development and initial psychometric properties of a new tool that aims to address this unmet need using a standardised methodology.
Factor analysis was followed by construct validation, examining convergence (evidence that different measurement methods of a closely related constructs correlate) and discriminant/divergent validity (ability to differentiate the construct from other distantly related constructs) of the QoDoS. The results showed that the instrument possesses strong measurement properties of reliability and validity which should provide confidence for its use in the scenarios outlined above.
The findings of this study also provide insights regarding the decision-making approach of organisations compared with those of individuals. The QoDoS can also identify similarities and differences between regulatory agencies and companies’ practices as well as areas for improvement for both stakeholders. This profiling, which is performed as a point-in-time assessment, should allow monitoring of the changes in decision-making over time, including following specific initiatives.
The QoDoS can therefore be used to increase awareness of the biases and influences that need to be considered when making decisions, as well as the best practices that should be incorporated into a decision-making framework such as: having a systematic, structured approach to aid decision-making); assigning values and relative importance to decision criteria; evaluating internal and external influences/biases; considering uncertainty; performing impact analysis; and ensuring transparency. Such practices underpin the FDA’s recent initiative to establish “the science of therapeutic regulatory decision-making” (Edlavitch and Salmon, 2015).
