T 21: Risk Assessment of Sites Through Risk-Based Monitoring (RBM): Do Your Monitors Agree? A Joint Case Study
Poster Presenter
Nicholas Hargaden
Director, Clinical Systems Strategy
Beigene United States
Objectives
The primary objective of this case study was to understand whether a simplified RBM approach that is data-driven could provide a view of site risks and also whether such an approach is congruent to study monitor’s perception of risk at sites who had used conventional on-site monitoring techniques.
Method
This blinded study was conducted in Sep 2015 along with the monitoring and management team at Neuroscience Trials (NT), Australia. Site anonymized data was collected, data-driven, visualization enabled manual review, and risk assessment of sites was performed. This was then validated with NT monitors.
Authors:
Abby Abraham – Vice-President, Clinical Solutions
Tina Soulis – General Manager, Neuroscience Trials
Nick Hargaden – President, US Operations
Results
Data from a completed Phase 2b study involving acute ischemic strokes from 12 sites and 77 subjects was provided. • The protocol was reviewed by Algorics team and inputs pertaining to the Protocol were received from Neuroscience trials team. Factors that can be a potential risk to conduct of the study were identified. Additional risks related to what is reviewed from site monitoring was also factored in and finalized.• Risk scoring process: Each risk parameter is provided a graded scoring guideline which is dependent upon the degree of impact it could cause on the outcome of study conduct. The aforementioned risk parameters were mapped from the study data report and data visualizations and thresholds were built for these risk parameters so as to easily and effectively review data by Algorics team. Subsequently, two types of analysis was used. 1. Manual review of data and deriving a site risk score. 2. Data-driven model on specific parameters (Percentile and k-means cluster models) that could provide insight into certain aspects of site’s functioning which could be linked to site’s functioning and thereby risk profile.
• The following are the key observations upon analysis of data through both approaches:
o Consistency of risk classification (75% incidence and above) of sites across Manual data review and the two data-driven models: 70%
o Consistency of risk classification of sites across manual data review and at least one data driven model was 70%. (Sites C,E & J).
o Consistency of risk classification across both data-driven models (K-means cluster and percentile method on time to groin puncture delay) was 80% (Sites A,B,C,D,E,G,H,I)
o High risk and medium risk classified site in manual data review that coincided with one data-driven model (Percentile method on Groin puncture delay): 80% (Sites C,E,F,H & J)
This was validated with the relevant site monitors whose impression of high and medium risk sites coincided with the data-driven model.
Conclusion
This exploratory exercise helped in drawing the following conclusions:
• Manual data review when performed in an objective and risk based approach was able to detect risks at site. Application of this approach during source data review (SDR) can significantly help in continually assessing risk of sites based on monitor’s feedback.
• Though there may be reservations about using data-driven models to assess risk, this case study demonstrates that when certain important processes executed at sites that are critical to study outcomes are selected and the data is run through relevant data-driven models, it does provide insight into how sites are functioning. This can be an important input to determine risk at sites proactively.
• Manual data review when complemented with simple data-driven statistical models could increase the likelihood of characterizing a site risk profile.
• In order to perform risk based approaches to monitoring, execution of such a process can be enabled through the use of clinical technology solution that helps in risk planning, data visualization and data-driven statistical modeling.
• The use relative ease of use of data visualizations was demonstrated. NT team was provided access to the data visualizations system (Acuity) and they reported that a single complete subject review took about 25 mins initially for review. This subsequently reduced to about 15 mins after getting used to the tool and the modified review methodology. This proves the need of such a technology platform to increase efficiency and effectiveness of data review in conventional monitoring models as well as in RBM.
