BackgroundWe aimed to select and externally validate a benchmark method for emergency ambulance services to use to forecast the daily number of calls that result in the dispatch of one or more ambulances. The study was conducted using standard methods known to the UK's NHS to aid implementation in practice.MethodsWe selected our benchmark model from a naive benchmark and 14 standard forecasting methods. Mean absolute scaled error and 80 and 95\% prediction interval coverage over a 84 day horizon were evaluated using time series cross validation across eight time series from the South West of England. External validation was conducted by time series cross validation across 13 time series from London, Yorkshire and Welsh Ambulance Services. ResultsA model combining a simple average of Facebook's Prophet and regression with ARIMA Errors (1, 1, 3)(1, 0, 1, 7) was selected. Benchmark MASE, 80 and 95\% prediction intervals were 0.68 (95% CI 0.67 - 0.69), 0.847 (95% CI 0.843 - 0.851), and 0.965 (95% CI 0.949 - 0.977), respectively. Performance in the validation set was within expected ranges for MASE, 0.73 (95% CI 0.72 - 0.74) 80\% coverage (0.833; 95% CI 0.828-0.838), and 95\% coverage (0.965; 95% CI 0.963-0.967).ConclusionsWe provide a robust externally validated benchmark for future ambulance demand forecasting studies to improve on. Our benchmark forecasting model is high quality and usable by ambulance services. We provide a simple python framework to aid its implementation in practice.

ABSTRACTBackgroundThis study presents two simulation modelling tools to support the organisation of networks of dialysis services during the COVID-19 pandemic. These tools were developed to support renal services in the South of England (the Wessex region caring for 650 patients), but are applicable elsewhere.MethodsA discrete-event simulation was used to model a worst case spread of COVID-19 (80% infected over three months), to stress-test plans for dialysis provision throughout the COVID-19 outbreak. We investigated the ability of the system to manage the mix of COVID-19 positive and negative patients, and examined the likely effects on patients, outpatient workloads across all units, and inpatient workload at the centralised COVID-positive inpatient unit. A second Monte-Carlo vehicle routing model estimated the feasibility of patient transport plans and relaxing the current policy of single COVID-19 patient transport to allow up to four infected patients at a time.ResultsIf current outpatient capacity is maintained there is sufficient capacity in the South of England to keep COVID-19 negative/recovered and positive patients in separate sessions, but rapid reallocation of patients may be needed (as sessions are cleared of negative/recovered patients to enable that session to be dedicated to positive patients). Outpatient COVID-19 cases will spillover to a secondary site while other sites will experience a reduction in workload. The primary site chosen to manage infected patients will experience a significant increase in outpatients and in-patients. At the peak of infection, it is predicted there will be up to 140 COVID-19 positive patients with 40 to 90 of these as inpatients, likely breaching current inpatient capacity (and possibly leading to a need for temporary movement of dialysis equipment).Patient transport services will also come under considerable pressure. If patient transport operates on a policy of one positive patient at a time, and two-way transport is needed, a likely scenario estimates 80 ambulance drive time hours per day (not including fixed drop-off and ambulance cleaning times). Relaxing policies on individual patient transport to 2-4 patients per trip can save 40-60% of drive time. In mixed urban/rural geographies steps may need to be taken to temporarily accommodate renal COVID-19 positive patients closer to treatment facilities.ConclusionsDiscrete-event simulation simulation and Monte-Carlo vehicle routing model provides a useful method for stress-testing inpatient and outpatient clinical systems prior to peak COVID-19 workloads.

AbstractObjectivesTo determine the precision of nurse staffing establishments estimated using the SNCT patient classification system, and to assess whether the recommended staff levels correspond with professional judgements of adequate staffing.Setting / population81 medical/surgical units in 4 acute care hospitals.MethodsNurses assessed patients using the SNCT and reported on the adequacy of staffing at least daily for one year. Bootstrap samples of varying sizes were used to estimate the precision of the tool’s recommendations for the number of nurses to employ on each unit. Multi-level regression models were used to assess the association between shortfalls from the measured staffing requirement and nurses’ assessments of adequate staffing.ResultsThe recommended minimum sample of 20 days allowed the required number to employ to be estimated with a mean precision of 4.1%. For most units, much larger samples were required to estimate establishments within +/- 1 whole time staff member. Every registered nurse hour per patient day shortfall in staffing was associated with an 11% decrease in the odds of nurses reporting that there were enough staff to provide quality care and a 14% increase in the odds of reporting that necessary nursing care was left undone. No threshold indicating an optimal staffing level was observed. Surgical specialty, patient turnover and more single bedded rooms were associated with lower odds of staffing adequacy.ConclusionsThe SNCT can provide reliable estimates of the number of nurses to employ on a unit, but larger samples than the recommended minimum are usually required. The SNCT provides a measure of nursing workload that correlates with professional judgements, but the recommended staffing levels may not be optimal. Some sources of systematic variations in staffing requirements for some units are not accounted for. SNCT measurements are a potentially useful adjunct to professional judgement, but cannot replace it.

Abstract
. Background
. We aimed to select and externally validate a benchmark method for emergency ambulance services to use to forecast the daily number of calls that result in the dispatch of one or more ambulances.
.
. Methods
. The study was conducted using standard methods known to the UK’s NHS to aid implementation in practice. We selected our benchmark model from a naive benchmark and 14 standard forecasting methods. Mean absolute scaled error and 80 and 95% prediction interval coverage over a 84 day horizon were evaluated using time series cross validation across eight time series from the South West of England. External validation was conducted by time series cross validation across 13 time series from London, Yorkshire and Welsh Ambulance Services.
.
. Results
. A model combining a simple average of Facebook’s prophet and regression with ARIMA errors (1, 1, 3)(1, 0, 1, 7) was selected. Benchmark MASE, 80 and 95% prediction intervals were 0.68 (95% CI 0.67 - 0.69), 0.847 (95% CI 0.843 - 0.851), and 0.965 (95% CI 0.949 - 0.977), respectively. Performance in the validation set was within expected ranges for MASE, 0.73 (95% CI 0.72 - 0.74) 80% coverage (0.833; 95% CI 0.828-0.838), and 95% coverage (0.965; 95% CI 0.963-0.967).
.
. Conclusions
. We provide a robust externally validated benchmark for future ambulance demand forecasting studies to improve on. Our benchmark forecasting model is high quality and usable by ambulance services. We provide a simple python framework to aid its implementation in practice. The results of this study were implemented in the South West of England.
.

Introduction: the aim of this work was to understand between-hospital variation in thrombolysis use among emergency stroke admissions in England and Wales. Patients: a total of 88,928 patients who arrived at all 132 emergency stroke hospitals in England Wales within 4 h of stroke onset, from 2016 to 2018. Methods: Machine learning was applied to the Sentinel Stroke National Audit Programme (SSNAP) data set, to learn which patients in each hospital would likely receive thrombolysis. We used XGBoost machine learning models, coupled with a SHAP model for explainability; Shapley (SHAP) values, providing estimates of how patient features, and hospital identity, influence the odds of receiving thrombolysis. Results: Thrombolysis use in patients arriving within 4 h of known or estimated stroke onset ranged 7% -49% between hospitals. The odds of receiving thrombolysis reduced 9-fold over the first 120 min of arrival-to-scan time, varied 30-fold with stroke severity, reduced 3-fold with estimated rather than precise stroke onset time, fell 6-fold with increasing pre-stroke disability, fell 4-fold with onset during sleep, fell 5-fold with use of anticoagulants, fell 2-fold between 80 and 110 years of age, reduced 3-fold between 120 and 240 min of onset-to-arrival time and varied 13-fold between hospitals. The majority of between-hospital variance was explained by the hospital, rather than the differences in local patient populations. Conclusions: Using explainable machine learning, we identified that the majority of the between-hospital variation in thrombolysis use in England and Wales may be explained by differences in in-hospital processes and differences in attitudes to judging suitability for thrombolysis.

Aims: to understand how COVID-19 affected nurse staffing in intensive care units (ICUs) in England, and to identify factors that influenced, and were influenced by, pandemic staffing models. Design: Exploratory qualitative study. Methods: Semi-structured, online interviews conducted July–September 2020 with regional critical care leaders including policy leads (n = 4) and directors/lead nurses (n = 10) across critical care networks in England. Findings: the six themes emerging from the framework analysis illustrate how the pre-pandemic ICU culture influenced ICU staffing models during the pandemic. Changes in staffing impacted on the workforce and the care delivered, whilst it was necessary to learn from, and adjust to, a rapidly changing situation. Variation across and between networks necessitated variation in responses. The overwhelming outcome was that the pandemic has challenged the central tenets of ICU nurse staffing. Conclusions: Pandemic nurse staffing models resulted in changes to ICU skill-mix and staffing numbers. Factors such as the impact of nurse staffing on care practices and on the workforce need to be taken into account when developing and testing future nurse staffing models for ICU. The extent to which ICUs will return to former staffing models is not yet known but there seems to be an appetite for change. Impact: in common with many countries, nurse staffing in English ICUs was adapted to address surge requirements during the COVID-19 pandemic. Findings highlight the challenge COVID-19 presented to pre-pandemic ICU nurse staffing guidelines, the impact on patient and staff well-being and the potential legacy for future staffing models. Study findings have implications for ICU nurse managers, researchers and policy makers: nurse staffing models need to be adaptable to the local context of care and future research should investigate the impact of different models on patients, staff and health service outcomes.

Aims: to examine the organisation of the nursing workforce in intensive care units and identify factors that influence how the workforce operates. Background: Pre-pandemic UK survey data show that up to 60% of intensive care units did not meet locally agreed staffing numbers and 40% of ICUs were closing beds at least once a week because of workforce shortages, specifically nursing. Nurse staffing in intensive care is based on the assumption that sicker patients need more nursing resource than those recovering from critical illness. These standards are based on historical working, and expert professional consensus, deemed the weakest form of evidence. Methods: Focus groups with intensive care health care professionals (n = 52 participants) and individual interviews with critical care network leads and policy leads (n = 14 participants) in England between December 2019 and July 2020. Data were analysed using framework analysis. Findings: Three themes were identified: the constraining or enabling nature of intensive care and hospital structures; whole team processes to mitigate nurse staffing shortfalls; and the impact of nurse staffing on patient, staff and intensive care flow outcomes. Staff made decisions about staffing throughout a shift and were influenced by a combination of factors illuminated in the three themes. Conclusions: Whilst nurse:patient ratios were clearly used to set the nursing establishment, it was clear that rostering and allocation/re-allocation during a shift took into account many other factors, such as patient and family nursing needs, staff well-being, intensive care layout and the experience, and availability, of other members of the multi-professional team. This has important implications for future planning for intensive care nurse staffing and highlights important factors to be accounted for in future research studies. Implications for Nursing Management: in order to safeguard patient and staff safety, factors such as the ICU layout need to be considered in staffing decisions and the local business case for nurse staffing needs to reflect these factors. Patient safety in intensive care may not be best served by a blanket ‘ratio’ approach to nurse staffing, intended to apply uniformly across health services.

BACKGROUND: Expert opinion is that about 20% of emergency stroke patients should receive thrombolysis. Currently, 11% to 12% of patients in England and Wales receive thrombolysis, ranging from 2% to 24% between hospitals. The aim of this study was to assess how much variation is due to differences in local patient populations, and how much is due to differences in clinical decision-making and stroke pathway performance, while estimating a realistic target thrombolysis use. METHODS: Anonymised data for 246 676 emergency stroke admissions to 132 acute hospitals in England and Wales between 2016 and 2018 was obtained from the Sentinel Stroke National Audit Programme data. We used machine learning to learn decisions on who to give thrombolysis to at each hospital. We used clinical pathway simulation to model effects of changing pathway performance. Qualitative research was used to assess clinician attitudes to these methods. Three changes were modeled: (1) arrival-to-treatment in 30 minutes, (2) proportion of patients with determined stroke onset times set to at least the national upper quartile, (3) thrombolysis decisions made based on majority vote of a benchmark set of hospitals. RESULTS: of the modeled changes, any single change was predicted to increase national thrombolysis use from 11.6% to between 12.3% to 14.5% (clinical decision-making having the most effect). Combined, these changes would be expected to increase thrombolysis to 18.3%, but there would still be significant variation between hospitals depending on local patient population. Clinicians engaged well with the modeling, but those from hospitals with lower thrombolysis use were most cautious about the methods. CONCLUSIONS: Machine learning and clinical pathway simulation may be applied at scale to national stroke audit data, allowing extended use and analysis of audit data. Stroke thrombolysis rates of at least 18% look achievable in England and Wales, but each hospital should have its own target.

BackgroundStroke is a common cause of adult disability. Expert opinion is that about 20% of patients should receive thrombolysis to break up a clot causing the stroke. Currently, 11–12% of patients in England and Wales receive this treatment, ranging between 2% and 24% between hospitals.ObjectivesWe sought to enhance the national stroke audit by providing further analysis of the key sources of inter-hospital variation to determine how a target of 20% of stroke patients receiving thrombolysis may be reached.DesignWe modelled three aspects of the thrombolysis pathway, using machine learning and clinical pathway simulation. In addition, the project had a qualitative research arm, with the objective of understanding clinicians’ attitudes to use of modelling and machine learning applied to the national stroke audit.Participants and data sourceAnonymised data were collected for 246,676 emergency stroke admissions to acute stroke teams in England and Wales between 2016 and 2018, obtained from the Sentinel Stroke National Audit Programme.ResultsUse of thrombolysis could be predicted with 85% accuracy for those patients with a chance of receiving thrombolysis (i.e. those arriving within 4 hours of stroke onset). Machine learning models allowed prediction of likely treatment choice for each patient at all hospitals. A clinical pathway simulation predicted hospital thrombolysis use with an average absolute error of 0.5 percentage points. We found that about half of the inter-hospital variation in thrombolysis use came from differences in local patient populations, and half from in-hospital processes and decision-making. Three changes were applied to all hospitals in the model: (1) arrival to treatment in 30 minutes, (2) proportion of patients with determined stroke onset times set to at least the national upper quartile and (3) thrombolysis decisions made based on majority vote of a benchmark set of 30 hospitals. Any single change alone was predicted to increase national thrombolysis use from 11.6% to between 12.3% and 14.5% (with clinical decision-making having the most effect). Combined, these changes would be expected to increase thrombolysis to 18.3% (and to double the clinical benefit of thrombolysis, as speed increases also improve clinical benefit independently of the proportion of patients receiving thrombolysis); however, there would still be significant variation between hospitals depending on local patient population. For each hospital, the effect of each change could be predicted alone or in combination. Qualitative research with 19 clinicians showed that engagement with, and trust in, the model was greatest in physicians from units with higher thrombolysis rates. Physicians also wanted to see a machine learning model predicting outcome with probability of adverse effect of thrombolysis to counter a fear that driving thrombolysis use up may cause more harm than good.LimitationsModels may be built using data available in the Sentinel Stroke National Audit Programme only. Not all factors affecting use of thrombolysis are contained in Sentinel Stroke National Audit Programme data and the model, therefore, provides information on patterns of thrombolysis use in hospitals, but is not suitable for, or intended as, a decision aid to thrombolysis.ConclusionsMachine learning and clinical pathway simulation may be applied at scale to national audit data, allowing extended use and analysis of audit data. Stroke thrombolysis rates of at least 18% look achievable in England and Wales, but each hospital should have its own target.Future workFuture studies should extend machine learning modelling to predict the patient-level outcome and probability of adverse effects of thrombolysis, and apply co-production techniques, with clinicians and other stakeholders, to communicate model outputs.FundingThis project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full inHealth and Social Care Delivery Research; Vol. 10, No. 31. See the NIHR Journals Library website for further project information.

We describe a practical two-stage algorithm, BootComp, for multi-objective optimization via simulation. Our algorithm finds a subset of good designs that a decision-maker can compare to identify the one that works best when considering all aspects of the system, including those that cannot be modeled. BootComp is designed to be straightforward to implement by a practitioner with basic statistical knowledge in a simulation package that does not support sequential ranking and selection. These requirements restrict us to a two-stage procedure that works with any distributions of the outputs and allows for the use of common random numbers. Comparisons with sequential ranking and selection methods suggest that it performs well, and we also demonstrate its use analyzing a real simulation aiming to determine the optimal ward configuration for a UK hospital.

Background and motivation: Deep Reinforcement Learning (Deep RL) is a rapidly
developing field. Historically most application has been made to games (such as
chess, Atari games, and go). Deep RL is now reaching the stage where it may
offer value in real world problems, including optimisation of healthcare
systems. One such problem is where to locate ambulances between calls in order
to minimise time from emergency call to ambulance on-scene. This is known as
the Ambulance Location problem.
. Aim: to develop an OpenAI Gym-compatible framework and simulation environment
for testing Deep RL agents.
. Methods: a custom ambulance dispatch simulation environment was developed
using OpenAI Gym and SimPy. Deep RL agents were built using PyTorch. The
environment is a simplification of the real world, but allows control over the
number of clusters of incident locations, number of possible dispatch
locations, number of hospitals, and creating incidents that occur at different
locations throughout each day.
. Results: a range of Deep RL agents based on Deep Q networks were tested in
this custom environment. All reduced time to respond to emergency calls
compared with random allocation to dispatch points. Bagging Noisy Duelling Deep
Q networks gave the most consistence performance. All methods had a tendency to
lose performance if trained for too long, and so agents were saved at their
optimal performance (and tested on independent simulation runs).
. Conclusions: Deep RL agents, developed using simulated environments, have the
potential to offer a novel approach to optimise the Ambulance Location problem.
Creating open simulation environments should allow more rapid progress in this
field.

Background and motivation: Combining Deep Reinforcement Learning (Deep RL)
and Health Systems Simulations has significant potential, for both research
into improving Deep RL performance and safety, and in operational practice.
While individual toolkits exist for Deep RL and Health Systems Simulations, no
framework to integrate the two has been established.
. Aim: Provide a framework for integrating Deep RL Networks with Health System
Simulations, and to ensure this framework is compatible with Deep RL agents
that have been developed and tested using OpenAI Gym.
. Methods: We developed our framework based on the OpenAI Gym framework, and
demonstrate its use on a simple hospital bed capacity model. We built the Deep
RL agents using PyTorch, and the Hospital Simulatation using SimPy.
. Results: We demonstrate example models using a Double Deep Q Network or a
Duelling Double Deep Q Network as the Deep RL agent.
. Conclusion: SimPy may be used to create Health System Simulations that are
compatible with agents developed and tested on OpenAI Gym environments.
. GitHub repository of code:
https://github.com/MichaelAllen1966/learninghospital

Background

The Safer Nursing Care Tool is a system designed to guide decisions about nurse staffing requirements on hospital wards, in particular the number of nurses to employ (establishment). The Safer Nursing Care Tool is widely used in English hospitals but there is a lack of evidence about how effective and cost-effective nurse staffing tools are at providing the staffing levels needed for safe and quality patient care.

Objectives

To determine whether or not the Safer Nursing Care Tool corresponds to professional judgement, to assess a range of options for using the Safer Nursing Care Tool and to model the costs and consequences of various ward staffing policies based on Safer Nursing Care Tool acuity/dependency measure.

Design

This was an observational study on medical/surgical wards in four NHS hospital trusts using regression, computer simulations and economic modelling. We compared the effects and costs of a ‘high’ establishment (set to meet demand on 90% of days), the ‘standard’ (mean-based) establishment and a ‘flexible (low)’ establishment (80% of the mean) providing a core staff group that would be sufficient on days of low demand, with flexible staff re-deployed/hired to meet fluctuations in demand.
Setting

Medical/surgical wards in four NHS hospital trusts.

Main outcome measures

The main outcome measures were professional judgement of staffing adequacy and reports of omissions in care, shifts staffed more than 15% below the measured requirement, cost per patient-day and cost per life saved.

Data sources

The data sources were hospital administrative systems, staff reports and national reference costs.

Results

In total, 81 wards participated (85% response rate), with data linking Safer Nursing Care Tool ratings and staffing levels for 26,362 wards × days (96% response rate). According to Safer Nursing Care Tool measures, 26% of all ward-days were understaffed by ≥ 15%. Nurses reported that they had enough staff to provide quality care on 78% of shifts. When using the Safer Nursing Care Tool to set establishments, on average 60 days of observation would be needed for a 95% confidence interval spanning 1 whole-time equivalent either side of the mean. Staffing levels below the daily requirement estimated using the Safer Nursing Care Tool were associated with lower odds of nurses reporting ‘enough staff for quality’ and more reports of missed nursing care. However, the relationship was effectively linear, with staffing above the recommended level associated with further improvements. In simulation experiments, ‘flexible (low)’ establishments led to high rates of understaffing and adverse outcomes, even when temporary staff were readily available. Cost savings were small when high temporary staff availability was assumed. ‘High’ establishments were associated with substantial reductions in understaffing and improved outcomes but higher costs, although, under most assumptions, the cost per life saved was considerably less than £30,000.
Limitations

This was an observational study. Outcomes of staffing establishments are simulated.

Conclusions

Understanding the effect on wards of variability of workload is important when planning staffing levels. The Safer Nursing Care Tool correlates with professional judgement but does not identify optimal staffing levels. Employing more permanent staff than recommended by the Safer Nursing Care Tool guidelines, meeting demand most days, could be cost-effective. Apparent cost savings from ‘flexible (low)’ establishments are achieved largely by below-adequate staffing. Cost savings are eroded under the conditions of high temporary staff availability that are required to make such policies function.

Future work

Research is needed to identify cut-off points for required staffing. Prospective studies measuring patient outcomes and comparing the results of different systems are feasible.
Trial registration

Current Controlled Trials ISRCTN12307968.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 16. See the NIHR Journals Library website for further project information

One approach to improving antibiotic stewardship in primary care may be to support all General Practitioners (GPs) to have access to point of care C-Reactive Protein tests to guide their prescribing decisions in patients presenting with symptoms of lower respiratory tract infection. However, to date there has been no work to understand how clinical commissioning groups might approach the practicalities of system-wide implementation. We aimed to develop an accessible service delivery modelling tool that, based on open data, could generate a layout of the geographical distribution of point of care facilities that minimised the cost and travel distance for patients across a given region. We considered different implementation models where point of care tests were placed at either GP surgeries, pharmacies or both. We analysed the trade-offs between cost and travel found by running the model under different configurations and analysing the model results in four regions of England (two urban, two rural). Our model suggests that even under assumptions of short travel distances for patients (e.g. under 500m), it is possible to achieve a meaningful reduction in the number of necessary point of care testing facilities to serve a region by referring some patients to be tested at nearby GP surgeries or pharmacies. In our test cases pharmacy-led implementation models resulted in some patients having to travel long distances to obtain a test, beyond the desired travel limits. These results indicate that an efficient implementation strategy for point of care tests over a geographic region, potentially building on primary care networks, might lead to significant cost reduction in equipment and associated personnel training, maintenance and quality control costs; as well as achieving fair access to testing facilities.

BACKGROUND: There is a growing expectation that consultant-level doctors should be present within an ED overnight. However, there is a lack of robust evidence substantiating the impact on patient waiting times, safety or the workforce. OBJECTIVES: to evaluate the impact of consultant-level doctors overnight working in ED in a large university hospital. METHODS: We conducted a controlled interrupted time series analysis to study ED waiting times before and after the introduction of consultant night working. Adverse event reports (AER) were used as a surrogate for patient safety. We conducted interviews with medical and nursing staff to explore attitudes to night work. RESULTS: the reduction seen in average time in department relative to the day, following the introduction of consultant was non-significant (-12 min; 95% CI -28 to 4, p=0.148). Analysis of hourly arrivals and departures indicated that overnight work was inherited from the day. There were three (0.9%) moderate and 0 severe AERs in 1 year. The workforce reported that night working had a negative impact on sleep patterns, performance and well-being and there were mixed views about the benefits of consultant night presence. Additional time off during the day acted as compensation for night work but resulted in reduced contact with ED teams. CONCLUSIONS: Our single-site study was unable to demonstrate a clinically important impact of consultant night working on total time patients spend in the department. Our analysis suggests there may be more potential to reduce total time in department during the day, at our study site. Negative impacts on well-being, and likely resistance to consultant night working should not be ignored. Further studies of night working are recommended to substantiate our results.

OBJECTIVE: to evaluate the application of clinical pathway simulation in machine learning, using clinical audit data, in order to identify key drivers for improving use and speed of thrombolysis at individual hospitals. DESIGN: Computer simulation modelling and machine learning. SETTING: Seven acute stroke units. PARTICIPANTS: Anonymised clinical audit data for 7864 patients. RESULTS: Three factors were pivotal in governing thrombolysis use: (1) the proportion of patients with a known stroke onset time (range 44%-73%), (2) pathway speed (for patients arriving within 4 hours of onset: per-hospital median arrival-to-scan ranged from 11 to 56 min; median scan-to-thrombolysis ranged from 21 to 44 min) and (3) predisposition to use thrombolysis (thrombolysis use ranged from 31% to 52% for patients with stroke scanned with 30 min left to administer thrombolysis). A pathway simulation model could predict the potential benefit of improving individual stages of the clinical pathway speed, whereas a machine learning model could predict the benefit of 'exporting' clinical decision making from one hospital to another, while allowing for differences in patient population between hospitals. By applying pathway simulation and machine learning together, we found a realistic ceiling of 15%-25% use of thrombolysis across different hospitals and, in the seven hospitals studied, a realistic opportunity to double the number of patients with no significant disability that may be attributed to thrombolysis. CONCLUSIONS: National clinical audit may be enhanced by a combination of pathway simulation and machine learning, which best allows for an understanding of key levers for improvement in hyperacute stroke pathways, allowing for differences between local patient populations. These models, based on standard clinical audit data, may be applied at scale while providing results at individual hospital level. The models facilitate understanding of variation and levers for improvement in stroke pathways, and help set realistic targets tailored to local populations.

Despite a long history of health services research that indicates that having sufficient nursing staff on hospital wards is critical for patient safety, and sustained interest in nurse staffing methods, there is a lack of agreement on how to determine safe staffing levels. For an alternative viewpoint, we look to a separate body of literature that makes use of operational research techniques for planning nurse staffing. Our goal is to provide examples of the use of operational research approaches applied to nurse staffing, and to discuss what they might add to traditional methods. The paper begins with a summary of traditional approaches to nurse staffing and their limitations. We explain some key operational research techniques and how they are relevant to different nurse staffing problems, based on examples from the operational research literature. We identify three key contributions of operational research techniques to these problems: “problem structuring”, handling complexity and numerical experimentation. We conclude that decision-making about nurse staffing could be enhanced if operational research techniques were brought in to mainstream nurse staffing research. There are also opportunities for further research on a range of nurse staff planning aspects: skill mix, nursing work other than direct patient care, quantifying risks and benefits of staffing below or above a target level, and validating staffing methods in a range of hospitals.

ObjectivesThere have been claims that Delayed Transfers of Care (DTOCs) of inpatients to home or a less acute setting are related to Emergency Department (ED) crowding. In particular DTOCs were associated with breaches of the UK 4-hour waiting time target in a previously published analysis. However, the analysis has major limitations by not adjusting for the longitudinal trend of the data. The aim of this work is to investigate whether the proposition that DTOCs impact the 4-hour target requires further research.MethodEstimation of an association between two or more variables that are measured over time requires specialised statistical methods. In this study, we performed two separate analyses. First, we created two sets of artificial data with no correlation. We then added an upward trend over time and again assessed for correlation. Second, we reproduced the simple linear regression of the original study using NHS England open data of English trusts between 2010 and 2016, assessing correlation of numbers of DTOCs and ED breaches of the 4-hour target. We then reanalysed the same data using standard time series methods to remove the trend before estimating an association.ResultsAfter introducing upward trends into the uncorrelated artificial data the correlation between the two data sets increased (R2=0.00 to 0.51 respectively). We found strong evidence of longitudinal trends within the NHS data of ED breaches and DTOCs. After removal of the trends the R2 reduced from 0.50 to 0.01.ConclusionOur reanalysis found weak correlation between numbers of DTOCs and ED 4-hour target breaches. Our study does not indicate that there is no relationship between 4-hour target and DTOCs, it highlights that statistically robust evidence for this relationship does not currently exist. Further work is required to understand the relationship between breaches of the 4-hour target and numbers of DTOCs.

OBJECTIVE: to quantify the effect of intrahospital patient flow on emergency department (ED) performance targets and indicate if the expectations set by the National Health Service (NHS) England 5-year forward review are realistic in returning emergency services to previous performance levels. DESIGN: Linear regression analysis of routinely reported trust activity and performance data using a series of cross-sectional studies. SETTING: NHS trusts in England submitting routine nationally reported measures to NHS England. PARTICIPANTS: 142 acute non-specialist trusts operating in England between 2012 and 2016. MAIN OUTCOME MEASURES: the primary outcome measures were proportion of 4-hour waiting time breaches and cancelled elective operations. METHODS: Univariate and multivariate linear regression models were used to show relationships between the outcome measures and various measures of trust activity including empty day beds, empty night beds, day bed to night bed ratio, ED conversion ratio and delayed transfers of care. RESULTS: Univariate regression results using the outcome of 4-hour breaches showed clear relationships with empty night beds and ED conversion ratio between 2012 and 2016. The day bed to night bed ratio showed an increasing ability to explain variation in performance between 2015 and 2016. Delayed transfers of care showed little evidence of an association. Multivariate model results indicated that the ability of patient flow variables to explain 4-hour target performance had reduced between 2012 and 2016 (19% to 12%), and had increased in explaining cancelled elective operations (7% to 17%). CONCLUSIONS: the flow of patients through trusts is shown to influence ED performance; however, performance has become less explainable by intratrust patient flow between 2012 and 2016. Some commonly stated explanatory factors such as delayed transfers of care showed limited evidence of being related. The results indicate some of the measures proposed by NHS England to reduce pressure on EDs may not have the desired impact on returning services to previous performance levels.

Stroke care has been identified as an area where operations research has great potential. In recent years there has been a small but sustained stream of discrete-event simulation case studies in modelling hyperacute stroke systems. The nature of such case studies has led to a fragmented knowledge base and high entry cost to stroke modelling research. Two common issues have faced researchers in stroke care: understanding the logistics and clinical aspects of stroke care and moving from these findings to an appropriately detailed model. We aim to accelerate studies in this area by introducing a conceptual modelling framework that is domain specific for stroke. A domain specific framework trades-off the wide applicability of a general framework against increased efficiency and reuse to support modelling in the problem domain. This compromise is appropriate when the problem domain is complex, of high value to society, and where the saving in future modelling effort is likely to be greater than the effort to create the framework. We detail the requirements of a domain specific conceptual model and then provide domain specific knowledge to support modellers in gaining an understanding of the problem situation, translating this knowledge into selected model outputs, inputs and content in the case of hyperacute stroke. We illustrate the use of the framework with an example based at a large hospital in the United Kingdom.

OBJECTIVES: the policy of centralising hyperacute stroke units (HASUs) in England aims to provide stroke care in units that are both large enough to sustain expertise (>600 admissions/year) and dispersed enough to rapidly deliver time-critical treatments (

BACKGROUND: Mathematical capacity planning methods that can take account of variations in patient complexity, admission rates and delayed discharges have long been available, but their implementation in complex pathways such as stroke care remains limited. Instead simple average based estimates are commonplace. These methods often substantially underestimate capacity requirements. We analyse the capacity requirements for acute and community stroke services in a pathway with over 630 admissions per year. We sought to identify current capacity bottlenecks affecting patient flow, future capacity requirements in the presence of increased admissions, the impact of co-location and pooling of the acute and rehabilitation units and the impact of patient subgroups on capacity requirements. We contrast these results to the often used method of planning by average occupancy, often with arbitrary uplifts to cater for variability. METHODS: We developed a discrete-event simulation model using aggregate parameter values derived from routine administrative data on over 2000 anonymised admission and discharge timestamps. The model mimicked the flow of stroke, high risk TIA and complex neurological patients from admission to an acute ward through to community rehab and early supported discharge, and predicted the probability of admission delays. RESULTS: an increase from 10 to 14 acute beds reduces the number of patients experiencing a delay to the acute stroke unit from 1 in every 7 to 1 in 50. Co-location of the acute and rehabilitation units and pooling eight beds out of a total bed stock of 26 reduce the number of delayed acute admissions to 1 in every 29 and the number of delayed rehabilitation admissions to 1 in every 20. Planning by average occupancy would resulted in delays for one in every five patients in the acute stroke unit. CONCLUSIONS: Planning by average occupancy fails to provide appropriate reserve capacity to manage the variations seen in stroke pathways to desired service levels. An appropriate uplift from the average cannot be based simply on occupancy figures. Our method draws on long available, intuitive, but underused mathematical techniques for capacity planning. Implementation via simulation at our study hospital provided valuable decision support for planners to assess future bed numbers and organisation of the acute and rehabilitation services.

KEYNOTE PRESENTATIONS K1 Researching complex interventions: the need for robust approaches Peter Craig K2 Complex intervention studies: an important step in developing knowledge for practice Ingalill Rahm-Hallberg K3 Public and patient involvement in research: what, why and how? Nicky Britten K4 Mixed methods in health service research – where do we go from here? Gunilla Borglin SPEAKER PRESENTATIONS S1 Exploring complexity in systematic reviews of complex interventions Gabriele Meyer, Sascha Köpke, Jane Noyes, Jackie Chandler S2 can complex health interventions be optimised before moving to a definitive RCT? Strategies and methods currently in use Sara Levati S3 a systematic approach to develop theory based implementation interventions Anne Sales S4 Pilot studies and feasibility studies for complex interventions: an introduction Lehana Thabane, Lora Giangregorio S5 What can be done to pilot complex interventions? Nancy Feeley, Sylvie Cossette S6 Using feasibility and pilot trials to test alternative methodologies and methodological procedures prior to full scale trials Rod Taylor S7 a mixed methods feasibility study in practice Jacqueline Hill, David a Richards, Willem Kuyken S8 Non-standard experimental designs and preference designs Louise von Essen S9 Evaluation gone wild: using natural experimental approaches to evaluate complex interventions Andrew Williams S10 the stepped wedge cluster randomised trial: an opportunity to increase the quality of evaluations of service delivery and public policy interventions Karla Hemming, Richard Lilford, Alan Girling, Monica Taljaard S11 Adaptive designs in confirmatory clinical trials: opportunities in investigating complex interventions Munyaradzi Dimairo S12 Processes, contexts and outcomes in complex interventions, and the implications for evaluation Mark Petticrew S13 Processes, contexts and outcomes in complex interventions, and the implications for evaluation Janis Baird, Graham Moore S14 Qualitative evaluation alongside RCTs: what to consider to get relevant and valuable results Willem Odendaal, Salla Atkins, Elizabeth Lutge, Natalie Leon, Simon Lewin S15 Using economic evaluations to understand the value of complex interventions: when maximising health status is not sufficient Katherine Payne S16 How to arrive at an implementation plan Theo van Achterberg S17 Modelling process and outcomes in complex interventions Walter Sermeus S18 Systems modelling for improving health care Martin Pitt, Thomas Monks

The ever increasing pressures to ensure the most efficient and effective use of limited health service resources will, over time, encourage policy makers to turn to system modelling solutions. Such techniques have been available for decades, but despite ample research which demonstrates potential, their application in health services to date is limited. This article surveys the breadth of approaches available to support delivery and design across many areas and levels of healthcare planning. A case study in emergency stroke care is presented as an exemplar of an impactful application of health system modelling. This is followed by a discussion of the key issues surrounding the application of these methods in health, what barriers need to be overcome to ensure more effective implementation, as well as likely developments in the future.

Very few discrete-event simulation studies follow up on recommendations with evaluation of whether modelled benefits have been realised and the extent to which modelling contributed to any change. This paper evaluates changes made to the emergency stroke care pathway at a UK hospital informed by a simulation modelling study. The aims of the study were to increase the proportion of people with strokes that undergo a time-sensitive treatment to breakdown a blood clot within the brain and decrease the time to treatment. Evaluation involved analysis of stroke treatment pre- and post-implementation, as well as a comparison of how the research team believed the intervention would aid implementation compared to what actually happened. Two years after the care pathway was changed, treatment rates had increased in line with expectations and the hospital was treating four times as many patients than before the intervention in half the time. There is evidence that the modelling process aided implementation, but not always in line with expectations of the research team. Despite user involvement throughout the study it proved difficult to involve a representative group of clinical stakeholders in conceptual modelling and this affected model credibility. The research team also found batch experimentation more useful than visual interactive simulation to structure debate and decision making. In particular, simple charts of results focused debates on the clinical effectiveness of drugs - an emergent barrier to change. Visual interactive simulation proved more useful for engaging different hospitals and initiating new projects.

Very few discrete-event simulation studies follow up on recommendations with evaluation of whether modelled benefits have been realised and the extent to which modelling contributed to any change. This paper evaluates changes made to the emergency stroke care pathway at a UK hospital informed by a simulation modelling study. The aims of the study were to increase the proportion of people with strokes that undergo a time-sensitive treatment to breakdown a blood clot within the brain and decrease the time to treatment. Evaluation involved analysis of stroke treatment pre- and post-implementation, as well as a comparison of how the research team believed the intervention would aid implementation compared to what actually happened. Two years after the care pathway was changed, treatment rates had increased in line with expectations and the hospital was treating four times as many patients than before the intervention in half the time. There is evidence that the modelling process aided implementation, but not always in line with expectations of the research team. Despite user involvement throughout the study it proved difficult to involve a representative group of clinical stakeholders in conceptual modelling and this affected model credibility. The research team also found batch experimentation more useful than visual interactive simulation to structure debate and decision making. In particular, simple charts of results focused debates on the clinical effectiveness of drugs - an emergent barrier to change. Visual interactive simulation proved more useful for engaging different hospitals and initiating new projects.

Discussion of learning from discrete-event simulation often takes the form of a hypothesis stating that involving clients in model building provides much of the learning necessary to aid their decisions. Whilst practitioners of simulation may intuitively agree with this hypothesis they are simultaneously motivated to reduce the model building effort through model reuse. As simulation projects are typically limited by time, model reuse offers an alternative learning route for clients as the time saved can be used to conduct more experimentation. We detail a laboratory experiment to test the high involvement hypothesis empirically, identify mechanisms that explain how involvement in model building or model reuse affect learning and explore the factors that inhibit learning from models. Measurement of learning focuses on the management of resource utilisation in a case study of a hospital emergency department and through the choice of scenarios during experimentation. Participants who reused a model benefitted from the increased experimentation time available when learning about resource utilisation. However, participants who were involved in model building simulated a greater variety of scenarios including more validation type scenarios early on. These results suggest that there may be a learning trade-off between model reuse and model building when simulation projects have a fixed budget of time. Further work evaluating client learning in practice should track the origin and choice of variables used in experimentation; studies should also record the methods modellers find most effective in communicating the impact of resource utilisation on queuing. © 2013 Elsevier B.V. All rights reserved.

Interest is growing internationally in the potential benefits of patient and public involvement (PPI) in research. In the United Kingdom (UK) health and social care services are now committed to involving patients and service users in the planning, development and evaluation of their services. Many funders require PPI as a prerequisite for funding. What does healthcare operational research miss by not involving patients and the public in the development, refinement and implementation of models? We believe PPI is important for healthcare OR for model design and validation, and ethical and economic reasons. It also has a distinct contribution that goes beyond the incorporation of behavioural parameters into models. Case studies in neonatal care and a fractured neck of femur pathway highlight PPI’s contribution to model design and validation, but a recent conference session also identified a number of obstacles. We suggest a provisional model for the implementation of PPI in healthcare OR that emphasises a facilitative approach. We acknowledge this is a significant challenge, but argue that it must be met for ethical and economic reasons that are ultimately rooted in modellers’ construction of valid models. Crucially, it has the potential to enhance our ability to bring about change which can benefit health services and, most importantly, the patients they serve.

BACKGROUND AND PURPOSE: to maximize the benefits of thrombolysis, it is necessary not only to treat more patients, but to deliver treatment as early as possible. The aims of our study were to prospectively evaluate the clinical benefit from reducing delays in the emergency stroke pathway at our district hospital and examine outcomes from scenarios that include extension of the alteplase license. METHODS: We developed a discrete-event simulation from prospective data for patients with stroke arriving at our large district hospital. We modeled current practice and assessed the impact on stroke outcomes of measures to reduce in-hospital delays to alteplase treatment and of extensions to the European license for alteplase from 3 to 4.5 hours and to people aged >80 years. RESULTS: Extension of the time window to 4.5 hours increases the thrombolysis rate by 4%, yielding an additional 2 patients per year with minimal or no disability at 3 months. Time window extension is most effective when combined with a system of prealerts, achieving a thrombolysis rate of 15% and an additional 8 patients per year with minimal or no disability, increasing to 13 patients per year with extension of the license to patients >80 years. CONCLUSIONS: If implemented alone, extension of the time window for alteplase has only a modest additional population disability benefit, but this benefit can be increased 5-fold if time window extension is combined with substantial reductions to in-hospital delays.

BACKGROUND AND PURPOSE: Pooled analyses show benefits of intravenous alteplase (recombinant tissue-type plasminogen activator) treatment for acute ischemic stroke up to 4.5 hours after onset despite marketing approval for up to 3 hours. However, the benefit from thrombolysis is critically time-dependent and if extending the time window reduces treatment urgency, this could reduce the population benefit from any extension. METHODS: Based on 3830 UK patients registered between 2005 to 2010 in the Safe Implementation of Treatments in Stroke-International Stroke Thrombolysis Registry (SITS-ISTR), a Monte Carlo simulation was used to model recombinant tissue-type plasminogen activator treatment up to 4·5 hours from onset and assess the impact (numbers surviving with little or no disability) from changes in hospital treatment times associated with this extended time window. RESULTS: We observed a significant relation between time remaining to treat and time taken to treat in the UK SITS-ISTR data set after adjustment for censoring. Simulation showed that as this "deadline effect" increases, an extended treatment time window entails that an increasing number of patients are treated at a progressively lower absolute benefit to a point where the population benefit from extending the time window is entirely negated. CONCLUSIONS: Despite the benefit for individual patients treated up to 4.5 hours after onset, the population benefit may be reduced or lost altogether if extending the time window results in more patients being treated but at a lower absolute benefit. A universally applied reduction in hospital arrival to treatment times of 8 minutes would confer a population benefit as large as the time window extension.

Our main aim in this paper is to highlight current practice and education in multiple scenario comparison within DES experimentation and to illustrate the possible benefits of employing false discovery rate (FDR) control as opposed to strict family-wise error rate (FWER) control when comparing large numbers of scenarios in an exploratory manner. We present the results of a small survey into the current practice of scenario analysis by simulation practitioners and academics. The results indicated that the range of scenarios used in DES studies may prohibit the use of FWER control methods such as the Bonferroni Correction referred to in DES textbooks. Furthermore, 80% of our sample were not familiar with any of the multiple comparison control procedures presented to them. We provide a practical example of the FDR in action and argue that it is preferable to employ FDR instead of no multiple comparison control in exploratory style studies. © 2011 IEEE.