Methods Research

This NIHR-funded project aims to improve how structural and key methodological uncertainties are identified and analysed in NICE technology appraisals by reviewing current practice, understanding limitations in existing guidance, and exploring when more formal methods should be used. Working with a multidisciplinary team and external experts, the project will review recent NICE appraisals, conduct a targeted literature review, and carry out interviews with committee chairs, members, and methodologists to understand real‑world decision‑making needs. Through case studies and a mock committee meeting, the team will test and refine recommendations, ultimately producing practical guidance to support more transparent, consistent, and decision‑relevant sensitivity analysis in future NICE submissions.

Funded by NIHR: NHR177738

The aim of this NIHR funded project was to develop clear, practical, and consensus based guidance for analysing individual patient level utility data to support robust economic modelling in NICE health technology assessments. PenTAG achieved this by conducting an extensive programme of work including a scoping survey of 32 stakeholders, a review of 125 recent NICE Technology Appraisals, and a landscape and targeted literature review encompassing 56 relevant studies, followed by a three round Delphi process with 31 experts to build consensus. Through this methodology, PenTAG identified widespread inconsistencies and gaps in current practice—such as poor reporting of value sets, frequent redaction of utility values, and variable analytical quality—and produced a comprehensive set of recommendations covering regression methods, covariate selection, missing data handling, adjustment for common issues, and sensitivity analysis, ultimately providing the first structured, evidence based guidance aimed at improving the quality, consistency, and transparency of utility data analysis in NICE submissions.

Funded by NIHR: NIHR175915

The published NIHR monograph can be found here

Related publications:

 

The aim of this ABPI-funded study was to develop best practice recommendations on the assessment of clinical similarity for HTA submissions that use cost‑comparison analyses. Because many treatments lack direct head‑to‑head trials, PenTAG examined methods for showing equivalence using indirect treatment comparisons (ITCs). PenTAG found several formal statistical approaches, with the strongest being Bayesian noninferiority ITCs, which estimates an indirect treatment effect and then compares it probabilistically with a predefined noninferiority margin.

However, PenTAG found that none of the past NICE appraisals had used these formal methods. Rather, companies relied on narrative summaries, often leading to uncertainty that had to be resolved only through expert opinion. PenTAG recommend integrating preliminary ITC results into the scoping stage and adopting formal noninferiority‑based methods to make similarity assessments more transparent and robust.

Related publications:

Lee, D., Allen, A., Lovell, A., Abdelsabour, A., Wilson, E.C.F., Melendez-Torres, G.J. How Similar Is Similar Enough? Assessment of Indirect Treatment Comparisons to Support Similarity for National Institute for Health and Care Excellence's Cost Comparison Route. Value Health. 2025 Nov;28(11):1770-1779. doi: 10.1016/j.jval.2025.05.003

PenTAG played a central role in NICE’s renal cell carcinoma (RCC) pathways pilot, collaborating directly with NICE to design and deliver the first pathways-based health technology appraisal model for advanced RCC. Our work led to the development of the Exeter Oncology Model: RCC edition (EOM‑RCC), a complex sequencing model use to evaluate 12 treatment options across multiple risk groups, prior therapy histories, and up to four lines of active treatment. This model served as the foundation for testing NICE’s new “pathways” approach; an initiative aimed at improving consistency and efficiency in decision making by creating reusable reference models for disease areas with many appraisals. PenTAG’s responsibilities included constructing the economic model, synthesizing clinical and real‑world evidence, and producing both the preliminary and final assessment reports within an ambitious timeline.

The pilot also required PenTAG to apply the new model to a live appraisal: evaluating cabozantinib plus nivolumab as a first‑line treatment for RCC. Our analysis contributed to testing how the pathways framework could handle multi‑comparator decision spaces and integrate innovative methodological elements such as real‑world evidence and expert elicitation. PenTAG’s experience in this pilot highlighted both the potential of pathway‑based modelling for future HTAs and the practical challenges of implementing such a comprehensive framework under tight development constraints.

Funded by NIHR: NIHR136008

The published NIHR monograph can be found here

Related publications:

Lee, D., Burns, D., Wilson, E. NICE’s pathways pilot: pursuing good decision making in difficult circumstances. Pharmacoecon Open 2024;8:645–9. https://doi.org/10.1007/s41669-024-00490-x. Epub 2024 Apr 13. [Erratum published in Pharmacoecon Open 2024;8:783.]. https://doi.org/10.1007/s41669-024-00499-2 

Lee, D., Melendez-Torres, G.J,, Challapalli, A., Frazer, R., McGrane, J., Bahl, A. Efficacy of cabozantinib and sunitinib for the treatment of intermediate/poor risk renal cell carcinoma based upon UK real-world data. ESMO Real World Data Digit Oncol 2024;6:100087. 

Lee, D., Ahmad, Z., Larkin, J.M.G., Bahl, A., Melendez-Torres, G.J. Structured expert elicitation to inform long-term survival extrapolations in advanced renal cell carcinoma [published online ahead of print 27 August 2025]. Appl Health Econ Health Policy 2025. https://doi.org/10.1007/s40258-025-01000-8. 

McGrane, J., Frazer, R., Challapalli, A., Ratnayake, G., Boh, Z., Clayton, A., et al. Real world, multicentre patterns of treatment and survival in metastatic renal cell carcinoma with the UK Renal Oncology Collaborative (UK ROC): is it time to look favourably on first‐line immunotherapy containing combinations in all IMDC groups? Cancer Med 2024;13:e7327. https://doi.org/10.1002/cam4.7327

Smith, R.A., Samyshkin, Y., Mohammed, W., Lamrock, F., Ward, T., Smith, J., et al. assertHE: an R package to improve quality assurance of HTA models [version 1; peer review: 1 approved with reservations]. Wellcome Open Res 2024;9:701

Lee, D., Ahmad, Z., Farmer, C., Barnish, M.S., Lovell, A., Melendez-Torres, G.J. Slipping away: slippage in hazard ratios over datacuts and its impact on immuno-oncology combination economic evaluations. Value Health 2025;28:260–8. https:// doi.org/10.1016/j.jval.2024.09.008

Frazer, R., McGrane, J.M., Challapalli, A., Ratnayake, G., Malik, J., Forde, C., Alam, S., Jones, E., Shaheen, A., Ferrara, A., Gupta, S., Moorcraft, S.Y., Moon, N., Parslow, D., Wang, Y., Walters, S., Liu, J., Geldart, T., Dyke, C., Reni, A., Melendez-Torres, G.J., Bahl, A. Real-world patterns of treatment and response in metastatic renal cell carcinoma: a multicentre UK-wide review with UK Renal Oncology Collaborative (UK ROC). ESMO Real World Data Digit Oncol. 2024 Mar 11;3:100027. doi: 10.1016/j.esmorw.2024.100027

Open-source R model: https://pythonhealthdatascience.github.io/stars-eom-rcc/

This review looked at how NICE technology appraisals handle one of their most persistent methodological challenges: estimating long‑term treatment effects when trial follow‑up is limited. The study identified substantial heterogeneity in how treatment effect waning is incorporated across appraisals, including an over‑reliance on precedent and limited justification for chosen methods, despite the major influence these assumptions have on cost‑effectiveness outcomes and funding decisions. By reviewing waning assumptions used in 28 recent appraisals and highlighting issues such as the frequent but poorly justified use of a five‑year timepoint and inconsistent links between waning and stopping rules, the manuscript demonstrates why robust estimation of long‑term treatment effects remains a key challenge across NICE assessments and provides guidance for manufacturers, EAGs and Committee's on how to use treatment effect waning assumptions within model base case and sensitivity analyses.

Related publications:

Trigg, L.A., Melendez-Torres, G.J., Abdelsabour, A., Lee D. Treatment Effect Waning Assumptions: A Review of National Institute of Health and Care Excellence Technology Appraisals. Value Health. 2024 Aug;27(8):1003-1011. doi: 10.1016/j.jval.2024.04.016

Lee, D., Ahmad, Z., Farmer, C., Barnish, M.S., Lovell, A., Melendez-Torres, G.J. Slipping Away: Slippage in Hazard Ratios Over Datacuts and Its Impact on Immuno-oncology Combination Economic Evaluations. Value Health. 2025 Feb;28(2):260-268. doi: 10.1016/j.jval.2024.09.008.

PenTAG was commissioned by NHSE to evaluate managed access policy in England through a series of seven focus groups (of payer and HTA representatives, clinicians, and representatives from industry and patient/carer organizations). The research described how the use of managed access agreements (MAA) is just one strategy that can be used by payers to resolve uncertainty for innovative technologies that present challenges for reimbursement. While their use may on some occasions deliver earlier access to promising technologies for patients, they should not be considered a “silver bullet”. In fact, their uncontrolled use increases risks with regard to healthcare spending and patient care. In order to manage their implementation, participants in the workshops agreed that there is a need for greater clarity about the aims of managed access and how these should be prioritized in decision making.

Related publications:

Farmer, C., Barnish, M.S., Trigg, L.A., Hayward, S., Shaw, N., Crathorne, L., Strong, T., Groves, B., Spoors, J., Melendez Torres, G.J. An evaluation of managed access agreements in England based on stakeholder experience. Int J Technol Assess Health Care. 2023 Jul 27;39(1):e55. doi: 10.1017/S0266462323000478

PenTAG's was commissioned by NHSE to inform the debate on changes to the NICE cost-effectiveness threshold by reviewing the cost-effectiveness of elective surgical interventions as a counter-point to considerations around pharmaceuticals which have so far driven the discussion. Almost all surgical procedures reviewed would be considered cost effective yielding incremental cost-effectiveness ratios (ICERs) below £20,000 per quality-adjusted life-year gained. Allocation of investment to areas that are more cost effective than others is likely to increase the efficiency of the NHS, resulting in a net health gain compared with the reimbursement of less cost-effective interventions.

Related publications:

Trigg, L.A., Farmer, C., Muthukumar, M., Wilson, E.C.F., Lovell, A., Lee, D. The Cost Effectiveness of Elective Surgical Procedures with Longer NHS Waiting Lists: A Targeted Review. Appl Health Econ Health Policy. 2025 Sep;23(5):779-796. doi:10.1007/s40258-025-00975-8. Epub 2025 May 9