Predicting cause of death and the presence of bad healthcare practice from free-text summaries

We used machine learning to predict cause of death from free text summaries of those diagnosed with prostate cancer; and the presence of bad practice in health and social care for those who have died with learning difficulties.  

Aims

Knowledge of underlying cause of death (UCoD) is a key health outcome in research and service improvement but is not always accurately recorded. Likewise, the identification of bad practice is also vital to understanding health outcomes in the population and improving health care provision 

We have applied machine learning (ML) classifiers to over 4,000 death reviews (free text summaries), from the Cancer Research UK Cluster randomised trial of PSA testing for prostate cancer (CAP) and Learning Disability Mortality Review programme (LeDeR). Each review was assigned a label either prostate cancer death or poor healthcare practice by independent experts. This expert assignment was used to train the ML techniques to: 

  1. Identify the key elements (words and phrases) that are good predictors of
  • Prostate cancer death or 
  • Poor health or social-care practice  

2. Add user confidence by explaining how the ML techniques work, rather than solely relying on the prediction probability that is output by the classifier. In this sense we add transparency to the ML.

We developed the methodology using data from the CAP project, and subsequently applied it to the LeDeR data to test how well it could generalise.  

Results

The first step was to build a tool to predict prostate cancer death from the free text summaries. Using a random forest (RF) classifier with a bag-of-words feature set we found that we could predict prostate cancer death with >90% accuracy. We then investigated how the RF was classifying the free-text summaries by looking at which elements in the free-text summarises were used by the RF to assign prostate cancer death. To do this we investigated a variety of potential visualisation techniques: 

Word clouds

Word clouds provide a visual representation of the words (or group of words) that are most predictive of prostate cancer deaths across the dataset. The word clouds show that clinically important signs of progressing prostate cancer, are key to identifying prostate cancer deaths.  

Figure 1: A word cloud showing the most important features for classification of prostate cancer death (using a random forest). The size of the word indicates the ‘importance’ of the feature.

Interpretable text-outputs

We also used both tree-interpreter and LIME to identify which words contributed most to any given classification. We then visualised these contributions by formatting the original free-text summary to indicate which text elements contribute to the classification, so that the classifier’s decision can be understood by a human reader. 

Figure 2: Snapshot of human-interpretable classifier output. Text elements in the cause of death review are formatted to show their contribution to the classification. The size of the font indicates the importance of the word or phrase and the colour illustrates whether the word indicates prostate cancer death (blue) or not (red).

Writing style

Using free-text is dependent on the quality of the text, which can influence the performance of ML techniques.  

These visualisations use the t-SNE algorithm to show clear clusters of free-text summaries that contain similar elements. Here we can identify the three main clusters, where the summaries share commonalities in style, and represent three different authors, where each author is presented by a specific colour. This analysis even brought the authorship of some reviews into question (note the two separate pink clusters, which appear to be so divergent in this data representation). 

Figure 3: T-distributed stochastic neighbour embedding (t-SNE) of the feature space. Each point is a cause of death review and their proximity in the space indicates their similarity in terms of the language they contain. Colours represents authors of the review. The blue line is a boundary that separates the two distinct clusters of reviews written by the pink author.

Hard vs Easy cases

Cases where there was disagreement between the panel of experts about the cause of death are potentially the same cases where the ML classifier is less certain of the prediction. Figure 4 illustrates that the cases the expert panel found “hard” to assign cause of date (purple and black points in the right hand panel) often sit in the location where prostate cancer (red) and non-prostate cancer deaths (blue) meet in the left hand panel. This is a space where the ML techniques are less certain about when predicting cause of death. This is confirmed by figure 5, which shows that the classifier performs much worse on the ‘hard’ cases than the ‘easy’ cases. 

Figure 4: The two panels show the same embedding of the feature space with points coloured according to: cause of death (left panel) and ‘difficulty’ (right panel). Difficulty of a case is assessed by the ’cause of death route’ which provides a range of how hard the human experts found it to determine the actual cause of death.
Figure 5: Receiver operating characteristic (ROC) curves showing the classifier performance for ‘hard’ and ‘easy’ cases.

The classification of ‘hard’ cases appears to be a more difficult task and we have yet to produce a classifier with good performance for this purpose.  It could be that we need to engineer additional features from the reviews to successfully predict hard cases. Also, natural language elements, such as negation or hedging terms are notoriously difficult to detect but may improve performance at this task 

Generalising to LeDeR

Our methodology was successfully applied (see figures 6 and 7) to the LeDeR dataset, which contains more verbose reviews and a much larger number of authors. The ability of the method to generalise to this challenging dataset validates our approach and encourages us to further develop the approach for application across new domains in the future.  

Figure 6: Receiver operating characteristic (ROC) curves showing the classifier performance when trained to predict ‘poor practice’ on the LeDeR dataset. Note that this classifier is currently overfitted and additional work is required to produce a classifier that generalises better.
Figure 7: Similar to figure 1, this word cloud shows the most important features for classification of poor healthcare practice in the LeDeR dataset.

Future plans for the project

This exploratory project has identified several avenues for further development. Notably, we have developed Python code that can predict prostate cancer deaths from free-text summaries, and demonstrated its application to a second dataset (LeDeR). This documented code will be shared on GitHub to allow others to apply our methodology to their data.  

Future work will focus on developing use cases for applying this methodology. For instance, gaining understanding of the textual elements that are key to decision making could be used to: 

  1. Provide decision support for identifying prostate cancer deaths or poor healthcare practice, reducing the need for clinical experts to review free-text summaries. 
  1. Identify the important elements of the free-text summary for the authors to target the key sections of the medical history, which would speed up the data collection. 

The ability to accurately predict hard cases would help to allocate reviewer resource more efficiently and so is a strong candidate for further development. We are also keen to produce a dashboard that would allow users to interactively explore their own datasets using the methods we have developed. We are exploring external funding opportunities to continue this project and are writing up our results for publication in a special edition of Frontiers in Digital Health.   

Contact details and links

Dr Emma Turner, Population Health Sciences 

Ms Eleanor Walsh, Population Health Sciences 

Dr Raul Santos-Rodriguez, Engineering Mathematics 

Dr Chris McWilliams, Engineering Mathematics 

Dr Avon Huxor, School of Policy Studies 

Methods derived in part from: FAT-Forensics toolbox 

Jean Golding Institute Seed Corn Funding Scheme

The Jean Golding Institute run an annual seed corn funding scheme and have supported many interdisciplinary projects. Our next round of funding will be in Autumn 2020. Find out more about our Funding opportunities

 

Digital Humanities meets Medieval Financial Records

Blog by Mike Jones, Research Software Engineer in Research IT, University of Bristol

The purpose of this project was to explore the use of ‘Digital Humanities methodologies’ in analysing an English-language translation of a medieval Latin document. We used data analysis tools and techniques to extract financial data and entities (e.g. people, places and communities) from the translation. This process then enabled the creation of example visualisations, to better interpret and understand the data and prompt further research questions. 

 Primary source 

The focus of the project was a single Irish Exchequer receipt roll from the latter years of King Edward I’s reign (1301–2). A receipt roll holds information on the day-to-day financial dealings of the Crown. It provides a rich source of material on not only the machinery of government but also the communities and people that, for various reasons, owed money to the king. An English-language calendar published in the late nineteenth century exists but was found to be deficient. A full English-language edition of the roll was edited by Prof Brendan Smith (Co-I, History) and Dr Paul Dryburgh (The National Archives) and published in the Handbook of Select Calendar of Sources for Medieval Ireland in the National Archives of the United Kingdom (Dublin, Four Courts Press, 2005). The original document is in The National Archives (TNA), London, with the document reference E 101/233/16. 

Transcript to tabular data 

The starting point was the text published in the Handbook of Select Calendar of Sources for Medieval Ireland. A Python script was used to trawl the text, looking for details of interest, namely dates and payments, and add them to a CSV (tabular data) format. Using the Natural Language Toolkit we attempted to extract entities, such as people and places, using an out-of-the-box Parts of Speech (POS) tagger. The results were not perfect, with some places identified as people, but it was an encouraging starting point. 

In the tabular data, each row recorded a payment, including the financial term, date, the geographic location or other entity they are categorised, the value owed to the Irish Exchequer. Payments, recorded in pounds, shillings, pence or marks, were converted to their value in pennies for more straightforward computation. We also checked our computed totals against those calculated by the medieval clerks of the Irish Exchequer these were one penny out, the clerks having missed some fractions of a penny on 24 May 1302! 

Data analysis and visualisations 

With the data in a tabular format, it could be queried with the pandas data library, and visualised with the Matplotlib and Seaborn visualisation libraries. Querying the data, we were now able to create several visualisations, ranging from summary statistics for the financial year, drilling down to monthly, weekly and daily activity. We were also able to visualise the days the Exchequer sat, compared to days it did not sit due to holidays and feast days.  

For example, the total value of the receipts for the financial year was £6159.18s.5d. In the following plot we can break-down the payments into the four financial terms: Michaelmas (September–December), Hilary (January–March), Easter (April–June) and Trinity (June–August), as shown in the chart. 

Other plots highlighted the variability of income, the amount of business (number of transactions), and the number of days the Irish Exchequer sat each term. This is illustrated in the following radar plots, where we plot all three variables – total revenue, length of the term and amount of business – with each variable represented as a percentage of the total value for the year. 

What is immediately striking in these plots is that the Hilary term is relatively long but has the least business and income. In contrast, the Easter term is quite short but provides the most income. These plots confirm what the historians expected – the sheriffs made their proffers to the Exchequer in the Michaelmas and Easter terms and thus were anticipated to be busier. 

Reception and response 

While working on the project, findings and visualisations were shared on social media. This prompted interest and questions from other historians. For example, Dr Richard Cassidy asked (https://twitter.com/rjcassidy/status/1240944622186217472): Was income concentrated in the first week or two of the Michaelmas and Easter terms, from the sheriffs’ adventus, as in the English receipt rolls from the 1250s and 60s?We were able to generate plots that showed in the Irish Exchequer the bulk of the income came in the fourth week and not the second. 

Note: in the tenth week of Michaelmas, the spike in payments against a lower number of transactions is accounted for by Roger Bagot, sheriff of Limerick, returning £76.6s.8d. for the ‘debts of divers persons’; and £100 being returned by William de Cauntone, sheriff of Cork, in forfeited property of felons and fugitives. 

Limitations and research questions 

Clearly there are limits to the analysis, since the project only examined one financial year. It would thus be interesting to analyse trends over time. How does the 1301/2 financial year compare to others in Edward I’s reign? What trends can be seen over the years, decades and centuries? How was the income from Ireland affected by war, rebellion, famine and plague? Are there trends to be gleaned from the different administrations under varying chancellors? Also, does income reflect the changeable effectiveness of English royal authority in Ireland? Can we confirm the ‘decline of lordship’ narratives in the historiography of fourteenth and fifteenth century Ireland?  

Future work 

It is our intention to build on this initial work with the support of external funding. An application has already been made under the AHRC/IRC scheme ‘UK-Ireland Collaboration in the Digital Humanities’ to support a Network to investigate suitable DH approaches to the entire series of Irish receipt rolls, covering the years 1280-1420. Despite being unsuccessful, our application was highly rated and we intend to apply for a major research grant worth up to £1m under the same scheme when details are announced. Furthermore, we are committed to collaborating with Beyond 2022, an ambitious project to create a Virtual Record Treasury of Irish history. Beyond 2022 have commissioned the digitisation of a large number of documents held at The National Archives, London, including the Irish Exchequer receipt rolls. Plans include creating English-language translations of the Irish receipt rolls in TEI/XML, the de facto standard for encoding texts. It will then be possible to construct a pipeline, that builds upon this seed-corn funding work, that results in researchers exploring and formulating research questions around English colonial rule in Ireland and how the Irish interacted with English machinery of government. 

Further Details 

More detailed information about the project can be found in a series of blog posts, and the source code and data are available on GitHub

Jean Golding Institute Seed Corn Funding Scheme

The Jean Golding Institute run an annual seed corn funding scheme and have supported many interdisciplinary projects. Our next round of funding will be in Autumn 2020. Find out more about our Funding opportunities

Decoding pain: real-time data visualisation of human pain nerve activity

Blog post by Manuel Martinez, Research Software Engineer and Dr Jim Dunham, Clinical Lecturer, from the School of Physiology, Pharmacology and Neuroscience at the University of Bristol

We are developing new tools to analyse human pain nerve activity in real time. This will aid diagnosis in chronic pain and enable individualised, targeted treatments.

Some patients with chronic pain have abnormally increased activity in their “pain detecting” nerves (nociceptors). We do not know which patients have this problem and which do not. If we could determine which individuals suffer with these ‘sensitised’ nociceptors, we could treat them more effectively, by giving medicines to ‘quieten’ their nerves.

We record from human nociceptors using a technique called microneurography. Sadly, this technique is only used in research as it is too time consuming and unreliable to use clinically. To bring microneurography closer to the clinic we sought to:

Improve Real-time Data Visualisation

  • Improve the way real-time neural data is displayed by replacing a legacy oscilloscope-like trace with a 4D ‘smart’ visualiser.

Close the Loop

  • Develop and implement automated real-time robust spike detection algorithms.
  • Develop and implement closed-loop dynamic thresholding algorithms to automatically control the electrical stimulus energy.

These developments have the potential to significantly increase experimental efficiency and data yields.

Figure 1 Conceptual set-up for a closed-loop experiment. An electrical stimulus of a predefined intensity is applied to the skin (A). If the stimulus intensity is large enough, the nerve will fire and send a “spike” of activity towards the brain. The electrical activity of the nerve is recorded “upstream” at some distance away from the stimulation site (B). These spikes are digitised and processed in a computer (C) so that they can be visualised in real time to aid in electrode placement. The resulting recordings can be exported for further analysis in third-party software tools (D).

Real-time Data Visualisation

Microneurography allows for nerve activity to be recorded by means of a fine electrode inserted through the skin into the nerve. After insertion into the nerve, the skin supplied by that nerve is electrically stimulated to cause activity in the nociceptors (Figure 1). Recording this activity is difficult; it requires careful positioning of the electrode and is further complicated by the small amplitude of the nerve signal in comparison to noise.

Figure 2A shows the legacy oscilloscope-like visualiser commonly used in microneurography. The signal trace represents the voltage measured in the recording electrode as a function of time. The evoked neural spikes are indicated by green arrows. The large spikes (indicated by the red lighting symbol) correspond to a signal artefact caused by the electrical stimulation system.

“Pain” nerves conduct slowly and therefore have characteristically long latencies. These latencies show good correlation between successive firings. Therefore, accurate electrode placement can be verified by the presence of consecutive spikes of similar latency after the stimulus event.

Figure 2B shows our novel 4D visualiser. Here, the signal amplitude is encoded via colour, with lighter colours representing high amplitudes. This colour scaling can be adjusted in real time by the user. The vertical axis corresponds to latency after the stimulus event and the horizontal axis to a series of stimulus events. Therefore, a constant latency spike manifests itself as a line in this visualiser.

This is a significant improvement over the legacy visualiser as the subtle changes in colour and the alignment between two consecutive spikes can be readily identified by eye in real time. This greatly increases the clinician’s situational awareness and contributes to maximising experimental yield.

Figure 2 Microneurography data recording from the superficial peroneal nerve as seen in the legacy oscilloscope-like visualiser (A) and the novel 4D latency visualiser (B-C). Two units of similar latency can be readily identified and have been indicated with green arrows. A possible third unit at a longer latency has been indicated with a dotted arrow. This third unit is only noticeable in the 4D visualiser as it is below the noise level in the oscilloscope trace.

Closed-loop stimulation control

The electrical energies required to evoke nociceptor activity are not constant. These changes in electrical ‘threshold’ may be useful in understanding why patients’ nerves are abnormally excitable. Unfortunately, balancing signal detection against stimulation energy in the context of real time analysis of small amplitude signals is difficult and primed for failure.

To improve reliability and reproducibility, we have developed a dynamic thresholding algorithm that automatically controls stimulation energy once a unit has been manually identified (i.e. a line can be seen in the visualiser). This is conceptually simple: decrease the stimulation energy until the unit ceases to fire, then increase it until it starts firing again.

In practice, the robust detection of spikes is challenging as existing approaches are only successful in environments with high signal-to-noise ratios (SNRs). To address this, our proof-of-concept algorithm first takes a set of candidate spikes (obtained using a simple threshold crossing method – green points in Figure 2C). Then, these candidate spikes are temporally (latency) filtered so that only those around a small region of interest near the detected track remain. This detection algorithm, despite its simplicity, has shown promising performance on pre-recorded and simulated data and is now ready for testing in microneurography.

Revolutionising human microneurography

We seek to revolutionise human microneurography: bringing it into the clinic as a diagnostic tool; informing treatment decisions and demonstrating ‘on target’ efficacy of new analgesics.

The novel 4D visualiser and automated closed-loop experimental tools developed here will be validated in microneurography experiments in healthy volunteers and then made publicly available in the spirit of open-source research. Additionally, we will integrate more advanced methods of ‘spike’ detection into the algorithm to maximise sensitivity and specificity.

We anticipate our first patient trials of these novel tools within the next 12 months. Our visualiser will enable rapid identification of abnormal activity in nociceptors, paving the way towards data-driven, personalised treatments for patients living with chronic pain.

Contacts and Links

Mr Manuel Martinez Perez (Research Software Engineer, School of Physiology, Pharmacology & Neuroscience)

Dr Jim Dunham (Clinical Lecturer, School of Physiology, Pharmacology & Neuroscience)

Dr Gethin Williams (Research Computing Manager, IT Services)

Dr Anna Sales (Research Associate, School of Physiology, Pharmacology & Neuroscience)

Mr Aidan Nickerson (PhD student, School of Physiology, Pharmacology & Neuroscience)

Prof Nathan Lepora (Professor of Robotics and AI, Department of Engineering Mathematics)

Prof Tony Pickering (Professor of Neuroscience and Anaesthesia, School of Physiology, Pharmacology & Neuroscience)

Jean Golding Institute Seed Corn Funding Scheme

The Jean Golding Institute run an annual seed corn funding scheme and have supported many interdisciplinary projects. Our next round of funding will be in Autumn 2020. Find out more about our Funding opportunities

Mood music – Inferring wellbeing from Spotify

Photo by Morning Brew on Unsplash

Does what you listen to reflect how you feel? And if it did, what would you think about using your music history to track your mood?  

Blog post by Nina Di Cara, PhD researcher, Population Health Sciences, University of Bristol

Our research group, the Dynamic Genetics Lab, previously looked at whether what we do and what we say on social media can be used to measure mood and wellbeing. This Seedcorn Grant, from the Jean Golding Institute, has given us the opportunity to look at the feasibility of a different medium – music streaming.  

Aims

  • Recruit a focus group of students to discuss the acceptability of tracking mood through music streaming behaviours 
  • Build an opensource software infrastructure to collect Spotify data from consenting participants, alongside tracking their mood through frequent mood questionnaires.  
  • Conduct a pilot study to understand whether music listening behaviours were predictive of mood.  

Establishing non-questionnaire measures of mood and wellbeing, especially those that allow us to track mood longitudinally, has many potential benefits. It means that understanding wellbeing does not need to rely on participants trying to remember how they have felt for the past several weeks or months. Continuous non-intrusive measurement of mood could also help identify patterns in response to external events at a personal or population level. These methods could also make it easier for people to track their own moods, and to share recent patterns with mental health professionals. Of course, with new technology like this it is always incredibly important to pair it with an in depth understanding of people’s views on the limitations and acceptability of its development and use. 

Bespoke software for novel application of mental health data science

When the project started in January 2020 we were really excited to get going – the study was a chance to integrate qualitative and quantitative research and build our own bespoke software for this novel application of mental health data science 

The software we are building is a platform that will allow participants to sign up, complete baseline questionnaires, connect their Spotify account and will collect the Spotify data they have agreed to. It will also send them a brief questionnaire several times a day for two weeks that will ask them to report how they are feeling. At the end we will have the two weeks of music listening data from Spotify alongside these mood reports to analyse. As the software will be open source it may even be of use to other researchers, as well as being used for future studies of our own. 

By February we had successfully navigated the first stage of ethical approval and recruited our participant focus group. A few weeks later we held the first of our five planned focus groups, where participants spoke about the acceptability of using music listening data in academic research and compared it to other types of data commonly collected for epidemiological research. The participants also shared their thoughts on how their music listening patterns may, or may not, be indicative of how they feel.  

An opportunity to refine plans

Sadly, just as we were getting started, COVID-19 arrived in the UK. Our focus groups were suspended, and we decided that it would not be ethical to conduct research which requires frequent introspection at a time that a lot of people were struggling to get to grips with lockdown.  

We are pleased to say that now life is starting to return to normal we are able to pick-up where we left off, with a few adjustments! We will be re-starting our focus groups online, and looking to run our pilot study in the Autumn when students return 

Having delays isn’t all bad though – it has meant we have more time to get feedback from other researchers, and more time to spend getting our software right. This should mean that when we do go ahead the study design and tools will have benefitted from those few extra months of refinement!  

How to get involved

If you are a student who is interested in taking part in a focus group, or taking part in the pilot study later this year, get in touch with Nina (nina.dicara@bristol.ac.uk) to receive updates when we start recruiting.  

Jean Golding Institute Seed Corn Funding Scheme

The Jean Golding Institute run an annual seed corn funding scheme and have supported many interdisciplinary projects. Our next round of funding will be in Autumn 2020. Find out more about our Funding opportunities

Building capacity for big data management for Ghana’s developing economy

A Science and Technology Facilities Impact Award (STFC IAA) won by a team from the Physics department in collaboration with the start-up iDAM and facilitated by the JGI will provide hardware and software facilities for high volume data storage and archiving, processing, visualisation, algorithm development and testing for research in academia and industry in Ghana, contributing to the development of data science and digital innovation capability in the country.

Emmanuel Bempong-Manful, Henning Flaecher, Johannes Allotey, Kate Robson Brown

The team (Dr Henning Flaecher, Prof Kate Robson Brown and Prof Mark Birkinshaw) will develop a collaboration with local partners, the Ghana Radio Astronomy Observatory (GRAO), the Development in Africa with Radio Astronomy (DARA) project, and iDAM, a local start-up founded by two Bristol PhD students, Johannes Allotey and Emmanuel Bempong-Manful.

The Government of Ghana is embarking on the digitisation of several areas of the economy, including the passport office, ports and harbours and the energy sector, with the aim to improve services and revenue collection. These developing digital services together with those still to be implemented (e.g., in the digitisation of national IDs, health records, birth and death registry) will produce an enormous volume of sensitive data that requires efficient storage and management. However, despite the looming data volumes and recent advancement in statistical and machine learning techniques for inference and predictive analysis, these techniques are still under-utilised in Ghana.

As the economy grows and evolves through digitisation, and as data volumes increase, these data science solutions will become increasingly useful for quick, efficient and reliable extraction and evaluation of information from the datasets and to support evidence-based predictions.  As a result, there is an urgent need to develop facilities and a skills-based workforce in data analytics that will be capable of manipulating big datasets to make meaningful contributions to the Ghanaian economy. However, these goals can only be achieved if modern computing infrastructure/ hardware and software solutions are available.

This STFC funded project will lay the foundation of a modern computing facility which will be hosted by GRAO and iDAM to provide the technical support and capacity building activities. iDAM has long-term plans to establish a one-stop data management hub to tackle data challenges in Ghana and is currently working with the Ghana Space Science and Technology Institute (GSSTI) and the DARA project to deliver data curation services.

Kakum Park, Nkrumah Museum, GRAO Observatory

This project will address a major societal challenge in the area of big data management in Ghana and aims to contribute to the Sustainable Development Goals (SDGs) through skills development programmes in data management and data science boosting new careers and economic growth and delivering quality data management services to the people of Ghana. The project will share regular updates via the JGI blog. If you would like to know more about this project, and would like to collaborate, please contact us via jgi-admin@bristol.ac.uk.