Case Study: Improving AWS AI Summarization with Multiprompt Architectures
Introduction
The healthcare AI industry is known for extreme competition in terms of accuracy. Amazon Web Services Health (AWS Health) is a leader in pioneering new techniques to bring new performance thresholds to all aspects of Healthcare AI, leading to its status as one of the leaders in the space.
One of AWS Health’s most successful suite of products is its transcription and summarization services suite. These services allow developers and providers alike to put structure to unstructured data. In its spirit of continuous innovation, AWS Health is constantly looking for new opportunities for these systems to provide new value to healthcare providers and improve the lives of their patients.
Problem
Summarizing a transcribed conversation can be complex, particularly when the transcribed conversation is long and information-dense. Medical documentation derived from these conversation requires a high degree of factual accuracy, without which providers may not be adequately compensated for the services they render to their patient panels.
Providers can also have strong and specific preferences regarding the structure and style of their documentation. Since providers use this documentation as a recall device, ensuring continuity of care for their patients across many visits and accommodating these style preferences are significant.
AWS Health and its competition must strive to meet these performance criteria and turn results around quickly, all while maintaining a cost-effective business model.
Action Steps
AWS partnered with the Earshot Health team to explore techniques that could be used to improve the performance of their medical summarization systems.
The team at Earshot Health platform features an agentic architecture that utilizes a Tree of Thought (ToT) structure to:
- Capture a wide range of complex medical topics
- Assemble those topics into highly accurate documentation
- Do so for a range of medical specialties without creating a great deal of overhead
- Account for varying per-use documentation output requirements
AWS partnered with the Earshot Health team to see how the techniques used in Earshot’s documentation generation tools could impact the performance of their summarization technology. To do so, we followed the following steps:
- AWS identified a few key medical specialties they wanted to evaluate
- Earshot developed a medical summarization API, built on AWS infrastructure, using our experience in developing medical summarization systems
- Earshot tuned our technique to each medical specialty, partnering with our network of expert providers to ensure accuracy
- The AWS team was given access to the system for further evaluation
Results
Earshot’s experts reviewed the produced notes, evaluating that the notes met our exacting standards along the following criteria:
- Factual accuracy - the extent to which the facts captured by the system match the actual facts of the case
- Stylistic accuracy - the extent to which the produced notes’ structure and style match with the example notes we were given
- Linguistic brevity - the efficiency with which the style and facts of the case are represented
As a result of the work we produced, the AWS team decided to expand the evaluation process, having their team perform a wider evaluation on a much larger set of patients. These underlying techniques have since been incorporated into many of AWS Health’s products.
Call To Action
Earshot Health is a best-in-class suite of AI tools for Care Management teams. Our robust set of features, including our documentation generation suite, are fueled by an expert team of pioneering AI Engineers. If you’re ready to bring cutting-edge AI technology to bear to transform your Care Management workflows, let’s have a conversation.