While a medical student at UC San Francisco, Dereck Paul grew concerned that innovation in medical software was lagging behind other sectors, like finance and aerospace. He came to believe that patients are best served when doctors are equipped with software that reflects the cutting edge of tech, and dreamed of starting a company that’d prioritize the needs of patients and doctors over hospital administrators or insurance companies.

So Paul teamed up with Graham Ramsey, his friend and an engineer at Modern Fertility, the women’s health tech company, to launch Glass Health in 2021. Glass Health provides a notebook physicians can use to store, organize and share their approaches for diagnosing and treating conditions throughout their careers; Ramsey describes it as a “personal knowledgement management system” for learning and practicing medicine.“During the pandemic, Ramsey and I witnessed the overwhelming burdens on our healthcare system and the worsening crisis of healthcare provider burnout,” Paul said. “I experienced provider burnout firsthand as a medical student on hospital rotations and later as an internal medicine resident physician at Brigham and Women’s Hospital. Our empathy for frontline providers catalyzed us to create a company committed to fully leveraging technology to improve the practice of medicine.”

Glass Health gained early traction on social media, particularly X (formerly Twitter), among physicians, nurses and physicians-in-training, and this translated to the company’s first funding tranche, a $1.5 million pre-seed round led by Breyer Capital in 2022. Glass Health was then accepted into Y Combinator’s Winter 2023 batch. But early this year, Paul and Ramsey decided to pivot the company to generative AI — embracing the growing trend. Glass Health now offers an AI tool powered by a large language model (LLM), tech similar to that behind OpenAI’s ChatGPT, to generate diagnoses and “evidence-based” treatment options to consider for patients. Physicians can type in descriptions like “71-year-old male with a history of myocardial infarction presents with subacute progressive dyspnea on exertion” or “65-year-old women with a history of diabetes and hyperlipidemia presents with acute-onset chest pain and diaphoresis,” and Glass Health’s AI will provide a likely prognosis and clinical plan.

“Clinicians enter a patient summary, also known as a problem representation, that describes the relevant demographics, past medical history, signs and symptoms and descriptions of laboratory and radiology findings related to a patient’s presentation — the information they might use to present a patient to another clinician,” Paul explained. “Glass analyzes the patient summary and recommends five to 10 diagnoses that the clinician may want to consider and further investigate.”Glass Health can also draft a case assessment paragraph for clinicians to review, which includes explanations about possibly relevant diagnostic studies. These explanations can be edited and used for clinical notes and records down the line, or shared with the wider Glass Health community.

In theory, Glass Health’s tool seems incredibly useful. But even cutting-edge LLMs have proven to be exceptionally bad at giving health advice.

Babylon Health, an AI startup backed by the U.K.’s National Health Service, has found itself under repeated scrutiny for making claims that its disease-diagnosing tech can perform better than doctors.

In an ill-fated experiment, the National Eating Disorders Association (NEDA) launched a chatbot in partnership with AI startup Cass to provide support to people suffering from eating disorders. As a result of a generative AI systems upgrade, Cass began to parrot harmful “diet culture” suggestions like calorie restriction, leading NEDA to shut down the tool.Elsewhere, Health News recently recruited a medical professional to evaluate the soundness of ChatGPT’s health advice in a wide range of subjects. The expert found that the chatbot missed recent studies, made misleading statements (like that “wine might prevent cancer” and that prostate cancer screenings should be based on “personal values”) and plagiarized content from several health news sources.

In a more charitable piece in Stat News, a research assistant and two Harvard professors found that ChatGPT listed the correct diagnosis (within the top three options) in 39 of 45 different vignettes. But the researchers caveated that the vignettes were the kind typically used to test medical students and might not reflect how people — particularly those for whom English is a second language — describe their symptoms in the real world.It hasn’t been studied much. But I wonder if bias, too, could play a role in causing an LLM to incorrectly diagnose a patient. Because medical LLMs like Glass’ are often trained on health records, which only show what doctors and nurses notice (and only in patients who can afford to see them), they could have dangerous blind spots that aren’t immediately apparent. Moreover, doctors may unwittingly encode their own racial, gender or socioeconomic biases in writing the records LLMs train on, leading the models to prioritize certain demographics over others.

Paul seemed well aware of the scrutiny surrounding generative AI in medicine — and asserted that Glass Health’s AI is superior to many of the solutions already on the market.“Glass connects LLMs with clinical guidelines that are created and peer-reviewed by our academic physician team,” he said. “Our physician team members are from major academic medical centers around the country and work part-time for Glass Health, like they would for a medical journal, creating guidelines for our AI and fine-tuning our AI’s ability to follow them … We ask our clinician users to supervise all of our our LLM application’s outputs closely, treating it like an assistant that can offer helpful recommendations and options that they may consider but never replaces directs or replaces their clinical judgment.”