We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Here is the conclusion, which you may find somewhat chilling when you recall ChatGPT’s propensity for bullshitting based on things it doesn’t understand and not offering any indication of confidence level when making definitive proouncements.
As AI becomes increasingly proficient, it will soon become ubiquitous, transforming clinical medicine across all healthcare sectors. Investigation of AI has now entered into the era of randomized controlled trials. Additionally, a profusion of pragmatic and observational studies supports a versatile role of AI in virtually all medical disciplines and specialities by improving risk assessment, operational efficiency, and patient communication.
Inspired by the remarkable performance of ChatGPT on the USMLE, clinicians within AnsibleHealth, a virtual chronic pulmonary disease clinic, have begun to experiment with ChatGPT as part of their workflows. Inputting queries in a secure and de-identified manner, our clinicians request ChatGPT to assist with traditionally onerous writing tasks such as composing appeal letters to payors, simplifying radiology reports (and other jargon-dense records) to facilitate patient comprehension, and even to brainstorm freely in a bid to kindle insight when faced with nebulous and diagnostically challenging cases. Overall, our clinicians reported a 33% decrease (future publication) in the time required to complete documentation and indirect patient care tasks. We believe this is an early but important signal that LLMs such as ChatGPT are reaching a maturity level that will soon impact clinical care at large and its ability to deliver truly individualized, compassionate, and scalable healthcare.
Yes, IBM Watson was also supposed to revolutionize medicine and blah blah blah. It managed to waste a couple of hundred million dollars and not produce anything of substance.
Dr. Chat GPT will surely be different because… this time it’s different?
ChatGPT actually searches the web to collect the information to synthesize the replies. That it works is impressive, and the flaw is perhaps not with ChatGPT, it’s with the quality of information online. As a result, we get GIGO: garbage in, garbage out.
The web contains lots of information and disinformation, there’s no reputational accountability, and as a result many of the ‘flaws’ of ChatGPT are simply flaws of a biased and partisan web, and of biased and partisan web search engines.
There’s also new ways to debug knowledge kept within a LLM:
GlacierMD burned one of my 3 free questions (does a vtach ablation increase my life expectancy?) saying it didn’t know,admittedly I was trying something difficult. I guess that’s better than Dr. Google coughing up 10 pages of miracle mystery herbal remedies and homeopathic cures
Just one of Mr. Hutchinson’s excellent insights:
“However, there is a deeper question of how the ancien regime will be able to maintain its current structure given that many of its favorite activities for status signaling (art and writing) are being automated first, while despised blue-collar activities like truck driving remain resistant.”
Physics professor Philip Moriarity, stunned by the results when one of his students showed him ChatGPT’s solution to a problem from his second year quantum mechanics course, signed up for an account and tried submitting a variety of problems, mostly in physics, from examinations ranging from the British GCSE (high school qualification), A-Level (university entrance qualification), and university undergraduate courses. He examines the results, what they mean for students and teachers, and how (at least for the moment), instructors can mitigate the threat of ChatGPT in his blog post, “ChatGPT: Transforming Physics Education?”, and demonstrates ChatGPT attempting to solve the problems in the following Sixty Symbols video.
Frohlich coherence refers to the concept that biological systems can exhibit long-range coherence or order in their behavior and structure as a result of the interaction of electromagnetic fields with biological molecules. The theory was first proposed by physicist Herbert Frohlich in the 1970s.
Frohlich proposed that biological systems, such as living cells, are able to maintain a state of long-range coherence by means of a process he called “superradiance.” In this process, the electromagnetic fields of biological molecules interact with each other in such a way as to create a collective behavior that is not present in individual molecules. This collective behavior is said to be responsible for the stability and function of biological systems.
The theory of Frohlich coherence is still considered as a hypothesis and it is not widely accepted by the scientific community. There is a limited amount of experimental evidence to support it and several criticisms have been made against it, such as the lack of direct observation of superradiance in biological systems.
Canadian radio amateur William Franzin, VE4VR, based on a suggestion by two U.S. amateurs in Seattle, built an interface that receives digital voice transmissions in the D-STAR protocol, processes them with a voice recognition engine to create text, submits the text as a prompt to ChatGPT, encodes the result with a text to speech tool, then transmits the audio back to the sender.
In the demonstration, ChatGPT is asked, over the radio, “What is the easiest way to get an amateur radio license?”.
Recently, a client reached out to Asion with a problem: the woman had moved into a pre-construction home and couldn’t open her windows. She had attempted to contact the developer for months with no response. Asion ran a copy of one of her emails through ChatGPT, asking it to rewrite it with an emphasis on the liability implications.
“ChatGPT wrote it as a legal issue and all of a sudden, the developer showed up at her house,” he said.
pretty good, though not sure if “super radiance” is correct in that context.
