Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Healthcare giant Optum has restricted access to its internal AI chatbot used by employees after a security researcher discovered it was publicly available online and anyone could access it using just a web browser.
The chatbot, seen by TechCrunch, allowed employees to ask the company questions about how to resolve patient health insurance claims and disputes for members, according to the company’s standard operating procedures (SOP).
While the chatbot does not contain sensitive personal or protected health information, its accidental exposure comes as its parent company, health insurance conglomerate UnitedHealth, faces scrutiny over its use of artificial intelligence tools and algorithms. ostensibly overrules doctors’ medical decisions and dismisses patient claims.
Mossab Hussein, chief security officer and co-founder of cybersecurity firm spiderSilk, warned TechCrunch about a publicly released internal Optum chatbot called “SOP Chatbot.” Although the tool was hosted on an internal Optum domain and could not be accessed from its web address, its IP address was public and accessible from the Internet and did not require users to enter a password.
It is not known how long the chatbot has been publicly available on the Internet. The AI chatbot could not be accessed online after TechCrunch reached out to Optum for comment on Thursday.
Optum spokesman Andrew Krejci told TechCrunch that Optum’s SOP chatbot was “a demo tool designed as a potential proof of concept” but “never went into production and is no longer available on the site.”
“The demo was designed to test how the tool answered questions on a small sample set of SOP documents,” the spokesperson said. The company has confirmed that there is no protected health information used in the bot or its training.
“This tool doesn’t make any decisions and never will, but it gives better access to existing SOPs. In short, this technology has never been scaled or used in any real way,” said the spokesperson.
AI chatbots like Optum are typically designed to produce responses based on the data the chatbot has been trained on. In this case, the chatbot is trained on internal Optum documents related to SOPs for handling certain claims, which can help Optum employees answer questions about claims and eligibility for payment. The Optum documents were hosted on UnitedHealthcare’s corporate network and were not accessible without employee access, but were referenced and referred to by the chatbot when asked about their content.
According to statistics displayed on the chatbot’s main dashboard, Optum employees have used the SOP Chatbot hundreds of times since September. The chatbot also kept a history of hundreds of conversations Optum employees had with the chatbot during that time. Chat history shows that Optum employees asked the chatbot, “What should the claim definition be?” they will ask things like and “How do I check the policy renewal date?”
TechCrunch reports that some of the documents the chatbot is referring to include dispute process management and compliance checks. The chat bot also developed answers that typically show when asked the reasons for rejecting coverage.
Like many AI models, Optum’s chatbot was able to respond to questions and prompts outside of the documents it was trained on. Some Optum employees were intrigued by the chatbot, prompting the bot with requests such as “Tell me a joke about cats” (which it rejected: “No joke”). The chat history also showed several attempts by employees:jailbreak” by forcing the chatbot to produce responses unrelated to the chatbot’s training data.
When TechCrunch asked the chatbot to “write a poem about denying the claim,” the chatbot produced a seven-paragraph paragraph that read in part:
In the large domain of healthcare
Where policies and regulations often restrict
The claim comes, he demands his right
But unfortunately, his fate is to reconcile.The supplier earnestly hopes,
For payment in service frenzy,
Although research reveals the tale,
And the reasons for denial prevail.
UnitedHealth Group, which owns Optum and UnitedHealthcare, is facing criticism and legal action for using artificial intelligence to deny patient claims. since targeted killing In early December, UnitedHealthcare CEO Brian Thompson, news outlets reported a flood of reports from patients expressing anguish and frustration over denials medical coverage by the health insurance giant.
The conglomerate, the largest private health insurance provider in the United States, was sued earlier this year for denying critical health insurance to patients who had lost access to health care. Citing a STAT News investigation. The federal lawsuit accuses UnitedHealthcare of using an AI model with a 90% error rate “in lieu of real medical professionals to unfairly deny care to elderly patients.” UnitedHealthcare, for its part, said it would defend itself in court.
UnitedHealth Group had $22 billion in profits in 2023 on $371 billion in revenue.
