clinFHIR does AI

I thought I’d see how useful the current AI models are in producing a FHIR bundle of resources that represents some – likely clinical – scenario to help those learning FHIR. I wanted to have a conversational interaction so that the bundles created can be viewed and refined, and a way for the AI to explain the choices it made when processing the input prompt and generating the bundle. The resulting bundle would be validated using the FHIR Validator.

TLDR: They are surprisingly good!

I chose the Patient Viewer for the initial module to integrate as I suspect that it would be most useful for those learning FHIR – particularly with a clinical background – as the UI is quite straightforward.

After loading Patient Viewer, the AI is invoked by clicking the green ‘AI Assistant’ button in the upper right hand corner of the Patient Viewer module as shown in the following screenshot.

The AI Assistant works in a similar way to a standard AI chatbot. It allows you to have a conversation with the AI where you start by asking it to create a bundle based on some prompt that you supply, review the bundle in the Viewer and then ask for changes.

You can have multiple conversations and switch back to previous ones if you wish. (The conversations are saved in the local browser cache so are only accessible from there).

It is specifically intended for the assistant to produce bundles, and each call includes background instructions (the ‘system’ prompt) that defines the operation and also the structure of the responses back to the Viewer. This is needed to be able to separate the bundle from the reasoning when displaying in the UI.

Clicking on the Assistant button results in the following dialog:

If you are in the middle of a conversation then previous prompts and responses will be shown.

The prompt box is where you enter the clinical request. Enter a description of the scenario you want to represent. For example:

A 5 year old patient complaining of a sore left ear for 3 days. Normal temperature. On examination an inflamed ear drum, dx Otitis media. Prescribed amoxil for 3 days.

Note that you don’t need to specify that you want a bundle, FHIR version or any other details – that is specified in the system prompt as described above.

Next, select the model you want the AI to use. The following claude models are supported:

• Haiku 5.5 This is the most basic model, but quite fast and surprisingly accurate in my somewhat limited testing. This is the default.
• Sonnet 5.5 More advanced, but slower to execute
• Opus 5.5 The most advanced, but slower and more expensive to run.

Now, click the ‘Invoke AI’ link just under the prompt. You’ll see a ‘please wait’ dialog in the right pane, then after some seconds you’ll see something like this:

What happened is that your prompt was sent to the AI, along with some background instructions (the ‘system prompt’) for the AI to create the bundle using FHIR R4 and return the result along with a description of its reasoning as it did so.

A summary of what it did – what its reasoning was –  appears alongside your prompt along with 2 links.

• ‘Load this bundle’ will load the bundle into the Viewer where it can be examined in the usual way. After viewing the bundle click the ‘AI Assistant’ button again to return to the dialog and request any changes.
• ‘Toggle details’ will display more information about the reasoning the model used when creating the Bundle and any warnings. It also shows some technical details like the time taken and the ‘cost’ in terms of tokens.

To view the bundle, click the ‘Load this bundle. ’ Here’s what the graph looks like in the Viewer.

Interestingly, the first response from the AI was a single Patient only – despite it saying it had created more.

I needed to re-prompt it a couple of times to get the result above. Here’s the conversation:

To investigate further, I created a new conversation (using the ‘Start new Conversation’ button) and used the same prompt and this time it got it right the first time!  It shows quite nicely some of the issues in dealing with an AI especially that the same instructions can return a different answer.

This highlights that it’s important to remember that the AI can generate a different bundle each time it runs even if the prompt is the same. The actual bundle created at each step is saved in the conversation history and can be reviewed any time.

If you want to copy the bundle – say to paste it into the Bundle Viewer for the more technical renderings then there’s a link on the Json tab that will copy it onto the clipboard so you can paste it into a file – or into the Bundle Viewer.

When the api returns the bundle, it is validated using the validation server – currently set to the public HAPI server using the $validate operation. If there are any errors or warnings, then a link appears below the response as shown below. Clicking on the link shows the details.

So far I haven’t had any structural errors from the AI – most are warnings like missing CodeSystems (though I’m surprised that LOINC isn’t loaded in HAPI – maybe someone deleted it…)

As stated above, you can have multiple conversations. Start a new one with the ‘Start new Conversation’ button. You don’t need to save the current conversation – that happens automatically.

Here’s a screenshot of the history of conversations from some of my testing with the ‘confused’ consultation conversation highlighted.

As you can see, you are able to re-load any of the previous bundles created and view the model reasoning at each step of the conversation. You can also make a previous conversation the current one (using the ‘Make current’ link at the upper right) and carry on editing. The history is saved in the local browser – only accessible from there.

So this is a ‘first step’ at integrating AI into clinFHIR and shows some of the challenges in using it. If you do decide to experiment with it I’d be interested in your comments and findings – use the FHIR chat for feedback.