Local AI: User Experience Improvements

One last tweak to our system should help users have just a bit more insight into what the AI system is doing and

Our system now streams in responses and allows configurable system prompts, but it’s missing an affordance to indicate that the AI is done. Fortunately, we can add that. We will add a boolean done field to the message models. On AI responses, it will start false and will be updated. We can use that to provide visible feedback to the user:

rails g migration add_done_to_messages done:boolean!

<%= tag.div id: dom_id(message), class: [ "p-2 max-w-4xl", "border-2 border-gray-400": !message.done ] do %>
  <div class="font-semibold"><%= message.role %>:</div>
  <%= message.to_markdown %>
  <%= tag.div "\"Thinking\" ..." unless message.done %>
<% end %>

This will give a border around the partial message and display “Thinking …” until complete. To wire it in, we’ll update the done field while streaming in the messages:

# app/models/conversation.rb
class Conversation < ApplicationRecord
	def send_thread
		# collect the preceding messages
		messages = self.messages.order(:created_at).map { {
			role: _1.role,
			content: _1.message
		} }
		
		# create the blank initial response
		response = self.messages.create({
			role: "assistant",
			message: "",
			done: false # 1
		})
		
		ollama_client.chat({
			model: "llama3",
			stream: true,
			messages: messages
		}) do |event, raw|
			response.message += event["message"]["content"]
			response.done = event["done"] # 2
			response.save
		end
	end
end

We start with an incomplete message (#1), and updated it every time we get a response from the AI (#2). When the final response comes from the AI, the Turbo Stream broadcast will include the updated message, and the updated styling.

And with that, we’re able to clearly show the user when a message is pending or complete.