Local AI: Improving response time

The big problem we have with our current AI implementation is that is slow. When the user submits a new message they have to wait for entire response before it shows up. Fortunately, Turbo gives us an excellent tool to improve the situation.

We will be changing the implementation to queue a job to get the AI response and return from the controller immediately.

1. Conversation Update Job

We will remove the inline call to update the conversation and move it into a background job:

# app/controllers/messages_controller.rb
class MessagesController < ApplicationController
	def create
		# …
		
		# @conversation.send_thread
		UpdateConversationJob.perform_later(@conversation)
		redirect_to @conversation
	end
end

# app/jobs/update_conversation_job.rb
class UpdateConversationJob < ApplicationJob
	def perform(conversation)
		conversation.send_thread
	end
end

This will queue up the job in the background, but require us to refresh the page manually to see the updated content.

2. Automatically Refresh Content

This is a perfect situation to apply Turbo to. With Turbo, we can monitor for new messages, and automatically add them to the page. In a default Rails app Solid Queue/Cable give us what we need out of the box, otherwise you’ll need to setup something comparable.

There are two changes we need to make:

1. Subscribe to update on the frontend
2. Send updates from the backend

1. Frontend Subscription

On the frontend we need to give an ID to the dom element that we will be pushing updates into. By convention this will be #messages. Additionally we need to setup a channel that we will be receiving updates on.

We could subscribe to each individual message, but it makes more sense to just subscribe to update on the conversation itself:

<!-- app/views/conversations/show.html.erb -->
<!-- title, form etc -->

<div id="messages" class="divide-y-2 mt-8">
  <%= render @conversation.messages.order(created_at: :desc) %>
</div>
<%= turbo_stream_from @conversation %>

2. Backend Updates

On the backend, we need to tell the Message model to send out a notification whenever it is created or updated:

class Message < ApplicationRecord
  broadcasts_to :conversation, inserts_by: :prepend
end

By broadcasting to the conversation, we use that association as the channel identifier. Additionally, because we are listing messages reverse chronologically, we want to prepend, not append, new messages.

And that’s it. Messages now show up automatically, no manual refresh needed. The only major issue still present is that the AI response shows up as one big message as opposed to coming in big by bit.

Guess we’ll have to address that tomorrow…