Here's how to avoid 3 weeks of pain

Pro tip, make sure you know what Kamal is doing behind the scenes to avoid a lot of pain…

For a recent project I was using solid_queue for background jobs and Kamal for deployment. Everything worked perfectly—until it didn’t.

Some of my jobs, Turbo broadcasts, would run just fine, but my custom jobs and notifications using Noticed were throwing NameErrors. Strangely, when I connected to the jobs server, I could run them manually without issues.

I’ll save you a lot of pain and suffering and jump to the solution to the problem. Turns out, an old container had been running for two months with outdated code, missing the new job classes. 🤦🏻‍♂️

This was caused when I switched from a single deploy.yml to using different destinations for staging and production, and not knowing exactly what Kamal was doing with that behind the scenes.

Container Names

When deploying with Kamal, you have a project name, a role, a git hash, and an optional destination. The code probably looks something like:

container_name = [
	project_name,
	destination,
	role,
	git_hash
].compact_blank.join("-")

In my original deploy.yml situation, this would make a container named PROJECT-web-abc123. After setting up staging, this would be PROJECT-staging-web-abc123. Kamal automatically shuts down and cleans up old containers, but it uses the name of the container to do that. This means my old, non-staging container, dutifully kept running.

The Resolution

Once I realized what was going wrong the resolution was quick and easy. I could connect to the server directly with kamal shell -d staging and see the list of containers with docker container ls:

9fe40e3246c3 ... PROJECT-web-staging
a776847b129c ... PROJECT-job-staging
8e171eede8ed ... PROJECT-web			<-- this one

Once the old container is identified, a single docker stop 8e171eede8ed will shut it down and resolve all your NameError woes.

PS Turbo broadcasts worked fine because they were part of the project from the beginning, meaning they existed in both the old and new codebases. But the new jobs and notifications didn’t exist in the outdated container, leading to NameErrors.