Local AI: Getting up and running

I don’t know if you’ve heard, but there’s this new tech that people are starting to talk about called AI. I think it might have some potential…

The easiest way to get started with AI is to sign up with one of the cloud AI solutions, ChatGPT for instance, but there are plenty of reasons to run AI on a box you control:

• Lower costs: By running AI on your own machine, you can avoid the costs associated with cloud services. This can be especially important if you’re in the experimentation phase and are not sure what level of compute you will require.
• Control over data: When you run AI on a cloud service, you have to trust that the company is handling your data securely and in compliance with relevant regulations. By running AI on your own machine, you have complete control over your data and can ensure it’s being handled in accordance with your organization’s policies.
• Regulatory compliance: For example, if you’re working in an industry that requires GDPR or HIPPA compliance, having control over your AI environment ensures that you can meet those requirements and avoid potential fines or reputational damage.

To really unlock the power of AI, though, we need to set up a basic development environment. This will allow us to experiment and learn from our mistakes.

Ollama is the primary tool we will be working with. Ollama provides a consistent framework for running and working with different AI models. For our purposes, we will use it, along with the llama3 model. The website includes instructions for different operating systems, but on the Mac it’s as easy as brew install ollama (you have Homebrew right?). Once that’s installed, ollama pull llama3 will download the model and ollama start will start the server.

At this point you can open another terminal and run ollama run llama3 to chat with your own local AI assistant.

Tomorrow, we’ll start a new Rails app and begin interacting with the model.

And after that, take over the world!