My local pi-AI
In this post, I’ll take you through how I set up my own locally hosted AI… using the Raspberry Pi I introduced you to a couple of posts ago.
If all you’re interested in is whether a Raspberry Pi is good enough for running AI then the short answer is no. It does do the job, but it’s frustratingly slow to do anything other than a couple of very simple questions. If you did genuinely want to run your own AI, I’d use some better hardware or simply stick to cloud-based AI. Or, run it on your actual day-to-day machine rather than a dedicated server, depending on your specs.
Also, for this I’ll be using Docker. So, I began by using SSH to remotely access my Raspberry Pi (which is running Ubuntu Server), and then I installed Docker with just a single command – Linux does have some perks!

Docker’s installation doesn’t take long, and then it informs me of how to run Docker without being root, so I go ahead and do that. If you’re interested, here’s the link to the Docker documentation, which helped me during this part of the setup: https://docs.docker.com/manuals/

The ‘logout’ command did boot me out of SSH, so I had to reconnect and then I checked which groups my user was in, and there I was in the ‘docker’ group!

Although we’ve confirmed that we’re in the right group, let’s go ahead and ensure we can run a Docker command whilst not being root, or using sudo. This also gives us a chance to ensure Docker itself has installed correctly and is working.

Next, I’ll install Ollama, which is what we’ll use to run our AI models. Don’t ask me any questions about Ollama, because frankly I don’t know. But, the installation is lovely and simple:

However, we do have to make a small amendment to a file once it has downloaded, as shown here:


Right at the top, we’ve made a change to the “Environment” variable. The purpose of this is simply to make Ollama listen on all IP addresses, rather than just the loopback address (127.0.0.1). This is because we’re going to be using a web UI to access our AI, which is much more convenient than having to SSH onto our Pi every time – it is also much easier for user management, if we were to share this with others. This web UI, which we’ll be getting onto next, will run in a Docker container which will have an IP address and therefore Ollama needs to listen out to all addresses as opposed to the loopback address.
I also want to use Docker Compose for our web UI container. Compose allows to create a configuration file and then use a single command to run that file, meaning we only have to do the setup once. This is important when we have configurations we want to specify, because otherwise the command we need to use to run our container each time is rather long and complicated, and it’s easier to use Compose in the long run. So, let’s make a directory and the config file:

Next, let’s actually type up this configuration file. They follow a standard format and they’re pretty self-explanatory, so just from looking at a few examples you won’t be far off being able to create your own for other situations. Here’s the finished configuration file:

- “image” specifies which image to grab from the specified repository. Here, I’m using “Open WebUI” (more about them here).
- “container_name”… any guesses?
- “volumes” specifies an area of storage on the host device that is used by the container, which is persistent and not deleted when the container is stopped. Pretty handy for many applications.
- “ports” specifies first the container port, which is the port the application within the port is configured to use, and then the host port, which is the port that the application is actually available on for the outside world. So, Open WebUI is utilising port 3000, but incoming connections have to use port 8080 to connect to it from the outside. (container:host!)
- “extra_hosts” is a bit of an odd one, one I don’t fully understand to be perfectly honest. But in this scenario I have defined a particular Docker networking alias which allows my container to interact with the host machine – necessary for Open WebUI to interact with Ollama.
- “restart” decides when the container will restart. Here, if any issues arise and the container stops for whatever reason, it will automatically restart itself. It will only truly stop if I tell it to with a command. “unless-stopped” is usually the go-to.
Next, let’s run the “docker compose up” command, which is what we use to actually spin up the container with this config file.

Firstly, the “-d” switch means “detached” – so that the container runs in the background rather than taking up our terminal, allowing us to immediately get on with our other commands. You can also see here the time taken to create the container – the initial download took a few minutes, but after that it was super fast: one of the perks of containers.
Next, we’ll run “docker ps” to see what containers are currently running, to ensure it is up and reporting to be healthy:

We can see it’s up, but let’s ensure we can actually connect to it. So, I’ll use my web browser to connect to the IP address of my Raspberry Pi, but on port 8080 (which we specified not long ago, when creating our docker compose file):

And we’re on. Let’s create an account and see what’s next:

Now, we’ve got our chat ready to go.. but the AI is nowhere to be seen. We can’t select a model in the top-right, because although we’ve installed Ollama, we haven’t actually downloaded any AI models to use. Let’s do that now. I’ve decided to go for “phi3″… I don’t know what that is, it was just fairly popular on the Ollama website.
Some of you may be recalling me mentioning how slow the AI ran on the Pi at the start of this post, and you may now be shouting at your screen about how it’s all my fault for choosing the wrong model. Please know I did try a few other models after finishing up this blog post, and I did see some improvement but not enough to make “Pi AI” worthwhile… sue me.
Anyway, after about 5 minutes phi3 is downloaded and we can refresh the page in our browser and give it a whirl. I simply select the model, and then go ahead and ask it a question:

There you go, job done. Want to see more? You know how to do it, follow the guide and do it yourself.
Thanks for reading.
Leave a Reply