Every since I heard of ChatGPT I’ve been looking for ways to run something like it locally. I wanted a private self-hosted AI that works offline and is open source. But every time I looked I didn’t find anything that matched what I wanted. Or it seemed overly complicated to get setup. Until now.
The closest thing I found to a private AI was Brave’s Leo but it did not check all the boxes. Then when I was not even looking for one I ran across the video below that introduced me to Ollama. Ollama Is an open source project that interfaces with large language models. There’s a fair number of models to pick from too.
Why do you need a private AI?
Anything you type into ChatGPT, Copilot, or other systems like it gets sent to the cloud. Then it can be used for training the AI later or who knows what else. It might even be sent to other users of the same system as part of an answer to their question. Because of this things like trade secrets, programming code, and even private medical conditions can be leaked.
Companies that deal with sensitive information have been known to block all access to ChatGPT on their networks. I work for one who has done this. So it can be a big deal and you should think about what you type into these public systems.
Or just get an offline private self-hosted AI. 🙂
My testing of Ollama
I’m running Ollama on Linux with the llama3 model. I’m on a Manjaro distro and its working very well. Ones I ran the install script I just typed “ollama run llama3” into the terminal to get it running. The first time took a while because it had to download the model.
It seems to use your video card to do the heavy work when running the AI. But it looks like it can also run on a lower end computer with the CPU. I even seen on their discord server people talking about running it on a raspberry pi. I want to try that at some point but I suspect it will be a lot slower.
It can also run on Windows, and Mac. But i’m not sure what the point of that would be if your interested in privacy. Since Microsoft and Apple will both soon have closed source AI’s in their OS watching your screen.
Finally. Privacy Focused AI Use is Here!
To give credit were it’s due. Here is the video that showed me Ollama. The ollama setup and demo information starts at around the 17 minute point if you want to skip to that.
