Is anyone self hosting AI?
Discussion
I've seen a few YT channels where people build their own AI servers using GPUs. I'm also aware of Google Coral (now discontinued, I believe?).
Has anyone here taken the leap? What did you build?
I've got a house move in the near future. I'll be doing a lot of restructuring of networks and services as part of that so it would be a good time to integrate an AI server. It would be to handle things like watching the CCTV (BlueIris or Frigate) and interaction via Home Assistant. The latter is a bit of a sledgehammer /walnut situation but I didn't much like the latency I experienced last time I built an HA system with local voice. The AI would be on a VLAN without internet access (until it learns to hack its way out).
I was thinking I would start with a single 3080, as I could then justify it as a free GPU (currently I have a 3080 in my main rig, I could replace that and have the 3080 for AI).
Not interested in my AI running on someone else's hardware, I realise this is more expensive!
Has anyone here taken the leap? What did you build?
I've got a house move in the near future. I'll be doing a lot of restructuring of networks and services as part of that so it would be a good time to integrate an AI server. It would be to handle things like watching the CCTV (BlueIris or Frigate) and interaction via Home Assistant. The latter is a bit of a sledgehammer /walnut situation but I didn't much like the latency I experienced last time I built an HA system with local voice. The AI would be on a VLAN without internet access (until it learns to hack its way out).
I was thinking I would start with a single 3080, as I could then justify it as a free GPU (currently I have a 3080 in my main rig, I could replace that and have the 3080 for AI).
Not interested in my AI running on someone else's hardware, I realise this is more expensive!
It’s something I’m toying with at the moment. In fact my partially built AI server is sitting on my desk at the moment. For local AI VRAM is the key factor rather than just computing power. If you can’t fit the model, you can’t run it no matter how fast the GPU is.
Im sure someone faaaar more knowledgeable than me will be along in a moment but the popular cheapo starter system is 2x 3060s (so 24gb VRAM in total) then 2 x 3090 (48gb). For those with a bit more cash there are 4090s and 5090s (potentially multiples). Nvidia is preferred to AMD due to CUDA. There are a million and other ways in, such as systems with large amounts of unified memory (Strix Halo with 128gb etc) or things like Nvidia Spark. Alternatively folks buy ex data centre cards. Recent developments have provided ways to run larger MoE models splitting them over both vram and dram (at the cost of speed)
Model wise, whatever you can fit will not come close to the current frontier models, although things like Qwen 27b is very capable and open weight models are advancing very quickly. I intend to run a Qwen model with a harness handing off to Claude when the local model can’t handle the task.
As I say, I’m not an expert and I’m sure there a million things folks will take issue with but it’s fun learning.
Im sure someone faaaar more knowledgeable than me will be along in a moment but the popular cheapo starter system is 2x 3060s (so 24gb VRAM in total) then 2 x 3090 (48gb). For those with a bit more cash there are 4090s and 5090s (potentially multiples). Nvidia is preferred to AMD due to CUDA. There are a million and other ways in, such as systems with large amounts of unified memory (Strix Halo with 128gb etc) or things like Nvidia Spark. Alternatively folks buy ex data centre cards. Recent developments have provided ways to run larger MoE models splitting them over both vram and dram (at the cost of speed)
Model wise, whatever you can fit will not come close to the current frontier models, although things like Qwen 27b is very capable and open weight models are advancing very quickly. I intend to run a Qwen model with a harness handing off to Claude when the local model can’t handle the task.
As I say, I’m not an expert and I’m sure there a million things folks will take issue with but it’s fun learning.
I have a 5070ti and a 2060 Super in my PC, I've got LM Studio setup and both VS Code on my PC and on my server can load a model and use it. To be honest though, it's just so far behind a £20 a month subscription that I don't really use it much beyond testing. I think it's potentially useful if you build a very controlled system and need very basic stuff like routing activities around a network and queuing up tasks for agents to then go and do with your subscription, because it'll probably do a good enough job of that and it'll keep going when you've run out of credits. I'm going to test it properly though when I get chance.
Interesting stuff from both of you, thanks!
I hadn't realised the models were so big, but the dual GPU thing makes sense now. I had been wondering if I could just sacrifice some performance by going with one GPU but clearly it would affect the model I could run.
Sigh, remember when hardware was only "reassuringly" expensive? I've got a 2-stick 64GB DDR4 kit waiting to go into my main NAS, I think I paid £100 on Amazon. Mental prices now.
I hadn't realised the models were so big, but the dual GPU thing makes sense now. I had been wondering if I could just sacrifice some performance by going with one GPU but clearly it would affect the model I could run.
Sigh, remember when hardware was only "reassuringly" expensive? I've got a 2-stick 64GB DDR4 kit waiting to go into my main NAS, I think I paid £100 on Amazon. Mental prices now.
It's a tricky thing to balance the costs, buying a bunch of gpus and powering them will be a big cost, you need to have a good use case for that investment, which means you will be a heavy user.
If you are a heavy user you will find that these gpus are just not as good as a data center full of expensive gpus.
If you are a light user, wtf are you doing spending £1000s on gpus and power
You could get a spark, £6000 gets you 128g of ram and 1000 tops, but its slower than a 5090, though this only has 32g so limits the models.
£15,000 gets you a 96gb RTX 6000 Pro 3500 tops.
If it f*cks, flies, or infers llms, buy it by the hour
https://getdeploying.com/gpus/nvidia-rtx-pro-6000
If you are a heavy user you will find that these gpus are just not as good as a data center full of expensive gpus.
If you are a light user, wtf are you doing spending £1000s on gpus and power

You could get a spark, £6000 gets you 128g of ram and 1000 tops, but its slower than a 5090, though this only has 32g so limits the models.
£15,000 gets you a 96gb RTX 6000 Pro 3500 tops.
If it f*cks, flies, or infers llms, buy it by the hour

https://getdeploying.com/gpus/nvidia-rtx-pro-6000
I remember watching a video a few years ago by George Hotz, who does the comma.ai 'self-driving' app.
He was saying back then, before I really used AI much, that everyone would eventually have their own computer which ran their own private AI. It would do everything for them, like filter their news, fact check everything, protect them digitally, buy things for them, manage their diary and meetings etc etc. It didn't really make sense to me at the time, but I thought it was interesting.
Fast forward to now, and I think he really tapped into the future with that idea. Yes, cloud AI currently works much better, but when you build your life around it, do you really want OpenAI or Anthropic to retain all your data? Doesn't take a leap too far to imagine a US President threatening to remove a country's access to AI if the other country says or does the 'wrong' thing. So I think self-hosted, at some point, is the way forward.
And the cloud costs add up too. My family are all very into using AI, so myself, my brother, his wife, my wife, my mum, my son all pay around £200/month each to get ChatGPT Pro. So together we are paying over £1,000 in subscriptions. And OpenAI have just now announced they are now throttling the use of £200/month plans and instead they expect heavier users to update to their new £500/month plan. How long will it stay at £500/user??
So if you think about a 3-4 year payback you could probably justify spending tens of thousands of a computer IF it could do the job. I keep watching youtube videos showing benchmark videos of Kimi K3 and the other openweight models vs the frontier stuff. And whilst they are close, it isn't yet quite close enough, and much more effort needs to go into model efficiency to get more than a stuttering pace of tokens outputed.
So I think now isn't the time for self-hosted, but from the pace of development, it could be maybe 1-2 years away?
He was saying back then, before I really used AI much, that everyone would eventually have their own computer which ran their own private AI. It would do everything for them, like filter their news, fact check everything, protect them digitally, buy things for them, manage their diary and meetings etc etc. It didn't really make sense to me at the time, but I thought it was interesting.
Fast forward to now, and I think he really tapped into the future with that idea. Yes, cloud AI currently works much better, but when you build your life around it, do you really want OpenAI or Anthropic to retain all your data? Doesn't take a leap too far to imagine a US President threatening to remove a country's access to AI if the other country says or does the 'wrong' thing. So I think self-hosted, at some point, is the way forward.
And the cloud costs add up too. My family are all very into using AI, so myself, my brother, his wife, my wife, my mum, my son all pay around £200/month each to get ChatGPT Pro. So together we are paying over £1,000 in subscriptions. And OpenAI have just now announced they are now throttling the use of £200/month plans and instead they expect heavier users to update to their new £500/month plan. How long will it stay at £500/user??
So if you think about a 3-4 year payback you could probably justify spending tens of thousands of a computer IF it could do the job. I keep watching youtube videos showing benchmark videos of Kimi K3 and the other openweight models vs the frontier stuff. And whilst they are close, it isn't yet quite close enough, and much more effort needs to go into model efficiency to get more than a stuttering pace of tokens outputed.
So I think now isn't the time for self-hosted, but from the pace of development, it could be maybe 1-2 years away?
Depends what you're trying to achieve, there's a lot of AI stuff that isn't LLMs and doesn't need big hardware, there's also smaller LLMs that are often good enough. Running big stuff if often comically slow unless you have some proper big hardware to match.
Start with the what and then think about the how.
Start with the what and then think about the how.
lord dubington said:
Security compliance is probably one of the more sensible reasons.
Or needing to bybass security....
Well these are some of the reasons I do so much on self hosted but it takes some serious big iron (which means £££££££) to make it properly useful if you're running big models.Or needing to bybass security....
It's not a cheap option - when you use subscriptions or whatever at least you're burning someone else's money, run it yourself and you see all the cost.
But on the other hand at least your data stays your own and you can tailor things too.
For other stuff - speech recognition or image recognition or all that utility functionality, then running it all locally is fairly practical and can be a lot lighter.
I have LMStudio running a few models on a Mac Studio M4 Max with 64GB RAM which is just about enough to load a decent sized model but the performance is far below the likes of Claude and Gemini since I have access to paid models through work. So I don't use it often but it does come in useful on occasion!
lord dubington said:
Security compliance is probably one of the more sensible reasons.
Or needing to bybass security....
More the former.Or needing to bybass security....
I've got a CCTV system with more cameras than most, running on PC hardware. I've refined the settings for alerts over the years but I'm still not happy with the number of false positives I get. So, what I "need" is the watchful eye of an AI agent. Effectively a security guard, watching the feeds 24/7 and deciding "is that a plant moving in the breeze, or is it someone who shouldn't be there?".
The case for self-hosting is clear (to me) there, in that I don't want anyone/anything outside my house to have access to my CCTV feeds. Plus it's a continuous task rather than a one off goal where more bandwidth means getting it done quicker.
Perhaps I'm considering a sledgehammer to crack a walnut!
I've seen demos of AI recognition on Frigate where it even decides "hey, that's Donk. No alert required" Vs "I don't know that guy, I better tell Donk".
donkmeister said:
More the former.
I've got a CCTV system with more cameras than most, running on PC hardware. I've refined the settings for alerts over the years but I'm still not happy with the number of false positives I get. So, what I "need" is the watchful eye of an AI agent. Effectively a security guard, watching the feeds 24/7 and deciding "is that a plant moving in the breeze, or is it someone who shouldn't be there?".
The case for self-hosting is clear (to me) there, in that I don't want anyone/anything outside my house to have access to my CCTV feeds. Plus it's a continuous task rather than a one off goal where more bandwidth means getting it done quicker.
Perhaps I'm considering a sledgehammer to crack a walnut!
I've seen demos of AI recognition on Frigate where it even decides "hey, that's Donk. No alert required" Vs "I don't know that guy, I better tell Donk".
I've had good results using AI CCTV analysis to drive security lighting, a bit more interesting than turning a sensitivity knob for a PIR sensor.I've got a CCTV system with more cameras than most, running on PC hardware. I've refined the settings for alerts over the years but I'm still not happy with the number of false positives I get. So, what I "need" is the watchful eye of an AI agent. Effectively a security guard, watching the feeds 24/7 and deciding "is that a plant moving in the breeze, or is it someone who shouldn't be there?".
The case for self-hosting is clear (to me) there, in that I don't want anyone/anything outside my house to have access to my CCTV feeds. Plus it's a continuous task rather than a one off goal where more bandwidth means getting it done quicker.
Perhaps I'm considering a sledgehammer to crack a walnut!
I've seen demos of AI recognition on Frigate where it even decides "hey, that's Donk. No alert required" Vs "I don't know that guy, I better tell Donk".
It works well generally, except for occasionally thinking that a weber grill which hasn't moved for 5 years is a person.
otolith said:
Or wanting to bypass guardrails and do things which the hosted models consider legally or ethically dubious.
I just had a think about what those things might be and now I feel depressed.I'm just trying to get a security guard on the cheap who won't judge me when I wobble past a camera with my shirt off, with a voice assistant that sounds like Paul Bettany and responds to the name "Jarvis" without me having to shout it multiple times and get bored of waiting, and that doesn't report on my conversations to the bald book seller.
No making of obscene deepfakes or genetically coding ill-tempered mutated seabass here.
Where strict contracts disallow the use of thirdpary AI to process or store data, then using local models is a way around that.
Beyond that, I can't really think of many use cases that make sense. Cloud is cheaper and quicker in most cases. If you do want to bypass basic gaurdrails are there not chinese models that will help you out there?
If you're doing really super illegal stuff with jail terms etc then you have a much wider set of problems beyond which AI you use.
Beyond that, I can't really think of many use cases that make sense. Cloud is cheaper and quicker in most cases. If you do want to bypass basic gaurdrails are there not chinese models that will help you out there?
If you're doing really super illegal stuff with jail terms etc then you have a much wider set of problems beyond which AI you use.
This has taken a turn... I'd hope no-one is trying to use their self-hosted AIs for illegal/objectionable stuff. Certainly my use case is quite mundane. I don't have a few £100k per year to pay for round the clock security patrols, and I want my CCTV to know the difference between a burglar and a squirrel 
Maybe I'm naive and I'm the 2026 equivalent of that one person who only used torrents to download Linux distros? All these other homelab types are generating AI deepfakes of prominent public figures hammering frozen sausages into each others lawns.

Maybe I'm naive and I'm the 2026 equivalent of that one person who only used torrents to download Linux distros? All these other homelab types are generating AI deepfakes of prominent public figures hammering frozen sausages into each others lawns.
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff


