Is anyone self hosting AI?
Discussion
I've seen a few YT channels where people build their own AI servers using GPUs. I'm also aware of Google Coral (now discontinued, I believe?).
Has anyone here taken the leap? What did you build?
I've got a house move in the near future. I'll be doing a lot of restructuring of networks and services as part of that so it would be a good time to integrate an AI server. It would be to handle things like watching the CCTV (BlueIris or Frigate) and interaction via Home Assistant. The latter is a bit of a sledgehammer /walnut situation but I didn't much like the latency I experienced last time I built an HA system with local voice. The AI would be on a VLAN without internet access (until it learns to hack its way out).
I was thinking I would start with a single 3080, as I could then justify it as a free GPU (currently I have a 3080 in my main rig, I could replace that and have the 3080 for AI).
Not interested in my AI running on someone else's hardware, I realise this is more expensive!
Has anyone here taken the leap? What did you build?
I've got a house move in the near future. I'll be doing a lot of restructuring of networks and services as part of that so it would be a good time to integrate an AI server. It would be to handle things like watching the CCTV (BlueIris or Frigate) and interaction via Home Assistant. The latter is a bit of a sledgehammer /walnut situation but I didn't much like the latency I experienced last time I built an HA system with local voice. The AI would be on a VLAN without internet access (until it learns to hack its way out).
I was thinking I would start with a single 3080, as I could then justify it as a free GPU (currently I have a 3080 in my main rig, I could replace that and have the 3080 for AI).
Not interested in my AI running on someone else's hardware, I realise this is more expensive!
It’s something I’m toying with at the moment. In fact my partially built AI server is sitting on my desk at the moment. For local AI VRAM is the key factor rather than just computing power. If you can’t fit the model, you can’t run it no matter how fast the GPU is.
Im sure someone faaaar more knowledgeable than me will be along in a moment but the popular cheapo starter system is 2x 3060s (so 24gb VRAM in total) then 2 x 3090 (48gb). For those with a bit more cash there are 4090s and 5090s (potentially multiples). Nvidia is preferred to AMD due to CUDA. There are a million and other ways in, such as systems with large amounts of unified memory (Strix Halo with 128gb etc) or things like Nvidia Spark. Alternatively folks buy ex data centre cards. Recent developments have provided ways to run larger MoE models splitting them over both vram and dram (at the cost of speed)
Model wise, whatever you can fit will not come close to the current frontier models, although things like Qwen 27b is very capable and open weight models are advancing very quickly. I intend to run a Qwen model with a harness handing off to Claude when the local model can’t handle the task.
As I say, I’m not an expert and I’m sure there a million things folks will take issue with but it’s fun learning.
Im sure someone faaaar more knowledgeable than me will be along in a moment but the popular cheapo starter system is 2x 3060s (so 24gb VRAM in total) then 2 x 3090 (48gb). For those with a bit more cash there are 4090s and 5090s (potentially multiples). Nvidia is preferred to AMD due to CUDA. There are a million and other ways in, such as systems with large amounts of unified memory (Strix Halo with 128gb etc) or things like Nvidia Spark. Alternatively folks buy ex data centre cards. Recent developments have provided ways to run larger MoE models splitting them over both vram and dram (at the cost of speed)
Model wise, whatever you can fit will not come close to the current frontier models, although things like Qwen 27b is very capable and open weight models are advancing very quickly. I intend to run a Qwen model with a harness handing off to Claude when the local model can’t handle the task.
As I say, I’m not an expert and I’m sure there a million things folks will take issue with but it’s fun learning.
I have a 5070ti and a 2060 Super in my PC, I've got LM Studio setup and both VS Code on my PC and on my server can load a model and use it. To be honest though, it's just so far behind a £20 a month subscription that I don't really use it much beyond testing. I think it's potentially useful if you build a very controlled system and need very basic stuff like routing activities around a network and queuing up tasks for agents to then go and do with your subscription, because it'll probably do a good enough job of that and it'll keep going when you've run out of credits. I'm going to test it properly though when I get chance.
Interesting stuff from both of you, thanks!
I hadn't realised the models were so big, but the dual GPU thing makes sense now. I had been wondering if I could just sacrifice some performance by going with one GPU but clearly it would affect the model I could run.
Sigh, remember when hardware was only "reassuringly" expensive? I've got a 2-stick 64GB DDR4 kit waiting to go into my main NAS, I think I paid £100 on Amazon. Mental prices now.
I hadn't realised the models were so big, but the dual GPU thing makes sense now. I had been wondering if I could just sacrifice some performance by going with one GPU but clearly it would affect the model I could run.
Sigh, remember when hardware was only "reassuringly" expensive? I've got a 2-stick 64GB DDR4 kit waiting to go into my main NAS, I think I paid £100 on Amazon. Mental prices now.
It's a tricky thing to balance the costs, buying a bunch of gpus and powering them will be a big cost, you need to have a good use case for that investment, which means you will be a heavy user.
If you are a heavy user you will find that these gpus are just not as good as a data center full of expensive gpus.
If you are a light user, wtf are you doing spending £1000s on gpus and power
You could get a spark, £6000 gets you 128g of ram and 1000 tops, but its slower than a 5090, though this only has 32g so limits the models.
£15,000 gets you a 96gb RTX 6000 Pro 3500 tops.
If it f*cks, flies, or infers llms, buy it by the hour
https://getdeploying.com/gpus/nvidia-rtx-pro-6000
If you are a heavy user you will find that these gpus are just not as good as a data center full of expensive gpus.
If you are a light user, wtf are you doing spending £1000s on gpus and power
You could get a spark, £6000 gets you 128g of ram and 1000 tops, but its slower than a 5090, though this only has 32g so limits the models.
£15,000 gets you a 96gb RTX 6000 Pro 3500 tops.
If it f*cks, flies, or infers llms, buy it by the hour
https://getdeploying.com/gpus/nvidia-rtx-pro-6000
I remember watching a video a few years ago by George Hotz, who does the comma.ai 'self-driving' app.
He was saying back then, before I really used AI much, that everyone would eventually have their own computer which ran their own private AI. It would do everything for them, like filter their news, fact check everything, protect them digitally, buy things for them, manage their diary and meetings etc etc. It didn't really make sense to me at the time, but I thought it was interesting.
Fast forward to now, and I think he really tapped into the future with that idea. Yes, cloud AI currently works much better, but when you build your life around it, do you really want OpenAI or Anthropic to retain all your data? Doesn't take a leap too far to imagine a US President threatening to remove a country's access to AI if the other country says or does the 'wrong' thing. So I think self-hosted, at some point, is the way forward.
And the cloud costs add up too. My family are all very into using AI, so myself, my brother, his wife, my wife, my mum, my son all pay around £200/month each to get ChatGPT Pro. So together we are paying over £1,000 in subscriptions. And OpenAI have just now announced they are now throttling the use of £200/month plans and instead they expect heavier users to update to their new £500/month plan. How long will it stay at £500/user??
So if you think about a 3-4 year payback you could probably justify spending tens of thousands of a computer IF it could do the job. I keep watching youtube videos showing benchmark videos of Kimi K3 and the other openweight models vs the frontier stuff. And whilst they are close, it isn't yet quite close enough, and much more effort needs to go into model efficiency to get more than a stuttering pace of tokens outputed.
So I think now isn't the time for self-hosted, but from the pace of development, it could be maybe 1-2 years away?
He was saying back then, before I really used AI much, that everyone would eventually have their own computer which ran their own private AI. It would do everything for them, like filter their news, fact check everything, protect them digitally, buy things for them, manage their diary and meetings etc etc. It didn't really make sense to me at the time, but I thought it was interesting.
Fast forward to now, and I think he really tapped into the future with that idea. Yes, cloud AI currently works much better, but when you build your life around it, do you really want OpenAI or Anthropic to retain all your data? Doesn't take a leap too far to imagine a US President threatening to remove a country's access to AI if the other country says or does the 'wrong' thing. So I think self-hosted, at some point, is the way forward.
And the cloud costs add up too. My family are all very into using AI, so myself, my brother, his wife, my wife, my mum, my son all pay around £200/month each to get ChatGPT Pro. So together we are paying over £1,000 in subscriptions. And OpenAI have just now announced they are now throttling the use of £200/month plans and instead they expect heavier users to update to their new £500/month plan. How long will it stay at £500/user??
So if you think about a 3-4 year payback you could probably justify spending tens of thousands of a computer IF it could do the job. I keep watching youtube videos showing benchmark videos of Kimi K3 and the other openweight models vs the frontier stuff. And whilst they are close, it isn't yet quite close enough, and much more effort needs to go into model efficiency to get more than a stuttering pace of tokens outputed.
So I think now isn't the time for self-hosted, but from the pace of development, it could be maybe 1-2 years away?
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff