If you download the source, you should be able to build it for metal? Croco.cpp is just a fork of kobold.cpp
I think lmstudio added MLX support, but otherwise you are stuck with anything llama.cpp based. I’d probably download llama.cpp directly and use the llama server first.
Nope, Intel is more of a pain. Your best bet is the llama.cpp server’s SYCL backend (or kobold.cpp if they have a build for that).
They have an OpenVINO server, but its not for the faint of heart.
CPU inference is, unfortunately, slow, even on my 7800X3D.
The one that might be interesting is deepseek code v2 lite, as its a very fast MoE model. IIRC microsoft also released a Phi MoE thats good for CPU.
Keep an eye out for upcoming bitnet models.
Dont bother upgrading RAM though. You will be bandwidth limited anyway, and it doesn’t make a huge difference.
CodeQwen 1.5 is pretty old at this point, afaik made obsolete by their latest release.
The Qwen models (at least 2.5) are really only good to like 32K, which is still a ton of context. But I’ve been testing Qwen 32B at 64K -90K and even that larger model is… Not great.
32K is generally enough to get the jist of whatever you’re trying to fill in.
I have not tested WSL or VMs in Windows in awhile, but my impression is that “it depends” and you should use the native windows version unless you are having some major installation issues.
Aphrodite is a fork of vllm. You should check it out!
If you are looking for raw batched speed, especially with some redundant context, I would actually recommend sglang instead. Check out its experimental flags too.
Llamafile is fine, but it still leaves a lot of performance on the table.
You can setup kobold.cpp with Q8 flash attention without ever having to install pytorch, which is the real headache. It does have a little python launch script, but its super minimal.
You can use the native llama.cpp server for absolutely zero python usage.
I would not recommend that for performance reasons, AFAIK.
Windows is fine, I should make that more clear.
Yep. 20GB is basically 24GB, though its too tight for 70B models.
One quirk for 7900 owners is that installing flash attention for long context usage can be a pain. Apparently it is doable now, I need to dig up the link, but it might just be easier to use kobold.cpp rocm with its native flash attention.
As for vision models, that is a whole different can of worms. Exllama does not support this, so you’d need a framework that does.
If you are looking for niche models, check out MiniG (which is a continued pretrain of the already very excellent GLM4-9B): https://huggingface.co/bartowski/miniG-GGUF
Llama.cpp support is recent, though I’m not 100% sure its completely fixed. It should work in Aphrodite as well.
No, but you can run Qwen 2.5 34B with 24GB total.
Host it in TabbyAPI instead of ollama too. Use its native tensor parallelism and Q4 cache, it will fly.
Could it also be that it’s a $80K-$100K car?
I have no solid source, but read comments suggesting Geico tends to not insure “exotic” $100K+ luxury vehicles.
…And I think this is important to remember this when talking about it. The Cybertruck is not a peer of a F-150, but a G-Wagon, a Maserati Levante or whatever.
Pretty much everything has an API :P
ollama is OK because its easy and automated, but you can get higher performance, better vram efficiency, and better samplers from either kobold.cpp or tabbyAPI, with the catch being that more manual configuration is required. But this is good, as it “forces” you to pick and test an optimal config for your system.
I’d recommend kobold.cpp for very short context (like 6K or less) or if you need to partially offload the model to CPU because your GPU is relatively low VRAM. Use a good IQ quantization (like IQ4_M, for instance).
Otherwise use TabbyAPI with an exl2 quantization, as it’s generally faster (but GPU only) and much better at long context through its great k/v cache quantization.
They all have OpenAI APIs, though kobold.cpp also has its own web ui.
Uh, depends on your hardware and model, but probably TabbyAPI?
Text-generation-webui is cool, but also kinda crufty. Honestly a lot of the stuff is holdovers from what’s now ancient history in LLM land, and it has (for me) major performance issues at longer context.
I have an old Lenovo laptop with an NVIDIA graphics card.
@Maroon@lemmy.world The biggest question I have for you is what graphics card, but generally speaking this is… less than ideal.
To answer your question, Open Web UI is the new hotness: https://github.com/open-webui/open-webui
I personally use exui for a lot of my LLM work, but that’s because I’m an uber minimalist.
And on your setup, I would host the best model you can on kobold.cpp or the built-in llama.cpp server (just not Ollama) and use Open Web UI as your front end. You can also use llama.cpp to host an embeddings model for RAG, if you wish.
This is a general ranking of the “best” models for document answering and summarization: https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard
…But generally, I prefer to not mess with RAG retrieval and just slap the context I want into the LLM myself, and for this, the performance of your machine is kind of critical (depending on just how much “context” you want it to cover). I know this is !selfhosted, but once you get your setup dialed in, you may consider making calls to an API like Groq, Cerebras or whatever, or even renting a Runpod GPU instance if that’s in your time/money budget.
Sam is actually a liar though.
Everyone in open source AI has been calling him a snake ever since llama1 came out. If you want a more authoritative source, look to the CEO of huggingface, oldschool AI researchers and such.
A letter seen by Reuters, sent by Vivaldi, Waterfox, and Wavebox, and supported by a group of web developers, also supports Opera’s move to take the EC to court over its decision to exclude Microsoft Edge from being subject to the Digital Markets Act (DMA).
OK…
Shouldn’t they be fighting Chrome, more than anything? Surely there’s a legal avenue for that, though I guess there’s a risk of getting deprioritized by Google and basically disappearing.
I somehow didnt’ get a notification for its post, but thats a terrible idea lol.
We already have AI horde, and it has nothing to do with blockchain. We also have APIs and GPU services… that have nothing to do with blockchain, and have no need for blockchain.
Someone apparently already tried the scheme you are describing, and absolutely no one in the wider AI community uses it.
Where’s a Johnny cab when you need it
Or a Delamain.
Or setting up a home server, which I figured some here would do.