this post was submitted on 07 May 2024
46 points (100.0% liked)
Technology
37362 readers
275 users here now
Rumors, happenings, and innovations in the technology sphere. If it's technological news or discussion of technology, it probably belongs here.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Training your own will be very difficult. You will need to gather so much data to get a model that has basic language understanding.
What I would do (and am doing) is just taking something like llama3 or mistral and adding your own content using RAG techniques.
But fair play if you do manage to train a real model!
OLlama is so fucking slow. Even with a 16-core overclocked Intel on 64Gb RAM with an Nvidia 3080 10Gb VRAM, using a 22B parameter model, the token generation for a simple haiku takes 20 minutes.
Hmmm weird. I have a 4090 / Ryzen 5800X3D and 64GB and it runs really well. Admittedly it's the 8B model because the intermediate sizes aren't out yet and 70B simply won't fly on a single GPU.
But it really screams. Much faster than I can read. PS: Ollama is just llama.cpp under the hood.
Edit: Ah, wait, I know what's going wrong here. The 22B parameter model is probably too big for your VRAM. Then it gets extremely slow yes.
It should be split between VRAM and regular RAM, at least if it's a GGUF model. Maybe it's not, and that's what's wrong?