Could in theory be saved later by “Sorry, this message was inappropriate, if it’s any excuse I was drunk when I sent it.” If feeling confident, could be followed by “The part about having a crush on you was true tho.”.
tinyllama 1.1b would probably run reasonably fast. Dumb as a rock for sure. But hey its something! My 2015 t460 thinkpad laptop with an i7 6600U 2.6GhZ duo core was able to do llama 3.1 8B at 1.5T-2T/s which while definitely slow was also just fast enough to still have fun in real time.
Than what are the minimal specs to run ollama (llama3 8b or preferably 27b) at a decent speed?
I have an old pc that now runs my plex and arr suite. Was thinking of upgrading it a bit and running ollama on it as well. It doesn’t have a gpu, so what else does it need? I don’t have a big budget, so no new nvidia card for me.
“decent speed” depends on your subjective opinion and what you want it to do. I think its fair to say if it can generate text around your slowest tolerable reading speed thats a bare minimum for real time conversational things. If you want a task done and don’t mind stepping away to get a coffee it can be much slower.
The i7 duo core 2.6ghz CPU in my laptop trying to run 8B was jusst barely enough to be passing grade for real time talking needs at 1.2-1.7 T/s it could say a short word or half of a complex one per second. When it needed to process something or recalculate context it took a hot minute or two. Kind of annoying if you were getting into what its saying Bumping it up to a desktop with a AMD ryzen 5 2600 6 core CPU was a night and day difference. It spits out a sentence very quick at 5-6 t/s. Im still working on getting the 4GB RX 580 GPU used for offloading so those numbers are just with the CPU bump. RAM also matters DDR6 will beat DDR4 speed wise
The modern Siren. We aren’t going to be fooled to steer our ships into rocks anymore, so instead were presented with irresistible belly fluff inside a bear trap of kitty claws and teeth.
Y’all laugh but I’m getting into Linux and dusted off an old i7 laptop with 16gb of RAM. Slapped a $40 512GB ssd and linux mint on it to get into !selfhost!
My current laptop is an i7 with 16 GB of RAM. Hardware requirements have plateaued pretty hard unless your trying to run something that requires the latest GPU.
lemmy.world
Oldest