Thank you for the discussion folks, I’ll try out Llama.cpp and report back.
I also saw the Neural Engine Stuff hasn’t been merged into main kernel yet but it’s available as a separate out of tree patch. Hopefully merging that will help with more model support? (Pure guesswork)
I also saw that PCL stuff isn’t ready yet and u/marcan42 said it’s WIP that also might be helpful in getting better model support because read somewhere that metal isn’t going to be a part of the Asahi kernel ever(?)
I’m no expert at any of this but hopefully we’ll be able to run some sort of GPTs locally someday.