That’s why they have the “Copilot PC” hardware requirement, because they’re using an NPU on the local machine.

searches

https://learn.microsoft.com/en-us/windows/ai/npu-devices/

Copilot+ PCs are a new class of Windows 11 hardware powered by a high-performance Neural Processing Unit (NPU) — a specialized computer chip for AI-intensive processes like real-time translations and image generation—that can perform more than 40 trillion operations per second (TOPS).

It’s not…terribly beefy. Like, I have a Framework Desktop with an APU and 128GB of memory that schlorps down 120W or something, substantially outdoes what you’re going to do on a laptop. And that in turn is weaker computationally than something like the big Nvidia hardware going into datacenters.

But it is doing local computation.

Isn’t that the whole shtick of the AI PCs no one wanted? Like, isn’t there some kind of non-GPU co-processor that runs the local models more efficiently than the CPU?

I don’t really want local LLMs but I won’t begrudge those who do. Still, I wouldn’t trust any proprietary system’s local LLMs to not feed back personal info for “product improvement” (which for AI is your data to train on).

Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up.

Eh…I don’t know if you’d call it “really serious hardware”, but when I picked up my 128GB Framework Desktop, it was $2k (without storage), and that box is often described as being aimed at the hobbyist AI market. That’s pricier than most video cards, but an AMD Radeon RX 7900 XTX GPU was north of $1k, an NVidia RTX 4090 was about $2k, and it looks like the NVidia RTX 5090 is presently something over $3k (and rising) on EBay, well over MSRP. None of those GPUs are dedicated hardware aimed at doing AI compute, just high-end cards aimed at playing games that people have used to do AI stuff on.

I think that the largest LLM I’ve run on the Framework Desktop was a 106 billion parameter GLM model at Q4_K_M quantization. It was certainly usable, and I wasn’t trying to squeeze as large a model as possible on the thing. I’m sure that one could run substantially-larger models.

EDIT: Also, some of the newer LLMs are MoE-based, and for those, it’s not necessarily unreasonable to offload expert layers to main memory. If a particular expert isn’t being used, it doesn’t need to live in VRAM. That relaxes some of the hardware requirements, from needing a ton of VRAM to just needing a fair bit of VRAM plus a ton of main memory.

It only looks good in comparison to Google search because they trashed Google search.

Ai does work great, at some stuff. The problem is pushing it into places it doesn’t belong.

I can generally agree with this, but I think a lot of people overestimate where it DOES belong.

For example, you’ll see a lot of tech bros talking about how AI is great at replacing artists, but a bunch of artists who know their shit can show you every possible way this just isn’t as good as human-made works, but those same artists might say that AI is still incredibly good at programming… because they’re not programmers.

It’s a good grammar and spell check.

Totally. After all, it’s built on a similar foundation to existing spellcheck systems: predict the likely next word. It’s good as a thesaurus too. (e.g. “what’s that word for someone who’s full of themselves, self-centered, and boastful?” and it’ll spit out “egocentric”)

It’s also great for troubleshooting consumer electronics.

Only for very basic, common, or broad issues. LLMs generally sound very confident, and provide answers regardless of if there’s actually a strong source. Plus, they tend to ignore the context of where they source information from.

For example, if I ask it how to change X setting in a niche piece of software, it will often just make up an entire name for a setting or menu, because it just… has to say something that sounds right, since the previous text was “Absolutely! You can fix x by…” and it’s just predicting the most likely term, which isn’t going to be “wait, nevermind, sorry I don’t think that’s a setting that even exists!”, but a made up name instead. (this is one of the reasons why “thinking” versions of models perform better, because the internal dialogue can reasonably include a correction, retraction, or self-questioning)

It will pull from names and text of entirely different posts that happened to display on the page it scraped, make up words that never appeared on any page, or infer a meaning that doesn’t actually exist.

But if you have a more common question like “my computer is having x issues, what could this be?” it’ll probably give you a good broad list, and if you narrow it down to RAM issues, it’ll probably recommend you MemTest86.

It’s far better at search than google.

As someone else already mentioned, this is mostly just because Google deliberately made search worse. Other search engines that haven’t enshittified, like the one I use (Kagi), tend to give much better results than Google, without you needing to use AI features at all.

On that note though, there is actually an interesting trend where AI models tend to pick lower-ranked, less SEO-optimized pages as sources, but still tend to pick ones with better information on average. It’s quite interesting, though I’m no expert on that in particular and couldn’t really tell you why other than “it can probably interpret the context of a page better than an algorithm made to do it as quickly as possible, at scale, returning 30 results in 0.3 seconds, given all the extra computing power and time.”

Even then it can only help, not replace folks or complete tasks.

Agreed.

Fundamentally due to it’s design, LLMs are digital duct tape.

The entire history of computer science has been making compromises between efficient machine code and human readable language. LLM’s solve this in a beautifully janky way, like duct tape.

But it’s ultimately still a compromise, you’ll never get machine accuracy from an LLM because it’s sole purpose is to fulfill the “human readable” part of that deal. So it’s applications are revolutionary in the same way as “how did you put together this car engine with only duct tape?” kind of way.

We’ll have to agree to disagree. To go through your points, spell check I don’t find particularly impressive. That was solved previously without requiring the power demands of a small town. Grammer, maybe - but in my experience my “LLM powered” keyboard’s suggestions are still worse than old T9 input.

I’ve had no luck troubleshooting anything with AI. It’s often trained on old data, tries to instruct you to change settings that don’t exist, or dreams up controls that might appear on “similar” hardware. Sure you can perhaps infer a solution, maybe, but it’s rarely correct at first response. It’ll happily run you through steps that are inconsequential to fixing a problem.

Finally, it might be better than indexed search NOW - but mostly because LLMs wrecked that too. I used to be able to use a couple search operators and get directly to the information I needed - now search is reduced to shifting through slop SEO sites.

And it does all this half assing while using enough power to justify dedicated nuclear reactors. I cant help but feel we’ve regressed on so many fronts.

Oh, I don’t think it’ll work. They’ve sold their own snake oil to themselves. It’s a modern day Mechanical Turk. They’re in love with Eliza.

LLMs are quite fast at summarizing large blobs of data though, and they’re so desperate to be first to market with this capability. They think they can become the all-seeing eye of Sauron (I mean…Palantir? Cmon), and they are salivating at the chance that it will work out for them.

But they will fuck up computing for everyone else to get the chance. Already have, in fact.