So that was several posts in a row where I was Eurekaing about figuring out how to finally get ROCm stable, shortly after which… it would break again without warning.

The thing about ROCm that I have now fully internalized is that I don’t understand it, and it may be made in such a way that, or relying on things that, I simply don’t understand. It has periods of meta-stability.

You’ll hunt for new things, new SSDT files, new memory mappings, new kernel parameters, build options, versions, firmware files, etc. And right after a change, it will work!

“AHA!” you say, “I have found the magic thing I was missing. It’s smooth sailing from here on out!”

And it will work all day. Or maybe all week. You might push a couple million tokens through it. And then suddenly, it crashes. And once it has crashed, thereafter it will immediately crash again every time you start it. You’ll swear and wonder why. You’ll be frantic.

“This was just working. I used it all week! What’s changed?! If it was just random crashes that would be one thing, but we went from rock solid all week to it crashes every single start before first turn and I didn’t change a thing!”

You’ll reboot, clear CMOS, do card resets with rocm-smi… and eventually, though it doesn’t make any sense, start hunting for that next “tweak” that must actually be the one that you need.

And hours of searching and experimenting later, you’ll suddenly “find” one that enables you to start again. And not only does it start, but it’s stable! I can ask it to do a task of 50 tool calls and it just rolls! Huzzah!”

Only no. In a day, or a week, it’ll suddenly stop and be unstartable again.

No more, I’m done.

— § —

Previously Vulkan had been rock sold every time, completely no drama, but I’d hesitated to use it because it was about 50% slower than ROCm, which seemed like too much to trade away… better to keep trying until I “figured ROCm out.”

But then I found someone who suggested removing the display card from the devices stack for Vulkan, and when I did… Boom. Not a 50% difference, but about a 5% difference. For no drama. And about 80% less power usage.

I’m sold. I’m Vulkan now on the two v620s and Qwen 3.5 35B A3B q8 (Bartowski). On the RX 6700XT, I’ll throw like a text and a visual embedding model maybe. Simple tools that will be of use.

— § —

Meanwhile, also worth keeping track of: every single Qwen 3.5 template out there is broken for llama.cpp because, it turns out, with Qwen the expectation is that the template does a bunch of fancy things to clean up, but llama.cpp doesn’t support Jinja, only a subset called Minja because until 5 minutes ago that sort of “eh whatevs we’ll do it in template) logic wasn’t really considered viable.

That creates a problem: all the templates are for jinja, but I needed a template for minja (in llama.cpp) so that we could get tool use working properly.

So I had Claude Opus review a bunch of the existing community jinja templates and then pick one to refactor, to the extent possible, for minja.

Here you go: template.jinja

Key and somewhat vexing note, for this to work properly, you also need to to into src/llama-grammar.cpp and change the value of MAX_REPETITION_THRESHOLD from 2,000 to:

#define MAX_REPETITION_THRESHOLD 200000

Because OpenClaw uses like six million parameters for some of its tools and that explodes the grammar enforcer (I still don’t fully understand whether this is a part of minja or not by convention), so once you have a template that actually works instead of breaks, it will seem to be a template that breaks instead of works until you recompile with that constant defined.

— § —

I started all of this nearly two months ago. It’s taken a long time, but we’re finally feeling like we’re getting to the point where I can start to build soon, rather than just be playing with toys.

One day after posting that I thought I had the problem solved, more was needed. Turns out that Linux was not honoring the mainboard BIOS’s aperture setting was was leading to stability on runs, crashes on others.

So we also added:

amdgpu.gartsize=2048

I have the nagging feeling that even this may not kill it once and for all, and there is every chance that in the end, we will have to drop back to Vulkan.

But for now, we seem to be golden, with Q6 weights and 128k Q8 context cache doing 42-46 tokens/second.

So I made a post a couple days ago about “the monster” but following that post, a few things had become clear:

• ROCm was unstable, maybe even very unstable

• Vulkan was much slower, about half as fast, but far more stable

• I didn’t really know what I was doing (okay, I knew this going in)

I was happy to know that there was a path to booting into Mac OS without having to tear the system apart again, and that it only required editing and recompiling Radeon driver kexts 😛 but meanwhile, inference was crashing more often than I’d like. Sometimes very often. I won’t even tell you all of the things I’ve tried, but among the many tactics tried to identify the error and/or solve it, I tried:

• Multiple models, at different quants, and from different providers

• Multiple versions of ROCm, including 6.2, 6.3, 6.3.4, 6.4, 6.4.4, 7.0, 7.1.1, and 7.2

• Multiple versions and forks of llama.cpp

• Aphrodite and vLLM and even for a minute MLCLLM

• Rearranging card order

• An entire library of environment variables, llama command line options, and kernel command line arguments

• Excluding the RX 6700XT card as the odd-man-out and just running the v620s in tandem

• Many different context sizes and quantizations

Sometimes I would think I had it fixed because it had been 5 or maybe even 10 turns without a crash on ROCm!

…and then it would crash on turn 6 or 11.

Some things worked better than others. For example, llama.cpp pre-b8353 with a Q8 model from Bartowski was much more stable than the Q8 from Unsloth or llama-current. And running it on ROCm 6.2 or 6.3 was more stable than running it on ROCm 7.1.1 or 7.2. Also, using the software SMU via kernel parameter seemed to help some things. But we’d always end up in the same place: ROCm crash, back to Vulkan.

I must have decided to “just use Vulkan” 100 times, but the thing is, when ROCm is giving you 45-50 tokens/second, it’s hard to settle for 21-25 even if it’s stable. And stable it was…throughout it all, Vulkan never crashed.

— § —

The two key problems were as far as I could tell:

• PCIe bus errors which I had sort of grumblingly admitted were just the result of trying to run data center hardware on consumer mainboards

• Page faults that were by far the most common thing to take down llama when using an ROCm build of it

The solutions in my case were different for each of the three items above.

1. PCIe Bus Errors

It turns out that the, um, less expensive (though very highly reviewed) Montech Century 1200w power supply I’d acquired from Amazon left something to be desired. I’d been so wrapped up in the exotic-lack-of-understanding that I felt around LLMs, and the “it’s not going to be quite right, it’s not consumer PC hardware” mentality that I hadn’t been monitoring voltage. When I did, I found that this PSU, which claimed to have 100A on the 12V rail, was running at 11.5V-11.6V at idle when the cards were, according to rocm-smi, only drawing about 5 watts each.

It should not be a surprise to anyone that when we started to push into load, this decreased further.

I’ve replaced that unit with a Corsair unit and now we’re at around 12.05V at idle and around 11.9V under full load. I can live with that. And the PCIe bus errors have gone away.

The lesson here is that “gamer power supplies” rated at 1200W are mostly marketing. They don’t really expect you to draw the 1200W, they expect you to run one really fat graphics card and want the “1200” number to be showing through the clear sides of your gamer case so that people can be impressed. When you actually try to pull 80-90 amps, they just can’t do it.

I keep having to re-learn this lesson build after build… Don’t try to get off cheaply on the power supply. Especially if you’re going to be running 3x RDNA2 cards, including two server cards that want 25 amps each on 12V. Be sure also, for those of you that are new to this, not to put both PCIe power connectors on a fat card like this on the same cable. Run two cables from the PSU or you’ll fry your PSU-side connector on the single cable.

2. (THE BIGGIE) Page faults

These were the bane of my existence, and I’ve spent multiple nights this week up until the wee hours trying to find *anything* that would reduce the “crashiness” of ROCm. So . many . environment . variables. And llama arguments. And kernel arguments. And new builds of ROCm, of llama.cpp, and of half the libraries that they rely on.

The solution: credit where it’s due, I stumbled across this guy’s posts: https://medium.com/@agentz/how-to-fix-rocm-pytorch-memory-faults-on-amd-gpus-segmentation-fault-page-not-present-544b9f62f627

I almost didn’t try this suggestion that he makes:

ttm.pages_limit=25165824

It looks sketch and random and not really related to amdgpu or amdgpu-dkms or really even anything related to anything that I’d been working on. But I decided to look up what it did and once I did, it made a sort of sense, so I tried it. About three hours ago now. After battling this all week.

And voila… no more crashes. At least not yet. And we’re running inference on the fast Lemonade version of llama.cpp that is based on its own fork of ROCm and is pretty exotic. Previously this would page fault on the first turn. Maybe the second. But we’ve been up for several hours now and all appears well.

The above increases the size of the address space that the kernel is allowed to open up. I don’t know what the default is, but at this point, I believe I know that it’s too small for 76GB VRAM + 128GB DRAM, which is what I’ve been trying to make work.

Note that there is also:

ttm.page_pool_size

This pre-allocates the GTT address space in case you expect memory pressure. I’m not using it right now, but if I was planning to run a really big model, I’d probably set both of these values. Right now I’ve just got ttm.pages_limit set to 30000000, which is just a nudge up from where he had it.

— § —

The amazing thing is that in a week of talking to every frontier hyperscaler LLM (Claude, ChatGPT, Gemini, etc.) and extensive Googling, I didn’t step across anyone discussing ttm.* parameters in relation to ROCm. Until tonight.

I think maybe this doesn’t come up all that often since it’s basically going to impact people who:

• Are using multiple AMD GPUs on a consumer system

• In a configuration (i.e. v620s) where you can end up with quite a bit of VRAM (more than triple the typical just 8-24GB)

But many thanks to AgentZ on Medium and hopefully this helps someone else that’s banging their head against a wall.

And, just in case any of them matter (I am *not* messing about with this now that we have ROCm whirrled up and stable), here are my kernel command line args:

pci=realloc=off amdgpu.gpu_recovery=1 amdgpu.mcbp=0 amd_iommu=off intel_iommu=off pcie_aspm=off pcie_port_pm=off amdgpu.swSMU=1 amdgpu.cswr_enable=0 ttm.pages_limit=25165824

With this I am running 2x Radeon Pro v620s and 1x RX 6700XT (total 76GB VRAM, 3x Radeon RDNA2 GPUs) and, finally, they appear to be… stable.

— § —

Important addendum: https://leapdragon.net/2026/03/23/more-was-needed/

It started as one of those weekends that you want back. Everyone’s been sick since Friday, O— worst of all. Sounds terrible, feels terrible, has hardly come out of his room, isn’t eating. M— started to feel rotten, too. And the same for dad.

And I have been trying to wrap up the first stage of the technology “rebuild” and return to the “user” role. There was a time when I wanted to hack all the time just to say I was hacking but that ended, like, 20 years ago. Now I sort of want to get things done.

The first hurdle in all of this was just getting three graphics cars into a case (two of them exceptionally long) and powered and cooled (two of them are designed for directional air server farms and don’t have active cooling). Then the next hurdle was getting reliable, performance inference going with a model I liked. That turned out to be Qwen 3.5, though not without hiccups.

Then I got stuck on the next two things:

• Plug OpenClaw into the local inference

• Get Mac OS booting again (that’s right, dual-booting a Hackintosh with wild hardware)

Both of these things needed to be done, and both were non-trivial tasks. So to make my weekend worse, on top of everyone feeling various degrees of sick, Friday night and yesterday I spent most of my time feeling lousy and alternating between sleep and banging on these two problems.

— § —

I must have spent six hours yesterday on the Mac problem alone, and I have no idea how many times I was like:

<press reset>
<select Linux>
<log in>
$ mkdir /tmp/EFI
$ sudo mount -t auto /dev/sdc1 /tmp/EFI
<enter password>
$ nano /tmp/EFI/EFI/OC/config.plist

Eventually I got frustrated enough to enlist the help of Claude (my own agent being down, more on this momentarily) to scour the web looking for answers. All kinds of wild stuff was tried in the OpenCore plist and with ACPI tables and so on. No matter what, it seemed like there was a hard hang around PCI enumeration, and I presumed the two v620s with their massive footprints in address space were the cause. Into late last night, between Claude and I, we just couldn’t get past PCI enumeration and mapping.

Claude was telling me that it was time to give up, it just wasn’t on the cards, the Mac OS IOPCI stuff couldn’t cope with what I was trying to do, that Mac OS had never been intended to work with. We had spent hours trying to turn off PCIe slots, build SSDTs, directly patch memory with OpenCore shells scripts on bootup, build .efi modules to rearrange PCI windows, but nothing seemed to work, and in fact nothing ever really seemed to change.

— § —

In the wee hours I pivoted over to OpenClaw, where I’d been having enough trouble with Qwen that I’d started downloading and trying instead, always to disappointment. The thing is, Qwen 3.5 35B A3B is just really good in certain ways. It’s blazing fast. It’s really smart. And it was perfect on my 76GB of VRAM (more on this in a bit) in that I could fit like an Unsloth dynamic q6_XL on it and have room for a massive context window at q8. It’s local inference that doesn’t feel like local inference.

Only, the other thing is, turns and tool calling were completely broken. Like, completely. It had been hard enough to get the model up and running—all kinds of errors, apparent incompatibilities, updates… and then we ended up in a state in which it would often lose the ability to call tools altogether, and would just pitifully announce it was going to do so, then do nothing, then apologize profusely when asked, then announce it was going to succeed this time, then do nothing, then say it couldn’t understand what was happening.

Of course, being new to running my own LLM, I was leaning heavily on LLMs to help me through, and they were coming up with all kinds of ideas and all kinds of opinions about Openclaw. We must have tried 10 different versions of Openclaw and 100 different (often hallucinated) configuration options for openclaw.json. We also tried vLLM (don’t bother with RDNA2, the tensor parallelism really requires better card-to-card links than PCIe, something that’s just not in the architecture, plus it really fights with just about every version of ROCm), Aphrodite, Ollama, LM Studio, a bunch of everyone’s hand-written proxies, and about six different front-ends for chat.

Nothing had worked, and I was very tired of complaining at LLMs while they complained at me as we tried to get my own LLM to behave with my own agent.

Then, as is so often the case, a less direct search somehow opened up the channels of serendipity enough to make a massive difference. I searched Google for “local models similar to Qwen 3.5 35b but with more stable tool calling” and then, without warning, Gemini opened up with all of the answers that it had previously hidden from me in our hours of chatting and banging on openclaw.json and template.jinja files. It suddenly just rattled off a list:

• Qwen 3.5 MoE models mix tool calling outputs into <think> phases, which nothing on earth can handle right now

• Qwen 3.5 is also a bit of a wild child as you increase temperature and response length (which I have liked), but that wildness includes its approach to JSON and XML

• Llama.cpp and Openclaw both have aggressive timeouts

• All together this means turns that platforms like Openclaw misinterpret and may even miss, leaving turns or entire sessions in undefined states

• Solution: Set 20 minute timeouts everywhere, turn of Qwen’s thinking mode at the inference level, and reduce temperature from 1+ to 0.7 or less

I implemented these fixes in about five minutes in the middle of the night, and voila, Qwen 3.5 35B A3B was suddenly a rock solid tool user in Openclaw. Like WTF. But also, in a good way.

Well, obviously what I did is put it to work on my entire chat summaries with Claude and GPT working on the Hackintosh problem and ask it to build me a library of findings about each area of research, as well as a final THEORY.md outlining its own theory, based on all the research, of just what was breaking in Mac OS so that we could try to fix it, since Claude and GPT were out of ideas.

— § —

So today, I came back in the morning, and boom. Nestled in the guts of THEORY.md was the point that just because the hang was around where Mac OS was spitting out that it was initializing PCI, that did not mean that this was causing the hang, and since my BIOS version (already checked) was good and my BIOS settings were right and the config.plist had been tested a million times, and all the effort spend trying to get the card to disappear for Mac OS in one way or another had not worked, it seemed likely that it was a kext, not the kernel, that was causing a hang, and the proximity of this hang to output about PCI enumeration had fooled us all.

And, at that moment, it all fell into place. I had opted for a display card with the same architecture ad the two Radeon Pro v620 units, with the idea that we could get another 12GB that way for inference. But what it also meant was that NootRX (the kext that enables certain Navi-baed hardware) might be choking on the cards in kextland, rather than this being about BARs and address space, etc.

Pulled the NootRX source and that was the answer. NootRX grabs one Navi card, whichever is the first it finds on the bus, and tries to use it for display.

It was likely grabbing a v620 first and we might not even be hung at all, there just aren’t any display outputs on a v620. And since we had the source, we had the power to fix the kext…

— § —

So anyway, tonight is here, and:

• Yes, inference is up and running

• Yes, Openclaw is up and running on it, including solid, stable, tool use and long projects

• Yes, Mac OS and Linux are dual booting again on this now wilder-ass PC with 76GB VRAM, 128GB DRAM, 3 RDNA2 GPUs, and 16 hardware CPU threads

• I can move on

Point being, the weekend ended on a high note, and I’m that much closer to re-architecting my computing and data life.

— § —

Next up:

• OwnCloud and a pile of non-HFS+ storage on a dedicated host

• Plus migrate a bunch of data off of the seemingly infinite universe of HFS+ I have here

It’s happening. Local inference and a future trajectory that is on Linux once again. Meanwhile, it’s late at night (or even wee hours) with the work week ahead. Not even sure this is coherent. So… end.

So at the same time that I’ve been slid laterally at work into something approximating an AI leadership role, I am also moving to boot AI up in my life. It’s about time. I’ve been in tech so very, very long—yet this is really the first time I’ve been slow to, or maybe even missed, the “early adopter” period. Although maybe this time my timing is actually better. I mean:

• 1983 — Started learning software engineering

• 1986 — Went “online” with asynchronous UUCP via a local gateway

• 1993 — Adopted Linux as my primary OS

• 1997 — Moved from film photography to digital photography

• 1999 — My whole life is in a tablet computer (Newton MessagePad 2000)

• etc.

The problem with all my “early adopting” over the years has generally been that I start it, master it, write the book about it, then move on, all years before anyone’s willing to pay me for it. I generally have moved on to the “next technology” while people are still making pronouncements about how the previous technology will never catch on. Then I get frustrated a decade later watching people build careers out of what I did a decade before, that was generally considered arcana and occult geekdom at the time.

So maybe my timing is better this time, because the world is actually accelerating into AI right now. The old me would have played with LLMs in the early stages, but been transitioning away from them onto the next set of projects around the time the chatbots (ChatGPT et. al.) were launching and stunning the world with what Large Language Models could do.

— § —

So, without much fanfare, I declare the next iteration of the “monster” up and running. Here and there in all of this I’ve made references to the “monster” which is the larger-than-average PC that I’ve always had in my life, since way back in the mid-’80s, often built halfway out of spare parts. I still have some of the parts from old incarnations in the basement. For example the instance that had dual Pentium 200 MMX CPUs and nine (9) 5.25″ full height 1GB SCSI drives in a RAID-5 configuration. Back then, it was a monster. (This has also generally always been the reason to run Linux… basically there are people writing drivers and systems for it that just aren’t there for other platforms.)

Where are we now?

• Core i9-9900k (you’re like ‘pffffftt’, but wait a bit)

• 128gb RAM (you’re like okay biggish but ‘monster’ ummmm not sure)

• 40TB online storage, about 50% SSD, attached to SAS (I have a lot of DSLR photos, like >350k)

• 76GB VRAM across 3x AMD Navi2x GPUs (← told you there was a monster in here somewhere) via 2x Radeon Pro V620 and 1x Radeon RX6700XT

• LTO4 internal (I mean, nothing says ‘big weird computer’ like streaming tape

What are we doing with it? Running 30ish-billion parameter LLM models with large amounts of q8_0 context. It’s a lot of compute, but it’s doing good work, giving me fast local LLM that’s pretty damned solid.

Now if I can figure out how to make it pay…

So it was a bit of an adventure getting the card to initialize and be owned by the kernel driver, but the Radeon Pro V620 is now up. That puts me at 44GB of VRAM (it lives next to a 6700XT). This weekend we’ll maybe get ROCm up and running and with 44GB of VRAM, we’ll be off to the races.

— § —

I almost hate to say it but these things are cheap and available right now. I suspect that won’t be the case for long. These are virtual GPU cards for server farms with 32GB of VRAM each and no display out. They’re NAVI 21 architecture cards and should play nicely with most modern things. But they do require:

• 300 watts of power, each

• A long, long, long card slot, with cooling attached, a good 3″ longer than the “full length” cards that most modern cases already can’t accommodate (you either need a very serverish case or some case mods)

• Your own cooling solution, as they are fully enclosed but contain no active cooling (expecting high airflow from their installation context)

• Mainboard/chipset that supports >4GB addressing and BAR

• Patience

— § —

I am running mine on a Gigabyte Z390UD mainboard with an i9-9900k and 128GB of system RAM, using a 6700XT for display (which contributes another 12GB VRAM), and powering everything with a 1200W power supply. It’s not a small setup.

For hours it was looking like it wasn’t going to initialize as I played with all kinds of BIOS settings and kernel arguments, but in one of those very “hacking moments” I stumbled across a Reddit post that suggested:

pci=realloc=off amdgpu.gpu_recovery=1 amdgpu.mcbp=0

Boom. Just like that, it was online. All the fancy stuff I was trying didn’t even matter. Probably it’s amdgpu.mcbp=0 that’s the magic here.

— § —

More on setup:

• I printed this to attach a fan: https://www.thingiverse.com/thing:7296707

• The only case I found that worked for my needs at a reasonable price was: https://www.newegg.com/phanteks-atx-full-tower-enthoo-pro-steel-plastic-computer-case-black-ph-es614pc-bk/p/N82E16811854004

• The card wants to 8-pin power headers

• Yes, it will work on a chipset (non-metal-clad) slot