Merge v2-rebuild: GDI+ single-surface UI, self-calibrating progress, compact push-to-talk

.gitignore

@@ -0,0 +1,66 @@
+# Build directories
+build/
+deps/
+*.vcxproj
+*.vcxproj.filters
+*.vcxproj.user
+*.sln
+*.suo
+*.user
+*.userosscache
+*.sln.docstates
+
+# Compiled binaries
+*.exe
+*.dll
+*.lib
+*.obj
+*.o
+*.a
+*.so
+*.dylib
+
+# CMake
+CMakeCache.txt
+CMakeFiles/
+cmake_install.cmake
+*.cmake
+!CMakeLists.txt
+!**/CMakeLists.txt
+!cmake/*.cmake
+!ggml/cmake/*.cmake
+
+# Visual Studio
+.vs/
+*.pdb
+*.ilk
+*.exp
+*.idb
+*.ipdb
+
+# Models (too large for git, users download separately)
+models/*.bin
+!models/download-*.sh
+!models/download-*.cmd
+
+# Release packages (keep structure but not binaries)
+release/WinDictation/*.exe
+release/WinDictation/*.dll
+release/WinDictation.zip
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+desktop.ini
+
+# Temporary files
+*.tmp
+*.log
+*.bak

Architechture-and-dev-guide.md

@@ -0,0 +1,411 @@
+# Win Dictation — Architecture Analysis & Dev Guide (Build 4)
+
+Two parts:
+
+- **Part A — Architecture analysis & recommendations.** Where the app stands now (post-fixes) and what's worth hardening.
+- **Part B — Dev guide** for the two things you asked for: (1) removing the leftover Windows chrome (combo dropdown buttons, button hairlines, the line around *Pinned*), and (2) showing real transcription progress + a time estimate instead of a static "Transcribing…".
+
+Context: native C++ / Win32 + SDL2 + whisper.cpp, CPU-only (i5-7th-gen, 2c/4t). Companion docs: `MODERN-UI-AND-FIXES.md`, `FINDINGS-FIXES-TESTS.md`.
+
+---
+
+# Part A — Architecture Analysis & Recommendations
+
+## A1. Current architecture (it's in good shape)
+
+```
+┌── UI thread (message loop) ───────────────┐     ┌── Audio thread (SDL callback) ──┐
+│  • window, hotkeys, tray, GDI+ paint      │     │  • append f32 samples to        │
+│  • owns Transcriber                       │◀────│    m_capture (mutex)            │
+│  • marshals worker results via PostMessage│     │  • update VU energy (atomic)    │
+└──────────────┬────────────────────────────┘     └─────────────────────────────────┘
+               │ stop_and_transcribe()
+               ▼
+┌── Worker thread (one per utterance) ──────┐
+│  • run_inference(): whisper_full() ONCE   │
+│  • PostMessage(WM_APP_RESULT, text)       │
+└────────────────────────────────────────────┘
+```
+
+State machine: **Idle → Recording → Transcribing → Idle**, with the model preloaded on a detached thread at startup.
+
+**What's genuinely good now:**
+- **Clean thread boundaries.** Audio capture, inference, and UI never block each other; all cross-thread hand-offs are atomics or `PostMessage`. This is the right shape.
+- **Batch (record-then-transcribe).** The correct model for dictation on a slow CPU — no real-time deadline, latency independent of clip length.
+- **Core is decoupled and unit-testable** (`Transcriber` + `run_inference`/`transcribe_sync`, `text_util.h`). The UI is a thin shell over it.
+- **Robust startup** (single-instance, honest `g_modelOk`, `g_initializing` guard, `m_cfg_mtx`) — the earlier crashes are designed out, not patched over.
+
+The remaining issues are **polish and hardening**, not structural. No rewrite needed.
+
+## A2. Recommendations (ranked by impact ÷ effort)
+
+| # | Recommendation | Why | Effort |
+|---|---|---|---|
+| 1 | **Transcription progress + ETA** (Part B2) | The #1 UX gap: a 1:40 clip shows a static "Transcribing…" with no sign of life. whisper.cpp already exposes a progress callback — wire it up. | S |
+| 2 | **De-theme the stock controls** (Part B1.1) | The hairlines around the buttons / *Pinned* are themed-edge leftovers. `SetWindowTheme(h, L"", L"")` + `WS_CLIPCHILDREN` + hide-focus removes them. | S |
+| 3 | **Replace the comboboxes with custom dropdowns** (Part B1.2) | A `CBS_OWNERDRAWFIXED` combo *cannot* hide its native dropdown button — that's the "second arrow" you see. The only clean fix is to not use a combobox. | M |
+| 4 | **Persist settings** | Mic, model, pin, auto-paste, window position all reset every launch. A tiny INI (`WritePrivateProfileStringW`) or registry blob fixes it. | S |
+| 5 | **Cancel/abort during transcription** | Pairs with #1 — if a long clip is wrong, let the user abort via `whisper_full_params.abort_callback`. | S |
+| 6 | **Delete the stale `src/CMakeLists.txt`** | It references removed APIs (`init`, `is_using_gpu`) and a `test-audio` target the root build ignores. It's a trap for the next contributor. | XS |
+| 7 | **Bound the capture buffer** | `m_capture` grows ~1.9 MB/30 s with no cap. Add a soft limit (e.g. auto-stop at 10 min) so a forgotten recording can't grow unbounded. | XS |
+| 8 | **Configurable hotkeys + conflict check** | `Ctrl+Shift+Space`/`H` are hard-coded; `RegisterHotKey` failures are silently ignored. Surface a warning and allow remap. | M |
+| 9 | **Lightweight logging** | A single rotating log line per session (model, threads, last error) makes field issues diagnosable without a debugger. | S |
+
+**Architectural note for the future:** the app leans on owner-drawing *stock* controls (buttons, combos). That's fine at this size, but every stock control fights you on chrome (edges, focus cues, dropdown buttons). If the UI grows, the higher-leverage move is a small set of **fully custom controls** (a `Button`, a `Select`, a `ProgressBar` you paint entirely in `WM_PAINT` and hit-test yourself) rather than more owner-draw patches. Part B1.2's custom dropdown is the first step down that path.
+
+---
+
+# Part B — Dev Guide
+
+## B1. Remove the leftover Windows chrome
+
+There are **three** separate sources of "Windows-y" pixels, each with a different cause:
+
+| What you see | Cause |
+|---|---|
+| Thin light line around *Pinned*; vertical line left + horizontal line top of Copy/Paste/Clear | The visual-style **themed edge** drawn behind owner-draw buttons, plus **focus rectangles** |
+| Two arrows / a boxy button on the mic + model selectors | A `CBS_OWNERDRAWFIXED` combo still lets the **system draw the dropdown button** and field border — owner-draw only covers the text/items |
+
+### B1.1 Kill the button hairlines + focus rectangles (quick, high-impact)
+
+Three changes, all small:
+
+**(a) De-theme the owner-draw controls.** `SetWindowTheme(h, L"", L"")` strips the UxTheme styling from a control so Windows stops painting its themed border/edge behind your `WM_DRAWITEM`. Add the header/lib and call it on every owner-draw control right after you create it:
+
+```cpp
+#include <uxtheme.h>
+#pragma comment(lib, "uxtheme.lib")
+
+// after creating each owner-draw button (Record, Pin, Copy, Paste, Clear):
+SetWindowTheme(hBtn, L"", L"");      // remove themed edges
+SetWindowSubclass(hBtn, BtnProc, 1, 0);
+```
+
+**(b) Stop the parent painting under the children.** Add `WS_CLIPCHILDREN` to the main window so its double-buffered `WM_PAINT` never bleeds a pixel into a child's rectangle:
+
+```cpp
+hMainWnd = CreateWindowExW(
+    WS_EX_TOPMOST,
+    L"WhisperDictationClass", L"Dictation",
+    WS_OVERLAPPEDWINDOW | WS_CLIPCHILDREN,   // <- add WS_CLIPCHILDREN
+    CW_USEDEFAULT, CW_USEDEFAULT, 400, 340,
+    nullptr, nullptr, hInstance, nullptr);
+```
+
+**(c) Hide focus rectangles app-wide.** The dotted/thin rect that appears on whichever control has keyboard focus (often *Pinned* after you click it). Tell the UI-state machine to keep focus cues hidden — once, after the controls exist:
+
+```cpp
+SendMessageW(hMainWnd, WM_CHANGEUISTATE, MAKEWPARAM(UIS_SET, UISF_HIDEFOCUS), 0);
+```
+
+After (a)+(b)+(c), the only remaining stock chrome is the combo dropdown button, handled next.
+
+### B1.2 Replace the comboboxes with a custom dropdown (removes the native button)
+
+A combobox always owns its dropdown button — you can't draw it away. Replace each combo with a **flat owner-draw "select" button** (you already draw exactly this look in `DrawCombo`) that opens a **custom dark popup window**. No native button, no field edge, fully on-theme.
+
+**Data model** (replace `g_modelComboPaths` usage as needed; keep a label list + selection per selector):
+
+```cpp
+std::vector<std::wstring> g_audioItems;  int g_audioSel = 0;
+std::vector<std::wstring> g_modelItems;  int g_modelSel = 0;   // parallel to g_modelComboPaths
+```
+Populate these in `RefreshAudioDevices` / `RefreshModelList` instead of `CB_ADDSTRING`. Create `ID_SEL_AUDIO` / `ID_SEL_MODEL` as `BS_OWNERDRAW` buttons (not comboboxes) and draw them with a field renderer:
+
+```cpp
+void DrawSelect(LPDRAWITEMSTRUCT d, const std::wstring& text) {
+    Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
+    Rect rc(d->rcItem.left, d->rcItem.top, d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
+    SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
+    Rect field = rc; field.Inflate(-1,-1);
+    bool hover = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
+    FillRound(g, hover ? C_SURFACEHI : C_SURFACE, field, 9);
+    StrokeRound(g, C_BORDER, field, 9, 1.0f);
+    Font f(d->hDC, g_fUI);
+    RectF tb((REAL)field.X+10, (REAL)field.Y, (REAL)(field.Width-28), (REAL)field.Height);
+    DrawTextC(g, text.c_str(), f, C_TEXT, tb, StringAlignmentNear, StringAlignmentCenter);
+    int cx = field.GetRight()-16, cy = field.Y + field.Height/2;     // our chevron, the only one now
+    Pen pen(C_TEXTDIM, 1.6f);
+    g.DrawLine(&pen, cx-4, cy-2, cx, cy+2); g.DrawLine(&pen, cx, cy+2, cx+4, cy-2);
+}
+// in WM_DRAWITEM:
+//   case ID_SEL_AUDIO: DrawSelect(d, g_audioItems.empty()?L"No devices":g_audioItems[g_audioSel]); return TRUE;
+//   case ID_SEL_MODEL: DrawSelect(d, g_modelItems.empty()?L"—":g_modelItems[g_modelSel]); return TRUE;
+```
+
+**The popup window** — a small borderless top-level you paint yourself; closes on pick or focus loss:
+
+```cpp
+struct PopupState { std::vector<std::wstring> items; int sel; int hot; HWND owner; int ctrlId; };
+static PopupState g_pop;
+
+LRESULT CALLBACK PopupProc(HWND h, UINT m, WPARAM w, LPARAM l) {
+    switch (m) {
+    case WM_MOUSEMOVE: {
+        int row = GET_Y_LPARAM(l) / 30;
+        if (row != g_pop.hot) { g_pop.hot = row; InvalidateRect(h, nullptr, FALSE); }
+        TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 }; TrackMouseEvent(&t);
+        return 0;
+    }
+    case WM_MOUSELEAVE: g_pop.hot = -1; InvalidateRect(h, nullptr, FALSE); return 0;
+    case WM_LBUTTONUP: {
+        int row = GET_Y_LPARAM(l) / 30;
+        if (row >= 0 && row < (int)g_pop.items.size())
+            PostMessageW(g_pop.owner, WM_APP_SELECT, (WPARAM)g_pop.ctrlId, (LPARAM)row);
+        DestroyWindow(h); return 0;
+    }
+    case WM_KILLFOCUS: DestroyWindow(h); return 0;
+    case WM_ERASEBKGND: return 1;
+    case WM_PAINT: {
+        PAINTSTRUCT ps; HDC hdc = BeginPaint(h, &ps);
+        RECT rc; GetClientRect(h, &rc);
+        HDC mem = CreateCompatibleDC(hdc);
+        HBITMAP bmp = CreateCompatibleBitmap(hdc, rc.right, rc.bottom);
+        HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
+        {
+            Graphics g(mem); g.SetSmoothingMode(SmoothingModeAntiAlias);
+            Rect all(0,0,rc.right,rc.bottom);
+            FillRound(g, C_SURFACE, all, 10); StrokeRound(g, C_BORDER, all, 10, 1.0f);
+            Font f(mem, g_fUI);
+            for (int i = 0; i < (int)g_pop.items.size(); ++i) {
+                Rect row(3, i*30+3, rc.right-6, 28);
+                if (i == g_pop.hot) FillRound(g, C_SURFACEHI, row, 7);
+                RectF tb((REAL)row.X+9,(REAL)row.Y,(REAL)row.Width-12,(REAL)row.Height);
+                DrawTextC(g, g_pop.items[i].c_str(), f, (i==g_pop.sel)?C_ACCENT:C_TEXT,
+                          tb, StringAlignmentNear, StringAlignmentCenter);
+            }
+        }
+        BitBlt(hdc,0,0,rc.right,rc.bottom,mem,0,0,SRCCOPY);
+        SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
+        EndPaint(h, &ps); return 0;
+    }
+    }
+    return DefWindowProc(h, m, w, l);
+}
+
+void ShowSelectPopup(HWND owner, int ctrlId, const std::vector<std::wstring>& items, int sel) {
+    static bool reg = false;
+    if (!reg) { WNDCLASSEXW wc{ sizeof(wc) }; wc.lpfnWndProc = PopupProc; wc.hInstance = hInst;
+        wc.hCursor = LoadCursor(nullptr, IDC_ARROW); wc.lpszClassName = L"DictPopup";
+        RegisterClassExW(&wc); reg = true; }
+    g_pop = { items, sel, -1, owner, ctrlId };
+    RECT rc; GetWindowRect(GetDlgItem(owner, ctrlId), &rc);
+    int h = (int)items.size()*30 + 6, wdt = rc.right - rc.left;
+    HWND p = CreateWindowExW(WS_EX_TOOLWINDOW | WS_EX_TOPMOST, L"DictPopup", L"",
+        WS_POPUP, rc.left, rc.bottom+2, wdt, h, owner, nullptr, hInst, nullptr);
+    int corner = DWMWCP_ROUND; DwmSetWindowAttribute(p, DWMWA_WINDOW_CORNER_PREFERENCE, &corner, sizeof(corner));
+    ShowWindow(p, SW_SHOWNA); SetForegroundWindow(p); SetFocus(p);
+}
+```
+
+Wire it up: clicking a selector opens the popup; the popup posts the chosen row back:
+
+```cpp
+// WM_COMMAND:
+case ID_SEL_AUDIO: ShowSelectPopup(hWnd, ID_SEL_AUDIO, g_audioItems, g_audioSel); break;
+case ID_SEL_MODEL: ShowSelectPopup(hWnd, ID_SEL_MODEL, g_modelItems, g_modelSel); break;
+
+// new message handler:
+case WM_APP_SELECT: {                      // #define WM_APP_SELECT (WM_USER + 5)
+    int ctrlId = (int)wParam, idx = (int)lParam;
+    if (ctrlId == ID_SEL_AUDIO) { g_audioSel = idx; g_config.capture_id = idx; }
+    else if (ctrlId == ID_SEL_MODEL && idx < (int)g_modelComboPaths.size()) {
+        g_modelSel = idx;
+        g_config.model_path = exe_dir() + "\\" + g_modelComboPaths[idx];
+        g_modelLoaded = false; g_modelOk = false; SetStatus(hWnd, L"Loading model…");
+        std::thread([]{ bool ok=g_tx.reload(g_config); g_modelOk=ok; g_modelLoaded=true; }).detach();
+    }
+    InvalidateRect(GetDlgItem(hWnd, ctrlId), nullptr, FALSE);
+} break;
+```
+
+You can now delete the `CBS_*`/`WM_MEASUREITEM`/`DrawCombo` combo code and the `WM_CTLCOLORLISTBOX` handler. The selectors are now pixel-identical to your other controls with exactly one (your) chevron.
+
+> Lighter alternative if you don't want a popup window yet: keep the combos but call `SetWindowTheme(hCombo, L"", L"")` — it flattens the dropdown button to a plain square. It's *less* boxy but the button is still there, so the popup-window route above is the real fix.
+
+---
+
+## B2. Transcription progress + time estimate
+
+You don't have to guess the timing — **whisper.cpp reports real progress.** `whisper_full_params` has a `progress_callback` that fires repeatedly during inference with an `int` 0–100 (fraction of the audio processed). Feed that to a determinate progress bar + a live ETA, so "Transcribing…" becomes "Transcribing 1:40 of audio · 45% · ~9s left".
+
+### B2.1 transcriber — expose progress + audio length
+
+`transcriber.h` (public):
+
+```cpp
+using ProgressCb = std::function<void(int)>;            // 0..100
+void  set_progress_callback(ProgressCb cb) { m_on_progress = std::move(cb); }
+float audio_seconds() const { return m_audio_seconds.load(); }
+void  request_cancel() { m_abort = true; }              // optional (B2.4)
+```
+`transcriber.h` (private):
+
+```cpp
+ProgressCb m_on_progress;
+std::atomic<float> m_audio_seconds{0.0f};
+std::atomic<bool>  m_abort{false};
+static void s_progress(struct whisper_context*, struct whisper_state*, int p, void* ud);
+static bool s_abort(void* ud);
+```
+
+`transcriber.cpp` — the static trampolines and the `run_inference` hook:
+
+```cpp
+void Transcriber::s_progress(whisper_context*, whisper_state*, int p, void* ud) {
+    auto* self = static_cast<Transcriber*>(ud);
+    if (self && self->m_on_progress) self->m_on_progress(p);
+}
+bool Transcriber::s_abort(void* ud) {
+    auto* self = static_cast<Transcriber*>(ud);
+    return self && self->m_abort.load();
+}
+
+std::string Transcriber::run_inference(std::vector<float>& audio) {
+    if (!m_ctx) return "";
+    m_abort = false;
+    if (m_cfg.trim_silence) trim_silence(audio);
+    m_audio_seconds = (float)(audio.size() / (double)WHISPER_SAMPLE_RATE);   // for the UI
+
+    whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
+    wp.print_progress = false; wp.print_realtime = false; wp.print_timestamps = false;
+    wp.no_timestamps  = true;  wp.translate = false;
+    wp.language = m_cfg.language.c_str();
+    wp.n_threads = m_cfg.n_threads;
+    wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true; wp.temperature = 0.0f;
+    wp.progress_callback           = &Transcriber::s_progress;   // <- live progress
+    wp.progress_callback_user_data = this;
+    wp.abort_callback              = &Transcriber::s_abort;      // <- optional cancel
+    wp.abort_callback_user_data    = this;
+
+    std::string out;
+    if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
+        int n = whisper_full_n_segments(m_ctx);
+        for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
+        out = clean_text(out);
+    }
+    return out;
+}
+```
+
+> If your `whisper.h` predates `abort_callback`, drop those two lines — `progress_callback` has been in whisper.cpp far longer and is what matters here.
+
+### B2.2 main.cpp — progress state + ETA
+
+```cpp
+#define WM_APP_PROGRESS (WM_USER + 6)
+std::atomic<int> g_progress{0};
+DWORD g_busyStart = 0;
+
+// at startup, next to set_result_callback:
+g_tx.set_progress_callback([](int p){ PostMessage(hMainWnd, WM_APP_PROGRESS, (WPARAM)p, 0); });
+```
+
+Start the clock when transcription begins (HK_TOGGLE stop branch):
+
+```cpp
+} else {                       // was recording -> stop
+    g_busyStart = GetTickCount();
+    g_progress  = 0;
+    g_tx.stop_and_transcribe();
+    SetStatus(hWnd, L"Transcribing…");
+}
+```
+
+Receive progress and repaint the bar:
+
+```cpp
+case WM_APP_PROGRESS:
+    g_progress = (int)wParam;
+    InvalidateRect(hWnd, &g_vuRect, FALSE);
+    return 0;
+```
+
+ETA text (replace the `is_busy()` branch in `UpdateStatus`):
+
+```cpp
+} else if (g_tx.is_busy()) {
+    int   p     = g_progress.load();
+    float total = g_tx.audio_seconds();
+    float elapsed = (GetTickCount() - g_busyStart) / 1000.0f;
+    int   mm = (int)total / 60, ss = (int)total % 60;
+    if (p >= 3) {
+        float est    = elapsed * 100.0f / p;          // projected total
+        float remain = est - elapsed; if (remain < 0) remain = 0;
+        swprintf_s(buf, L"Transcribing %d:%02d  •  %d%%  •  ~%ds left", mm, ss, p, (int)(remain + 0.5f));
+    } else {
+        swprintf_s(buf, L"Transcribing %d:%02d of audio…", mm, ss);
+    }
+    SetStatus(hwnd, buf);
+}
+```
+
+### B2.3 Reuse the VU strip as a determinate progress bar
+
+While recording the strip shows the VU; while transcribing it shows progress. Add a renderer and branch in `WM_PAINT`:
+
+```cpp
+void DrawProgress(Graphics& g, const RECT& r, float frac) {
+    Rect track(r.left, r.top, r.right - r.left, r.bottom - r.top);
+    FillRound(g, C_SURFACEHI, track, 4);
+    frac = frac < 0 ? 0 : (frac > 1 ? 1 : frac);
+    int w = (int)((r.right - r.left) * frac);
+    if (w > 0) { Rect fill(r.left, r.top, w, r.bottom - r.top); FillRound(g, C_ACCENT, fill, 4); }
+}
+```
+```cpp
+// in WM_PAINT, where you currently call DrawVU:
+if (g_tx.is_busy())      DrawProgress(g, g_vuRect, g_progress.load() / 100.0f);
+else                     DrawVU(g, g_vuRect, g_energy);
+```
+
+Keep it ticking between callbacks so the ETA counts down smoothly — in `WM_TIMER`, add:
+
+```cpp
+else if (g_tx.is_busy()) {
+    InvalidateRect(hWnd, &g_vuRect, FALSE);   // (UpdateStatus already runs each tick below)
+}
+```
+
+That's it: a moving bar, a percentage, and a shrinking "~Ns left" — the user can see it's alive and roughly how long is left. The estimate self-corrects as real progress arrives (the first few percent are rougher; it tightens quickly).
+
+### B2.4 (Optional) Cancel a long transcription
+
+You added `request_cancel()` + the abort callback in B2.1. Hook it so that pressing the button **while busy** aborts instead of being ignored, and treat the empty result as "Cancelled":
+
+```cpp
+// top of the HK_TOGGLE handler:
+if (g_tx.is_busy()) { g_tx.request_cancel(); SetStatus(hWnd, L"Cancelling…"); break; }
+```
+`whisper_full` returns promptly with no/aborted segments → your existing `WM_APP_RESULT` empty-string path shows "No speech detected"; change that label to "Cancelled" when a cancel was requested if you want to distinguish them.
+
+---
+
+## B3. Build & test
+
+- **New link deps:** `uxtheme.lib` (added via `#pragma comment` in B1.1). GDI+ is already linked.
+- **New files:** none required for B2; B1.2 adds the popup window proc inside `main.cpp` (no new translation unit).
+- **Rebuild:** `cmake --build build --config Release` as usual.
+
+**Verify:**
+- [ ] No hairline around *Pinned*; no left/top lines on Copy/Paste/Clear; focus no longer draws a dotted rect.
+- [ ] Mic + model selectors show a single (your) chevron, open a dark rounded popup, and selecting reloads the model / switches device.
+- [ ] Record a ~1–2 min clip → the strip fills as a progress bar, the status shows `%` and a shrinking `~Ns left`, and it completes (no static "Transcribing…").
+- [ ] (If added) pressing the button mid-transcription cancels promptly.
+
+**Headless regression (extends `tests/test_core.cpp` from `FINDINGS-FIXES-TESTS.md`):**
+- Set a progress callback that records the max value seen; assert it reaches ~100 for `samples/jfk.wav`, and that callbacks arrive in non-decreasing order. This locks in that progress reporting keeps working across whisper.cpp upgrades.
+
+```cpp
+int last = -1, maxp = 0; bool monotonic = true;
+t.set_progress_callback([&](int p){ if (p < last) monotonic = false; last = p; if (p > maxp) maxp = p; });
+t.transcribe_sync(audio);
+CHECK(monotonic, "progress is non-decreasing");
+CHECK(maxp >= 95, "progress reaches ~100%");
+```
+
+---
+
+*Apply order: B1.1 (5 min, instant visual win) → B2 (progress, the big UX gain) → B1.2 (custom dropdowns) → optional B2.4 cancel. Each is independent and safe to ship on its own.*
+
+

CMakeLists.txt

@@ -0,0 +1,160 @@
+cmake_minimum_required(VERSION 3.5)
+project(win-dictation C CXX)
+
+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+
+# Path to modules
+list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
+
+# Set output directory
+set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
+
+# Options
+option(BUILD_SHARED_LIBS "build shared libraries" OFF)
+option(WHISPER_SDL2 "whisper: support for libSDL2" ON)
+option(WHISPER_NO_AVX "whisper: disable AVX" OFF)
+option(WHISPER_NO_AVX2 "whisper: disable AVX2" OFF)
+option(WHISPER_NO_FMA "whisper: disable FMA" OFF)
+option(WHISPER_NO_F16C "whisper: disable F16C" OFF)
+
+# ----------------------------
+# SDL2
+# ----------------------------
+if(NOT SDL2_DIR)
+    set(SDL2_DIR "${CMAKE_CURRENT_SOURCE_DIR}/SDL2-mingw/cmake")
+endif()
+
+find_package(SDL2 REQUIRED)
+
+string(STRIP "${SDL2_LIBRARIES}" SDL2_LIBRARIES)
+
+# ----------------------------
+# Whisper (ONLY dependency layer)
+# ----------------------------
+# IMPORTANT:
+# We no longer build ggml separately.
+# whisper/ must contain its own CMakeLists.txt (modern whisper.cpp layout)
+add_subdirectory(whisper)
+
+# ----------------------------
+# Common Library (optional legacy utilities)
+# ----------------------------
+if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/common/common.h")
+
+    set(COMMON_TARGET common)
+
+    add_library(${COMMON_TARGET} STATIC
+        common/common.h
+        common/common.cpp
+        common/common-ggml.h
+        common/common-ggml.cpp
+        common/common-whisper.h
+        common/common-whisper.cpp
+        common/grammar-parser.h
+        common/grammar-parser.cpp
+    )
+
+    target_include_directories(${COMMON_TARGET} PUBLIC
+        ${CMAKE_CURRENT_SOURCE_DIR}/common
+    )
+
+    # Link against whisper target only (no ggml exposure)
+    target_link_libraries(${COMMON_TARGET} PRIVATE whisper)
+
+    set(COMMON_SDL_TARGET common-sdl)
+
+    add_library(${COMMON_SDL_TARGET} STATIC
+        common/common-sdl.h
+        common/common-sdl.cpp
+    )
+
+    target_include_directories(${COMMON_SDL_TARGET} PUBLIC
+        ${CMAKE_CURRENT_SOURCE_DIR}/common
+        ${SDL2_INCLUDE_DIRS}
+    )
+
+    target_link_libraries(${COMMON_SDL_TARGET} PRIVATE ${SDL2_LIBRARIES})
+
+else()
+
+    # Empty fallback targets
+    add_library(common INTERFACE)
+    add_library(common-sdl INTERFACE)
+
+endif()
+
+# ----------------------------
+# Main executable
+# ----------------------------
+add_executable(win-dictation WIN32
+    src/main.cpp
+    src/transcriber.cpp
+    src/transcriber.h
+    src/win-dictation.rc
+)
+
+target_link_libraries(win-dictation PRIVATE
+    whisper
+    ${SDL2_LIBRARIES}
+    comctl32
+    dwmapi
+)
+
+target_include_directories(win-dictation PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/src
+    ${CMAKE_CURRENT_SOURCE_DIR}/common
+)
+
+target_compile_definitions(win-dictation PRIVATE UNICODE _UNICODE)
+
+# ----------------------------
+# Tests
+# ----------------------------
+add_executable(test-core
+    tests/test_core.cpp
+    src/transcriber.cpp
+    src/transcriber.h
+)
+target_link_libraries(test-core PRIVATE whisper ${SDL2_LIBRARIES})
+target_include_directories(test-core PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/src
+    ${CMAKE_CURRENT_SOURCE_DIR}/common
+)
+target_compile_definitions(test-core PRIVATE UNICODE _UNICODE)
+
+# ----------------------------
+# MSVC optimisations
+# ----------------------------
+if(MSVC)
+    target_compile_options(win-dictation PRIVATE
+        $<$<CONFIG:Release>:/O2 /GL>
+    )
+    target_link_options(win-dictation PRIVATE
+        $<$<CONFIG:Release>:/LTCG>
+    )
+endif()
+
+# ----------------------------
+# Post build: runtime assets
+# ----------------------------
+if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/deps/SDL2-2.28.5/lib/x64/SDL2.dll")
+    add_custom_command(TARGET win-dictation POST_BUILD
+        COMMAND ${CMAKE_COMMAND} -E copy_if_different
+            "${CMAKE_CURRENT_SOURCE_DIR}/deps/SDL2-2.28.5/lib/x64/SDL2.dll"
+            $<TARGET_FILE_DIR:win-dictation>
+    )
+elseif(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/release/WinDictation/SDL2.dll")
+    add_custom_command(TARGET win-dictation POST_BUILD
+        COMMAND ${CMAKE_COMMAND} -E copy_if_different
+            "${CMAKE_CURRENT_SOURCE_DIR}/release/WinDictation/SDL2.dll"
+            $<TARGET_FILE_DIR:win-dictation>
+    )
+endif()
+
+add_custom_command(TARGET win-dictation POST_BUILD
+    COMMAND ${CMAKE_COMMAND} -E make_directory $<TARGET_FILE_DIR:win-dictation>/models
+    COMMAND ${CMAKE_COMMAND} -E copy_directory
+        "${CMAKE_CURRENT_SOURCE_DIR}/models"
+        $<TARGET_FILE_DIR:win-dictation>/models
+)

FINDINGS-FIXES-TESTS.md

@@ -0,0 +1,425 @@
+# Win Dictation — Findings, Fixes & Test Guide
+
+A consolidated record of everything diagnosed and changed while turning the slow,
+crashing build into a fast, stable push-to-talk dictation tool. Four parts:
+
+1. **Findings** — what was actually wrong, in the order we discovered it.
+2. **Fixes required** — the concrete code changes, by file.
+3. **Reasoning** — why each fix is correct (the non-obvious calls).
+4. **How to build & run tests** — making the core testable, plus regression tests for every bug here.
+
+Target machine: Dell, Intel Core i5 7th-gen (2 cores / 4 threads), 16 GB RAM, CPU-only.
+Stack: native C++ / Win32 + SDL2 + whisper.cpp (`examples/win-dictation`).
+
+---
+
+## Part 1 — Findings
+
+### F1. The original app was slow because of its *architecture*, not its model
+The first build used whisper.cpp's live-streaming pattern: it re-ran `whisper_full` on a
+rolling 5–6 second window every ~0.4 s (`length_ms` / `step_ms` in `worker_loop`). To keep
+up, the CPU must transcribe ~6 s of audio in well under 1 s — i.e. **>6× real-time**.
+
+- The README's "10–15× real-time" numbers were measured on a **24-thread** box.
+- A 2-core i5-7th-gen does tiny.en at roughly **2–4× real-time**.
+
+So each 6 s window took ~1.5–2.5 s while a new one was demanded every 0.4 s → the buffer
+backed up without bound, audio was re-transcribed in overlapping chunks (duplicated/garbled
+text), the CPU pegged, and 4 Whisper threads on 2 physical cores starved the UI. tiny.en was
+already the right model — **the streaming design was the bottleneck.**
+
+**Conclusion:** for dictation (not live captioning), switch to **batch / push-to-talk**:
+buffer audio cheaply while recording, run `whisper_full` exactly **once** on Stop.
+
+### F2. Build 1 crashed on Stop — null Whisper context
+After the batch rewrite, the app recorded fine (VU moved) but crashed the instant you hit
+Stop. Two defects combined:
+
+1. The preload thread set `g_modelLoaded = true` **even when `preload()` returned false**
+   (model file not found), so the UI showed "Ready" and let you record with no model.
+2. `transcribe_worker()` called `whisper_reset_timings(m_ctx)` **before** the `if (m_ctx)`
+   check. With `m_ctx == nullptr` that's an access violation. Recording never touches Whisper
+   — which is exactly why capture/VU worked and only Stop crashed.
+
+The empty model dropdown in the first screenshot was the tell: no model files were found at
+the new exe-relative path, so `preload()` failed silently.
+
+### F3. Build 1 — window wouldn't resize, and looked boxy
+- The style was `WS_POPUP | WS_CAPTION | WS_SYSMENU` — **no sizing border** — and there was
+  **no `WM_SIZE` handler**, so controls never reflowed. Frozen at ~360×200, Copy/Paste cut off.
+- The gray 3-D system buttons + chunky title bar were the "boxy Windows" look the brief
+  wanted to avoid.
+
+### F4. Build 2 — startup data race (found and fixed by the user)
+`RefreshModelList()` calls `CB_SETCURSEL`, which fires `CBN_SELCHANGE`. That spawned a
+`reload()` background thread that raced the initial `preload()` thread — both writing
+`m_cfg` (a `std::string`) and `m_ctx` at once. The corrupted context pointer made
+`whisper_full` crash ~15 s into processing. Fixed with a `g_initializing` guard (skip the
+combo handler during startup) and an `m_cfg_mtx` mutex around `m_cfg`/`m_ctx` writes.
+
+### F5. Build 3 — the "crash after transcribing" was **not a crash**
+Reported: "crashes after transcribing; relaunching the exe shows the last transcription, and
+a second run replaced the first instead of appending."
+
+That behavior is **impossible for a crashed process** — a dead process can't remember the
+last transcription (nothing is persisted to disk/registry), and a *fresh* launch couldn't
+show it. The only explanation: **the original process never died.**
+
+What actually happened:
+1. On a successful result, `WM_APP_RESULT` ran `ShowWindow(hWnd, SW_HIDE)` in the auto-paste
+   branch → the window **vanished** (looked like a crash).
+2. Double-clicking the `.exe` again hit the **single-instance guard**, which
+   `PostMessage(WM_APP_SHOW)` to the existing **hidden** window → it re-appeared, still holding
+   the last transcription.
+3. `SetDlgItemTextW` **replaced** the edit text, so run #2 overwrote run #1.
+
+Also: clicking the **Record button** (vs. the global hotkey) set `g_prevForeground` to the
+dictation window *itself*, so it hid and "pasted" into its own read-only box — nothing landed
+in a useful app.
+
+10-second confirmation: when it "crashes," the tray icon is still present and
+`win-dictation.exe` is still in Task Manager.
+
+### F6. The test harness is stale and never built
+`src/test-audio.cpp` calls `m_transcriber.init(...)` and `is_using_gpu()` — methods that the
+batch rewrite **removed** (`transcriber.h` now exposes `preload`/`reload`, no `init`). So it
+won't compile against the current code. And the **root** `CMakeLists.txt` (the one `build.ps1`
+actually uses) defines only the `win-dictation` target — the `test-audio` target lives only in
+the unused `src/CMakeLists.txt`. So there are currently **no working tests**. Part 4 fixes this.
+
+---
+
+## Part 2 — Fixes required
+
+Status legend: ✅ done in current code · ⬜ still to apply.
+
+### `transcriber.cpp` / `transcriber.h`
+
+- ✅ **Batch architecture** — `start_recording()` buffers audio cheaply; `stop_and_transcribe()`
+  runs `whisper_full` once on a worker thread; result delivered via `PostMessage` (F1).
+- ✅ **Null-context guard** in `transcribe_worker` — `if (!m_ctx) { m_busy=false; if(m_on_result) m_on_result(""); return; }`,
+  and the `whisper_reset_timings` call removed (F2).
+- ✅ **`reload()`** frees + re-inits under a mutex; **`m_cfg_mtx`** protects `m_cfg`/`m_ctx`;
+  **`threads()`** getter (F4 + thread-count display).
+- ⬜ **Refactor for testability** — extract the inference core so it can be called
+  synchronously from a test (see Part 4):
+
+  ```cpp
+  // transcriber.h  (public)
+  std::string transcribe_sync(std::vector<float> audio);   // headless / tests
+  // transcriber.h  (private)
+  std::string run_inference(std::vector<float>& audio);
+  ```
+
+  ```cpp
+  // transcriber.cpp
+  std::string Transcriber::run_inference(std::vector<float>& audio) {
+      if (!m_ctx) return "";
+      if (m_cfg.trim_silence) trim_silence(audio);
+      whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
+      wp.print_progress = wp.print_realtime = wp.print_timestamps = false;
+      wp.no_timestamps = true; wp.translate = false;
+      wp.language = m_cfg.language.c_str();
+      wp.n_threads = m_cfg.n_threads;
+      wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true;
+      wp.temperature = 0.0f;
+      std::string out;
+      if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
+          int n = whisper_full_n_segments(m_ctx);
+          for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
+          out = clean_text(out);
+      }
+      return out;
+  }
+  void Transcriber::transcribe_worker(std::vector<float> audio) {
+      std::string out = run_inference(audio);
+      m_busy = false;
+      if (m_on_result) m_on_result(out);
+  }
+  std::string Transcriber::transcribe_sync(std::vector<float> audio) {
+      return run_inference(audio);   // assumes preload() already succeeded
+  }
+  ```
+
+### `main.cpp`
+
+- ✅ **Honest load state** — separate `g_modelOk` atomic; record is gated and shows
+  "Model not found: …\models\ggml-tiny.en.bin" instead of crashing (F2).
+- ✅ **Resizable** — `WS_OVERLAPPEDWINDOW` (has `WS_THICKFRAME`), `WM_SIZE → LayoutControls`,
+  `WM_GETMINMAXINFO` min size (F3).
+- ✅ **Startup race guard** — `g_initializing` skips the model-combo handler during startup (F4).
+- ⬜ **`WM_APP_RESULT` rewrite** — the F5 fix: **don't hide**, **append** (not replace),
+  clipboard = latest utterance, paste only into a *different* valid window:
+
+  ```cpp
+  case WM_APP_RESULT:
+  {
+      std::string* res = (std::string*)wParam;
+      if (res && !res->empty()) {
+          HWND hEdit = GetDlgItem(hWnd, ID_EDIT_TEXT);
+          int len = GetWindowTextLengthW(hEdit);
+          std::wstring cur;
+          if (len > 0) { cur.resize(len + 1); int g = GetWindowTextW(hEdit, &cur[0], len + 1); cur.resize(g); }
+          std::wstring combined = cur;
+          if (!combined.empty()) combined += L" ";
+          combined += to_w(*res);
+          SetWindowTextW(hEdit, combined.c_str());
+          SendMessageW(hEdit, EM_SETSEL, (WPARAM)combined.size(), (LPARAM)combined.size());
+          SendMessageW(hEdit, EM_SCROLLCARET, 0, 0);
+          UpdatePlaceholder(hWnd);
+
+          SetClipboardTextUtf8(hWnd, *res);   // clipboard = just the latest utterance
+
+          bool pasted = false;
+          if (g_autoPaste && g_prevForeground &&
+              g_prevForeground != hWnd && IsWindow(g_prevForeground)) {
+              PasteIntoWindow(g_prevForeground);
+              pasted = true;
+          }
+          SetStatus(hWnd, pasted ? L"Pasted ✓" : L"Copied ✓");
+      } else {
+          SetStatus(hWnd, L"No speech detected");
+      }
+      delete res;
+  }
+  break;
+  ```
+
+- ⬜ **Clear button** — `ID_BTN_CLEAR` is handled in `WM_COMMAND` but no button is created in
+  the modern layout, so the (now appending) transcript can't be cleared. Create it owner-drawn,
+  place it in `LayoutControls`, and call `UpdatePlaceholder(hWnd)` after clearing.
+- ⬜ **Optional `g_autoHide`** (default `false`) — only if you want the window to tuck away after
+  a successful paste into another app: `if (pasted && g_autoHide) ShowWindow(hWnd, SW_HIDE);`
+
+### Modern UI (separate doc)
+The flat dark restyle (GDI+ owner-drawn buttons, rounded panel, custom VU, dark caption +
+rounded corners) is fully specified in **`MODERN-UI-AND-FIXES.md`**, Part 2.
+
+---
+
+## Part 3 — Reasoning
+
+**Why batch beats streaming here (F1).** Streaming only wins if you need words *as you speak*
+(live captioning). Dictation doesn't: you speak a sentence, then want the text. Batch removes
+the real-time deadline entirely — Whisper processes each second of audio exactly once, at its
+own pace, after you stop. That means: near-zero CPU while speaking (just buffering), predictable
+"a few seconds after Stop" latency that **doesn't grow with clip length**, no overlap
+re-processing, and better accuracy (full context, no chunk-boundary word splits). On 2 cores
+this is the difference between unusable and snappy.
+
+**Why threads = physical cores (2), not logical (4).** Whisper's matmuls are memory-bandwidth
+bound. Two extra hyperthreads share the same execution ports and cache, so they add little
+throughput while stealing cycles from the UI/audio threads (janky window, laggy VU). Two
+threads leaves headroom for a responsive UI.
+
+**Why the Stop crash was a null deref (F2), and why the guard is the right fix.** Recording is
+pure SDL + a `std::vector` append — it never calls into Whisper, which is why only Stop crashed.
+`whisper_reset_timings(nullptr)` dereferences the context. The guard makes the worker a no-op
+when no model is loaded; the separate `g_modelOk` flag makes that state *visible* (a MessageBox
+with the path) instead of letting you walk into it. Defensive + diagnostic.
+
+**Why the "crash after transcribing" was actually a hide (F5) — the logic is conclusive.** A
+crashed process loses all memory. There is no on-disk persistence in this app. Therefore a
+re-launched process **cannot** display the previous transcription, and a second run **cannot**
+replace text in "the same box." The observed behavior (prior text reappears; second replaces
+first) is only possible if it's the *same* live process — which means it hid, and the
+single-instance guard re-showed it. This is why "is the process still in Task Manager?" is the
+decisive test, not "did the window disappear?"
+
+**Why clipboard = latest utterance but the box appends.** Two different jobs. The on-screen box
+is your running log (you want history). The clipboard is what you paste into another app — you
+want *just what you last said*, not the whole session. Splitting them gives both.
+
+**Why paste must exclude our own window.** Auto-paste replays Ctrl+V into the foreground window
+captured before the popup. If you triggered recording by clicking the Record button, that
+"previous" window is the dictation app itself — pasting into its read-only box is a no-op and
+(with the old hide) made the app appear to swallow the text. Restricting to
+`g_prevForeground != hWnd && IsWindow(...)` makes the global-hotkey workflow the reliable path
+and the button a safe "copy only."
+
+---
+
+## Part 4 — How to build & run tests
+
+The goal: catch the bugs above automatically, without a human watching a window. UI behavior
+(resize, hide, paste) needs a short **manual** checklist, but the **core** (model load,
+transcription, null-safety, text append) can and should be tested headlessly.
+
+### 4.1 Make the core testable
+Apply the `run_inference` / `transcribe_sync` refactor from Part 2 so a test can feed a buffer
+and get text synchronously. Also extract the append rule as a pure function so it can be tested
+with zero Win32:
+
+```cpp
+// text_util.h  (new, tiny, UI-free)
+#pragma once
+#include <string>
+inline std::wstring append_transcript(const std::wstring& cur, const std::wstring& add) {
+    if (add.empty()) return cur;
+    if (cur.empty()) return add;
+    return cur + L" " + add;
+}
+```
+Use it in `WM_APP_RESULT` (`combined = append_transcript(cur, to_w(*res));`) so the UI and the
+test exercise the *same* rule.
+
+### 4.2 The test program (`tests/test_core.cpp`)
+Replaces the stale `src/test-audio.cpp`. Compiles against the **current** API and tests the real
+code path (`preload` + `transcribe_sync`).
+
+```cpp
+#include "transcriber.h"
+#include "text_util.h"
+#include <cstdio>
+#include <cstdint>
+#include <fstream>
+#include <vector>
+#include <string>
+#include <cctype>
+
+static int g_fail = 0;
+#define CHECK(cond, msg) do { if (!(cond)) { printf("  FAIL: %s\n", msg); ++g_fail; } \
+                              else            printf("  ok:   %s\n", msg); } while(0)
+
+// Minimal 16-bit PCM mono WAV loader -> float [-1,1]. Assumes 16 kHz mono (whisper.cpp samples are).
+static bool load_wav(const std::string& path, std::vector<float>& out) {
+    std::ifstream f(path, std::ios::binary);
+    if (!f) return false;
+    char hdr[44];
+    f.read(hdr, 44);
+    if (std::string(hdr, 4) != "RIFF") return false;
+    std::vector<int16_t> pcm((std::istreambuf_iterator<char>(f)), {});  // crude: rest of file
+    out.clear(); out.reserve(pcm.size());
+    for (int16_t s : pcm) out.push_back(s / 32768.0f);
+    return !out.empty();
+}
+
+static std::string lower(std::string s) { for (char& c : s) c = (char)tolower((unsigned char)c); return s; }
+
+int main(int argc, char** argv) {
+    std::string model = (argc > 1) ? argv[1] : "models/ggml-tiny.en.bin";
+    std::string wav   = (argc > 2) ? argv[2] : "samples/jfk.wav";
+
+    // --- Pure logic: append rule (no model needed) ---
+    CHECK(append_transcript(L"", L"hello") == L"hello",        "append: empty + a");
+    CHECK(append_transcript(L"hello", L"world") == L"hello world", "append: a + b spaced");
+    CHECK(append_transcript(L"hello", L"") == L"hello",        "append: a + empty");
+
+    // --- Regression F2: bad model path must NOT crash, returns "" ---
+    {
+        Transcriber t; WhisperConfig bad; bad.model_path = "models\\does-not-exist.bin";
+        bool ok = t.preload(bad);
+        CHECK(!ok, "bad model path: preload returns false");
+        std::vector<float> a(16000, 0.0f);               // 1 s of silence
+        std::string r = t.transcribe_sync(a);            // must be a safe no-op
+        CHECK(r.empty(), "bad model path: transcribe_sync returns empty, no crash");
+    }
+
+    // --- Happy path: real model + known clip -> non-empty, expected words ---
+    {
+        Transcriber t; WhisperConfig cfg; cfg.model_path = model; cfg.n_threads = 4;
+        bool ok = t.preload(cfg);
+        CHECK(ok, "model loads");
+        if (ok) {
+            std::vector<float> audio;
+            bool loaded = load_wav(wav, audio);
+            CHECK(loaded, "wav loads");
+            if (loaded) {
+                std::string text = lower(t.transcribe_sync(audio));
+                CHECK(!text.empty(), "transcription is non-empty");
+                // jfk.wav: "...ask not what your country can do for you..."
+                CHECK(text.find("country") != std::string::npos, "transcription contains 'country'");
+            }
+        }
+    }
+
+    // --- Edge: sub-300ms / empty audio -> "" , no crash ---
+    {
+        Transcriber t; WhisperConfig cfg; cfg.model_path = model;
+        if (t.preload(cfg)) {
+            std::vector<float> tiny(100, 0.1f);
+            std::string r = t.transcribe_sync(tiny);
+            CHECK(true, "short audio did not crash");   // reaching here = no crash
+        }
+    }
+
+    printf("\n%s (%d failure%s)\n", g_fail ? "TESTS FAILED" : "ALL TESTS PASSED",
+           g_fail, g_fail == 1 ? "" : "s");
+    return g_fail ? 1 : 0;
+}
+```
+
+> The WAV loader above is deliberately minimal (good enough for whisper.cpp's 16 kHz mono
+> `samples/*.wav`). If you test arbitrary WAVs, parse the `fmt `/`data` chunks properly.
+
+### 4.3 Build the tests (root `CMakeLists.txt`)
+The active build file builds only `win-dictation`. Add a test target next to it:
+
+```cmake
+# --- tests ---
+add_executable(test-core
+    tests/test_core.cpp
+    src/transcriber.cpp
+    src/transcriber.h
+)
+target_link_libraries(test-core PRIVATE whisper ${SDL2_LIBRARIES})
+target_include_directories(test-core PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/src
+    ${CMAKE_CURRENT_SOURCE_DIR}/common
+)
+target_compile_definitions(test-core PRIVATE UNICODE _UNICODE)
+```
+(`transcriber.cpp` references SDL symbols even though the test never opens a device, so the test
+still links `${SDL2_LIBRARIES}`.)
+
+Build and run:
+```powershell
+cmake --build build --config Release --target test-core
+# run from a dir where SDL2.dll + models/ + samples/ are reachable:
+cd build\bin\Release
+copy ..\..\..\deps\SDL2-2.28.5\lib\x64\SDL2.dll .   # if not already beside the exe
+.\test-core.exe models\ggml-tiny.en.bin ..\..\..\samples\jfk.wav
+```
+Exit code `0` = all passed (usable in CI / a pre-commit hook).
+
+### 4.4 What each test guards
+| Test | Guards against |
+|---|---|
+| append rule (3 cases) | F5 regression — replace-instead-of-append |
+| bad model path → false + empty | F2 — null-context crash on Stop |
+| model loads / clip → non-empty, has "country" | core transcription works end to end |
+| short/empty audio → no crash | the `< 0.3 s` guard + null-safety |
+
+### 4.5 Manual UI checklist (can't be unit-tested)
+- [ ] **Not a crash:** after Stop, the window **stays visible**; if it ever vanishes, the tray
+      icon / `win-dictation.exe` in Task Manager confirms whether it's hiding vs. truly gone.
+- [ ] **Append:** two dictations in a row → second is appended after the first, view scrolls down.
+- [ ] **Clipboard:** after a dictation, paste into Notepad → only the *latest* utterance appears.
+- [ ] **Auto-paste:** focus Notepad, press the global hotkey, speak, press it again → text lands
+      in Notepad. Triggering via the Record button instead → "Copied ✓" (no self-paste).
+- [ ] **Resize:** drag edges; transcript grows; min size respected.
+- [ ] **Performance:** a ~10 s clip transcribes in a few seconds and the UI stays responsive;
+      latency does **not** grow with longer recordings.
+
+### 4.6 Performance smoke test (optional)
+`test-core` can also print timing — wrap `transcribe_sync` in `std::chrono::steady_clock` and
+assert `elapsed < audio_seconds * k` for a sanity bound (e.g. `k = 1.0`, i.e. faster than
+real-time on tiny.en). Useful to catch a future regression that quietly reintroduces the
+streaming-style slowdown.
+
+---
+
+## Part 5 — Status & remaining work
+
+**Verified fixed:** F1 (batch), F2 (null-context crash), F3 (resize), F4 (startup race).
+
+**Apply next (⬜ in Part 2):**
+1. `run_inference` / `transcribe_sync` refactor (unblocks tests).
+2. `WM_APP_RESULT` rewrite (the F5 fix — no-hide + append + safe paste). **Highest priority** —
+   it's what's making the app *look* like it crashes.
+3. Add the Clear button (needed now that text appends).
+4. Add `tests/test_core.cpp` + the CMake target; run it.
+
+**Then re-run** the Part 4 manual checklist. Expected result: the window no longer disappears,
+transcriptions append, the clipboard holds the latest utterance, and `test-core.exe` exits `0`.
+

FINDINGS-FIXES-TESTS2.md

@@ -0,0 +1,425 @@
+# Win Dictation — Findings, Fixes & Test Guide
+
+A consolidated record of everything diagnosed and changed while turning the slow,
+crashing build into a fast, stable push-to-talk dictation tool. Four parts:
+
+1. **Findings** — what was actually wrong, in the order we discovered it.
+2. **Fixes required** — the concrete code changes, by file.
+3. **Reasoning** — why each fix is correct (the non-obvious calls).
+4. **How to build & run tests** — making the core testable, plus regression tests for every bug here.
+
+Target machine: Dell, Intel Core i5 7th-gen (2 cores / 4 threads), 16 GB RAM, CPU-only.
+Stack: native C++ / Win32 + SDL2 + whisper.cpp (`examples/win-dictation`).
+
+---
+
+## Part 1 — Findings
+
+### F1. The original app was slow because of its *architecture*, not its model
+The first build used whisper.cpp's live-streaming pattern: it re-ran `whisper_full` on a
+rolling 5–6 second window every ~0.4 s (`length_ms` / `step_ms` in `worker_loop`). To keep
+up, the CPU must transcribe ~6 s of audio in well under 1 s — i.e. **>6× real-time**.
+
+- The README's "10–15× real-time" numbers were measured on a **24-thread** box.
+- A 2-core i5-7th-gen does tiny.en at roughly **2–4× real-time**.
+
+So each 6 s window took ~1.5–2.5 s while a new one was demanded every 0.4 s → the buffer
+backed up without bound, audio was re-transcribed in overlapping chunks (duplicated/garbled
+text), the CPU pegged, and 4 Whisper threads on 2 physical cores starved the UI. tiny.en was
+already the right model — **the streaming design was the bottleneck.**
+
+**Conclusion:** for dictation (not live captioning), switch to **batch / push-to-talk**:
+buffer audio cheaply while recording, run `whisper_full` exactly **once** on Stop.
+
+### F2. Build 1 crashed on Stop — null Whisper context
+After the batch rewrite, the app recorded fine (VU moved) but crashed the instant you hit
+Stop. Two defects combined:
+
+1. The preload thread set `g_modelLoaded = true` **even when `preload()` returned false**
+   (model file not found), so the UI showed "Ready" and let you record with no model.
+2. `transcribe_worker()` called `whisper_reset_timings(m_ctx)` **before** the `if (m_ctx)`
+   check. With `m_ctx == nullptr` that's an access violation. Recording never touches Whisper
+   — which is exactly why capture/VU worked and only Stop crashed.
+
+The empty model dropdown in the first screenshot was the tell: no model files were found at
+the new exe-relative path, so `preload()` failed silently.
+
+### F3. Build 1 — window wouldn't resize, and looked boxy
+- The style was `WS_POPUP | WS_CAPTION | WS_SYSMENU` — **no sizing border** — and there was
+  **no `WM_SIZE` handler**, so controls never reflowed. Frozen at ~360×200, Copy/Paste cut off.
+- The gray 3-D system buttons + chunky title bar were the "boxy Windows" look the brief
+  wanted to avoid.
+
+### F4. Build 2 — startup data race (found and fixed by the user)
+`RefreshModelList()` calls `CB_SETCURSEL`, which fires `CBN_SELCHANGE`. That spawned a
+`reload()` background thread that raced the initial `preload()` thread — both writing
+`m_cfg` (a `std::string`) and `m_ctx` at once. The corrupted context pointer made
+`whisper_full` crash ~15 s into processing. Fixed with a `g_initializing` guard (skip the
+combo handler during startup) and an `m_cfg_mtx` mutex around `m_cfg`/`m_ctx` writes.
+
+### F5. Build 3 — the "crash after transcribing" was **not a crash**
+Reported: "crashes after transcribing; relaunching the exe shows the last transcription, and
+a second run replaced the first instead of appending."
+
+That behavior is **impossible for a crashed process** — a dead process can't remember the
+last transcription (nothing is persisted to disk/registry), and a *fresh* launch couldn't
+show it. The only explanation: **the original process never died.**
+
+What actually happened:
+1. On a successful result, `WM_APP_RESULT` ran `ShowWindow(hWnd, SW_HIDE)` in the auto-paste
+   branch → the window **vanished** (looked like a crash).
+2. Double-clicking the `.exe` again hit the **single-instance guard**, which
+   `PostMessage(WM_APP_SHOW)` to the existing **hidden** window → it re-appeared, still holding
+   the last transcription.
+3. `SetDlgItemTextW` **replaced** the edit text, so run #2 overwrote run #1.
+
+Also: clicking the **Record button** (vs. the global hotkey) set `g_prevForeground` to the
+dictation window *itself*, so it hid and "pasted" into its own read-only box — nothing landed
+in a useful app.
+
+10-second confirmation: when it "crashes," the tray icon is still present and
+`win-dictation.exe` is still in Task Manager.
+
+### F6. The test harness is stale and never built
+`src/test-audio.cpp` calls `m_transcriber.init(...)` and `is_using_gpu()` — methods that the
+batch rewrite **removed** (`transcriber.h` now exposes `preload`/`reload`, no `init`). So it
+won't compile against the current code. And the **root** `CMakeLists.txt` (the one `build.ps1`
+actually uses) defines only the `win-dictation` target — the `test-audio` target lives only in
+the unused `src/CMakeLists.txt`. So there are currently **no working tests**. Part 4 fixes this.
+
+---
+
+## Part 2 — Fixes required
+
+Status legend: ✅ done in current code · ⬜ still to apply.
+
+### `transcriber.cpp` / `transcriber.h`
+
+- ✅ **Batch architecture** — `start_recording()` buffers audio cheaply; `stop_and_transcribe()`
+  runs `whisper_full` once on a worker thread; result delivered via `PostMessage` (F1).
+- ✅ **Null-context guard** in `transcribe_worker` — `if (!m_ctx) { m_busy=false; if(m_on_result) m_on_result(""); return; }`,
+  and the `whisper_reset_timings` call removed (F2).
+- ✅ **`reload()`** frees + re-inits under a mutex; **`m_cfg_mtx`** protects `m_cfg`/`m_ctx`;
+  **`threads()`** getter (F4 + thread-count display).
+- ⬜ **Refactor for testability** — extract the inference core so it can be called
+  synchronously from a test (see Part 4):
+
+  ```cpp
+  // transcriber.h  (public)
+  std::string transcribe_sync(std::vector<float> audio);   // headless / tests
+  // transcriber.h  (private)
+  std::string run_inference(std::vector<float>& audio);
+  ```
+
+  ```cpp
+  // transcriber.cpp
+  std::string Transcriber::run_inference(std::vector<float>& audio) {
+      if (!m_ctx) return "";
+      if (m_cfg.trim_silence) trim_silence(audio);
+      whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
+      wp.print_progress = wp.print_realtime = wp.print_timestamps = false;
+      wp.no_timestamps = true; wp.translate = false;
+      wp.language = m_cfg.language.c_str();
+      wp.n_threads = m_cfg.n_threads;
+      wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true;
+      wp.temperature = 0.0f;
+      std::string out;
+      if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
+          int n = whisper_full_n_segments(m_ctx);
+          for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
+          out = clean_text(out);
+      }
+      return out;
+  }
+  void Transcriber::transcribe_worker(std::vector<float> audio) {
+      std::string out = run_inference(audio);
+      m_busy = false;
+      if (m_on_result) m_on_result(out);
+  }
+  std::string Transcriber::transcribe_sync(std::vector<float> audio) {
+      return run_inference(audio);   // assumes preload() already succeeded
+  }
+  ```
+
+### `main.cpp`
+
+- ✅ **Honest load state** — separate `g_modelOk` atomic; record is gated and shows
+  "Model not found: …\models\ggml-tiny.en.bin" instead of crashing (F2).
+- ✅ **Resizable** — `WS_OVERLAPPEDWINDOW` (has `WS_THICKFRAME`), `WM_SIZE → LayoutControls`,
+  `WM_GETMINMAXINFO` min size (F3).
+- ✅ **Startup race guard** — `g_initializing` skips the model-combo handler during startup (F4).
+- ⬜ **`WM_APP_RESULT` rewrite** — the F5 fix: **don't hide**, **append** (not replace),
+  clipboard = latest utterance, paste only into a *different* valid window:
+
+  ```cpp
+  case WM_APP_RESULT:
+  {
+      std::string* res = (std::string*)wParam;
+      if (res && !res->empty()) {
+          HWND hEdit = GetDlgItem(hWnd, ID_EDIT_TEXT);
+          int len = GetWindowTextLengthW(hEdit);
+          std::wstring cur;
+          if (len > 0) { cur.resize(len + 1); int g = GetWindowTextW(hEdit, &cur[0], len + 1); cur.resize(g); }
+          std::wstring combined = cur;
+          if (!combined.empty()) combined += L" ";
+          combined += to_w(*res);
+          SetWindowTextW(hEdit, combined.c_str());
+          SendMessageW(hEdit, EM_SETSEL, (WPARAM)combined.size(), (LPARAM)combined.size());
+          SendMessageW(hEdit, EM_SCROLLCARET, 0, 0);
+          UpdatePlaceholder(hWnd);
+
+          SetClipboardTextUtf8(hWnd, *res);   // clipboard = just the latest utterance
+
+          bool pasted = false;
+          if (g_autoPaste && g_prevForeground &&
+              g_prevForeground != hWnd && IsWindow(g_prevForeground)) {
+              PasteIntoWindow(g_prevForeground);
+              pasted = true;
+          }
+          SetStatus(hWnd, pasted ? L"Pasted ✓" : L"Copied ✓");
+      } else {
+          SetStatus(hWnd, L"No speech detected");
+      }
+      delete res;
+  }
+  break;
+  ```
+
+- ⬜ **Clear button** — `ID_BTN_CLEAR` is handled in `WM_COMMAND` but no button is created in
+  the modern layout, so the (now appending) transcript can't be cleared. Create it owner-drawn,
+  place it in `LayoutControls`, and call `UpdatePlaceholder(hWnd)` after clearing.
+- ⬜ **Optional `g_autoHide`** (default `false`) — only if you want the window to tuck away after
+  a successful paste into another app: `if (pasted && g_autoHide) ShowWindow(hWnd, SW_HIDE);`
+
+### Modern UI (separate doc)
+The flat dark restyle (GDI+ owner-drawn buttons, rounded panel, custom VU, dark caption +
+rounded corners) is fully specified in **`MODERN-UI-AND-FIXES.md`**, Part 2.
+
+---
+
+## Part 3 — Reasoning
+
+**Why batch beats streaming here (F1).** Streaming only wins if you need words *as you speak*
+(live captioning). Dictation doesn't: you speak a sentence, then want the text. Batch removes
+the real-time deadline entirely — Whisper processes each second of audio exactly once, at its
+own pace, after you stop. That means: near-zero CPU while speaking (just buffering), predictable
+"a few seconds after Stop" latency that **doesn't grow with clip length**, no overlap
+re-processing, and better accuracy (full context, no chunk-boundary word splits). On 2 cores
+this is the difference between unusable and snappy.
+
+**Why threads = physical cores (2), not logical (4).** Whisper's matmuls are memory-bandwidth
+bound. Two extra hyperthreads share the same execution ports and cache, so they add little
+throughput while stealing cycles from the UI/audio threads (janky window, laggy VU). Two
+threads leaves headroom for a responsive UI.
+
+**Why the Stop crash was a null deref (F2), and why the guard is the right fix.** Recording is
+pure SDL + a `std::vector` append — it never calls into Whisper, which is why only Stop crashed.
+`whisper_reset_timings(nullptr)` dereferences the context. The guard makes the worker a no-op
+when no model is loaded; the separate `g_modelOk` flag makes that state *visible* (a MessageBox
+with the path) instead of letting you walk into it. Defensive + diagnostic.
+
+**Why the "crash after transcribing" was actually a hide (F5) — the logic is conclusive.** A
+crashed process loses all memory. There is no on-disk persistence in this app. Therefore a
+re-launched process **cannot** display the previous transcription, and a second run **cannot**
+replace text in "the same box." The observed behavior (prior text reappears; second replaces
+first) is only possible if it's the *same* live process — which means it hid, and the
+single-instance guard re-showed it. This is why "is the process still in Task Manager?" is the
+decisive test, not "did the window disappear?"
+
+**Why clipboard = latest utterance but the box appends.** Two different jobs. The on-screen box
+is your running log (you want history). The clipboard is what you paste into another app — you
+want *just what you last said*, not the whole session. Splitting them gives both.
+
+**Why paste must exclude our own window.** Auto-paste replays Ctrl+V into the foreground window
+captured before the popup. If you triggered recording by clicking the Record button, that
+"previous" window is the dictation app itself — pasting into its read-only box is a no-op and
+(with the old hide) made the app appear to swallow the text. Restricting to
+`g_prevForeground != hWnd && IsWindow(...)` makes the global-hotkey workflow the reliable path
+and the button a safe "copy only."
+
+---
+
+## Part 4 — How to build & run tests
+
+The goal: catch the bugs above automatically, without a human watching a window. UI behavior
+(resize, hide, paste) needs a short **manual** checklist, but the **core** (model load,
+transcription, null-safety, text append) can and should be tested headlessly.
+
+### 4.1 Make the core testable
+Apply the `run_inference` / `transcribe_sync` refactor from Part 2 so a test can feed a buffer
+and get text synchronously. Also extract the append rule as a pure function so it can be tested
+with zero Win32:
+
+```cpp
+// text_util.h  (new, tiny, UI-free)
+#pragma once
+#include <string>
+inline std::wstring append_transcript(const std::wstring& cur, const std::wstring& add) {
+    if (add.empty()) return cur;
+    if (cur.empty()) return add;
+    return cur + L" " + add;
+}
+```
+Use it in `WM_APP_RESULT` (`combined = append_transcript(cur, to_w(*res));`) so the UI and the
+test exercise the *same* rule.
+
+### 4.2 The test program (`tests/test_core.cpp`)
+Replaces the stale `src/test-audio.cpp`. Compiles against the **current** API and tests the real
+code path (`preload` + `transcribe_sync`).
+
+```cpp
+#include "transcriber.h"
+#include "text_util.h"
+#include <cstdio>
+#include <cstdint>
+#include <fstream>
+#include <vector>
+#include <string>
+#include <cctype>
+
+static int g_fail = 0;
+#define CHECK(cond, msg) do { if (!(cond)) { printf("  FAIL: %s\n", msg); ++g_fail; } \
+                              else            printf("  ok:   %s\n", msg); } while(0)
+
+// Minimal 16-bit PCM mono WAV loader -> float [-1,1]. Assumes 16 kHz mono (whisper.cpp samples are).
+static bool load_wav(const std::string& path, std::vector<float>& out) {
+    std::ifstream f(path, std::ios::binary);
+    if (!f) return false;
+    char hdr[44];
+    f.read(hdr, 44);
+    if (std::string(hdr, 4) != "RIFF") return false;
+    std::vector<int16_t> pcm((std::istreambuf_iterator<char>(f)), {});  // crude: rest of file
+    out.clear(); out.reserve(pcm.size());
+    for (int16_t s : pcm) out.push_back(s / 32768.0f);
+    return !out.empty();
+}
+
+static std::string lower(std::string s) { for (char& c : s) c = (char)tolower((unsigned char)c); return s; }
+
+int main(int argc, char** argv) {
+    std::string model = (argc > 1) ? argv[1] : "models/ggml-tiny.en.bin";
+    std::string wav   = (argc > 2) ? argv[2] : "samples/jfk.wav";
+
+    // --- Pure logic: append rule (no model needed) ---
+    CHECK(append_transcript(L"", L"hello") == L"hello",        "append: empty + a");
+    CHECK(append_transcript(L"hello", L"world") == L"hello world", "append: a + b spaced");
+    CHECK(append_transcript(L"hello", L"") == L"hello",        "append: a + empty");
+
+    // --- Regression F2: bad model path must NOT crash, returns "" ---
+    {
+        Transcriber t; WhisperConfig bad; bad.model_path = "models\\does-not-exist.bin";
+        bool ok = t.preload(bad);
+        CHECK(!ok, "bad model path: preload returns false");
+        std::vector<float> a(16000, 0.0f);               // 1 s of silence
+        std::string r = t.transcribe_sync(a);            // must be a safe no-op
+        CHECK(r.empty(), "bad model path: transcribe_sync returns empty, no crash");
+    }
+
+    // --- Happy path: real model + known clip -> non-empty, expected words ---
+    {
+        Transcriber t; WhisperConfig cfg; cfg.model_path = model; cfg.n_threads = 4;
+        bool ok = t.preload(cfg);
+        CHECK(ok, "model loads");
+        if (ok) {
+            std::vector<float> audio;
+            bool loaded = load_wav(wav, audio);
+            CHECK(loaded, "wav loads");
+            if (loaded) {
+                std::string text = lower(t.transcribe_sync(audio));
+                CHECK(!text.empty(), "transcription is non-empty");
+                // jfk.wav: "...ask not what your country can do for you..."
+                CHECK(text.find("country") != std::string::npos, "transcription contains 'country'");
+            }
+        }
+    }
+
+    // --- Edge: sub-300ms / empty audio -> "" , no crash ---
+    {
+        Transcriber t; WhisperConfig cfg; cfg.model_path = model;
+        if (t.preload(cfg)) {
+            std::vector<float> tiny(100, 0.1f);
+            std::string r = t.transcribe_sync(tiny);
+            CHECK(true, "short audio did not crash");   // reaching here = no crash
+        }
+    }
+
+    printf("\n%s (%d failure%s)\n", g_fail ? "TESTS FAILED" : "ALL TESTS PASSED",
+           g_fail, g_fail == 1 ? "" : "s");
+    return g_fail ? 1 : 0;
+}
+```
+
+> The WAV loader above is deliberately minimal (good enough for whisper.cpp's 16 kHz mono
+> `samples/*.wav`). If you test arbitrary WAVs, parse the `fmt `/`data` chunks properly.
+
+### 4.3 Build the tests (root `CMakeLists.txt`)
+The active build file builds only `win-dictation`. Add a test target next to it:
+
+```cmake
+# --- tests ---
+add_executable(test-core
+    tests/test_core.cpp
+    src/transcriber.cpp
+    src/transcriber.h
+)
+target_link_libraries(test-core PRIVATE whisper ${SDL2_LIBRARIES})
+target_include_directories(test-core PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/src
+    ${CMAKE_CURRENT_SOURCE_DIR}/common
+)
+target_compile_definitions(test-core PRIVATE UNICODE _UNICODE)
+```
+(`transcriber.cpp` references SDL symbols even though the test never opens a device, so the test
+still links `${SDL2_LIBRARIES}`.)
+
+Build and run:
+```powershell
+cmake --build build --config Release --target test-core
+# run from a dir where SDL2.dll + models/ + samples/ are reachable:
+cd build\bin\Release
+copy ..\..\..\deps\SDL2-2.28.5\lib\x64\SDL2.dll .   # if not already beside the exe
+.\test-core.exe models\ggml-tiny.en.bin ..\..\..\samples\jfk.wav
+```
+Exit code `0` = all passed (usable in CI / a pre-commit hook).
+
+### 4.4 What each test guards
+| Test | Guards against |
+|---|---|
+| append rule (3 cases) | F5 regression — replace-instead-of-append |
+| bad model path → false + empty | F2 — null-context crash on Stop |
+| model loads / clip → non-empty, has "country" | core transcription works end to end |
+| short/empty audio → no crash | the `< 0.3 s` guard + null-safety |
+
+### 4.5 Manual UI checklist (can't be unit-tested)
+- [ ] **Not a crash:** after Stop, the window **stays visible**; if it ever vanishes, the tray
+      icon / `win-dictation.exe` in Task Manager confirms whether it's hiding vs. truly gone.
+- [ ] **Append:** two dictations in a row → second is appended after the first, view scrolls down.
+- [ ] **Clipboard:** after a dictation, paste into Notepad → only the *latest* utterance appears.
+- [ ] **Auto-paste:** focus Notepad, press the global hotkey, speak, press it again → text lands
+      in Notepad. Triggering via the Record button instead → "Copied ✓" (no self-paste).
+- [ ] **Resize:** drag edges; transcript grows; min size respected.
+- [ ] **Performance:** a ~10 s clip transcribes in a few seconds and the UI stays responsive;
+      latency does **not** grow with longer recordings.
+
+### 4.6 Performance smoke test (optional)
+`test-core` can also print timing — wrap `transcribe_sync` in `std::chrono::steady_clock` and
+assert `elapsed < audio_seconds * k` for a sanity bound (e.g. `k = 1.0`, i.e. faster than
+real-time on tiny.en). Useful to catch a future regression that quietly reintroduces the
+streaming-style slowdown.
+
+---
+
+## Part 5 — Status & remaining work
+
+**Verified fixed:** F1 (batch), F2 (null-context crash), F3 (resize), F4 (startup race).
+
+**Apply next (⬜ in Part 2):**
+1. `run_inference` / `transcribe_sync` refactor (unblocks tests).
+2. `WM_APP_RESULT` rewrite (the F5 fix — no-hide + append + safe paste). **Highest priority** —
+   it's what's making the app *look* like it crashes.
+3. Add the Clear button (needed now that text appends).
+4. Add `tests/test_core.cpp` + the CMake target; run it.
+
+**Then re-run** the Part 4 manual checklist. Expected result: the window no longer disappears,
+transcriptions append, the clipboard holds the latest utterance, and `test-core.exe` exits `0`.
+

README.md

@@ -0,0 +1,115 @@
+# Win Dictation - AI Voice to Text for Windows
+
+A push-to-talk speech-to-text utility for Windows using OpenAI's Whisper model. Record, transcribe, and paste with a single hotkey.
+
+![Win Dictation Screenshot](screenshot.png)
+
+## Quick Download
+
+**[Download Latest Release (WinDictation.zip)](release/WinDictation.zip)**
+
+Extract the ZIP file and run `win-dictation.exe`. The release includes all required DLLs and a Whisper model.
+
+---
+
+## Features
+
+### Performance
+- **Physical-core threading**: Uses one thread per physical core for efficient batch transcription
+- **CPU-only**: Optimised for the target Intel i5-7th-gen 2-core/4-thread machine
+- **Smart model selection**: Auto-selects tiny.en for CPU-only, base.en when GPU is present
+- **Self-calibrating progress**: Learns transcription speed per model and machine, delivering a smooth countdown
+
+### User Interface
+- **Single-surface rendering**: No inter-window seams or hairlines — the entire UI is one painted surface
+- **Dark theme**: Calm, elevated card design with hover/press feedback
+- **Per-monitor DPI awareness**: Looks sharp at any display scale
+- **System tray**: Minimise to tray, global hotkey to record
+
+### Audio Processing
+- **Push-to-talk**: Press Ctrl+Shift+Space, speak, press again to transcribe
+- **500ms auto-end**: Stops recording after 500ms of silence
+- **Multiple microphones**: Select from all available input devices
+- **16kHz sample rate**: Optimised for Whisper
+
+## Usage
+
+### Controls
+- **Record**: Click the Record pill or press `Ctrl+Shift+Space`
+- **Pin**: Keep window always-on-top
+- **Copy / Paste / Clear**: Text actions
+- **Model / Mic**: Select from popup menus
+- **Hide**: `Ctrl+Shift+H` hides the window
+
+### Indicators
+- **Level**: Live audio energy during recording
+- **Progress bar**: Smooth, counting-down estimate during transcription
+- **Status**: Thread count at idle, elapsed time during recording
+
+## Building from Source
+
+### Prerequisites
+
+- **Windows 10/11**
+- **CMake** 3.5+
+- **Visual Studio 2022/2026** with C++ workload
+- **SDL2** (included in deps/)
+
+### Build Steps
+
+```powershell
+cmake -S . -B build -G "Visual Studio 18 2026" \
+  -DSDL2_DIR="deps/SDL2-2.28.5/cmake"
+cmake --build build --config Release
+```
+
+The executable will be at `build\bin\Release\win-dictation.exe`.
+
+## Technical Details
+
+### Architecture
+
+```
+Hotkey → SDL Capture → Stop → Batch whisper_full → Text → Auto-paste
+```
+
+1. **SDL audio capture**: 16kHz mono recording into memory
+2. **Stop-and-transcribe**: Press stop or hit max length (600s), then one `whisper_full` call
+3. **Progress estimation**: Linear model fitted per machine/model, fused with whisper's chunk progress
+4. **Text output**: Appended to transcript, copied to clipboard, optionally auto-pasted
+
+### Model
+
+Place `.bin` files in `models/` next to the executable. The app auto-detects available models:
+
+| Model | Size | Params | Best for |
+|-------|------|--------|----------|
+| tiny.en | 75 MB | 39M | CPU-only systems |
+| base.en | 140 MB | 74M | GPU-accelerated systems |
+
+## Project Structure
+
+```
+win-dictation/
+├── src/              # Application source
+│   ├── main.cpp      # UI and message handling
+│   ├── transcriber.* # Recording and transcription
+│   ├── timing.h      # Progress estimation engine
+│   ├── settings.h    # INI persistence
+│   ├── text_util.h   # Transcript helpers
+│   └── logging.h     # Log utilities
+├── whisper/          # Whisper.cpp library
+├── ggml/             # GGML tensor library
+├── models/           # Whisper model files
+├── release/          # Pre-built package
+└── CMakeLists.txt    # Build configuration
+```
+
+## License
+
+MIT — follows [whisper.cpp](https://github.com/ggerganov/whisper.cpp).
+
+## Resources
+
+- [Whisper.cpp](https://github.com/ggerganov/whisper.cpp)
+- [Model Download](https://huggingface.co/ggerganov/whisper.cpp)

UI-and-progress-bar-rebuild-part-2.md

@@ -0,0 +1,78 @@
+# Win Dictation — Fix Note 01
+
+**Every control renders as the blue Record pill**
+
+**Applies to:** the single-surface UI rewrite described in `UI-and-Progress-Rebuild.md` (Part 1), after the first build.
+**Status:** root cause confirmed, one-line fix below.
+
+---
+
+## Symptom
+
+After wiring up the single-surface paint, every control — **Record**, **Pin**, the **mic** and **model** selects, and **Copy / Paste / Clear** — renders as the *same* blue accent pill with a white dot. No labels, no distinct styles. (The window chrome, card, and scrollbar are correct; only the widgets are wrong.)
+
+## The mistake
+
+`Widget::kind` is declared without an initializer, and the widget array has static storage duration, so the whole array is zero-initialized:
+
+```cpp
+struct Widget {
+    WK     kind;        // <-- no initializer
+    RectF  r;
+    bool   hover = false;
+    bool   pressed = false;
+    float  anim = 0.0f;
+};
+static Widget g_w[ (int)WK::Transcript + 1 ];   // zero-filled → every kind == 0
+```
+
+`WK::RecordHero` is the first enumerator, i.e. value `0`. So **every** slot's `kind` reads as `RecordHero`. `LayoutWidgets()` assigns each slot's *rectangle* (`g_w[(int)WK::Pin].r = …`, etc.) but never its `kind`. The paint loop dispatches on `kind`:
+
+```cpp
+for (auto& w : g_w) {
+    switch (w.kind) {                 // 0 for every widget
+        case WK::RecordHero: DrawHero(g, w);          break;   // <-- all eight land here
+        case WK::Copy:       DrawGhost(g, w, L"Copy");  break; // unreachable
+        case WK::SelAudio:   DrawSelectSurface(...);    break; // unreachable
+        ...
+    }
+}
+```
+
+Result: `DrawHero` paints all eight widgets → eight identical blue pills.
+
+## The required change
+
+The array is **indexed by enum value** (`g_w[(int)WK::Pin]`, `g_w[(int)WK::Copy]`, …), so slot `i` must carry `kind == (WK)i`. Stamp the kinds once, at the top of `LayoutWidgets()` — it runs before the first paint and again on every resize / DPI change, so the invariant is always re-asserted:
+
+```cpp
+void LayoutWidgets(int W, int H) {
+    for (int i = 0; i < (int)std::size(g_w); ++i)   // <-- ADD THESE TWO LINES
+        g_w[i].kind = (WK)i;                         //     slot index == kind
+
+    for (auto& w : g_w) w.r = RectF(0, 0, 0, 0);
+    float s = g_dpiScale;
+    // ... unchanged ...
+}
+```
+
+Equivalent alternative: an `InitWidgets()` helper called once in `wWinMain` before the first `LayoutWidgets`/paint. Doing it *inside* `LayoutWidgets` is the most robust, because nothing can paint before layout has run.
+
+**Invariant to preserve:** `g_w[i].kind == (WK)i`. As long as rectangles keep being assigned via `g_w[(int)WK::X].r`, the slot index always equals the enum value and this holds.
+
+After this change, `DrawGhost` (Copy/Paste/Clear), `DrawPinSurface`, and `DrawSelectSurface` (mic/model) take over their slots, and you get the intended distinct, borderless, label-bearing controls.
+
+## Secondary issue noticed (cosmetic — not fixed here)
+
+The transcript **placeholder** ("Your transcription will appear here…") won't appear. `PaintSurface` draws it into the transcript rect, but the real multiline `EDIT` child window covers that rect and is opaque, so it hides the parent-painted text. `EM_SETCUEBANNER` only works on *single-line* edits. Clean fix: subclass the multiline `EDIT` and draw the cue in its own `WM_PAINT` when the control is empty and unfocused. Tracked as a separate follow-up.
+
+## Verify after rebuilding
+
+1. **Top row:** a wide blue **Record** pill + a quiet **Pin** toggle (accent only when pinned).
+2. **Bottom:** **mic** and **model** selects (label + chevron, faint at rest) and quiet **Copy / Paste / Clear** ghost buttons (text only at rest).
+3. **Hover** a ghost button → a soft fill fades in; **press** → slightly darker. No resting borders, no hairlines.
+4. Independent of this fix, run a transcription and confirm the progress **%** rises smoothly and the **"… s left"** value counts **down** (not up).
+
+---
+
+*Companion to `UI-and-Progress-Rebuild.md`. This note documents a correction to that guide's Part 1 scaffold; the guide itself is left unchanged.*

UI-and-progress-bar-rebuild-part-3.md

@@ -0,0 +1,214 @@
+# Win Dictation — Fix Note 02
+
+**All text missing from the interface (fills render, labels don't)**
+
+**Applies to:** the single-surface UI after applying Fix Note 01 (`g_w[i].kind` stamping).
+**Status:** root cause confirmed. One pattern to remove, used in five places. Two smaller adjacent issues documented below.
+
+---
+
+## Symptom
+
+Widgets now render in their correct *shapes* — blue Record pill with white dot, two select fields with chevrons, the card, the dark scrollbar — but **no text appears anywhere**:
+
+- Record pill has no "Record" label
+- Pin is completely invisible (empty space right of the pill)
+- Copy / Paste / Clear are completely invisible (empty strip at the bottom)
+- Mic / model selects are empty except for the chevron
+- No status line ("Ready · 2 threads" etc.)
+
+The tell: everything drawn with Graphics *primitives* (`FillRound`, `FillEllipse`, `DrawLine`, `DrawPath`) renders; everything drawn with `DrawTextC` doesn't. Pin and the ghost buttons are text-only at rest, so they vanish entirely.
+
+## What you missed: `Graphics::GetHDC()` locks the Graphics object
+
+Every text call sits inside this pattern (from `DrawHero`, and repeated in the other draw functions):
+
+```cpp
+HDC hdc = g.GetHDC();                  // <-- locks `g`
+Font f(hdc, g_fUISemi);                //     (constructing the Font is fine)
+RectF tb(...);
+DrawTextC(g, L"Record", f, ...);       // <-- call on locked `g` → fails silently
+g.ReleaseHDC(hdc);                     // <-- unlock, too late
+```
+
+`Graphics::GetHDC()` is documented to put the Graphics object into a **locked state**: between `GetHDC()` and `ReleaseHDC()`, *any* method called on that `Graphics` fails with `Status::ObjectBusy`. GDI+ reports errors via return codes, not exceptions — so `g.DrawString(...)` inside `DrawTextC` returns an error and draws nothing, with no crash and no debugger output. The result is exactly what you see: silent, total text loss, while every primitive drawn *outside* a lock window renders fine.
+
+This also explains two details that look confusing at first:
+
+1. **Why the chevrons survive in `DrawSelectSurface`:** the two `g.DrawLine(...)` calls happen *after* `g.ReleaseHDC(hdc)` — outside the lock — so they render.
+2. **Why the *old* owner-draw code's text worked:** it wrote `Font f(d->hDC, g_fUISemi)` using the **raw owner-draw HDC** it already had. Constructing a `Font` from an HDC does not lock anything — only `Graphics::GetHDC()` does. (Same reason the popup's `Font f(mem, g_fUI)` still works: `mem` is the raw memory HDC, not a `GetHDC()` result.)
+
+The `GetHDC()` calls were added because the new draw functions receive only a `Graphics&` and the `Font(HDC, HFONT)` constructor needs an HDC. The intent was right; the mechanism poisons the Graphics.
+
+### Affected call sites (all five must change)
+
+| Function | What's invisible |
+|---|---|
+| `DrawHero` | "Record" / "Stop" label |
+| `DrawGhost` | Copy / Paste / Clear (entire control) |
+| `DrawPinSurface` | Pin / Pinned (entire control) |
+| `DrawSelectSurface` | mic / model value text |
+| `PaintSurface` | status line, transcript placeholder |
+
+## The fix: cached GDI+ fonts, zero `GetHDC()` calls
+
+Create GDI+ `Font` objects **once** from the existing HFONTs using a *screen* DC (never the Graphics), cache them, and use them everywhere. This removes every `GetHDC()`/`ReleaseHDC()` pair and is also a per-frame win — no font construction inside a 60 fps paint loop.
+
+### 1. Add globals (near the HFONT globals)
+
+```cpp
+Gdiplus::Font* g_gpUI     = nullptr;   // labels (15px)
+Gdiplus::Font* g_gpUISemi = nullptr;   // hero label (15px semibold)
+Gdiplus::Font* g_gpSmall  = nullptr;   // status line (12px)
+Gdiplus::Font* g_gpText   = nullptr;   // transcript placeholder (16px)
+```
+
+### 2. Add the builder and fold it into `RecreateFonts`
+
+```cpp
+static Gdiplus::Font* GdipFontFromHFont(HFONT hf) {
+    HDC sdc = GetDC(nullptr);                       // screen DC — no Graphics involved
+    Gdiplus::Font* f = new Gdiplus::Font(sdc, hf);
+    ReleaseDC(nullptr, sdc);
+    if (f->GetLastStatus() != Ok) { delete f; return nullptr; }
+    return f;
+}
+
+static void RebuildGdipFonts() {
+    delete g_gpUI; delete g_gpUISemi; delete g_gpSmall; delete g_gpText;
+    g_gpUI     = GdipFontFromHFont(g_fUI);
+    g_gpUISemi = GdipFontFromHFont(g_fUISemi);
+    g_gpSmall  = GdipFontFromHFont(g_fSmall);
+    g_gpText   = GdipFontFromHFont(g_fText);
+    // safety net if HFONT conversion ever fails:
+    if (!g_gpUI)     g_gpUI     = new Gdiplus::Font(L"Segoe UI", 15.0f * g_dpiScale, FontStyleRegular,  UnitPixel);
+    if (!g_gpUISemi) g_gpUISemi = new Gdiplus::Font(L"Segoe UI", 15.0f * g_dpiScale, FontStyleBold,     UnitPixel);
+    if (!g_gpSmall)  g_gpSmall  = new Gdiplus::Font(L"Segoe UI", 12.0f * g_dpiScale, FontStyleRegular,  UnitPixel);
+    if (!g_gpText)   g_gpText   = new Gdiplus::Font(L"Segoe UI", 16.0f * g_dpiScale, FontStyleRegular,  UnitPixel);
+}
+```
+
+At the **end of `RecreateFonts(float scale)`**, add:
+
+```cpp
+    RebuildGdipFonts();
+```
+
+So HFONTs and GDI+ fonts always change together (startup and `WM_DPICHANGED`).
+
+### 3. Call it at startup — ordering matters
+
+`RebuildGdipFonts` needs GDI+ started (it already is — `GdiplusStartup` runs first) and benefits from the real DPI. In `wWinMain`, right after the DPI is known:
+
+```cpp
+    g_dpiScale = GetDpiForWindow(hMainWnd) / 96.0f;
+    if (g_dpiScale <= 0.0f) g_dpiScale = 1.0f;
+    RecreateFonts(g_dpiScale);          // <-- ADD: rebuilds HFONTs at real DPI + GDI+ fonts
+```
+
+(See "Adjacent issue A" below for why `RecreateFonts` belongs here anyway.)
+
+### 4. Shutdown ordering — GDI+ objects must die before `GdiplusShutdown`
+
+In the shutdown block, delete the GDI+ fonts **before** `GdiplusShutdown(g_gdipToken)`:
+
+```cpp
+    delete g_gpUI; delete g_gpUISemi; delete g_gpSmall; delete g_gpText;
+    g_gpUI = g_gpUISemi = g_gpSmall = g_gpText = nullptr;
+    GdiplusShutdown(g_gdipToken);
+    DeleteObject(g_fUI); ...            // HFONTs are plain GDI; their order is fine as-is
+```
+
+(Destroying GDI+ objects after shutdown is undefined behavior — worth getting right even though it "usually" doesn't crash.)
+
+### 5. Strip the lock pattern from all five call sites
+
+**`DrawHero` — before:**
+
+```cpp
+    HDC hdc = g.GetHDC();
+    Font f(hdc, g_fUISemi);
+    RectF tb((REAL)(pill.X + (int)(36 * g_dpiScale)), (REAL)pill.Y,
+             (REAL)(pill.Width - (int)(44 * g_dpiScale)), (REAL)pill.Height);
+    DrawTextC(g, rec ? L"Stop" : L"Record", f, Color(255, 255, 255, 255),
+              tb, StringAlignmentNear, StringAlignmentCenter);
+    g.ReleaseHDC(hdc);
+```
+
+**`DrawHero` — after:**
+
+```cpp
+    RectF tb((REAL)(pill.X + (int)(36 * g_dpiScale)), (REAL)pill.Y,
+             (REAL)(pill.Width - (int)(44 * g_dpiScale)), (REAL)pill.Height);
+    DrawTextC(g, rec ? L"Stop" : L"Record", *g_gpUISemi, Color(255, 255, 255, 255),
+              tb, StringAlignmentNear, StringAlignmentCenter);
+```
+
+Apply the identical transformation to the rest — delete the `GetHDC`/`Font(hdc, …)`/`ReleaseHDC` lines and pass the cached font:
+
+| Call site | Replace local `Font f(hdc, …)` with |
+|---|---|
+| `DrawGhost` | `*g_gpUI` |
+| `DrawPinSurface` | `*g_gpUI` |
+| `DrawSelectSurface` | `*g_gpUI` |
+| `PaintSurface` (status line) | `*g_gpSmall` |
+| `PaintSurface` (placeholder) | `*g_gpText` |
+
+After this, there must be **zero** calls to `g.GetHDC()` anywhere in the paint path. (The popup's `Font f(mem, g_fUI)` is fine and can stay — `mem` is a raw HDC; consider migrating it to `*g_gpUI` later for consistency.)
+
+---
+
+## Adjacent issue A — fonts are built at the wrong DPI on startup
+
+In `wWinMain`, the HFONTs are created with `g_dpiScale` still at its initial `1.0f` (the window doesn't exist yet), and `g_dpiScale` is only set *after* `CreateWindowExW`. Nothing recreates the fonts at startup, so on a 125%/150% display, all text is undersized until the first `WM_DPICHANGED`. The `RecreateFonts(g_dpiScale)` call added in step 3 fixes this — and because it runs *before* the child controls are created, the `EDIT` receives the correctly-scaled `g_fText` at creation.
+
+## Adjacent issue B — transient status messages are now invisible
+
+`SetStatus(...)` writes to the `ID_STATIC_STATUS` control — which is hidden (`SW_HIDE`). The painted status line in `PaintSurface` derives its text purely from state (recording/busy/ready/loading), so these messages can never appear: **"Copied", "Pasted", "Cancelled", "No speech detected", "Microphone error", "Hotkey in use — edit win-dictation.ini"**.
+
+Minimal repair — route `SetStatus` into the painted surface as a transient override:
+
+```cpp
+std::wstring g_statusOverride;          // shown instead of the derived idle status
+DWORD        g_statusOverrideUntil = 0; // GetTickCount() deadline
+
+void SetStatus(HWND hwnd, const wchar_t* text) {
+    g_statusOverride      = text;
+    g_statusOverrideUntil = GetTickCount() + 2500;   // visible for 2.5s
+    InvalidateRect(hwnd, nullptr, FALSE);
+}
+```
+
+In `PaintSurface`'s status-text branch, prefer the override when idle:
+
+```cpp
+        } else if (g_modelLoaded.load()) {
+            if (!g_statusOverride.empty() && GetTickCount() < g_statusOverrideUntil) {
+                wcscpy_s(statusBuf, g_statusOverride.c_str());
+            } else if (!g_modelOk.load()) {
+                swprintf_s(statusBuf, L"Model not found — check models folder");
+            } else {
+                swprintf_s(statusBuf, L"Ready  •  %d threads", g_tx.threads());
+            }
+        }
+```
+
+(Recording/busy branches still win, which is correct — those states are more important than a stale "Copied".)
+
+---
+
+## Verify after rebuilding
+
+1. **Labels everywhere:** "Record" on the pill; "Pinned" top-right in accent; "Copy / Paste / Clear" as dim labels; mic + model names in the selects; "Loading model…" → "Ready · 2 threads" on the status line.
+2. **Hover** Copy/Paste/Clear → soft fill fades in, label brightens.
+3. **Copy something** → status briefly shows "Copied" (issue B fix).
+4. On a HiDPI display, text is correctly sized at first launch (issue A fix).
+5. Run a transcription → smooth rising % and counting-down ETA (unrelated path, but confirm while you're there).
+
+## Optional cleanup (safe to defer)
+
+The legacy chrome is now dead weight: the nine hidden child windows and their `Create…`/`SetWindowTheme`/`SetWindowSubclass`/`ShowWindow(SW_HIDE)` calls, `LayoutControls`, `BtnProc`, `DrawRecordButton`, `DrawFlatButton`, `DrawSelect`, `UpdatePlaceholder`'s STATIC logic, and the `WM_DRAWITEM`/`WM_MEASUREITEM` handlers (keep the `EDIT` and everything for it). Removing them deletes ~150 lines and removes the double layout work in `WM_SIZE` (`LayoutControls` + `LayoutWidgets` both run and both `MoveWindow` the EDIT). Functionally harmless today, so treat as a tidy-up pass, not part of this fix.
+
+---
+
+*Companion to `UI-and-Progress-Rebuild.md` and `UI-Progress-Rebuild-Fix-01.md`. Per project convention, this note is a new document; prior documents are unchanged.*

UI-and-progress-bar-rebuild.md

@@ -0,0 +1,630 @@
+# Win Dictation — UI & Progress Rebuild Guide
+
+A design-led plan to (1) kill the "thin lines" problem at its architectural root rather than patching it, and (2) replace the broken progress bar with a self-calibrating, smoothly-animated estimator that learns this machine's transcription speed and fuses whisper's own progress signal.
+
+This is implementation guidance with concrete code. You build on Windows; nothing here is compiled or tested in place.
+
+---
+
+## Table of contents
+
+1. [Part 1 — The interface](#part-1--the-interface)
+   - [1.1 Why the lines are really there](#11-why-the-lines-are-really-there)
+   - [1.2 The architectural fix: one surface](#12-the-architectural-fix-one-surface)
+   - [1.3 Two tiers: GDI+ vs Direct2D](#13-two-tiers-gdi-vs-direct2d)
+   - [1.4 A real design language](#14-a-real-design-language)
+   - [1.5 Component specs](#15-component-specs)
+   - [1.6 Rendering scaffold + hit-testing (code)](#16-rendering-scaffold--hit-testing-code)
+   - [1.7 The transcript field & DPI](#17-the-transcript-field--dpi)
+   - [1.8 Migration order from today's main.cpp](#18-migration-order-from-todays-maincpp)
+2. [Part 2 — The progress system](#part-2--the-progress-system)
+   - [2.1 Why it's broken today](#21-why-its-broken-today)
+   - [2.2 The plan: predict, then correct](#22-the-plan-predict-then-correct)
+   - [2.3 Persistent per-model timing history](#23-persistent-per-model-timing-history)
+   - [2.4 The live estimator (smooth countdown + fusion)](#24-the-live-estimator-smooth-countdown--fusion)
+   - [2.5 `timing.h` — full code](#25-timingh--full-code)
+   - [2.6 Wiring into main.cpp](#26-wiring-into-maincpp)
+   - [2.7 Tuning & edge cases](#27-tuning--edge-cases)
+3. [Part 3 — Cleanup checklist](#part-3--cleanup-checklist)
+4. [Part 4 — Suggested build order](#part-4--suggested-build-order)
+
+---
+
+# Part 1 — The interface
+
+## 1.1 Why the lines are really there
+
+The hairlines aren't one bug; they're an emergent property of how the window is built. Today the UI is roughly **nine separate child windows** living on top of the main window:
+
+- `BUTTON` (owner-draw): Record, Pin, Copy, Paste, Clear
+- `BUTTON` (owner-draw) used as selects: mic, model
+- `EDIT` (multiline): the transcript
+- `STATIC`: status + placeholder
+
+Each child is its own HWND with its own device context, its own paint timing, and — because the parent uses `WS_CLIPCHILDREN` — its own **hard-clipped rectangle punched out of the parent's paint**. That single fact is the source of the lines:
+
+1. **Seams at every child boundary.** The parent paints its background/panel, then Windows clips out each child rectangle and the child paints itself. The boundary between "parent pixels" and "child pixels" is a 1px hard edge. Any difference in rounding, antialiasing, or color across that edge reads as a hairline — even when both sides *intend* to be the same dark color.
+2. **Theme chrome you didn't ask for.** The `EDIT` control draws its own themed 1px border and a **light-mode scrollbar** (the pale bar on the right of your screenshot). `SetWindowTheme(h, L"", L"")` on the buttons disables visual styles but doesn't make seams go away.
+3. **Square corners around round shapes.** Your chips are drawn rounded, but the *child window* is rectangular, so the artifact rectangle has sharp corners that don't follow the chip — which is exactly what's visible around the Record pill and the Copy/Paste/Clear buttons.
+
+So removing `StrokeRound(..., C_BORDER, ...)` only removes the *intentional* borders. The *structural* hairlines (items 1–3) remain. That's why it feels like a band-aid: **you can't fully remove seams while compositing many themed child windows.**
+
+> **Root cause, one sentence:** the window is assembled from many separate themed/owner-draw child HWNDs, and the boundaries between them can never be made perfectly seamless. The fix is to stop having those boundaries.
+
+## 1.2 The architectural fix: one surface
+
+Render the **entire window as a single double-buffered surface**, immediate-mode:
+
+- The parent's `WM_PAINT` draws *everything* — background, the card, every button, the selects, the status line, the VU/progress strip — onto **one off-screen bitmap**, then blits it once. (You already do this for the background and panel; we extend it to cover all chrome.)
+- **There are no child windows for chrome.** "Buttons" become **painted regions** described by a small data model (a rect + a kind + interaction state). There is exactly one surface, so there are zero inter-window seams. Antialiasing, radii, spacing, shadows, and animation are all under your control.
+- **Interaction** is handled in the parent: `WM_MOUSEMOVE` / `WM_LBUTTONDOWN` / `WM_LBUTTONUP` hit-test against the widget rects; you track hover/pressed/focus yourself and invalidate. (The window is tiny — invalidating the whole client area each frame is cheap.)
+- **The one exception is the transcript**, which stays a real `EDIT` child because you genuinely want selection, caret, scrolling, and IME. We make it *visually chrome-less* and inset it inside the painted card so the card is the only visible frame (see [1.7](#17-the-transcript-field--dpi)).
+
+This is the same "retained data model + immediate-mode paint" approach used by every good custom-drawn desktop UI. Separation between elements comes from **fills, spacing, and elevation — not outlines.** Once outlines stop being load-bearing, the hairline problem is gone by construction.
+
+## 1.3 Two tiers: GDI+ vs Direct2D
+
+You said you'll happily take more effort for a result that looks genuinely good. Here are the two honest options.
+
+### Tier 1 — GDI+ single-surface (recommended baseline)
+
+- Keep GDI+ (already in the project). Move all drawing into one parent paint routine that renders to a 32-bit DIB back-buffer, then `BitBlt`.
+- Reuse your existing helpers (`FillRound`, `StrokeRound`, `DrawTextC`) — they're good. You're changing *what hosts them*, not the primitives.
+- Add an animation clock + hover/press state.
+- **Effort:** moderate. **Payoff:** the seams disappear, you get full control of spacing/elevation/motion, and it will look clean and modern. This removes 100% of the reported problem.
+- **Limitations:** GDI+ has no true GPU compositing; soft drop-shadows must be faked (pre-blurred bitmap or layered alpha), and very large blurs are slow. For a 400×340 utility this is a non-issue.
+
+### Tier 2 — Direct2D + DirectWrite (premium path)
+
+- GPU-accelerated geometry with flawless antialiasing, real `ID2D1Effect` drop shadows / Gaussian blur, per-primitive opacity layers, and **DirectWrite** text with subpixel positioning (noticeably crisper labels, especially at fractional DPI).
+- Pairs naturally with a swap-chain or a DC render target; integrates with DWM for tear-free animation at the monitor refresh rate.
+- Optionally add **Windows.UI.Composition / DirectComposition** for soft shadows and an acrylic/mica backdrop — a true Windows 11 feel.
+- **Effort:** higher (COM lifetimes, device-lost handling, more setup). **Payoff:** the highest visual ceiling and the best foundation if this app grows.
+- You can still keep the `EDIT` child for the transcript layered above the D2D surface.
+
+**Recommendation:** Build **Tier 1 now** — it eliminates the actual defect and looks great, and almost all of the work (the design language, the widget model, the interaction layer, the progress system in Part 2) is *identical* regardless of renderer. If you later want the extra polish, swapping the draw calls to Direct2D is a contained change because the data model and layout stay the same. The rest of this guide is written renderer-agnostic with GDI+ code samples.
+
+## 1.4 A real design language
+
+The current look is "many bordered boxes." The target look is **one calm, elevated card** where hierarchy comes from type, spacing, and a single light source — not lines.
+
+### Tokens (define once)
+
+```cpp
+// ---- color tokens (ARGB) ----
+const Color T_BG        (255, 0x0E, 0x10, 0x14); // app backdrop (near-black)
+const Color T_CARD      (255, 0x16, 0x19, 0x20); // elevated card
+const Color T_CARD_HI   (255, 0x1E, 0x22, 0x2B); // hovered surface
+const Color T_CARD_LO   (255, 0x12, 0x15, 0x1B); // pressed surface / wells
+const Color T_TEXT      (255, 0xEC, 0xEE, 0xF2); // primary text
+const Color T_DIM       (255, 0x8A, 0x90, 0x9C); // secondary text
+const Color T_FAINT     (255, 0x5A, 0x60, 0x6C); // tertiary / icons at rest
+const Color T_ACCENT    (255, 0x6E, 0x8B, 0xFF); // primary action
+const Color T_ACCENT_HI (255, 0x83, 0x9C, 0xFF); // accent hover
+const Color T_DANGER    (255, 0xFF, 0x5C, 0x5C); // recording
+const Color T_GOOD      (255, 0x46, 0xD3, 0x9A); // level / success
+
+// The ONLY "edge" allowed: a low-alpha top highlight on the card,
+// to read as "lit from above." Never a full gray rectangle.
+const Color T_TOPLIGHT  (26,  0xFF, 0xFF, 0xFF); // ~10% white
+```
+
+**Principle:** elements are distinguished by *fill* (`T_CARD` vs `T_CARD_HI`), by *space* (generous padding), and by *elevation* (the card sits on the backdrop, optionally with a soft shadow). Outlines are reserved for nothing, or at most one hairline-as-toplight on the card itself.
+
+### Type scale (Segoe UI Variable, which you already load)
+
+| Role | Size (logical px) | Weight | Color |
+|------|------|--------|-------|
+| Primary state ("Record" / "Stop" / "Transcribing") | 16 | SemiBold | white on accent / `T_TEXT` |
+| Body / transcript | 16 | Regular | `T_TEXT` |
+| Buttons (ghost) | 14 | Medium | `T_DIM` → `T_TEXT` on hover |
+| Status caption | 12.5 | Regular | `T_DIM` |
+| Micro (threads, %, ETA) | 11.5 | Regular | `T_FAINT` |
+
+### Elevation & radius
+
+- Card radius **16**; inner controls radius **10–11**; progress/level pill radius = half-height.
+- Optional soft shadow under the card (Tier 1: a pre-rendered blurred rounded-rect bitmap at ~22% alpha, offset y+6, blur ~18; Tier 2: a D2D shadow effect). Subtle — it should read as depth, not drama.
+
+### Motion (this is what makes it feel "good", not just look good)
+
+- Hover/press fills cross-fade over **120–160ms**, ease-out-cubic.
+- Recording state: a **1.2s sine "breathing"** on the record pill + a live waveform (see below).
+- Progress: bar width and the % label are **eased**, never snapped (except the final 100%).
+- Drive all of it from one animation clock (Section 1.6). Run the timer at ~16ms **only while something is animating**, and idle otherwise (don't burn CPU on a 2-core machine when nothing moves).
+
+## 1.5 Component specs
+
+**Record (hero).** Full-width pill, `T_ACCENT` fill, white glyph + label. States:
+- *Idle:* circle glyph + "Record". Hover → `T_ACCENT_HI`. Press → ×0.9 brightness.
+- *Recording:* `T_DANGER`, breathing alpha, square "stop" glyph, label "Stop", and a **live waveform** drawn across the pill or in the strip below.
+- Keep it the visual anchor; everything else is quieter.
+
+**Ghost actions (Copy / Paste / Clear).** No resting fill, no border — just a Medium-weight label in `T_DIM`. On hover, a `T_CARD_HI` rounded fill fades in and text lifts to `T_TEXT`; on press, `T_CARD_LO`. Because there's no resting border, there are no hairlines; separation is purely spacing. (Add small 16px line icons before labels for a more finished feel.)
+
+**Pin.** An icon toggle (pin glyph), `T_ACCENT` when active, `T_FAINT` when not. No label needed.
+
+**Selects (mic / model).** Quiet rows: small dim label on top ("Microphone"), value below in `T_TEXT`, a small chevron at the right; hover = `T_CARD_HI` fill. **Consider relocating both behind a small gear/settings affordance** — a dictation utility doesn't need model internals on the main face. If you keep them visible, give them the same fill-on-hover, no-border treatment.
+
+**Status + progress strip (unified).** One horizontal zone under the hero that changes by state:
+- *Idle:* `"Ready · 2 threads"` in `T_DIM`.
+- *Recording:* live waveform + `mm:ss` timer.
+- *Transcribing:* the progress bar (Part 2) with smooth % and a **counting-down** ETA.
+
+**Level / waveform.** Replace the 14-segment VU (reads as "old") with either a smooth antialiased waveform (ring buffer of recent RMS samples drawn as a filled path) or a single breathing level pill. Color `T_GOOD`, riding on `T_CARD_LO`.
+
+**Empty state.** Centered mic glyph + "Your transcription will appear here" in `T_DIM`, drawn *inside* the card (not as a separate STATIC) so it shares the surface.
+
+## 1.6 Rendering scaffold + hit-testing (code)
+
+The whole UI becomes a small list of widgets plus one paint routine and one interaction handler. Skeleton (GDI+, Tier 1):
+
+```cpp
+enum class WK { RecordHero, Pin, Copy, Paste, Clear, SelAudio, SelModel, Transcript };
+
+struct Widget {
+    WK     kind;
+    RectF  r;                 // logical rect, filled by Layout()
+    bool   hover = false;
+    bool   pressed = false;
+    float  anim = 0.0f;       // 0..1 eased hover/press amount
+};
+
+static Widget g_w[ (int)WK::Transcript + 1 ];
+static int    g_hot = -1;     // index under cursor
+static int    g_active = -1;  // index pressed
+
+// --- one animation clock ---
+static DWORD  g_lastFrame = 0;
+static bool   AnyAnimating();          // true if any widget anim is mid-transition, or recording, or busy
+
+// Advance eased states; call from the render timer.
+void StepAnimations(float dt) {
+    for (auto& w : g_w) {
+        float target = (g_active == (&w - g_w) ) ? 1.0f : (w.hover ? 0.6f : 0.0f);
+        // ease toward target
+        w.anim += (target - w.anim) * std::min(1.0f, dt * 12.0f);
+    }
+}
+
+// --- layout: compute rects from client size & DPI scale ---
+void Layout(int W, int H, float s /*dpi scale*/);
+
+// --- paint: ONE surface ---
+void Paint(HWND hwnd) {
+    PAINTSTRUCT ps; HDC hdc = BeginPaint(hwnd, &ps);
+    RECT rc; GetClientRect(hwnd, &rc);
+    int W = rc.right, H = rc.bottom;
+
+    HDC mem = CreateCompatibleDC(hdc);
+    HBITMAP bmp = CreateCompatibleBitmap(hdc, W, H);
+    HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
+    {
+        Graphics g(mem);
+        g.SetSmoothingMode(SmoothingModeAntiAlias);
+        g.SetTextRenderingHint(TextRenderingHintClearTypeGridFit);
+
+        SolidBrush bg(T_BG); g.FillRectangle(&bg, 0, 0, W, H);
+
+        DrawCardWithShadow(g, g_cardRect, 16);   // optional soft shadow + T_CARD fill + T_TOPLIGHT edge
+
+        for (auto& w : g_w) {
+            switch (w.kind) {
+                case WK::RecordHero: DrawHero(g, w);  break;
+                case WK::Copy:  DrawGhost(g, w, L"Copy");  break;
+                case WK::Paste: DrawGhost(g, w, L"Paste"); break;
+                case WK::Clear: DrawGhost(g, w, L"Clear"); break;
+                case WK::Pin:   DrawPin(g, w);  break;
+                case WK::SelAudio: DrawSelect(g, w, g_audioVal); break;
+                case WK::SelModel: DrawSelect(g, w, g_modelVal); break;
+                case WK::Transcript: /* the EDIT child paints itself; we just leave its inset */ break;
+            }
+        }
+        DrawStatusStrip(g, g_stripRect);   // idle / recording waveform / progress
+    }
+    BitBlt(hdc, 0, 0, W, H, mem, 0, 0, SRCCOPY);
+    SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
+    EndPaint(hwnd, &ps);
+}
+
+// --- interaction: hit-test in the parent ---
+int HitTest(POINT p) {
+    for (int i = 0; i < (int)std::size(g_w); ++i)
+        if (g_w[i].kind != WK::Transcript && g_w[i].r.Contains((REAL)p.x, (REAL)p.y)) return i;
+    return -1;
+}
+
+LRESULT CALLBACK WndProc(HWND h, UINT m, WPARAM w, LPARAM l) {
+    switch (m) {
+    case WM_MOUSEMOVE: {
+        POINT p{ GET_X_LPARAM(l), GET_Y_LPARAM(l) };
+        int hot = HitTest(p);
+        if (hot != g_hot) {
+            if (g_hot >= 0) g_w[g_hot].hover = false;
+            g_hot = hot;
+            if (g_hot >= 0) g_w[g_hot].hover = true;
+            TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 }; TrackMouseEvent(&t);
+            EnsureAnimating(h);
+        }
+        return 0;
+    }
+    case WM_MOUSELEAVE:
+        if (g_hot >= 0) { g_w[g_hot].hover = false; g_hot = -1; EnsureAnimating(h); }
+        return 0;
+    case WM_LBUTTONDOWN:
+        g_active = g_hot;
+        if (g_active >= 0) { g_w[g_active].pressed = true; SetCapture(h); EnsureAnimating(h); }
+        return 0;
+    case WM_LBUTTONUP: {
+        ReleaseCapture();
+        POINT p{ GET_X_LPARAM(l), GET_Y_LPARAM(l) };
+        if (g_active >= 0 && HitTest(p) == g_active) OnClick(h, g_w[g_active].kind);
+        if (g_active >= 0) g_w[g_active].pressed = false;
+        g_active = -1; EnsureAnimating(h);
+        return 0;
+    }
+    case WM_ERASEBKGND: return 1;          // we paint everything
+    case WM_PAINT: Paint(h); return 0;
+    case WM_SIZE: Layout(LOWORD(l), HIWORD(l), g_dpiScale); InvalidateRect(h, nullptr, FALSE); return 0;
+    // ... WM_TIMER drives StepAnimations + InvalidateRect while AnyAnimating()
+    }
+    return DefWindowProc(h, m, w, l);
+}
+```
+
+Notes:
+- `DrawGhost` simply lerps its fill alpha by `w.anim` between transparent → `T_CARD_HI`, and text color between `T_DIM` → `T_TEXT`. No `StrokeRound`. That's the whole trick.
+- `EnsureAnimating(h)` starts the 16ms timer if it isn't running; the timer stops itself when `AnyAnimating()` returns false to spare the CPU.
+- Keyboard focus (for accessibility / Tab) can be added later by tracking a `g_focus` index and painting a soft focus ring on the focused widget only — still no native chrome.
+
+## 1.7 The transcript field & DPI
+
+**Transcript = the one real child window.** Keep `EDIT` (multiline, read-only) for free selection/caret/scroll/IME, but strip its chrome:
+
+1. **No border:** create without `WS_BORDER`/`WS_EX_CLIENTEDGE` (already the case). To suppress the *themed* edit border entirely, either `SetWindowTheme(hEdit, L"", L"")` (kills the theme, gives a classic flat look) or subclass and handle `WM_NCPAINT` to no-op. Prefer the dark-mode route below so the scrollbar also matches.
+2. **Dark background:** you already return `g_brSurface` from `WM_CTLCOLOREDIT`; set it to `T_CARD`/`T_CARD_LO` so the field is invisible against the card.
+3. **Dark (or custom) scrollbar — this removes the pale bar in your screenshot:**
+   - Easiest: enable app dark mode then theme the control:
+     ```cpp
+     // once, after the process starts (uxtheme, undocumented but widely used):
+     // AllowDarkModeForApp(true); SetPreferredAppMode(AllowDark);
+     SetWindowTheme(hEdit, L"DarkMode_Explorer", nullptr); // dark scrollbar
+     ```
+   - Most control: hide the native scrollbar (`ShowScrollBar(hEdit, SB_VERT, FALSE)` or `WM_NCCALCSIZE`) and **paint a slim custom scrollbar on the parent surface**, driven by `EM_GETFIRSTVISIBLELINE` / line count. Best looking, more work.
+4. **Inset it inside the card** by ~14–16px so the card's rounded surface is the visible frame and the EDIT contributes no edges of its own.
+
+**DPI awareness (do this — it's part of "looks good").** Today metrics are fixed pixels; on a HiDPI panel they blur/misalign.
+
+- Declare **Per-Monitor-V2** via the app manifest (preferred) or `SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2)` at startup.
+- Compute `g_dpiScale = GetDpiForWindow(hwnd) / 96.0f`; multiply every metric (padding, radii, font sizes, widget sizes) by it.
+- Recreate fonts and re-`Layout()` on `WM_DPICHANGED`, and use the suggested rect it passes for repositioning.
+
+## 1.8 Migration order from today's main.cpp
+
+You can do this incrementally without a rewrite:
+
+1. **Stop creating chrome child windows.** Delete the `CreateWindow(L"BUTTON", …)` calls for Record/Pin/Copy/Paste/Clear and the two selects, and the two `STATIC`s. Keep only the `EDIT`.
+2. **Add the `Widget` array + `Layout()`** computing the same rectangles your `LayoutControls` used (reuse the math; just store `RectF`s instead of `MoveWindow`-ing HWNDs).
+3. **Move your existing draw functions** (`DrawRecordButton`→`DrawHero`, `DrawFlatButton`→`DrawGhost`, `DrawSelect`) to take a `Widget&` and draw into the shared `Graphics&` — and **delete every `StrokeRound(..., T_BORDER/C_BORDER, …)`**. Replace the panel border with the card + optional shadow + toplight.
+4. **Route interaction** through `WndProc` hit-testing (Section 1.6). `OnClick(kind)` calls your existing handlers (toggle record, copy, paste, clear, open the popup for selects).
+5. **Add the animation clock**; convert hover from per-HWND `GWLP_USERDATA` to `w.anim`.
+6. **Theme the EDIT + scrollbar**, inset it, and add DPI scaling.
+7. The popup list (`PopupProc`) can stay as-is — it's already a single custom-painted surface and looks consistent.
+
+Result: one surface, zero seams, full control. The "thin lines" cannot come back because nothing draws them and there are no child boundaries to leak them.
+
+---
+
+# Part 2 — The progress system
+
+## 2.1 Why it's broken today
+
+Three separate problems, all visible in your 1:27 example.
+
+**(a) The ETA counts *up*.** In `UpdateStatus`:
+
+```cpp
+float elapsed = (GetTickCount() - g_busyStart) / 1000.0f;
+float est     = elapsed * 100.0f / (float)p;   // total, derived from stale p
+float remain  = est - elapsed;                  // = elapsed * (100 - p) / p
+```
+
+`remain` is recomputed every 50ms, but `p` (whisper's progress) only changes at chunk boundaries. With `p` held constant and `elapsed` rising, `remain = elapsed·(100−p)/p` **increases over time** — the ETA climbs until the next `p` update, then snaps down when `p` jumps. That's precisely "counts up, then jumps to 20s, then counts up again."
+
+**(b) The percentage jumps in big steps.** whisper.cpp calls its progress callback at most **once per 30-second audio chunk**. 1:27 = 87s ≈ **three chunks**, so `p` arrives roughly as `0 → 33 → 67 → 100`. The 34% and 72% you saw are those chunk boundaries (off slightly due to seek rounding). The bar can't be smooth if its only input updates 3 times.
+
+**(c) Dead air at the start.** For 87s of audio the first callback only fires after the *first* 30s chunk finishes decoding — several seconds on a 2-core CPU — so nothing moves at first (you read it as "model loading"). The model is actually already preloaded; it's first-chunk latency with no fallback signal.
+
+**Conclusion:** whisper's callback is a *coarse, occasional measurement*, not a progress source. We need our own continuous prediction, corrected by that measurement.
+
+## 2.2 The plan: predict, then correct
+
+Exactly your idea, formalized:
+
+1. **Predict** total processing time the instant recording stops, from a **history of how long this machine took** for clips of various lengths (per model). This drives a smooth bar from frame 1 — even for sub-30s clips that get *zero* whisper updates.
+2. **Correct** that prediction as whisper reports progress: each callback implies a *measured* total time; we fuse it into our estimate with exponential smoothing so accuracy improves **without jumps**.
+3. **Display** a strictly **counting-down** remaining time and a **smoothly rising** percent derived from the same model, ease to **95%**, and **snap to 100%** when the real result arrives.
+4. **Learn:** on completion, record `(audio_seconds, actual_processing_seconds)` and persist it, so the next prediction is better.
+
+## 2.3 Persistent per-model timing history
+
+Processing time vs audio length is, to first order, **linear**: `proc ≈ a + b·audio`, where `b` is roughly the inverse real-time factor and `a` is fixed overhead. We fit `a, b` per model (tiny.en and base.en behave very differently) with an **online least-squares** accumulator, with a gentle decay so the model adapts to thermal throttling / machine load.
+
+- **Key by model filename** (e.g. `ggml-tiny.en.bin`), since speed is model-dependent.
+- **Cold start:** before we have ≥2 samples, use baked-in defaults (rough seeds for a 2-core i5-7th-gen; they self-correct after a run or two):
+  - tiny.en: `a ≈ 0.3s`, `b ≈ 0.45` (≈2.2× real-time)
+  - base.en: `a ≈ 0.5s`, `b ≈ 1.1` (≈0.9× real-time)
+  - (These are only seeds; the regression takes over quickly.)
+- **Persist** alongside the existing `win-dictation.ini` using the same `WritePrivateProfileString` style you already use in `settings.h`, one section per model holding the five accumulators.
+
+## 2.4 The live estimator (smooth countdown + fusion)
+
+State: `T_hat` (current best total-time estimate), `disp_rem` (displayed remaining, monotonic), `t` (seconds since start).
+
+- **begin(T_pred):** `T_hat = disp_rem = max(0.4, T_pred)`, `t = 0`.
+- **on_whisper(t_now, p):** ignore `p < 5` (noisy). Else measured total `T_meas = 100·t_now / p`; fuse: `T_hat = (1−α)·T_hat + α·T_meas` with `α ≈ 0.5`. This is where whisper "adjusts our countdown" — it moves the estimate, not the displayed number directly, so there's never a visible jump.
+- **tick(dt):** the smoothing rules that make it feel solid:
+  1. Always count down in real time: `disp_rem -= dt`.
+  2. Pull toward the model's `raw_rem = max(0, T_hat − t)`, but **only ever downward**, and **rate-limited**:
+     - `err = raw_rem − disp_rem`
+     - if `err < 0` (we're behind → need to speed up): `disp_rem += max(err, −maxCatchUp·dt)` (bounded extra shrink, no snap)
+     - if `err ≥ 0` (we have more headroom than shown): **do nothing** — never push remaining up. The bar simply keeps easing and parks near 95% if we under-predicted.
+  3. Clamp `disp_rem ≥ 0`.
+  4. Derive fraction from the same numbers: `frac = t / (t + disp_rem)`, clamp to **0.95**. Because `t` only rises and `disp_rem` only falls, `frac` only rises — smooth, monotonic, no jumps.
+- **on_result:** snap `frac → 1.0`; record `(audio_seconds, t)` into the timing model and persist.
+
+This guarantees: **ETA only counts down** (bug fixed), **% only rises smoothly** (no 34→72 jumps), whisper's coarse measurements **gently re-aim** the countdown, and there's **motion from frame 1** (no dead start). On a sub-30s clip with no whisper updates, it runs purely on the learned prediction — exactly what you asked for.
+
+## 2.5 `timing.h` — full code
+
+```cpp
+#pragma once
+#include <windows.h>
+#include <string>
+#include <cmath>
+#include <algorithm>
+
+// ---------------------------------------------------------------------------
+// Online linear model: proc_sec ~= a + b * audio_sec, fitted per whisper model.
+// Decayed least squares so it adapts to thermal / load drift over time.
+// ---------------------------------------------------------------------------
+struct TimingModel {
+    double n=0, sx=0, sy=0, sxx=0, sxy=0;   // decayed accumulators
+    double a=0, b=0;                          // fitted intercept / slope
+    bool   fitted=false;
+    double def_a=0.4, def_b=0.6;              // cold-start seeds (set per model)
+
+    void recompute() {
+        if (n >= 2.0) {
+            double denom = n*sxx - sx*sx;
+            if (std::fabs(denom) > 1e-9) {
+                double bb = (n*sxy - sx*sy) / denom;
+                double aa = (sy - bb*sx) / n;
+                if (bb < 0.02) bb = def_b;     // guard against degenerate fits
+                if (aa < 0.0)  aa = 0.0;
+                a=aa; b=bb; fitted=true; return;
+            }
+        }
+        a=def_a; b=def_b; fitted=false;
+    }
+
+    double predict(double audio_sec) const {
+        double t = (fitted ? a : def_a) + (fitted ? b : def_b) * audio_sec;
+        return std::max(0.4, t);
+    }
+
+    void add_sample(double audio_sec, double proc_sec) {
+        const double decay = 0.97;             // ~30-sample memory
+        n*=decay; sx*=decay; sy*=decay; sxx*=decay; sxy*=decay;
+        n+=1; sx+=audio_sec; sy+=proc_sec;
+        sxx+=audio_sec*audio_sec; sxy+=audio_sec*proc_sec;
+        recompute();
+    }
+};
+
+// ---------------------------------------------------------------------------
+// Live estimator: smooth, monotonic countdown fused with whisper's progress.
+// ---------------------------------------------------------------------------
+struct ProgressEstimator {
+    double T_hat=1.0, disp_rem=1.0, t=0.0;
+    bool   done=false;
+
+    void begin(double T_pred) {
+        T_hat = std::max(0.4, T_pred);
+        disp_rem = T_hat; t = 0.0; done=false;
+    }
+
+    void on_whisper(double t_now, int p) {     // p in (0,100]
+        if (done || p < 5) return;
+        double T_meas = 100.0 * t_now / (double)p;
+        const double alpha = 0.5;              // how much we trust the measurement
+        T_hat = (1.0-alpha)*T_hat + alpha*T_meas;
+        if (T_hat < t_now) T_hat = t_now;      // never imply we're already done
+    }
+
+    // dt seconds since last tick. Outputs eased fraction [0,1] and remaining secs.
+    void tick(double dt, float& out_frac, float& out_remaining) {
+        if (done) { out_frac=1.0f; out_remaining=0.0f; return; }
+        t += dt;
+        disp_rem -= dt;                                 // (1) real-time countdown
+        double raw_rem = std::max(0.0, T_hat - t);
+        const double maxCatchUp = 2.5;                  // cap speed-up (×realtime)
+        double err = raw_rem - disp_rem;
+        if (err < 0) disp_rem += std::max(err, -maxCatchUp*dt);   // (2) shrink only
+        if (disp_rem < 0) disp_rem = 0;                 // (3)
+        double frac = (t + disp_rem > 1e-6) ? t/(t+disp_rem) : 0.0;
+        if (frac > 0.95) frac = 0.95;                   // (4) hold until result
+        out_frac = (float)frac;
+        out_remaining = (float)disp_rem;
+    }
+
+    void finish(float& out_frac, float& out_remaining) {
+        done=true; out_frac=1.0f; out_remaining=0.0f;
+    }
+};
+
+// ---------------------------------------------------------------------------
+// Persistence (same ini style as settings.h). Section = model base filename.
+// ---------------------------------------------------------------------------
+inline std::wstring TimingIniPath() {
+    wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
+    std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
+    return p + L"\\win-dictation.ini";
+}
+inline std::wstring SectionFor(const std::string& modelPath) {
+    std::string base = modelPath.substr(modelPath.find_last_of("\\/")+1);
+    return L"timing-" + std::wstring(base.begin(), base.end());
+}
+inline void PutD(const std::wstring& sec, const wchar_t* k, double v) {
+    wchar_t b[64]; swprintf_s(b, L"%.6f", v);
+    WritePrivateProfileStringW(sec.c_str(), k, b, TimingIniPath().c_str());
+}
+inline double GetD(const std::wstring& sec, const wchar_t* k, double d) {
+    wchar_t b[64]; swprintf_s(b, L"%.6f", d);
+    wchar_t out[64];
+    GetPrivateProfileStringW(sec.c_str(), k, b, out, 64, TimingIniPath().c_str());
+    return wcstod(out, nullptr);
+}
+inline void LoadTiming(TimingModel& m, const std::string& modelPath) {
+    auto s = SectionFor(modelPath);
+    m.n=GetD(s,L"n",0); m.sx=GetD(s,L"sx",0); m.sy=GetD(s,L"sy",0);
+    m.sxx=GetD(s,L"sxx",0); m.sxy=GetD(s,L"sxy",0);
+    m.recompute();
+}
+inline void SaveTiming(const TimingModel& m, const std::string& modelPath) {
+    auto s = SectionFor(modelPath);
+    PutD(s,L"n",m.n); PutD(s,L"sx",m.sx); PutD(s,L"sy",m.sy);
+    PutD(s,L"sxx",m.sxx); PutD(s,L"sxy",m.sxy);
+}
+
+// Set per-model cold-start seeds when (re)loading a model.
+inline void SeedDefaults(TimingModel& m, const std::string& modelPath) {
+    std::string p = modelPath;
+    auto has = [&](const char* s){ return p.find(s)!=std::string::npos; };
+    if      (has("tiny")) { m.def_a=0.3; m.def_b=0.45; }
+    else if (has("base")) { m.def_a=0.5; m.def_b=1.10; }
+    else if (has("small")){ m.def_a=0.8; m.def_b=3.00; }
+    else                  { m.def_a=0.5; m.def_b=1.00; }
+    m.recompute();
+}
+```
+
+## 2.6 Wiring into main.cpp
+
+Add globals and capture the audio length **before** `stop_and_transcribe()` swaps the buffer away:
+
+```cpp
+TimingModel       g_timing;
+ProgressEstimator g_est;
+double            g_lastAudioLen = 0.0;     // seconds of the clip being transcribed
+DWORD             g_lastTick     = 0;
+```
+
+**At model load / model switch** (where you set `g_config.model_path`), seed + load history:
+
+```cpp
+SeedDefaults(g_timing, g_config.model_path);
+LoadTiming(g_timing, g_config.model_path);
+```
+
+**On STOP → transcribe** (the `HK_TOGGLE` stop branch and the max-length branch):
+
+```cpp
+g_lastAudioLen = g_tx.recorded_seconds();           // BEFORE stop swaps the buffer
+g_busyStart    = GetTickCount();
+g_lastTick     = g_busyStart;
+g_est.begin(g_timing.predict(g_lastAudioLen));      // bar moves from frame 1
+g_progress     = 0;
+g_cancelRequested = false;
+g_tx.stop_and_transcribe();
+```
+
+**whisper progress** (`WM_APP_PROGRESS`) becomes a *correction*, not the display source:
+
+```cpp
+case WM_APP_PROGRESS: {
+    double t_now = (GetTickCount() - g_busyStart) / 1000.0;
+    g_est.on_whisper(t_now, (int)wParam);
+    InvalidateRect(hWnd, &g_vuRect, FALSE);
+    return 0;
+}
+```
+
+**The UI timer** (`WM_TIMER`, while `g_tx.is_busy()`) advances the estimator and paints:
+
+```cpp
+DWORD now = GetTickCount();
+float dt  = (now - g_lastTick) / 1000.0f; g_lastTick = now;
+float frac, remain;
+g_est.tick(dt, frac, remain);
+g_progressFrac = frac;                 // float 0..1 used by DrawProgress
+g_progressRemain = remain;             // seconds, for the ETA label
+InvalidateRect(hWnd, &g_vuRect, FALSE);
+```
+
+**Status text** (replaces the counts-up math entirely):
+
+```cpp
+int mm = (int)g_lastAudioLen/60, ss=(int)g_lastAudioLen%60;
+int pct = (int)(g_progressFrac*100.0f + 0.5f);
+int rem = (int)(g_progressRemain + 0.5f);
+swprintf_s(buf, L"Transcribing %d:%02d  ·  %d%%  ·  %ds left", mm, ss, pct, rem);
+```
+
+**On result** (`WM_APP_RESULT`): snap, learn, persist:
+
+```cpp
+float frac, remain; g_est.finish(frac, remain);
+g_progressFrac = 1.0f; g_progressRemain = 0.0f;
+double actual = (GetTickCount() - g_busyStart) / 1000.0;
+if (g_lastAudioLen > 0.5 && actual > 0.2 && !g_cancelRequested) {
+    g_timing.add_sample(g_lastAudioLen, actual);
+    SaveTiming(g_timing, g_config.model_path);
+}
+```
+
+**`DrawProgress`** already takes a fraction — just feed `g_progressFrac` instead of `g_progress/100.0f`, and optionally add a subtle animated shimmer on the fill for life.
+
+## 2.7 Tuning & edge cases
+
+- **`alpha` (whisper trust)** 0.5 is a good start. Lower (0.3) = smoother but slower to correct; higher (0.7) = snappier, slightly jumpier.
+- **`maxCatchUp`** (2.5× real-time) caps how fast the countdown may accelerate when we over-predicted, so a correction never looks like a snap. Raise for faster catch-up, lower for calmer motion.
+- **Under-prediction** (transcription takes longer than estimated): `frac` parks at 95% and the ETA sits at a small floor until the result lands — which is the honest, expected behavior.
+- **Decay `0.97`** ≈ last ~30 runs dominate. Increase toward 0.99 for steadier long-term averages, decrease for faster adaptation to a throttling machine.
+- **Cancel / no-speech:** call `g_est.finish(...)`, reset `g_progressFrac=0`, and **don't** record a sample.
+- **Model switch mid-history:** because history is keyed per model, switching tiny.en↔base.en uses the right curve automatically.
+- **Sanity clamp:** keep `predict()`'s floor (0.4s) so ultra-short clips still show a brief, graceful sweep rather than instant 100%.
+- **Optional richer model:** if you ever want better fits on very short vs long clips, swap the linear `a+b·x` for a two-segment fit (sub-30s vs ≥30s) — the accumulators and API stay the same; just keep two `TimingModel`s.
+
+---
+
+# Part 3 — Cleanup checklist
+
+Smaller items that make the project cleaner and the app feel finished:
+
+- [ ] **Kill the pale scrollbar** (dark-mode theme or custom slim scrollbar) — Section 1.7.
+- [ ] **Delete all owner-draw chrome child windows**; keep only the transcript `EDIT` — Section 1.8.
+- [ ] **Remove every `StrokeRound(..., C_BORDER, …)`**; rely on fills + elevation.
+- [ ] **One animation clock** that idles when nothing moves (protect the 2-core CPU).
+- [ ] **DPI Per-Monitor-V2** + scaled metrics + font reload on `WM_DPICHANGED`.
+- [ ] **Reconcile the docs with reality.** `README.md`, `src/README.md`, and `src/CHANGES.md` still describe the *old* streaming architecture — "ring buffer", "24 threads", "VAD", "step_ms/length_ms", "<1s real-time GPU". The app is now **push-to-talk batch, CPU, physical-core threads, `whisper_full` once on stop**. Update or archive those docs so future-you isn't misled. (The `CUDA-SETUP.md` / `QUICK-REBUILD-GPU.md` RTX-3090 guides don't apply to the target Dell i5 either.)
+- [ ] **Status copy:** "Ready · N threads" is good; make the idle/recording/transcribing strings come from one place.
+- [ ] **Remove dead members** once streaming is gone (any leftover `step_ms`/`length_ms`/VAD config that no longer feeds `whisper_full`).
+
+---
+
+# Part 4 — Suggested build order
+
+Do them in this sequence so each step is verifiable on its own:
+
+1. **Progress system first** (Part 2). It's self-contained, low-risk, and immediately fixes the most visible "is it even working?" problem. You'll see a smooth countdown the same day.
+2. **DPI + tokens** (1.4 / 1.7). Small, mechanical, and everything after looks better for it.
+3. **Single-surface conversion** (1.6 / 1.8): convert one widget at a time — start with the ghost buttons (highest hairline payoff), then the hero, then the selects, then retire the STATICs.
+4. **Transcript chrome + scrollbar** (1.7).
+5. **Motion polish** (1.4): hover cross-fades, recording breathing, waveform, progress shimmer.
+6. **Docs reconciliation** (Part 3).
+7. *(Optional later)* **Direct2D/DirectWrite** (1.3) if you want the premium ceiling — the data model and Part 2 carry over unchanged.
+
+---
+
+*Build target reminder: native Win32 C++, MSVC Release, CPU-only on a 2-core / 4-thread i5-7th-gen. Keep the idle CPU near zero — animate only when something is actually moving.*

architecture.html

@@ -1,520 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="UTF-8">
-<meta name="viewport" content="width=device-width, initial-scale=1.0">
-<title>Win Dictation — Architecture &amp; Engineering Review</title>
-<link rel="preconnect" href="https://fonts.googleapis.com">
-<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
-<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet">
-<style>
-  :root{
-    --bg:#0E1014; --card:#16191F; --card-hi:#1E222B; --card-lo:#12151B;
-    --text:#ECEEF2; --dim:#8A909C; --faint:#5A606C;
-    --accent:#6E8BFF; --accent-hi:#839CFF; --danger:#FF5C5C; --good:#46D39A; --warn:#E8B84B;
-    --border:#262B36; --hair:rgba(255,255,255,.06);
-    --mono:'JetBrains Mono',ui-monospace,SFMono-Regular,Menlo,monospace;
-    --sans:'Inter',-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;
-  }
-  *{box-sizing:border-box}
-  html{scroll-behavior:smooth}
-  body{margin:0; background:var(--bg); color:var(--text); font-family:var(--sans);
-    font-size:16.5px; line-height:1.72; -webkit-font-smoothing:antialiased; letter-spacing:.1px}
-  .wrap{max-width:960px; margin:0 auto; padding:0 6vw 140px}
-  a{color:var(--accent-hi); text-decoration:none} a:hover{text-decoration:underline}
-
-  .hero{padding:88px 0 34px; border-bottom:1px solid var(--border); margin-bottom:50px}
-  .eyebrow{font-family:var(--mono); font-size:12.5px; letter-spacing:.32em; text-transform:uppercase; color:var(--accent); margin:0 0 22px}
-  h1{font-size:clamp(32px,5.6vw,52px); line-height:1.05; margin:0; font-weight:800; letter-spacing:-1.2px; max-width:840px}
-  .sub{font-size:20px; color:var(--dim); max-width:720px; margin:20px 0 0; font-weight:400}
-  .metarow{display:flex; flex-wrap:wrap; gap:10px; margin-top:30px}
-  .chip{font-family:var(--mono); font-size:12.5px; color:var(--dim); background:var(--card); border:1px solid var(--border); border-radius:999px; padding:7px 15px}
-  .chip b{color:var(--text); font-weight:600}
-
-  .toc{background:linear-gradient(180deg,var(--card),var(--card-lo)); border:1px solid var(--border); border-radius:18px; padding:30px 34px; margin-bottom:60px}
-  .toc h4{margin:0 0 18px; font-family:var(--mono); font-size:12px; letter-spacing:.28em; text-transform:uppercase; color:var(--faint)}
-  .toc ol{margin:0; padding:0; list-style:none; counter-reset:t; columns:2; column-gap:46px}
-  .toc li{counter-increment:t; padding:7px 0; break-inside:avoid}
-  .toc li a{color:var(--text); font-weight:500; font-size:15.5px}
-  .toc li a::before{content:counter(t,decimal-leading-zero); font-family:var(--mono); color:var(--accent); font-size:12px; margin-right:13px; font-weight:600}
-
-  section{margin:0 0 72px; scroll-margin-top:30px}
-  .sec-h{display:flex; align-items:baseline; gap:16px; margin:0 0 10px}
-  .sec-n{font-family:var(--mono); font-size:14px; color:var(--accent); font-weight:600; flex:none}
-  h2{font-size:29px; font-weight:700; margin:0; letter-spacing:-.5px}
-  h3{font-size:19px; font-weight:650; margin:38px 0 12px; letter-spacing:-.2px}
-  .lead{color:var(--dim); font-size:18px; margin:0 0 26px; max-width:740px}
-  p{margin:0 0 17px} .muted{color:var(--dim)} strong{color:#fff; font-weight:650}
-  ul,ol{margin:0 0 18px; padding-left:22px} li{margin:9px 0}
-
-  code{font-family:var(--mono); font-size:14px; background:var(--card-hi); color:#cfe0ff; padding:2px 7px; border-radius:6px; border:1px solid var(--border)}
-  .path{font-family:var(--mono); font-size:13.5px; color:var(--good)}
-  pre{background:var(--card-lo); border:1px solid var(--border); border-radius:13px; padding:20px 22px; overflow-x:auto; margin:20px 0;
-    font-family:var(--mono); font-size:13.5px; line-height:1.7; color:#c8d2e0}
-  pre .c{color:var(--faint)} pre .k{color:#c98bff} pre .s{color:var(--good)} pre .n{color:var(--accent-hi)} pre .f{color:#ffd479}
-
-  /* verdict box */
-  .verdict{background:linear-gradient(135deg,rgba(110,139,255,.12),rgba(70,211,154,.06)); border:1px solid var(--border);
-    border-radius:18px; padding:30px 34px; margin:6px 0 10px; position:relative; overflow:hidden}
-  .verdict::before{content:""; position:absolute; left:0; top:0; bottom:0; width:4px; background:linear-gradient(180deg,var(--accent),var(--good))}
-  .verdict .vh{font-family:var(--mono); font-size:11.5px; letter-spacing:.2em; text-transform:uppercase; color:var(--accent); margin:0 0 12px}
-  .verdict p{font-size:18.5px; line-height:1.66; margin:0 0 14px; color:var(--text)}
-  .verdict p:last-child{margin:0}
-  .verdict .big{font-size:22px; font-weight:700; letter-spacing:-.3px}
-
-  /* pipeline */
-  .pipe{display:flex; flex-wrap:wrap; align-items:stretch; gap:0; margin:26px 0; border:1px solid var(--border); border-radius:14px; overflow:hidden; background:var(--card-lo)}
-  .stage{flex:1 1 120px; min-width:120px; padding:18px 16px; border-right:1px solid var(--border); position:relative}
-  .stage:last-child{border-right:0}
-  .stage .si{font-family:var(--mono); font-size:11px; color:var(--faint); margin:0 0 8px}
-  .stage .sn{font-weight:650; font-size:14.5px; margin:0 0 4px; color:var(--text)}
-  .stage .sd{font-size:12.5px; color:var(--dim); line-height:1.5; margin:0}
-  .stage.hot{background:linear-gradient(180deg,rgba(110,139,255,.1),transparent)}
-  .stage.work{background:linear-gradient(180deg,rgba(70,211,154,.1),transparent)}
-
-  .tbl{width:100%; border-collapse:collapse; margin:20px 0; font-size:14.5px; border-radius:12px; overflow:hidden; border:1px solid var(--border)}
-  .tbl th{text-align:left; font-family:var(--mono); font-size:11px; letter-spacing:.1em; text-transform:uppercase; color:var(--dim); padding:12px 15px; background:var(--card-lo); border-bottom:1px solid var(--border); font-weight:600}
-  .tbl td{padding:12px 15px; border-bottom:1px solid var(--border); vertical-align:top}
-  .tbl tr:last-child td{border-bottom:0}
-  .tbl td code{font-size:13px}
-  .tbl .r{color:var(--dim); font-size:13.5px}
-
-  .note{border:1px solid var(--border); border-left:3px solid var(--accent); background:linear-gradient(90deg,rgba(110,139,255,.08),transparent 60%); border-radius:12px; padding:17px 20px; margin:22px 0}
-  .note.warn{border-left-color:var(--warn); background:linear-gradient(90deg,rgba(232,184,75,.09),transparent 60%)}
-  .note.bad{border-left-color:var(--danger); background:linear-gradient(90deg,rgba(255,92,92,.08),transparent 60%)}
-  .note.good{border-left-color:var(--good); background:linear-gradient(90deg,rgba(70,211,154,.08),transparent 60%)}
-  .note .nt{font-family:var(--mono); font-size:11.5px; letter-spacing:.18em; text-transform:uppercase; color:var(--dim); margin:0 0 6px}
-  .note p:last-child{margin:0}
-
-  /* assessment cards */
-  .assess{display:grid; gap:14px; margin:22px 0}
-  .ac{border:1px solid var(--border); border-radius:13px; padding:18px 20px; background:var(--card)}
-  .ac .ah{display:flex; align-items:center; gap:11px; margin:0 0 7px}
-  .ac .dot{width:9px;height:9px;border-radius:50%; flex:none}
-  .ac.pos .dot{background:var(--good)} .ac.neg .dot{background:var(--danger)} .ac.neu .dot{background:var(--warn)}
-  .ac h4{margin:0; font-size:16.5px; font-weight:650}
-  .ac p{margin:0; color:var(--dim); font-size:14.5px; line-height:1.62}
-  .ac .ref{font-family:var(--mono); font-size:12px; color:var(--faint); margin-top:7px}
-
-  /* scorecard */
-  .score{background:var(--card); border:1px solid var(--border); border-radius:16px; padding:28px 30px; margin:24px 0}
-  .srow{display:grid; grid-template-columns:200px 1fr 50px; align-items:center; gap:18px; padding:11px 0; border-bottom:1px solid var(--border)}
-  .srow:last-child{border-bottom:0}
-  .srow .sl{font-size:14.5px; font-weight:500}
-  .srow .sb{height:9px; background:var(--card-hi); border-radius:99px; overflow:hidden}
-  .srow .sf{height:100%; border-radius:99px; background:linear-gradient(90deg,var(--accent),var(--accent-hi))}
-  .srow .sf.hi{background:linear-gradient(90deg,#46D39A,#6ee0b0)}
-  .srow .sf.lo{background:linear-gradient(90deg,#E8B84B,#f0cd77)}
-  .srow .sf.vlo{background:linear-gradient(90deg,#FF5C5C,#ff8585)}
-  .srow .sv{font-family:var(--mono); font-size:14px; font-weight:600; text-align:right; color:var(--text)}
-  .overall{display:flex; align-items:baseline; gap:16px; margin-top:22px; padding-top:22px; border-top:1px solid var(--border)}
-  .overall .num{font-size:46px; font-weight:800; letter-spacing:-2px; color:var(--accent-hi); font-family:var(--mono)}
-  .overall .ot{color:var(--dim); font-size:15px}
-
-  /* recommendations */
-  .rec{counter-reset:r; margin:22px 0; padding:0; list-style:none}
-  .rec li{counter-increment:r; position:relative; padding:16px 18px 16px 60px; margin:0 0 12px; background:var(--card); border:1px solid var(--border); border-radius:12px}
-  .rec li::before{content:counter(r); position:absolute; left:16px; top:16px; width:30px;height:30px;border-radius:9px; background:var(--card-hi); color:var(--accent); font-family:var(--mono); font-weight:700; display:flex; align-items:center; justify-content:center; font-size:14px}
-  .rec h4{margin:0 0 4px; font-size:16px; font-weight:650}
-  .rec p{margin:0; color:var(--dim); font-size:14.5px}
-  .pri{font-family:var(--mono); font-size:10.5px; letter-spacing:.1em; text-transform:uppercase; padding:2px 8px; border-radius:5px; margin-left:9px; vertical-align:middle}
-  .pri.hi{background:rgba(255,92,92,.16); color:var(--danger)}
-  .pri.md{background:rgba(232,184,75,.16); color:var(--warn)}
-  .pri.lo{background:rgba(138,144,156,.16); color:var(--dim)}
-
-  .stackgrid{display:grid; grid-template-columns:repeat(2,1fr); gap:14px; margin:22px 0}
-  .scell{background:var(--card); border:1px solid var(--border); border-radius:12px; padding:16px 18px}
-  .scell .sk{font-family:var(--mono); font-size:11px; letter-spacing:.12em; text-transform:uppercase; color:var(--faint); margin:0 0 6px}
-  .scell .sv{font-size:15px; color:var(--text); font-weight:500}
-  .scell .sv span{color:var(--dim); font-weight:400; font-size:13.5px}
-
-  .footer{border-top:1px solid var(--border); margin-top:80px; padding-top:30px; color:var(--faint); font-size:13.5px; font-family:var(--mono)}
-  .footer b{color:var(--dim); font-weight:500}
-
-  @media(max-width:680px){
-    .toc ol,.stackgrid{columns:1; grid-template-columns:1fr}
-    .srow{grid-template-columns:1fr; gap:6px}
-    .srow .sv{text-align:left}
-    .pipe .stage{flex-basis:100%; border-right:0; border-bottom:1px solid var(--border)}
-    body{font-size:16px}
-  }
-</style>
-<style>
-.ha-img-placeholder{display:flex;align-items:center;justify-content:center;flex-direction:column;gap:6px;background:#f4f4f5;border:1px dashed #d4d4d8;border-radius:8px;color:#71717a;font-size:12px;font-family:system-ui,sans-serif;min-height:80px;padding:16px;box-sizing:border-box;animation:ha-img-pulse 1.5s ease-in-out infinite}
-.ha-img-placeholder.ha-failed{animation:none;opacity:.7}
-@keyframes ha-img-pulse{0%,100%{opacity:1}50%{opacity:.5}}
-@media(prefers-color-scheme:dark){.ha-img-placeholder{background:#27272a;border-color:#3f3f46;color:#a1a1aa}}
-</style>
-</head>
-<body>
-<div class="wrap">
-
-  <header class="hero">
-    <p class="eyebrow">Architecture · Code Analysis · Engineering Verdict</p>
-    <h1>Win Dictation: under the hood</h1>
-    <p class="sub">A guided walk through a native Win32 C++ speech-to-text app — how it's built, why it's built that way, and an honest assessment of the code for anyone picking it up for the first time.</p>
-    <div class="metarow">
-      <span class="chip"><b>~3,400</b> lines C++ (app)</span>
-      <span class="chip"><b>Win32</b> + GDI+ + SDL2 + whisper.cpp</span>
-      <span class="chip"><b>Single .exe</b>, no runtime deps</span>
-      <span class="chip"><b>Target</b> 2-core i5, CPU-only</span>
-    </div>
-  </header>
-
-  <nav class="toc">
-    <h4>Contents</h4>
-    <ol>
-      <li><a href="#tldr">The verdict, up front</a></li>
-      <li><a href="#stack">Stack at a glance</a></li>
-      <li><a href="#arch">Architecture &amp; data flow</a></li>
-      <li><a href="#decision">The defining decision</a></li>
-      <li><a href="#threads">Threading model</a></li>
-      <li><a href="#map">Source map, file by file</a></li>
-      <li><a href="#ui">Deep dive: the UI engine</a></li>
-      <li><a href="#timing">Deep dive: the progress estimator</a></li>
-      <li><a href="#transcriber">Deep dive: the transcriber</a></li>
-      <li><a href="#persistence">History, downloads &amp; persistence</a></li>
-      <li><a href="#anatomy">Anatomy of one dictation</a></li>
-      <li><a href="#strengths">What's done well</a></li>
-      <li><a href="#weaknesses">What holds it back</a></li>
-      <li><a href="#recs">Recommendations</a></li>
-      <li><a href="#scorecard">Scorecard &amp; final word</a></li>
-    </ol>
-  </nav>
-
-  <!-- 1 -->
-  <section id="tldr">
-    <div class="sec-h"><span class="sec-n">01</span><h2>The verdict, up front</h2></div>
-    <div class="verdict">
-      <p class="vh">Bottom line</p>
-      <p class="big">A genuinely strong, characterful single-purpose tool that punches well above hobby grade.</p>
-      <p>The hard engineering — the architecture choice, the self-calibrating progress estimator, the seam-free single-surface renderer — is thoughtful and well-executed. The app does exactly one thing and does it well on hardware most tools would choke on.</p>
-      <p>What holds it back is <strong>organizational debt, not algorithmic weakness</strong>: a 1,500-line <code>main.cpp</code>, a layer of vestigial child-window controls left over from an earlier design, and a pile of stale documentation that describes a GPU-streaming app this no longer is. None of it breaks the running product — but all of it raises the cost of the next person walking in.</p>
-    </div>
-    <p class="muted">The sections below back up every part of that judgement with specifics from the source.</p>
-  </section>
-
-  <!-- 2 -->
-  <section id="stack">
-    <div class="sec-h"><span class="sec-n">02</span><h2>Stack at a glance</h2></div>
-    <p class="lead">Deliberately lean. No UI framework, no managed runtime, no garbage collector — just the OS and three libraries.</p>
-    <div class="stackgrid">
-      <div class="scell"><p class="sk">Language</p><p class="sv">C++ <span>(MSVC, Release /O2 /GL /LTCG, AVX2/FMA/F16C)</span></p></div>
-      <div class="scell"><p class="sk">UI</p><p class="sv">Raw Win32 + GDI+ <span>immediate-mode painted surface</span></p></div>
-      <div class="scell"><p class="sk">Audio capture</p><p class="sv">SDL2 <span>16 kHz mono, F32</span></p></div>
-      <div class="scell"><p class="sk">Inference</p><p class="sv">whisper.cpp <span>whisper_full, CPU backend</span></p></div>
-      <div class="scell"><p class="sk">Networking</p><p class="sv">WinHTTP <span>model downloads, system proxy aware</span></p></div>
-      <div class="scell"><p class="sk">Persistence</p><p class="sv">Plain INI + UTF-8 text files <span>no database</span></p></div>
-      <div class="scell"><p class="sk">Build</p><p class="sv">CMake <span>+ a PowerShell convenience script</span></p></div>
-      <div class="scell"><p class="sk">Footprint</p><p class="sv">One .exe + a few DLLs + model <span>tiny RAM, no install</span></p></div>
-    </div>
-    <p>The whole product is native code with no framework abstraction between it and the Win32 API. That's the source of both its biggest strength (a tiny, fast, dependency-light binary) and its biggest cost (everything is hand-rolled, including the widgets).</p>
-  </section>
-
-  <!-- 3 -->
-  <section id="arch">
-    <div class="sec-h"><span class="sec-n">03</span><h2>Architecture &amp; data flow</h2></div>
-    <p class="lead">A linear pipeline with a single inference step. Audio in, text out, no streaming loop.</p>
-    <div class="pipe">
-      <div class="stage hot"><p class="si">trigger</p><p class="sn">Hotkey</p><p class="sd">Global <code>RegisterHotKey</code>. Captures the previously focused window.</p></div>
-      <div class="stage"><p class="si">capture</p><p class="sn">SDL2 mic</p><p class="sd">16 kHz mono into an in-memory buffer. Near-zero CPU.</p></div>
-      <div class="stage"><p class="si">buffer</p><p class="sn">PCM in RAM</p><p class="sd">Accumulated under a mutex; RMS energy tracked for the meter.</p></div>
-      <div class="stage work"><p class="si">inference</p><p class="sn">whisper_full</p><p class="sd">One pass on a worker thread when you stop. Trim → transcribe → clean.</p></div>
-      <div class="stage"><p class="si">deliver</p><p class="sn">Insert + paste</p><p class="sd">Text to caret, to clipboard, into the prior window.</p></div>
-    </div>
-    <p>Two side channels run alongside the main pipeline: a <strong>progress estimator</strong> that predicts and smooths the transcription countdown, and a <strong>timing model</strong> that learns this machine's speed and feeds back into the next prediction. Results return to the UI thread exclusively via <code>PostMessage</code>; shared flags are <code>std::atomic</code>.</p>
-    <div class="note">
-      <p class="nt">Mental model</p>
-      <p>Think of it as a tape recorder with a transcription button, not a live captioner. The architecture has no per-frame transcription loop at all — which, on a 2-core CPU, is the whole point.</p>
-    </div>
-  </section>
-
-  <!-- 4 -->
-  <section id="decision">
-    <div class="sec-h"><span class="sec-n">04</span><h2>The defining decision</h2></div>
-    <p class="lead">The single most important thing to understand: this app was rebuilt from a live-streaming design into a push-to-talk batch design — and that was the right call.</p>
-    <p>An earlier version used the classic whisper.cpp streaming approach: a rolling 5–6&nbsp;second window re-transcribed every ~0.4&nbsp;seconds. That technique <em>assumes</em> spare cores. On the target machine — an Intel i5-7th-gen with two physical cores — the buffer backlogged, audio was re-transcribed, and the UI starved. The symptom looked like "the model is slow"; the real cause was an architecture that needed hardware the target didn't have.</p>
-    <p>The fix wasn't a faster model. It was removing the streaming loop entirely:</p>
-    <table class="tbl">
-      <tr><th>Aspect</th><th>Old: streaming window</th><th>New: push-to-talk batch</th></tr>
-      <tr><td>CPU while speaking</td><td class="r">Pinned — constant re-inference</td><td>Near idle — just buffering</td></tr>
-      <tr><td>Inference calls</td><td class="r">Many per second</td><td>Exactly one, on stop</td></tr>
-      <tr><td>Accuracy</td><td class="r">Lower — partial context windows</td><td>Higher — full clip, full context</td></tr>
-      <tr><td>UI responsiveness</td><td class="r">Starved under load</td><td>Free until the single pass</td></tr>
-      <tr><td>Predictability</td><td class="r">Variable lag</td><td>Predictable few-second wait</td></tr>
-    </table>
-    <p>This is textbook root-cause engineering: the team correctly diagnosed that the bottleneck was the <em>shape</em> of the work, not its size, and changed the shape. Everything else in the codebase — the batch worker, the progress estimator, the physical-core thread default — follows from this one decision.</p>
-  </section>
-
-  <!-- 5 -->
-  <section id="threads">
-    <div class="sec-h"><span class="sec-n">05</span><h2>Threading model</h2></div>
-    <p class="lead">Four threads, one rule: only the UI thread touches the UI. Everything else reports back by message.</p>
-    <table class="tbl">
-      <tr><th>Thread</th><th>Lifetime</th><th>Job</th></tr>
-      <tr><td><b>UI thread</b></td><td class="r">Whole app</td><td>Message loop, all painting, the 16 ms animation timer and 50 ms update timer.</td></tr>
-      <tr><td><b>Model preload</b></td><td class="r">Detached, once</td><td>Loads the Whisper context off the UI thread at startup so the window appears instantly.</td></tr>
-      <tr><td><b>Transcribe worker</b></td><td class="r">Per clip</td><td>Runs <code>whisper_full</code>; posts <code>WM_APP_PROGRESS</code> during and <code>WM_APP_RESULT</code> when done.</td></tr>
-      <tr><td><b>Downloader</b></td><td class="r">Per download</td><td>WinHTTP fetch on its own thread; posts <code>WM_APP_DLPROGRESS</code>.</td></tr>
-    </table>
-    <p>Cross-thread state is handled with discipline rather than locks where possible: <code>std::atomic</code> booleans (<code>m_recording</code>, <code>m_busy</code>, <code>m_abort</code>, <code>g_modelLoaded</code>, <code>g_modelOk</code>) gate state transitions, and the only shared buffer — the captured PCM — is protected by a dedicated mutex. The audio callback (driven by SDL's own thread) appends under that mutex; <code>stop_and_transcribe</code> swaps the buffer out under the same lock before handing it to the worker. That swap-not-copy handoff is a nice touch.</p>
-    <pre><span class="c">// transcriber.cpp — physical cores, not logical, by design</span>
-<span class="k">int</span> Transcriber::<span class="f">default_threads</span>() {
-    <span class="k">unsigned</span> hc = std::thread::<span class="f">hardware_concurrency</span>();
-    <span class="k">if</span> (hc &lt;= 2) <span class="k">return</span> (<span class="k">int</span>)std::<span class="f">max</span>(1u, hc);
-    <span class="k">return</span> (<span class="k">int</span>)(hc / 2);   <span class="c">// 4 logical → 2 worker threads</span>
-}</pre>
-    <p>Defaulting to physical cores rather than <code>hardware_concurrency()</code> is the correct choice for compute-bound SIMD inference — hyperthreads contend for the same execution units and would only add scheduling overhead.</p>
-  </section>
-
-  <!-- 6 -->
-  <section id="map">
-    <div class="sec-h"><span class="sec-n">06</span><h2>Source map, file by file</h2></div>
-    <p class="lead">The app is small and mostly header-only outside the two big translation units. Here's where everything lives.</p>
-    <table class="tbl">
-      <tr><th>File</th><th>Role</th><th>Notes</th></tr>
-      <tr><td><code>main.cpp</code></td><td>Window, painting, interaction, settings view, clipboard, paste, model selection, popups</td><td class="r">~1,500 lines. The monolith — see §13.</td></tr>
-      <tr><td><code>transcriber.{h,cpp}</code></td><td>SDL capture, Whisper preload/inference, progress &amp; abort callbacks</td><td class="r">Clean, well-scoped class. The model layer.</td></tr>
-      <tr><td><code>timing.h</code></td><td>Per-model least-squares timing model + live progress estimator + INI persistence</td><td class="r">The standout module. See §08.</td></tr>
-      <tr><td><code>history.h</code></td><td>Session text files, UTF-8 r/w with BOM, index, pruning to 100</td><td class="r">Self-contained, header-only.</td></tr>
-      <tr><td><code>downloader.h</code></td><td>WinHTTP model downloader on a background thread</td><td class="r">.part + atomic rename, cancel, proxy-aware.</td></tr>
-      <tr><td><code>stats.h</code></td><td>Lifetime usage totals + derived figures (wpm, real-time factor, time saved)</td><td class="r">INI-backed, header-only.</td></tr>
-      <tr><td><code>settings.h</code></td><td>App settings read/write via <code>GetPrivateProfile*</code></td><td class="r">Simple and transparent.</td></tr>
-      <tr><td><code>text_util.h</code> · <code>logging.h</code></td><td>Transcript concatenation; timestamped file log</td><td class="r">Tiny helpers.</td></tr>
-      <tr><td><code>tests/test_core.cpp</code></td><td>Unit checks: append logic, bad-model handling, real-WAV transcription, progress monotonicity</td><td class="r">Modest but meaningful. Built as <code>test-core</code>.</td></tr>
-    </table>
-    <p>The decision to make most subsystems <strong>header-only and independent</strong> (<code>timing.h</code>, <code>stats.h</code>, <code>history.h</code>, <code>downloader.h</code>, <code>settings.h</code>) is a good one for a project this size: each is cohesive, individually readable, and free of cross-dependencies. The contrast with <code>main.cpp</code> — which absorbs everything else — is stark.</p>
-  </section>
-
-  <!-- 7 -->
-  <section id="ui">
-    <div class="sec-h"><span class="sec-n">07</span><h2>Deep dive: the single-surface UI engine</h2></div>
-    <p class="lead">There are no buttons. Everything you see is painted onto one double-buffered surface — and that's a deliberate fix, not a shortcut.</p>
-    <p>The previous UI composited around nine separate themed child windows (buttons, statics, an edit). That produced thin hairline seams around every control — the hard-edged holes <code>WS_CLIPCHILDREN</code> punches per child, plus the edit's themed border. Rather than chase pixel borders, the rebuild eliminated the cause: <strong>collapse the controls into one painted region.</strong></p>
-    <p>The model is a small immediate-mode system:</p>
-    <ul>
-      <li>A flat <code>Widget g_w[]</code> array — each entry is a <em>kind</em>, a rectangle, and <code>hover/pressed/anim</code> state. No HWNDs.</li>
-      <li><code>LayoutWidgets()</code> positions them; <code>PaintSurface()</code> draws each into an off-screen DC with GDI+, then blits once (no flicker).</li>
-      <li><code>HitTest()</code> maps a click point to a widget; <code>OnClick()</code> dispatches the action.</li>
-      <li>An animation clock eases each widget's <code>anim</code> toward a target (hover 0.6, active 1.0) on a 16 ms timer that <em>stops itself</em> when nothing is moving — no idle CPU burn.</li>
-    </ul>
-    <p>Two "views" — Main and Settings — render onto the same surface, toggled by <code>SwitchView()</code>. Dropdowns (mic, history) are the one exception: they're real top-level <code>WS_POPUP</code> windows, because a surface-painted dropdown would render <em>behind</em> the transcript edit (a child HWND always paints above its parent's surface). That's a correct, well-reasoned exception.</p>
-    <div class="note good">
-      <p class="nt">A sign of maturity</p>
-      <p>The popup code carries a comment never to open a <code>MessageBox</code> from inside it — because <code>WA_INACTIVE</code> self-destroys the popup mid-handler, causing a use-after-free. Recognising that class of Win32 lifetime bug, and the GDI+ "<code>GetHDC</code> locks the Graphics object" trap documented elsewhere, shows real depth.</p>
-    </div>
-    <p>The one real child window that survives is the transcript <code>EDIT</code> — kept because a hand-rolled text editor with selection, scrolling, IME and undo is genuinely not worth rebuilding. Pragmatic.</p>
-    <h3>The cost of this approach</h3>
-    <p>Custom-painted controls are invisible to screen readers and UI Automation, and the app explicitly hides focus rectangles. The transcript box is accessible; the buttons are not. For a personal productivity tool this is a defensible trade, but it's the kind of thing worth stating out loud.</p>
-  </section>
-
-  <!-- 8 -->
-  <section id="timing">
-    <div class="sec-h"><span class="sec-n">08</span><h2>Deep dive: the progress estimator</h2></div>
-    <p class="lead">The crown jewel. Most apps fake a progress bar; this one runs a small statistical model that learns your machine.</p>
-    <p>Whisper only reports coarse progress (per 30-second chunk), so a naive bar jumps — 0, 34, 72, 100 — and a naive "time left" computed from stale percentages actually counts <em>up</em>. <code>timing.h</code> solves both. It has two parts.</p>
-    <h3>1 — A learned timing model</h3>
-    <p>Processing time is modelled as a linear function of audio length, <code>proc = a + b·audio</code>, fitted by <strong>decayed online least-squares</strong>. Each completed transcription feeds back a real sample; older samples decay (factor 0.97) so the model tracks the current machine state. Defaults are seeded per model family (tiny/base/small) so even the very first clip has a sane estimate, and the accumulators persist per-model in the INI.</p>
-    <pre><span class="k">void</span> <span class="f">add_sample</span>(<span class="k">double</span> audio_sec, <span class="k">double</span> proc_sec) {
-    <span class="k">const double</span> decay = <span class="n">0.97</span>;
-    n*=decay; sx*=decay; sy*=decay; sxx*=decay; sxy*=decay;
-    n+=<span class="n">1</span>; sx+=audio_sec; sy+=proc_sec;
-    sxx+=audio_sec*audio_sec; sxy+=audio_sec*proc_sec;
-    <span class="f">recompute</span>();   <span class="c">// closed-form slope/intercept</span>
-}</pre>
-    <h3>2 — A live estimator that only counts down</h3>
-    <p>On stop, <code>begin(predict)</code> seeds a predicted total. As Whisper reports chunk progress, <code>on_whisper()</code> folds it in as a <em>measurement</em> via an EMA (α = 0.5) — nudging the estimate without the jumpy jumps. Meanwhile <code>tick()</code> advances a displayed "remaining" value that is <strong>strictly monotonic downward</strong>, with a clamped catch-up rate so it can speed up but never lurch backward, easing to 95% and snapping to 100% only on the real result.</p>
-    <pre><span class="k">void</span> <span class="f">tick</span>(<span class="k">double</span> dt, <span class="k">float</span>&amp; out_frac, <span class="k">float</span>&amp; out_remaining) {
-    t += dt; disp_rem -= dt;
-    <span class="k">double</span> raw_rem = std::<span class="f">max</span>(<span class="n">0.0</span>, T_hat - t);
-    <span class="k">double</span> err = raw_rem - disp_rem;
-    <span class="k">if</span> (err &lt; <span class="n">0</span>) disp_rem += std::<span class="f">max</span>(err, -maxCatchUp*dt);  <span class="c">// catch up, never jump back</span>
-    <span class="k">double</span> frac = t/(t+disp_rem);
-    <span class="k">if</span> (frac &gt; <span class="n">0.95</span>) frac = <span class="n">0.95</span>;   <span class="c">// park at 95% until done</span>
-    out_frac = (<span class="k">float</span>)frac; out_remaining = (<span class="k">float</span>)disp_rem;
-}</pre>
-    <p>This is more thought than most commercial apps put into a progress bar, and the test suite even asserts the progress is non-decreasing. It's the clearest signal in the codebase that someone cared about the <em>feel</em> of the product, not just its function.</p>
-  </section>
-
-  <!-- 9 -->
-  <section id="transcriber">
-    <div class="sec-h"><span class="sec-n">09</span><h2>Deep dive: the transcriber</h2></div>
-    <p class="lead">The cleanest class in the project — a tidy boundary between the OS/model and the rest of the app.</p>
-    <p><code>Transcriber</code> owns the Whisper context and the SDL device, and exposes a small, sensible surface: <code>preload</code>, <code>reload</code>, <code>start_recording</code>, <code>stop_and_transcribe</code>, <code>cancel</code>, plus state queries and two callbacks (<code>result</code>, <code>progress</code>). Inference parameters are configured sensibly for dictation — greedy sampling, no timestamps, no prior context, blank/non-speech suppression, temperature 0 — and an <code>abort_callback</code> lets a long transcription be cancelled mid-flight.</p>
-    <p>Two small details worth calling out:</p>
-    <ul>
-      <li><strong>Silence trimming.</strong> Before inference, leading/trailing silence is trimmed from the clip — cheaper and more accurate than transcribing dead air. (Note: this trims the <em>buffer</em>; it does not auto-stop recording — see the doc-drift note in §13.)</li>
-      <li><strong>Output cleanup.</strong> <code>clean_text()</code> strips Whisper's <code>[BLANK_AUDIO]</code> / <code>[NOISE]</code> artifacts and trims whitespace, so the user never sees model noise.</li>
-    </ul>
-    <p>The class is also defensively coded: a missing model file makes <code>preload</code> return false cleanly (the test suite verifies this), <code>start_recording</code> bails if a device won't open, and clips under ~0.3&nbsp;s short-circuit to an empty result rather than invoking the model.</p>
-  </section>
-
-  <!-- 10 -->
-  <section id="persistence">
-    <div class="sec-h"><span class="sec-n">10</span><h2>History, downloads &amp; persistence</h2></div>
-    <p class="lead">No database, no registry sprawl — everything is a file next to the executable. Transparent and portable.</p>
-    <h3>Session history</h3>
-    <p>Each session is one UTF-8 text file in <span class="path">history\</span>, named by timestamp. The clever bit is <em>live</em> archiving: the first clip of a session creates the file; subsequent clips rewrite the <em>same</em> file with the full text. So a session is always one tidy, crash-safe file — not a scatter of fragments — and it appears in the History popup immediately. A <code>g_sessionPath</code> global plus a <code>FinalizeSession()</code> helper handle the edge cases (manual edits, typed-only sessions, loading an old entry without resurrecting it). The list is capped at 100 with automatic pruning.</p>
-    <div class="note">
-      <p class="nt">A real bug was fixed here</p>
-      <p>An earlier version stamped every fresh transcription as "loaded from history," so the duplicate-guard silently skipped archiving — sessions never reached disk while the UI claimed "Saved." The fix (live archiving + a corrected guard) is documented and shows the team chasing subtle state bugs to ground.</p>
-    </div>
-    <h3>Model downloads</h3>
-    <p>The downloader is more robust than it needed to be, in a good way: it streams to a <code>.part</code> file then does an atomic rename on success (no half-files), honors the system proxy, follows the Hugging Face → CDN redirects, supports cancellation, allows only one download at a time, and sweeps up stray <code>.part</code> files at startup.</p>
-    <h3>Settings, timing &amp; stats</h3>
-    <p>All three live in a single <span class="path">win-dictation.ini</span> under different sections — app settings, per-model timing accumulators, and lifetime stats. Using the OS's own <code>GetPrivateProfile*</code> API means zero parsing code and a file a user can read and edit by hand. For an app of this scope, that's exactly the right level of machinery.</p>
-  </section>
-
-  <!-- 11 -->
-  <section id="anatomy">
-    <div class="sec-h"><span class="sec-n">11</span><h2>Anatomy of one dictation</h2></div>
-    <p class="lead">Following a single clip end-to-end ties the whole system together.</p>
-    <ol>
-      <li><code>WM_HOTKEY</code> fires → the app records <code>g_prevForeground</code> and the current selection (<code>EM_GETSEL</code>) so it knows where to paste and where to insert.</li>
-      <li><code>g_tx.start_recording()</code> opens the SDL device; the audio callback appends PCM under the capture mutex and updates the RMS energy meter.</li>
-      <li>The 50 ms UI timer animates the level meter and ticks the on-screen recording clock.</li>
-      <li>Second <code>WM_HOTKEY</code> → <code>g_est.begin(g_timing.predict(len))</code> seeds the progress estimate; <code>g_tx.stop_and_transcribe()</code> swaps the buffer to a worker thread.</li>
-      <li>The worker runs <code>run_inference()</code> → <code>whisper_full</code>. Whisper's progress callback posts <code>WM_APP_PROGRESS</code>; the estimator's <code>on_whisper()</code> EMA-folds it in.</li>
-      <li>On completion the worker posts <code>WM_APP_RESULT</code>.</li>
-      <li>The UI thread then, in order: snaps progress to 100%, records a real timing sample (<code>add_sample</code> + <code>SaveTiming</code>), updates lifetime stats, inserts the text at the saved caret with smart spacing via <code>EM_REPLACESEL</code> (undoable), archives the session, copies to the clipboard, and pastes into <code>g_prevForeground</code>.</li>
-    </ol>
-    <p>Every piece of the architecture shows up in that one trip: the atomics, the message hand-back, the estimator, the learned timing feedback loop, the editable transcript, the live history. It's a coherent design.</p>
-  </section>
-
-  <!-- 12 -->
-  <section id="strengths">
-    <div class="sec-h"><span class="sec-n">12</span><h2>What's done well</h2></div>
-    <div class="assess">
-      <div class="ac pos"><div class="ah"><span class="dot"></span><h4>The architecture fits the hardware</h4></div><p>Push-to-talk batch over streaming is the correct response to a 2-core CPU, reached by genuine root-cause analysis rather than knob-twiddling.</p></div>
-      <div class="ac pos"><div class="ah"><span class="dot"></span><h4>The progress estimator is exceptional</h4></div><p>Decayed online least-squares + EMA fusion + a strictly monotonic countdown is far beyond what the task demanded — and it shows in the feel.</p></div>
-      <div class="ac pos"><div class="ah"><span class="dot"></span><h4>Seam-free UI by elimination, not patching</h4></div><p>Collapsing nine child windows into one painted, double-buffered, DPI-aware, self-throttling surface removed the problem at its source.</p></div>
-      <div class="ac pos"><div class="ah"><span class="dot"></span><h4>Clean module boundaries (outside main)</h4></div><p>Header-only, dependency-free subsystems (timing, history, downloader, stats, settings) are each individually readable and testable.</p></div>
-      <div class="ac pos"><div class="ah"><span class="dot"></span><h4>Robustness in the right places</h4></div><p>Atomic rename downloads, crash-safe live history, graceful missing-model handling, cancellable inference, single-instance mutex, model preload off the UI thread.</p></div>
-      <div class="ac pos"><div class="ah"><span class="dot"></span><h4>Real product thoughtfulness</h4></div><p>Smart insertion spacing, undoable edits, auto-paste into the prior window, auto-hide, learned timing, friendly stats. These are details a careful builder adds.</p></div>
-    </div>
-  </section>
-
-  <!-- 13 -->
-  <section id="weaknesses">
-    <div class="sec-h"><span class="sec-n">13</span><h2>What holds it back</h2></div>
-    <p class="lead">All fixable, and none of it affects the running app. But it's exactly what a newcomer trips over.</p>
-    <div class="assess">
-      <div class="ac neu"><div class="ah"><span class="dot"></span><h4><code>main.cpp</code> is a 1,500-line god object</h4></div><p>UI, layout, painting, the entire settings screen, clipboard, paste mechanics, model selection, the popup window class, and stats formatting all live in one translation unit with dozens of globals. It works, but it's the hardest part of the codebase to onboard into. Splitting the settings view, the popup, and the painting helpers into their own files would pay for itself quickly.</p></div>
-      <div class="ac neu"><div class="ah"><span class="dot"></span><h4>Vestigial child windows &amp; duplicate code paths</h4></div><p>Startup still creates ~9 owner-draw child controls (record, pin, copy, paste, clear, two selects, two statics) and then immediately hides all but the transcript edit. Their dead <code>WM_COMMAND</code> handlers duplicate the painted-widget <code>OnClick</code> logic — e.g. the Copy action exists in two near-identical places. Leftovers from the rebuild that should be deleted.</p><p class="ref">main.cpp — CreateWindow(...) blocks then ShowWindow(..., SW_HIDE)</p></div>
-      <div class="ac neg"><div class="ah"><span class="dot"></span><h4>Documentation describes a different app</h4></div><p>This is the most actively misleading issue. <code>CUDA-SETUP.md</code>, <code>QUICK-REBUILD-GPU.md</code> and <code>FIXES-APPLIED.md</code> describe a streaming, VAD, ring-buffer, 24-thread, RTX 3090 design that no longer exists. <code>build.ps1</code> still hunts for CUDA and downloads <code>base.en</code> though the product is a CPU-only <code>tiny.en</code> app. <code>TESTING.md</code> references a <code>test-audio.exe</code> the CMake doesn't build (it builds <code>test-core</code>). A newcomer reading the docs would form a completely wrong mental model.</p></div>
-      <div class="ac neg"><div class="ah"><span class="dot"></span><h4>The README claims a feature that isn't there</h4></div><p>Both <code>README.md</code> and <code>CHANGES.md</code> describe a "500 ms silence auto-end timer." The recording loop has no such logic — it only auto-stops at the 10-minute safety cap. (Silence is <em>trimmed</em> before inference, which is likely the source of the confusion.) Either implement it or remove the claim.</p><p class="ref">main.cpp WM_TIMER recording branch vs README "Audio Processing"</p></div>
-      <div class="ac neu"><div class="ah"><span class="dot"></span><h4>Heavy reliance on global mutable state</h4></div><p>The UI is coordinated through dozens of file-scope globals (<code>g_*</code>), a mix of atomics and plain values. It's manageable at this size and the threading is disciplined, but it makes the code hard to reason about in isolation and easy to break with a careless edit.</p></div>
-      <div class="ac neu"><div class="ah"><span class="dot"></span><h4>Build declares C++11 but uses C++17</h4></div><p><code>CMakeLists.txt</code> sets <code>CMAKE_CXX_STANDARD 11</code>, yet the code uses <code>std::size()</code> (C++17). It compiles only because MSVC's default is newer. Set the standard to 17 explicitly so the build is honest and portable.</p></div>
-      <div class="ac neu"><div class="ah"><span class="dot"></span><h4>Minor: redundant color systems &amp; no in-app hotkey editor</h4></div><p>Three overlapping palettes coexist (<code>CR_*</code> COLORREF, <code>T_*</code> GDI+ Color, <code>C_*</code> aliases). And changing the hotkey requires hand-editing the INI — a natural gap given the polished Settings screen already exists.</p></div>
-    </div>
-  </section>
-
-  <!-- 14 -->
-  <section id="recs">
-    <div class="sec-h"><span class="sec-n">14</span><h2>Recommendations</h2></div>
-    <p class="lead">If the next session had a short to-do list, this would be it — ordered by payoff for effort.</p>
-    <ol class="rec">
-      <li><h4>Purge or archive the stale docs <span class="pri hi">high</span></h4><p>Delete or clearly mark <code>CUDA-SETUP.md</code>, <code>QUICK-REBUILD-GPU.md</code>, <code>FIXES-APPLIED.md</code>, <code>TESTING.md</code> and <code>DESIGN.md</code> as describing the retired streaming design. This is the single biggest improvement to onboarding, and it's nearly free.</p></li>
-      <li><h4>Reconcile the README with reality <span class="pri hi">high</span></h4><p>Remove the "500 ms silence auto-end" claim (or implement it). Update the model table and build commands to match the CPU-only product.</p></li>
-      <li><h4>Delete the vestigial child windows <span class="pri md">medium</span></h4><p>Remove the hidden owner-draw controls and their dead <code>WM_COMMAND</code> handlers so there's exactly one code path per action. De-duplicate Copy.</p></li>
-      <li><h4>Break up <code>main.cpp</code> <span class="pri md">medium</span></h4><p>Lift the Settings view, the popup window, and the GDI+ drawing helpers into their own files. Even a mechanical split dramatically improves navigability.</p></li>
-      <li><h4>Fix the build standard &amp; align <code>build.ps1</code> <span class="pri md">medium</span></h4><p>Set <code>CMAKE_CXX_STANDARD 17</code>. Strip the CUDA detection from the build script and default it to fetching <code>tiny.en</code>.</p></li>
-      <li><h4>Add an in-app hotkey picker <span class="pri lo">low</span></h4><p>The Settings surface already exists; surfacing the hotkey there closes an obvious UX gap and removes a troubleshooting step.</p></li>
-    </ol>
-  </section>
-
-  <!-- 15 -->
-  <section id="scorecard">
-    <div class="sec-h"><span class="sec-n">15</span><h2>Scorecard &amp; final word</h2></div>
-    <div class="score">
-      <div class="srow"><span class="sl">Architecture &amp; design</span><span class="sb"><span class="sf hi" style="width:92%"></span></span><span class="sv">9.2</span></div>
-      <div class="srow"><span class="sl">Performance fit for target</span><span class="sb"><span class="sf hi" style="width:93%"></span></span><span class="sv">9.3</span></div>
-      <div class="srow"><span class="sl">UX &amp; polish</span><span class="sb"><span class="sf hi" style="width:87%"></span></span><span class="sv">8.7</span></div>
-      <div class="srow"><span class="sl">Robustness &amp; error handling</span><span class="sb"><span class="sf" style="width:75%"></span></span><span class="sv">7.5</span></div>
-      <div class="srow"><span class="sl">Code organization</span><span class="sb"><span class="sf lo" style="width:55%"></span></span><span class="sv">5.5</span></div>
-      <div class="srow"><span class="sl">Maintainability</span><span class="sb"><span class="sf lo" style="width:58%"></span></span><span class="sv">5.8</span></div>
-      <div class="srow"><span class="sl">Testing</span><span class="sb"><span class="sf lo" style="width:48%"></span></span><span class="sv">4.8</span></div>
-      <div class="srow"><span class="sl">Documentation accuracy</span><span class="sb"><span class="sf vlo" style="width:38%"></span></span><span class="sv">3.8</span></div>
-      <div class="overall"><span class="num">7.1</span><span class="ot"><strong style="color:var(--text)">Strong, with cleanup debt.</strong><br>An impressive core wrapped in organizational and documentation drift. The engineering earns a high mark; the housekeeping pulls the average down.</span></div>
-    </div>
-    <div class="verdict" style="margin-top:30px">
-      <p class="vh">Final word</p>
-      <p>Win Dictation is a <strong>good codebase — at its core, an impressive one</strong>. The architectural judgement (batch over streaming), the standout progress estimator, and the seam-free renderer are the work of someone who diagnoses root causes and cares about how software feels. Those are the hard parts, and they're done right.</p>
-      <p>What separates it from "great" is entirely recoverable: a monolithic main file, dead code from a prior design, and documentation that actively describes a different application. A focused day of cleanup — most of it deletion — would lift this from "strong for its niche" to "exemplary small-app code." The good news for anyone inheriting it: the bones are excellent, and the to-do list is short.</p>
-    </div>
-    <div class="footer">
-      <b>Win Dictation — Architecture &amp; Engineering Review</b><br>
-      Native Win32 C++ · GDI+ · SDL2 · whisper.cpp · CPU-only · target Intel i5-7th-gen (2C/4T)<br>
-      Assessment based on a full read of the current source tree.
-    </div>
-  </section>
-
-</div>
-<script>
-document.addEventListener('keydown', function(e) {
-  if (e.key === 'Escape' && window.parent !== window) {
-    window.parent.postMessage({ type: 'close-fullscreen' }, '*');
-  }
-});
-</script>
-<!-- broken-img-handler -->
-<script>
-(function(){
-  if(window.__brokenImgHandler)return;
-  window.__brokenImgHandler=true;
-  var MAX=5,DELAYS=[2000,4000,8000,16000,32000];
-  document.addEventListener('error',function(e){
-    var img=e.target;
-    if(!img||img.tagName!=='IMG')return;
-    var liveSrc=img.getAttribute('src');
-    var src=img.dataset.haOriginalSrc||liveSrc;
-    if(!src)return;
-    if(img.dataset.haOriginalSrc&&liveSrc&&liveSrc!==img.dataset.haOriginalSrc&&liveSrc.indexOf('_r=')<0){src=liveSrc;img.dataset.haOriginalSrc=src;img.dataset.haRetryCount='0'}
-    else if(!img.dataset.haOriginalSrc){img.dataset.haOriginalSrc=src}
-    var attempt=parseInt(img.dataset.haRetryCount||'0',10);
-    if(img.dataset.haPhId){var old=document.getElementById(img.dataset.haPhId);if(old)old.remove()}
-    var ph=document.createElement('div');
-    ph.className='ha-img-placeholder'+(attempt>=MAX?' ha-failed':'');
-    ph.id='ha-ph-'+Math.random().toString(36).slice(2,9);
-    var w=img.getAttribute('width');var h=img.getAttribute('height');
-    if(w)ph.style.width=w+(isNaN(Number(w))?'':'px');
-    else if(img.style.width)ph.style.width=img.style.width;
-    else if(img.width>1)ph.style.width=img.width+'px';
-    if(h)ph.style.height=h+(isNaN(Number(h))?'':'px');
-    else if(img.style.height)ph.style.height=img.style.height;
-    else if(img.height>1)ph.style.height=img.height+'px';
-    ph.textContent=attempt>=MAX?'Image unavailable':'Loading image\u2026';
-    img.dataset.haPhId=ph.id;
-    if(img.dataset.haOrigDisplay==null)img.dataset.haOrigDisplay=img.style.display||'';
-    img.style.display='none';
-    img.insertAdjacentElement('afterend',ph);
-    if(attempt<MAX){
-      img.dataset.haRetryCount=String(attempt+1);
-      setTimeout(function(){
-        if(!img.isConnected)return;
-        if(img.dataset.haOriginalSrc!==src)return;
-        if(img.complete&&img.naturalWidth>0)return;
-        var curSrc=img.getAttribute('src');
-        if(curSrc&&curSrc.indexOf(src)!==0)return;
-        var fresh=src+(src.indexOf('?')>=0?'&':'?')+'_r='+(attempt+1)+'_'+Date.now();
-        img.src=fresh;
-      },DELAYS[attempt]);
-    }
-  },true);
-  document.addEventListener('load',function(e){
-    var img=e.target;
-    if(!img||img.tagName!=='IMG')return;
-    if(img.dataset.haPhId){
-      var ph=document.getElementById(img.dataset.haPhId);
-      if(ph)ph.remove();
-      delete img.dataset.haPhId;
-      img.style.display=img.dataset.haOrigDisplay||'';
-      delete img.dataset.haOrigDisplay;
-      delete img.dataset.haOriginalSrc;
-      delete img.dataset.haRetryCount;
-    }
-  },true);
-})();
-</script>
-</body>
-</html>

bug-fixes-and-visual-upgrade.md

@@ -0,0 +1,790 @@
+# Win Dictation — Crash Fixes & Modern UI (Build 2)
+
+This document has two parts:
+
+- **Part 1 — Critical fixes** (do these first): the Stop crash, resizing, model switching, thread count, model path.
+- **Part 2 — Modern UI**: replace the boxy Win32 look with a flat, rounded, dark "web-app" interface (GDI+ custom drawing).
+
+Notes before you start:
+
+- All snippets target your current `main.cpp` / `transcriber.cpp` / `transcriber.h`.
+- I can't compile on my side — integrate a piece at a time and build often. If a `DWMWA_*` or glyph constant is missing, see the `#define` block in §2.3.
+- Part 2's `LayoutControls()` (§2.12) is the final layout and supersedes any basic one from Part 1.
+
+---
+
+# Part 1 — Critical Fixes
+
+## 1.1 The Stop crash (null Whisper context)
+
+**Symptom:** record + VU work, then it crashes the moment you hit Stop.
+
+**Root cause:** two defects combine.
+
+1. In `wWinMain`, the preload thread sets `g_modelLoaded = true` *even when `preload()` returns false* (model file not found). So the UI shows "Ready" and lets you record with no model loaded.
+2. `transcribe_worker()` calls `whisper_reset_timings(m_ctx)` **before** the `if (m_ctx)` check. With `m_ctx == nullptr` that's an immediate access violation. Recording never touches Whisper (that's why capture + VU work) — the crash only fires when transcription starts.
+
+The empty model dropdown in your screenshot confirms it: `RefreshModelList()` found no model files, so `preload()` failed silently.
+
+### Fix A — make the worker null-safe (`transcriber.cpp`)
+
+Replace the top of `transcribe_worker` and delete the `whisper_reset_timings` call entirely (it isn't needed for one-shot transcription):
+
+```cpp
+void Transcriber::transcribe_worker(std::vector<float> audio) {
+    // Hard guard: if the model failed to load, never touch whisper.
+    if (!m_ctx) {
+        m_busy = false;
+        if (m_on_result) m_on_result("");   // UI will show "No speech / not loaded"
+        return;
+    }
+
+    if (m_cfg.trim_silence) trim_silence(audio);
+
+    whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
+    wp.print_progress = false; wp.print_realtime = false; wp.print_timestamps = false;
+    wp.no_timestamps  = true;  wp.translate = false;
+    wp.language       = m_cfg.language.c_str();
+    wp.n_threads      = m_cfg.n_threads;
+    wp.no_context     = true;  wp.suppress_blank = true; wp.suppress_nst = true;
+    wp.temperature    = 0.0f;
+    // (whisper_reset_timings removed — it was the crash site and is unnecessary)
+
+    std::string out;
+    if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
+        int n = whisper_full_n_segments(m_ctx);
+        for (int i = 0; i < n; ++i) {
+            const char* t = whisper_full_get_segment_text(m_ctx, i);
+            if (t) out += t;
+        }
+        out = clean_text(out);
+    }
+    m_busy = false;
+    if (m_on_result) m_on_result(out);
+}
+```
+
+### Fix B — track load success honestly (`main.cpp`)
+
+Add an atomic next to `g_modelLoaded`:
+
+```cpp
+std::atomic<bool> g_modelLoaded{false};
+std::atomic<bool> g_modelOk{false};
+```
+
+Fix the preload lambda so failure is recorded:
+
+```cpp
+std::thread([] {
+    bool ok = g_tx.preload(g_config);   // false if the .bin isn't found
+    g_modelOk = ok;
+    g_modelLoaded = true;
+}).detach();
+```
+
+### Fix C — refuse to record without a model (`main.cpp`, `WM_HOTKEY`)
+
+This both prevents the crash path and tells you *why* nothing happens:
+
+```cpp
+case WM_HOTKEY:
+    if (wParam == HK_TOGGLE) {
+        if (g_tx.is_busy()) break;                 // mid-transcription: ignore
+        if (!g_tx.is_recording()) {
+            if (!g_modelLoaded.load()) { SetStatus(hWnd, L"Loading model…"); break; }
+            if (!g_modelOk.load()) {
+                std::wstring m = L"Model not found:\n" + to_w(g_config.model_path)
+                               + L"\n\nPut the .bin there and restart.";
+                MessageBoxW(hWnd, m.c_str(), L"Dictation", MB_OK | MB_ICONWARNING);
+                break;
+            }
+            g_prevForeground = GetForegroundWindow();
+            ShowWindow(hWnd, SW_SHOWNA);
+            g_recordingSecs = 0;
+            if (g_tx.start_recording()) {
+                SetDlgItemText(hWnd, ID_BTN_RECORD, L"■  Stop");
+                SetStatus(hWnd, L"Recording…");
+            } else SetStatus(hWnd, L"Microphone error");
+        } else {
+            g_tx.stop_and_transcribe();
+            SetDlgItemText(hWnd, ID_BTN_RECORD, L"Record");
+            SetStatus(hWnd, L"Transcribing…");
+        }
+    } else if (wParam == HK_HIDE) {
+        if (g_tx.is_recording()) g_tx.cancel();
+        ShowWindow(hWnd, SW_HIDE);
+    }
+    break;
+```
+
+After this, a missing model shows **"Model not found: …\models\ggml-tiny.en.bin"** instead of crashing.
+
+## 1.2 Verify the model file
+
+The app now loads the model from a path relative to the **.exe**:
+`exe_dir() + "\\models\\ggml-tiny.en.bin"`. Confirm it's actually there:
+
+```
+dir build\bin\Release\models
+```
+
+You should see `ggml-tiny.en.bin`. If it's missing, re-run `build.ps1`, or drop the `.bin` into that `models\` folder manually. (Your CMake post-build step copies `models/` next to the exe — if the source `models/` had no `.bin` at build time, nothing got copied.)
+
+## 1.3 Resizable window
+
+Two reasons it's frozen: the window style has no sizing border, and there's no `WM_SIZE` handler so controls never reflow.
+
+### Window style (`main.cpp`, `CreateWindowExW`)
+
+> If you do Part 2 Option A (§2.3), keep a normal frame as shown here. If you do the borderless Option B (appendix), that section replaces this.
+
+```cpp
+hMainWnd = CreateWindowExW(
+    WS_EX_TOPMOST,
+    L"WhisperDictationClass", L"Dictation",
+    WS_OVERLAPPEDWINDOW,            // caption + sysmenu + THICKFRAME + min/max = resizable
+    CW_USEDEFAULT, CW_USEDEFAULT, 400, 340,
+    nullptr, nullptr, hInstance, nullptr);
+```
+
+### Handlers (`main.cpp`, `WndProc`)
+
+```cpp
+case WM_SIZE:
+    LayoutControls(hWnd, LOWORD(lParam), HIWORD(lParam));   // defined in §2.12
+    return 0;
+
+case WM_GETMINMAXINFO:
+    ((MINMAXINFO*)lParam)->ptMinTrackSize.x = 340;
+    ((MINMAXINFO*)lParam)->ptMinTrackSize.y = 280;
+    return 0;
+```
+
+Control positions in `InitializeUI` no longer matter — `LayoutControls` owns geometry. Add one explicit call right after `ShowWindow` so the first frame is laid out:
+
+```cpp
+RECT rc; GetClientRect(hMainWnd, &rc);
+LayoutControls(hMainWnd, rc.right, rc.bottom);
+```
+
+## 1.4 Model switching + correct thread count
+
+Two real bugs you'll hit:
+
+- The model dropdown only lists files that **exist**, so its selected index does **not** map to `kModelFiles[]`.
+- `preload()` early-returns when `m_ctx` is already set, so switching models **never reloads**.
+- The status bar shows `hardware_concurrency()` (4) instead of the threads actually used (2).
+
+### transcriber.h — add a reload + a threads getter
+
+```cpp
+bool reload(const WhisperConfig& cfg);          // free + re-init with a new model
+int  threads() const { return m_cfg.n_threads; }
+```
+
+### transcriber.cpp — implement reload
+
+```cpp
+bool Transcriber::reload(const WhisperConfig& cfg) {
+    if (m_recording.load() || m_busy.load()) return false;  // not mid-use
+    if (m_worker.joinable()) m_worker.join();
+    if (m_ctx) { whisper_free(m_ctx); m_ctx = nullptr; }
+    return preload(cfg);
+}
+```
+
+### main.cpp — track the real file paths the combo shows
+
+Replace the static arrays + `RefreshModelList` + the `ID_COMBO_MODEL` handler:
+
+```cpp
+static const wchar_t* kModelNames[] = { L"tiny.en", L"tiny.en-q8_0", L"base.en-q5_1", L"base.en" };
+static const char*    kModelFiles[] = {
+    "models\\ggml-tiny.en.bin", "models\\ggml-tiny.en-q8_0.bin",
+    "models\\ggml-base.en-q5_1.bin", "models\\ggml-base.en.bin",
+};
+std::vector<std::string> g_modelComboPaths;   // parallel to combo entries
+
+void RefreshModelList(HWND hwnd) {
+    HWND hCombo = GetDlgItem(hwnd, ID_COMBO_MODEL);
+    SendMessage(hCombo, CB_RESETCONTENT, 0, 0);
+    g_modelComboPaths.clear();
+    std::string dir = exe_dir();
+    for (int i = 0; i < 4; ++i) {
+        std::string full = dir + "\\" + kModelFiles[i];
+        if (GetFileAttributesA(full.c_str()) != INVALID_FILE_ATTRIBUTES) {
+            SendMessage(hCombo, CB_ADDSTRING, 0, (LPARAM)kModelNames[i]);
+            g_modelComboPaths.push_back(kModelFiles[i]);   // remember the real file
+        }
+    }
+    if (!g_modelComboPaths.empty()) SendMessage(hCombo, CB_SETCURSEL, 0, 0);
+}
+```
+
+```cpp
+case ID_COMBO_MODEL:
+    if (HIWORD(wParam) == CBN_SELCHANGE) {
+        int idx = (int)SendMessage(GetDlgItem(hWnd, ID_COMBO_MODEL), CB_GETCURSEL, 0, 0);
+        if (idx >= 0 && idx < (int)g_modelComboPaths.size()) {
+            g_config.model_path = exe_dir() + "\\" + g_modelComboPaths[idx];
+            g_modelLoaded = false; g_modelOk = false;
+            SetStatus(hWnd, L"Loading model…");
+            std::thread([] {
+                bool ok = g_tx.reload(g_config);   // <- reload, not preload
+                g_modelOk = ok; g_modelLoaded = true;
+            }).detach();
+        }
+    }
+    break;
+```
+
+### Correct thread count in the status (`UpdateStatus`)
+
+```cpp
+} else if (g_modelLoaded.load()) {
+    if (!g_modelOk.load()) { SetStatus(hwnd, L"Model not found — check models folder"); return; }
+    swprintf_s(buf, L"Ready  •  %d threads", g_tx.threads());
+    SetStatus(hwnd, buf);
+}
+```
+
+---
+
+# Part 2 — Modern UI (flat, rounded, dark)
+
+## 2.1 Design language
+
+The "boxy" feeling comes from three things: the gray 3-D system buttons, the chunky title bar, and zero breathing room. We fix all three.
+
+- **Palette** — near-black canvas, one elevated surface tone, a single indigo accent, hairline borders. (Linear/Raycast register.)
+- **Shape** — everything rounded: 10px buttons/fields, 14px panels, rounded window corners.
+- **Type** — Segoe UI Variable / Segoe UI, a clear size hierarchy, no monospace for the transcript (sans reads more "app").
+- **Space** — 16px outer margin, 10–12px gaps, the transcript is the hero and grows with the window.
+- **Motion** — a soft pulse on the record button while recording; subtle hover/press on buttons.
+
+| Token | Hex | Use |
+|---|---|---|
+| Bg | `#0F1115` | window canvas |
+| Surface | `#181B22` | fields, transcript panel |
+| SurfaceHi | `#20242D` | hover |
+| Border | `#262B36` | 1px hairlines |
+| Text | `#E7E9EE` | primary text |
+| TextDim | `#9AA0AB` | status, placeholders |
+| Accent | `#6E8BFF` | record idle, focus |
+| AccentHi | `#5B7BFF` | accent hover |
+| Danger | `#FF5C5C` | recording / stop |
+| Good | `#46D39A` | VU meter |
+
+## 2.2 How it's built
+
+- **GDI+** does the drawing — it anti-aliases rounded rectangles and supports alpha, so we get smooth corners without Direct2D.
+- **Owner-drawn child buttons** for Record / Copy / Paste / Pin (we paint them; Windows still gives us click + focus).
+- **Parent-painted** background, transcript panel frame, and the VU meter (in `WM_PAINT`).
+- **Native EDIT** stays for the transcript (so selection/scroll work) but flat-themed with padding; a STATIC shows placeholder text when empty.
+- **Owner-drawn comboboxes** for mic/model so they match the dark theme.
+- A tiny **button subclass** gives reliable hover.
+
+This keeps real controls (accessibility, IME, scrolling) while looking custom.
+
+## 2.3 Setup: GDI+ and a dark, rounded window (Option A — recommended)
+
+Option A keeps the native frame but recolors it dark and rounds the corners — low-risk and modern. (Option B, fully borderless custom title bar, is in the appendix.)
+
+At the top of `main.cpp`:
+
+```cpp
+#include <gdiplus.h>
+#pragma comment(lib, "gdiplus.lib")
+using namespace Gdiplus;
+
+// DWM attributes (define in case your SDK headers are older)
+#ifndef DWMWA_USE_IMMERSIVE_DARK_MODE
+#define DWMWA_USE_IMMERSIVE_DARK_MODE 20
+#endif
+#ifndef DWMWA_BORDER_COLOR
+#define DWMWA_BORDER_COLOR 34
+#endif
+#ifndef DWMWA_CAPTION_COLOR
+#define DWMWA_CAPTION_COLOR 35
+#endif
+#ifndef DWMWA_TEXT_COLOR
+#define DWMWA_TEXT_COLOR 36
+#endif
+#ifndef DWMWA_WINDOW_CORNER_PREFERENCE
+#define DWMWA_WINDOW_CORNER_PREFERENCE 33
+#endif
+#ifndef DWMWCP_ROUND
+#define DWMWCP_ROUND 2
+#endif
+```
+
+Start/stop GDI+ in `wWinMain`:
+
+```cpp
+ULONG_PTR g_gdipToken = 0;
+// near the top of wWinMain, before creating the window:
+GdiplusStartupInput gdipIn;
+GdiplusStartup(&g_gdipToken, &gdipIn, nullptr);
+// ... after the message loop, before return:
+GdiplusShutdown(g_gdipToken);
+```
+
+After the window is created, theme the frame (Win11 honors all of these; Win10 ignores caption/border color but keeps dark mode):
+
+```cpp
+BOOL dark = TRUE;
+DwmSetWindowAttribute(hMainWnd, DWMWA_USE_IMMERSIVE_DARK_MODE, &dark, sizeof(dark));
+COLORREF cap = RGB(0x0F,0x11,0x15), bord = RGB(0x26,0x2B,0x36), txt = RGB(0xE7,0xE9,0xEE);
+DwmSetWindowAttribute(hMainWnd, DWMWA_CAPTION_COLOR, &cap,  sizeof(cap));
+DwmSetWindowAttribute(hMainWnd, DWMWA_BORDER_COLOR,  &bord, sizeof(bord));
+DwmSetWindowAttribute(hMainWnd, DWMWA_TEXT_COLOR,    &txt,  sizeof(txt));
+int corner = DWMWCP_ROUND;
+DwmSetWindowAttribute(hMainWnd, DWMWA_WINDOW_CORNER_PREFERENCE, &corner, sizeof(corner));
+```
+
+## 2.4 Theme constants + fonts (`main.cpp`)
+
+```cpp
+// GDI+ colors
+static const Color C_BG       (255,0x0F,0x11,0x15);
+static const Color C_SURFACE  (255,0x18,0x1B,0x22);
+static const Color C_SURFACEHI(255,0x20,0x24,0x2D);
+static const Color C_BORDER   (255,0x26,0x2B,0x36);
+static const Color C_TEXT     (255,0xE7,0xE9,0xEE);
+static const Color C_TEXTDIM  (255,0x9A,0xA0,0xAB);
+static const Color C_ACCENT   (255,0x6E,0x8B,0xFF);
+static const Color C_ACCENTHI (255,0x5B,0x7B,0xFF);
+static const Color C_DANGER   (255,0xFF,0x5C,0x5C);
+static const Color C_GOOD     (255,0x46,0xD3,0x9A);
+// COLORREF mirrors for GDI (WM_CTLCOLOR*)
+#define CR_BG      RGB(0x0F,0x11,0x15)
+#define CR_SURFACE RGB(0x18,0x1B,0x22)
+#define CR_TEXT    RGB(0xE7,0xE9,0xEE)
+
+HFONT g_fUI=nullptr, g_fUISemi=nullptr, g_fSmall=nullptr, g_fText=nullptr;
+HBRUSH g_brBg=nullptr, g_brSurface=nullptr;
+
+static HFONT MakeFont(int px, int weight) {
+    return CreateFontW(-px,0,0,0,weight,FALSE,FALSE,FALSE,DEFAULT_CHARSET,
+        OUT_DEFAULT_PRECIS,CLIP_DEFAULT_PRECIS,CLEARTYPE_QUALITY,
+        DEFAULT_PITCH|FF_DONTCARE,L"Segoe UI Variable Display");
+}
+// build once in wWinMain:
+//   g_fUI = MakeFont(15, FW_NORMAL);  g_fUISemi = MakeFont(15, FW_SEMIBOLD);
+//   g_fSmall = MakeFont(12, FW_NORMAL); g_fText = MakeFont(16, FW_NORMAL);
+//   g_brBg = CreateSolidBrush(CR_BG); g_brSurface = CreateSolidBrush(CR_SURFACE);
+// (If "Segoe UI Variable Display" is unavailable it falls back to Segoe UI.)
+```
+
+## 2.5 GDI+ helpers
+
+```cpp
+static void RoundPath(GraphicsPath& p, const Rect& r, int rad) {
+    int d = rad * 2;
+    p.AddArc(r.X, r.Y, d, d, 180, 90);
+    p.AddArc(r.GetRight()-d, r.Y, d, d, 270, 90);
+    p.AddArc(r.GetRight()-d, r.GetBottom()-d, d, d, 0, 90);
+    p.AddArc(r.X, r.GetBottom()-d, d, d, 90, 90);
+    p.CloseFigure();
+}
+static void FillRound(Graphics& g, const Color& c, const Rect& r, int rad) {
+    GraphicsPath p; RoundPath(p, r, rad); SolidBrush b(c); g.FillPath(&b, &p);
+}
+static void StrokeRound(Graphics& g, const Color& c, const Rect& r, int rad, REAL w=1.0f) {
+    GraphicsPath p; RoundPath(p, r, rad); Pen pen(c, w); g.DrawPath(&pen, &p);
+}
+static void DrawTextC(Graphics& g, const wchar_t* s, const Font& f, const Color& c,
+                      const RectF& box, StringAlignment h, StringAlignment v) {
+    StringFormat sf; sf.SetAlignment(h); sf.SetLineAlignment(v);
+    sf.SetTrimming(StringTrimmingEllipsisCharacter);
+    SolidBrush b(c); g.DrawString(s, -1, &f, box, &sf, &b);
+}
+```
+
+## 2.6 Reliable hover (button subclass)
+
+Owner-draw buttons don't reliably get hover. This subclass tracks it and stores a 0/1 hover flag in the window's user data, repainting on enter/leave.
+
+```cpp
+LRESULT CALLBACK BtnProc(HWND h, UINT m, WPARAM w, LPARAM l, UINT_PTR, DWORD_PTR) {
+    if (m == WM_MOUSEMOVE) {
+        if (!GetWindowLongPtr(h, GWLP_USERDATA)) {
+            SetWindowLongPtr(h, GWLP_USERDATA, 1);
+            TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 };
+            TrackMouseEvent(&t);
+            InvalidateRect(h, nullptr, FALSE);
+        }
+    } else if (m == WM_MOUSELEAVE) {
+        SetWindowLongPtr(h, GWLP_USERDATA, 0);
+        InvalidateRect(h, nullptr, FALSE);
+    }
+    return DefSubclassProc(h, m, w, l);   // needs <commctrl.h> (already included)
+}
+// after creating each owner-draw button:
+//   SetWindowSubclass(hBtn, BtnProc, 1, 0);
+```
+
+(Add `#pragma comment(lib, "comctl32.lib")` is already present; `SetWindowSubclass` lives in `<commctrl.h>`.)
+
+## 2.7 The record button (hero, pulsing)
+
+`ID_BTN_RECORD` is an owner-draw button drawn as a rounded pill: accent "● Record" when idle, danger "■ Stop" with a soft brightness pulse while recording. The dot/square are drawn shapes (no icon-font dependency).
+
+```cpp
+void DrawRecordButton(LPDRAWITEMSTRUCT d) {
+    Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
+    Rect rc(d->rcItem.left, d->rcItem.top,
+            d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
+    // paint the parent canvas in the corners first
+    SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
+
+    bool rec     = g_tx.is_recording();
+    bool hover   = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
+    bool pressed = (d->itemState & ODS_SELECTED) != 0;
+
+    Color fill = rec ? C_DANGER : (hover ? C_ACCENTHI : C_ACCENT);
+    if (rec) {                                   // gentle pulse
+        double ph = (GetTickCount() % 1400) / 1400.0;
+        int add = (int)(18 * (0.5 + 0.5 * sin(ph * 6.2831853)));
+        fill = Color(255, min(255,0xFF), min(255,0x5C+add), min(255,0x5C+add));
+    }
+    if (pressed) fill = Color(255, GetRValue(0)+ (BYTE)(fill.GetR()*0.85),
+                                   (BYTE)(fill.GetG()*0.85), (BYTE)(fill.GetB()*0.85));
+
+    Rect pill = rc; pill.Inflate(-1,-1);
+    FillRound(g, fill, pill, pill.Height/2);     // full pill
+
+    // icon: filled circle (idle) or rounded square (recording)
+    int cx = pill.X + 22, cy = pill.Y + pill.Height/2;
+    SolidBrush white(Color(255,255,255,255));
+    if (rec) { Rect sq(cx-7, cy-7, 14, 14); FillRound(g, Color(255,255,255,255), sq, 3); }
+    else     { g.FillEllipse(&white, cx-7, cy-7, 14, 14); }
+
+    Font f(d->hDC, g_fUISemi);
+    RectF tb((REAL)(pill.X+36), (REAL)pill.Y, (REAL)(pill.Width-44), (REAL)pill.Height);
+    DrawTextC(g, rec ? L"Stop" : L"Record", f, Color(255,255,255,255),
+              tb, StringAlignmentNear, StringAlignmentCenter);
+}
+```
+
+While recording, repaint it each tick so the pulse animates — in `WM_TIMER`:
+
+```cpp
+if (g_tx.is_recording()) InvalidateRect(GetDlgItem(hWnd, ID_BTN_RECORD), nullptr, FALSE);
+```
+
+## 2.8 Flat buttons (Copy / Paste / Pin)
+
+Generic owner-draw for secondary buttons — a flat surface chip that lifts on hover, accent text. Pin shows a filled dot when active.
+
+```cpp
+void DrawFlatButton(LPDRAWITEMSTRUCT d, const wchar_t* label, bool active=false) {
+    Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
+    Rect rc(d->rcItem.left, d->rcItem.top,
+            d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
+    SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
+
+    bool hover   = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
+    bool pressed = (d->itemState & ODS_SELECTED) != 0;
+    Rect chip = rc; chip.Inflate(-1,-1);
+    FillRound(g, pressed ? C_SURFACE : (hover ? C_SURFACEHI : C_SURFACE), chip, 10);
+    StrokeRound(g, C_BORDER, chip, 10, 1.0f);
+
+    Font f(d->hDC, g_fUI);
+    Color tc = active ? C_ACCENT : C_TEXT;
+    RectF tb((REAL)chip.X, (REAL)chip.Y, (REAL)chip.Width, (REAL)chip.Height);
+    DrawTextC(g, label, f, tc, tb, StringAlignmentCenter, StringAlignmentCenter);
+}
+```
+
+In `WM_DRAWITEM`, route each control:
+
+```cpp
+case WM_DRAWITEM: {
+    LPDRAWITEMSTRUCT d = (LPDRAWITEMSTRUCT)lParam;
+    switch (d->CtlID) {
+        case ID_BTN_RECORD: DrawRecordButton(d); return TRUE;
+        case ID_BTN_COPY:   DrawFlatButton(d, L"Copy");  return TRUE;
+        case ID_BTN_PASTE:  DrawFlatButton(d, L"Paste"); return TRUE;
+        case ID_BTN_PIN:    DrawFlatButton(d, g_pinned ? L"Pinned" : L"Pin", g_pinned); return TRUE;
+        case ID_COMBO_AUDIO:
+        case ID_COMBO_MODEL: DrawCombo(d); return TRUE;   // §2.9
+    }
+    break;
+}
+```
+
+Make Copy/Paste/Pin owner-draw too (add `BS_OWNERDRAW`) and subclass them for hover:
+
+```cpp
+// in InitializeUI, create with BS_OWNERDRAW, then:
+SetWindowSubclass(hCopy,  BtnProc, 1, 0);
+SetWindowSubclass(hPaste, BtnProc, 1, 0);
+SetWindowSubclass(hPin,   BtnProc, 1, 0);
+SetWindowSubclass(GetDlgItem(hwnd, ID_BTN_RECORD), BtnProc, 1, 0);
+```
+
+## 2.9 Mic & model selectors (dark combos)
+
+Make both combos `CBS_DROPDOWNLIST | CBS_OWNERDRAWFIXED | CBS_HASSTRINGS`. Set row height once, then draw field + list items dark with a drawn chevron.
+
+```cpp
+case WM_MEASUREITEM: {
+    auto* mi = (LPMEASUREITEMSTRUCT)lParam;
+    if (mi->CtlType == ODT_COMBOBOX) mi->itemHeight = 26;
+    return TRUE;
+}
+```
+
+```cpp
+void DrawCombo(LPDRAWITEMSTRUCT d) {
+    Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
+    Rect rc(d->rcItem.left, d->rcItem.top,
+            d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
+    bool inField = (d->itemState & ODS_COMBOBOXEDIT) != 0;  // the closed field
+    bool sel     = (d->itemState & ODS_SELECTED) != 0;
+
+    SolidBrush bg(inField ? C_SURFACE : (sel ? C_SURFACEHI : C_SURFACE));
+    g.FillRectangle(&bg, rc);
+    if (inField) { Rect b=rc; b.Inflate(-1,-1); StrokeRound(g, C_BORDER, b, 9); }
+
+    wchar_t txt[256] = L"";
+    if ((int)d->itemID >= 0)
+        SendMessageW(d->hwndItem, CB_GETLBTEXT, d->itemID, (LPARAM)txt);
+
+    Font f(d->hDC, g_fUI);
+    RectF tb((REAL)rc.X+10, (REAL)rc.Y, (REAL)(rc.Width-28), (REAL)rc.Height);
+    DrawTextC(g, txt, f, C_TEXT, tb, StringAlignmentNear, StringAlignmentCenter);
+
+    if (inField) {   // chevron
+        int cx = rc.X + rc.Width - 16, cy = rc.Y + rc.Height/2;
+        Pen pen(C_TEXTDIM, 1.6f);
+        g.DrawLine(&pen, cx-4, cy-2, cx, cy+2);
+        g.DrawLine(&pen, cx,   cy+2, cx+4, cy-2);
+    }
+}
+```
+
+Keep your existing `WM_CTLCOLORLISTBOX` returning the surface brush so the dropdown list background is dark too.
+
+> Minimal-by-default option: hide both combos and reveal them only when a small "settings" toggle in the header is clicked — so the resting UI is just record + transcript. Skip if you'd rather keep them always visible.
+
+## 2.10 Transcript field (flat panel + placeholder)
+
+Keep the native multiline EDIT but make it flat: no client edge, surface background, light text, inner padding. Draw a rounded surface panel behind it in `WM_PAINT`; inset the EDIT a few px inside that panel so the rounded corners show.
+
+Create it without `WS_EX_CLIENTEDGE` (flat) and give it inner margins:
+
+```cpp
+HWND hEdit = CreateWindowExW(0, L"EDIT", L"",
+    WS_CHILD | WS_VISIBLE | WS_VSCROLL | ES_MULTILINE | ES_AUTOVSCROLL | ES_READONLY,
+    0,0,0,0, hwnd, (HMENU)ID_EDIT_TEXT, hInst, nullptr);
+SendMessage(hEdit, WM_SETFONT, (WPARAM)g_fText, TRUE);
+SendMessage(hEdit, EM_SETMARGINS, EC_LEFTMARGIN|EC_RIGHTMARGIN, MAKELONG(10,10));
+```
+
+Theme it (you already have `WM_CTLCOLOREDIT` — point it at the surface):
+
+```cpp
+case WM_CTLCOLOREDIT: {
+    HDC hdc=(HDC)wParam; SetTextColor(hdc, CR_TEXT); SetBkColor(hdc, CR_SURFACE);
+    return (LRESULT)g_brSurface;
+}
+```
+
+Placeholder via a STATIC shown only when empty (`ID_STATIC_PLACEHOLDER`), created after the edit so it sits on top:
+
+```cpp
+CreateWindowW(L"STATIC", L"Your transcription will appear here…",
+    WS_CHILD | WS_VISIBLE | SS_LEFT, 0,0,0,0, hwnd,
+    (HMENU)ID_STATIC_PLACEHOLDER, hInst, nullptr);
+// font g_fText; color via WM_CTLCOLORSTATIC -> C_TEXTDIM on C_SURFACE
+// toggle it:
+void UpdatePlaceholder(HWND hwnd) {
+    bool empty = GetWindowTextLengthW(GetDlgItem(hwnd, ID_EDIT_TEXT)) == 0;
+    ShowWindow(GetDlgItem(hwnd, ID_STATIC_PLACEHOLDER), empty ? SW_SHOW : SW_HIDE);
+}
+// call after setting/clearing transcript text (WM_APP_RESULT, Clear).
+```
+
+## 2.11 Custom VU meter
+
+Drop the `PROGRESS_CLASS` control; draw a row of rounded segments in `WM_PAINT` that light up green with the input level. Store its rect in a global from `LayoutControls`.
+
+```cpp
+RECT g_vuRect = {0,0,0,0};
+float g_energy = 0.0f;   // updated in WM_TIMER from g_tx.get_audio_energy()
+
+void DrawVU(Graphics& g, const RECT& r, float level) {
+    const int N = 14, gap = 3;
+    int w = (r.right-r.left); int segW = (w - gap*(N-1)) / N;
+    int h = r.bottom - r.top;
+    int lit = (int)(level * N + 0.5f);
+    for (int i = 0; i < N; ++i) {
+        int x = r.left + i*(segW+gap);
+        Rect seg(x, r.top, segW, h);
+        Color c = (i < lit) ? C_GOOD : C_SURFACEHI;
+        FillRound(g, c, seg, 2);
+    }
+}
+```
+
+In `WM_TIMER`, refresh the level and invalidate just the VU rect:
+
+```cpp
+g_energy = g_tx.is_recording() ? g_tx.get_audio_energy() : 0.0f;
+InvalidateRect(hWnd, &g_vuRect, FALSE);
+```
+
+## 2.12 Responsive layout (final)
+
+This is the single source of truth for geometry — referenced by §1.3.
+
+```cpp
+void LayoutControls(HWND h, int W, int H) {
+    const int M = 16, row = 36, gap = 10;
+    int x = M, y = M, innerW = W - 2*M;
+
+    // Row 1: Record (left, prominent) + Pin (right)
+    int pinW = 78, recW = innerW - pinW - gap;
+    MoveWindow(GetDlgItem(h, ID_BTN_RECORD), x, y, recW, row, TRUE);
+    MoveWindow(GetDlgItem(h, ID_BTN_PIN), x + recW + gap, y, pinW, row, TRUE);
+
+    // Row 2: VU meter (custom-painted) — reserve a slim strip
+    y += row + gap;
+    g_vuRect = { x, y, x + innerW, y + 8 };
+
+    // Row 3: status line
+    y += 8 + gap;
+    MoveWindow(GetDlgItem(h, ID_STATIC_STATUS), x, y, innerW, 18, TRUE);
+
+    // Row 4: transcript panel (hero) — fills the middle
+    y += 18 + gap;
+    int bottom = (row + gap) * 2;                 // selectors row + copy/paste row
+    int panelTop = y;
+    int textH = H - y - M - bottom;
+    if (textH < 70) textH = 70;
+    // EDIT inset 10px inside the rounded panel drawn in WM_PAINT
+    MoveWindow(GetDlgItem(h, ID_EDIT_TEXT), x+10, panelTop+10, innerW-20, textH-20, TRUE);
+    MoveWindow(GetDlgItem(h, ID_STATIC_PLACEHOLDER), x+16, panelTop+16, innerW-32, 22, TRUE);
+
+    // Row 5: mic + model selectors
+    y += textH + gap;
+    int halfW = (innerW - gap) / 2;
+    MoveWindow(GetDlgItem(h, ID_COMBO_AUDIO), x, y, halfW, row, TRUE);
+    MoveWindow(GetDlgItem(h, ID_COMBO_MODEL), x + halfW + gap, y, halfW, row, TRUE);
+
+    // Row 6: Copy + Paste
+    y += row + gap;
+    MoveWindow(GetDlgItem(h, ID_BTN_COPY),  x, y, halfW, row, TRUE);
+    MoveWindow(GetDlgItem(h, ID_BTN_PASTE), x + halfW + gap, y, halfW, row, TRUE);
+
+    // remember the transcript panel rect for WM_PAINT
+    extern RECT g_panelRect; g_panelRect = { x, panelTop, x+innerW, panelTop+textH };
+    InvalidateRect(h, nullptr, FALSE);
+}
+```
+
+## 2.13 Putting it together (paint + flicker-free)
+
+Flicker-free background and the painted bits (canvas, transcript panel frame, VU):
+
+```cpp
+RECT g_panelRect = {0,0,0,0};
+
+case WM_ERASEBKGND: return 1;     // we paint everything in WM_PAINT
+
+case WM_PAINT: {
+    PAINTSTRUCT ps; HDC hdc = BeginPaint(hWnd, &ps);
+    RECT rc; GetClientRect(hWnd, &rc);
+    // double buffer
+    HDC mem = CreateCompatibleDC(hdc);
+    HBITMAP bmp = CreateCompatibleBitmap(hdc, rc.right, rc.bottom);
+    HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
+    {
+        Graphics g(mem); g.SetSmoothingMode(SmoothingModeAntiAlias);
+        SolidBrush bg(C_BG); g.FillRectangle(&bg, 0,0,rc.right,rc.bottom);
+        // transcript panel
+        Rect panel(g_panelRect.left, g_panelRect.top,
+                   g_panelRect.right-g_panelRect.left, g_panelRect.bottom-g_panelRect.top);
+        FillRound(g, C_SURFACE, panel, 14);
+        StrokeRound(g, C_BORDER, panel, 14, 1.0f);
+        // VU
+        DrawVU(g, g_vuRect, g_energy);
+    }
+    BitBlt(hdc, 0,0, rc.right, rc.bottom, mem, 0,0, SRCCOPY);
+    SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
+    EndPaint(hWnd, &ps);
+    return 0;
+}
+```
+
+> Child controls (EDIT, combos, buttons) paint themselves on top of this — they're opaque where they sit, so the panel fill shows only in its 10px inset border. Because `WM_ERASEBKGND` returns 1 and we double-buffer, there's no flicker on resize.
+
+`WM_CTLCOLORSTATIC` should return the bg brush for the status line and the surface brush for the placeholder:
+
+```cpp
+case WM_CTLCOLORSTATIC: {
+    HDC hdc=(HDC)wParam;
+    if (GetDlgCtrlID((HWND)lParam) == ID_STATIC_PLACEHOLDER) {
+        SetTextColor(hdc, RGB(0x9A,0xA0,0xAB)); SetBkColor(hdc, CR_SURFACE);
+        return (LRESULT)g_brSurface;
+    }
+    SetTextColor(hdc, CR_TEXT); SetBkColor(hdc, CR_BG);
+    return (LRESULT)g_brBg;
+}
+```
+
+Define `#define ID_STATIC_PLACEHOLDER 1014` with your other IDs, and remember to call `UpdatePlaceholder(hWnd)` after `WM_APP_RESULT` sets text and after Clear.
+
+---
+
+# Part 3 — Build & checklist
+
+- **Link GDI+:** the `#pragma comment(lib,"gdiplus.lib")` covers MSVC; no CMake change needed. (`comctl32` for `SetWindowSubclass` is already linked.)
+- **Build:** `cmake --build build --config Release` as before.
+- **Verify, in order:**
+  - [ ] Launch → window has rounded corners + dark title bar, no gray 3-D buttons.
+  - [ ] Record pill fills accent; hover lightens it; recording turns it red and it pulses; VU segments track your voice.
+  - [ ] Stop → transcript appears in the flat panel; placeholder hides; "Copied/Pasted".
+  - [ ] **No crash on Stop.** If you see "Model not found", fix the path (§1.2).
+  - [ ] Drag the window edges → it resizes and the transcript grows; min size respected.
+  - [ ] Switch model in the dropdown → status shows "Loading…" then "Ready • 2 threads"; next transcription uses it.
+- **OS notes:** caption color / rounded corners need Windows 11 (build 22000+). On Windows 10 you still get dark mode (immersive) but square corners and the default caption color — everything else is identical since it's all custom-drawn.
+
+---
+
+# Appendix — Option B: fully borderless custom title bar (advanced)
+
+If you want **zero** system chrome (no native title bar at all), use the "extend client over the caption" technique instead of Option A's recoloring. Keep `WS_OVERLAPPEDWINDOW` (so resize/snap/shadow/rounding stay native) and reclaim the caption area:
+
+```cpp
+case WM_NCCALCSIZE:
+    if (wParam) {
+        NCCALCSIZE_PARAMS* p = (NCCALCSIZE_PARAMS*)lParam;
+        int top = p->rgrc[0].top;
+        LRESULT r = DefWindowProc(hWnd, message, wParam, lParam); // default frame calc
+        p->rgrc[0].top = top;     // give the caption height back to the client area
+        return r;
+    }
+    break;
+
+case WM_NCHITTEST: {
+    LRESULT ht = DefWindowProc(hWnd, message, wParam, lParam);    // handles resize edges
+    if (ht != HTCLIENT) return ht;
+    POINT pt{ GET_X_LPARAM(lParam), GET_Y_LPARAM(lParam) };
+    ScreenToClient(hWnd, &pt);
+    const int HEADER = 40;
+    if (pt.y < HEADER) {
+        // exclude your custom min/close/pin hit-rects here, else:
+        return HTCAPTION;       // drag region
+    }
+    return HTCLIENT;
+}
+```
+
+Then draw your own header (title text on the left, custom min/close buttons on the right) in `WM_PAINT`, and shift `LayoutControls`' starting `y` down by `HEADER`. This is more finicky across DPI/OS — only take it on if Option A's dark caption isn't minimal enough for you.
+
+
+

cmake/DefaultTargetOptions.cmake

@@ -0,0 +1,16 @@
+# Set the default compile features and properties for a target.
+
+if (NOT TARGET)
+    message(FATAL_ERROR "TARGET not set before including DefaultTargetOptions")
+endif()
+
+target_compile_features(${TARGET}
+    PRIVATE
+        cxx_std_11
+    )
+
+set_target_properties(${TARGET}
+    PROPERTIES
+        EXPORT_COMPILE_COMMANDS ON
+        RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+)

cmake/FindFFmpeg.cmake

@@ -0,0 +1,163 @@
+# From
+# https://github.com/snikulov/cmake-modules/blob/master/FindFFmpeg.cmake
+#
+# vim: ts=2 sw=2
+# - Try to find the required ffmpeg components(default: AVFORMAT, AVUTIL, AVCODEC)
+#
+# Once done this will define
+#  FFMPEG_FOUND         - System has the all required components.
+#  FFMPEG_INCLUDE_DIRS  - Include directory necessary for using the required components headers.
+#  FFMPEG_LIBRARIES     - Link these to use the required ffmpeg components.
+#  FFMPEG_DEFINITIONS   - Compiler switches required for using the required ffmpeg components.
+#
+# For each of the components it will additionally set.
+#   - AVCODEC
+#   - AVDEVICE
+#   - AVFORMAT
+#   - AVFILTER
+#   - AVUTIL
+#   - POSTPROC
+#   - SWSCALE
+# the following variables will be defined
+#  <component>_FOUND        - System has <component>
+#  <component>_INCLUDE_DIRS - Include directory necessary for using the <component> headers
+#  <component>_LIBRARIES    - Link these to use <component>
+#  <component>_DEFINITIONS  - Compiler switches required for using <component>
+#  <component>_VERSION      - The components version
+#
+# Copyright (c) 2006, Matthias Kretz, <kretz@kde.org>
+# Copyright (c) 2008, Alexander Neundorf, <neundorf@kde.org>
+# Copyright (c) 2011, Michael Jansen, <kde@michael-jansen.biz>
+#
+# Redistribution and use is allowed according to the terms of the BSD license.
+# For details see the accompanying COPYING-CMAKE-SCRIPTS file.
+
+include(FindPackageHandleStandardArgs)
+
+# The default components were taken from a survey over other FindFFMPEG.cmake files
+if (NOT FFmpeg_FIND_COMPONENTS)
+  set(FFmpeg_FIND_COMPONENTS AVFORMAT AVCODEC AVUTIL SWRESAMPLE)
+endif()
+
+#
+### Macro: set_component_found
+#
+# Marks the given component as found if both *_LIBRARIES AND *_INCLUDE_DIRS is present.
+#
+macro(set_component_found _component )
+  if (${_component}_LIBRARIES AND ${_component}_INCLUDE_DIRS)
+    message(DEBUG "  - ${_component} found.")
+    set(${_component}_FOUND TRUE)
+  else ()
+  message(DEBUG "  - ${_component} not found.")
+  endif ()
+endmacro()
+
+#
+### Macro: find_component
+#
+# Checks for the given component by invoking pkgconfig and then looking up the libraries and
+# include directories.
+#
+macro(find_component _component _pkgconfig _library _header)
+
+  if (NOT WIN32)
+     # use pkg-config to get the directories and then use these values
+     # in the FIND_PATH() and FIND_LIBRARY() calls
+     find_package(PkgConfig)
+     if (PKG_CONFIG_FOUND)
+       pkg_check_modules(PC_${_component} ${_pkgconfig})
+       message(STATUS "Pkgconfig found: ${PC_${_component}_INCLUDEDIR}")
+       message(STATUS "Pkgconfig found: ${PC_${_component}_INCLUDE_DIRS}")
+       message(STATUS "${PC_${_component}_CFLAGS}")
+     endif ()
+  endif (NOT WIN32)
+
+
+  find_path(${_component}_INCLUDE_DIRS ${_header}
+    HINTS
+      ${PC_${_component}_INCLUDEDIR}
+      ${PC_${_component}_INCLUDE_DIRS}
+    PATH_SUFFIXES
+      ffmpeg
+  )
+
+  # CMake's default is to search first for shared libraries and then for static libraries.
+  # Todo later: add option to prefer static libs over dynamic:
+  find_library(${_component}_LIBRARIES NAMES ${_library} lib${_library}.a
+      HINTS
+      ${PC_${_component}_LIBDIR}
+      ${PC_${_component}_LIBRARY_DIRS}
+  )
+
+  set(${_component}_DEFINITIONS  ${PC_${_component}_CFLAGS_OTHER} CACHE STRING "The ${_component} CFLAGS.")
+  set(${_component}_VERSION      ${PC_${_component}_VERSION}      CACHE STRING "The ${_component} version number.")
+
+  set_component_found(${_component})
+
+  mark_as_advanced(
+    ${_component}_INCLUDE_DIRS
+    ${_component}_LIBRARIES
+    ${_component}_DEFINITIONS
+    ${_component}_VERSION)
+
+endmacro()
+
+
+# Check for cached results. If there are skip the costly part.
+if (NOT FFMPEG_LIBRARIES)
+
+  # Check for all possible component.
+  find_component(AVCODEC    libavcodec    avcodec  libavcodec/avcodec.h)
+  find_component(AVFORMAT   libavformat   avformat libavformat/avformat.h)
+  find_component(AVDEVICE   libavdevice   avdevice libavdevice/avdevice.h)
+  #find_component(AVRESAMPLE libavresample avresample libavresample/avresample.h) # old name for swresample
+  find_component(AVUTIL     libavutil     avutil   libavutil/avutil.h)
+  find_component(AVFILTER   libavfilter   avfilter libavfilter/avfilter.h)
+  find_component(SWSCALE    libswscale    swscale  libswscale/swscale.h)
+  find_component(POSTPROC   libpostproc   postproc libpostproc/postprocess.h)
+  find_component(SWRESAMPLE libswresample swresample libswresample/swresample.h)
+
+  # Check if the required components were found and add their stuff to the FFMPEG_* vars.
+  foreach (_component ${FFmpeg_FIND_COMPONENTS})
+    if (${_component}_FOUND)
+      # message(STATUS "Required component ${_component} present.")
+      set(FFMPEG_LIBRARIES   ${FFMPEG_LIBRARIES}   ${${_component}_LIBRARIES})
+      set(FFMPEG_DEFINITIONS ${FFMPEG_DEFINITIONS} ${${_component}_DEFINITIONS})
+      list(APPEND FFMPEG_INCLUDE_DIRS ${${_component}_INCLUDE_DIRS})
+    else ()
+      # message(STATUS "Required component ${_component} missing.")
+    endif ()
+  endforeach ()
+
+  # Build the include path with duplicates removed.
+  if (FFMPEG_INCLUDE_DIRS)
+    list(REMOVE_DUPLICATES FFMPEG_INCLUDE_DIRS)
+  endif ()
+
+  # cache the vars.
+  set(FFMPEG_INCLUDE_DIRS ${FFMPEG_INCLUDE_DIRS} CACHE STRING "The FFmpeg include directories." FORCE)
+  set(FFMPEG_LIBRARIES    ${FFMPEG_LIBRARIES}    CACHE STRING "The FFmpeg libraries." FORCE)
+  set(FFMPEG_DEFINITIONS  ${FFMPEG_DEFINITIONS}  CACHE STRING "The FFmpeg cflags." FORCE)
+
+  mark_as_advanced(FFMPEG_INCLUDE_DIRS
+                   FFMPEG_LIBRARIES
+                   FFMPEG_DEFINITIONS)
+
+endif ()
+
+# Now set the noncached _FOUND vars for the components.
+# whisper.cpp does not need SWSCALE
+foreach (_component AVCODEC AVDEVICE AVFORMAT AVRESAMPLE AVUTIL POSTPROCESS)
+  set_component_found(${_component})
+endforeach ()
+
+# Compile the list of required vars
+set(_FFmpeg_REQUIRED_VARS FFMPEG_LIBRARIES FFMPEG_INCLUDE_DIRS)
+foreach (_component ${FFmpeg_FIND_COMPONENTS})
+  list(APPEND _FFmpeg_REQUIRED_VARS ${_component}_LIBRARIES ${_component}_INCLUDE_DIRS)
+endforeach ()
+
+# Give a nice error message if some of the required vars are missing.
+find_package_handle_standard_args(FFmpeg DEFAULT_MSG ${_FFmpeg_REQUIRED_VARS})
+

cmake/build-info.cmake

@@ -0,0 +1,60 @@
+set(BUILD_NUMBER 0)
+set(BUILD_COMMIT "unknown")
+set(BUILD_COMPILER "unknown")
+set(BUILD_TARGET "unknown")
+
+# Look for git
+find_package(Git)
+if(NOT Git_FOUND)
+    find_program(GIT_EXECUTABLE NAMES git git.exe)
+    if(GIT_EXECUTABLE)
+        set(Git_FOUND TRUE)
+        message(STATUS "Found Git: ${GIT_EXECUTABLE}")
+    else()
+        message(WARNING "Git not found. Build info will not be accurate.")
+    endif()
+endif()
+
+# Get the commit count and hash
+if(Git_FOUND)
+    execute_process(
+        COMMAND ${GIT_EXECUTABLE} rev-parse --short HEAD
+        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+        OUTPUT_VARIABLE HEAD
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+        RESULT_VARIABLE RES
+    )
+    if (RES EQUAL 0)
+        set(BUILD_COMMIT ${HEAD})
+    endif()
+    execute_process(
+        COMMAND ${GIT_EXECUTABLE} rev-list --count HEAD
+        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+        OUTPUT_VARIABLE COUNT
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+        RESULT_VARIABLE RES
+    )
+    if (RES EQUAL 0)
+        set(BUILD_NUMBER ${COUNT})
+    endif()
+endif()
+
+if(MSVC)
+    set(BUILD_COMPILER "${CMAKE_C_COMPILER_ID} ${CMAKE_C_COMPILER_VERSION}")
+    set(BUILD_TARGET ${CMAKE_VS_PLATFORM_NAME})
+    add_compile_options("$<$<COMPILE_LANGUAGE:C>:/utf-8>")
+    add_compile_options("$<$<COMPILE_LANGUAGE:CXX>:/utf-8>")
+else()
+    execute_process(
+        COMMAND sh -c "$@ --version | head -1" _ ${CMAKE_C_COMPILER}
+        OUTPUT_VARIABLE OUT
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+    )
+    set(BUILD_COMPILER ${OUT})
+    execute_process(
+        COMMAND ${CMAKE_C_COMPILER} -dumpmachine
+        OUTPUT_VARIABLE OUT
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+    )
+    set(BUILD_TARGET ${OUT})
+endif()

cmake/git-vars.cmake

@@ -0,0 +1,22 @@
+find_package(Git)
+
+# the commit's SHA1
+execute_process(COMMAND
+    "${GIT_EXECUTABLE}" describe --match=NeVeRmAtCh --always --abbrev=8
+    WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
+    OUTPUT_VARIABLE GIT_SHA1
+    ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE)
+
+# the date of the commit
+execute_process(COMMAND
+    "${GIT_EXECUTABLE}" log -1 --format=%ad --date=local
+    WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
+    OUTPUT_VARIABLE GIT_DATE
+    ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE)
+
+# the subject of the commit
+execute_process(COMMAND
+    "${GIT_EXECUTABLE}" log -1 --format=%s
+    WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
+    OUTPUT_VARIABLE GIT_COMMIT_SUBJECT
+    ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE)

cmake/whisper-config.cmake.in

@@ -0,0 +1,65 @@
+set(WHISPER_VERSION      @WHISPER_INSTALL_VERSION@)
+set(WHISPER_BUILD_COMMIT @WHISPER_BUILD_COMMIT@)
+set(WHISPER_BUILD_NUMBER @WHISPER_BUILD_NUMBER@)
+set(WHISPER_SHARED_LIB   @BUILD_SHARED_LIBS@)
+
+set(GGML_BLAS       @GGML_BLAS@)
+set(GGML_CUDA       @GGML_CUDA@)
+set(GGML_METAL      @GGML_METAL@)
+set(GGML_HIPBLAS    @GGML_HIPBLAS@)
+set(GGML_ACCELERATE @GGML_ACCELERATE@)
+
+@PACKAGE_INIT@
+
+set_and_check(WHISPER_INCLUDE_DIR "@PACKAGE_WHISPER_INCLUDE_INSTALL_DIR@")
+set_and_check(WHISPER_LIB_DIR     "@PACKAGE_WHISPER_LIB_INSTALL_DIR@")
+set_and_check(WHISPER_BIN_DIR     "@PACKAGE_WHISPER_BIN_INSTALL_DIR@")
+
+# Ensure transient dependencies satisfied
+
+find_package(Threads REQUIRED)
+
+if (APPLE AND GGML_ACCELERATE)
+    find_library(ACCELERATE_FRAMEWORK Accelerate REQUIRED)
+endif()
+
+if (GGML_BLAS)
+    find_package(BLAS REQUIRED)
+endif()
+
+if (GGML_CUDA)
+    find_package(CUDAToolkit REQUIRED)
+endif()
+
+if (GGML_METAL)
+    find_library(FOUNDATION_LIBRARY Foundation REQUIRED)
+    find_library(METAL_FRAMEWORK Metal REQUIRED)
+    find_library(METALKIT_FRAMEWORK MetalKit REQUIRED)
+endif()
+
+if (GGML_HIPBLAS)
+    find_package(hip REQUIRED)
+    find_package(hipblas REQUIRED)
+    find_package(rocblas REQUIRED)
+endif()
+
+find_library(whisper_LIBRARY whisper
+    REQUIRED
+    HINTS ${WHISPER_LIB_DIR})
+
+set(_whisper_link_deps "Threads::Threads" "@WHISPER_EXTRA_LIBS@")
+set(_whisper_transient_defines "@WHISPER_TRANSIENT_DEFINES@")
+
+add_library(whisper UNKNOWN IMPORTED)
+
+set_target_properties(whisper
+    PROPERTIES
+    INTERFACE_INCLUDE_DIRECTORIES "${WHISPER_INCLUDE_DIR}"
+        INTERFACE_LINK_LIBRARIES "${_whisper_link_deps}"
+        INTERFACE_COMPILE_DEFINITIONS "${_whisper_transient_defines}"
+        IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
+        IMPORTED_LOCATION "${whisper_LIBRARY}"
+        INTERFACE_COMPILE_FEATURES cxx_std_11
+        POSITION_INDEPENDENT_CODE ON )
+
+check_required_components(whisper)

cmake/whisper.pc.in

@@ -0,0 +1,10 @@
+prefix=@CMAKE_INSTALL_PREFIX@
+exec_prefix=${prefix}
+libdir=${exec_prefix}/lib
+includedir=${prefix}/include
+
+Name: whisper
+Description: Port of OpenAI's Whisper model in C/C++
+Version: @PROJECT_VERSION@
+Libs: -L${libdir} -lggml  -lggml-base -lwhisper
+Cflags: -I${includedir}

dev-task-list.md

@@ -0,0 +1,321 @@
+# Win Dictation — Developer Task List
+
+A prioritised, junior-friendly backlog to finish polishing the app. Each task says **what**, **which files**, **steps**, **the code** (inline, or a pointer to the verbatim block in a companion doc), and **how to know it's done**.
+
+**Companion docs (full code lives here — don't retype, copy from them):**
+- `ARCHITECTURE-AND-DEVGUIDE.md` — Part B1 (chrome removal), Part B2 (progress + ETA). Verbatim code.
+- `MODERN-UI-AND-FIXES.md` — the modern UI design + GDI+ helpers.
+- `FINDINGS-FIXES-TESTS.md` — the test harness (`tests/test_core.cpp`).
+
+**Baseline (already done — do NOT redo):** batch record-then-transcribe; crash fixes (null-context guard, `g_modelOk`, `g_initializing`, `m_cfg_mtx`); append behaviour (`text_util.h`); no-hide + safe auto-paste; Clear button; `run_inference`/`transcribe_sync` refactor; modern dark UI (GDI+, rounded panel, owner-draw buttons, custom VU, dark caption); resizable window; `test-core` CMake target.
+
+**Conventions:** Effort = XS (<30 min) · S (≤2 h) · M (half-day) · L (1–2 days). Do phases in order; tasks within a phase are mostly independent unless "Depends on" says otherwise. After every task: `cmake --build build --config Release` must succeed with **zero new warnings**, and the app must still launch.
+
+**Color tokens (already in `main.cpp`, reuse — never hard-code hex elsewhere):** `C_BG #0F1115`, `C_SURFACE #181B22`, `C_SURFACEHI #20242D`, `C_BORDER #262B36`, `C_TEXT #E7E9EE`, `C_TEXTDIM #9AA0AB`, `C_ACCENT #6E8BFF`, `C_DANGER #FF5C5C`, `C_GOOD #46D39A`.
+
+---
+
+## Phase 0 — Repo hygiene (do first; clears traps)
+
+### Task 0.1 — Delete the stale `src/CMakeLists.txt`
+- **Goal:** Remove a build file that references **removed** APIs (`init()`, `is_using_gpu()`) and a `test-audio` target the real build ignores. The authoritative build is the **root** `CMakeLists.txt`.
+- **Files:** `src/CMakeLists.txt` (delete).
+- **Steps:** Confirm `build.ps1` configures from repo root (`-S $RepoRoot`). It does. Delete `src/CMakeLists.txt`.
+- **Done when:** Clean build from root still works; no other file `add_subdirectory(src)`.
+- **Effort:** XS
+
+### Task 0.2 — Remove the broken `src/test-audio.cpp` (replaced in Phase 4)
+- **Goal:** `src/test-audio.cpp` calls `m_transcriber.init(...)` / `is_using_gpu()` which no longer exist — it cannot compile. It's superseded by `tests/test_core.cpp` (Task 4.1).
+- **Files:** `src/test-audio.cpp` (delete), `src/record-test-audio.ps1` (keep — still useful for capturing WAVs), `src/TESTING.md` (mark superseded in Task 4.4).
+- **Done when:** No target references `test-audio.cpp`.
+- **Effort:** XS
+
+### Task 0.3 — Create the `tests/` folder + placeholder
+- **Goal:** The root `CMakeLists.txt` already declares `add_executable(test-core tests/test_core.cpp ...)`, but the file doesn't exist yet → configure fails if anyone builds `test-core`.
+- **Steps:** Create `tests/` and add `tests/test_core.cpp` (full content in Task 4.1). Until then, the `test-core` target can stay; just don't build it.
+- **Done when:** `tests/test_core.cpp` exists and `cmake --build build --target test-core` compiles (after Task 4.1).
+- **Effort:** XS · **Depends on:** 4.1 for real content
+
+---
+
+## Phase 1 — Visual polish & assets
+
+### Task 1.1 — App icon asset (custom)
+- **Goal:** Replace the placeholder icon with a clean, modern app icon used for the window, taskbar, and tray.
+- **Files:** `assets/icon-source.png` (new, 1024×1024), `src/icon.ico` (generated), `src/convert_icon.py` (fix paths), `src/win-dictation.rc` (already references `101 ICON "icon.ico"`).
+- **Design spec:** Flat, minimal. A single rounded **microphone** glyph, centered, on a dark charcoal rounded-square (`#15171C`). Mic filled with the indigo accent (`#6E8BFF`), subtle top-down gradient to `#5B7BFF`. No text. Must read clearly at **16×16**. Keep ~12% padding around the glyph.
+  - Ready-to-use generation prompt (AI image tool): *"Minimalist modern app icon, a single simple microphone glyph centered on a dark charcoal rounded square, microphone filled indigo #6E8BFF with a soft vertical gradient, flat design, crisp clean edges, no text, high contrast, legible at small sizes, 1024×1024."*
+  - Or design in Figma/Inkscape and export 1024×1024 PNG.
+- **Steps:**
+  1. Put the source PNG at `assets/icon-source.png`.
+  2. Fix `convert_icon.py` to use real paths and multi-size output:
+     ```python
+     from PIL import Image
+     img = Image.open("assets/icon-source.png").convert("RGBA")
+     img.save("src/icon.ico", format="ICO",
+              sizes=[(256,256),(64,64),(48,48),(32,32),(16,16)])
+     print("wrote src/icon.ico")
+     ```
+  3. Run it (`python src/convert_icon.py` from repo root). Confirm `src/icon.ico` exists.
+  4. Rebuild; the resource compiler picks up `src/icon.ico` via the `.rc`.
+- **Done when:** The new icon shows on the title bar, taskbar, Alt-Tab, and tray — sharp at all sizes.
+- **Effort:** S
+
+### Task 1.2 — Tray icon reflects recording state (optional but nice)
+- **Goal:** When recording (window may be hidden), the **tray** icon turns red so state is visible at a glance.
+- **Files:** `assets/icon-rec-source.png` (new), `src/icon-rec.ico`, `src/win-dictation.rc` (add `102 ICON "icon-rec.ico"`), `src/main.cpp`.
+- **Steps:**
+  1. Create a red variant of the icon (mic in `#FF5C5C`). Convert to `src/icon-rec.ico` (same sizes), add `102 ICON "icon-rec.ico"` to the `.rc`.
+  2. In `main.cpp`, load both icons once: `HICON g_icoIdle, g_icoRec;` via `LoadIcon(hInst, MAKEINTRESOURCE(101/102))`.
+  3. Add a helper `void SetTrayIcon(bool rec){ nid.uFlags = NIF_ICON; nid.hIcon = rec?g_icoRec:g_icoIdle; Shell_NotifyIcon(NIM_MODIFY,&nid); }`.
+  4. Call `SetTrayIcon(true)` when recording starts, `SetTrayIcon(false)` on stop/cancel/result.
+- **Done when:** Start recording, hide the window — the tray icon is red; after transcription it returns to normal.
+- **Effort:** S · **Depends on:** 1.1
+
+### Task 1.3 — Kill the button hairlines + focus rectangles (B1.1)
+- **Goal:** Remove the thin light line around *Pinned* and the left/top lines on Copy/Paste/Clear.
+- **Files:** `src/main.cpp`; link `uxtheme.lib`.
+- **Steps (full code in `ARCHITECTURE-AND-DEVGUIDE.md` §B1.1):**
+  1. Add `#include <uxtheme.h>` and `#pragma comment(lib, "uxtheme.lib")`.
+  2. After creating each owner-draw button (Record, Pin, Copy, Paste, Clear), call `SetWindowTheme(hBtn, L"", L"");` (before/after `SetWindowSubclass` is fine).
+  3. Add `WS_CLIPCHILDREN` to the main window style in `CreateWindowExW`.
+  4. After all controls are created (end of the create block), call once:
+     `SendMessageW(hMainWnd, WM_CHANGEUISTATE, MAKEWPARAM(UIS_SET, UISF_HIDEFOCUS), 0);`
+- **Done when:** No hairline around any button; tabbing between controls draws no dotted focus rect.
+- **Effort:** S
+
+### Task 1.4 — Replace comboboxes with custom dropdowns (B1.2)
+- **Goal:** Remove the native Windows dropdown button (the "second arrow") on the mic + model selectors.
+- **Files:** `src/main.cpp`.
+- **Steps (full code — `DrawSelect`, `PopupProc`, `ShowSelectPopup`, `WM_APP_SELECT` — in `ARCHITECTURE-AND-DEVGUIDE.md` §B1.2):**
+  1. Add label/selection state: `std::vector<std::wstring> g_audioItems; int g_audioSel=0;` and `g_modelItems`/`g_modelSel`. Populate them in `RefreshAudioDevices`/`RefreshModelList` (keep populating `g_modelComboPaths` in parallel).
+  2. Replace the two `COMBOBOX` creations with `BS_OWNERDRAW` buttons `ID_SEL_AUDIO`/`ID_SEL_MODEL`; subclass with `BtnProc`; `SetWindowTheme(.., L"", L"")`.
+  3. Add `#define WM_APP_SELECT (WM_USER + 5)` and `#define ID_SEL_AUDIO/ID_SEL_MODEL`.
+  4. Paste `DrawSelect`, route both IDs in `WM_DRAWITEM`. Paste `PopupProc` + `ShowSelectPopup`. On `WM_COMMAND` for the two IDs call `ShowSelectPopup(...)`. Handle `WM_APP_SELECT` to apply the choice (set `capture_id` / reload model).
+  5. Delete the now-unused `DrawCombo`, `WM_MEASUREITEM` combo branch, and `WM_CTLCOLORLISTBOX`.
+  6. Update `LayoutControls` to position `ID_SEL_AUDIO`/`ID_SEL_MODEL` where the combos were.
+- **Done when:** Each selector shows exactly one (our) chevron, opens a dark rounded popup, hover highlights rows, selecting reloads the model / switches mic, and clicking elsewhere dismisses it.
+- **Effort:** M · **Depends on:** 1.3 (shared `SetWindowTheme`)
+
+---
+
+## Phase 2 — Progress feedback & control
+
+### Task 2.1 — Transcription progress + ETA (B2)
+- **Goal:** Replace the static "Transcribing…" with a moving progress bar + live status like **"Transcribing 1:40 · 45% · ~9s left"**.
+- **Files:** `src/transcriber.h`, `src/transcriber.cpp`, `src/main.cpp`.
+- **Steps (full code in `ARCHITECTURE-AND-DEVGUIDE.md` §B2.1–§B2.3):**
+  1. `transcriber.h`: add `set_progress_callback`, `audio_seconds()`, private `m_on_progress`, `m_audio_seconds`, and the static `s_progress` trampoline.
+  2. `transcriber.cpp`: implement `s_progress`; in `run_inference` set `m_audio_seconds`, `wp.progress_callback = &Transcriber::s_progress; wp.progress_callback_user_data = this;`.
+  3. `main.cpp`: add `#define WM_APP_PROGRESS (WM_USER + 6)`, `std::atomic<int> g_progress{0};`, `DWORD g_busyStart;`. Register `set_progress_callback([](int p){ PostMessage(hMainWnd, WM_APP_PROGRESS, p, 0); })`.
+  4. Set `g_busyStart = GetTickCount(); g_progress = 0;` right before `stop_and_transcribe()`.
+  5. Handle `WM_APP_PROGRESS` (store + invalidate `g_vuRect`). Replace the `is_busy()` branch of `UpdateStatus` with the ETA formatter. Add `DrawProgress` and, in `WM_PAINT`, draw the progress bar in `g_vuRect` while busy (VU otherwise). In `WM_TIMER`, also invalidate `g_vuRect` while busy so the bar/ETA tick.
+- **Edge cases:** `progress < 3%` → show "Transcribing m:ss of audio…" (ETA not stable yet). Clamp `remain >= 0`.
+- **Done when:** A 1–2 min clip shows a filling bar + percentage + shrinking ETA and completes; short clips still feel instant.
+- **Effort:** M
+
+### Task 2.2 — Cancel a running transcription (B2.4)
+- **Goal:** Let the user abort a long/incorrect transcription instead of waiting it out.
+- **Files:** `src/transcriber.h/.cpp`, `src/main.cpp`.
+- **Steps:**
+  1. `transcriber.h`: add `void request_cancel(){ m_abort = true; }`, private `std::atomic<bool> m_abort{false};`, static `s_abort`.
+  2. `transcriber.cpp`: in `run_inference`, `m_abort = false;` at the top, and set `wp.abort_callback = &Transcriber::s_abort; wp.abort_callback_user_data = this;` (skip if your `whisper.h` lacks `abort_callback`).
+  3. `main.cpp`: at the very top of the `HK_TOGGLE` handler add `if (g_tx.is_busy()) { g_tx.request_cancel(); SetStatus(hWnd, L"Cancelling…"); break; }` — so the Record button becomes "cancel" while busy.
+  4. In `WM_APP_RESULT`, when the result is empty *and* a cancel was requested, show "Cancelled" instead of "No speech detected" (track a `g_cancelRequested` flag, reset each Stop).
+- **Done when:** Pressing the button (or hotkey) mid-transcription stops it within ~1 s and the status reads "Cancelled".
+- **Effort:** S · **Depends on:** 2.1
+
+---
+
+## Phase 3 — Robustness & persistence
+
+### Task 3.1 — Persist settings between launches
+- **Goal:** Remember mic, model, pin state, auto-paste, auto-hide, and window position. Today everything resets each launch.
+- **Files:** `src/settings.h` (new, header-only), `src/main.cpp`.
+- **Code (`src/settings.h`):**
+  ```cpp
+  #pragma once
+  #include <windows.h>
+  #include <string>
+  struct AppSettings {
+      int  captureId = 0;
+      std::wstring modelFile;                 // e.g. L"models\\ggml-tiny.en.bin" ("" = auto)
+      bool pinned = true, autoPaste = true, autoHide = false;
+      int  winX = CW_USEDEFAULT, winY = CW_USEDEFAULT, winW = 400, winH = 340;
+  };
+  inline std::wstring SettingsPath() {
+      wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
+      std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
+      return p + L"\\win-dictation.ini";
+  }
+  inline int  GetIni(const wchar_t* k, int d){ return GetPrivateProfileIntW(L"app", k, d, SettingsPath().c_str()); }
+  inline void PutIni(const wchar_t* k, int v){ wchar_t b[32]; wsprintfW(b, L"%d", v); WritePrivateProfileStringW(L"app", k, b, SettingsPath().c_str()); }
+  inline void LoadSettings(AppSettings& s){
+      s.captureId = GetIni(L"captureId", s.captureId);
+      s.pinned    = GetIni(L"pinned",    s.pinned)    != 0;
+      s.autoPaste = GetIni(L"autoPaste", s.autoPaste) != 0;
+      s.autoHide  = GetIni(L"autoHide",  s.autoHide)  != 0;
+      s.winX = GetIni(L"winX", s.winX); s.winY = GetIni(L"winY", s.winY);
+      s.winW = GetIni(L"winW", s.winW); s.winH = GetIni(L"winH", s.winH);
+      wchar_t m[MAX_PATH]; GetPrivateProfileStringW(L"app", L"modelFile", L"", m, MAX_PATH, SettingsPath().c_str());
+      s.modelFile = m;
+  }
+  inline void SaveSettings(const AppSettings& s){
+      PutIni(L"captureId", s.captureId); PutIni(L"pinned", s.pinned);
+      PutIni(L"autoPaste", s.autoPaste); PutIni(L"autoHide", s.autoHide);
+      PutIni(L"winX", s.winX); PutIni(L"winY", s.winY); PutIni(L"winW", s.winW); PutIni(L"winH", s.winH);
+      WritePrivateProfileStringW(L"app", L"modelFile", s.modelFile.c_str(), SettingsPath().c_str());
+  }
+  ```
+- **Wiring (`main.cpp`):**
+  1. Add a global `AppSettings g_set;` Call `LoadSettings(g_set);` **before** creating the window.
+  2. Apply: use `g_set.winX/Y/W/H` in `CreateWindowExW` (validate on-screen; fall back to `CW_USEDEFAULT` if off all monitors). Set `g_pinned = g_set.pinned`, `g_autoPaste = g_set.autoPaste`, `g_autoHide = g_set.autoHide`, `g_config.capture_id = g_set.captureId`. If `g_set.modelFile` is non-empty and the file exists, use it instead of `SelectOptimalModel`.
+  3. Save on change: after toggling pin/auto-paste/auto-hide, after a model/mic change, and on `WM_EXITSIZEMOVE` (window moved/resized → store rect) and `WM_DESTROY` (final save). A `void PersistNow()` that copies the live globals into `g_set` then `SaveSettings(g_set)` keeps it DRY.
+- **Done when:** Change mic/model, move/resize the window, toggle pin, quit, relaunch → everything is restored. Deleting the `.ini` restores defaults.
+- **Effort:** M
+
+### Task 3.2 — Quick toggles in the tray menu
+- **Goal:** Expose Auto-paste, Always-on-top, and Auto-hide without building a settings panel.
+- **Files:** `src/main.cpp` (`ShowContextMenu`, `WM_COMMAND`).
+- **Steps:** Add checkable items to the tray popup (`MF_STRING | (flag?MF_CHECKED:0)`) with new IDs (`ID_TRAY_AUTOPASTE`, `ID_TRAY_TOPMOST`, `ID_TRAY_AUTOHIDE`). In `WM_COMMAND`, flip the matching global, apply (for top-most call `SetWindowPos(... HWND_TOPMOST/NOTOPMOST ...)`), then `PersistNow()`.
+- **Done when:** Right-click tray → toggles show check state, take effect immediately, and survive a relaunch.
+- **Effort:** S · **Depends on:** 3.1
+
+### Task 3.3 — Bound the recording length
+- **Goal:** A forgotten recording shouldn't grow memory without limit (~1.9 MB/30 s today, uncapped).
+- **Files:** `src/transcriber.h/.cpp`, `src/main.cpp`.
+- **Steps:**
+  1. `transcriber.h`: add `float recorded_seconds() const;` returning `m_capture` size / `WHISPER_SAMPLE_RATE` under `m_capture_mtx` (or maintain an atomic sample counter incremented in `on_audio`).
+  2. `main.cpp` `WM_TIMER` (recording branch): if `g_tx.recorded_seconds() >= kMaxRecordSeconds` (e.g. 600), auto-stop by posting the same path as a manual Stop, and set status "Max length reached — transcribing".
+- **Done when:** Recording auto-stops at the cap and transcribes what was captured; normal short clips unaffected.
+- **Effort:** S
+
+### Task 3.4 — Surface hotkey-registration failures + make hotkeys configurable
+- **Goal:** Today `RegisterHotKey` return values are ignored — if another app owns `Ctrl+Shift+Space`, the hotkey silently dies. Also allow remapping.
+- **Files:** `src/main.cpp`, `src/settings.h`.
+- **Steps:**
+  1. Capture the return of both `RegisterHotKey` calls. If either fails, show a non-blocking status ("Hotkey in use — set another in win-dictation.ini") and still allow the on-screen Record button to work.
+  2. Read modifiers + key from the INI (`hkMods`, `hkVk`, defaulting to `MOD_CONTROL|MOD_SHIFT` + `VK_SPACE` / `'H'`); register those. (Full remap UI is a Phase 5 stretch — INI is enough now.)
+- **Done when:** With a conflicting global hotkey registered by another app, the app launches, warns, and the button still records; editing the INI changes the hotkey.
+- **Effort:** S · **Depends on:** 3.1
+
+### Task 3.5 — Lightweight logging
+- **Goal:** One small log file so field issues are diagnosable without a debugger.
+- **Files:** `src/logging.h` (new), `src/main.cpp`, `src/transcriber.cpp`.
+- **Code (`src/logging.h`):**
+  ```cpp
+  #pragma once
+  #include <windows.h>
+  #include <cstdio>
+  #include <string>
+  inline void LogLine(const char* msg) {
+      wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
+      std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/")) + L"\\win-dictation.log";
+      FILE* f = _wfopen(p.c_str(), L"a"); if (!f) return;
+      SYSTEMTIME t; GetLocalTime(&t);
+      fprintf(f, "%04d-%02d-%02d %02d:%02d:%02d  %s\n", t.wYear,t.wMonth,t.wDay,t.wHour,t.wMinute,t.wSecond, msg);
+      fclose(f);
+  }
+  ```
+- **Log at minimum:** startup (model path, threads, GPU on/off), model load success/failure, mic-open failure, and each transcription (audio seconds, elapsed ms, output char count). Keep messages one line, ASCII.
+- **Optional:** if the file exceeds ~1 MB at startup, rename to `.log.1` (simple 1-file rotation).
+- **Done when:** `win-dictation.log` appears next to the exe and records a startup line + one line per transcription.
+- **Effort:** S
+
+---
+
+## Phase 4 — Tests, docs & packaging
+
+### Task 4.1 — Create `tests/test_core.cpp`
+- **Goal:** Headless regression tests that catch the bugs we already fixed and lock in progress reporting.
+- **Files:** `tests/test_core.cpp` (new — **full content in `FINDINGS-FIXES-TESTS.md` §4.2**).
+- **Steps:**
+  1. Copy the test program from `FINDINGS-FIXES-TESTS.md` §4.2 into `tests/test_core.cpp`.
+  2. Add the progress assertion from `ARCHITECTURE-AND-DEVGUIDE.md` §B3 (max progress ≥95, monotonic).
+  3. Build + run:
+     ```powershell
+     cmake --build build --config Release --target test-core
+     cd build\bin\Release
+     copy ..\..\..\deps\SDL2-2.28.5\lib\x64\SDL2.dll .   # if missing
+     .\test-core.exe models\ggml-tiny.en.bin ..\..\..\samples\jfk.wav
+     ```
+- **Tests covered:** append rule; bad model path → no crash + empty; real clip → contains "country"; short audio → no crash; progress reaches ~100% and is non-decreasing.
+- **Done when:** `test-core.exe` prints "ALL TESTS PASSED" and exits 0.
+- **Effort:** S · **Depends on:** 2.1 (for the progress test)
+
+### Task 4.2 — (Optional) CI workflow
+- **Goal:** Run the build + `test-core` on every push.
+- **Files:** `.github/workflows/build.yml` (new).
+- **Steps:** Windows runner → configure CMake (CPU-only) → build `win-dictation` + `test-core` → download `ggml-tiny.en.bin` → run `test-core` with `jfk.wav`. Fail the job on non-zero exit.
+- **Done when:** A pushed branch shows a green check that actually ran the tests.
+- **Effort:** M · **Depends on:** 4.1
+
+### Task 4.3 — Desktop / Start-Menu shortcut on install
+- **Goal:** One-click launch (the user already pins to taskbar; a shortcut makes first run easy).
+- **Files:** `src/package.ps1` (extend) or a new `install-shortcut.ps1`.
+- **Steps:** After packaging, create a `.lnk` via `WScript.Shell` with `TargetPath` = the exe and `WorkingDirectory` = its folder (so `models\` resolves even though we also use `exe_dir()`), `IconLocation` = the exe. (Snippet in `MODERN-UI-AND-FIXES.md` §9 / spec doc §9.)
+- **Done when:** Running the script creates a working Desktop shortcut that launches the app with the icon.
+- **Effort:** S
+
+### Task 4.4 — Fix the documentation (it describes the OLD app)
+- **Goal:** `README.md`, `src/README.md`, `src/CHANGES.md`, `src/CUDA-SETUP.md`, `src/QUICK-REBUILD-GPU.md`, `src/TESTING.md`, `src/DESIGN.md`, `src/FIXES-APPLIED.md` still describe the **streaming / ring-buffer / 24-thread / Ctrl+Shift+R** design and RTX-3090 benchmarks — all now wrong/misleading.
+- **Steps:**
+  1. Rewrite the top-level `README.md` to describe the **current** app: push-to-talk batch transcription, `Ctrl+Shift+Space` to record/stop, `Ctrl+Shift+H` to hide, tray, always-on-top, copy + auto-paste, model/mic selectors, CPU-tuned (physical-core threads, tiny.en default). Remove ring-buffer/VAD/24-thread/streaming claims and the RTX benchmarks (or move GPU notes to an "optional" aside).
+  2. Update `build.ps1` end-of-run messages ("Hotkey: Ctrl+Shift+R", "Model: base.en") to match reality.
+  3. Mark `CHANGES.md`, `FIXES-APPLIED.md`, `TESTING.md`, `QUICK-REBUILD-GPU.md`, `CUDA-SETUP.md` as **historical/superseded** (a one-line banner at top), or fold the still-true bits into the README and delete the rest. Keep `DESIGN.md` only if updated to the batch architecture.
+- **Done when:** A new reader following `README.md` gets accurate hotkeys, model behaviour, and build steps; no doc claims a ring buffer or 24 threads.
+- **Effort:** M
+
+---
+
+## Phase 5 — Stretch features (nice-to-have)
+
+### Task 5.1 — Hold-to-talk mode
+- **Goal:** Option to record only while a key is held (vs. toggle).
+- **Files:** `src/main.cpp`, `src/settings.h`.
+- **Steps:** Add a low-level keyboard hook (`SetWindowsHookEx(WH_KEYBOARD_LL, ...)`); on key-down of the chosen key start recording, on key-up `stop_and_transcribe`; debounce auto-repeat with a flag. Gate behind a `holdToTalk` INI setting; keep toggle as default. (Outline in the original spec doc §7.)
+- **Done when:** With the setting on, holding the key records and releasing transcribes; toggle mode still available.
+- **Effort:** M
+
+### Task 5.2 — Settings panel (graduate from tray toggles + INI)
+- **Goal:** A small in-app settings popup (reuse the custom popup window from Task 1.4) for mic, model, auto-paste, auto-hide, hold-to-talk, and hotkey capture.
+- **Done when:** All settings are editable in-app and persist (Task 3.1).
+- **Effort:** L · **Depends on:** 1.4, 3.1
+
+### Task 5.3 — Export / save transcript
+- **Goal:** Save the transcript box to a `.txt` (and timestamped filename) from the tray menu or a button.
+- **Effort:** S
+
+### Task 5.4 — Multi-language support
+- **Goal:** Allow non-English models + a language selector (currently hard-wired `en`). Swap to a multilingual model (`ggml-base.bin`) and set `WhisperConfig.language` from a selector.
+- **Effort:** M · **Depends on:** 1.4 (selector), 3.1
+
+---
+
+## Definition of Done (per task)
+- [ ] Builds clean (`cmake --build build --config Release`), **zero new warnings**.
+- [ ] App launches, records, transcribes, appends, copies/pastes — no regressions.
+- [ ] `test-core.exe` exits 0 (after Phase 4).
+- [ ] Any new setting persists across relaunch (after Phase 3).
+- [ ] Change is reflected in `README.md` if user-facing.
+
+## Suggested order (fastest path to "feels finished")
+1. **0.1–0.3** (hygiene) → **1.3** (hairlines, 5-min win) → **2.1** (progress — biggest UX gain).
+2. **1.1** (icon) → **1.4** (custom dropdowns) → **1.2** (tray state) → **2.2** (cancel).
+3. **3.1** (persistence) → **3.2** (tray toggles) → **3.4** (hotkey safety) → **3.3** (length cap) → **3.5** (logging).
+4. **4.1** (tests) → **4.4** (docs) → **4.3** (shortcut) → **4.2** (CI).
+5. Stretch (**5.x**) as desired.
+
+## Verification matrix (final smoke test)
+| Area | Check |
+|---|---|
+| Crash-free | Record/stop 10× incl. a 2-min clip; window never vanishes; process stable |
+| Progress | 2-min clip shows filling bar + % + shrinking ETA; cancel works |
+| Chrome | No hairlines/focus rects; selectors have one chevron + dark popup |
+| Paste | Hotkey-from-another-app pastes the latest utterance; button = copy only |
+| Persistence | mic/model/pin/auto-paste/window pos restored after relaunch |
+| Assets | New icon crisp in title bar, taskbar, Alt-Tab, tray; red tray icon while recording |
+| Robustness | Missing model → clear message (no crash); hotkey conflict → warned; long record auto-stops |
+| Tests/docs | `test-core` green; README matches actual hotkeys/behaviour |
+
+

ggml/.gitignore

@@ -0,0 +1 @@
+src/ggml-metal-embed.metal

ggml/CMakeLists.txt

@@ -0,0 +1,445 @@
+cmake_minimum_required(VERSION 3.14) # for add_link_options and implicit target directories.
+project("ggml" C CXX)
+include(CheckIncludeFileCXX)
+
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+if (NOT XCODE AND NOT MSVC AND NOT CMAKE_BUILD_TYPE)
+    set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE)
+    set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo")
+endif()
+
+if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
+    set(GGML_STANDALONE ON)
+
+    set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
+
+    # configure project version
+    # TODO
+else()
+    set(GGML_STANDALONE OFF)
+endif()
+
+if (EMSCRIPTEN)
+    set(BUILD_SHARED_LIBS_DEFAULT OFF)
+
+    option(GGML_WASM_SINGLE_FILE "ggml: embed WASM inside the generated ggml.js" ON)
+else()
+    if (MINGW)
+        set(BUILD_SHARED_LIBS_DEFAULT OFF)
+    else()
+        set(BUILD_SHARED_LIBS_DEFAULT ON)
+    endif()
+endif()
+
+# remove the lib prefix on win32 mingw
+if (WIN32)
+    set(CMAKE_STATIC_LIBRARY_PREFIX "")
+    set(CMAKE_SHARED_LIBRARY_PREFIX "")
+    set(CMAKE_SHARED_MODULE_PREFIX  "")
+endif()
+
+option(BUILD_SHARED_LIBS "ggml: build shared libraries" ${BUILD_SHARED_LIBS_DEFAULT})
+option(GGML_BACKEND_DL   "ggml: build backends as dynamic libraries (requires BUILD_SHARED_LIBS)" OFF)
+
+#
+# option list
+#
+
+# TODO: mark all options as advanced when not GGML_STANDALONE
+
+if (APPLE)
+    set(GGML_METAL_DEFAULT ON)
+    set(GGML_BLAS_DEFAULT ON)
+    set(GGML_BLAS_VENDOR_DEFAULT "Apple")
+else()
+    set(GGML_METAL_DEFAULT OFF)
+    set(GGML_BLAS_DEFAULT OFF)
+    set(GGML_BLAS_VENDOR_DEFAULT "Generic")
+endif()
+
+if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH})
+    message(STATUS "Setting GGML_NATIVE_DEFAULT to OFF")
+    set(GGML_NATIVE_DEFAULT OFF)
+else()
+    set(GGML_NATIVE_DEFAULT ON)
+endif()
+
+# defaults
+if (NOT GGML_LLAMAFILE_DEFAULT)
+    set(GGML_LLAMAFILE_DEFAULT OFF)
+endif()
+
+if (NOT GGML_CUDA_GRAPHS_DEFAULT)
+    set(GGML_CUDA_GRAPHS_DEFAULT OFF)
+endif()
+
+# general
+option(GGML_STATIC "ggml: static link libraries"                     OFF)
+option(GGML_NATIVE "ggml: optimize the build for the current system" ${GGML_NATIVE_DEFAULT})
+option(GGML_LTO    "ggml: enable link time optimization"             OFF)
+option(GGML_CCACHE "ggml: use ccache if available"                   ON)
+
+# debug
+option(GGML_ALL_WARNINGS           "ggml: enable all compiler warnings"                   ON)
+option(GGML_ALL_WARNINGS_3RD_PARTY "ggml: enable all compiler warnings in 3rd party libs" OFF)
+option(GGML_GPROF                  "ggml: enable gprof"                                   OFF)
+
+# build
+option(GGML_FATAL_WARNINGS    "ggml: enable -Werror flag"    OFF)
+
+# sanitizers
+option(GGML_SANITIZE_THREAD    "ggml: enable thread sanitizer"    OFF)
+option(GGML_SANITIZE_ADDRESS   "ggml: enable address sanitizer"   OFF)
+option(GGML_SANITIZE_UNDEFINED "ggml: enable undefined sanitizer" OFF)
+
+# instruction set specific
+if (GGML_NATIVE OR NOT GGML_NATIVE_DEFAULT)
+    set(INS_ENB OFF)
+else()
+    set(INS_ENB ON)
+endif()
+
+message(DEBUG "GGML_NATIVE         : ${GGML_NATIVE}")
+message(DEBUG "GGML_NATIVE_DEFAULT : ${GGML_NATIVE_DEFAULT}")
+message(DEBUG "INS_ENB             : ${INS_ENB}")
+
+option(GGML_CPU_HBM          "ggml: use memkind for CPU HBM" OFF)
+option(GGML_CPU_REPACK       "ggml: use runtime weight conversion of Q4_0 to Q4_X_X" ON)
+option(GGML_CPU_KLEIDIAI     "ggml: use KleidiAI optimized kernels if applicable" OFF)
+option(GGML_SSE42            "ggml: enable SSE 4.2"          ${INS_ENB})
+option(GGML_AVX              "ggml: enable AVX"              ${INS_ENB})
+option(GGML_AVX_VNNI         "ggml: enable AVX-VNNI"         OFF)
+option(GGML_AVX2             "ggml: enable AVX2"             ${INS_ENB})
+option(GGML_BMI2             "ggml: enable BMI2"             ${INS_ENB})
+option(GGML_AVX512           "ggml: enable AVX512F"          OFF)
+option(GGML_AVX512_VBMI      "ggml: enable AVX512-VBMI"      OFF)
+option(GGML_AVX512_VNNI      "ggml: enable AVX512-VNNI"      OFF)
+option(GGML_AVX512_BF16      "ggml: enable AVX512-BF16"      OFF)
+if (NOT MSVC)
+    # in MSVC F16C and FMA is implied with AVX2/AVX512
+    option(GGML_FMA          "ggml: enable FMA"              ${INS_ENB})
+    option(GGML_F16C         "ggml: enable F16C"             ${INS_ENB})
+    # MSVC does not seem to support AMX
+    option(GGML_AMX_TILE     "ggml: enable AMX-TILE"         OFF)
+    option(GGML_AMX_INT8     "ggml: enable AMX-INT8"         OFF)
+    option(GGML_AMX_BF16     "ggml: enable AMX-BF16"         OFF)
+endif()
+option(GGML_LASX             "ggml: enable lasx"             ON)
+option(GGML_LSX              "ggml: enable lsx"              ON)
+option(GGML_RVV              "ggml: enable rvv"              ON)
+option(GGML_RV_ZFH           "ggml: enable riscv zfh"        OFF)
+option(GGML_XTHEADVECTOR     "ggml: enable xtheadvector"     OFF)
+option(GGML_VXE              "ggml: enable vxe"              ON)
+option(GGML_NNPA             "ggml: enable nnpa"             OFF)  # temp disabled by default, see: https://github.com/ggml-org/llama.cpp/issues/14877
+
+option(GGML_CPU_ALL_VARIANTS "ggml: build all variants of the CPU backend (requires GGML_BACKEND_DL)" OFF)
+set(GGML_CPU_ARM_ARCH        "" CACHE STRING "ggml: CPU architecture for ARM")
+set(GGML_CPU_POWERPC_CPUTYPE "" CACHE STRING "ggml: CPU type for PowerPC")
+
+
+if (MINGW)
+    set(GGML_WIN_VER "0x602" CACHE STRING   "ggml: Windows version")
+endif()
+
+# ggml core
+set(GGML_SCHED_MAX_COPIES  "4" CACHE STRING "ggml: max input copies for pipeline parallelism")
+option(GGML_CPU                             "ggml: enable CPU backend"                        ON)
+
+# 3rd party libs / backends
+option(GGML_ACCELERATE                      "ggml: enable Accelerate framework"               ON)
+option(GGML_BLAS                            "ggml: use BLAS"                                  ${GGML_BLAS_DEFAULT})
+set(GGML_BLAS_VENDOR ${GGML_BLAS_VENDOR_DEFAULT} CACHE STRING
+                                            "ggml: BLAS library vendor")
+option(GGML_LLAMAFILE                       "ggml: use LLAMAFILE"                             ${GGML_LLAMAFILE_DEFAULT})
+
+option(GGML_CUDA                            "ggml: use CUDA"                                  OFF)
+option(GGML_MUSA                            "ggml: use MUSA"                                  OFF)
+option(GGML_CUDA_FORCE_MMQ                  "ggml: use mmq kernels instead of cuBLAS"         OFF)
+option(GGML_CUDA_FORCE_CUBLAS               "ggml: always use cuBLAS instead of mmq kernels"  OFF)
+option(GGML_CUDA_F16                        "ggml: use 16 bit floats for some calculations"   OFF)
+set   (GGML_CUDA_PEER_MAX_BATCH_SIZE "128" CACHE STRING
+                                            "ggml: max. batch size for using peer access")
+option(GGML_CUDA_NO_PEER_COPY               "ggml: do not use peer to peer copies"            OFF)
+option(GGML_CUDA_NO_VMM                     "ggml: do not try to use CUDA VMM"                OFF)
+option(GGML_CUDA_FA                         "ggml: compile ggml FlashAttention CUDA kernels"  ON)
+option(GGML_CUDA_FA_ALL_QUANTS              "ggml: compile all quants for FlashAttention"     OFF)
+option(GGML_CUDA_GRAPHS                     "ggml: use CUDA graphs (llama.cpp only)"          ${GGML_CUDA_GRAPHS_DEFAULT})
+set   (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING
+                                            "ggml: cuda link binary compression mode; requires cuda 12.8+")
+set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size")
+
+option(GGML_HIP                             "ggml: use HIP"                                   OFF)
+option(GGML_HIP_GRAPHS                      "ggml: use HIP graph, experimental, slow"         OFF)
+option(GGML_HIP_NO_VMM                      "ggml: do not try to use HIP VMM"                 ON)
+option(GGML_HIP_ROCWMMA_FATTN               "ggml: enable rocWMMA for FlashAttention"         OFF)
+option(GGML_HIP_FORCE_ROCWMMA_FATTN_GFX12   "ggml: enable rocWMMA FlashAttention on GFX12"    OFF)
+option(GGML_MUSA_GRAPHS                     "ggml: use MUSA graph, experimental, unstable"    OFF)
+option(GGML_MUSA_MUDNN_COPY                 "ggml: enable muDNN for accelerated copy"         OFF)
+option(GGML_VULKAN                          "ggml: use Vulkan"                                OFF)
+option(GGML_VULKAN_CHECK_RESULTS            "ggml: run Vulkan op checks"                      OFF)
+option(GGML_VULKAN_DEBUG                    "ggml: enable Vulkan debug output"                OFF)
+option(GGML_VULKAN_MEMORY_DEBUG             "ggml: enable Vulkan memory debug output"         OFF)
+option(GGML_VULKAN_SHADER_DEBUG_INFO        "ggml: enable Vulkan shader debug info"           OFF)
+option(GGML_VULKAN_VALIDATE                 "ggml: enable Vulkan validation"                  OFF)
+option(GGML_VULKAN_RUN_TESTS                "ggml: run Vulkan tests"                          OFF)
+option(GGML_WEBGPU                          "ggml: use WebGPU"                                OFF)
+option(GGML_WEBGPU_DEBUG                    "ggml: enable WebGPU debug output"                OFF)
+option(GGML_METAL                           "ggml: use Metal"                                 ${GGML_METAL_DEFAULT})
+option(GGML_METAL_USE_BF16                  "ggml: use bfloat if available"                   OFF)
+option(GGML_METAL_NDEBUG                    "ggml: disable Metal debugging"                   OFF)
+option(GGML_METAL_SHADER_DEBUG              "ggml: compile Metal with -fno-fast-math"         OFF)
+option(GGML_METAL_EMBED_LIBRARY             "ggml: embed Metal library"                       ${GGML_METAL})
+set   (GGML_METAL_MACOSX_VERSION_MIN "" CACHE STRING
+                                            "ggml: metal minimum macOS version")
+set   (GGML_METAL_STD "" CACHE STRING       "ggml: metal standard version (-std flag)")
+option(GGML_OPENMP                          "ggml: use OpenMP"                                ON)
+option(GGML_RPC                             "ggml: use RPC"                                   OFF)
+option(GGML_SYCL                            "ggml: use SYCL"                                  OFF)
+option(GGML_SYCL_F16                        "ggml: use 16 bit floats for sycl calculations"   OFF)
+option(GGML_SYCL_GRAPH                      "ggml: enable graphs in the SYCL backend"         ON)
+option(GGML_SYCL_DNN                        "ggml: enable oneDNN in the SYCL backend"         ON)
+set   (GGML_SYCL_TARGET "INTEL" CACHE STRING
+                                            "ggml: sycl target device")
+set   (GGML_SYCL_DEVICE_ARCH "" CACHE STRING
+                                            "ggml: sycl device architecture")
+
+option(GGML_OPENCL                          "ggml: use OpenCL"                                OFF)
+option(GGML_OPENCL_PROFILING                "ggml: use OpenCL profiling (increases overhead)" OFF)
+option(GGML_OPENCL_EMBED_KERNELS            "ggml: embed kernels"                             ON)
+option(GGML_OPENCL_USE_ADRENO_KERNELS       "ggml: use optimized kernels for Adreno"          ON)
+set   (GGML_OPENCL_TARGET_VERSION "300" CACHE STRING
+                                            "gmml: OpenCL API version to target")
+
+# toolchain for vulkan-shaders-gen
+set   (GGML_VULKAN_SHADERS_GEN_TOOLCHAIN "" CACHE FILEPATH "ggml: toolchain file for vulkan-shaders-gen")
+
+# extra artifacts
+option(GGML_BUILD_TESTS    "ggml: build tests"    ${GGML_STANDALONE})
+option(GGML_BUILD_EXAMPLES "ggml: build examples" ${GGML_STANDALONE})
+
+#
+# dependencies
+#
+
+set(CMAKE_C_STANDARD 11)
+set(CMAKE_C_STANDARD_REQUIRED true)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED true)
+
+set(THREADS_PREFER_PTHREAD_FLAG ON)
+
+find_package(Threads REQUIRED)
+
+include(GNUInstallDirs)
+
+#
+# build the library
+#
+
+add_subdirectory(src)
+
+#
+# tests and examples
+#
+
+if (GGML_BUILD_TESTS)
+    enable_testing()
+    add_subdirectory(tests)
+endif ()
+
+if (GGML_BUILD_EXAMPLES)
+    add_subdirectory(examples)
+endif ()
+
+#
+# install
+#
+
+include(CMakePackageConfigHelpers)
+
+# all public headers
+set(GGML_PUBLIC_HEADERS
+    include/ggml.h
+    include/ggml-cpu.h
+    include/ggml-alloc.h
+    include/ggml-backend.h
+    include/ggml-blas.h
+    include/ggml-cann.h
+    include/ggml-cpp.h
+    include/ggml-cuda.h
+    include/ggml-opt.h
+    include/ggml-metal.h
+    include/ggml-rpc.h
+    include/ggml-sycl.h
+    include/ggml-vulkan.h
+    include/ggml-webgpu.h
+    include/gguf.h)
+
+set_target_properties(ggml PROPERTIES PUBLIC_HEADER "${GGML_PUBLIC_HEADERS}")
+#if (GGML_METAL)
+#    set_target_properties(ggml PROPERTIES RESOURCE "${CMAKE_CURRENT_SOURCE_DIR}/src/ggml-metal.metal")
+#endif()
+install(TARGETS ggml LIBRARY PUBLIC_HEADER)
+install(TARGETS ggml-base LIBRARY)
+
+if (GGML_STANDALONE)
+    configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ggml.pc.in
+        ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc
+        @ONLY)
+
+    install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc
+        DESTINATION share/pkgconfig)
+endif()
+
+#
+# Create CMake package
+#
+
+# Generate version info based on git commit.
+
+if(NOT DEFINED GGML_BUILD_NUMBER)
+    find_program(GIT_EXE NAMES git git.exe REQUIRED NO_CMAKE_FIND_ROOT_PATH)
+    execute_process(COMMAND ${GIT_EXE} rev-list --count HEAD
+        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+        OUTPUT_VARIABLE GGML_BUILD_NUMBER
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+    )
+
+    if(GGML_BUILD_NUMBER EQUAL 1)
+        message(WARNING "GGML build version fixed at 1 likely due to a shallow clone.")
+    endif()
+
+    execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD
+        WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+        OUTPUT_VARIABLE GGML_BUILD_COMMIT
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+    )
+endif()
+
+
+# Capture variables prefixed with GGML_.
+
+set(variable_set_statements
+"
+####### Expanded from @GGML_VARIABLES_EXPANED@ by configure_package_config_file() #######
+####### Any changes to this file will be overwritten by the next CMake run        #######
+
+")
+
+set(GGML_SHARED_LIB ${BUILD_SHARED_LIBS})
+
+get_cmake_property(all_variables VARIABLES)
+foreach(variable_name IN LISTS all_variables)
+    if(variable_name MATCHES "^GGML_")
+        string(REPLACE ";" "\\;"
+               variable_value "${${variable_name}}")
+
+        set(variable_set_statements
+            "${variable_set_statements}set(${variable_name} \"${variable_value}\")\n")
+    endif()
+endforeach()
+
+set(GGML_VARIABLES_EXPANDED ${variable_set_statements})
+
+# Create the CMake package and set install location.
+
+set(GGML_INSTALL_VERSION 0.0.${GGML_BUILD_NUMBER})
+set(GGML_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header  files")
+set(GGML_LIB_INSTALL_DIR     ${CMAKE_INSTALL_LIBDIR}     CACHE PATH "Location of library files")
+set(GGML_BIN_INSTALL_DIR     ${CMAKE_INSTALL_BINDIR}     CACHE PATH "Location of binary  files")
+
+configure_package_config_file(
+        ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-config.cmake.in
+        ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake
+    INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml
+    PATH_VARS GGML_INCLUDE_INSTALL_DIR
+              GGML_LIB_INSTALL_DIR
+              GGML_BIN_INSTALL_DIR)
+
+write_basic_package_version_file(
+        ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake
+    VERSION ${GGML_INSTALL_VERSION}
+    COMPATIBILITY SameMajorVersion)
+
+target_compile_definitions(ggml-base PRIVATE
+    GGML_VERSION="${GGML_INSTALL_VERSION}"
+    GGML_COMMIT="${GGML_BUILD_COMMIT}"
+)
+message(STATUS "ggml version: ${GGML_INSTALL_VERSION}")
+message(STATUS "ggml commit:  ${GGML_BUILD_COMMIT}")
+
+install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake
+              ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake
+        DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml)
+
+if (MSVC)
+    set(MSVC_WARNING_FLAGS
+        /wd4005  # Macro redefinition
+        /wd4244  # Conversion from one type to another type, possible loss of data
+        /wd4267  # Conversion from 'size_t' to a smaller type, possible loss of data
+        /wd4305  # Conversion from 'type1' to 'type2', possible loss of data
+        /wd4566  # Conversion from 'char' to 'wchar_t', possible loss of data
+        /wd4996  # Disable POSIX deprecation warnings
+        /wd4702  # Unreachable code warnings
+    )
+    function(disable_msvc_warnings target_name)
+        if(TARGET ${target_name})
+            target_compile_options(${target_name} PRIVATE ${MSVC_WARNING_FLAGS})
+        endif()
+    endfunction()
+
+    disable_msvc_warnings(ggml-base)
+    disable_msvc_warnings(ggml)
+    disable_msvc_warnings(ggml-cpu)
+    disable_msvc_warnings(ggml-cpu-x64)
+    disable_msvc_warnings(ggml-cpu-sse42)
+    disable_msvc_warnings(ggml-cpu-sandybridge)
+    disable_msvc_warnings(ggml-cpu-haswell)
+    disable_msvc_warnings(ggml-cpu-skylakex)
+    disable_msvc_warnings(ggml-cpu-icelake)
+    disable_msvc_warnings(ggml-cpu-alderlake)
+
+    if (GGML_BUILD_EXAMPLES)
+        disable_msvc_warnings(common-ggml)
+        disable_msvc_warnings(common)
+
+        disable_msvc_warnings(mnist-common)
+        disable_msvc_warnings(mnist-eval)
+        disable_msvc_warnings(mnist-train)
+
+        disable_msvc_warnings(gpt-2-ctx)
+        disable_msvc_warnings(gpt-2-alloc)
+        disable_msvc_warnings(gpt-2-backend)
+        disable_msvc_warnings(gpt-2-sched)
+        disable_msvc_warnings(gpt-2-quantize)
+        disable_msvc_warnings(gpt-2-batched)
+
+        disable_msvc_warnings(gpt-j)
+        disable_msvc_warnings(gpt-j-quantize)
+
+        disable_msvc_warnings(magika)
+        disable_msvc_warnings(yolov3-tiny)
+        disable_msvc_warnings(sam)
+
+        disable_msvc_warnings(simple-ctx)
+        disable_msvc_warnings(simple-backend)
+    endif()
+
+    if (GGML_BUILD_TESTS)
+        disable_msvc_warnings(test-mul-mat)
+        disable_msvc_warnings(test-arange)
+        disable_msvc_warnings(test-backend-ops)
+        disable_msvc_warnings(test-cont)
+        disable_msvc_warnings(test-conv-transpose)
+        disable_msvc_warnings(test-conv-transpose-1d)
+        disable_msvc_warnings(test-conv1d)
+        disable_msvc_warnings(test-conv2d)
+        disable_msvc_warnings(test-conv2d-dw)
+        disable_msvc_warnings(test-customop)
+        disable_msvc_warnings(test-dup)
+        disable_msvc_warnings(test-opt)
+        disable_msvc_warnings(test-pool)
+    endif ()
+endif()

ggml/cmake/common.cmake

@@ -0,0 +1,25 @@
+include_guard(GLOBAL)
+
+function(add_library name)
+    _add_library(${name} ${ARGN})
+    set(TARGET ${name} PARENT_SCOPE)
+endfunction()
+
+function(ggml_get_system_arch)
+    set(options)
+    set(oneValueArgs RESULT)
+    set(multiValueArgs)
+    cmake_parse_arguments(ARG "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
+
+    if(CMAKE_OSX_ARCHITECTURES STREQUAL "arm64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64")
+        set(${ARG_RESULT} "arm64" PARENT_SCOPE)
+    elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64|AMD64")
+        set(${ARG_RESULT} "x86" PARENT_SCOPE)
+    elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "i[3-6]86")
+        set(${ARG_RESULT} "x86" PARENT_SCOPE)
+    elseif(CMAKE_SIZEOF_VOID_P EQUAL 8)
+        set(${ARG_RESULT} "x86" PARENT_SCOPE)
+    else()
+        set(${ARG_RESULT} "x86" PARENT_SCOPE)
+    endif()
+endfunction()

ggml/cmake/ggml-config.cmake.in

@@ -0,0 +1,189 @@
+@PACKAGE_INIT@
+
+@GGML_VARIABLES_EXPANDED@
+
+# Find all dependencies before creating any target.
+include(CMakeFindDependencyMacro)
+find_dependency(Threads)
+if (NOT GGML_SHARED_LIB)
+    set(GGML_CPU_INTERFACE_LINK_LIBRARIES "")
+    set(GGML_CPU_INTERFACE_LINK_OPTIONS   "")
+
+    if (APPLE AND GGML_ACCELERATE)
+        find_library(ACCELERATE_FRAMEWORK Accelerate)
+        if(NOT ACCELERATE_FRAMEWORK)
+            set(${CMAKE_FIND_PACKAGE_NAME}_FOUND 0)
+            return()
+        endif()
+        list(APPEND GGML_CPU_INTERFACE_LINK_LIBRARIES ${ACCELERATE_FRAMEWORK})
+    endif()
+
+    if (GGML_OPENMP_ENABLED)
+        find_dependency(OpenMP)
+        list(APPEND GGML_CPU_INTERFACE_LINK_LIBRARIES OpenMP::OpenMP_C OpenMP::OpenMP_CXX)
+    endif()
+
+    if (GGML_CPU_HBM)
+        find_library(memkind memkind)
+        if(NOT memkind)
+            set(${CMAKE_FIND_PACKAGE_NAME}_FOUND 0)
+            return()
+        endif()
+        list(APPEND GGML_CPU_INTERFACE_LINK_LIBRARIES memkind)
+    endif()
+
+    if (GGML_BLAS)
+        find_dependency(BLAS)
+        list(APPEND GGML_CPU_INTERFACE_LINK_LIBRARIES ${BLAS_LIBRARIES})
+        list(APPEND GGML_CPU_INTERFACE_LINK_OPTIONS   ${BLAS_LINKER_FLAGS})
+    endif()
+
+    if (GGML_CUDA)
+        set(GGML_CUDA_INTERFACE_LINK_LIBRARIES "")
+        find_dependency(CUDAToolkit)
+        if (GGML_STATIC)
+            list(APPEND GGML_CUDA_INTERFACE_LINK_LIBRARIES $<LINK_ONLY:CUDA::cudart_static>)
+            if (WIN32)
+                list(APPEND GGML_CUDA_INTERFACE_LINK_LIBRARIES $<LINK_ONLY:CUDA::cublas> $<LINK_ONLY:CUDA::cublasLt>)
+            else()
+                list(APPEND GGML_CUDA_INTERFACE_LINK_LIBRARIES $<LINK_ONLY:CUDA::cublas_static> $<LINK_ONLY:CUDA::cublasLt_static>)
+            endif()
+        endif()
+        if (NOT GGML_CUDA_NO_VMM)
+            list(APPEND GGML_CUDA_INTERFACE_LINK_LIBRARIES $<LINK_ONLY:CUDA::cuda_driver>)
+        endif()
+    endif()
+
+    if (GGML_METAL)
+        find_library(FOUNDATION_LIBRARY Foundation)
+        find_library(METAL_FRAMEWORK    Metal)
+        find_library(METALKIT_FRAMEWORK MetalKit)
+        if(NOT FOUNDATION_LIBRARY OR NOT METAL_FRAMEWORK OR NOT METALKIT_FRAMEWORK)
+            set(${CMAKE_FIND_PACKAGE_NAME}_FOUND 0)
+            return()
+        endif()
+        set(GGML_METAL_INTERFACE_LINK_LIBRARIES
+            ${FOUNDATION_LIBRARY} ${METAL_FRAMEWORK} ${METALKIT_FRAMEWORK})
+    endif()
+
+    if (GGML_OPENCL)
+        find_dependency(OpenCL)
+        set(GGML_OPENCL_INTERFACE_LINK_LIBRARIES $<LINK_ONLY:OpenCL::OpenCL>)
+    endif()
+
+    if (GGML_VULKAN)
+        find_dependency(Vulkan)
+        set(GGML_VULKAN_INTERFACE_LINK_LIBRARIES $<LINK_ONLY:Vulkan::Vulkan>)
+    endif()
+
+    if (GGML_HIP)
+        find_dependency(hip)
+        find_dependency(hipblas)
+        find_dependency(rocblas)
+        set(GGML_HIP_INTERFACE_LINK_LIBRARIES hip::host roc::rocblas roc::hipblas)
+    endif()
+
+    if (GGML_SYCL)
+        set(GGML_SYCL_INTERFACE_LINK_LIBRARIES "")
+        find_package(DNNL)
+        if (${DNNL_FOUND} AND GGML_SYCL_TARGET STREQUAL "INTEL")
+            list(APPEND GGML_SYCL_INTERFACE_LINK_LIBRARIES DNNL::dnnl)
+        endif()
+        if (WIN32)
+            find_dependency(IntelSYCL)
+            find_dependency(MKL)
+            list(APPEND GGML_SYCL_INTERFACE_LINK_LIBRARIES IntelSYCL::SYCL_CXX MKL::MKL MKL::MKL_SYCL)
+        endif()
+    endif()
+endif()
+
+set_and_check(GGML_INCLUDE_DIR "@PACKAGE_GGML_INCLUDE_INSTALL_DIR@")
+set_and_check(GGML_LIB_DIR "@PACKAGE_GGML_LIB_INSTALL_DIR@")
+#set_and_check(GGML_BIN_DIR "@PACKAGE_GGML_BIN_INSTALL_DIR@")
+
+if(NOT TARGET ggml::ggml)
+    find_package(Threads REQUIRED)
+
+    find_library(GGML_LIBRARY ggml
+        REQUIRED
+        HINTS ${GGML_LIB_DIR}
+        NO_CMAKE_FIND_ROOT_PATH)
+
+    add_library(ggml::ggml UNKNOWN IMPORTED)
+    set_target_properties(ggml::ggml
+        PROPERTIES
+            IMPORTED_LOCATION "${GGML_LIBRARY}")
+
+    find_library(GGML_BASE_LIBRARY ggml-base
+        REQUIRED
+        HINTS ${GGML_LIB_DIR}
+        NO_CMAKE_FIND_ROOT_PATH)
+
+    add_library(ggml::ggml-base UNKNOWN IMPORTED)
+    set_target_properties(ggml::ggml-base
+        PROPERTIES
+            IMPORTED_LOCATION "${GGML_BASE_LIBRARY}")
+
+    set(_ggml_all_targets "")
+    foreach(_ggml_backend ${GGML_AVAILABLE_BACKENDS})
+        string(REPLACE "-" "_" _ggml_backend_pfx "${_ggml_backend}")
+        string(TOUPPER "${_ggml_backend_pfx}" _ggml_backend_pfx)
+
+        find_library(${_ggml_backend_pfx}_LIBRARY ${_ggml_backend}
+            REQUIRED
+            HINTS ${GGML_LIB_DIR}
+            NO_CMAKE_FIND_ROOT_PATH)
+
+        message(STATUS "Found ${${_ggml_backend_pfx}_LIBRARY}")
+
+        add_library(ggml::${_ggml_backend} UNKNOWN IMPORTED)
+        set_target_properties(ggml::${_ggml_backend}
+            PROPERTIES
+                INTERFACE_INCLUDE_DIRECTORIES "${GGML_INCLUDE_DIR}"
+                IMPORTED_LINK_INTERFACE_LANGUAGES "CXX"
+                IMPORTED_LOCATION "${${_ggml_backend_pfx}_LIBRARY}"
+                INTERFACE_COMPILE_FEATURES c_std_90
+                POSITION_INDEPENDENT_CODE ON)
+
+        string(REGEX MATCH "^ggml-cpu" is_cpu_variant "${_ggml_backend}")
+        if(is_cpu_variant)
+            list(APPEND GGML_CPU_INTERFACE_LINK_LIBRARIES "ggml::ggml-base")
+            set_target_properties(ggml::${_ggml_backend}
+            PROPERTIES
+                INTERFACE_LINK_LIBRARIES "${GGML_CPU_INTERFACE_LINK_LIBRARIES}")
+
+            if(GGML_CPU_INTERFACE_LINK_OPTIONS)
+                set_target_properties(ggml::${_ggml_backend}
+                    PROPERTIES
+                        INTERFACE_LINK_OPTIONS "${GGML_CPU_INTERFACE_LINK_OPTIONS}")
+            endif()
+
+        else()
+            list(APPEND ${_ggml_backend_pfx}_INTERFACE_LINK_LIBRARIES "ggml::ggml-base")
+            set_target_properties(ggml::${_ggml_backend}
+                PROPERTIES
+                    INTERFACE_LINK_LIBRARIES "${${_ggml_backend_pfx}_INTERFACE_LINK_LIBRARIES}")
+
+            if(${_ggml_backend_pfx}_INTERFACE_LINK_OPTIONS)
+                set_target_properties(ggml::${_ggml_backend}
+                    PROPERTIES
+                        INTERFACE_LINK_OPTIONS "${${_ggml_backend_pfx}_INTERFACE_LINK_OPTIONS}")
+            endif()
+        endif()
+
+        list(APPEND _ggml_all_targets ggml::${_ggml_backend})
+    endforeach()
+
+    list(APPEND GGML_INTERFACE_LINK_LIBRARIES ggml::ggml-base "${_ggml_all_targets}")
+    set_target_properties(ggml::ggml
+        PROPERTIES
+            INTERFACE_LINK_LIBRARIES "${GGML_INTERFACE_LINK_LIBRARIES}")
+
+    add_library(ggml::all INTERFACE IMPORTED)
+    set_target_properties(ggml::all
+        PROPERTIES
+            INTERFACE_LINK_LIBRARIES "${_ggml_all_targets}")
+
+endif()
+
+check_required_components(ggml)

ggml/include/ggml-alloc.h

@@ -0,0 +1,76 @@
+#pragma once
+
+#include "ggml.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+typedef struct ggml_backend_buffer_type * ggml_backend_buffer_type_t;
+typedef struct      ggml_backend_buffer * ggml_backend_buffer_t;
+typedef struct             ggml_backend * ggml_backend_t;
+
+// Tensor allocator
+struct ggml_tallocr {
+    ggml_backend_buffer_t buffer;
+    void * base;
+    size_t alignment;
+    size_t offset;
+};
+
+GGML_API struct ggml_tallocr ggml_tallocr_new(ggml_backend_buffer_t buffer);
+GGML_API enum ggml_status    ggml_tallocr_alloc(struct ggml_tallocr * talloc, struct ggml_tensor * tensor);
+
+// Graph allocator
+/*
+  Example usage:
+    ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_cpu_buffer_type());
+
+    // optional: create a worst-case graph and reserve the buffers to avoid reallocations
+    ggml_gallocr_reserve(galloc, build_graph(max_batch));
+
+    // allocate the graph
+    struct ggml_cgraph * graph = build_graph(batch);
+    ggml_gallocr_alloc_graph(galloc, graph);
+
+    printf("compute buffer size: %zu bytes\n", ggml_gallocr_get_buffer_size(galloc, 0));
+
+    // evaluate the graph
+    ggml_backend_graph_compute(backend, graph);
+*/
+
+// special tensor flags for use with the graph allocator:
+//   ggml_set_input(): all input tensors are allocated at the beginning of the graph in non-overlapping addresses
+//   ggml_set_output(): output tensors are never freed and never overwritten
+
+typedef struct ggml_gallocr * ggml_gallocr_t;
+
+GGML_API ggml_gallocr_t ggml_gallocr_new(ggml_backend_buffer_type_t buft);
+GGML_API ggml_gallocr_t ggml_gallocr_new_n(ggml_backend_buffer_type_t * bufts, int n_bufs);
+GGML_API void           ggml_gallocr_free(ggml_gallocr_t galloc);
+
+// pre-allocate buffers from a measure graph - does not allocate or modify the graph
+// call with a worst-case graph to avoid buffer reallocations
+// not strictly required for single buffer usage: ggml_gallocr_alloc_graph will reallocate the buffers automatically if needed
+// returns false if the buffer allocation failed
+GGML_API bool ggml_gallocr_reserve(ggml_gallocr_t galloc, struct ggml_cgraph * graph);
+GGML_API bool ggml_gallocr_reserve_n(
+    ggml_gallocr_t galloc,
+    struct ggml_cgraph * graph,
+    const int * node_buffer_ids,
+    const int * leaf_buffer_ids);
+
+// automatic reallocation if the topology changes when using a single buffer
+// returns false if using multiple buffers and a re-allocation is needed (call ggml_gallocr_reserve_n first to set the node buffers)
+GGML_API bool ggml_gallocr_alloc_graph(ggml_gallocr_t galloc, struct ggml_cgraph * graph);
+
+GGML_API size_t ggml_gallocr_get_buffer_size(ggml_gallocr_t galloc, int buffer_id);
+
+// Utils
+// Create a buffer and allocate all the tensors in a ggml_context
+GGML_API struct ggml_backend_buffer * ggml_backend_alloc_ctx_tensors_from_buft(struct ggml_context * ctx, ggml_backend_buffer_type_t buft);
+GGML_API struct ggml_backend_buffer * ggml_backend_alloc_ctx_tensors(struct ggml_context * ctx, ggml_backend_t backend);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-backend.h

@@ -0,0 +1,354 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-alloc.h"
+
+#ifdef GGML_BACKEND_SHARED
+#    if defined(_WIN32) && !defined(__MINGW32__)
+#        ifdef GGML_BACKEND_BUILD
+#            define GGML_BACKEND_API __declspec(dllexport) extern
+#        else
+#            define GGML_BACKEND_API __declspec(dllimport) extern
+#        endif
+#    else
+#        define GGML_BACKEND_API __attribute__ ((visibility ("default"))) extern
+#    endif
+#else
+#    define GGML_BACKEND_API extern
+#endif
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+    typedef struct ggml_backend_buffer_type * ggml_backend_buffer_type_t;
+    typedef struct ggml_backend_buffer * ggml_backend_buffer_t;
+    typedef struct ggml_backend_event * ggml_backend_event_t;
+    typedef struct ggml_backend * ggml_backend_t;
+    typedef void * ggml_backend_graph_plan_t;
+    typedef struct ggml_backend_reg * ggml_backend_reg_t;
+    typedef struct ggml_backend_device * ggml_backend_dev_t;
+
+
+    //
+    // Backend buffer type
+    //
+
+    GGML_API const char *          ggml_backend_buft_name          (ggml_backend_buffer_type_t buft);
+    GGML_API ggml_backend_buffer_t ggml_backend_buft_alloc_buffer  (ggml_backend_buffer_type_t buft, size_t size);
+    GGML_API size_t                ggml_backend_buft_get_alignment (ggml_backend_buffer_type_t buft);
+    GGML_API size_t                ggml_backend_buft_get_max_size  (ggml_backend_buffer_type_t buft);
+    GGML_API size_t                ggml_backend_buft_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * tensor);
+    GGML_API bool                  ggml_backend_buft_is_host       (ggml_backend_buffer_type_t buft);
+    GGML_API ggml_backend_dev_t    ggml_backend_buft_get_device    (ggml_backend_buffer_type_t buft);
+
+    //
+    // Backend buffer
+    //
+
+    enum ggml_backend_buffer_usage {
+        GGML_BACKEND_BUFFER_USAGE_ANY = 0,
+        GGML_BACKEND_BUFFER_USAGE_WEIGHTS = 1,
+        GGML_BACKEND_BUFFER_USAGE_COMPUTE = 2,
+    };
+
+    GGML_API const char *                   ggml_backend_buffer_name          (ggml_backend_buffer_t buffer);
+    GGML_API void                           ggml_backend_buffer_free          (ggml_backend_buffer_t buffer);
+    GGML_API void *                         ggml_backend_buffer_get_base      (ggml_backend_buffer_t buffer);
+    GGML_API size_t                         ggml_backend_buffer_get_size      (ggml_backend_buffer_t buffer);
+    GGML_API enum ggml_status               ggml_backend_buffer_init_tensor   (ggml_backend_buffer_t buffer, struct ggml_tensor * tensor);
+    GGML_API size_t                         ggml_backend_buffer_get_alignment (ggml_backend_buffer_t buffer);
+    GGML_API size_t                         ggml_backend_buffer_get_max_size  (ggml_backend_buffer_t buffer);
+    GGML_API size_t                         ggml_backend_buffer_get_alloc_size(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor);
+    GGML_API void                           ggml_backend_buffer_clear         (ggml_backend_buffer_t buffer, uint8_t value);
+    GGML_API bool                           ggml_backend_buffer_is_host       (ggml_backend_buffer_t buffer);
+    GGML_API void                           ggml_backend_buffer_set_usage     (ggml_backend_buffer_t buffer, enum ggml_backend_buffer_usage usage);
+    GGML_API enum ggml_backend_buffer_usage ggml_backend_buffer_get_usage     (ggml_backend_buffer_t buffer);
+    GGML_API ggml_backend_buffer_type_t     ggml_backend_buffer_get_type      (ggml_backend_buffer_t buffer);
+    GGML_API void                           ggml_backend_buffer_reset         (ggml_backend_buffer_t buffer);
+
+    // tensor copy between different backends
+    GGML_API void ggml_backend_tensor_copy(struct ggml_tensor * src, struct ggml_tensor * dst);
+
+    //
+    // Backend (stream)
+    //
+
+    GGML_API ggml_guid_t  ggml_backend_guid(ggml_backend_t backend);
+    GGML_API const char * ggml_backend_name(ggml_backend_t backend);
+    GGML_API void         ggml_backend_free(ggml_backend_t backend);
+
+    GGML_API ggml_backend_buffer_type_t ggml_backend_get_default_buffer_type(ggml_backend_t backend);
+    GGML_API ggml_backend_buffer_t      ggml_backend_alloc_buffer(ggml_backend_t backend, size_t size);
+    GGML_API size_t                     ggml_backend_get_alignment(ggml_backend_t backend);
+    GGML_API size_t                     ggml_backend_get_max_size(ggml_backend_t backend);
+
+    GGML_API void ggml_backend_tensor_set_async(ggml_backend_t backend,       struct ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+    GGML_API void ggml_backend_tensor_get_async(ggml_backend_t backend, const struct ggml_tensor * tensor,       void * data, size_t offset, size_t size);
+
+    // "offset" refers to the offset in tensor->data for setting/getting data
+    GGML_API void ggml_backend_tensor_set(      struct ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+    GGML_API void ggml_backend_tensor_get(const struct ggml_tensor * tensor,       void * data, size_t offset, size_t size);
+    GGML_API void ggml_backend_tensor_memset(   struct ggml_tensor * tensor,     uint8_t value, size_t offset, size_t size);
+
+    GGML_API void ggml_backend_synchronize(ggml_backend_t backend);
+
+    GGML_API ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph);
+    GGML_API void                      ggml_backend_graph_plan_free  (ggml_backend_t backend, ggml_backend_graph_plan_t plan);
+
+    GGML_API enum ggml_status ggml_backend_graph_plan_compute (ggml_backend_t backend, ggml_backend_graph_plan_t plan);
+    GGML_API enum ggml_status ggml_backend_graph_compute      (ggml_backend_t backend, struct ggml_cgraph * cgraph);
+    GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
+
+    // NOTE: will be removed, use device version instead
+    GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
+    GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);
+    GGML_API bool ggml_backend_offload_op(ggml_backend_t backend, const struct ggml_tensor * op);
+
+    // asynchronous copy
+    // the copy is performed after all the currently queued operations in backend_src
+    // backend_dst will wait for the copy to complete before performing other operations
+    // automatic fallback to sync copy if async is not supported
+    GGML_API void ggml_backend_tensor_copy_async(ggml_backend_t backend_src, ggml_backend_t backend_dst, struct ggml_tensor * src, struct ggml_tensor * dst);
+
+    GGML_API ggml_backend_dev_t ggml_backend_get_device(ggml_backend_t backend);
+
+    //
+    // Events
+    //
+
+    GGML_API ggml_backend_event_t ggml_backend_event_new(ggml_backend_dev_t device);
+    GGML_API void                 ggml_backend_event_free(ggml_backend_event_t event);
+    GGML_API void                 ggml_backend_event_record(ggml_backend_event_t event, ggml_backend_t backend);
+    GGML_API void                 ggml_backend_event_synchronize(ggml_backend_event_t event);
+    GGML_API void                 ggml_backend_event_wait(ggml_backend_t backend, ggml_backend_event_t event);
+
+    //
+    // Backend device
+    //
+
+    enum ggml_backend_dev_type {
+        // CPU device using system memory
+        GGML_BACKEND_DEVICE_TYPE_CPU,
+        // GPU device using dedicated memory
+        GGML_BACKEND_DEVICE_TYPE_GPU,
+        // accelerator devices intended to be used together with the CPU backend (e.g. BLAS or AMX)
+        GGML_BACKEND_DEVICE_TYPE_ACCEL
+    };
+
+    // functionality supported by the device
+    struct ggml_backend_dev_caps {
+        // asynchronous operations
+        bool async;
+        // pinned host buffer
+        bool host_buffer;
+        // creating buffers from host ptr
+        bool buffer_from_host_ptr;
+        // event synchronization
+        bool events;
+    };
+
+    // all the device properties
+    struct ggml_backend_dev_props {
+        const char * name;
+        const char * description;
+        size_t memory_free;
+        size_t memory_total;
+        enum ggml_backend_dev_type type;
+        struct ggml_backend_dev_caps caps;
+    };
+
+    GGML_API const char *                  ggml_backend_dev_name(ggml_backend_dev_t device);
+    GGML_API const char *                  ggml_backend_dev_description(ggml_backend_dev_t device);
+    GGML_API void                          ggml_backend_dev_memory(ggml_backend_dev_t device, size_t * free, size_t * total);
+    GGML_API enum ggml_backend_dev_type    ggml_backend_dev_type(ggml_backend_dev_t device);
+    GGML_API void                          ggml_backend_dev_get_props(ggml_backend_dev_t device, struct ggml_backend_dev_props * props);
+    GGML_API ggml_backend_reg_t            ggml_backend_dev_backend_reg(ggml_backend_dev_t device);
+    GGML_API ggml_backend_t                ggml_backend_dev_init(ggml_backend_dev_t device, const char * params);
+    GGML_API ggml_backend_buffer_type_t    ggml_backend_dev_buffer_type(ggml_backend_dev_t device);
+    GGML_API ggml_backend_buffer_type_t    ggml_backend_dev_host_buffer_type(ggml_backend_dev_t device);
+    GGML_API ggml_backend_buffer_t         ggml_backend_dev_buffer_from_host_ptr(ggml_backend_dev_t device, void * ptr, size_t size, size_t max_tensor_size);
+
+    GGML_API bool                          ggml_backend_dev_supports_op(ggml_backend_dev_t device, const struct ggml_tensor * op);
+    GGML_API bool                          ggml_backend_dev_supports_buft(ggml_backend_dev_t device, ggml_backend_buffer_type_t buft);
+    GGML_API bool                          ggml_backend_dev_offload_op(ggml_backend_dev_t device, const struct ggml_tensor * op);
+
+    //
+    // Backend (reg)
+    //
+
+    GGML_API const char *       ggml_backend_reg_name(ggml_backend_reg_t reg);
+    GGML_API size_t             ggml_backend_reg_dev_count(ggml_backend_reg_t reg);
+    GGML_API ggml_backend_dev_t ggml_backend_reg_dev_get(ggml_backend_reg_t reg, size_t index);
+    GGML_API void *             ggml_backend_reg_get_proc_address(ggml_backend_reg_t reg, const char * name);
+
+    // Common functions that may be obtained using ggml_backend_reg_get_proc_address
+
+    // Split buffer type for tensor parallelism
+    typedef ggml_backend_buffer_type_t   (*ggml_backend_split_buffer_type_t)(int main_device, const float * tensor_split);
+    // Set the number of threads for the backend
+    typedef void                         (*ggml_backend_set_n_threads_t)(ggml_backend_t backend, int n_threads);
+    // Get additional buffer types provided by the device (returns a NULL-terminated array)
+    typedef ggml_backend_buffer_type_t * (*ggml_backend_dev_get_extra_bufts_t)(ggml_backend_dev_t device);
+    // Set the abort callback for the backend
+    typedef void                         (*ggml_backend_set_abort_callback_t)(ggml_backend_t backend, ggml_abort_callback abort_callback, void * abort_callback_data);
+    // Get a list of feature flags supported by the backend (returns a NULL-terminated array)
+    struct ggml_backend_feature {
+        const char * name;
+        const char * value;
+    };
+    typedef struct ggml_backend_feature * (*ggml_backend_get_features_t)(ggml_backend_reg_t reg);
+
+    //
+    // Backend registry
+    //
+
+    GGML_API void ggml_backend_device_register(ggml_backend_dev_t device);
+
+    // Backend (reg) enumeration
+    GGML_API size_t             ggml_backend_reg_count(void);
+    GGML_API ggml_backend_reg_t ggml_backend_reg_get(size_t index);
+    GGML_API ggml_backend_reg_t ggml_backend_reg_by_name(const char * name);
+
+    // Device enumeration
+    GGML_API size_t             ggml_backend_dev_count(void);
+    GGML_API ggml_backend_dev_t ggml_backend_dev_get(size_t index);
+    GGML_API ggml_backend_dev_t ggml_backend_dev_by_name(const char * name);
+    GGML_API ggml_backend_dev_t ggml_backend_dev_by_type(enum ggml_backend_dev_type type);
+
+    // Direct backend (stream) initialization
+    // = ggml_backend_dev_init(ggml_backend_dev_by_name(name), params)
+    GGML_API ggml_backend_t ggml_backend_init_by_name(const char * name, const char * params);
+    // = ggml_backend_dev_init(ggml_backend_dev_by_type(type), params)
+    GGML_API ggml_backend_t ggml_backend_init_by_type(enum ggml_backend_dev_type type, const char * params);
+    // = ggml_backend_dev_init(ggml_backend_dev_by_type(GPU) OR ggml_backend_dev_by_type(CPU), NULL)
+    GGML_API ggml_backend_t ggml_backend_init_best(void);
+
+    // Load a backend from a dynamic library and register it
+    GGML_API ggml_backend_reg_t ggml_backend_load(const char * path);
+    // Unload a backend if loaded dynamically and unregister it
+    GGML_API void               ggml_backend_unload(ggml_backend_reg_t reg);
+    // Load all known backends from dynamic libraries
+    GGML_API void               ggml_backend_load_all(void);
+    GGML_API void               ggml_backend_load_all_from_path(const char * dir_path);
+
+    //
+    // Backend scheduler
+    //
+
+    // The backend scheduler allows for multiple backend devices to be used together
+    // Handles compute buffer allocation, assignment of tensors to backends, and copying of tensors between backends
+    // The backends are selected based on:
+    // - the backend that supports the operation
+    // - the location of the pre-allocated tensors (e.g. the weights)
+    /*
+      Example usage:
+
+        // operations that use tensors allocated in a buffer with USAGE_WEIGHTS will be assigned
+        // preferrably to run on the same backend as the buffer
+        ggml_backend_buffer_set_usage(buf_weights, GGML_BACKEND_BUFFER_USAGE_WEIGHTS);
+
+        sched = ggml_backend_sched_new({backend_gpu, backend_gpu2, backend_cpu}, NULL, num_backends, GGML_DEFAULT_GRAPH_SIZE, false, true);
+
+        // initialize buffers from a max size graph (optional)
+        reserve_graph = build_graph(sched, max_batch_size);
+
+        // manually assign nodes to a backend (optional, should not be needed in most cases)
+        struct ggml_tensor * node = ggml_mul_mat(ctx, ...);
+        ggml_backend_sched_set_tensor_backend(sched, node, backend_gpu);
+
+        ggml_backend_sched_reserve(sched, reserve_graph);
+
+        // compute
+        graph = build_graph(sched); // the graph and its tensors are single-use in terms of allocation, multi-use in terms of computation
+        for (int i = 0; i < 10; ++i) {
+            ggml_backend_sched_graph_compute(sched, graph); // on the first iteration the graph is allocated automatically
+        }
+
+        // if there are graph inputs:
+        graph = build_graph(sched); // get a new graph that is not allocated (the metadata for the old graph is freed once ggml_free is called)
+        ggml_backend_sched_reset(sched); // clear the allocation of the previous graph
+        ggml_backend_sched_alloc_graph(sched, graph); // explicitly allocate the new graph but do not execute it
+        ggml_backend_tensor_set(input_tensor, ...); // copy data to the newly allocated graph tensors
+        ggml_backend_sched_graph_compute(sched, graph); // execute the graph
+
+        // as an alternative to the above it is also possible to assign the inputs to a dedicated context and
+        // allocate them statically via ggml_backend_alloc_ctx_tensors
+    }
+    */
+
+    typedef struct ggml_backend_sched * ggml_backend_sched_t;
+
+    // Evaluation callback for each node in the graph (set with ggml_backend_sched_set_eval_callback)
+    // when ask == true, the scheduler wants to know if the user wants to observe this node
+    // this allows the scheduler to batch nodes together in order to evaluate them in a single call
+    //
+    // when ask == false, the scheduler is passing the node tensor to the user for observation
+    // if the user returns false, the scheduler will cancel the graph compute
+    //
+    typedef bool (*ggml_backend_sched_eval_callback)(struct ggml_tensor * t, bool ask, void * user_data);
+
+    // Initialize a backend scheduler, backends with low index are given priority over backends with high index
+    GGML_API ggml_backend_sched_t ggml_backend_sched_new(ggml_backend_t * backends, ggml_backend_buffer_type_t * bufts, int n_backends, size_t graph_size, bool parallel, bool op_offload);
+    GGML_API void                 ggml_backend_sched_free(ggml_backend_sched_t sched);
+
+    // Initialize backend buffers from a measure graph
+    GGML_API bool                 ggml_backend_sched_reserve(ggml_backend_sched_t sched, struct ggml_cgraph * measure_graph); // returns success
+
+    GGML_API int                  ggml_backend_sched_get_n_backends(ggml_backend_sched_t sched);
+    GGML_API ggml_backend_t       ggml_backend_sched_get_backend(ggml_backend_sched_t sched, int i);
+
+    // Get the number of splits of the last graph
+    GGML_API int                  ggml_backend_sched_get_n_splits(ggml_backend_sched_t sched);
+    GGML_API int                  ggml_backend_sched_get_n_copies(ggml_backend_sched_t sched);
+
+    GGML_API size_t               ggml_backend_sched_get_buffer_size(ggml_backend_sched_t sched, ggml_backend_t backend);
+
+    GGML_API void                 ggml_backend_sched_set_tensor_backend(ggml_backend_sched_t sched, struct ggml_tensor * node, ggml_backend_t backend);
+    GGML_API ggml_backend_t       ggml_backend_sched_get_tensor_backend(ggml_backend_sched_t sched, struct ggml_tensor * node);
+
+    // Allocate and compute graph on the backend scheduler
+    GGML_API bool                 ggml_backend_sched_alloc_graph(ggml_backend_sched_t sched, struct ggml_cgraph * graph); // returns success
+    GGML_API enum ggml_status     ggml_backend_sched_graph_compute(ggml_backend_sched_t sched, struct ggml_cgraph * graph);
+    GGML_API enum ggml_status     ggml_backend_sched_graph_compute_async(ggml_backend_sched_t sched, struct ggml_cgraph * graph);
+    GGML_API void                 ggml_backend_sched_synchronize(ggml_backend_sched_t sched);
+
+    // Reset all assignments and allocators - must be called before changing the node backends or allocating a new graph.
+    // This in effect deallocates all tensors that were previously allocated and leaves them with dangling pointers.
+    // The correct way to use this API is to discard the deallocated tensors and create new ones.
+    GGML_API void                 ggml_backend_sched_reset(ggml_backend_sched_t sched);
+
+    // Set a callback to be called for each resulting node during graph compute
+    GGML_API void                 ggml_backend_sched_set_eval_callback(ggml_backend_sched_t sched, ggml_backend_sched_eval_callback callback, void * user_data);
+
+    //
+    // Utils
+    //
+
+    struct ggml_backend_graph_copy {
+        ggml_backend_buffer_t buffer;
+        struct ggml_context * ctx_allocated;
+        struct ggml_context * ctx_unallocated;
+        struct ggml_cgraph * graph;
+    };
+
+    // Copy a graph to a different backend
+    GGML_API struct ggml_backend_graph_copy ggml_backend_graph_copy(ggml_backend_t backend, struct ggml_cgraph * graph);
+    GGML_API void                           ggml_backend_graph_copy_free(struct ggml_backend_graph_copy copy);
+
+    typedef bool (*ggml_backend_eval_callback)(int node_index, struct ggml_tensor * t1, struct ggml_tensor * t2, void * user_data);
+
+    // Compare the output of two backends
+    GGML_API bool ggml_backend_compare_graph_backend(ggml_backend_t backend1, ggml_backend_t backend2, struct ggml_cgraph * graph, ggml_backend_eval_callback callback, void * user_data, struct ggml_tensor * test_node);
+
+    // Tensor initialization
+    GGML_API enum ggml_status ggml_backend_tensor_alloc(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, void * addr);
+    GGML_API enum ggml_status ggml_backend_view_init(struct ggml_tensor * tensor);
+
+    // CPU buffer types are always available
+    GGML_API ggml_backend_buffer_t      ggml_backend_cpu_buffer_from_ptr(void * ptr, size_t size);
+    GGML_API ggml_backend_buffer_type_t ggml_backend_cpu_buffer_type(void);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-blas.h

@@ -0,0 +1,25 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+// backend API
+GGML_BACKEND_API ggml_backend_t ggml_backend_blas_init(void);
+
+GGML_BACKEND_API bool ggml_backend_is_blas(ggml_backend_t backend);
+
+// number of threads used for conversion to float
+// for openblas and blis, this will also set the number of threads used for blas operations
+GGML_BACKEND_API void ggml_backend_blas_set_n_threads(ggml_backend_t backend_blas, int n_threads);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_blas_reg(void);
+
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-cann.h

@@ -0,0 +1,123 @@
+/*
+ * Copyright (c) 2023-2024 The ggml authors
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#pragma once
+
+#include "ggml-backend.h"
+#include "ggml.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @brief Maximum number of CANN devices supported.
+ */
+#define GGML_CANN_MAX_DEVICES 16
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_cann_reg(void);
+
+/**
+ * @brief Initializes the CANN backend for a specified device.
+ *
+ * This function initializes the CANN backend for the given device.
+ * It verifies the device index, allocates a context, and creates a backend
+ * instance.
+ *
+ * @param device The index of the device to initialize.
+ * @return A pointer to the initialized backend instance, or nullptr on failure.
+ */
+GGML_BACKEND_API ggml_backend_t ggml_backend_cann_init(int32_t device);
+
+/**
+ * @brief Checks if a given backend is a CANN backend.
+ *
+ * This function verifies if the provided backend is a CANN backend by comparing
+ * its GUID with the CANN backend's GUID.
+ *
+ * @param backend The backend instance to check.
+ * @return True if the backend is a CANN backend, false otherwise.
+ */
+GGML_BACKEND_API bool ggml_backend_is_cann(ggml_backend_t backend);
+
+/**
+ * @brief Retrieves the CANN buffer type for a specified device.
+ *
+ * This function initializes and returns the buffer type interface associated
+ * with the given device. It ensures thread-safe access using a mutex.
+ *
+ * @param device The device index for which to retrieve the buffer type.
+ * @return A pointer to the buffer type interface for the specified device, or
+ * nullptr if the device index is out of range.
+ */
+GGML_BACKEND_API ggml_backend_buffer_type_t
+ggml_backend_cann_buffer_type(int32_t device);
+
+/**
+ * @brief Retrieves the number of CANN devices available.
+ *
+ * This function returns the number of CANN devices available based on
+ * information obtained from `ggml_cann_info()`.
+ *
+ * @return The number of CANN devices available.
+ */
+GGML_BACKEND_API int32_t ggml_backend_cann_get_device_count(void);
+
+/**
+ * @brief pinned host buffer for use with the CPU backend for faster copies between CPU and NPU.
+ *
+ * @return A pointer to the host buffer type interface.
+ */
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cann_host_buffer_type(void);
+
+/**
+ * @brief Retrieves the description of a specific CANN device.
+ *
+ * This function sets the specified device, retrieves the SoC name,
+ * and writes it into the provided description buffer.
+ *
+ * @param device The device index to retrieve the description for.
+ * @param description Pointer to a buffer where the description will be written.
+ * @param description_size Size of the description buffer.
+ */
+GGML_BACKEND_API void ggml_backend_cann_get_device_description(
+    int32_t device, char* description, size_t description_size);
+
+/**
+ * @brief Retrieves the memory information of a specific CANN device.
+ *
+ * This function sets the specified device, retrieves the free and total
+ * memory information of the specified type (ACL_HBM_MEM), and stores them
+ * in the provided pointers.
+ *
+ * @param device The device index to retrieve memory information for.
+ * @param free Pointer to a variable where the free memory size will be stored.
+ * @param total Pointer to a variable where the total memory size will be
+ * stored.
+ */
+GGML_BACKEND_API void ggml_backend_cann_get_device_memory(int32_t device,
+                                                  size_t* free,
+                                                  size_t* total);
+
+#ifdef __cplusplus
+}
+#endif

ggml/include/ggml-cpp.h

@@ -0,0 +1,39 @@
+#pragma once
+
+#ifndef __cplusplus
+#error "This header is for C++ only"
+#endif
+
+#include "ggml.h"
+#include "ggml-alloc.h"
+#include "ggml-backend.h"
+#include "gguf.h"
+#include <memory>
+
+// Smart pointers for ggml types
+
+// ggml
+
+struct ggml_context_deleter { void operator()(ggml_context * ctx) { ggml_free(ctx); } };
+struct gguf_context_deleter { void operator()(gguf_context * ctx) { gguf_free(ctx); } };
+
+typedef std::unique_ptr<ggml_context, ggml_context_deleter> ggml_context_ptr;
+typedef std::unique_ptr<gguf_context, gguf_context_deleter> gguf_context_ptr;
+
+// ggml-alloc
+
+struct ggml_gallocr_deleter { void operator()(ggml_gallocr_t galloc) { ggml_gallocr_free(galloc); } };
+
+typedef std::unique_ptr<ggml_gallocr, ggml_gallocr_deleter> ggml_gallocr_ptr;
+
+// ggml-backend
+
+struct ggml_backend_deleter        { void operator()(ggml_backend_t backend)       { ggml_backend_free(backend); } };
+struct ggml_backend_buffer_deleter { void operator()(ggml_backend_buffer_t buffer) { ggml_backend_buffer_free(buffer); } };
+struct ggml_backend_event_deleter  { void operator()(ggml_backend_event_t event)   { ggml_backend_event_free(event); } };
+struct ggml_backend_sched_deleter  { void operator()(ggml_backend_sched_t sched)   { ggml_backend_sched_free(sched); } };
+
+typedef std::unique_ptr<ggml_backend,        ggml_backend_deleter>        ggml_backend_ptr;
+typedef std::unique_ptr<ggml_backend_buffer, ggml_backend_buffer_deleter> ggml_backend_buffer_ptr;
+typedef std::unique_ptr<ggml_backend_event,  ggml_backend_event_deleter>  ggml_backend_event_ptr;
+typedef std::unique_ptr<ggml_backend_sched,  ggml_backend_sched_deleter>  ggml_backend_sched_ptr;

ggml/include/ggml-cpu.h

@@ -0,0 +1,145 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+    // the compute plan that needs to be prepared for ggml_graph_compute()
+    // since https://github.com/ggml-org/ggml/issues/287
+    struct ggml_cplan {
+        size_t    work_size; // size of work buffer, calculated by `ggml_graph_plan()`
+        uint8_t * work_data; // work buffer, to be allocated by caller before calling to `ggml_graph_compute()`
+
+        int n_threads;
+        struct ggml_threadpool * threadpool;
+
+        // abort ggml_graph_compute when true
+        ggml_abort_callback abort_callback;
+        void *              abort_callback_data;
+    };
+
+    // numa strategies
+    enum ggml_numa_strategy {
+        GGML_NUMA_STRATEGY_DISABLED   = 0,
+        GGML_NUMA_STRATEGY_DISTRIBUTE = 1,
+        GGML_NUMA_STRATEGY_ISOLATE    = 2,
+        GGML_NUMA_STRATEGY_NUMACTL    = 3,
+        GGML_NUMA_STRATEGY_MIRROR     = 4,
+        GGML_NUMA_STRATEGY_COUNT
+    };
+
+    GGML_BACKEND_API void    ggml_numa_init(enum ggml_numa_strategy numa); // call once for better performance on NUMA systems
+    GGML_BACKEND_API bool    ggml_is_numa(void); // true if init detected that system has >1 NUMA node
+
+    GGML_BACKEND_API struct ggml_tensor * ggml_new_i32(struct ggml_context * ctx, int32_t value);
+    GGML_BACKEND_API struct ggml_tensor * ggml_new_f32(struct ggml_context * ctx, float value);
+
+    GGML_BACKEND_API struct ggml_tensor * ggml_set_i32 (struct ggml_tensor * tensor, int32_t value);
+    GGML_BACKEND_API struct ggml_tensor * ggml_set_f32 (struct ggml_tensor * tensor, float value);
+
+    GGML_BACKEND_API int32_t ggml_get_i32_1d(const struct ggml_tensor * tensor, int i);
+    GGML_BACKEND_API void    ggml_set_i32_1d(const struct ggml_tensor * tensor, int i, int32_t value);
+
+    GGML_BACKEND_API int32_t ggml_get_i32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3);
+    GGML_BACKEND_API void    ggml_set_i32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3, int32_t value);
+
+    GGML_BACKEND_API float   ggml_get_f32_1d(const struct ggml_tensor * tensor, int i);
+    GGML_BACKEND_API void    ggml_set_f32_1d(const struct ggml_tensor * tensor, int i, float value);
+
+    GGML_BACKEND_API float   ggml_get_f32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3);
+    GGML_BACKEND_API void    ggml_set_f32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3, float value);
+
+    GGML_BACKEND_API struct ggml_threadpool *      ggml_threadpool_new           (struct ggml_threadpool_params  * params);
+    GGML_BACKEND_API void                          ggml_threadpool_free          (struct ggml_threadpool * threadpool);
+    GGML_BACKEND_API int                           ggml_threadpool_get_n_threads (struct ggml_threadpool * threadpool);
+    GGML_BACKEND_API void                          ggml_threadpool_pause         (struct ggml_threadpool * threadpool);
+    GGML_BACKEND_API void                          ggml_threadpool_resume        (struct ggml_threadpool * threadpool);
+
+    // ggml_graph_plan() has to be called before ggml_graph_compute()
+    // when plan.work_size > 0, caller must allocate memory for plan.work_data
+    GGML_BACKEND_API struct ggml_cplan ggml_graph_plan(
+                  const struct ggml_cgraph * cgraph,
+                                       int   n_threads, /* = GGML_DEFAULT_N_THREADS */
+                    struct ggml_threadpool * threadpool /* = NULL */ );
+    GGML_BACKEND_API enum ggml_status  ggml_graph_compute(struct ggml_cgraph * cgraph, struct ggml_cplan * cplan);
+
+    // same as ggml_graph_compute() but the work data is allocated as a part of the context
+    // note: the drawback of this API is that you must have ensured that the context has enough memory for the work data
+    GGML_BACKEND_API enum ggml_status  ggml_graph_compute_with_ctx(struct ggml_context * ctx, struct ggml_cgraph * cgraph, int n_threads);
+
+    //
+    // system info
+    //
+
+    // x86
+    GGML_BACKEND_API int ggml_cpu_has_sse3       (void);
+    GGML_BACKEND_API int ggml_cpu_has_ssse3      (void);
+    GGML_BACKEND_API int ggml_cpu_has_avx        (void);
+    GGML_BACKEND_API int ggml_cpu_has_avx_vnni   (void);
+    GGML_BACKEND_API int ggml_cpu_has_avx2       (void);
+    GGML_BACKEND_API int ggml_cpu_has_bmi2       (void);
+    GGML_BACKEND_API int ggml_cpu_has_f16c       (void);
+    GGML_BACKEND_API int ggml_cpu_has_fma        (void);
+    GGML_BACKEND_API int ggml_cpu_has_avx512     (void);
+    GGML_BACKEND_API int ggml_cpu_has_avx512_vbmi(void);
+    GGML_BACKEND_API int ggml_cpu_has_avx512_vnni(void);
+    GGML_BACKEND_API int ggml_cpu_has_avx512_bf16(void);
+    GGML_BACKEND_API int ggml_cpu_has_amx_int8   (void);
+    // ARM
+    GGML_BACKEND_API int ggml_cpu_has_neon       (void);
+    GGML_BACKEND_API int ggml_cpu_has_arm_fma    (void);
+    GGML_BACKEND_API int ggml_cpu_has_fp16_va    (void);
+    GGML_BACKEND_API int ggml_cpu_has_dotprod    (void);
+    GGML_BACKEND_API int ggml_cpu_has_matmul_int8(void);
+    GGML_BACKEND_API int ggml_cpu_has_sve        (void);
+    GGML_BACKEND_API int ggml_cpu_get_sve_cnt    (void);  // sve vector length in bytes
+    GGML_BACKEND_API int ggml_cpu_has_sme        (void);
+    // other
+    GGML_BACKEND_API int ggml_cpu_has_riscv_v    (void);
+    GGML_BACKEND_API int ggml_cpu_has_vsx        (void);
+    GGML_BACKEND_API int ggml_cpu_has_vxe        (void);
+    GGML_BACKEND_API int ggml_cpu_has_nnpa       (void);
+    GGML_BACKEND_API int ggml_cpu_has_wasm_simd  (void);
+    GGML_BACKEND_API int ggml_cpu_has_llamafile  (void);
+
+    // Internal types and functions exposed for tests and benchmarks
+
+    typedef void (*ggml_vec_dot_t)  (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT x, size_t bx,
+                                       const void * GGML_RESTRICT y, size_t by, int nrc);
+
+    struct ggml_type_traits_cpu {
+        ggml_from_float_t        from_float;
+        ggml_vec_dot_t           vec_dot;
+        enum ggml_type           vec_dot_type;
+        int64_t                  nrows; // number of rows to process simultaneously
+    };
+
+    GGML_BACKEND_API const struct ggml_type_traits_cpu * ggml_get_type_traits_cpu(enum ggml_type type);
+
+    GGML_BACKEND_API void ggml_cpu_init(void);
+
+    //
+    // CPU backend
+    //
+
+    GGML_BACKEND_API ggml_backend_t ggml_backend_cpu_init(void);
+
+    GGML_BACKEND_API bool ggml_backend_is_cpu                (ggml_backend_t backend);
+    GGML_BACKEND_API void ggml_backend_cpu_set_n_threads     (ggml_backend_t backend_cpu, int n_threads);
+    GGML_BACKEND_API void ggml_backend_cpu_set_threadpool    (ggml_backend_t backend_cpu, ggml_threadpool_t threadpool);
+    GGML_BACKEND_API void ggml_backend_cpu_set_abort_callback(ggml_backend_t backend_cpu, ggml_abort_callback abort_callback, void * abort_callback_data);
+
+    GGML_BACKEND_API ggml_backend_reg_t ggml_backend_cpu_reg(void);
+
+    GGML_BACKEND_API void ggml_cpu_fp32_to_fp32(const float *,       float *, int64_t);
+    GGML_BACKEND_API void ggml_cpu_fp32_to_fp16(const float *, ggml_fp16_t *, int64_t);
+    GGML_BACKEND_API void ggml_cpu_fp16_to_fp32(const ggml_fp16_t *, float *, int64_t);
+    GGML_BACKEND_API void ggml_cpu_fp32_to_bf16(const float *, ggml_bf16_t *, int64_t);
+    GGML_BACKEND_API void ggml_cpu_bf16_to_fp32(const ggml_bf16_t *, float *, int64_t);
+
+#ifdef __cplusplus
+}
+#endif

ggml/include/ggml-cuda.h

@@ -0,0 +1,47 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+#ifdef GGML_USE_HIP
+#define GGML_CUDA_NAME "ROCm"
+#define GGML_CUBLAS_NAME "hipBLAS"
+#elif defined(GGML_USE_MUSA)
+#define GGML_CUDA_NAME "MUSA"
+#define GGML_CUBLAS_NAME "muBLAS"
+#else
+#define GGML_CUDA_NAME "CUDA"
+#define GGML_CUBLAS_NAME "cuBLAS"
+#endif
+#define GGML_CUDA_MAX_DEVICES       16
+
+// backend API
+GGML_BACKEND_API ggml_backend_t ggml_backend_cuda_init(int device);
+
+GGML_BACKEND_API bool ggml_backend_is_cuda(ggml_backend_t backend);
+
+// device buffer
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cuda_buffer_type(int device);
+
+// split tensor buffer that splits matrices by rows across multiple devices
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split);
+
+// pinned host buffer for use with the CPU backend for faster copies between CPU and GPU
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_cuda_host_buffer_type(void);
+
+GGML_BACKEND_API int  ggml_backend_cuda_get_device_count(void);
+GGML_BACKEND_API void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size);
+GGML_BACKEND_API void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total);
+
+GGML_BACKEND_API bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size);
+GGML_BACKEND_API void ggml_backend_cuda_unregister_host_buffer(void * buffer);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_cuda_reg(void);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-metal.h

@@ -0,0 +1,66 @@
+// Note: this description is outdated
+//
+// An interface allowing to compute ggml_cgraph with Metal
+//
+// This is a fully functional interface that extends ggml with GPU support for Apple devices.
+// A similar interface can be created for other GPU backends (e.g. Vulkan, CUDA, etc.)
+//
+// How it works?
+//
+// As long as your program can create and evaluate a ggml_cgraph on the CPU, you can use this
+// interface to evaluate the same graph on the GPU. Instead of using ggml_graph_compute(), you
+// use ggml_metal_graph_compute() (or ggml_vulkan_graph_compute(), etc.)
+//
+// You only need to make sure that all memory buffers that you used during the graph creation
+// are mapped to the device memory with the ggml_metal_add_buffer() function. This mapping is
+// used during the graph evaluation to determine the arguments of the compute kernels.
+//
+// Synchronization between device and host memory (for example for input and output tensors)
+// is done with the ggml_metal_set_tensor() and ggml_metal_get_tensor() functions.
+//
+
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#include <stddef.h>
+#include <stdbool.h>
+
+struct ggml_tensor;
+struct ggml_cgraph;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+//
+// backend API
+// user-code should use only these functions
+//
+
+GGML_BACKEND_API ggml_backend_t ggml_backend_metal_init(void);
+
+GGML_BACKEND_API bool ggml_backend_is_metal(ggml_backend_t backend);
+
+GGML_DEPRECATED(
+        GGML_BACKEND_API ggml_backend_buffer_t ggml_backend_metal_buffer_from_ptr(void * data, size_t size, size_t max_size),
+        "obsoleted by the new device interface - https://github.com/ggml-org/llama.cpp/pull/9713");
+
+GGML_BACKEND_API void ggml_backend_metal_set_abort_callback(ggml_backend_t backend, ggml_abort_callback abort_callback, void * user_data);
+
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_metal_buffer_type(void);
+
+// helper to check if the device supports a specific family
+// ideally, the user code should be doing these checks
+// ref: https://developer.apple.com/metal/Metal-Feature-Set-Tables.pdf
+GGML_BACKEND_API bool ggml_backend_metal_supports_family(ggml_backend_t backend, int family);
+
+// capture all command buffers committed the next time `ggml_backend_graph_compute` is called
+GGML_BACKEND_API void ggml_backend_metal_capture_next_compute(ggml_backend_t backend);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_metal_reg(void);
+
+#ifdef __cplusplus
+}
+#endif

ggml/include/ggml-opencl.h

@@ -0,0 +1,26 @@
+#ifndef GGML_OPENCL_H
+#define GGML_OPENCL_H
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+//
+// backend API
+//
+GGML_BACKEND_API ggml_backend_t ggml_backend_opencl_init(void);
+GGML_BACKEND_API bool ggml_backend_is_opencl(ggml_backend_t backend);
+
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_opencl_buffer_type(void);
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_opencl_host_buffer_type(void);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_opencl_reg(void);
+
+#ifdef  __cplusplus
+}
+#endif
+
+#endif // GGML_OPENCL_H

ggml/include/ggml-opt.h

@@ -0,0 +1,237 @@
+// This file contains functionality for training models using GGML.
+// It is not strictly needed vs. just vanilla GGML but it provides a more high-level interface for common needs such as datasets.
+// At the bottom of this file especially there are relatively high-level functions that are suitable use or adaptation in user code.
+//
+// Module maintainer: Johannes Gäßler (@JohannesGaessler, johannesg@5d6.de)
+
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#include <stdint.h>
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+    struct ggml_opt_dataset;
+    struct ggml_opt_context;
+    struct ggml_opt_result;
+
+    typedef struct ggml_opt_dataset * ggml_opt_dataset_t;
+    typedef struct ggml_opt_context * ggml_opt_context_t;
+    typedef struct ggml_opt_result  * ggml_opt_result_t;
+
+    // ====== Loss ======
+
+    // built-in loss types, i.e. the built-in quantities minimized by the optimizer
+    // custom loss types can be defined via mean or sum which simply reduce the outputs for all datapoints to a single value
+    enum ggml_opt_loss_type {
+        GGML_OPT_LOSS_TYPE_MEAN,
+        GGML_OPT_LOSS_TYPE_SUM,
+        GGML_OPT_LOSS_TYPE_CROSS_ENTROPY,
+        GGML_OPT_LOSS_TYPE_MEAN_SQUARED_ERROR,
+    };
+
+    // ====== Dataset ======
+
+    GGML_API ggml_opt_dataset_t ggml_opt_dataset_init(
+            enum ggml_type type_data,    // the type for the internal data tensor
+            enum ggml_type type_label,   // the type for the internal labels tensor
+            int64_t        ne_datapoint, // number of elements per datapoint
+            int64_t        ne_label,     // number of elements per label
+            int64_t        ndata,        // total number of datapoints/labels
+            int64_t        ndata_shard); // number of datapoints/labels per shard (unit at which the dataset is shuffled/copied)
+    GGML_API void ggml_opt_dataset_free(ggml_opt_dataset_t dataset);
+
+    // get underlying tensors that store the data
+    GGML_API int64_t              ggml_opt_dataset_ndata (ggml_opt_dataset_t dataset);
+    GGML_API struct ggml_tensor * ggml_opt_dataset_data  (ggml_opt_dataset_t dataset); // shape = [ne_datapoint, ndata]
+    GGML_API struct ggml_tensor * ggml_opt_dataset_labels(ggml_opt_dataset_t dataset); // shape = [nd_label,     ndata]
+
+    // shuffle idata first datapoints from dataset with RNG from opt_ctx, shuffle all datapoints if idata is negative
+    GGML_API void ggml_opt_dataset_shuffle(ggml_opt_context_t opt_ctx, ggml_opt_dataset_t dataset, int64_t idata);
+
+    // get batch at position ibatch from dataset and copy the data to data_batch and labels_batch
+    GGML_API void ggml_opt_dataset_get_batch(
+            ggml_opt_dataset_t   dataset,
+            struct ggml_tensor * data_batch,   // shape = [ne_datapoint, ndata_batch]
+            struct ggml_tensor * labels_batch, // shape = [ne_label,     ndata_batch]
+            int64_t              ibatch);
+    GGML_API void ggml_opt_dataset_get_batch_host(
+            ggml_opt_dataset_t   dataset,
+            void               * data_batch,
+            size_t               nb_data_batch,
+            void               * labels_batch,
+            int64_t              ibatch);
+
+    // ====== Model / Context ======
+
+    enum ggml_opt_build_type {
+        GGML_OPT_BUILD_TYPE_FORWARD = 10,
+        GGML_OPT_BUILD_TYPE_GRAD    = 20,
+        GGML_OPT_BUILD_TYPE_OPT     = 30,
+    };
+
+    // parameters that control which optimizer is used and how said optimizer tries to find the minimal loss
+    struct ggml_opt_optimizer_params {
+        // AdamW optimizer parameters
+        struct {
+            float alpha; // learning rate
+            float beta1;
+            float beta2;
+            float eps;   // epsilon for numerical stability
+            float wd;    // weight decay for AdamW, use 0.0f to disable
+        } adamw;
+    };
+
+    // callback to calculate optimizer parameters prior to a backward pass
+    // userdata can be used to pass arbitrary data
+    typedef struct ggml_opt_optimizer_params (*ggml_opt_get_optimizer_params)(void * userdata);
+
+    // returns the default optimizer params (constant, hard-coded values)
+    // userdata is not used
+    GGML_API struct ggml_opt_optimizer_params ggml_opt_get_default_optimizer_params(void * userdata);
+
+    // casts userdata to ggml_opt_optimizer_params and returns it
+    GGML_API struct ggml_opt_optimizer_params ggml_opt_get_constant_optimizer_params(void * userdata);
+
+    // parameters for initializing a new optimization context
+    struct ggml_opt_params {
+        ggml_backend_sched_t backend_sched; // defines which backends are used to construct the compute graphs
+
+        // by default the forward graph needs to be reconstructed for each eval
+        // if ctx_compute, inputs, and outputs are set the graphs are instead allocated statically
+        struct ggml_context * ctx_compute;
+        struct ggml_tensor  * inputs;
+        struct ggml_tensor  * outputs;
+
+        enum ggml_opt_loss_type  loss_type;
+        enum ggml_opt_build_type build_type;
+
+        int32_t opt_period; // after how many gradient accumulation steps an optimizer step should be done
+
+        ggml_opt_get_optimizer_params get_opt_pars; // callback for calculating optimizer parameters
+        void * get_opt_pars_ud;                     // userdata for calculating optimizer parameters
+    };
+
+    // get parameters for an optimization context with defaults set where possible
+    // parameters for which no sensible defaults exist are supplied as arguments to this function
+    GGML_API struct ggml_opt_params ggml_opt_default_params(
+            ggml_backend_sched_t    backend_sched,
+            enum ggml_opt_loss_type loss_type);
+
+    GGML_API ggml_opt_context_t ggml_opt_init(struct ggml_opt_params params);
+    GGML_API void ggml_opt_free(ggml_opt_context_t opt_ctx);
+
+    // set gradients to zero, initilize loss, and optionally reset the optimizer
+    GGML_API void ggml_opt_reset(ggml_opt_context_t opt_ctx, bool optimizer);
+
+    GGML_API bool ggml_opt_static_graphs(ggml_opt_context_t opt_ctx); // whether the graphs are allocated_statically
+
+    // get underlying tensors that store data
+    // if not using static graphs these pointers become invalid with the next call to ggml_opt_alloc
+    GGML_API struct ggml_tensor * ggml_opt_inputs(  ggml_opt_context_t opt_ctx); // forward graph input tensor
+    GGML_API struct ggml_tensor * ggml_opt_outputs( ggml_opt_context_t opt_ctx); // forward graph output tensor
+    GGML_API struct ggml_tensor * ggml_opt_labels(  ggml_opt_context_t opt_ctx); // labels to compare outputs against
+    GGML_API struct ggml_tensor * ggml_opt_loss(    ggml_opt_context_t opt_ctx); // scalar tensor that contains the loss
+    GGML_API struct ggml_tensor * ggml_opt_pred(    ggml_opt_context_t opt_ctx); // predictions made by outputs
+    GGML_API struct ggml_tensor * ggml_opt_ncorrect(ggml_opt_context_t opt_ctx); // number of matching predictions between outputs and labels
+
+    // get the gradient accumulator for a node from the forward graph
+    GGML_API struct ggml_tensor * ggml_opt_grad_acc(ggml_opt_context_t opt_ctx, struct ggml_tensor * node);
+
+    // ====== Optimization Result ======
+
+    GGML_API ggml_opt_result_t ggml_opt_result_init(void);
+    GGML_API void ggml_opt_result_free(ggml_opt_result_t result);
+    GGML_API void ggml_opt_result_reset(ggml_opt_result_t result);
+
+    // get data from result, uncertainties are optional and can be ignored by passing NULL
+    GGML_API void ggml_opt_result_ndata(   ggml_opt_result_t result, int64_t * ndata);                  // writes 1 value, number of datapoints
+    GGML_API void ggml_opt_result_loss(    ggml_opt_result_t result, double  * loss,     double * unc); // writes 1 value
+    GGML_API void ggml_opt_result_pred(    ggml_opt_result_t result, int32_t * pred);                   // writes ndata values
+    GGML_API void ggml_opt_result_accuracy(ggml_opt_result_t result, double  * accuracy, double * unc); // writes 1 value
+
+    // ====== Computation ======
+
+    // if not using static graphs, this function must be called prior to ggml_opt_alloc
+    GGML_API void ggml_opt_prepare_alloc(
+        ggml_opt_context_t    opt_ctx,
+        struct ggml_context * ctx_compute,
+        struct ggml_cgraph  * gf,
+        struct ggml_tensor  * inputs,
+        struct ggml_tensor  * outputs);
+
+    // allocate the next graph for evaluation, either forward or forward + backward
+    // must be called exactly once prior to calling ggml_opt_eval
+    GGML_API void ggml_opt_alloc(ggml_opt_context_t opt_ctx, bool backward);
+
+    // do forward pass, increment result if not NULL, do backward pass if allocated
+    GGML_API void ggml_opt_eval(ggml_opt_context_t opt_ctx, ggml_opt_result_t result);
+
+    // ############################################################################
+    // ## The high-level functions start here. They do not depend on any private ##
+    // ## functions or structs and can be copied to and adapted for user code.   ##
+    // ############################################################################
+
+    // ====== Intended Usage ======
+    //
+    // 1. Select the appropriate loss for your problem.
+    // 2. Create a dataset and set the data for the "data" tensor. Also set the "labels" tensor if your loss needs them.
+    //    Setting the shard size to 1 will be fine, it's the granularity with which data is shuffled/loaded (bigger values are faster).
+    // 3. Create a GGML graph for your model with no_alloc == true. Use two separate contexts for the tensors.
+    //    The first context should contain the model parameters and inputs and be allocated statically in user code.
+    //    The second context should contain all other tensors and will be (re)allocated automatically.
+    //    Due to this automated allocation the data of the second context is not defined when accessed in user code.
+    //    Note that the second dimension of the inputs/outputs are interpreted as the number of datapoints in those tensors.
+    // 4. Call ggml_opt_fit. If you need more control you can use ggml_opt_epoch instead.
+
+    // signature for a callback while evaluating opt_ctx on dataset, called after an evaluation
+    typedef void (*ggml_opt_epoch_callback)(
+            bool               train,       // true after training evaluation, false after validation evaluation
+            ggml_opt_context_t opt_ctx,
+            ggml_opt_dataset_t dataset,
+            ggml_opt_result_t  result,      // result associated with the dataset subsection
+            int64_t            ibatch,      // number of batches that have been evaluated so far
+            int64_t            ibatch_max,  // total number of batches in this dataset subsection
+            int64_t            t_start_us); // time at which the evaluation on the dataset subsection was started
+
+    // do training on front of dataset, do evaluation only on back of dataset
+    GGML_API void ggml_opt_epoch(
+            ggml_opt_context_t      opt_ctx,
+            ggml_opt_dataset_t      dataset,
+            ggml_opt_result_t       result_train,   // result to increment during training, ignored if NULL
+            ggml_opt_result_t       result_eval,    // result to increment during evaluation, ignored if NULL
+            int64_t                 idata_split,    // data index at which to split training and evaluation
+            ggml_opt_epoch_callback callback_train,
+            ggml_opt_epoch_callback callback_eval);
+
+    // callback that prints a progress bar on stderr
+    GGML_API void ggml_opt_epoch_callback_progress_bar(
+            bool               train,
+            ggml_opt_context_t opt_ctx,
+            ggml_opt_dataset_t dataset,
+            ggml_opt_result_t  result,
+            int64_t            ibatch,
+            int64_t            ibatch_max,
+            int64_t            t_start_us);
+
+    // fit model defined by inputs and outputs to dataset
+    GGML_API void ggml_opt_fit(
+            ggml_backend_sched_t            backend_sched,  // backend scheduler for constructing the compute graphs
+            struct ggml_context           * ctx_compute,    // context with temporarily allocated tensors to calculate the outputs
+            struct ggml_tensor            * inputs,         // input tensor with shape [ne_datapoint, ndata_batch]
+            struct ggml_tensor            * outputs,        // output tensor, must have shape [ne_label, ndata_batch] if labels are used
+            ggml_opt_dataset_t              dataset,        // dataset with data and optionally also labels
+            enum ggml_opt_loss_type         loss_type,      // loss to minimize
+            ggml_opt_get_optimizer_params   get_opt_pars,   // callback to get optimizer params, userdata is pointer to epoch (of type int64_t)
+            int64_t                         nepoch,         // how many times the dataset should be iterated over
+            int64_t                         nbatch_logical, // datapoints optimizer step, must be a multiple of ndata_batch in inputs/outputs
+            float                           val_split,      // fraction of the dataset to use for validation, must be in [0.0f, 1.0f)
+            bool                            silent);        // whether or not info prints to stderr should be suppressed
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-rpc.h

@@ -0,0 +1,33 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+#define RPC_PROTO_MAJOR_VERSION    2
+#define RPC_PROTO_MINOR_VERSION    0
+#define RPC_PROTO_PATCH_VERSION    0
+#define GGML_RPC_MAX_SERVERS       16
+
+// backend API
+GGML_BACKEND_API ggml_backend_t ggml_backend_rpc_init(const char * endpoint);
+GGML_BACKEND_API bool ggml_backend_is_rpc(ggml_backend_t backend);
+
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_rpc_buffer_type(const char * endpoint);
+
+GGML_BACKEND_API void ggml_backend_rpc_get_device_memory(const char * endpoint, size_t * free, size_t * total);
+
+GGML_BACKEND_API void ggml_backend_rpc_start_server(ggml_backend_t backend, const char * endpoint,
+                                                    const char * cache_dir,
+                                                    size_t free_mem, size_t total_mem);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_rpc_reg(void);
+
+GGML_BACKEND_API ggml_backend_dev_t ggml_backend_rpc_add_device(const char * endpoint);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-sycl.h

@@ -0,0 +1,49 @@
+//
+//  MIT license
+//  Copyright (C) 2024 Intel Corporation
+//  SPDX-License-Identifier: MIT
+//
+
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#define GGML_SYCL_NAME "SYCL"
+#define GGML_SYCL_MAX_DEVICES 48
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+// backend API
+GGML_BACKEND_API ggml_backend_t ggml_backend_sycl_init(int device);
+
+GGML_BACKEND_API bool ggml_backend_is_sycl(ggml_backend_t backend);
+
+// devide buffer
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_sycl_buffer_type(int device);
+
+// split tensor buffer that splits matrices by rows across multiple devices
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_sycl_split_buffer_type(const float * tensor_split);
+
+// pinned host buffer for use with the CPU backend for faster copies between CPU and GPU
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_sycl_host_buffer_type(void);
+
+GGML_BACKEND_API void ggml_backend_sycl_print_sycl_devices(void);
+GGML_BACKEND_API void ggml_backend_sycl_get_gpu_list(int *id_list, int max_len);
+GGML_BACKEND_API void ggml_backend_sycl_get_device_description(int device,
+                                                       char *description,
+                                                       size_t description_size);
+GGML_BACKEND_API int  ggml_backend_sycl_get_device_count();
+GGML_BACKEND_API void ggml_backend_sycl_get_device_memory(int device, size_t *free, size_t *total);
+
+// SYCL doesn't support registering host memory, keep here for reference
+// GGML_BACKEND_API bool ggml_backend_sycl_register_host_buffer(void * buffer, size_t size);
+// GGML_BACKEND_API void ggml_backend_sycl_unregister_host_buffer(void * buffer);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_sycl_reg(void);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-vulkan.h

@@ -0,0 +1,29 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+#define GGML_VK_NAME "Vulkan"
+#define GGML_VK_MAX_DEVICES 16
+
+// backend API
+GGML_BACKEND_API ggml_backend_t ggml_backend_vk_init(size_t dev_num);
+
+GGML_BACKEND_API bool ggml_backend_is_vk(ggml_backend_t backend);
+GGML_BACKEND_API int  ggml_backend_vk_get_device_count(void);
+GGML_BACKEND_API void ggml_backend_vk_get_device_description(int device, char * description, size_t description_size);
+GGML_BACKEND_API void ggml_backend_vk_get_device_memory(int device, size_t * free, size_t * total);
+
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_vk_buffer_type(size_t dev_num);
+// pinned host buffer for use with the CPU backend for faster copies between CPU and GPU
+GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_vk_host_buffer_type(void);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_vk_reg(void);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/ggml-webgpu.h

@@ -0,0 +1,19 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+#define GGML_WEBGPU_NAME "WebGPU"
+
+// Needed for examples in ggml
+GGML_BACKEND_API ggml_backend_t ggml_backend_webgpu_init(void);
+
+GGML_BACKEND_API ggml_backend_reg_t ggml_backend_webgpu_reg(void);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/include/gguf.h

@@ -0,0 +1,202 @@
+// This file contains functionality related to "GGUF" files, the binary file format used by ggml.
+// GGUF files have the following structure:
+//
+// 1. File magic "GGUF" (4 bytes).
+// 2. File version (uint32_t).
+// 3. Number of ggml tensors in file (int64_t).
+// 4. Number of key-value-pairs in file (int64_t).
+// 5. For each KV pair:
+//   1. The key (string).
+//   2. The value type (gguf_type).
+//   3a. If the value type is GGUF_TYPE_ARRAY:
+//     1. The type of the array (gguf_type).
+//     2. The number of elements in the array (uint64_t).
+//     3. The binary representation of each element in the array.
+//   3b. Otherwise:
+//     1. The binary representation of the value.
+// 6. For each ggml tensor:
+//   1. The tensor name (string).
+//   2. The number of dimensions of the tensor (uint32_t).
+//   3. For each dimension:
+//     1. The size of the tensor in the dimension (int64_t).
+//   4. The tensor data type (ggml_type).
+//   5. The tensor data offset in the tensor data binary blob (uint64_t).
+// 7. The tensor data binary blob (optional, aligned).
+//
+// Strings are serialized as the string length (uint64_t) followed by the C string without the null terminator.
+// All enums are stored as int32_t.
+// All bool values are stored as int8_t.
+// If the special key "general.alignment" (uint32_t) is defined it is used for alignment,
+//   otherwise GGUF_DEFAULT_ALIGNMENT is used.
+//
+// Module maintainer: Johannes Gäßler (@JohannesGaessler, johannesg@5d6.de)
+
+#pragma once
+
+#include "ggml.h"
+
+#include <stdbool.h>
+#include <stdint.h>
+
+#define GGUF_MAGIC   "GGUF"
+#define GGUF_VERSION 3
+
+#define GGUF_KEY_GENERAL_ALIGNMENT "general.alignment"
+
+#define GGUF_DEFAULT_ALIGNMENT 32
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+    // types that can be stored as GGUF KV data
+    enum gguf_type {
+        GGUF_TYPE_UINT8   = 0,
+        GGUF_TYPE_INT8    = 1,
+        GGUF_TYPE_UINT16  = 2,
+        GGUF_TYPE_INT16   = 3,
+        GGUF_TYPE_UINT32  = 4,
+        GGUF_TYPE_INT32   = 5,
+        GGUF_TYPE_FLOAT32 = 6,
+        GGUF_TYPE_BOOL    = 7,
+        GGUF_TYPE_STRING  = 8,
+        GGUF_TYPE_ARRAY   = 9,
+        GGUF_TYPE_UINT64  = 10,
+        GGUF_TYPE_INT64   = 11,
+        GGUF_TYPE_FLOAT64 = 12,
+        GGUF_TYPE_COUNT,       // marks the end of the enum
+    };
+
+    struct gguf_context;
+
+    struct gguf_init_params {
+        bool no_alloc;
+
+        // if not NULL, create a ggml_context and allocate the tensor data in it
+        struct ggml_context ** ctx;
+    };
+
+    GGML_API struct gguf_context * gguf_init_empty(void);
+    GGML_API struct gguf_context * gguf_init_from_file(const char * fname, struct gguf_init_params params);
+    //GGML_API struct gguf_context * gguf_init_from_buffer(..);
+
+    GGML_API void gguf_free(struct gguf_context * ctx);
+
+    GGML_API const char * gguf_type_name(enum gguf_type type);
+
+    GGML_API uint32_t gguf_get_version    (const struct gguf_context * ctx);
+    GGML_API size_t   gguf_get_alignment  (const struct gguf_context * ctx);
+    GGML_API size_t   gguf_get_data_offset(const struct gguf_context * ctx);
+
+    GGML_API int64_t      gguf_get_n_kv(const struct gguf_context * ctx);
+    GGML_API int64_t      gguf_find_key(const struct gguf_context * ctx, const char * key); // returns -1 if key is not found
+    GGML_API const char * gguf_get_key (const struct gguf_context * ctx, int64_t key_id);
+
+    GGML_API enum gguf_type gguf_get_kv_type (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API enum gguf_type gguf_get_arr_type(const struct gguf_context * ctx, int64_t key_id);
+
+    // will abort if the wrong type is used for the key
+    GGML_API uint8_t      gguf_get_val_u8  (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API int8_t       gguf_get_val_i8  (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API uint16_t     gguf_get_val_u16 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API int16_t      gguf_get_val_i16 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API uint32_t     gguf_get_val_u32 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API int32_t      gguf_get_val_i32 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API float        gguf_get_val_f32 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API uint64_t     gguf_get_val_u64 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API int64_t      gguf_get_val_i64 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API double       gguf_get_val_f64 (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API bool         gguf_get_val_bool(const struct gguf_context * ctx, int64_t key_id);
+    GGML_API const char * gguf_get_val_str (const struct gguf_context * ctx, int64_t key_id);
+    GGML_API const void * gguf_get_val_data(const struct gguf_context * ctx, int64_t key_id);
+    GGML_API size_t       gguf_get_arr_n   (const struct gguf_context * ctx, int64_t key_id);
+
+    // get raw pointer to the first element of the array with the given key_id
+    // for bool arrays, note that they are always stored as int8 on all platforms (usually this makes no difference)
+    GGML_API const void * gguf_get_arr_data(const struct gguf_context * ctx, int64_t key_id);
+
+    // get ith C string from array with given key_id
+    GGML_API const char * gguf_get_arr_str (const struct gguf_context * ctx, int64_t key_id, size_t i);
+
+    GGML_API int64_t        gguf_get_n_tensors    (const struct gguf_context * ctx);
+    GGML_API int64_t        gguf_find_tensor      (const struct gguf_context * ctx, const char * name); // returns -1 if the tensor is not found
+    GGML_API size_t         gguf_get_tensor_offset(const struct gguf_context * ctx, int64_t tensor_id);
+    GGML_API const char *   gguf_get_tensor_name  (const struct gguf_context * ctx, int64_t tensor_id);
+    GGML_API enum ggml_type gguf_get_tensor_type  (const struct gguf_context * ctx, int64_t tensor_id);
+    GGML_API size_t         gguf_get_tensor_size  (const struct gguf_context * ctx, int64_t tensor_id);
+
+    // removes key if it exists, returns id that the key had prior to removal (-1 if it didn't exist)
+    GGML_API int64_t gguf_remove_key(struct gguf_context * ctx, const char * key);
+
+    // overrides an existing KV pair or adds a new one, the new KV pair is always at the back
+    GGML_API void gguf_set_val_u8  (struct gguf_context * ctx, const char * key, uint8_t      val);
+    GGML_API void gguf_set_val_i8  (struct gguf_context * ctx, const char * key, int8_t       val);
+    GGML_API void gguf_set_val_u16 (struct gguf_context * ctx, const char * key, uint16_t     val);
+    GGML_API void gguf_set_val_i16 (struct gguf_context * ctx, const char * key, int16_t      val);
+    GGML_API void gguf_set_val_u32 (struct gguf_context * ctx, const char * key, uint32_t     val);
+    GGML_API void gguf_set_val_i32 (struct gguf_context * ctx, const char * key, int32_t      val);
+    GGML_API void gguf_set_val_f32 (struct gguf_context * ctx, const char * key, float        val);
+    GGML_API void gguf_set_val_u64 (struct gguf_context * ctx, const char * key, uint64_t     val);
+    GGML_API void gguf_set_val_i64 (struct gguf_context * ctx, const char * key, int64_t      val);
+    GGML_API void gguf_set_val_f64 (struct gguf_context * ctx, const char * key, double       val);
+    GGML_API void gguf_set_val_bool(struct gguf_context * ctx, const char * key, bool         val);
+    GGML_API void gguf_set_val_str (struct gguf_context * ctx, const char * key, const char * val);
+
+    // creates a new array with n elements of the given type and copies the corresponding number of bytes from data
+    GGML_API void gguf_set_arr_data(struct gguf_context * ctx, const char * key, enum gguf_type type, const void * data, size_t n);
+
+    // creates a new array with n strings and copies the corresponding strings from data
+    GGML_API void gguf_set_arr_str (struct gguf_context * ctx, const char * key, const char ** data, size_t n);
+
+    // set or add KV pairs from another context
+    GGML_API void gguf_set_kv(struct gguf_context * ctx, const struct gguf_context * src);
+
+    // add tensor to GGUF context, tensor name must be unique
+    GGML_API void gguf_add_tensor(struct gguf_context * ctx, const struct ggml_tensor * tensor);
+
+    // after changing a tensor's type, the offsets of all tensors with higher indices are immediately recalculated
+    //   in such a way that the tensor data remains as one contiguous block (except for padding)
+    GGML_API void gguf_set_tensor_type(struct gguf_context * ctx, const char * name, enum ggml_type type);
+
+    // assumes that at least gguf_get_tensor_size bytes can be read from data
+    GGML_API void gguf_set_tensor_data(struct gguf_context * ctx, const char * name, const void * data);
+
+    // writing gguf files can be done in 3 ways:
+    //
+    // - write the entire gguf_context to a binary file in a single pass:
+    //
+    //   gguf_write_to_file(ctx, fname, /*only_meta =*/ false);
+    //
+    // - write only the meta data to a file, then re-open the file and append the tensor data:
+    //
+    //   gguf_write_to_file(ctx, fname, /*only_meta =*/ true);
+    //   FILE * f = fopen(fname, "ab");
+    //   fwrite(f, ...); // write tensor data
+    //   fclose(f);
+    //
+    // - first prepare a file with a placeholder for the meta data, write the tensor data, then write the meta data:
+    //
+    //   FILE * f = fopen(fname, "wb");
+    //   const size_t size_meta = gguf_get_meta_size(ctx);
+    //   fseek(f, size_meta, SEEK_SET);
+    //   fwrite(f, ...); // write tensor data
+    //   void * data = malloc(size_meta);
+    //   gguf_get_meta_data(ctx, data);
+    //   rewind(f);
+    //   fwrite(data, 1, data, f);
+    //   free(data);
+    //   fclose(f);
+    //
+
+    // write the entire context to a binary file
+    GGML_API bool gguf_write_to_file(const struct gguf_context * ctx, const char * fname, bool only_meta);
+
+    // get the size in bytes of the meta data (header, kv pairs, tensor info) including padding
+    GGML_API size_t gguf_get_meta_size(const struct gguf_context * ctx);
+
+    // writes the meta data to pointer "data"
+    GGML_API void   gguf_get_meta_data(const struct gguf_context * ctx, void * data);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/src/CMakeLists.txt

@@ -0,0 +1,404 @@
+include(CheckCXXCompilerFlag)
+include("../cmake/common.cmake")
+
+add_compile_definitions(GGML_SCHED_MAX_COPIES=${GGML_SCHED_MAX_COPIES})
+
+# enable libstdc++ assertions for debug builds
+if (CMAKE_SYSTEM_NAME MATCHES "Linux")
+    add_compile_definitions($<$<CONFIG:Debug>:_GLIBCXX_ASSERTIONS>)
+endif()
+
+if (NOT MSVC)
+    if (GGML_SANITIZE_THREAD)
+        add_compile_options(-fsanitize=thread)
+        link_libraries     (-fsanitize=thread)
+    endif()
+
+    if (GGML_SANITIZE_ADDRESS)
+        add_compile_options(-fsanitize=address -fno-omit-frame-pointer)
+        link_libraries     (-fsanitize=address)
+    endif()
+
+    if (GGML_SANITIZE_UNDEFINED)
+        add_compile_options(-fsanitize=undefined)
+        link_libraries     (-fsanitize=undefined)
+    endif()
+endif()
+
+if (GGML_FATAL_WARNINGS)
+    if (CMAKE_CXX_COMPILER_ID MATCHES "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+        list(APPEND C_FLAGS   -Werror)
+        list(APPEND CXX_FLAGS -Werror)
+    elseif (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
+        add_compile_options(/WX)
+    endif()
+endif()
+
+if (GGML_ALL_WARNINGS)
+    if (NOT MSVC)
+        list(APPEND WARNING_FLAGS -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function)
+        list(APPEND C_FLAGS       -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes
+                                  -Werror=implicit-int -Werror=implicit-function-declaration)
+        list(APPEND CXX_FLAGS     -Wmissing-declarations -Wmissing-noreturn)
+
+        list(APPEND C_FLAGS   ${WARNING_FLAGS})
+        list(APPEND CXX_FLAGS ${WARNING_FLAGS})
+
+        ggml_get_flags(${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION})
+
+        add_compile_options("$<$<COMPILE_LANGUAGE:C>:${C_FLAGS};${GF_C_FLAGS}>"
+                            "$<$<COMPILE_LANGUAGE:CXX>:${CXX_FLAGS};${GF_CXX_FLAGS}>")
+    else()
+        # todo : msvc
+        set(C_FLAGS   "")
+        set(CXX_FLAGS "")
+    endif()
+endif()
+
+if (GGML_LTO)
+    include(CheckIPOSupported)
+    check_ipo_supported(RESULT result OUTPUT output)
+    if (result)
+        set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE)
+    else()
+        message(WARNING "IPO is not supported: ${output}")
+    endif()
+endif()
+
+if (GGML_CCACHE AND NOT CMAKE_C_COMPILER_LAUNCHER AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
+    find_program(GGML_CCACHE_FOUND ccache)
+    find_program(GGML_SCCACHE_FOUND sccache)
+
+    if (GGML_CCACHE_FOUND OR GGML_SCCACHE_FOUND)
+        if(GGML_CCACHE_FOUND)
+            set(GGML_CCACHE_VARIANT ccache)
+        else()
+            set(GGML_CCACHE_VARIANT sccache)
+        endif()
+        # TODO: should not be set globally
+        if (GGML_SYCL AND GGML_CCACHE_FOUND AND WIN32)
+            set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "ccache compiler_type=icl")
+        else ()
+            set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "${GGML_CCACHE_VARIANT}")
+        endif ()
+        set(ENV{CCACHE_SLOPPINESS} time_macros)
+        message(STATUS "${GGML_CCACHE_VARIANT} found, compilation results will be cached. Disable with GGML_CCACHE=OFF.")
+    else()
+        message(STATUS "Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF")
+    endif ()
+endif()
+
+# this version of Apple ld64 is buggy
+execute_process(
+    COMMAND ${CMAKE_C_COMPILER} ${CMAKE_EXE_LINKER_FLAGS} -Wl,-v
+    ERROR_VARIABLE output
+    OUTPUT_QUIET
+)
+
+if (output MATCHES "dyld-1015\.7")
+    add_compile_definitions(HAVE_BUGGY_APPLE_LINKER)
+endif()
+
+# architecture specific
+# TODO: probably these flags need to be tweaked on some architectures
+#       feel free to update the Makefile for your architecture and send a pull request or issue
+message(STATUS "CMAKE_SYSTEM_PROCESSOR: ${CMAKE_SYSTEM_PROCESSOR}")
+if (MSVC)
+    string(TOLOWER "${CMAKE_GENERATOR_PLATFORM}" CMAKE_GENERATOR_PLATFORM_LWR)
+    message(STATUS "CMAKE_GENERATOR_PLATFORM: ${CMAKE_GENERATOR_PLATFORM}")
+else ()
+    set(CMAKE_GENERATOR_PLATFORM_LWR "")
+endif ()
+ggml_get_system_arch()
+message(STATUS "GGML_SYSTEM_ARCH: ${GGML_SYSTEM_ARCH}")
+
+if (NOT MSVC)
+    if (GGML_STATIC)
+        add_link_options(-static)
+        if (MINGW)
+            add_link_options(-static-libgcc -static-libstdc++)
+        endif()
+    endif()
+    if (GGML_GPROF)
+        add_compile_options(-pg)
+    endif()
+endif()
+
+if (MINGW)
+    add_compile_definitions(_WIN32_WINNT=${GGML_WIN_VER})
+endif()
+
+#
+# POSIX conformance
+#
+
+# clock_gettime came in POSIX.1b (1993)
+# CLOCK_MONOTONIC came in POSIX.1-2001 / SUSv3 as optional
+# posix_memalign came in POSIX.1-2001 / SUSv3
+# M_PI is an XSI extension since POSIX.1-2001 / SUSv3, came in XPG1 (1985)
+
+# Somehow in OpenBSD whenever POSIX conformance is specified
+# some string functions rely on locale_t availability,
+# which was introduced in POSIX.1-2008, forcing us to go higher
+if (CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
+    add_compile_definitions(_XOPEN_SOURCE=700)
+else()
+    add_compile_definitions(_XOPEN_SOURCE=600)
+endif()
+
+# Data types, macros and functions related to controlling CPU affinity and
+# some memory allocation are available on Linux through GNU extensions in libc
+if (CMAKE_SYSTEM_NAME MATCHES "Linux" OR CMAKE_SYSTEM_NAME MATCHES "Android")
+    add_compile_definitions(_GNU_SOURCE)
+endif()
+
+# RLIMIT_MEMLOCK came in BSD, is not specified in POSIX.1,
+# and on macOS its availability depends on enabling Darwin extensions
+# similarly on DragonFly, enabling BSD extensions is necessary
+if (
+    CMAKE_SYSTEM_NAME MATCHES "Darwin" OR
+    CMAKE_SYSTEM_NAME MATCHES "iOS"    OR
+    CMAKE_SYSTEM_NAME MATCHES "tvOS"   OR
+    CMAKE_SYSTEM_NAME MATCHES "DragonFly"
+)
+    add_compile_definitions(_DARWIN_C_SOURCE)
+endif()
+
+# alloca is a non-standard interface that is not visible on BSDs when
+# POSIX conformance is specified, but not all of them provide a clean way
+# to enable it in such cases
+if (CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
+    add_compile_definitions(__BSD_VISIBLE)
+endif()
+if (CMAKE_SYSTEM_NAME MATCHES "NetBSD")
+    add_compile_definitions(_NETBSD_SOURCE)
+endif()
+if (CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
+    add_compile_definitions(_BSD_SOURCE)
+endif()
+
+if (WIN32)
+    add_compile_definitions(_CRT_SECURE_NO_WARNINGS)
+endif()
+
+# ggml
+
+if (GGML_BACKEND_DL AND NOT BUILD_SHARED_LIBS)
+    message(FATAL_ERROR "GGML_BACKEND_DL requires BUILD_SHARED_LIBS")
+endif()
+
+add_library(ggml-base
+            ../include/ggml.h
+            ../include/ggml-alloc.h
+            ../include/ggml-backend.h
+            ../include/ggml-cpp.h
+            ../include/ggml-opt.h
+            ../include/gguf.h
+            ggml.c
+            ggml.cpp
+            ggml-alloc.c
+            ggml-backend.cpp
+            ggml-opt.cpp
+            ggml-threading.cpp
+            ggml-threading.h
+            ggml-quants.c
+            ggml-quants.h
+            gguf.cpp)
+
+target_include_directories(ggml-base PRIVATE .)
+if (GGML_BACKEND_DL)
+    target_compile_definitions(ggml-base PUBLIC GGML_BACKEND_DL)
+endif()
+
+add_library(ggml
+            ggml-backend-reg.cpp)
+add_library(ggml::ggml ALIAS ggml)
+
+target_link_libraries(ggml PUBLIC ggml-base)
+
+if (CMAKE_SYSTEM_NAME MATCHES "Linux")
+    target_link_libraries(ggml PRIVATE dl)
+endif()
+
+function(ggml_add_backend_library backend)
+    if (GGML_BACKEND_DL)
+        add_library(${backend} MODULE ${ARGN})
+        # write the shared library to the output directory
+        set_target_properties(${backend} PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})
+        target_compile_definitions(${backend} PRIVATE GGML_BACKEND_DL)
+        add_dependencies(ggml ${backend})
+        install(TARGETS ${backend} LIBRARY DESTINATION ${CMAKE_INSTALL_BINDIR})
+    else()
+        add_library(${backend} ${ARGN})
+        target_link_libraries(ggml PUBLIC ${backend})
+        install(TARGETS ${backend} LIBRARY)
+    endif()
+
+    target_link_libraries(${backend} PRIVATE ggml-base)
+    target_include_directories(${backend} PRIVATE ..)
+
+    if (${BUILD_SHARED_LIBS})
+        target_compile_definitions(${backend} PRIVATE GGML_BACKEND_BUILD)
+        target_compile_definitions(${backend} PUBLIC  GGML_BACKEND_SHARED)
+    endif()
+
+    if(NOT GGML_AVAILABLE_BACKENDS)
+        set(GGML_AVAILABLE_BACKENDS "${backend}"
+            CACHE INTERNAL "List of backends for cmake package")
+    else()
+        list(FIND GGML_AVAILABLE_BACKENDS "${backend}" has_backend)
+        if(has_backend EQUAL -1)
+            set(GGML_AVAILABLE_BACKENDS "${GGML_AVAILABLE_BACKENDS};${backend}"
+                CACHE INTERNAL "List of backends for cmake package")
+        endif()
+    endif()
+endfunction()
+
+function(ggml_add_backend backend)
+    string(TOUPPER "GGML_${backend}" backend_id)
+    if (${backend_id})
+        string(TOLOWER "ggml-${backend}" backend_target)
+        add_subdirectory(${backend_target})
+        message(STATUS "Including ${backend} backend")
+        if (NOT GGML_BACKEND_DL)
+            string(TOUPPER "GGML_USE_${backend}" backend_use)
+            target_compile_definitions(ggml PUBLIC ${backend_use})
+        endif()
+    endif()
+endfunction()
+
+function(ggml_add_cpu_backend_variant tag_name)
+    set(GGML_CPU_TAG_NAME ${tag_name})
+    # other: OPENMP LLAMAFILE CPU_HBM
+    if (GGML_SYSTEM_ARCH STREQUAL "x86")
+        foreach (feat NATIVE
+                      SSE42
+                      AVX AVX2 BMI2 AVX_VNNI FMA F16C
+                      AVX512 AVX512_VBMI AVX512_VNNI AVX512_BF16
+                      AMX_TILE AMX_INT8 AMX_BF16)
+            set(GGML_${feat} OFF)
+        endforeach()
+
+        foreach (feat ${ARGN})
+            set(GGML_${feat} ON)
+        endforeach()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "ARM")
+        foreach (feat ${ARGN})
+            set(GGML_INTERNAL_${feat} ON)
+        endforeach()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
+        foreach (feat ${ARGN})
+            set(GGML_INTERNAL_${feat} ON)
+        endforeach()
+    endif()
+
+    ggml_add_cpu_backend_variant_impl(${tag_name})
+endfunction()
+
+ggml_add_backend(CPU)
+
+if (GGML_CPU_ALL_VARIANTS)
+    if (NOT GGML_BACKEND_DL)
+        message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS requires GGML_BACKEND_DL")
+    elseif (GGML_CPU_ARM_ARCH)
+        message(FATAL_ERROR "Cannot use both GGML_CPU_ARM_ARCH and GGML_CPU_ALL_VARIANTS")
+    endif()
+    if (GGML_SYSTEM_ARCH STREQUAL "x86")
+        ggml_add_cpu_backend_variant(x64)
+        ggml_add_cpu_backend_variant(sse42        SSE42)
+        ggml_add_cpu_backend_variant(sandybridge  SSE42 AVX)
+        ggml_add_cpu_backend_variant(haswell      SSE42 AVX F16C AVX2 BMI2 FMA)
+        ggml_add_cpu_backend_variant(skylakex     SSE42 AVX F16C AVX2 BMI2 FMA AVX512)
+        ggml_add_cpu_backend_variant(icelake      SSE42 AVX F16C AVX2 BMI2 FMA AVX512 AVX512_VBMI AVX512_VNNI)
+        ggml_add_cpu_backend_variant(alderlake    SSE42 AVX F16C AVX2 BMI2 FMA AVX_VNNI)
+        if (NOT MSVC)
+            # MSVC doesn't support AMX
+            ggml_add_cpu_backend_variant(sapphirerapids SSE42 AVX F16C AVX2 BMI2 FMA AVX512 AVX512_VBMI AVX512_VNNI AVX512_BF16 AMX_TILE AMX_INT8)
+        endif()
+    elseif(GGML_SYSTEM_ARCH STREQUAL "ARM")
+        if (CMAKE_SYSTEM_NAME MATCHES "Linux")
+            # Many of these features are optional so we build versions with popular
+            # combinations and name the backends based on the version they were
+            # first released with
+            ggml_add_cpu_backend_variant(armv8.0_1)
+            ggml_add_cpu_backend_variant(armv8.2_1    DOTPROD)
+            ggml_add_cpu_backend_variant(armv8.2_2    DOTPROD FP16_VECTOR_ARITHMETIC)
+            ggml_add_cpu_backend_variant(armv8.2_3    DOTPROD FP16_VECTOR_ARITHMETIC SVE)
+            ggml_add_cpu_backend_variant(armv8.6_1    DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8)
+            ggml_add_cpu_backend_variant(armv8.6_2    DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8 SVE2)
+            ggml_add_cpu_backend_variant(armv9.2_1    DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8 SME)
+            ggml_add_cpu_backend_variant(armv9.2_2    DOTPROD FP16_VECTOR_ARITHMETIC SVE MATMUL_INT8 SVE2 SME)
+        elseif (CMAKE_SYSTEM_NAME MATCHES "Android")
+            # Android-specific backends with SoC-compatible feature sets
+            ggml_add_cpu_backend_variant(android_armv8.0_1)
+            ggml_add_cpu_backend_variant(android_armv8.2_1    DOTPROD)
+            ggml_add_cpu_backend_variant(android_armv8.2_2    DOTPROD FP16_VECTOR_ARITHMETIC)
+            ggml_add_cpu_backend_variant(android_armv8.6_1    DOTPROD FP16_VECTOR_ARITHMETIC MATMUL_INT8)
+        elseif (APPLE)
+            ggml_add_cpu_backend_variant(apple_m1             DOTPROD)
+            ggml_add_cpu_backend_variant(apple_m2_m3          DOTPROD MATMUL_INT8)
+            ggml_add_cpu_backend_variant(apple_m4             DOTPROD MATMUL_INT8 NOSVE SME)
+        else()
+            message(FATAL_ERROR "Unsupported ARM target OS: ${CMAKE_SYSTEM_NAME}")
+        endif()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
+        if (CMAKE_SYSTEM_NAME MATCHES "Linux")
+            ggml_add_cpu_backend_variant(power0)
+            ggml_add_cpu_backend_variant(power7_1       POWER7)
+            ggml_add_cpu_backend_variant(power7_2       POWER7  VSX)
+            ggml_add_cpu_backend_variant(power8_1       POWER8)
+            ggml_add_cpu_backend_variant(power8_2       POWER8  VSX)
+            ggml_add_cpu_backend_variant(power9         POWER9  VSX)
+            ggml_add_cpu_backend_variant(power10        POWER10 VSX)
+            ggml_add_cpu_backend_variant(power11        POWER11 VSX)
+        else()
+            message(FATAL_ERROR "Unsupported PowerPC target OS: ${CMAKE_SYSTEM_NAME}")
+        endif()
+    else()
+        message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS not yet supported with ${GGML_SYSTEM_ARCH} on ${CMAKE_SYSTEM_NAME}")
+    endif()
+elseif (GGML_CPU)
+    ggml_add_cpu_backend_variant_impl("")
+endif()
+
+ggml_add_backend(BLAS)
+ggml_add_backend(CANN)
+ggml_add_backend(CUDA)
+ggml_add_backend(HIP)
+ggml_add_backend(METAL)
+ggml_add_backend(MUSA)
+ggml_add_backend(RPC)
+ggml_add_backend(SYCL)
+ggml_add_backend(Vulkan)
+ggml_add_backend(WebGPU)
+ggml_add_backend(OpenCL)
+
+foreach (target ggml-base ggml)
+    target_include_directories(${target} PUBLIC    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/../include> $<INSTALL_INTERFACE:include>)
+    target_compile_features   (${target} PRIVATE c_std_11 cxx_std_17) # don't bump
+endforeach()
+
+target_link_libraries(ggml-base PRIVATE Threads::Threads)
+
+find_library(MATH_LIBRARY m)
+if (MATH_LIBRARY)
+    if (NOT WIN32 OR NOT DEFINED ENV{ONEAPI_ROOT})
+        target_link_libraries(ggml-base PRIVATE m)
+    endif()
+endif()
+
+if (CMAKE_SYSTEM_NAME MATCHES "Android")
+    target_link_libraries(ggml-base PRIVATE dl)
+endif()
+
+if(CMAKE_SYSTEM_NAME MATCHES "visionOS")
+    target_compile_definitions(ggml-base PUBLIC _DARWIN_C_SOURCE)
+endif()
+
+if (BUILD_SHARED_LIBS)
+    foreach (target ggml-base ggml)
+        set_target_properties(${target} PROPERTIES POSITION_INDEPENDENT_CODE ON)
+        target_compile_definitions(${target} PRIVATE GGML_BUILD)
+        target_compile_definitions(${target} PUBLIC  GGML_SHARED)
+    endforeach()
+endif()

ggml/src/ggml-amx/CMakeLists.txt

@@ -0,0 +1,107 @@
+if (CMAKE_OSX_ARCHITECTURES STREQUAL "x86_64" OR CMAKE_GENERATOR_PLATFORM_LWR MATCHES "^(x86_64|i686|amd64|x64|win32)$" OR
+        (NOT CMAKE_OSX_ARCHITECTURES AND NOT CMAKE_GENERATOR_PLATFORM_LWR AND
+         CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|i686|AMD64)$") AND
+        CMAKE_COMPILER_IS_GNUCC AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER 11.0)
+    message(STATUS "Using AMX")
+
+    file(GLOB   GGML_HEADERS_AMX "*.h")
+    list(APPEND GGML_HEADERS_AMX "../../include/ggml-amx.h")
+
+    file(GLOB   GGML_SOURCES_AMX "*.cpp")
+
+    add_library(ggml-amx
+                ${GGML_HEADERS_AMX}
+                ${GGML_SOURCES_AMX})
+
+    target_link_libraries(ggml-amx PRIVATE ggml-base)
+    target_include_directories(ggml-amx PRIVATE . ..)
+
+    # this is duplicated from the CPU backend, since the AMX backend also depends on the architecture flags
+    # TODO: integrate AMX backend into the CPU backend
+    if (MSVC)
+        # instruction set detection for MSVC only
+        if (GGML_NATIVE)
+            # TODO: improve, should not reference files from the parent folder
+            include(../ggml-cpu/cmake/FindSIMD.cmake)
+        endif ()
+        if (GGML_AVX512)
+            list(APPEND ARCH_FLAGS /arch:AVX512)
+            # MSVC has no compile-time flags enabling specific
+            # AVX512 extensions, neither it defines the
+            # macros corresponding to the extensions.
+            # Do it manually.
+            if (GGML_AVX512_VBMI)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:C>:__AVX512VBMI__>)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:CXX>:__AVX512VBMI__>)
+            endif()
+            if (GGML_AVX512_VNNI)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:C>:__AVX512VNNI__>)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:CXX>:__AVX512VNNI__>)
+            endif()
+            if (GGML_AVX512_BF16)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:C>:__AVX512BF16__>)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:CXX>:__AVX512BF16__>)
+            endif()
+            if (GGML_AMX_TILE)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:C>:__AMX_TILE__>)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:CXX>:__AMX_TILE__>)
+            endif()
+            if (GGML_AMX_INT8)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:C>:__AMX_INT8__>)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:CXX>:__AMX_INT8__>)
+            endif()
+            if (GGML_AMX_BF16)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:C>:__AMX_BF16__>)
+                add_compile_definitions($<$<COMPILE_LANGUAGE:CXX>:__AMX_BF16__>)
+            endif()
+        elseif (GGML_AVX2)
+            list(APPEND ARCH_FLAGS /arch:AVX2)
+        elseif (GGML_AVX)
+            list(APPEND ARCH_FLAGS /arch:AVX)
+        endif()
+    else()
+        if (GGML_NATIVE)
+            list(APPEND ARCH_FLAGS -march=native)
+        endif()
+        if (GGML_F16C)
+            list(APPEND ARCH_FLAGS -mf16c)
+        endif()
+        if (GGML_FMA)
+            list(APPEND ARCH_FLAGS -mfma)
+        endif()
+        if (GGML_AVX)
+            list(APPEND ARCH_FLAGS -mavx)
+        endif()
+        if (GGML_AVX2)
+            list(APPEND ARCH_FLAGS -mavx2)
+        endif()
+        if (GGML_AVX512)
+            list(APPEND ARCH_FLAGS -mavx512f)
+            list(APPEND ARCH_FLAGS -mavx512dq)
+            list(APPEND ARCH_FLAGS -mavx512bw)
+        endif()
+        if (GGML_AVX512_VBMI)
+            list(APPEND ARCH_FLAGS -mavx512vbmi)
+        endif()
+        if (GGML_AVX512_VNNI)
+            list(APPEND ARCH_FLAGS -mavx512vnni)
+        endif()
+        if (GGML_AVX512_BF16)
+            list(APPEND ARCH_FLAGS -mavx512bf16)
+        endif()
+        if (GGML_AMX_TILE)
+            list(APPEND ARCH_FLAGS -mamx-tile)
+        endif()
+        if (GGML_AMX_INT8)
+            list(APPEND ARCH_FLAGS -mamx-int8)
+        endif()
+        if (GGML_AMX_BF16)
+            list(APPEND ARCH_FLAGS -mamx-bf16)
+        endif()
+    endif()
+
+    target_compile_options(ggml-amx PRIVATE ${ARCH_FLAGS})
+else()
+    set(GGML_AMX OFF PARENT_SCOPE)
+    message(WARNING "AMX requires x86 and gcc version > 11.0. Turning off GGML_AMX.")
+endif()

ggml/src/ggml-amx/common.h

@@ -0,0 +1,94 @@
+#pragma once
+
+#include "ggml.h"
+// hack until AMX is moved into the CPU backend
+#include "../ggml-cpu/ggml-cpu-impl.h" // <immintrin.h>
+
+#include <algorithm>
+#include <memory>
+#include <type_traits>
+
+#if defined(_OPENMP)
+#include <omp.h>
+#endif
+
+#define TILE_M 16
+#define TILE_N 16
+#define TILE_K 32
+#define VNNI_BLK 4
+
+#define AMX_BLK_SIZE 32
+
+#define TMM0 0
+#define TMM1 1
+#define TMM2 2
+#define TMM3 3
+#define TMM4 4
+#define TMM5 5
+#define TMM6 6
+#define TMM7 7
+
+// parallel routines
+template <typename T, typename std::enable_if<std::is_integral<T>::value, int>::type = 0>
+inline T div_up(T x, T y) { return (x + y - 1) / y; }
+
+template <typename T>
+inline void balance211(T n, T nth, T ith, T& n_start, T& n_end) {
+#if 0
+    // onednn partition pattern
+    T& n_my = n_end;
+    if (nth <= 1 || n == 0) {
+        n_start = 0;
+        n_my = n;
+    } else {
+        T n1 = div_up(n, nth);
+        T n2 = n1 - 1;
+        T T1 = n - n2 * nth;
+        n_my = ith < T1 ? n1 : n2;
+        n_start = ith <= T1 ? ith*n1 : T1 * n1 + (ith - T1) * n2;
+    }
+    n_end += n_start;
+#else
+    // pytorch aten partition pattern
+    T n_my = div_up(n, nth);
+    n_start = ith * n_my;
+    n_end = std::min(n_start + n_my, n);
+#endif
+}
+
+template <typename func_t>
+inline void parallel_for(int nth, int n, const func_t& f) {
+#if defined(_OPENMP)
+#pragma omp parallel num_threads(nth)
+{
+    //int nth = omp_get_num_threads();
+    int ith = omp_get_thread_num();
+    int tbegin, tend;
+    balance211(n, nth, ith, tbegin, tend);
+    f(tbegin, tend);
+}
+#else
+    f(0, n);
+
+    GGML_UNUSED(nth);
+#endif
+}
+
+// quantized types that have AMX support
+inline bool qtype_has_amx_kernels(const enum ggml_type type) {
+    // TODO: fix padding for vnni format
+    return (type == GGML_TYPE_Q4_0) ||
+        (type == GGML_TYPE_Q4_1);
+        //(type == GGML_TYPE_Q8_0) ||
+        //(type == GGML_TYPE_Q4_K) ||
+        //(type == GGML_TYPE_Q5_K) ||
+        //(type == GGML_TYPE_Q6_K) ||
+        //(type == GGML_TYPE_IQ4_XS);
+}
+
+// ggml backend context
+struct ggml_backend_amx_context {
+    int n_threads = GGML_DEFAULT_N_THREADS;
+    std::unique_ptr<char[]> work_data;
+    size_t work_size = 0;
+};

ggml/src/ggml-amx/ggml-amx.cpp

@@ -0,0 +1,446 @@
+#include "ggml-amx.h"
+#include "ggml-amx/common.h"
+#include "ggml-amx/mmq.h"
+#include "ggml-backend-impl.h"
+#include "ggml-impl.h"
+
+#if defined(__gnu_linux__)
+#include <sys/syscall.h>
+#include <unistd.h>
+#endif
+
+#include <cstdlib>
+#include <cstring>
+#include <memory>
+
+#if defined(__AMX_INT8__)
+
+// AMX buffer interface
+static void ggml_backend_amx_buffer_free_buffer(ggml_backend_buffer_t buffer) {
+    free(buffer->context);
+}
+
+static void * ggml_backend_amx_buffer_get_base(ggml_backend_buffer_t buffer) {
+    return (void *)(buffer->context);
+}
+
+static void ggml_backend_amx_buffer_memset_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {
+    memset((char *)tensor->data + offset, value, size);
+
+    GGML_UNUSED(buffer);
+}
+
+static void ggml_backend_amx_buffer_set_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
+    if (qtype_has_amx_kernels(tensor->type)) {
+        ggml_backend_amx_convert_weight(tensor, data, offset, size);
+    } else {
+        memcpy((char *)tensor->data + offset, data, size);
+    }
+
+    GGML_UNUSED(buffer);
+}
+
+static void ggml_backend_amx_buffer_get_tensor(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, size_t offset, size_t size) {
+    GGML_ASSERT(!qtype_has_amx_kernels(tensor->type));
+    memcpy(data, (const char *)tensor->data + offset, size);
+
+    GGML_UNUSED(buffer);
+}
+
+static bool ggml_backend_amx_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const struct ggml_tensor * src, struct ggml_tensor * dst) {
+    if (ggml_backend_buffer_is_host(src->buffer)) {
+        if (qtype_has_amx_kernels(src->type)) {
+            ggml_backend_amx_convert_weight(dst, src->data, 0, ggml_backend_amx_get_alloc_size(dst));
+        } else {
+            memcpy(dst->data, src->data, ggml_nbytes(src));
+        }
+        return true;
+    }
+    return false;
+
+    GGML_UNUSED(buffer);
+}
+
+static void ggml_backend_amx_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
+    memset(buffer->context, value, buffer->size);
+}
+
+static ggml_backend_buffer_i ggml_backend_amx_buffer_interface = {
+    /* .free_buffer     = */ ggml_backend_amx_buffer_free_buffer,
+    /* .get_base        = */ ggml_backend_amx_buffer_get_base,
+    /* .init_tensor     = */ NULL, // no initialization required
+    /* .memset_tensor   = */ ggml_backend_amx_buffer_memset_tensor,
+    /* .set_tensor      = */ ggml_backend_amx_buffer_set_tensor,
+    /* .get_tensor      = */ ggml_backend_amx_buffer_get_tensor,
+    /* .cpy_tensor      = */ ggml_backend_amx_buffer_cpy_tensor,
+    /* .clear           = */ ggml_backend_amx_buffer_clear,
+    /* .reset           = */ NULL,
+};
+
+static const char * ggml_backend_amx_buffer_type_get_name(ggml_backend_buffer_type_t buft) {
+    return "AMX";
+
+    GGML_UNUSED(buft);
+}
+
+static ggml_backend_buffer_t ggml_backend_amx_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
+    void * data = aligned_alloc(TENSOR_ALIGNMENT, size);
+    if (data == NULL) {
+        fprintf(stderr, "%s: failed to allocate buffer of size %zu\n", __func__, size);
+        return NULL;
+    }
+
+    return ggml_backend_buffer_init(buft, ggml_backend_amx_buffer_interface, data, size);
+}
+
+static size_t ggml_backend_amx_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
+    return TENSOR_ALIGNMENT;
+
+    GGML_UNUSED(buft);
+}
+
+static size_t ggml_backend_amx_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor* tensor) {
+    return ggml_backend_amx_get_alloc_size(tensor);
+
+    GGML_UNUSED(buft);
+}
+
+static bool ggml_backend_amx_buffer_type_is_host(ggml_backend_buffer_type_t buft) {
+    return false;
+
+    GGML_UNUSED(buft);
+}
+
+ggml_backend_buffer_type_t ggml_backend_amx_buffer_type() {
+    static struct ggml_backend_buffer_type ggml_backend_buffer_type_amx = {
+        /* .iface = */ {
+            /* .get_name         = */ ggml_backend_amx_buffer_type_get_name,
+            /* .alloc_buffer     = */ ggml_backend_amx_buffer_type_alloc_buffer,
+            /* .get_alignment    = */ ggml_backend_amx_buffer_type_get_alignment,
+            /* .get_max_size     = */ NULL, // defaults to SIZE_MAX
+            /* .get_alloc_size   = */ ggml_backend_amx_buffer_type_get_alloc_size,
+            /* .is_host          = */ ggml_backend_amx_buffer_type_is_host,
+        },
+        /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_amx_reg(), 0),
+        /* .context = */ NULL,
+    };
+
+    return &ggml_backend_buffer_type_amx;
+}
+
+// backend interface
+
+static const char * ggml_backend_amx_name(ggml_backend_t backend) {
+    return "AMX";
+
+    GGML_UNUSED(backend);
+}
+
+static void ggml_backend_amx_free(ggml_backend_t backend) {
+    ggml_backend_amx_context * ctx = (ggml_backend_amx_context *)backend->context;
+    delete ctx;
+    delete backend;
+}
+
+static enum ggml_status ggml_backend_amx_graph_compute(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
+    ggml_backend_amx_context * ctx = (ggml_backend_amx_context *)backend->context;
+
+    for (int i = 0; i < cgraph->n_nodes; i++) {
+        struct ggml_tensor * node = cgraph->nodes[i];
+
+        switch (node->op) {
+        case GGML_OP_MUL_MAT:
+            ggml_backend_amx_mul_mat(ctx, node);
+            break;
+
+        case GGML_OP_NONE:
+        case GGML_OP_RESHAPE:
+        case GGML_OP_VIEW:
+        case GGML_OP_PERMUTE:
+        case GGML_OP_TRANSPOSE:
+            break;
+
+        default:
+            fprintf(stderr, "%s: unsupported op %s\n", __func__, ggml_op_desc(node));
+            GGML_ASSERT(false);
+        }
+    }
+
+    return GGML_STATUS_SUCCESS;
+
+    GGML_UNUSED(backend);
+}
+
+static struct ggml_backend_i ggml_backend_amx_i = {
+    /* .get_name                = */ ggml_backend_amx_name,
+    /* .free                    = */ ggml_backend_amx_free,
+    /* .set_tensor_async        = */ NULL,
+    /* .get_tensor_async        = */ NULL,
+    /* .cpy_tensor_async        = */ NULL,
+    /* .synchronize             = */ NULL,
+    /* .graph_plan_create       = */ NULL,
+    /* .graph_plan_free         = */ NULL,
+    /* .graph_plan_update       = */ NULL,
+    /* .graph_plan_compute      = */ NULL,
+    /* .graph_compute           = */ ggml_backend_amx_graph_compute,
+    /* .event_record            = */ NULL,
+    /* .event_wait              = */ NULL,
+};
+
+static ggml_guid_t ggml_backend_amx_guid() {
+    static ggml_guid guid = { 0x13, 0xb8, 0xa4, 0xc4, 0xba, 0xfe, 0x51, 0x67, 0x87, 0x44, 0x55, 0x15, 0xb2, 0x35, 0x62, 0x3e };
+    return &guid;
+}
+
+#define ARCH_GET_XCOMP_PERM     0x1022
+#define ARCH_REQ_XCOMP_PERM     0x1023
+#define XFEATURE_XTILECFG       17
+#define XFEATURE_XTILEDATA      18
+
+static bool ggml_amx_init() {
+#if defined(__gnu_linux__)
+    if (syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA)) {
+        fprintf(stderr, "AMX is not ready to be used!\n");
+        return false;
+    }
+    return true;
+#elif defined(_WIN32)
+    return true;
+#endif
+}
+
+ggml_backend_t ggml_backend_amx_init() {
+
+    // invoke a Linux system call to request access to AMX features
+    ggml_amx_init();
+
+    // backend context
+    ggml_backend_amx_context * ctx = new ggml_backend_amx_context;
+
+    // ggml amx backend
+    ggml_backend_t backend = new ggml_backend {
+        /* .guid      = */ ggml_backend_amx_guid(),
+        /* .interface = */ ggml_backend_amx_i,
+        /* .device    = */ ggml_backend_reg_dev_get(ggml_backend_amx_reg(), 0),
+        /* .context   = */ ctx,
+    };
+
+    return backend;
+}
+
+bool ggml_backend_is_amx(ggml_backend_t backend) {
+    return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_amx_guid());
+}
+
+void ggml_backend_amx_set_n_threads(ggml_backend_t backend_amx, int n_threads) {
+    GGML_ASSERT(ggml_backend_is_amx(backend_amx));
+
+    ggml_backend_amx_context * ctx = (ggml_backend_amx_context *)backend_amx->context;
+    ctx->n_threads = n_threads;
+}
+
+// device interface
+
+static const char * ggml_backend_amx_device_get_name(ggml_backend_dev_t dev) {
+    return "AMX";
+
+    GGML_UNUSED(dev);
+}
+
+static const char * ggml_backend_amx_device_get_description(ggml_backend_dev_t dev) {
+    return "Intel Advanced Matrix Extensions";
+
+    GGML_UNUSED(dev);
+}
+
+static void ggml_backend_amx_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) {
+    // TODO
+    *free = 0;
+    *total = 0;
+
+    GGML_UNUSED(dev);
+}
+
+static enum ggml_backend_dev_type ggml_backend_amx_device_get_type(ggml_backend_dev_t dev) {
+    return GGML_BACKEND_DEVICE_TYPE_ACCEL;
+
+    GGML_UNUSED(dev);
+}
+
+static void ggml_backend_amx_device_get_props(ggml_backend_dev_t dev, struct ggml_backend_dev_props * props) {
+    props->name        = ggml_backend_amx_device_get_name(dev);
+    props->description = ggml_backend_amx_device_get_description(dev);
+    props->type        = ggml_backend_amx_device_get_type(dev);
+    ggml_backend_amx_device_get_memory(dev, &props->memory_free, &props->memory_total);
+
+    // `buffer_from_host_ptr` is intended to be used in mmap, when memory layout unchanged
+    props->caps = {
+        /* .async                 = */ false,
+        /* .host_buffer           = */ false,
+        /* .buffer_from_host_ptr  = */ false,
+        /* .events                = */ false,
+    };
+}
+
+static ggml_backend_t ggml_backend_amx_device_init(ggml_backend_dev_t dev, const char * params) {
+    return ggml_backend_amx_init();
+
+    GGML_UNUSED(dev);
+    GGML_UNUSED(params);
+}
+
+static ggml_backend_buffer_type_t ggml_backend_amx_device_get_buffer_type(ggml_backend_dev_t dev) {
+    return ggml_backend_amx_buffer_type();
+
+    GGML_UNUSED(dev);
+}
+
+static bool ggml_backend_amx_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
+
+    // handle only 2d gemm for now
+    auto is_contiguous_2d = [](const struct ggml_tensor * t) {
+        return ggml_is_contiguous(t) && t->ne[3] == 1 && t->ne[2] == 1;
+    };
+
+    switch (op->op) {
+        case GGML_OP_NONE:
+        case GGML_OP_RESHAPE:
+        case GGML_OP_VIEW:
+        case GGML_OP_PERMUTE:
+        case GGML_OP_TRANSPOSE:
+            return true;
+
+        case GGML_OP_MUL_MAT: {
+            const struct ggml_tensor * src0 = op->src[0];
+            const struct ggml_tensor * src1 = op->src[1];
+
+            const enum ggml_type type = src0->type;
+            const int64_t ne0 = op->ne[0];
+
+            // amx kernels enables for Q4_0, Q4_1, Q8_0, F16
+            // Q4_K, Q5_K, Q6_K, IQ4_XS enabled for QK_K = 256
+            bool has_amx_kernels = qtype_has_amx_kernels(type) || (type == GGML_TYPE_F16);
+
+            bool can_use_amx =
+                is_contiguous_2d(src0) &&       // src0 must be contiguous
+                is_contiguous_2d(src1) &&       // src1 must be contiguous
+                src1->type == GGML_TYPE_F32 &&  // src1 must be float32
+                has_amx_kernels &&              // with amx kernel impls
+                ne0 % (TILE_N * 2) == 0;        // out_features is 32x
+
+            return can_use_amx;
+        }
+        default:
+            return false;
+    }
+
+    GGML_UNUSED(dev);
+}
+
+static bool ggml_backend_amx_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
+    return buft->iface.get_name == ggml_backend_amx_buffer_type_get_name;
+
+    GGML_UNUSED(dev);
+}
+
+static const struct ggml_backend_device_i ggml_backend_amx_device_i = {
+    /* .get_name             = */ ggml_backend_amx_device_get_name,
+    /* .get_description      = */ ggml_backend_amx_device_get_description,
+    /* .get_memory           = */ ggml_backend_amx_device_get_memory,
+    /* .get_type             = */ ggml_backend_amx_device_get_type,
+    /* .get_props            = */ ggml_backend_amx_device_get_props,
+    /* .init_backend         = */ ggml_backend_amx_device_init,
+    /* .get_buffer_type      = */ ggml_backend_amx_device_get_buffer_type,
+    /* .get_host_buffer_type = */ NULL,
+    /* .buffer_from_host_ptr = */ NULL,
+    /* .supports_op          = */ ggml_backend_amx_device_supports_op,
+    /* .supports_buft        = */ ggml_backend_amx_device_supports_buft,
+    /* .offload_op           = */ NULL,
+    /* .event_new            = */ NULL,
+    /* .event_free           = */ NULL,
+    /* .event_synchronize    = */ NULL,
+};
+
+// backend reg interface
+
+static const char * ggml_backend_amx_reg_get_name(ggml_backend_reg_t reg) {
+    return "AMX";
+
+    GGML_UNUSED(reg);
+}
+
+static size_t ggml_backend_amx_reg_get_device_count(ggml_backend_reg_t reg) {
+    return 1;
+
+    GGML_UNUSED(reg);
+}
+
+static ggml_backend_dev_t ggml_backend_amx_reg_get_device(ggml_backend_reg_t reg, size_t index) {
+    GGML_ASSERT(index == 0);
+
+    static ggml_backend_device ggml_backend_amx_device = {
+        /* .iface   = */ ggml_backend_amx_device_i,
+        /* .reg     = */ reg,
+        /* .context = */ nullptr,
+    };
+
+    return &ggml_backend_amx_device;
+
+    GGML_UNUSED(reg);
+    GGML_UNUSED(index);
+}
+
+static void * ggml_backend_amx_get_proc_address(ggml_backend_reg_t reg, const char * name) {
+    if (std::strcmp(name, "ggml_backend_set_n_threads") == 0) {
+        return (void *)ggml_backend_amx_set_n_threads;
+    }
+    return NULL;
+
+    GGML_UNUSED(reg);
+    GGML_UNUSED(name);
+}
+
+static const struct ggml_backend_reg_i ggml_backend_amx_reg_i = {
+    /* .get_name         = */ ggml_backend_amx_reg_get_name,
+    /* .get_device_count = */ ggml_backend_amx_reg_get_device_count,
+    /* .get_device       = */ ggml_backend_amx_reg_get_device,
+    /* .get_proc_address = */ ggml_backend_amx_get_proc_address,
+};
+
+ggml_backend_reg_t ggml_backend_amx_reg(void) {
+    static struct ggml_backend_reg ggml_backend_amx_reg = {
+        /* .iface   = */ ggml_backend_amx_reg_i,
+        /* .context = */ NULL,
+    };
+
+    return &ggml_backend_amx_reg;
+}
+
+#else // if defined(__AMX_INT8__)
+
+ggml_backend_buffer_type_t ggml_backend_amx_buffer_type(void) {
+    return nullptr;
+}
+
+bool ggml_backend_is_amx(ggml_backend_t backend) {
+    GGML_UNUSED(backend);
+    return false;
+}
+
+ggml_backend_t ggml_backend_amx_init(void) {
+    fprintf(stderr, "GGML is not compiled with AMX support!\n");
+    return nullptr;
+}
+
+void ggml_backend_amx_set_n_threads(ggml_backend_t backend_amx, int n_threads) {
+    fprintf(stderr, "GGML is not compiled with AMX support!\n");
+
+    GGML_UNUSED(backend_amx);
+    GGML_UNUSED(n_threads);
+}
+
+ggml_backend_reg_t ggml_backend_amx_reg(void) {
+    return nullptr;
+}
+
+#endif

ggml/src/ggml-amx/mmq.h

@@ -0,0 +1,17 @@
+#pragma once
+#include "common.h"
+#include <stdint.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+size_t ggml_backend_amx_get_alloc_size(const struct ggml_tensor * tensor);
+
+void ggml_backend_amx_convert_weight(struct ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+
+void ggml_backend_amx_mul_mat(ggml_backend_amx_context * ctx, struct ggml_tensor * dst);
+
+#ifdef __cplusplus
+}
+#endif

ggml/src/ggml-backend-impl.h

@@ -0,0 +1,255 @@
+#pragma once
+
+// ggml-backend internal header
+
+#include "ggml-backend.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+    #define GGML_BACKEND_API_VERSION 1
+
+    //
+    // Backend buffer type
+    //
+
+    struct ggml_backend_buffer_type_i {
+        const char *          (*get_name)      (ggml_backend_buffer_type_t buft);
+        // allocate a buffer of this type
+        ggml_backend_buffer_t (*alloc_buffer)  (ggml_backend_buffer_type_t buft, size_t size);
+        // tensor alignment
+        size_t                (*get_alignment) (ggml_backend_buffer_type_t buft);
+        // (optional) max buffer size that can be allocated (defaults to SIZE_MAX)
+        size_t                (*get_max_size)  (ggml_backend_buffer_type_t buft);
+        // (optional) data size needed to allocate the tensor, including padding (defaults to ggml_nbytes)
+        size_t                (*get_alloc_size)(ggml_backend_buffer_type_t buft, const struct ggml_tensor * tensor);
+        // (optional) check if tensor data is in host memory and uses standard ggml tensor layout (defaults to false)
+        bool                  (*is_host)       (ggml_backend_buffer_type_t buft);
+    };
+
+    struct ggml_backend_buffer_type {
+        struct ggml_backend_buffer_type_i  iface;
+        ggml_backend_dev_t device;
+        void * context;
+    };
+
+    //
+    // Backend buffer
+    //
+
+    struct ggml_backend_buffer_i {
+        // (optional) free the buffer
+        void         (*free_buffer)  (ggml_backend_buffer_t buffer);
+        // base address of the buffer
+        void *       (*get_base)     (ggml_backend_buffer_t buffer);
+        // (optional) initialize a tensor in the buffer (eg. add tensor extras)
+        enum ggml_status (*init_tensor)(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor);
+        // tensor data access
+        void         (*memset_tensor)(ggml_backend_buffer_t buffer,       struct ggml_tensor * tensor,     uint8_t value, size_t offset, size_t size);
+        void         (*set_tensor)   (ggml_backend_buffer_t buffer,       struct ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+        void         (*get_tensor)   (ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor,       void * data, size_t offset, size_t size);
+        // (optional) tensor copy: dst is in the buffer, src may be in any buffer, including buffers from a different backend (return false if not supported)
+        bool         (*cpy_tensor)   (ggml_backend_buffer_t buffer, const struct ggml_tensor * src, struct ggml_tensor * dst);
+        // clear the entire buffer
+        void         (*clear)        (ggml_backend_buffer_t buffer, uint8_t value);
+        // (optional) reset any internal state due to tensor initialization, such as tensor extras
+        void         (*reset)        (ggml_backend_buffer_t buffer);
+    };
+
+    struct ggml_backend_buffer {
+        struct ggml_backend_buffer_i  iface;
+        ggml_backend_buffer_type_t    buft;
+        void * context;
+        size_t size;
+        enum ggml_backend_buffer_usage usage;
+    };
+
+    GGML_API ggml_backend_buffer_t ggml_backend_buffer_init(
+                   ggml_backend_buffer_type_t buft,
+            struct ggml_backend_buffer_i      iface,
+                   void *                     context,
+                   size_t                     size);
+
+    // do not use directly, use ggml_backend_tensor_copy instead
+    GGML_API bool ggml_backend_buffer_copy_tensor(const struct ggml_tensor * src, struct ggml_tensor * dst);
+
+    // multi-buffer
+    // buffer that contains a collection of buffers
+    GGML_API ggml_backend_buffer_t ggml_backend_multi_buffer_alloc_buffer(ggml_backend_buffer_t * buffers, size_t n_buffers);
+    GGML_API bool                  ggml_backend_buffer_is_multi_buffer(ggml_backend_buffer_t buffer);
+    GGML_API void                  ggml_backend_multi_buffer_set_usage(ggml_backend_buffer_t buffer, enum ggml_backend_buffer_usage usage);
+
+    //
+    // Backend (stream)
+    //
+
+    struct ggml_backend_i {
+        const char * (*get_name)(ggml_backend_t backend);
+
+        void (*free)(ggml_backend_t backend);
+
+        // (optional) asynchronous tensor data access
+        void (*set_tensor_async)(ggml_backend_t backend,       struct ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+        void (*get_tensor_async)(ggml_backend_t backend, const struct ggml_tensor * tensor,       void * data, size_t offset, size_t size);
+        bool (*cpy_tensor_async)(ggml_backend_t backend_src, ggml_backend_t backend_dst, const struct ggml_tensor * src, struct ggml_tensor * dst);
+
+        // (optional) complete all pending operations (required if the backend supports async operations)
+        void (*synchronize)(ggml_backend_t backend);
+
+        // (optional) graph plans (not used currently)
+        // compute graph with a plan
+        ggml_backend_graph_plan_t (*graph_plan_create) (ggml_backend_t backend, const struct ggml_cgraph * cgraph);
+        void                      (*graph_plan_free)   (ggml_backend_t backend, ggml_backend_graph_plan_t plan);
+        // update the plan with a new graph - this should be faster than creating a new plan when the graph has the same topology
+        void                      (*graph_plan_update) (ggml_backend_t backend, ggml_backend_graph_plan_t plan, const struct ggml_cgraph * cgraph);
+        // compute the graph with the plan
+        enum ggml_status          (*graph_plan_compute)(ggml_backend_t backend, ggml_backend_graph_plan_t plan);
+
+        // compute graph (always async if supported by the backend)
+        enum ggml_status          (*graph_compute)     (ggml_backend_t backend, struct ggml_cgraph * cgraph);
+
+        // (optional) event synchronization
+        // record an event on this stream
+        void (*event_record)(ggml_backend_t backend, ggml_backend_event_t event);
+        // wait for an event on on a different stream
+        void (*event_wait)  (ggml_backend_t backend, ggml_backend_event_t event);
+    };
+
+    struct ggml_backend {
+        ggml_guid_t guid;
+        struct ggml_backend_i iface;
+        ggml_backend_dev_t device;
+        void * context;
+    };
+
+    struct ggml_backend_event {
+        struct ggml_backend_device * device;
+        void * context;
+    };
+
+    //
+    // Backend device
+    //
+
+    // Note: if additional properties are needed, we should add a struct with all of them
+    //       the current functions to obtain the properties can remain, since they are more convenient for often used properties
+    struct ggml_backend_device_i {
+        // device name: short identifier for this device, such as "CPU" or "CUDA0"
+        const char * (*get_name)(ggml_backend_dev_t dev);
+
+        // device description: short informative description of the device, could be the model name
+        const char * (*get_description)(ggml_backend_dev_t dev);
+
+        // device memory in bytes
+        void         (*get_memory)(ggml_backend_dev_t dev, size_t * free, size_t * total);
+
+        // device type
+        enum ggml_backend_dev_type (*get_type)(ggml_backend_dev_t dev);
+
+        // device properties
+        void (*get_props)(ggml_backend_dev_t dev, struct ggml_backend_dev_props * props);
+
+        // backend (stream) initialization
+        ggml_backend_t (*init_backend)(ggml_backend_dev_t dev, const char * params);
+
+        // preferred buffer type
+        ggml_backend_buffer_type_t (*get_buffer_type)(ggml_backend_dev_t dev);
+
+        // (optional) host buffer type (in system memory, typically this is a pinned memory buffer for faster transfers between host and device)
+        ggml_backend_buffer_type_t (*get_host_buffer_type)(ggml_backend_dev_t dev);
+
+        // (optional) buffer from pointer: create a buffer from a host pointer (useful for memory mapped models and importing data from other libraries)
+        ggml_backend_buffer_t (*buffer_from_host_ptr)(ggml_backend_dev_t dev, void * ptr, size_t size, size_t max_tensor_size);
+
+        // check if the backend can compute an operation
+        bool (*supports_op)(ggml_backend_dev_t dev, const struct ggml_tensor * op);
+
+        // check if the backend can use tensors allocated in a buffer type
+        bool (*supports_buft)(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft);
+
+        // (optional) check if the backend wants to run an operation, even if the weights are allocated in an incompatible buffer
+        // these should be expensive operations that may benefit from running on this backend instead of the CPU backend
+        bool (*offload_op)(ggml_backend_dev_t dev, const struct ggml_tensor * op);
+
+        // (optional) event synchronization
+        ggml_backend_event_t (*event_new)         (ggml_backend_dev_t dev);
+        void                 (*event_free)        (ggml_backend_dev_t dev, ggml_backend_event_t event);
+        void                 (*event_synchronize) (ggml_backend_dev_t dev, ggml_backend_event_t event);
+    };
+
+    struct ggml_backend_device {
+        struct ggml_backend_device_i iface;
+        ggml_backend_reg_t reg;
+        void * context;
+    };
+
+    //
+    // Backend (reg)
+    //
+
+    struct ggml_backend_reg_i {
+        const char * (*get_name)(ggml_backend_reg_t reg);
+
+        // enumerate available devices
+        size_t             (*get_device_count)(ggml_backend_reg_t reg);
+        ggml_backend_dev_t (*get_device)(ggml_backend_reg_t reg, size_t index);
+
+        // (optional) get a pointer to a function in the backend
+        // backends can add custom functions that are not part of the standard ggml-backend interface
+        void * (*get_proc_address)(ggml_backend_reg_t reg, const char * name);
+    };
+
+    struct ggml_backend_reg {
+        int api_version; // initialize to GGML_BACKEND_API_VERSION
+        struct ggml_backend_reg_i iface;
+        void * context;
+    };
+
+    // Internal backend registry API
+    GGML_API void ggml_backend_register(ggml_backend_reg_t reg);
+
+    // Add backend dynamic loading support to the backend
+
+    // Initialize the backend
+    typedef ggml_backend_reg_t (*ggml_backend_init_t)(void);
+    // Optional: obtain a score for the backend based on the system configuration
+    // Higher scores are preferred, 0 means the backend is not supported in the current system
+    typedef int                (*ggml_backend_score_t)(void);
+
+#ifdef GGML_BACKEND_DL
+#    ifdef __cplusplus
+#        define GGML_BACKEND_DL_IMPL(reg_fn)                             \
+            extern "C" {                                                 \
+            GGML_BACKEND_API ggml_backend_reg_t ggml_backend_init(void); \
+            }                                                            \
+            ggml_backend_reg_t ggml_backend_init(void) {                 \
+                return reg_fn();                                         \
+            }
+#        define GGML_BACKEND_DL_SCORE_IMPL(score_fn)       \
+            extern "C" {                                   \
+            GGML_BACKEND_API int ggml_backend_score(void); \
+            }                                              \
+            int ggml_backend_score(void) {                 \
+                return score_fn();                         \
+            }
+#    else
+#        define GGML_BACKEND_DL_IMPL(reg_fn)                              \
+            GGML_BACKEND_API ggml_backend_reg_t ggml_backend_init(void);  \
+            ggml_backend_reg_t                  ggml_backend_init(void) { \
+                return reg_fn();                                          \
+            }
+#        define GGML_BACKEND_DL_SCORE_IMPL(score_fn)        \
+            GGML_BACKEND_API int ggml_backend_score(void);  \
+            int                  ggml_backend_score(void) { \
+                return score_fn();                          \
+            }
+#    endif
+#else
+#    define GGML_BACKEND_DL_IMPL(reg_fn)
+#    define GGML_BACKEND_DL_SCORE_IMPL(score_fn)
+#endif
+
+#ifdef  __cplusplus
+}
+#endif

ggml/src/ggml-backend-reg.cpp

@@ -0,0 +1,590 @@
+#include "ggml-backend-impl.h"
+#include "ggml-backend.h"
+#include "ggml-impl.h"
+#include <algorithm>
+#include <cstring>
+#include <filesystem>
+#include <memory>
+#include <string>
+#include <type_traits>
+#include <vector>
+#include <cctype>
+
+#ifdef _WIN32
+#    define WIN32_LEAN_AND_MEAN
+#    ifndef NOMINMAX
+#        define NOMINMAX
+#    endif
+#    include <windows.h>
+#elif defined(__APPLE__)
+#    include <mach-o/dyld.h>
+#    include <dlfcn.h>
+#else
+#    include <dlfcn.h>
+#    include <unistd.h>
+#endif
+
+// Backend registry
+#ifdef GGML_USE_CPU
+#include "ggml-cpu.h"
+#endif
+
+#ifdef GGML_USE_CUDA
+#include "ggml-cuda.h"
+#endif
+
+#ifdef GGML_USE_METAL
+#include "ggml-metal.h"
+#endif
+
+#ifdef GGML_USE_SYCL
+#include "ggml-sycl.h"
+#endif
+
+#ifdef GGML_USE_VULKAN
+#include "ggml-vulkan.h"
+#endif
+
+#ifdef GGML_USE_WEBGPU
+#include "ggml-webgpu.h"
+#endif
+
+#ifdef GGML_USE_OPENCL
+#include "ggml-opencl.h"
+#endif
+
+#ifdef GGML_USE_BLAS
+#include "ggml-blas.h"
+#endif
+
+#ifdef GGML_USE_RPC
+#include "ggml-rpc.h"
+#endif
+
+#ifdef GGML_USE_CANN
+#include "ggml-cann.h"
+#endif
+
+// disable C++17 deprecation warning for std::codecvt_utf8
+#if defined(__clang__)
+#    pragma clang diagnostic push
+#    pragma clang diagnostic ignored "-Wdeprecated-declarations"
+#elif defined(__GNUC__)
+#    pragma GCC diagnostic push
+#    pragma GCC diagnostic ignored "-Wdeprecated-declarations"
+#endif
+
+namespace fs = std::filesystem;
+
+static std::string path_str(const fs::path & path) {
+    std::string u8path;
+    try {
+#if defined(__cpp_lib_char8_t)
+        // C++20 and later: u8string() returns std::u8string
+        std::u8string u8str = path.u8string();
+        u8path = std::string(reinterpret_cast<const char*>(u8str.c_str()));
+#else
+        // C++17: u8string() returns std::string
+        u8path = path.u8string();
+#endif
+    } catch (...) {
+    }
+    return u8path;
+}
+
+#if defined(__clang__)
+#    pragma clang diagnostic pop
+#elif defined(__GNUC__)
+#    pragma GCC diagnostic pop
+#endif
+
+#ifdef _WIN32
+
+using dl_handle = std::remove_pointer_t<HMODULE>;
+
+struct dl_handle_deleter {
+    void operator()(HMODULE handle) {
+        FreeLibrary(handle);
+    }
+};
+
+static dl_handle * dl_load_library(const fs::path & path) {
+    // suppress error dialogs for missing DLLs
+    DWORD old_mode = SetErrorMode(SEM_FAILCRITICALERRORS);
+    SetErrorMode(old_mode | SEM_FAILCRITICALERRORS);
+
+    HMODULE handle = LoadLibraryW(path.wstring().c_str());
+
+    SetErrorMode(old_mode);
+
+    return handle;
+}
+
+static void * dl_get_sym(dl_handle * handle, const char * name) {
+    DWORD old_mode = SetErrorMode(SEM_FAILCRITICALERRORS);
+    SetErrorMode(old_mode | SEM_FAILCRITICALERRORS);
+
+    void * p = (void *) GetProcAddress(handle, name);
+
+    SetErrorMode(old_mode);
+
+    return p;
+}
+
+#else
+
+using dl_handle = void;
+
+struct dl_handle_deleter {
+    void operator()(void * handle) {
+        dlclose(handle);
+    }
+};
+
+static void * dl_load_library(const fs::path & path) {
+    dl_handle * handle = dlopen(path.string().c_str(), RTLD_NOW | RTLD_LOCAL);
+
+    return handle;
+}
+
+static void * dl_get_sym(dl_handle * handle, const char * name) {
+    return dlsym(handle, name);
+}
+
+#endif
+
+using dl_handle_ptr = std::unique_ptr<dl_handle, dl_handle_deleter>;
+
+struct ggml_backend_reg_entry {
+    ggml_backend_reg_t reg;
+    dl_handle_ptr handle;
+};
+
+struct ggml_backend_registry {
+    std::vector<ggml_backend_reg_entry> backends;
+    std::vector<ggml_backend_dev_t> devices;
+
+    ggml_backend_registry() {
+#ifdef GGML_USE_CUDA
+        register_backend(ggml_backend_cuda_reg());
+#endif
+#ifdef GGML_USE_METAL
+        register_backend(ggml_backend_metal_reg());
+#endif
+#ifdef GGML_USE_SYCL
+        register_backend(ggml_backend_sycl_reg());
+#endif
+#ifdef GGML_USE_VULKAN
+        register_backend(ggml_backend_vk_reg());
+#endif
+#ifdef GGML_USE_WEBGPU
+        register_backend(ggml_backend_webgpu_reg());
+#endif
+#ifdef GGML_USE_OPENCL
+        register_backend(ggml_backend_opencl_reg());
+#endif
+#ifdef GGML_USE_CANN
+        register_backend(ggml_backend_cann_reg());
+#endif
+#ifdef GGML_USE_BLAS
+        register_backend(ggml_backend_blas_reg());
+#endif
+#ifdef GGML_USE_RPC
+        register_backend(ggml_backend_rpc_reg());
+#endif
+#ifdef GGML_USE_CPU
+        register_backend(ggml_backend_cpu_reg());
+#endif
+    }
+
+    ~ggml_backend_registry() {
+        // FIXME: backends cannot be safely unloaded without a function to destroy all the backend resources,
+        // since backend threads may still be running and accessing resources from the dynamic library
+        for (auto & entry : backends) {
+            if (entry.handle) {
+                entry.handle.release(); // NOLINT
+            }
+        }
+    }
+
+    void register_backend(ggml_backend_reg_t reg, dl_handle_ptr handle = nullptr) {
+        if (!reg) {
+            return;
+        }
+
+#ifndef NDEBUG
+        GGML_LOG_DEBUG("%s: registered backend %s (%zu devices)\n",
+            __func__, ggml_backend_reg_name(reg), ggml_backend_reg_dev_count(reg));
+#endif
+        backends.push_back({ reg, std::move(handle) });
+        for (size_t i = 0; i < ggml_backend_reg_dev_count(reg); i++) {
+            register_device(ggml_backend_reg_dev_get(reg, i));
+        }
+    }
+
+    void register_device(ggml_backend_dev_t device) {
+#ifndef NDEBUG
+        GGML_LOG_DEBUG("%s: registered device %s (%s)\n", __func__, ggml_backend_dev_name(device), ggml_backend_dev_description(device));
+#endif
+        devices.push_back(device);
+    }
+
+    ggml_backend_reg_t load_backend(const fs::path & path, bool silent) {
+        dl_handle_ptr handle { dl_load_library(path) };
+        if (!handle) {
+            if (!silent) {
+                GGML_LOG_ERROR("%s: failed to load %s\n", __func__, path_str(path).c_str());
+            }
+            return nullptr;
+        }
+
+        auto score_fn = (ggml_backend_score_t) dl_get_sym(handle.get(), "ggml_backend_score");
+        if (score_fn && score_fn() == 0) {
+            if (!silent) {
+                GGML_LOG_INFO("%s: backend %s is not supported on this system\n", __func__, path_str(path).c_str());
+            }
+            return nullptr;
+        }
+
+        auto backend_init_fn = (ggml_backend_init_t) dl_get_sym(handle.get(), "ggml_backend_init");
+        if (!backend_init_fn) {
+            if (!silent) {
+                GGML_LOG_ERROR("%s: failed to find ggml_backend_init in %s\n", __func__, path_str(path).c_str());
+            }
+            return nullptr;
+        }
+
+        ggml_backend_reg_t reg = backend_init_fn();
+        if (!reg || reg->api_version != GGML_BACKEND_API_VERSION) {
+            if (!silent) {
+                if (!reg) {
+                    GGML_LOG_ERROR("%s: failed to initialize backend from %s: ggml_backend_init returned NULL\n",
+                        __func__, path_str(path).c_str());
+                } else {
+                    GGML_LOG_ERROR("%s: failed to initialize backend from %s: incompatible API version (backend: %d, current: %d)\n",
+                        __func__, path_str(path).c_str(), reg->api_version, GGML_BACKEND_API_VERSION);
+                }
+            }
+            return nullptr;
+        }
+
+        GGML_LOG_INFO("%s: loaded %s backend from %s\n", __func__, ggml_backend_reg_name(reg), path_str(path).c_str());
+
+        register_backend(reg, std::move(handle));
+
+        return reg;
+    }
+
+    void unload_backend(ggml_backend_reg_t reg, bool silent) {
+        auto it = std::find_if(backends.begin(), backends.end(),
+                               [reg](const ggml_backend_reg_entry & entry) { return entry.reg == reg; });
+
+        if (it == backends.end()) {
+            if (!silent) {
+                GGML_LOG_ERROR("%s: backend not found\n", __func__);
+            }
+            return;
+        }
+
+        if (!silent) {
+            GGML_LOG_DEBUG("%s: unloading %s backend\n", __func__, ggml_backend_reg_name(reg));
+        }
+
+        // remove devices
+        devices.erase(
+            std::remove_if(devices.begin(), devices.end(),
+                            [reg](ggml_backend_dev_t dev) { return ggml_backend_dev_backend_reg(dev) == reg; }),
+            devices.end());
+
+        // remove backend
+        backends.erase(it);
+    }
+};
+
+static ggml_backend_registry & get_reg() {
+    static ggml_backend_registry reg;
+    return reg;
+}
+
+// Internal API
+void ggml_backend_register(ggml_backend_reg_t reg) {
+    get_reg().register_backend(reg);
+}
+
+void ggml_backend_device_register(ggml_backend_dev_t device) {
+    get_reg().register_device(device);
+}
+
+// Backend (reg) enumeration
+static bool striequals(const char * a, const char * b) {
+    for (; *a && *b; a++, b++) {
+        if (std::tolower(*a) != std::tolower(*b)) {
+            return false;
+        }
+    }
+    return *a == *b;
+}
+
+size_t ggml_backend_reg_count() {
+    return get_reg().backends.size();
+}
+
+ggml_backend_reg_t ggml_backend_reg_get(size_t index) {
+    GGML_ASSERT(index < ggml_backend_reg_count());
+    return get_reg().backends[index].reg;
+}
+
+ggml_backend_reg_t ggml_backend_reg_by_name(const char * name) {
+    for (size_t i = 0; i < ggml_backend_reg_count(); i++) {
+        ggml_backend_reg_t reg = ggml_backend_reg_get(i);
+        if (striequals(ggml_backend_reg_name(reg), name)) {
+            return reg;
+        }
+    }
+    return nullptr;
+}
+
+// Device enumeration
+size_t ggml_backend_dev_count() {
+    return get_reg().devices.size();
+}
+
+ggml_backend_dev_t ggml_backend_dev_get(size_t index) {
+    GGML_ASSERT(index < ggml_backend_dev_count());
+    return get_reg().devices[index];
+}
+
+ggml_backend_dev_t ggml_backend_dev_by_name(const char * name) {
+    for (size_t i = 0; i < ggml_backend_dev_count(); i++) {
+        ggml_backend_dev_t dev = ggml_backend_dev_get(i);
+        if (striequals(ggml_backend_dev_name(dev), name)) {
+            return dev;
+        }
+    }
+    return nullptr;
+}
+
+ggml_backend_dev_t ggml_backend_dev_by_type(enum ggml_backend_dev_type type) {
+    for (size_t i = 0; i < ggml_backend_dev_count(); i++) {
+        ggml_backend_dev_t dev = ggml_backend_dev_get(i);
+        if (ggml_backend_dev_type(dev) == type) {
+            return dev;
+        }
+    }
+    return nullptr;
+}
+
+// Convenience functions
+ggml_backend_t ggml_backend_init_by_name(const char * name, const char * params) {
+    ggml_backend_dev_t dev = ggml_backend_dev_by_name(name);
+    if (!dev) {
+        return nullptr;
+    }
+    return ggml_backend_dev_init(dev, params);
+}
+
+ggml_backend_t ggml_backend_init_by_type(enum ggml_backend_dev_type type, const char * params) {
+    ggml_backend_dev_t dev = ggml_backend_dev_by_type(type);
+    if (!dev) {
+        return nullptr;
+    }
+    return ggml_backend_dev_init(dev, params);
+}
+
+ggml_backend_t ggml_backend_init_best(void) {
+    ggml_backend_dev_t dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_GPU);
+    if (!dev) {
+        dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU);
+    }
+    if (!dev) {
+        return nullptr;
+    }
+    return ggml_backend_dev_init(dev, nullptr);
+}
+
+// Dynamic loading
+ggml_backend_reg_t ggml_backend_load(const char * path) {
+    return get_reg().load_backend(path, false);
+}
+
+void ggml_backend_unload(ggml_backend_reg_t reg) {
+    get_reg().unload_backend(reg, true);
+}
+
+static fs::path get_executable_path() {
+#if defined(__APPLE__)
+    // get executable path
+    std::vector<char> path;
+    uint32_t size;
+    while (true) {
+        size = path.size();
+        if (_NSGetExecutablePath(path.data(), &size) == 0) {
+            break;
+        }
+        path.resize(size);
+    }
+    std::string base_path(path.data(), size);
+    // remove executable name
+    auto last_slash = base_path.find_last_of('/');
+    if (last_slash != std::string::npos) {
+        base_path = base_path.substr(0, last_slash);
+    }
+    return base_path + "/";
+#elif defined(__linux__) || defined(__FreeBSD__)
+    std::string base_path = ".";
+    std::vector<char> path(1024);
+    while (true) {
+        // get executable path
+#    if defined(__linux__)
+        ssize_t len = readlink("/proc/self/exe", path.data(), path.size());
+#    elif defined(__FreeBSD__)
+        ssize_t len = readlink("/proc/curproc/file", path.data(), path.size());
+#    endif
+        if (len == -1) {
+            break;
+        }
+        if (len < (ssize_t) path.size()) {
+            base_path = std::string(path.data(), len);
+            // remove executable name
+            auto last_slash = base_path.find_last_of('/');
+            if (last_slash != std::string::npos) {
+                base_path = base_path.substr(0, last_slash);
+            }
+            break;
+        }
+        path.resize(path.size() * 2);
+    }
+
+    return base_path + "/";
+#elif defined(_WIN32)
+    std::vector<wchar_t> path(MAX_PATH);
+    DWORD len = GetModuleFileNameW(NULL, path.data(), path.size());
+    if (len == 0) {
+        return {};
+    }
+    std::wstring base_path(path.data(), len);
+    // remove executable name
+    auto last_slash = base_path.find_last_of('\\');
+    if (last_slash != std::string::npos) {
+        base_path = base_path.substr(0, last_slash);
+    }
+    return base_path + L"\\";
+#else
+    return {};
+#endif
+}
+
+static fs::path backend_filename_prefix() {
+#ifdef _WIN32
+    return fs::u8path("ggml-");
+#else
+    return fs::u8path("libggml-");
+#endif
+}
+
+static fs::path backend_filename_extension() {
+#ifdef _WIN32
+    return fs::u8path(".dll");
+#else
+    return fs::u8path(".so");
+#endif
+}
+
+static ggml_backend_reg_t ggml_backend_load_best(const char * name, bool silent, const char * user_search_path) {
+    // enumerate all the files that match [lib]ggml-name-*.[so|dll] in the search paths
+    const fs::path name_path = fs::u8path(name);
+    const fs::path file_prefix = backend_filename_prefix().native() + name_path.native() + fs::u8path("-").native();
+    const fs::path file_extension = backend_filename_extension();
+
+    std::vector<fs::path> search_paths;
+    if (user_search_path == nullptr) {
+        // default search paths: executable directory, current directory
+        search_paths.push_back(get_executable_path());
+        search_paths.push_back(fs::current_path());
+    } else {
+        search_paths.push_back(fs::u8path(user_search_path));
+    }
+
+    int best_score = 0;
+    fs::path best_path;
+
+    for (const auto & search_path : search_paths) {
+        if (!fs::exists(search_path)) {
+            GGML_LOG_DEBUG("%s: search path %s does not exist\n", __func__, path_str(search_path).c_str());
+            continue;
+        }
+        fs::directory_iterator dir_it(search_path, fs::directory_options::skip_permission_denied);
+        for (const auto & entry : dir_it) {
+            if (entry.is_regular_file()) {
+                auto filename = entry.path().filename();
+                auto ext = entry.path().extension();
+                if (filename.native().find(file_prefix) == 0 && ext == file_extension) {
+                    dl_handle_ptr handle { dl_load_library(entry) };
+                    if (!handle && !silent) {
+                        GGML_LOG_ERROR("%s: failed to load %s\n", __func__, path_str(entry.path()).c_str());
+                    }
+                    if (handle) {
+                        auto score_fn = (ggml_backend_score_t) dl_get_sym(handle.get(), "ggml_backend_score");
+                        if (score_fn) {
+                            int s = score_fn();
+#ifndef NDEBUG
+                            GGML_LOG_DEBUG("%s: %s score: %d\n", __func__, path_str(entry.path()).c_str(), s);
+#endif
+                            if (s > best_score) {
+                                best_score = s;
+                                best_path = entry.path();
+                            }
+                        } else {
+                            if (!silent) {
+                                GGML_LOG_INFO("%s: failed to find ggml_backend_score in %s\n", __func__, path_str(entry.path()).c_str());
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+    if (best_score == 0) {
+        // try to load the base backend
+        for (const auto & search_path : search_paths) {
+            fs::path filename = backend_filename_prefix().native() + name_path.native() + backend_filename_extension().native();
+            fs::path path = search_path / filename;
+            if (fs::exists(path)) {
+                return get_reg().load_backend(path, silent);
+            }
+        }
+        return nullptr;
+    }
+
+    return get_reg().load_backend(best_path, silent);
+}
+
+void ggml_backend_load_all() {
+    ggml_backend_load_all_from_path(nullptr);
+}
+
+void ggml_backend_load_all_from_path(const char * dir_path) {
+#ifdef NDEBUG
+    bool silent = true;
+#else
+    bool silent = false;
+#endif
+
+    ggml_backend_load_best("blas", silent, dir_path);
+    ggml_backend_load_best("cann", silent, dir_path);
+    ggml_backend_load_best("cuda", silent, dir_path);
+    ggml_backend_load_best("hip", silent, dir_path);
+    ggml_backend_load_best("metal", silent, dir_path);
+    ggml_backend_load_best("rpc", silent, dir_path);
+    ggml_backend_load_best("sycl", silent, dir_path);
+    ggml_backend_load_best("vulkan", silent, dir_path);
+    ggml_backend_load_best("opencl", silent, dir_path);
+    ggml_backend_load_best("musa", silent, dir_path);
+    ggml_backend_load_best("cpu", silent, dir_path);
+    // check the environment variable GGML_BACKEND_PATH to load an out-of-tree backend
+    const char * backend_path = std::getenv("GGML_BACKEND_PATH");
+    if (backend_path) {
+        ggml_backend_load(backend_path);
+    }
+}

ggml/src/ggml-blas/CMakeLists.txt

@@ -0,0 +1,87 @@
+if (GGML_STATIC)
+    set(BLA_STATIC ON)
+endif()
+#if (CMAKE_VERSION VERSION_GREATER_EQUAL 3.22)
+#    set(BLA_SIZEOF_INTEGER 8)
+#endif()
+
+set(BLA_VENDOR ${GGML_BLAS_VENDOR})
+find_package(BLAS)
+
+if (BLAS_FOUND)
+    message(STATUS "BLAS found, Libraries: ${BLAS_LIBRARIES}")
+
+    ggml_add_backend_library(ggml-blas
+                             ggml-blas.cpp
+                            )
+
+    if (${GGML_BLAS_VENDOR} MATCHES "Apple")
+        add_compile_definitions(ACCELERATE_NEW_LAPACK)
+        add_compile_definitions(ACCELERATE_LAPACK_ILP64)
+        add_compile_definitions(GGML_BLAS_USE_ACCELERATE)
+    elseif ("${BLAS_INCLUDE_DIRS}" STREQUAL "")
+        # BLAS_INCLUDE_DIRS is missing in FindBLAS.cmake.
+        # see https://gitlab.kitware.com/cmake/cmake/-/issues/20268
+        find_package(PkgConfig REQUIRED)
+        if (${GGML_BLAS_VENDOR} MATCHES "Generic")
+            pkg_check_modules(DepBLAS blas)
+        elseif (${GGML_BLAS_VENDOR} MATCHES "OpenBLAS")
+            # As of openblas v0.3.22, the 64-bit is named openblas64.pc
+            pkg_check_modules(DepBLAS openblas64)
+            if (NOT DepBLAS_FOUND)
+                pkg_check_modules(DepBLAS openblas)
+            endif()
+        elseif (${GGML_BLAS_VENDOR} MATCHES "FLAME")
+            add_compile_definitions(GGML_BLAS_USE_BLIS)
+            pkg_check_modules(DepBLAS blis)
+        elseif (${GGML_BLAS_VENDOR} MATCHES "ATLAS")
+            pkg_check_modules(DepBLAS blas-atlas)
+        elseif (${GGML_BLAS_VENDOR} MATCHES "FlexiBLAS")
+            pkg_check_modules(DepBLAS flexiblas_api)
+        elseif (${GGML_BLAS_VENDOR} MATCHES "Intel")
+            add_compile_definitions(GGML_BLAS_USE_MKL)
+            # all Intel* libraries share the same include path
+            pkg_check_modules(DepBLAS mkl-sdl)
+        elseif (${GGML_BLAS_VENDOR} MATCHES "NVHPC")
+            # this doesn't provide pkg-config
+            # suggest to assign BLAS_INCLUDE_DIRS on your own
+            if ("${NVHPC_VERSION}" STREQUAL "")
+                message(WARNING "Better to set NVHPC_VERSION")
+            else()
+                set(DepBLAS_FOUND ON)
+                set(DepBLAS_INCLUDE_DIRS "/opt/nvidia/hpc_sdk/${CMAKE_SYSTEM_NAME}_${CMAKE_SYSTEM_PROCESSOR}/${NVHPC_VERSION}/math_libs/include")
+            endif()
+        endif()
+        if (DepBLAS_FOUND)
+            set(BLAS_INCLUDE_DIRS ${DepBLAS_INCLUDE_DIRS})
+        else()
+            message(WARNING "BLAS_INCLUDE_DIRS neither been provided nor been automatically"
+            " detected by pkgconfig, trying to find cblas.h from possible paths...")
+            find_path(BLAS_INCLUDE_DIRS
+                NAMES cblas.h
+                HINTS
+                    /usr/include
+                    /usr/local/include
+                    /usr/include/openblas
+                    /opt/homebrew/opt/openblas/include
+                    /usr/local/opt/openblas/include
+                    /usr/include/x86_64-linux-gnu/openblas/include
+            )
+        endif()
+    endif()
+
+    message(STATUS "BLAS found, Includes: ${BLAS_INCLUDE_DIRS}")
+
+    target_compile_options(ggml-blas PRIVATE ${BLAS_LINKER_FLAGS})
+
+    if (${BLAS_INCLUDE_DIRS} MATCHES "mkl" AND (${GGML_BLAS_VENDOR} MATCHES "Generic" OR ${GGML_BLAS_VENDOR} MATCHES "Intel"))
+        add_compile_definitions(GGML_BLAS_USE_MKL)
+    endif()
+
+    target_link_libraries     (ggml-blas PRIVATE ${BLAS_LIBRARIES})
+    target_include_directories(ggml-blas PRIVATE ${BLAS_INCLUDE_DIRS})
+else()
+    message(FATAL_ERROR "BLAS not found, please refer to "
+                        "https://cmake.org/cmake/help/latest/module/FindBLAS.html#blas-lapack-vendors"
+                        " to set correct GGML_BLAS_VENDOR")
+endif()

ggml/src/ggml-blas/ggml-blas.cpp

@@ -0,0 +1,517 @@
+#include "ggml-impl.h"
+#include "ggml-blas.h"
+#include "ggml-backend-impl.h"
+
+#include <future>
+#include <vector>
+#include <cstring>
+
+#if defined(GGML_BLAS_USE_ACCELERATE)
+#   include <Accelerate/Accelerate.h>
+#elif defined(GGML_BLAS_USE_MKL)
+#   include <mkl.h>
+#elif defined(GGML_BLAS_USE_BLIS)
+#   include <blis.h>
+#elif defined(GGML_BLAS_USE_NVPL)
+#   include <nvpl_blas.h>
+#else
+#   include <cblas.h>
+#endif
+
+struct ggml_backend_blas_context {
+    int n_threads = GGML_DEFAULT_N_THREADS;
+    std::unique_ptr<char[]> work_data;
+    size_t work_size = 0;
+#ifndef GGML_USE_OPENMP
+    std::vector<std::future<void>> tasks;
+#endif
+};
+
+static void ggml_backend_blas_mul_mat(ggml_backend_blas_context * ctx, struct ggml_tensor * dst) {
+    const struct ggml_tensor * src0 = dst->src[0];
+    const struct ggml_tensor * src1 = dst->src[1];
+
+    GGML_TENSOR_BINARY_OP_LOCALS
+
+    const enum ggml_type type = src0->type;
+
+    GGML_ASSERT(ne0 == ne01);
+    GGML_ASSERT(ne1 == ne11);
+    GGML_ASSERT(ne2 == ne12);
+    GGML_ASSERT(ne3 == ne13);
+
+    // we don't support permuted src0 or src1
+    GGML_ASSERT(nb00 == ggml_type_size(type));
+    GGML_ASSERT(nb10 == ggml_type_size(src1->type));
+
+    // dst cannot be transposed or permuted
+    GGML_ASSERT(nb0 == sizeof(float));
+    GGML_ASSERT(nb0 <= nb1);
+    GGML_ASSERT(nb1 <= nb2);
+    GGML_ASSERT(nb2 <= nb3);
+
+    // broadcast factors
+    const int64_t r2 = ne12/ne02;
+    const int64_t r3 = ne13/ne03;
+
+    const int64_t ne_plane      = ne01*ne00;
+    const size_t  desired_wsize = type == GGML_TYPE_F32 ? 0 : ne03*ne02*ne_plane*sizeof(float);
+
+    if (ctx->work_size < desired_wsize) {
+        ctx->work_data.reset(new char[desired_wsize]);
+        ctx->work_size = desired_wsize;
+    }
+    void * wdata = ctx->work_data.get();
+
+    // convert src0 to float
+    if (type != GGML_TYPE_F32) {
+        const auto * type_traits = ggml_get_type_traits(type);
+        ggml_to_float_t const to_float = type_traits->to_float;
+
+        for (int64_t i03 = 0; i03 < ne03; i03++) {
+            for (int64_t i02 = 0; i02 < ne02; i02++) {
+                const void  *       x      = (char *)  src0->data + i02*nb02          + i03*nb03;
+                      float * const wplane = (float *) wdata      + i02*ne_plane      + i03*ne02*ne_plane;
+
+                const int min_cols_per_thread = 4096;
+                const int min_rows_per_thread = std::max((int)(min_cols_per_thread/ne00), 1);
+                const int n_threads = std::max(std::min(ctx->n_threads, (int)(ne01/min_rows_per_thread)), 1);
+
+#ifdef GGML_USE_OPENMP
+                #pragma omp parallel for num_threads(n_threads)
+                for (int64_t i01 = 0; i01 < ne01; i01++) {
+                    to_float((const char *) x + i01*nb01, wplane + i01*ne00, ne00);
+                }
+#else
+                for (int i = 1; i < n_threads; i++) {
+                    const int64_t start =       i*ne01/n_threads;
+                    const int64_t end   = (i + 1)*ne01/n_threads;
+                    if (start < end) {
+                        ctx->tasks.push_back(std::async(std::launch::async, [=]() {
+                            for (int64_t i01 = start; i01 < end; i01++) {
+                                to_float((const char *) x + i01*nb01, wplane + i01*ne00, ne00);
+                            }
+                        }));
+                    }
+                }
+                {
+                    // reuse the current thread for the first task
+                    const int64_t start = 0;
+                    const int64_t end   = ne01/n_threads;
+                    for (int64_t i01 = start; i01 < end; i01++) {
+                        to_float((const char *) x + i01*nb01, wplane + i01*ne00, ne00);
+                    }
+                }
+#endif
+            }
+        }
+
+#ifndef GGML_USE_OPENMP
+        // wait for all tasks to finish
+        for (auto & task : ctx->tasks) {
+            task.get();
+        }
+        ctx->tasks.clear();
+#endif
+    }
+
+#if defined(OPENBLAS_VERSION)
+    openblas_set_num_threads(ctx->n_threads);
+#endif
+
+#if defined(GGML_BLAS_USE_BLIS)
+    bli_thread_set_num_threads(ctx->n_threads);
+#endif
+
+#if defined(GGML_BLAS_USE_NVPL)
+    nvpl_blas_set_num_threads(ctx->n_threads);
+#endif
+
+    for (int64_t i13 = 0; i13 < ne13; i13++) {
+        for (int64_t i12 = 0; i12 < ne12; i12++) {
+            const int64_t i03 = i13/r3;
+            const int64_t i02 = i12/r2;
+
+            const float * x = (float *) ((char *) src0->data + i02*nb02 + i03*nb03);
+            const float * y = (float *) ((char *) src1->data + i12*nb12 + i13*nb13);
+                  float * d = (float *) ((char *)  dst->data + i12*nb2  + i13*nb3);
+
+            if (type != GGML_TYPE_F32) {
+                x = (float *) wdata + i02*ne_plane + i03*ne02*ne_plane;
+            }
+
+            cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans,
+                        ne1, ne01, ne10,
+                        1.0f,   y, ne10,
+                                x, ne00,
+                        0.0f,   d, ne01);
+        }
+    }
+}
+
+static void ggml_backend_blas_out_prod(ggml_backend_blas_context * ctx, struct ggml_tensor * dst) {
+    const struct ggml_tensor * src0 = dst->src[0];
+    const struct ggml_tensor * src1 = dst->src[1];
+
+    GGML_TENSOR_BINARY_OP_LOCALS
+
+    GGML_ASSERT(ne0  == ne00);
+    GGML_ASSERT(ne1  == ne10);
+    GGML_ASSERT(ne2  == ne02);
+    GGML_ASSERT(ne02 == ne12);
+    GGML_ASSERT(ne3  == ne13);
+    GGML_ASSERT(ne03 == ne13);
+
+    // we don't support permuted src0 or src1
+    GGML_ASSERT(nb00 == sizeof(float));
+
+    // dst cannot be transposed or permuted
+    GGML_ASSERT(nb0 == sizeof(float));
+    // GGML_ASSERT(nb0 <= nb1);
+    // GGML_ASSERT(nb1 <= nb2);
+    // GGML_ASSERT(nb2 <= nb3);
+
+    // Arguments to ggml_compute_forward_out_prod (expressed as major,minor)
+    // src0: (k,n)
+    // src1: (k,m)
+    // dst:  (m,n)
+    //
+    // Arguments to sgemm (see https://github.com/Reference-LAPACK/lapack/blob/master/BLAS/SRC/sgemm.f)
+    // Also expressed as (major,minor)
+    // a: (m,k): so src1 transposed
+    // b: (k,n): so src0
+    // c: (m,n)
+    //
+    // However, if ggml_is_transposed(src1) is true, then
+    // src1->data already contains a transposed version, so sgemm mustn't
+    // transpose it further.
+
+    int n = src0->ne[0];
+    int k = src0->ne[1];
+    int m = src1->ne[0];
+
+    CBLAS_TRANSPOSE transposeA;
+    int lda;
+
+    if (!ggml_is_transposed(src1)) {
+        transposeA = CblasTrans;
+        lda = m;
+    } else {
+        transposeA = CblasNoTrans;
+        lda = k;
+    }
+
+    float * a = (float *) ((char *) src1->data);
+    float * b = (float *) ((char *) src0->data);
+    float * c = (float *) ((char *) dst->data);
+
+    cblas_sgemm(CblasRowMajor, transposeA, CblasNoTrans, m, n, k, 1.0, a, lda, b, n, 0.0, c, n);
+
+    GGML_UNUSED(ctx);
+}
+
+// backend interface
+
+static const char * ggml_backend_blas_get_name(ggml_backend_t backend) {
+    return "BLAS";
+
+    GGML_UNUSED(backend);
+}
+
+static void ggml_backend_blas_free(ggml_backend_t backend) {
+    ggml_backend_blas_context * ctx = (ggml_backend_blas_context *)backend->context;
+    delete ctx;
+    delete backend;
+}
+
+static enum ggml_status ggml_backend_blas_graph_compute(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
+    ggml_backend_blas_context * ctx = (ggml_backend_blas_context *)backend->context;
+
+    for (int i = 0; i < cgraph->n_nodes; i++) {
+        struct ggml_tensor * node = cgraph->nodes[i];
+
+        switch (node->op) {
+            case GGML_OP_MUL_MAT:
+                ggml_backend_blas_mul_mat(ctx, node);
+                break;
+
+            case GGML_OP_OUT_PROD:
+                ggml_backend_blas_out_prod(ctx, node);
+                break;
+
+            case GGML_OP_NONE:
+            case GGML_OP_RESHAPE:
+            case GGML_OP_VIEW:
+            case GGML_OP_PERMUTE:
+            case GGML_OP_TRANSPOSE:
+                break;
+
+            default:
+                GGML_ABORT("%s: unsupported op %s\n", __func__, ggml_op_desc(node));
+        }
+    }
+
+    return GGML_STATUS_SUCCESS;
+
+    GGML_UNUSED(backend);
+}
+
+static struct ggml_backend_i blas_backend_i = {
+    /* .get_name                = */ ggml_backend_blas_get_name,
+    /* .free                    = */ ggml_backend_blas_free,
+    /* .set_tensor_async        = */ NULL,
+    /* .get_tensor_async        = */ NULL,
+    /* .cpy_tensor_async        = */ NULL,
+    /* .synchronize             = */ NULL,
+    /* .graph_plan_create       = */ NULL,
+    /* .graph_plan_free         = */ NULL,
+    /* .graph_plan_update       = */ NULL,
+    /* .graph_plan_compute      = */ NULL,
+    /* .graph_compute           = */ ggml_backend_blas_graph_compute,
+    /* .event_record            = */ NULL,
+    /* .event_wait              = */ NULL,
+};
+
+static ggml_guid_t ggml_backend_blas_guid(void) {
+    static ggml_guid guid = { 0x12, 0xa8, 0xae, 0xf4, 0xc0, 0x1e, 0x61, 0x97, 0x8f, 0xeb, 0x33, 0x04, 0xa1, 0x33, 0x51, 0x2d };
+    return &guid;
+}
+
+ggml_backend_t ggml_backend_blas_init(void) {
+    ggml_backend_blas_context * ctx = new ggml_backend_blas_context;
+
+    ggml_backend_t backend = new ggml_backend {
+        /* .guid      = */ ggml_backend_blas_guid(),
+        /* .interface = */ blas_backend_i,
+        /* .device    = */ ggml_backend_reg_dev_get(ggml_backend_blas_reg(), 0),
+        /* .context   = */ ctx,
+    };
+
+#if defined(OPENBLAS_VERSION) && defined(GGML_USE_OPENMP)
+    if (openblas_get_parallel() != OPENBLAS_OPENMP) {
+        GGML_LOG_DEBUG("%s: warning: ggml is using OpenMP, but OpenBLAS was compiled without OpenMP support\n", __func__);
+    }
+#endif
+
+#if defined(BLIS_ENABLE_CBLAS) && defined(GGML_USE_OPENMP) && !defined(BLIS_ENABLE_OPENMP)
+    GGML_LOG_DEBUG("%s: warning: ggml is using OpenMP, but BLIS was compiled without OpenMP support\n", __func__);
+#endif
+
+    return backend;
+}
+
+bool ggml_backend_is_blas(ggml_backend_t backend) {
+    return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_blas_guid());
+}
+
+void ggml_backend_blas_set_n_threads(ggml_backend_t backend_blas, int n_threads) {
+    GGML_ASSERT(ggml_backend_is_blas(backend_blas));
+
+    ggml_backend_blas_context * ctx = (ggml_backend_blas_context *)backend_blas->context;
+    ctx->n_threads = n_threads;
+}
+
+// device interface
+
+static const char * ggml_backend_blas_device_get_name(ggml_backend_dev_t dev) {
+    return "BLAS";
+
+    GGML_UNUSED(dev);
+}
+
+static const char * ggml_backend_blas_device_get_description(ggml_backend_dev_t dev) {
+    #if defined(GGML_BLAS_USE_ACCELERATE)
+        return "Accelerate";
+    #elif defined(GGML_BLAS_USE_MKL)
+        return "MKL";
+    #elif defined(GGML_BLAS_USE_BLIS)
+        return "BLIS";
+    #elif defined(GGML_BLAS_USE_NVPL)
+        return "NVPL";
+    #elif defined(OPENBLAS_VERSION)
+        return "OpenBLAS";
+    #else
+        return "BLAS";
+    #endif
+
+    GGML_UNUSED(dev);
+}
+
+static void ggml_backend_blas_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) {
+    // TODO
+    *free = 0;
+    *total = 0;
+
+    GGML_UNUSED(dev);
+}
+
+static enum ggml_backend_dev_type ggml_backend_blas_device_get_type(ggml_backend_dev_t dev) {
+    return GGML_BACKEND_DEVICE_TYPE_ACCEL;
+
+    GGML_UNUSED(dev);
+}
+
+static void ggml_backend_blas_device_get_props(ggml_backend_dev_t dev, struct ggml_backend_dev_props * props) {
+    props->name        = ggml_backend_blas_device_get_name(dev);
+    props->description = ggml_backend_blas_device_get_description(dev);
+    props->type        = ggml_backend_blas_device_get_type(dev);
+    ggml_backend_blas_device_get_memory(dev, &props->memory_free, &props->memory_total);
+    props->caps = {
+        /* .async                 = */ false,
+        /* .host_buffer           = */ false,
+        /* .buffer_from_host_ptr  = */ true,
+        /* .events                = */ false,
+    };
+}
+
+static ggml_backend_t ggml_backend_blas_device_init_backend(ggml_backend_dev_t dev, const char * params) {
+    return ggml_backend_blas_init();
+
+    GGML_UNUSED(dev);
+    GGML_UNUSED(params);
+}
+
+static ggml_backend_buffer_type_t ggml_backend_blas_device_get_buffer_type(ggml_backend_dev_t dev) {
+    return ggml_backend_cpu_buffer_type();
+
+    GGML_UNUSED(dev);
+}
+
+static ggml_backend_buffer_t ggml_backend_blas_device_buffer_from_host_ptr(ggml_backend_dev_t dev, void * ptr, size_t size, size_t max_tensor_size) {
+    return ggml_backend_cpu_buffer_from_ptr(ptr, size);
+
+    GGML_UNUSED(dev);
+    GGML_UNUSED(max_tensor_size);
+}
+
+static bool ggml_backend_blas_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
+    const struct ggml_tensor * src0 = op->src[0];
+    const struct ggml_tensor * src1 = op->src[1];
+
+    switch (op->op) {
+        case GGML_OP_NONE:
+        case GGML_OP_RESHAPE:
+        case GGML_OP_VIEW:
+        case GGML_OP_PERMUTE:
+        case GGML_OP_TRANSPOSE:
+            return true;
+
+        case GGML_OP_MUL_MAT:
+        {
+            // BLAS usually is only faster for large matrices
+            const struct ggml_tensor * src0 = op->src[0];
+            const struct ggml_tensor * src1 = op->src[1];
+
+            const int64_t ne10 = src1->ne[0];
+
+            const int64_t ne0 = op->ne[0];
+            const int64_t ne1 = op->ne[1];
+
+            // TODO: find the optimal value
+            const int64_t min_batch = 32;
+
+            return ggml_is_contiguous(src0) &&
+                   ggml_is_contiguous(src1) &&
+                   src1->type == GGML_TYPE_F32 &&
+                   (ne0 >= min_batch && ne1 >= min_batch && ne10 >= min_batch) &&
+                   (src0->type == GGML_TYPE_F32 || ggml_get_type_traits(src0->type)->to_float != NULL);
+        }
+
+        case GGML_OP_OUT_PROD:
+            return op->src[0]->type == GGML_TYPE_F32 &&
+                   op->src[1]->type == GGML_TYPE_F32 &&
+                   ggml_is_matrix(src0) &&
+                   ggml_is_matrix(src1) &&
+                   ggml_is_contiguous(src0) &&
+                   (ggml_is_contiguous(src1) || ggml_is_transposed(src1)) &&
+                   (src0->type == GGML_TYPE_F32 || ggml_get_type_traits(src0->type)->to_float != NULL);
+
+        default:
+            return false;
+
+    }
+
+    GGML_UNUSED(dev);
+}
+
+static bool ggml_backend_blas_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
+    return ggml_backend_buft_is_host(buft);
+
+    GGML_UNUSED(dev);
+}
+
+static const struct ggml_backend_device_i ggml_backend_blas_device_i = {
+    /* .get_name             = */ ggml_backend_blas_device_get_name,
+    /* .get_description      = */ ggml_backend_blas_device_get_description,
+    /* .get_memory           = */ ggml_backend_blas_device_get_memory,
+    /* .get_type             = */ ggml_backend_blas_device_get_type,
+    /* .get_props            = */ ggml_backend_blas_device_get_props,
+    /* .init_backend         = */ ggml_backend_blas_device_init_backend,
+    /* .get_buffer_type      = */ ggml_backend_blas_device_get_buffer_type,
+    /* .get_host_buffer_type = */ NULL,
+    /* .buffer_from_host_ptr = */ ggml_backend_blas_device_buffer_from_host_ptr,
+    /* .supports_op          = */ ggml_backend_blas_device_supports_op,
+    /* .supports_buft        = */ ggml_backend_blas_device_supports_buft,
+    /* .offload_op           = */ NULL,
+    /* .event_new            = */ NULL,
+    /* .event_free           = */ NULL,
+    /* .event_synchronize    = */ NULL,
+};
+
+// backend reg interface
+
+static const char * ggml_backend_blas_reg_get_name(ggml_backend_reg_t reg) {
+    return "BLAS";
+
+    GGML_UNUSED(reg);
+}
+
+static size_t ggml_backend_blas_reg_get_device_count(ggml_backend_reg_t reg) {
+    return 1;
+
+    GGML_UNUSED(reg);
+}
+
+static ggml_backend_dev_t ggml_backend_blas_reg_get_device(ggml_backend_reg_t reg, size_t index) {
+    GGML_ASSERT(index == 0);
+
+    static ggml_backend_device ggml_backend_blas_device = {
+        /* .iface   = */ ggml_backend_blas_device_i,
+        /* .reg     = */ reg,
+        /* .context = */ nullptr,
+    };
+
+    return &ggml_backend_blas_device;
+
+    GGML_UNUSED(reg);
+    GGML_UNUSED(index);
+}
+
+static void * ggml_backend_blas_get_proc_address(ggml_backend_reg_t reg, const char * name) {
+    if (std::strcmp(name, "ggml_backend_set_n_threads") == 0) {
+        return (void *)ggml_backend_blas_set_n_threads;
+    }
+    return NULL;
+
+    GGML_UNUSED(reg);
+    GGML_UNUSED(name);
+}
+
+static const struct ggml_backend_reg_i ggml_backend_blas_reg_i = {
+    /* .get_name         = */ ggml_backend_blas_reg_get_name,
+    /* .get_device_count = */ ggml_backend_blas_reg_get_device_count,
+    /* .get_device       = */ ggml_backend_blas_reg_get_device,
+    /* .get_proc_address = */ ggml_backend_blas_get_proc_address,
+};
+
+ggml_backend_reg_t ggml_backend_blas_reg(void) {
+    static struct ggml_backend_reg ggml_backend_blas_reg = {
+        /* .api_version = */ GGML_BACKEND_API_VERSION,
+        /* .iface       = */ ggml_backend_blas_reg_i,
+        /* .context     = */ NULL,
+    };
+
+    return &ggml_backend_blas_reg;
+}
+
+GGML_BACKEND_DL_IMPL(ggml_backend_blas_reg)

ggml/src/ggml-cann/CMakeLists.txt

@@ -0,0 +1,75 @@
+if ("cann${CANN_INSTALL_DIR}" STREQUAL "cann" AND DEFINED ENV{ASCEND_TOOLKIT_HOME})
+    set(CANN_INSTALL_DIR $ENV{ASCEND_TOOLKIT_HOME})
+    message(STATUS "CANN: updated CANN_INSTALL_DIR from ASCEND_TOOLKIT_HOME=$ENV{ASCEND_TOOLKIT_HOME}")
+endif()
+
+# Auto-detech Soc type and Soc version, if detect failed, will abort build
+set(SOC_VERSION "")
+function(detect_ascend_soc_type SOC_VERSION)
+    execute_process(
+        COMMAND bash -c "npu-smi info|awk -F' ' 'NF > 0 && NR==7 {print $3}'"
+        OUTPUT_VARIABLE npu_info
+        RESULT_VARIABLE npu_result
+        OUTPUT_STRIP_TRAILING_WHITESPACE
+    )
+    if("${npu_info}" STREQUAL "" OR ${npu_result})
+        message(FATAL_ERROR "Auto-detech ascend soc type failed, please specify manually or check ascend device working normally.")
+    endif()
+    set(${SOC_VERSION} "Ascend${npu_info}" PARENT_SCOPE)
+endfunction()
+
+if(NOT SOC_TYPE)
+    detect_ascend_soc_type(SOC_VERSION)
+    set(SOC_TYPE "${SOC_VERSION}")
+    message(STATUS "CANN: SOC_VERSION auto-detected is:${SOC_VERSION}")
+endif()
+
+string(TOLOWER ${SOC_TYPE} SOC_VERSION) # SOC_VERSION need lower
+
+# Construct Soc specify compile option: ASCEND_#Soc_Major_SN. Such as ASCEND_910B, ASCEND_310P.
+string(REGEX MATCH "[0-9]+[a-zA-Z]" SOC_TYPE_MAJOR_SN "${SOC_VERSION}")
+set(SOC_TYPE_COMPILE_OPTION "ASCEND_${SOC_TYPE_MAJOR_SN}")
+string(TOUPPER ${SOC_TYPE_COMPILE_OPTION} SOC_TYPE_COMPILE_OPTION)
+message(STATUS "CANN: SOC_VERSION =  ${SOC_VERSION}")
+
+if (CANN_INSTALL_DIR)
+    # Only Support Linux.
+    if (NOT UNIX)
+        message(FATAL_ERROR "CANN: CANN toolkit supports unix but not ${CMAKE_SYSTEM_NAME}")
+    endif()
+
+    # Supported platforms: x86-64, arm64
+    if (CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64")
+    elseif (CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64")
+    else()
+        message(FATAL_ERROR "CANN: CANN toolkit supports x86-64 and arm64 but not ${CMAKE_SYSTEM_PROCESSOR}")
+    endif()
+
+    # Set header and libs
+    set(CANN_INCLUDE_DIRS
+        ${CANN_INSTALL_DIR}/include
+        ${CANN_INSTALL_DIR}/include/aclnn
+        ${CANN_INSTALL_DIR}/acllib/include
+    )
+
+    list(APPEND CANN_LIBRARIES
+        ascendcl
+        nnopbase
+        opapi
+        acl_op_compiler
+    )
+
+    file(GLOB GGML_SOURCES_CANN "*.cpp")
+
+    ggml_add_backend_library(ggml-cann ${GGML_SOURCES_CANN})
+    target_link_libraries(ggml-cann PRIVATE ${CANN_LIBRARIES})
+    target_include_directories(ggml-cann PRIVATE ${CANN_INCLUDE_DIRS})
+    target_link_directories(ggml-cann PRIVATE ${CANN_INSTALL_DIR}/lib64)
+
+    target_compile_definitions(ggml-cann PRIVATE "-D${SOC_TYPE_COMPILE_OPTION}")
+
+    message(STATUS "CANN: CANN_INCLUDE_DIRS =  ${CANN_INCLUDE_DIRS}")
+    message(STATUS "CANN: CANN_LIBRARIES =  ${CANN_LIBRARIES}")
+else()
+    message(FATAL_ERROR "CANN: Can't find CANN_INSTALL_DIR, did you forget to source set_var.sh?")
+endif()

ggml/src/ggml-cann/acl_tensor.cpp

@@ -0,0 +1,183 @@
+/*
+ * Copyright (c) 2023-2024 The ggml authors
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#include "acl_tensor.h"
+
+#include <algorithm>
+#include <cstring>
+
+aclDataType ggml_cann_type_mapping(ggml_type type) {
+    switch (type) {
+        case GGML_TYPE_F32:
+            return ACL_FLOAT;
+        case GGML_TYPE_F16:
+            return ACL_FLOAT16;
+        case GGML_TYPE_BF16:
+            return ACL_BF16;
+        case GGML_TYPE_I8:
+            return ACL_INT8;
+        case GGML_TYPE_I16:
+            return ACL_INT16;
+        case GGML_TYPE_I32:
+            return ACL_INT32;
+        case GGML_TYPE_Q4_0:
+            return ACL_INT4;
+        case GGML_TYPE_Q8_0:
+            return ACL_INT8;
+        case GGML_TYPE_I64:
+            return ACL_INT64;
+        default:
+            return ACL_DT_UNDEFINED;
+    }
+    return ACL_DT_UNDEFINED;
+}
+
+aclTensor* ggml_cann_create_tensor(const ggml_tensor* tensor, int64_t* ne,
+                                   size_t* nb, int64_t dims, aclFormat format,
+                                   size_t offset) {
+    // If tensor is bcasted, Up to GGML_MAX_DIMS additional dimensions will be
+    // added.
+    int64_t acl_ne[GGML_MAX_DIMS * 2], acl_stride[GGML_MAX_DIMS * 2];
+
+    if (ne == nullptr) {
+        for (int i = 0; i < GGML_MAX_DIMS; i++) {
+            acl_ne[i] = tensor->ne[i];
+            // The step size of acl is in elements.
+            acl_stride[i] = tensor->nb[i] / ggml_element_size(tensor);
+        }
+    } else {
+        // With bcast
+        for (int i = 0; i < dims; i++) {
+            acl_ne[i] = ne[i];
+            acl_stride[i] = nb[i] / ggml_element_size(tensor);
+        }
+    }
+
+    int64_t final_dims = (dims == 0 ? GGML_MAX_DIMS : dims);
+    int64_t acl_storage_len = 1;
+    for (int i = 0; i < final_dims; i++) {
+        acl_storage_len += (acl_ne[i] - 1) * acl_stride[i];
+    }
+    size_t elem_offset = offset / ggml_element_size(tensor);
+    acl_storage_len += elem_offset;
+
+    // Reverse ne and stride.
+    std::reverse(acl_ne, acl_ne + final_dims);
+    std::reverse(acl_stride, acl_stride + final_dims);
+
+    aclTensor* acl_tensor = aclCreateTensor(
+        acl_ne, final_dims, ggml_cann_type_mapping(tensor->type), acl_stride,
+        elem_offset, format, &acl_storage_len, 1,
+        tensor->data);
+
+    return acl_tensor;
+}
+
+bool ggml_cann_need_bcast(const ggml_tensor* t0, const ggml_tensor* t1) {
+    for (int i = 0; i < GGML_MAX_DIMS; i++) {
+        if (t1->ne[i] != t0->ne[i] && t1->ne[i] != 1) {
+            return true;
+        }
+    }
+    return false;
+}
+
+int64_t ggml_cann_get_bcast_shape(const ggml_tensor* src0,
+                                  const ggml_tensor* src1,
+                                  int64_t* bcast_src0_ne,
+                                  int64_t* bcast_src1_ne, size_t* bcast_src0_nb,
+                                  size_t* bcast_src1_nb) {
+    GGML_ASSERT(ggml_can_repeat(src1, src0));
+    int bcast_dim_cnt = 0;
+    for (int i = 0; i < GGML_MAX_DIMS; i++) {
+        int64_t nr = src0->ne[i] / src1->ne[i];
+        bcast_src0_ne[bcast_dim_cnt] = src0->ne[i] / nr;
+        bcast_src1_ne[bcast_dim_cnt] = src1->ne[i];
+        bcast_src0_nb[bcast_dim_cnt] = src0->nb[i];
+        bcast_src1_nb[bcast_dim_cnt] = src1->nb[i];
+        bcast_dim_cnt++;
+        if (nr != 1) {
+            // Need to add an extra dim.
+            bcast_src0_ne[bcast_dim_cnt] = nr;
+            bcast_src1_ne[bcast_dim_cnt] = 1;
+            bcast_src0_nb[bcast_dim_cnt] = bcast_src0_nb[bcast_dim_cnt - 1] *
+                                           bcast_src0_ne[bcast_dim_cnt - 1];
+            bcast_src1_nb[bcast_dim_cnt] = bcast_src1_nb[bcast_dim_cnt - 1] *
+                                           bcast_src1_ne[bcast_dim_cnt - 1];
+            bcast_dim_cnt++;
+        }
+    }
+    return bcast_dim_cnt;
+}
+
+int64_t ggml_cann_get_mulmat_bcast_shape(
+    const int64_t* input_ne, const int64_t* weight_ne, const int64_t* dst_ne,
+    const size_t* input_nb, const size_t* weight_nb, const size_t* dst_nb,
+    int64_t* bcast_input_ne, int64_t* bcast_weight_ne, int64_t* bcast_dst_ne,
+    size_t* bcast_input_nb, size_t* bcast_weight_nb, size_t* bcast_dst_nb) {
+    // input and dst shoule in same shape, except first two dims.
+    GGML_ASSERT(input_ne[2] == dst_ne[2]);
+    GGML_ASSERT(input_ne[3] == dst_ne[3]);
+
+    int bcast_dim_cnt = 0;
+
+    // For mul_mat, a dimension needs to be added before the dimension that
+    // weight needs to be expanded to satisfy the bcast rule of matrix
+    // multiplication.
+    for (int i = 0; i < GGML_MAX_DIMS; i++) {
+        int64_t nr = input_ne[i] / weight_ne[i];
+        // Do not use bcast in the first two dimensions because we only support
+        // the bcast batch dimension. Just copy them.
+        if (i < 2 || nr == 1) {
+            bcast_input_ne[bcast_dim_cnt] = input_ne[i];
+            bcast_weight_ne[bcast_dim_cnt] = weight_ne[i];
+            bcast_dst_ne[bcast_dim_cnt] = dst_ne[i];
+
+            bcast_input_nb[bcast_dim_cnt] = input_nb[i];
+            bcast_weight_nb[bcast_dim_cnt] = weight_nb[i];
+            bcast_dst_nb[bcast_dim_cnt] = dst_nb[i];
+            bcast_dim_cnt++;
+        } else {
+            // Need to add an extra dim.
+            bcast_input_ne[bcast_dim_cnt] = nr;
+            bcast_dst_ne[bcast_dim_cnt] = nr;
+            bcast_weight_ne[bcast_dim_cnt] = 1;
+            bcast_input_nb[bcast_dim_cnt] = input_nb[i];
+            bcast_dst_nb[bcast_dim_cnt] = dst_nb[i];
+            bcast_weight_nb[bcast_dim_cnt] = weight_nb[i];
+            bcast_dim_cnt++;
+
+            bcast_input_ne[bcast_dim_cnt] = input_ne[i] / nr;
+            bcast_dst_ne[bcast_dim_cnt] = dst_ne[i] / nr;
+            bcast_weight_ne[bcast_dim_cnt] = weight_ne[i];
+            bcast_input_nb[bcast_dim_cnt] = bcast_input_nb[bcast_dim_cnt - 1] *
+                                            bcast_input_ne[bcast_dim_cnt - 1];
+            bcast_dst_nb[bcast_dim_cnt] = bcast_dst_nb[bcast_dim_cnt - 1] *
+                                          bcast_dst_ne[bcast_dim_cnt - 1];
+            bcast_weight_nb[bcast_dim_cnt] =
+                bcast_weight_nb[bcast_dim_cnt - 1] *
+                bcast_weight_ne[bcast_dim_cnt - 1];
+            bcast_dim_cnt++;
+        }
+    }
+    return bcast_dim_cnt;
+}

ggml/src/ggml-cann/acl_tensor.h

@@ -0,0 +1,258 @@
+/*
+ * Copyright (c) 2023-2024 The ggml authors
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef CANN_ACL_TENSOR_H
+#define CANN_ACL_TENSOR_H
+
+#include <algorithm>
+#include <cstring>
+
+#include <aclnn/aclnn_base.h>
+#include "common.h"
+
+/**
+ * @brief	Maps a ggml_type to its corresponding aclDataType.
+ *
+ * @details	This function takes a ggml_type as input and returns the corresponding
+ *			aclDataType. It supports mapping for various ggml_types. If the input type
+ *			does not match any of the predefined ggml_types, the function returns
+ *          ACL_DT_UNDEFINED.
+ *
+ * @param	type    The ggml_type to be mapped.
+ * @return	The corresponding aclDataType. If the input type is not recognized,
+ *			ACL_DT_UNDEFINED is returned.
+ */
+aclDataType ggml_cann_type_mapping(ggml_type type);
+
+/**
+ * @brief   Creates an ACL tensor from a ggml_tensor with optional shape.
+ *
+ * @details This function creates an ACL tensor based on the properties of the
+ *          provided ggml_tensor. It supports customer shape by adjusting dimensions
+ *          and strides accordingly. If customer shape is applied, additional
+ *          dimensions and strides are calculated based on the provided parameters.
+ *
+ * @param   tensor      Pointer to the ggml_tensor to be converted to ACL tensor.
+ * @param   ne          Pointer to an array containing dimensions. Defaults to nullptr
+ *                      if no customer shape is applied.
+ * @param   nb          Pointer to an array containing strides. Defaults to nullptr
+ *                      if no customer shape is applied.
+ * @param   dims        Number of dimensions in the tensor. Defaults to 0 if no customer
+ *                      shape is applied.
+ * @param   format      ACL tensor format. Defaults to ACL_FORMAT_ND.
+ * @param   offset      Offset in bytes for the ACL tensor data. Defaults to 0.
+ * @return  Pointer to the created ACL tensor.
+ */
+aclTensor* ggml_cann_create_tensor(const ggml_tensor* tensor, int64_t* ne = nullptr,
+                             size_t* nb = nullptr, int64_t dims = 0,
+                             aclFormat format = ACL_FORMAT_ND,
+                             size_t offset = 0);
+
+/**
+ * @brief   Template for creating an ACL tensor from provided parameters. typename TYPE
+ *          should be size_t or float.
+ *
+ * @details This function creates an ACL tensor using the provided data pointer,
+ *          data type, dimensions, strides, format, offset, and additional parameters.
+ *          It calculates necessary dimensions and strides based on the provided ne and nb
+ *          arrays, adjusting them for the ACL tensor creation. The ACL storage length
+ *          is also calculated based on the provided dimensions and strides.
+ *
+ * @param   data_ptr    Pointer to the data buffer for the ACL tensor.
+ * @param   dtype       ACL data type of the tensor.
+ * @param   type_size   Size of each element in the tensor data buffer.
+ * @param   ne          Pointer to an array containing tensor dimensions.
+ * @param   nb          Pointer to an array containing tensor strides.
+ * @param   dims        Number of dimensions of the tensor.
+ * @param   format      ACL tensor format. Defaults to ACL_FORMAT_ND.
+ * @param   offset      Offset in bytes for the ACL tensor data. Defaults to 0.
+ * @return  Pointer to the created ACL tensor.
+ */
+template<typename TYPE>
+aclTensor* ggml_cann_create_tensor(void* data_ptr, aclDataType dtype,
+                                   TYPE type_size, int64_t* ne, TYPE* nb,
+                                   int64_t dims,
+                                   aclFormat format = ACL_FORMAT_ND,
+                                   size_t offset = 0) {
+    int64_t tmp_ne[GGML_MAX_DIMS * 2];
+    int64_t tmp_stride[GGML_MAX_DIMS * 2];
+
+    memcpy(tmp_ne, ne, dims * sizeof(int64_t));
+    for (int i = 0; i < dims; i++) {
+        tmp_stride[i] = nb[i] / type_size;
+    }
+
+    int64_t acl_storage_len = 1;
+    for (int i = 0; i < dims; i++) {
+        acl_storage_len += (tmp_ne[i] - 1) * tmp_stride[i];
+    }
+
+    std::reverse(tmp_ne, tmp_ne + dims);
+    std::reverse(tmp_stride, tmp_stride + dims);
+
+    aclTensor* acl_tensor =
+        aclCreateTensor(tmp_ne, dims, dtype, tmp_stride, offset / type_size,
+                        format, &acl_storage_len, 1, data_ptr);
+
+    return acl_tensor;
+}
+
+/**
+ * @brief   Checks if tensors require broadcasting based on their shapes.
+ *
+ * @details This function determines if two ggml_tensors need to be broadcasted for
+ *          element-wise operations. Broadcasting is necessary if the shapes of the
+ *          tensors are not identical and no dimension in either tensor equals 1.
+ *
+ * @param   t0      Pointer to the first ggml_tensor.
+ * @param   t1      Pointer to the second ggml_tensor.
+ * @return  True if broadcasting is needed, False otherwise.
+ *
+ * @remarks This function iterates over the dimensions of t0 and t1. It checks if each
+ *          dimension in t1 differs from t0's corresponding dimension and is not equal
+ *          to 1. If such a dimension is found, broadcasting is required to align t1
+ *          with t0 for element-wise operations.
+ */
+bool ggml_cann_need_bcast(const ggml_tensor* t0, const ggml_tensor* t1);
+
+/**
+ * @brief   Computes broadcast shapes and strides for two ggml_tensors.
+ *
+ * @details This function calculates the broadcast shapes and strides for two ggml_tensors,
+ *          following the broadcasting rules similar to numpy. It adjusts dimensions and
+ *          strides to ensure compatibility for element-wise operations where one tensor
+ *          can be broadcasted to match the shape of another tensor.
+ *
+ * @param   src0                Pointer to the first ggml_tensor.
+ * @param   src1                Pointer to the second ggml_tensor.
+ * @param   bcast_ne_src0       Output array to store broadcasted dimensions for src0.
+ * @param   bcast_ne_src1       Output array to store broadcasted dimensions for src1.
+ * @param   bcast_nb_src0       Output array to store broadcasted strides for src0.
+ * @param   bcast_nb_src1       Output array to store broadcasted strides for src1.
+ * @return  Number of dimensions in the broadcasted shape.
+ *
+ * @pre     ggml_can_repeat(src1, src0) must return true, indicating src1 can be broadcasted
+ *          to match src0.
+ *
+ * @remarks This function iterates over the dimensions of src0 and src1, calculating the
+ *          necessary broadcast dimensions and strides. If a dimension requires broadcasting
+ *          (i.e., its size in src1 is smaller than in src0), an additional dimension is
+ *          added with size calculated to match src0's dimension. This adjustment ensures
+ *          that src1 can be element-wise broadcasted to src0's shape.
+ *
+ *  How it works:
+ *
+ *  if dim0 has padding.
+ *  a -> (2, 2) padding = 2
+ *   a: [[1, 2, *, *]
+ *       [2, 3, *, *]]
+ *  nb = (8, 4, 2)
+ *
+ *  if a should bcast with b -> (2, 4)
+ *  b' -> (2, 2, 2)
+ *  b : [[1, 2, 3, 4, *, *]
+ *       [5, 6, 7, 8, *, *]]
+ *  nb = (12, 6, 1)
+ *
+ *  after bcast:
+ *  a' -> (2, 1, 2)
+ *  a': [[[1, 2], *, *]
+ *       [[2, 3], *, *]]
+ *  nb = (8, 4, 2, 1)
+ *
+ *  b' : [[[1, 2], [3, 4], *, *]
+ *        [[5, 6], [7, 8], *, *]]
+ *  nb = (12, 6, 2, 1)
+ *  \endcode
+ *
+ *  dim1 in a inserted dim, should add nb for dim1,
+ *  and all other nb moves to next in order.
+ */
+int64_t ggml_cann_get_bcast_shape(const ggml_tensor* src0, const ggml_tensor* src1,
+                        int64_t* bcast_ne_src0, int64_t* bcast_ne_src1,
+                        size_t* bcast_nb_src0, size_t* bcast_nb_src1);
+
+// Bcast macro to avoid duplicate code.
+#define BCAST_SHAPE(src0, src1)                                              \
+    int64_t bcast_##src0##_ne[GGML_MAX_DIMS * 2];                            \
+    int64_t bcast_##src1##_ne[GGML_MAX_DIMS * 2];                            \
+    size_t bcast_##src0##_nb[GGML_MAX_DIMS * 2];                             \
+    size_t bcast_##src1##_nb[GGML_MAX_DIMS * 2];                             \
+    int64_t bcast_dims = ggml_cann_get_bcast_shape(                          \
+        src0, src1, bcast_##src0##_ne, bcast_##src1##_ne, bcast_##src0##_nb, \
+        bcast_##src1##_nb);
+
+#define BCAST_PARAM(tensor) bcast_##tensor##_ne, bcast_##tensor##_nb, bcast_dims
+
+/**
+ * @brief Calculates broadcast shapes for matrix multiplication.
+ *
+ * @details This function computes the broadcast shapes required for matrix multiplication
+ *          based on the input, weight, and destination tensor shapes. It ensures that the
+ *          dimensions of weight tensors are expanded appropriately to satisfy matrix
+ *          multiplication broadcast rules.
+ *
+ * @param input_ne      Array containing the dimensions of the input tensor.
+ * @param weight_ne     Array containing the dimensions of the weight tensor.
+ * @param dst_ne        Array containing the dimensions of the destination tensor.
+ * @param input_nb      Array containing the strides of the input tensor.
+ * @param weight_nb     Array containing the strides of the weight tensor.
+ * @param dst_nb        Array containing the strides of the destination tensor.
+ * @param bcast_input_ne    Output array for broadcasted input tensor dimensions.
+ * @param bcast_weight_ne   Output array for broadcasted weight tensor dimensions.
+ * @param bcast_dst_ne      Output array for broadcasted destination tensor dimensions.
+ * @param bcast_input_nb    Output array for broadcasted input tensor strides.
+ * @param bcast_weight_nb   Output array for broadcasted weight tensor strides.
+ * @param bcast_dst_nb      Output array for broadcasted destination tensor strides.
+ * @return The number of dimensions in the broadcasted tensors.
+ *
+ * @remarks This function iterates over the tensor dimensions and calculates the broadcast
+ *          shapes needed for matrix multiplication. It ensures that dimensions where
+ *          weight tensor requires expansion are appropriately handled to conform with
+ *          broadcasting rules.
+ * @note compare with ggml_cann_get_bcast_shape, mul_mat broadcast need add this new dim
+ *       before cast dim.
+ * @sa ggml_cann_get_bcast_shape
+ */
+int64_t ggml_cann_get_mulmat_bcast_shape(
+    const int64_t* input_ne, const int64_t* weight_ne, const int64_t* dst_ne,
+    const size_t* input_nb, const size_t* weight_nb, const size_t* dst_nb,
+    int64_t* bcast_input_ne, int64_t* bcast_weight_ne, int64_t* bcast_dst_ne,
+    size_t* bcast_input_nb, size_t* bcast_weight_nb, size_t* bcast_dst_nb);
+
+// Bcast macro to avoid duplicate code.
+#define BCAST_MUL_MAT_SHAPE(input, weight, dst)                         \
+    int64_t bcast_##input##_ne[GGML_MAX_DIMS * 2];                      \
+    int64_t bcast_##weight##_ne[GGML_MAX_DIMS * 2];                     \
+    int64_t bcast_##dst##_ne[GGML_MAX_DIMS * 2];                        \
+    size_t bcast_##input##_nb[GGML_MAX_DIMS * 2];                       \
+    size_t bcast_##weight##_nb[GGML_MAX_DIMS * 2];                      \
+    size_t bcast_##dst##_nb[GGML_MAX_DIMS * 2];                         \
+    int64_t bcast_dims = ggml_cann_get_mulmat_bcast_shape(              \
+        input->ne, weight->ne, dst->ne, input->nb, weight->nb, dst->nb, \
+        bcast_##input##_ne, bcast_##weight##_ne, bcast_##dst##_ne,      \
+        bcast_##input##_nb, bcast_##weight##_nb, bcast_##dst##_nb);
+
+#define BCAST_MUL_MAT_PARAM(tensor) \
+    bcast_##tensor##_ne, bcast_##tensor##_nb, bcast_dims
+
+#endif  // CANN_ACL_TENSOR_H

ggml/src/ggml-cann/common.h

@@ -0,0 +1,425 @@
+/*
+ * Copyright (c) 2023-2024 The ggml authors
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef CANN_COMMON_H
+#define CANN_COMMON_H
+
+#include <acl/acl.h>
+
+#include <cstdio>
+#include <iostream>
+#include <map>
+#include <memory>
+#include <string>
+#include <vector>
+#include <atomic>
+#include <condition_variable>
+#include <mutex>
+#include <thread>
+#include <unistd.h>
+#include <functional>
+#include <optional>
+
+#include "../include/ggml-cann.h"
+#include "../include/ggml.h"
+#include "../ggml-impl.h"
+
+#define MATRIX_ROW_PADDING 512
+#define GGML_CANN_MAX_STREAMS 8
+
+/**
+ * @brief Handles CANN-related errors by printing an error message and
+ *        terminating the program.
+ * @param stmt The statement that caused the error.
+ * @param func The function in which the error occurred.
+ * @param file The file in which the error occurred.
+ * @param line The line number at which the error occurred.
+ * @param msg The error message.
+ */
+[[noreturn]] void ggml_cann_error(const char* stmt, const char* func,
+                                  const char* file, int line, const char* msg);
+
+/**
+ * @brief Checks the result of a CANN function call and invokes the error
+ *        handler if the call fails.
+ * @param stmt The CANN function call to check.
+ * @param success The success code that indicates the call was successful.
+ * @param error_fn The function to call to retrieve the error message.
+ */
+#define ACL_CHECK_GEN(stmt, success, error_fn)                                \
+    do {                                                                      \
+        int err_code = (stmt);                                                \
+        if (err_code != (success)) {                                          \
+            ggml_cann_error(#stmt, __func__, __FILE__, __LINE__, error_fn()); \
+        }                                                                     \
+    } while (0);
+
+#define ACL_CHECK(stmt) ACL_CHECK_GEN(stmt, 0, aclGetRecentErrMsg)
+
+/**
+ * @brief Contains information about CANN devices.
+ */
+struct ggml_cann_device_info {
+    /**
+     * @brief Number of CANN devices available.
+     */
+    int32_t device_count;
+
+    /**
+     * @brief Information about a single CANN device.
+     */
+    struct cann_device_info {
+        int cc;                 /**< Compute capability.                   */
+        size_t smpb;            /**< Maximum shared memory per block.      */
+        bool vmm;               /**< Virtual memory support.               */
+        size_t vmm_granularity; /**< Granularity of virtual memory.        */
+        size_t total_vram;      /**< Total video RAM available on the device. */
+    };
+
+    cann_device_info devices[GGML_CANN_MAX_DEVICES] =
+        {}; /**< Array of CANN device information. */
+};
+
+const ggml_cann_device_info& ggml_cann_info();
+
+void ggml_cann_set_device(int32_t device);
+int32_t ggml_cann_get_device();
+
+std::optional<std::string> get_env(const std::string& name);
+bool parse_bool(const std::string& value);
+
+/**
+ * @brief Abstract base class for memory pools used by CANN.
+ */
+struct ggml_cann_pool {
+    /**
+     * @brief Virtual destructor for the memory pool.
+     */
+    virtual ~ggml_cann_pool() = default;
+
+    /**
+     * @brief Allocates memory from the pool.
+     *
+     * @param size         The size of the memory block to allocate.
+     * @param actual_size  Pointer to a variable where the actual allocated size
+     *                     will be stored.
+     * @return             Pointer to the allocated memory block.
+     */
+    virtual void* alloc(size_t size, size_t* actual_size) = 0;
+
+    /**
+     * @brief Frees a previously allocated memory block.
+     *
+     * @param ptr   Pointer to the memory block to free.
+     * @param size  Size of the memory block to free.
+     * @note Note that all CANN opertors are running async. Make sure memory is
+     *       still avaiable before this operator finished.
+     */
+    virtual void free(void* ptr, size_t size) = 0;
+};
+
+/**
+ * @brief RAII wrapper for managing memory allocations from a CANN memory pool.
+ */
+struct ggml_cann_pool_alloc {
+    ggml_cann_pool* pool = nullptr; /**< Pointer to the memory pool. */
+    void* ptr = nullptr;    /**< Pointer to the allocated memory block. */
+    size_t actual_size = 0; /**< Actual size of the allocated memory block. */
+
+    /**
+     * @brief Default constructor.
+     */
+    ggml_cann_pool_alloc() = default;
+
+    /**
+     * @brief Constructor that initializes the memory pool.
+     * @param pool Reference to the memory pool.
+     */
+    explicit ggml_cann_pool_alloc(ggml_cann_pool& pool) : pool(&pool) {}
+
+    /**
+     * @brief Constructor that initializes the memory pool and allocates memory.
+     * @param pool Reference to the memory pool.
+     * @param size Size of the memory block to allocate.
+     */
+    ggml_cann_pool_alloc(ggml_cann_pool& pool, size_t size) : pool(&pool) {
+        alloc(size);
+    }
+
+    /**
+     * @brief Destructor that frees the allocated memory block.
+     */
+    ~ggml_cann_pool_alloc() {
+        if (ptr != nullptr) {
+            pool->free(ptr, actual_size);
+        }
+    }
+
+    /**
+     * @brief Allocates memory from the pool.
+     * @param size Size of the memory block to allocate.
+     * @return Pointer to the allocated memory block.
+     */
+    void* alloc(size_t size) {
+        GGML_ASSERT(pool != nullptr);
+        GGML_ASSERT(ptr == nullptr);
+        ptr = pool->alloc(size, &this->actual_size);
+        return ptr;
+    }
+
+    /**
+     * @brief Allocates memory from a specific memory pool.
+     * @param pool Reference to the memory pool.
+     * @param size Size of the memory block to allocate.
+     * @return Pointer to the allocated memory block.
+     */
+    void* alloc(ggml_cann_pool& pool, size_t size) {
+        this->pool = &pool;
+        return alloc(size);
+    }
+
+    /**
+     * @brief Gets the pointer to the allocated memory block.
+     * @return Pointer to the allocated memory block.
+     */
+    void* get() { return ptr; }
+
+    // Deleted copy constructor
+    ggml_cann_pool_alloc(const ggml_cann_pool_alloc&) = delete;
+
+    // Deleted move constructor
+    ggml_cann_pool_alloc(ggml_cann_pool_alloc&&) = delete;
+
+    // Deleted copy assignment operator
+    ggml_cann_pool_alloc& operator=(const ggml_cann_pool_alloc&) = delete;
+
+    // Deleted move assignment operator
+    ggml_cann_pool_alloc& operator=(ggml_cann_pool_alloc&&) = delete;
+};
+
+/**
+ * @brief Function pointer type for ACLNN operator calls.
+ */
+using aclnn_func_t = aclnnStatus (*)(void*, uint64_t, aclOpExecutor*, aclrtStream);
+
+/**
+ * @brief Base class for all CANN tasks to be submitted to the task queue.
+ *
+ * Users should override the run_task() method with actual task logic.
+ */
+class cann_task {
+public:
+    virtual void run_task() {}
+};
+
+/**
+ * @brief A lock-free ring-buffer based task queue for asynchronously executing cann_task instances.
+ */
+class cann_task_queue {
+public:
+    /**
+     * @brief Constructs a task queue with a fixed power-of-two capacity for a specific device.
+     *
+     * @param capacity Queue capacity. Must be a power of 2.
+     * @param device Target device ID (used for context setting).
+     */
+    explicit cann_task_queue(size_t capacity, int32_t device)
+        : buffer_(capacity), capacity_(capacity), head_(0), tail_(0),
+          running_(false), device_(device) {
+        GGML_ASSERT((capacity & (capacity - 1)) == 0 && "capacity must be power of 2");
+        mask_ = capacity_ - 1;
+    }
+
+    /**
+     * @brief Attempts to enqueue a task into the queue.
+     *
+     * @param item Unique pointer to the task.
+     * @return true if the task was successfully enqueued, false if the queue was full.
+     */
+    bool enqueue(std::unique_ptr<cann_task>&& item) {
+        size_t next_tail = (tail_ + 1) & mask_;
+
+        if (next_tail == head_) {
+            return false;
+        }
+
+        buffer_[tail_] = std::move(item);
+        std::atomic_thread_fence(std::memory_order_release);
+        tail_ = next_tail;
+
+        return true;
+    }
+
+    /**
+     * @brief Submits a task to the queue, and starts the worker thread if not already running.
+     *
+     * @param task Task to be submitted.
+     */
+    void submit_task(std::unique_ptr<cann_task>&& task) {
+        while(!enqueue(std::move(task))) {
+            std::this_thread::yield();
+            continue;
+        }
+
+        if (!running_) {
+            running_ = true;
+            thread_ = std::thread(&cann_task_queue::execute, this);
+        }
+
+    }
+
+    /**
+     * @brief Waits until the queue is completely empty and no tasks are being processed.
+     */
+    void wait() {
+        while (running_ && head_ != tail_) {
+            std::this_thread::yield();
+            continue;
+        }
+    }
+
+    /**
+     * @brief Stops the task queue and joins the worker thread.
+     */
+    void stop() {
+        running_ = false;
+        if (thread_.joinable()) {
+            thread_.join();
+        }
+    }
+
+private:
+    /**
+     * @brief Worker thread function that continuously dequeues and executes tasks.
+     */
+    void execute() {
+        ggml_cann_set_device(device_);
+
+        while (running_) {
+            if(head_ == tail_) {
+                std::this_thread::yield();
+                continue;
+            }
+
+            std::atomic_thread_fence(std::memory_order_acquire);
+            buffer_[head_]->run_task();
+            buffer_[head_].reset();
+            head_ = (head_ + 1) & mask_;
+        }
+    }
+
+    std::vector<std::unique_ptr<cann_task>> buffer_;
+    const size_t capacity_;
+    size_t mask_;
+    size_t head_;
+    size_t tail_;
+    bool running_;
+    std::thread thread_;
+    int32_t device_;
+};
+
+/**
+ * @brief Context for managing CANN backend operations.
+ */
+struct ggml_backend_cann_context {
+    int32_t device;                  /**< Device ID. */
+    std::string name;                /**< Name of the device. */
+    std::string description;         /**< Description of the device. */
+    aclrtEvent copy_event = nullptr; /**< Event for managing copy operations. */
+    cann_task_queue task_queue;
+    bool async_mode;
+
+    aclrtStream streams[GGML_CANN_MAX_STREAMS] = {nullptr}; /**< Array of streams for the device. */
+
+    /**
+     * @brief Constructor for initializing the context with a given device.
+     * @param device Device ID.
+     */
+    explicit ggml_backend_cann_context(int device)
+        : device(device), name("CANN" + std::to_string(device)), task_queue(1024, device) {
+        ggml_cann_set_device(device);
+        description = aclrtGetSocName();
+
+        async_mode = parse_bool(get_env("GGML_CANN_ASYNC_MODE").value_or(""));
+        GGML_LOG_INFO("%s: device %d async operator submission is %s\n", __func__,
+            device, async_mode ? "ON" : "OFF");
+    }
+
+    /**
+     * @brief Destructor for cleaning up resources.
+     */
+    ~ggml_backend_cann_context() {
+        ggml_cann_set_device(device);
+        task_queue.stop();
+        if (copy_event != nullptr) {
+            ACL_CHECK(aclrtDestroyEvent(copy_event));
+        }
+        for (int i = 0; i < GGML_CANN_MAX_STREAMS; ++i) {
+            if (streams[i] != nullptr) {
+                ACL_CHECK(aclrtDestroyStream(streams[i]));
+            }
+        }
+    }
+
+    /**
+     * @brief Get or create a stream for a given index.
+     * @param stream Index of the stream.
+     * @return The stream corresponding to the given index.
+     */
+    aclrtStream stream(int stream) {
+        if (streams[stream] == nullptr) {
+            ggml_cann_set_device(device);
+            ACL_CHECK(aclrtCreateStream(&streams[stream]));
+        }
+        return streams[stream];
+    }
+
+    /**
+     * @brief Get or create the default stream (index 0).
+     * @return The default stream.
+     */
+    aclrtStream stream() { return stream(0); }
+
+    // TODO: each stream should have a memory pool.
+    std::unique_ptr<ggml_cann_pool>
+        mem_pool; /**< Memory pool for the device. */
+
+    /**
+     * @brief Create a new memory pool for a given device.
+     * @param device Device ID.
+     * @return A unique pointer to the new memory pool.
+     */
+    static std::unique_ptr<ggml_cann_pool> new_pool_for_device(int device);
+
+    /**
+     * @brief Get or create the memory pool for the context.
+     * @return Reference to the memory pool.
+     */
+    ggml_cann_pool& pool() {
+        if (mem_pool == nullptr) {
+            mem_pool = new_pool_for_device(device);
+        }
+        return *mem_pool;
+    }
+};
+
+#endif  // CANN_COMMON_H

ggml/src/ggml-cpu/CMakeLists.txt

@@ -0,0 +1,600 @@
+function(ggml_add_cpu_backend_features cpu_name arch)
+    # The feature detection code is compiled as a separate target so that
+    # it can be built without the architecture flags
+    # Since multiple variants of the CPU backend may be included in the same
+    # build, using set_source_files_properties() to set the arch flags is not possible
+    set(GGML_CPU_FEATS_NAME ${cpu_name}-feats)
+    add_library(${GGML_CPU_FEATS_NAME} OBJECT ggml-cpu/arch/${arch}/cpu-feats.cpp)
+    target_include_directories(${GGML_CPU_FEATS_NAME} PRIVATE . ../include)
+    target_compile_definitions(${GGML_CPU_FEATS_NAME} PRIVATE ${ARGN})
+    target_compile_definitions(${GGML_CPU_FEATS_NAME} PRIVATE GGML_BACKEND_DL GGML_BACKEND_BUILD GGML_BACKEND_SHARED)
+    set_target_properties(${GGML_CPU_FEATS_NAME} PROPERTIES POSITION_INDEPENDENT_CODE ON)
+    target_link_libraries(${cpu_name} PRIVATE ${GGML_CPU_FEATS_NAME})
+endfunction()
+
+function(ggml_add_cpu_backend_variant_impl tag_name)
+    if (tag_name)
+        set(GGML_CPU_NAME ggml-cpu-${tag_name})
+    else()
+        set(GGML_CPU_NAME ggml-cpu)
+    endif()
+
+    ggml_add_backend_library(${GGML_CPU_NAME})
+
+    list (APPEND GGML_CPU_SOURCES
+        ggml-cpu/ggml-cpu.c
+        ggml-cpu/ggml-cpu.cpp
+        ggml-cpu/repack.cpp
+        ggml-cpu/repack.h
+        ggml-cpu/hbm.cpp
+        ggml-cpu/hbm.h
+        ggml-cpu/quants.c
+        ggml-cpu/quants.h
+        ggml-cpu/traits.cpp
+        ggml-cpu/traits.h
+        ggml-cpu/amx/amx.cpp
+        ggml-cpu/amx/amx.h
+        ggml-cpu/amx/mmq.cpp
+        ggml-cpu/amx/mmq.h
+        ggml-cpu/ggml-cpu-impl.h
+        ggml-cpu/common.h
+        ggml-cpu/binary-ops.h
+        ggml-cpu/binary-ops.cpp
+        ggml-cpu/unary-ops.h
+        ggml-cpu/unary-ops.cpp
+        ggml-cpu/simd-mappings.h
+        ggml-cpu/vec.h
+        ggml-cpu/vec.cpp
+        ggml-cpu/ops.h
+        ggml-cpu/ops.cpp
+        )
+
+    target_compile_features(${GGML_CPU_NAME} PRIVATE c_std_11 cxx_std_17)
+    target_include_directories(${GGML_CPU_NAME} PRIVATE . ggml-cpu)
+
+    if (APPLE AND GGML_ACCELERATE)
+        find_library(ACCELERATE_FRAMEWORK Accelerate)
+        if (ACCELERATE_FRAMEWORK)
+            message(STATUS "Accelerate framework found")
+
+            target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_USE_ACCELERATE)
+            target_compile_definitions(${GGML_CPU_NAME} PRIVATE ACCELERATE_NEW_LAPACK)
+            target_compile_definitions(${GGML_CPU_NAME} PRIVATE ACCELERATE_LAPACK_ILP64)
+
+            target_link_libraries(${GGML_CPU_NAME} PRIVATE ${ACCELERATE_FRAMEWORK})
+        else()
+            message(WARNING "Accelerate framework not found")
+        endif()
+    endif()
+
+    if (GGML_OPENMP)
+        find_package(OpenMP)
+        if (OpenMP_FOUND)
+            set(GGML_OPENMP_ENABLED "ON" CACHE INTERNAL "")
+            target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_USE_OPENMP)
+
+            target_link_libraries(${GGML_CPU_NAME} PRIVATE OpenMP::OpenMP_C OpenMP::OpenMP_CXX)
+        else()
+            set(GGML_OPENMP_ENABLED "OFF" CACHE INTERNAL "")
+            message(WARNING "OpenMP not found")
+        endif()
+    endif()
+
+    if (GGML_LLAMAFILE)
+        target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_USE_LLAMAFILE)
+
+        list(APPEND GGML_CPU_SOURCES
+                    ggml-cpu/llamafile/sgemm.cpp
+                    ggml-cpu/llamafile/sgemm.h)
+    endif()
+
+    if (GGML_CPU_HBM)
+        find_library(memkind memkind REQUIRED)
+
+        message(STATUS "Using memkind for CPU HBM")
+
+        target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_USE_CPU_HBM)
+
+        target_link_libraries(${GGML_CPU_NAME} PUBLIC memkind)
+    endif()
+
+    if (GGML_SYSTEM_ARCH STREQUAL "ARM")
+        message(STATUS "ARM detected")
+        list(APPEND GGML_CPU_SOURCES
+            ggml-cpu/arch/arm/quants.c
+            ggml-cpu/arch/arm/repack.cpp
+            )
+
+        if (MSVC AND NOT CMAKE_C_COMPILER_ID STREQUAL "Clang")
+            message(FATAL_ERROR "MSVC is not supported for ARM, use clang")
+        else()
+            check_cxx_compiler_flag(-mfp16-format=ieee GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E)
+            if (NOT "${GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E}" STREQUAL "")
+                list(APPEND ARCH_FLAGS -mfp16-format=ieee)
+            endif()
+
+            if (GGML_NATIVE)
+                # -mcpu=native does not always enable all the features in some compilers,
+                # so we check for them manually and enable them if available
+
+                execute_process(
+                    COMMAND ${CMAKE_C_COMPILER} -mcpu=native -E -v -
+                    INPUT_FILE "/dev/null"
+                    OUTPUT_QUIET
+                    ERROR_VARIABLE ARM_MCPU
+                    RESULT_VARIABLE ARM_MCPU_RESULT
+                )
+                if (NOT ARM_MCPU_RESULT)
+                    string(REGEX MATCH "-mcpu=[^ ']+" ARM_MCPU_FLAG "${ARM_MCPU}")
+                endif()
+                if ("${ARM_MCPU_FLAG}" STREQUAL "")
+                    set(ARM_MCPU_FLAG -mcpu=native)
+                    message(STATUS "ARM -mcpu not found, -mcpu=native will be used")
+                endif()
+
+                include(CheckCXXSourceRuns)
+
+                function(check_arm_feature tag code)
+                    set(CMAKE_REQUIRED_FLAGS_SAVE ${CMAKE_REQUIRED_FLAGS})
+                    set(CMAKE_REQUIRED_FLAGS "${ARM_MCPU_FLAG}+${tag}")
+                    check_cxx_source_runs("${code}" GGML_MACHINE_SUPPORTS_${tag})
+                    if (GGML_MACHINE_SUPPORTS_${tag})
+                        set(ARM_MCPU_FLAG_FIX "${ARM_MCPU_FLAG_FIX}+${tag}" PARENT_SCOPE)
+                    else()
+                        set(CMAKE_REQUIRED_FLAGS "${ARM_MCPU_FLAG}+no${tag}")
+                        check_cxx_source_compiles("int main() { return 0; }" GGML_MACHINE_SUPPORTS_no${tag})
+                        if (GGML_MACHINE_SUPPORTS_no${tag})
+                            set(ARM_MCPU_FLAG_FIX "${ARM_MCPU_FLAG_FIX}+no${tag}" PARENT_SCOPE)
+                        endif()
+                    endif()
+                    set(CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS_SAVE})
+                endfunction()
+
+                check_arm_feature(dotprod "#include <arm_neon.h>\nint main() { int8x16_t _a, _b; volatile int32x4_t _s = vdotq_s32(_s, _a, _b); return 0; }")
+                check_arm_feature(i8mm    "#include <arm_neon.h>\nint main() { int8x16_t _a, _b; volatile int32x4_t _s = vmmlaq_s32(_s, _a, _b); return 0; }")
+                check_arm_feature(sve     "#include <arm_sve.h>\nint main()  { svfloat32_t _a, _b; volatile svfloat32_t _c = svadd_f32_z(svptrue_b8(), _a, _b); return 0; }")
+                check_arm_feature(sme     "#include <arm_sme.h>\n__arm_locally_streaming int main() { __asm__ volatile(\"smstart; smstop;\"); return 0; }")
+
+                list(APPEND ARCH_FLAGS "${ARM_MCPU_FLAG}${ARM_MCPU_FLAG_FIX}")
+            else()
+                if (GGML_CPU_ARM_ARCH)
+                    list(APPEND ARCH_FLAGS -march=${GGML_CPU_ARM_ARCH})
+                elseif(GGML_CPU_ALL_VARIANTS)
+                    # Begin with the lowest baseline
+                    set(ARM_MCPU "armv8-a")
+                    set(ARCH_TAGS "")
+                    set(ARCH_DEFINITIONS "")
+
+                    # When a feature is selected, bump the MCPU to the first
+                    # version that supported it
+                    if (GGML_INTERNAL_DOTPROD)
+                        set(ARM_MCPU "armv8.2-a")
+                        set(ARCH_TAGS "${ARCH_TAGS}+dotprod")
+                        list(APPEND ARCH_DEFINITIONS GGML_USE_DOTPROD)
+                    endif()
+                    if (GGML_INTERNAL_FP16_VECTOR_ARITHMETIC)
+                        set(ARM_MCPU "armv8.2-a")
+                        set(ARCH_TAGS "${ARCH_TAGS}+fp16")
+                        list(APPEND ARCH_DEFINITIONS GGML_USE_FP16_VECTOR_ARITHMETIC)
+                    endif()
+                    if (GGML_INTERNAL_SVE)
+                        set(ARM_MCPU "armv8.2-a")
+                        set(ARCH_TAGS "${ARCH_TAGS}+sve")
+                        list(APPEND ARCH_DEFINITIONS GGML_USE_SVE)
+                    endif()
+                    if (GGML_INTERNAL_MATMUL_INT8)
+                        set(ARM_MCPU "armv8.6-a")
+                        set(ARCH_TAGS "${ARCH_TAGS}+i8mm")
+                        list(APPEND ARCH_DEFINITIONS GGML_USE_MATMUL_INT8)
+                    endif()
+                    if (GGML_INTERNAL_SVE2)
+                        set(ARM_MCPU "armv8.6-a")
+                        set(ARCH_TAGS "${ARCH_TAGS}+sve2")
+                        list(APPEND ARCH_DEFINITIONS GGML_USE_SVE2)
+                    endif()
+                    if (GGML_INTERNAL_NOSVE)
+                        set(ARCH_TAGS "${ARCH_TAGS}+nosve")
+                    endif()
+                    if (GGML_INTERNAL_SME)
+                        set(ARM_MCPU "armv9.2-a")
+                        set(ARCH_TAGS "${ARCH_TAGS}+sme")
+                        list(APPEND ARCH_DEFINITIONS GGML_USE_SME)
+                    endif()
+                    list(APPEND ARCH_FLAGS "-march=${ARM_MCPU}${ARCH_TAGS}")
+                    ggml_add_cpu_backend_features(${GGML_CPU_NAME} arm ${ARCH_DEFINITIONS})
+                endif()
+            endif()
+
+            # show enabled features
+            if (CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows")
+                set(FEAT_INPUT_FILE "NUL")
+            else()
+                set(FEAT_INPUT_FILE "/dev/null")
+            endif()
+
+            execute_process(
+                COMMAND ${CMAKE_C_COMPILER} ${ARCH_FLAGS} -dM -E -
+                INPUT_FILE ${FEAT_INPUT_FILE}
+                OUTPUT_VARIABLE ARM_FEATURE
+                RESULT_VARIABLE ARM_FEATURE_RESULT
+            )
+            if (ARM_FEATURE_RESULT)
+                message(WARNING "Failed to get ARM features")
+            else()
+                foreach(feature DOTPROD SVE MATMUL_INT8 FMA FP16_VECTOR_ARITHMETIC SME)
+                    string(FIND "${ARM_FEATURE}" "__ARM_FEATURE_${feature} 1" feature_pos)
+                    if (NOT ${feature_pos} EQUAL -1)
+                        message(STATUS "ARM feature ${feature} enabled")
+                    endif()
+                endforeach()
+            endif()
+        endif()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "x86")
+        message(STATUS "x86 detected")
+        list(APPEND GGML_CPU_SOURCES
+            ggml-cpu/arch/x86/quants.c
+            ggml-cpu/arch/x86/repack.cpp
+            )
+
+        if (MSVC)
+            # instruction set detection for MSVC only
+            if (GGML_NATIVE)
+                include(ggml-cpu/cmake/FindSIMD.cmake)
+            endif ()
+            if (GGML_AVX512)
+                list(APPEND ARCH_FLAGS /arch:AVX512)
+                # /arch:AVX512 includes: __AVX512F__, __AVX512CD__, __AVX512BW__, __AVX512DQ__, and __AVX512VL__
+                # MSVC has no compile-time flags enabling specific
+                # AVX512 extensions, neither it defines the
+                # macros corresponding to the extensions.
+                # Do it manually.
+                list(APPEND ARCH_DEFINITIONS GGML_AVX512)
+                if (GGML_AVX512_VBMI)
+                    list(APPEND ARCH_DEFINITIONS __AVX512VBMI__)
+                    if (CMAKE_C_COMPILER_ID STREQUAL "Clang")
+                        list(APPEND ARCH_FLAGS -mavx512vbmi)
+                    endif()
+                endif()
+                if (GGML_AVX512_VNNI)
+                    list(APPEND ARCH_DEFINITIONS __AVX512VNNI__ GGML_AVX512_VNNI)
+                    if (CMAKE_C_COMPILER_ID STREQUAL "Clang")
+                        list(APPEND ARCH_FLAGS -mavx512vnni)
+                    endif()
+                endif()
+                if (GGML_AVX512_BF16)
+                    list(APPEND ARCH_DEFINITIONS __AVX512BF16__ GGML_AVX512_BF16)
+                    if (CMAKE_C_COMPILER_ID STREQUAL "Clang")
+                        list(APPEND ARCH_FLAGS -mavx512bf16)
+                    endif()
+                endif()
+                if (GGML_AMX_TILE)
+                    list(APPEND ARCH_DEFINITIONS __AMX_TILE__ GGML_AMX_TILE)
+                endif()
+                if (GGML_AMX_INT8)
+                    list(APPEND ARCH_DEFINITIONS __AMX_INT8__ GGML_AMX_INT8)
+                endif()
+                if (GGML_AMX_BF16)
+                    list(APPEND ARCH_DEFINITIONS __AMX_BF16__ GGML_AMX_BF16)
+                endif()
+            elseif (GGML_AVX2)
+                list(APPEND ARCH_FLAGS /arch:AVX2)
+                list(APPEND ARCH_DEFINITIONS GGML_AVX2 GGML_FMA GGML_F16C)
+            elseif (GGML_AVX)
+                list(APPEND ARCH_FLAGS /arch:AVX)
+                list(APPEND ARCH_DEFINITIONS GGML_AVX)
+            elseif (GGML_SSE42)
+                list(APPEND ARCH_FLAGS /arch:SSE4.2)
+                list(APPEND ARCH_DEFINITIONS GGML_SSE42)
+            endif()
+            if (GGML_AVX_VNNI)
+                list(APPEND ARCH_DEFINITIONS __AVXVNNI__ GGML_AVX_VNNI)
+            endif()
+            if (GGML_BMI2)
+                # MSVC does not define macro __BMI2__
+                list(APPEND ARCH_DEFINITIONS __BMI2__ GGML_BMI2)
+            endif()
+        else ()
+            if (GGML_NATIVE)
+                list(APPEND ARCH_FLAGS -march=native)
+            else ()
+                if (GGML_SSE42)
+                    list(APPEND ARCH_FLAGS -msse4.2)
+                    list(APPEND ARCH_DEFINITIONS GGML_SSE42)
+                endif()
+                if (GGML_F16C)
+                    list(APPEND ARCH_FLAGS -mf16c)
+                    list(APPEND ARCH_DEFINITIONS GGML_F16C)
+                endif()
+                if (GGML_FMA)
+                    list(APPEND ARCH_FLAGS -mfma)
+                    list(APPEND ARCH_DEFINITIONS GGML_FMA)
+                endif()
+                if (GGML_BMI2)
+                    list(APPEND ARCH_FLAGS -mbmi2)
+                    list(APPEND ARCH_DEFINITIONS GGML_BMI2)
+                endif()
+                if (GGML_AVX)
+                    list(APPEND ARCH_FLAGS -mavx)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX)
+                endif()
+                if (GGML_AVX2)
+                    list(APPEND ARCH_FLAGS -mavx2)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX2)
+                endif()
+                if (GGML_AVX_VNNI)
+                    list(APPEND ARCH_FLAGS -mavxvnni)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX_VNNI)
+                endif()
+                if (GGML_AVX512)
+                    list(APPEND ARCH_FLAGS -mavx512f)
+                    list(APPEND ARCH_FLAGS -mavx512cd)
+                    list(APPEND ARCH_FLAGS -mavx512vl)
+                    list(APPEND ARCH_FLAGS -mavx512dq)
+                    list(APPEND ARCH_FLAGS -mavx512bw)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX512)
+                endif()
+                if (GGML_AVX512_VBMI)
+                    list(APPEND ARCH_FLAGS -mavx512vbmi)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX512_VBMI)
+                endif()
+                if (GGML_AVX512_VNNI)
+                    list(APPEND ARCH_FLAGS -mavx512vnni)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX512_VNNI)
+                endif()
+                if (GGML_AVX512_BF16)
+                    list(APPEND ARCH_FLAGS -mavx512bf16)
+                    list(APPEND ARCH_DEFINITIONS GGML_AVX512_BF16)
+                endif()
+                if (GGML_AMX_TILE)
+                    list(APPEND ARCH_FLAGS -mamx-tile)
+                    list(APPEND ARCH_DEFINITIONS GGML_AMX_TILE)
+                endif()
+                if (GGML_AMX_INT8)
+                    list(APPEND ARCH_FLAGS -mamx-int8)
+                    list(APPEND ARCH_DEFINITIONS GGML_AMX_INT8)
+                endif()
+                if (GGML_AMX_BF16)
+                    list(APPEND ARCH_FLAGS -mamx-bf16)
+                    list(APPEND ARCH_DEFINITIONS GGML_AMX_BF16)
+                endif()
+            endif()
+        endif()
+
+        if (GGML_BACKEND_DL)
+            if (GGML_NATIVE)
+                # the feature check relies on ARCH_DEFINITIONS, but it is not set with GGML_NATIVE
+                message(FATAL_ERROR "GGML_NATIVE is not compatible with GGML_BACKEND_DL, consider using GGML_CPU_ALL_VARIANTS")
+            endif()
+            ggml_add_cpu_backend_features(${GGML_CPU_NAME} x86 ${ARCH_DEFINITIONS})
+        endif()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
+        message(STATUS "PowerPC detected")
+        list(APPEND GGML_CPU_SOURCES ggml-cpu/arch/powerpc/quants.c)
+        if (GGML_NATIVE)
+            if (${CMAKE_SYSTEM_PROCESSOR} MATCHES "ppc64")
+                file(READ "/proc/cpuinfo" POWER10_M)
+            elseif (${CMAKE_SYSTEM_PROCESSOR} MATCHES "powerpc")
+                execute_process(COMMAND bash -c "prtconf |grep 'Implementation' | head -n 1" OUTPUT_VARIABLE POWER10_M)
+            endif()
+
+            string(TOUPPER "${POWER10_M}" POWER10_M_UPPER)
+            string(REGEX MATCHALL "POWER *([0-9]+)" MATCHED_STRING "${POWER10_M_UPPER}")
+            string(REGEX REPLACE "POWER *([0-9]+)" "\\1" EXTRACTED_NUMBER "${MATCHED_STRING}")
+
+            if (EXTRACTED_NUMBER GREATER_EQUAL 10)
+                list(APPEND ARCH_FLAGS -mcpu=power10 -mpowerpc64)
+            elseif (EXTRACTED_NUMBER EQUAL 9)
+                list(APPEND ARCH_FLAGS -mcpu=power9 -mpowerpc64)
+            elseif (${CMAKE_SYSTEM_PROCESSOR} MATCHES "ppc64le")
+                list(APPEND ARCH_FLAGS -mcpu=powerpc64le -mtune=native)
+            else()
+                list(APPEND ARCH_FLAGS -mcpu=native -mtune=native -mpowerpc64)
+            endif()
+        elseif(GGML_CPU_ALL_VARIANTS)
+            # Begin with the lowest baseline
+            set(ARCH_DEFINITIONS "")
+
+            # When a feature is selected, bump the MCPU to the first
+            # version that supported it
+            foreach(PVER RANGE 7 11)
+                if(DEFINED GGML_INTERNAL_POWER${PVER})
+                    set(POWERPC_MCPU "power${PVER}")
+                    list(APPEND ARCH_DEFINITIONS GGML_USE_POWER${PVER})
+                endif()
+            endforeach()
+            if (GGML_INTERNAL_VSX)
+                list(APPEND ARCH_DEFINITIONS GGML_USE_VSX)
+                list(APPEND ARCH_FLAGS -mvsx)
+            endif()
+
+            if (DEFINED POWERPC_MCPU)
+                list(APPEND ARCH_FLAGS -mcpu=${POWERPC_MCPU})
+            endif()
+            ggml_add_cpu_backend_features(${GGML_CPU_NAME} powerpc ${ARCH_DEFINITIONS})
+        else()
+            if (GGML_CPU_POWERPC_CPUTYPE)
+                list(APPEND ARCH_FLAGS -mcpu=${GGML_CPU_POWERPC_CPUTYPE})
+            endif()
+        endif()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "loongarch64")
+        message(STATUS "loongarch64 detected")
+        list(APPEND GGML_CPU_SOURCES ggml-cpu/arch/loongarch/quants.c)
+
+        list(APPEND ARCH_FLAGS -march=loongarch64)
+        if (GGML_LASX)
+            list(APPEND ARCH_FLAGS -mlasx)
+        endif()
+        if (GGML_LSX)
+            list(APPEND ARCH_FLAGS -mlsx)
+        endif()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "riscv64")
+        message(STATUS "riscv64 detected")
+        list(APPEND GGML_CPU_SOURCES
+            ggml-cpu/arch/riscv/quants.c
+            ggml-cpu/arch/riscv/repack.cpp
+            )
+        if (GGML_RVV)
+            if (GGML_XTHEADVECTOR)
+                list(APPEND ARCH_FLAGS -march=rv64gc_xtheadvector -mabi=lp64d)
+            elseif (GGML_RV_ZFH)
+                list(APPEND ARCH_FLAGS -march=rv64gcv_zfhmin -mabi=lp64d)
+            else()
+                list(APPEND ARCH_FLAGS -march=rv64gcv -mabi=lp64d)
+            endif()
+        endif()
+    elseif (GGML_SYSTEM_ARCH STREQUAL "s390x")
+        message(STATUS "s390x detected")
+        list(APPEND GGML_CPU_SOURCES ggml-cpu/arch/s390/quants.c)
+        file(READ "/proc/cpuinfo" CPUINFO_CONTENTS)
+        string(REGEX REPLACE "machine[ \t\r\n]*=[ \t\r\n]*([0-9]+)" "\\1" S390X_M ${CPUINFO_CONTENTS})
+
+        # TODO: Separation to determine activation of VX/VXE/VXE2
+        if (${S390X_M} MATCHES "8561|8562")
+            set(GGML_NNPA OFF)
+            message(STATUS "z15 target")
+            list(APPEND ARCH_FLAGS -march=z15)
+        elseif (${S390X_M} MATCHES "3931")
+            message(STATUS "z16 target")
+            list(APPEND ARCH_FLAGS -march=z16)
+        elseif (${S390X_M} MATCHES "9175|9176")
+            # NOTE: Only available from GCC 15.1.0 onwards. Any z17 machine with compile issues must first verify their GCC version.
+            #       binutils must also be updated to the latest for the -march=z17 flag to work. Otherwise, use -march=arch15.
+            message(STATUS "z17 target")
+            list(APPEND ARCH_FLAGS -march=z17)
+        else()
+            message(STATUS "Unknown target")
+            message(WARNING "Unknown target. If you are compiling for z14 and earlier, you might have to add -DGGML_VXE=OFF.")
+            list(APPEND ARCH_FLAGS -march=native -mtune=native)
+        endif()
+
+        if (GGML_VXE)
+            message(STATUS "VX/VXE/VXE2 enabled")
+            list(APPEND ARCH_FLAGS -mvx -mzvector)
+            list(APPEND ARCH_DEFINITIONS GGML_VXE)
+        endif()
+
+        if (GGML_NNPA)
+            message(STATUS "NNPA enabled")
+            list(APPEND ARCH_DEFINITIONS GGML_NNPA)
+        endif()
+    elseif (CMAKE_SYSTEM_PROCESSOR MATCHES "wasm")
+        message(STATUS "Wasm detected")
+        list (APPEND GGML_CPU_SOURCES ggml-cpu/arch/wasm/quants.c)
+    else()
+        message(WARNING "Unknown CPU architecture. Falling back to generic implementations.")
+        list(APPEND ARCH_FLAGS -DGGML_CPU_GENERIC)
+    endif()
+
+    if (GGML_CPU_REPACK)
+        target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_USE_CPU_REPACK)
+    endif()
+
+    if (GGML_CPU_KLEIDIAI)
+        message(STATUS "Using KleidiAI optimized kernels if applicable")
+
+        # Disable the KleidiAI tests
+        set(KLEIDIAI_BUILD_TESTS  OFF)
+
+        # Fetch KleidiAI sources:
+        include(FetchContent)
+        set(KLEIDIAI_COMMIT_TAG "v1.11.0")
+        set(KLEIDIAI_DOWNLOAD_URL "https://github.com/ARM-software/kleidiai/archive/refs/tags/${KLEIDIAI_COMMIT_TAG}.tar.gz")
+        set(KLEIDIAI_ARCHIVE_MD5  "3fe9e5ab964c375c53839296eb71eaa2")
+
+        if (POLICY CMP0135)
+            cmake_policy(SET CMP0135 NEW)
+        endif()
+
+        FetchContent_Declare(KleidiAI_Download
+            URL ${KLEIDIAI_DOWNLOAD_URL}
+            DOWNLOAD_EXTRACT_TIMESTAMP NEW
+            URL_HASH MD5=${KLEIDIAI_ARCHIVE_MD5})
+
+        FetchContent_MakeAvailable(KleidiAI_Download)
+        FetchContent_GetProperties(KleidiAI_Download
+            SOURCE_DIR  KLEIDIAI_SRC
+            POPULATED   KLEIDIAI_POPULATED)
+
+        if (NOT KLEIDIAI_POPULATED)
+            message(FATAL_ERROR "KleidiAI source downloaded failed.")
+        endif()
+
+        add_compile_definitions(GGML_USE_CPU_KLEIDIAI)
+
+        # Remove kleidiai target after fetching it
+        if (TARGET kleidiai)
+            set_target_properties(kleidiai PROPERTIES EXCLUDE_FROM_ALL TRUE)
+        endif()
+
+        list(APPEND GGML_CPU_SOURCES
+            ggml-cpu/kleidiai/kleidiai.cpp
+            ggml-cpu/kleidiai/kernels.cpp
+            ggml-cpu/kleidiai/kleidiai.h
+            ggml-cpu/kleidiai/kernels.h
+            )
+
+        # KleidiAI
+        include_directories(
+            ${KLEIDIAI_SRC}/
+            ${KLEIDIAI_SRC}/kai/
+            ${KLEIDIAI_SRC}/kai/ukernels/
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/)
+
+        set(ARCH_FLAGS_TEMP "${ARCH_FLAGS}")
+        if (NOT ARCH_FLAGS_TEMP)
+            string(REGEX MATCH "-march=[^ ]+" ARCH_FLAGS_TEMP "${CMAKE_C_FLAGS}")
+        endif()
+        string(FIND "${ARCH_FLAGS_TEMP}" "+dotprod" DOTPROD_ENABLED)
+        string(FIND "${ARCH_FLAGS_TEMP}" "+i8mm" I8MM_ENABLED)
+        string(FIND "${ARCH_FLAGS_TEMP}" "+sme" SME_ENABLED)
+
+        set(PRIVATE_ARCH_FLAGS ${ARCH_FLAGS_TEMP})
+
+        list(APPEND GGML_KLEIDIAI_SOURCES
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_quant_pack_qsi8d32p_f32.c
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon.c
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_quant_pack_qsi8d32p_f32_neon.c
+            ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0.c)
+
+        if (NOT DOTPROD_ENABLED MATCHES -1)
+            list(APPEND GGML_KLEIDIAI_SOURCES
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod.c
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod.c
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod.c)
+        endif()
+
+        if (NOT I8MM_ENABLED MATCHES -1)
+            list(APPEND GGML_KLEIDIAI_SOURCES ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm.c)
+        endif()
+
+        if (NOT SME_ENABLED MATCHES -1)
+            list(APPEND GGML_KLEIDIAI_SOURCES
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa.c
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot.c
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa.c
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_bf16p2vlx2_f32_sme.c
+                ${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.c)
+            set(PRIVATE_ARCH_FLAGS "-fno-tree-vectorize;${PRIVATE_ARCH_FLAGS}+sve+sve2")
+        endif()
+
+        set_source_files_properties(${GGML_KLEIDIAI_SOURCES} PROPERTIES COMPILE_OPTIONS "${PRIVATE_ARCH_FLAGS}")
+        list(APPEND GGML_CPU_SOURCES ${GGML_KLEIDIAI_SOURCES})
+    endif()
+
+    message(STATUS "Adding CPU backend variant ${GGML_CPU_NAME}: ${ARCH_FLAGS} ${ARCH_DEFINITIONS}")
+    target_sources(${GGML_CPU_NAME} PRIVATE ${GGML_CPU_SOURCES})
+    target_compile_options(${GGML_CPU_NAME} PRIVATE ${ARCH_FLAGS})
+    target_compile_definitions(${GGML_CPU_NAME} PRIVATE ${ARCH_DEFINITIONS})
+
+    if (EMSCRIPTEN)
+        set_target_properties(${GGML_CPU_NAME} PROPERTIES COMPILE_FLAGS "-msimd128")
+    endif()
+
+    if (CMAKE_CXX_COMPILER_ID STREQUAL "IntelLLVM")
+        # The compiler automatically enables "-ffast-math" which can cause NaNs in tests due to "-fassociative-math"
+        target_compile_options(${GGML_CPU_NAME} PRIVATE "-fno-associative-math")
+    endif()
+endfunction()

ggml/src/ggml-cpu/amx/amx.cpp

@@ -0,0 +1,221 @@
+#include "amx.h"
+#include "common.h"
+#include "mmq.h"
+#include "ggml-backend-impl.h"
+#include "ggml-backend.h"
+#include "ggml-impl.h"
+#include "ggml-cpu.h"
+#include "traits.h"
+
+#if defined(__gnu_linux__)
+#include <sys/syscall.h>
+#include <unistd.h>
+#endif
+
+#include <cstdlib>
+#include <cstring>
+#include <memory>
+
+#if defined(__AMX_INT8__) && defined(__AVX512VNNI__)
+
+// AMX type_trais
+namespace ggml::cpu::amx {
+class tensor_traits : public ggml::cpu::tensor_traits {
+    bool work_size(int /* n_threads */, const struct ggml_tensor * op, size_t & size) override {
+        size = ggml_backend_amx_desired_wsize(op);
+        return true;
+    }
+
+    bool compute_forward(struct ggml_compute_params * params, struct ggml_tensor * op) override {
+        if (op->op == GGML_OP_MUL_MAT) {
+            ggml_backend_amx_mul_mat(params, op);
+            return true;
+        }
+        return false;
+    }
+};
+
+static ggml::cpu::tensor_traits * get_tensor_traits(ggml_backend_buffer_t, struct ggml_tensor *) {
+    static tensor_traits traits;
+    return &traits;
+}
+}  // namespace ggml::cpu::amx
+
+// AMX buffer interface
+static void ggml_backend_amx_buffer_free_buffer(ggml_backend_buffer_t buffer) {
+    free(buffer->context);
+}
+
+static void * ggml_backend_amx_buffer_get_base(ggml_backend_buffer_t buffer) {
+    return (void *) (buffer->context);
+}
+
+static enum ggml_status ggml_backend_amx_buffer_init_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor) {
+    tensor->extra = (void *) ggml::cpu::amx::get_tensor_traits(buffer, tensor);
+
+    GGML_UNUSED(buffer);
+    return GGML_STATUS_SUCCESS;
+}
+
+static void ggml_backend_amx_buffer_memset_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor,
+                                                  uint8_t value, size_t offset, size_t size) {
+    memset((char *) tensor->data + offset, value, size);
+
+    GGML_UNUSED(buffer);
+}
+
+static void ggml_backend_amx_buffer_set_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor,
+                                               const void * data, size_t offset, size_t size) {
+    if (qtype_has_amx_kernels(tensor->type)) {
+        GGML_LOG_DEBUG("%s: amx repack tensor %s of type %s\n", __func__, tensor->name, ggml_type_name(tensor->type));
+        ggml_backend_amx_convert_weight(tensor, data, offset, size);
+    } else {
+        memcpy((char *) tensor->data + offset, data, size);
+    }
+
+    GGML_UNUSED(buffer);
+}
+
+/*
+// need to figure what we need to do with buffer->extra.
+static void ggml_backend_amx_buffer_get_tensor(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, size_t offset, size_t size) {
+    GGML_ASSERT(!qtype_has_amx_kernels(tensor->type));
+    memcpy(data, (const char *)tensor->data + offset, size);
+
+    GGML_UNUSED(buffer);
+}
+
+static bool ggml_backend_amx_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const struct ggml_tensor * src, struct ggml_tensor * dst) {
+    if (ggml_backend_buffer_is_host(src->buffer)) {
+        if (qtype_has_amx_kernels(src->type)) {
+            ggml_backend_amx_convert_weight(dst, src->data, 0, ggml_nbytes(dst));
+        } else {
+            memcpy(dst->data, src->data, ggml_nbytes(src));
+        }
+        return true;
+    }
+    return false;
+
+    GGML_UNUSED(buffer);
+}
+*/
+
+static void ggml_backend_amx_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
+    memset(buffer->context, value, buffer->size);
+}
+
+static ggml_backend_buffer_i ggml_backend_amx_buffer_interface = {
+    /* .free_buffer     = */ ggml_backend_amx_buffer_free_buffer,
+    /* .get_base        = */ ggml_backend_amx_buffer_get_base,
+    /* .init_tensor     = */ ggml_backend_amx_buffer_init_tensor,
+    /* .memset_tensor   = */ ggml_backend_amx_buffer_memset_tensor,
+    /* .set_tensor      = */ ggml_backend_amx_buffer_set_tensor,
+    /* .get_tensor      = */ nullptr,
+    /* .cpy_tensor      = */ nullptr,
+    /* .clear           = */ ggml_backend_amx_buffer_clear,
+    /* .reset           = */ nullptr,
+};
+
+static const char * ggml_backend_amx_buffer_type_get_name(ggml_backend_buffer_type_t buft) {
+    return "AMX";
+
+    GGML_UNUSED(buft);
+}
+
+static ggml_backend_buffer_t ggml_backend_amx_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
+    void * data = ggml_aligned_malloc(size);
+    if (data == NULL) {
+        fprintf(stderr, "%s: failed to allocate buffer of size %zu\n", __func__, size);
+        return NULL;
+    }
+
+    return ggml_backend_buffer_init(buft, ggml_backend_amx_buffer_interface, data, size);
+}
+
+static size_t ggml_backend_amx_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
+    return TENSOR_ALIGNMENT;
+
+    GGML_UNUSED(buft);
+}
+
+namespace ggml::cpu::amx {
+class extra_buffer_type : ggml::cpu::extra_buffer_type {
+    bool supports_op(ggml_backend_dev_t, const struct ggml_tensor * op) override {
+        // handle only 2d gemm for now
+        auto is_contiguous_2d = [](const struct ggml_tensor * t) {
+            return ggml_is_contiguous(t) && t->ne[3] == 1 && t->ne[2] == 1;
+        };
+
+        if (op->op == GGML_OP_MUL_MAT && is_contiguous_2d(op->src[0]) &&  // src0 must be contiguous
+            is_contiguous_2d(op->src[1]) &&                               // src1 must be contiguous
+            op->src[0]->buffer && op->src[0]->buffer->buft == ggml_backend_amx_buffer_type() &&
+            op->ne[0] % (TILE_N * 2) == 0 &&                              // out_features is 32x
+            (qtype_has_amx_kernels(op->src[0]->type) || (op->src[0]->type == GGML_TYPE_F16))) {
+            // src1 must be host buffer
+            if (op->src[1]->buffer && !ggml_backend_buft_is_host(op->src[1]->buffer->buft)) {
+                return false;
+            }
+            // src1 must be float32
+            if (op->src[1]->type == GGML_TYPE_F32) {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    ggml::cpu::tensor_traits * get_tensor_traits(const struct ggml_tensor * op) override {
+        if (op->op == GGML_OP_MUL_MAT && op->src[0]->buffer &&
+            op->src[0]->buffer->buft == ggml_backend_amx_buffer_type()) {
+            return (ggml::cpu::tensor_traits *) op->src[0]->extra;
+        }
+
+        return nullptr;
+    }
+};
+}  // namespace ggml::cpu::amx
+
+static size_t ggml_backend_amx_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {
+    return ggml_backend_amx_get_alloc_size(tensor);
+
+    GGML_UNUSED(buft);
+}
+
+#define ARCH_GET_XCOMP_PERM     0x1022
+#define ARCH_REQ_XCOMP_PERM     0x1023
+#define XFEATURE_XTILECFG       17
+#define XFEATURE_XTILEDATA      18
+
+static bool ggml_amx_init() {
+#if defined(__gnu_linux__)
+    if (syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA)) {
+        fprintf(stderr, "AMX is not ready to be used!\n");
+        return false;
+    }
+    return true;
+#elif defined(_WIN32)
+    return true;
+#endif
+}
+
+ggml_backend_buffer_type_t ggml_backend_amx_buffer_type() {
+    static struct ggml_backend_buffer_type ggml_backend_buffer_type_amx = {
+        /* .iface = */ {
+                        /* .get_name         = */ ggml_backend_amx_buffer_type_get_name,
+                        /* .alloc_buffer     = */ ggml_backend_amx_buffer_type_alloc_buffer,
+                        /* .get_alignment    = */ ggml_backend_amx_buffer_type_get_alignment,
+                        /* .get_max_size     = */ nullptr,  // defaults to SIZE_MAX
+                        /* .get_alloc_size   = */ ggml_backend_amx_buffer_type_get_alloc_size,
+                        /* .is_host          = */ nullptr,
+                        },
+        /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_cpu_reg(), 0),
+        /* .context = */ new ggml::cpu::amx::extra_buffer_type(),
+    };
+
+    if (!ggml_amx_init()) {
+        return nullptr;
+    }
+
+    return &ggml_backend_buffer_type_amx;
+}
+
+#endif  // defined(__AMX_INT8__) && defined(__AVX512VNNI__)

ggml/src/ggml-cpu/amx/amx.h

@@ -0,0 +1,8 @@
+#include "ggml-backend.h"
+#include "ggml-cpu-impl.h"
+
+// GGML internal header
+
+#if defined(__AMX_INT8__) && defined(__AVX512VNNI__)
+ggml_backend_buffer_type_t ggml_backend_amx_buffer_type(void);
+#endif

ggml/src/ggml-cpu/amx/common.h

@@ -0,0 +1,91 @@
+#pragma once
+
+#include "ggml.h"
+#include "ggml-cpu-impl.h"
+
+#include <algorithm>
+#include <memory>
+#include <type_traits>
+
+#if defined(GGML_USE_OPENMP)
+#include <omp.h>
+#endif
+
+#define TILE_M 16
+#define TILE_N 16
+#define TILE_K 32
+#define VNNI_BLK 4
+
+#define AMX_BLK_SIZE 32
+
+#define TMM0 0
+#define TMM1 1
+#define TMM2 2
+#define TMM3 3
+#define TMM4 4
+#define TMM5 5
+#define TMM6 6
+#define TMM7 7
+
+// parallel routines
+template <typename T, typename std::enable_if<std::is_integral<T>::value, int>::type = 0>
+inline T div_up(T x, T y) { return (x + y - 1) / y; }
+
+template <typename T>
+inline void balance211(T n, T nth, T ith, T& n_start, T& n_end) {
+#if 0
+    // onednn partition pattern
+    T& n_my = n_end;
+    if (nth <= 1 || n == 0) {
+        n_start = 0;
+        n_my = n;
+    } else {
+        T n1 = div_up(n, nth);
+        T n2 = n1 - 1;
+        T T1 = n - n2 * nth;
+        n_my = ith < T1 ? n1 : n2;
+        n_start = ith <= T1 ? ith*n1 : T1 * n1 + (ith - T1) * n2;
+    }
+    n_end += n_start;
+#else
+    // pytorch aten partition pattern
+    T n_my = div_up(n, nth);
+    n_start = ith * n_my;
+    n_end = std::min(n_start + n_my, n);
+#endif
+}
+
+template <typename func_t>
+inline void parallel_for(int n, const func_t& f) {
+#if defined(GGML_USE_OPENMP)
+#pragma omp parallel
+{
+    int nth = omp_get_num_threads();
+    int ith = omp_get_thread_num();
+    int tbegin, tend;
+    balance211(n, nth, ith, tbegin, tend);
+    f(tbegin, tend);
+}
+#else
+    f(0, n);
+#endif
+}
+
+template <typename func_t>
+inline void parallel_for_ggml(const ggml_compute_params * params, int n, const func_t & f) {
+    int tbegin, tend;
+    balance211(n, params->nth, params->ith, tbegin, tend);
+    f(tbegin, tend);
+}
+
+// quantized types that have AMX support
+inline bool qtype_has_amx_kernels(const enum ggml_type type) {
+    // TODO: fix padding for vnni format
+    return (type == GGML_TYPE_Q4_0) ||
+        (type == GGML_TYPE_Q4_1) ||
+        (type == GGML_TYPE_Q8_0) ||
+        (type == GGML_TYPE_Q4_K) ||
+        (type == GGML_TYPE_Q5_K) ||
+        (type == GGML_TYPE_Q6_K) ||
+        (type == GGML_TYPE_IQ4_XS);
+}

ggml/src/ggml-cpu/amx/mmq.h

@@ -0,0 +1,10 @@
+#pragma once
+#include "common.h"
+
+size_t ggml_backend_amx_desired_wsize(const struct ggml_tensor * dst);
+
+size_t ggml_backend_amx_get_alloc_size(const struct ggml_tensor * tensor);
+
+void ggml_backend_amx_convert_weight(struct ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+
+void ggml_backend_amx_mul_mat(const struct ggml_compute_params * params, struct ggml_tensor * dst);

ggml/src/ggml-cpu/arch-fallback.h

@@ -0,0 +1,184 @@
+#pragma once
+
+// Rename `_generic` functions if no native implementation is available.
+// This effectively selects the generic implementation.
+
+#if defined(GGML_CPU_GENERIC)
+// quants.c
+#define quantize_row_q8_0_generic quantize_row_q8_0
+#define quantize_row_q8_1_generic quantize_row_q8_1
+#define quantize_row_q8_K_generic quantize_row_q8_K
+#define ggml_vec_dot_q4_0_q8_0_generic ggml_vec_dot_q4_0_q8_0
+#define ggml_vec_dot_q4_1_q8_1_generic ggml_vec_dot_q4_1_q8_1
+#define ggml_vec_dot_q5_0_q8_0_generic ggml_vec_dot_q5_0_q8_0
+#define ggml_vec_dot_q5_1_q8_1_generic ggml_vec_dot_q5_1_q8_1
+#define ggml_vec_dot_q8_0_q8_0_generic ggml_vec_dot_q8_0_q8_0
+#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
+#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_q2_K_q8_K_generic ggml_vec_dot_q2_K_q8_K
+#define ggml_vec_dot_q3_K_q8_K_generic ggml_vec_dot_q3_K_q8_K
+#define ggml_vec_dot_q4_K_q8_K_generic ggml_vec_dot_q4_K_q8_K
+#define ggml_vec_dot_q5_K_q8_K_generic ggml_vec_dot_q5_K_q8_K
+#define ggml_vec_dot_q6_K_q8_K_generic ggml_vec_dot_q6_K_q8_K
+#define ggml_vec_dot_iq2_xxs_q8_K_generic ggml_vec_dot_iq2_xxs_q8_K
+#define ggml_vec_dot_iq2_xs_q8_K_generic ggml_vec_dot_iq2_xs_q8_K
+#define ggml_vec_dot_iq2_s_q8_K_generic ggml_vec_dot_iq2_s_q8_K
+#define ggml_vec_dot_iq3_xxs_q8_K_generic ggml_vec_dot_iq3_xxs_q8_K
+#define ggml_vec_dot_iq3_s_q8_K_generic ggml_vec_dot_iq3_s_q8_K
+#define ggml_vec_dot_iq1_s_q8_K_generic ggml_vec_dot_iq1_s_q8_K
+#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
+#define ggml_vec_dot_iq4_nl_q8_0_generic ggml_vec_dot_iq4_nl_q8_0
+#define ggml_vec_dot_iq4_xs_q8_K_generic ggml_vec_dot_iq4_xs_q8_K
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_q4_0_8x8_q8_0_generic ggml_gemv_q4_0_8x8_q8_0
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_q4_0_8x8_q8_0_generic ggml_gemm_q4_0_8x8_q8_0
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#elif defined(__aarch64__) || defined(__arm__) || defined(_M_ARM) || defined(_M_ARM64)
+// repack.cpp
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#elif defined(__x86_64__) || defined(__i386__) || defined(_M_IX86) || defined(_M_X64)
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#elif defined(__POWERPC__) || defined(__powerpc__)
+// ref: https://github.com/ggml-org/llama.cpp/pull/14146#issuecomment-2972561679
+// quants.c
+#define quantize_row_q8_K_generic quantize_row_q8_K
+#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
+#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_q4_0_8x8_q8_0_generic ggml_gemv_q4_0_8x8_q8_0
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_q4_0_8x8_q8_0_generic ggml_gemm_q4_0_8x8_q8_0
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#elif defined(__loongarch64)
+// quants.c
+#define quantize_row_q8_K_generic quantize_row_q8_K
+#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
+#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_q4_0_8x8_q8_0_generic ggml_gemv_q4_0_8x8_q8_0
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_q4_0_8x8_q8_0_generic ggml_gemm_q4_0_8x8_q8_0
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#elif defined(__riscv)
+// quants.c
+#define quantize_row_q8_K_generic quantize_row_q8_K
+#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
+#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_iq2_xxs_q8_K_generic ggml_vec_dot_iq2_xxs_q8_K
+#define ggml_vec_dot_iq2_xs_q8_K_generic ggml_vec_dot_iq2_xs_q8_K
+#define ggml_vec_dot_iq2_s_q8_K_generic ggml_vec_dot_iq2_s_q8_K
+#define ggml_vec_dot_iq3_xxs_q8_K_generic ggml_vec_dot_iq3_xxs_q8_K
+#define ggml_vec_dot_iq3_s_q8_K_generic ggml_vec_dot_iq3_s_q8_K
+#define ggml_vec_dot_iq1_s_q8_K_generic ggml_vec_dot_iq1_s_q8_K
+#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
+#define ggml_vec_dot_iq4_nl_q8_0_generic ggml_vec_dot_iq4_nl_q8_0
+#define ggml_vec_dot_iq4_xs_q8_K_generic ggml_vec_dot_iq4_xs_q8_K
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#elif defined(__s390x__)
+// quants.c
+#define quantize_row_q8_K_generic quantize_row_q8_K
+#define ggml_vec_dot_q5_0_q8_0_generic ggml_vec_dot_q5_0_q8_0
+#define ggml_vec_dot_q5_1_q8_1_generic ggml_vec_dot_q5_1_q8_1
+#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
+#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_q2_K_q8_K_generic ggml_vec_dot_q2_K_q8_K
+#define ggml_vec_dot_iq2_xxs_q8_K_generic ggml_vec_dot_iq2_xxs_q8_K
+#define ggml_vec_dot_iq2_xs_q8_K_generic ggml_vec_dot_iq2_xs_q8_K
+#define ggml_vec_dot_iq2_s_q8_K_generic ggml_vec_dot_iq2_s_q8_K
+#define ggml_vec_dot_iq3_xxs_q8_K_generic ggml_vec_dot_iq3_xxs_q8_K
+#define ggml_vec_dot_iq3_s_q8_K_generic ggml_vec_dot_iq3_s_q8_K
+#define ggml_vec_dot_iq1_s_q8_K_generic ggml_vec_dot_iq1_s_q8_K
+#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_q4_0_8x8_q8_0_generic ggml_gemv_q4_0_8x8_q8_0
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_q4_0_8x8_q8_0_generic ggml_gemm_q4_0_8x8_q8_0
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#elif defined(__wasm__)
+// quants.c
+#define ggml_vec_dot_q4_1_q8_1_generic ggml_vec_dot_q4_1_q8_1
+#define ggml_vec_dot_tq1_0_q8_K_generic ggml_vec_dot_tq1_0_q8_K
+#define ggml_vec_dot_tq2_0_q8_K_generic ggml_vec_dot_tq2_0_q8_K
+#define ggml_vec_dot_iq2_xxs_q8_K_generic ggml_vec_dot_iq2_xxs_q8_K
+#define ggml_vec_dot_iq2_xs_q8_K_generic ggml_vec_dot_iq2_xs_q8_K
+#define ggml_vec_dot_iq2_s_q8_K_generic ggml_vec_dot_iq2_s_q8_K
+#define ggml_vec_dot_iq3_xxs_q8_K_generic ggml_vec_dot_iq3_xxs_q8_K
+#define ggml_vec_dot_iq3_s_q8_K_generic ggml_vec_dot_iq3_s_q8_K
+#define ggml_vec_dot_iq1_s_q8_K_generic ggml_vec_dot_iq1_s_q8_K
+#define ggml_vec_dot_iq1_m_q8_K_generic ggml_vec_dot_iq1_m_q8_K
+#define ggml_vec_dot_iq4_nl_q8_0_generic ggml_vec_dot_iq4_nl_q8_0
+#define ggml_vec_dot_iq4_xs_q8_K_generic ggml_vec_dot_iq4_xs_q8_K
+// repack.cpp
+#define ggml_quantize_mat_q8_0_4x4_generic ggml_quantize_mat_q8_0_4x4
+#define ggml_quantize_mat_q8_0_4x8_generic ggml_quantize_mat_q8_0_4x8
+#define ggml_quantize_mat_q8_K_4x8_generic ggml_quantize_mat_q8_K_4x8
+#define ggml_gemv_q4_0_4x4_q8_0_generic ggml_gemv_q4_0_4x4_q8_0
+#define ggml_gemv_q4_0_4x8_q8_0_generic ggml_gemv_q4_0_4x8_q8_0
+#define ggml_gemv_q4_0_8x8_q8_0_generic ggml_gemv_q4_0_8x8_q8_0
+#define ggml_gemv_q4_K_8x8_q8_K_generic ggml_gemv_q4_K_8x8_q8_K
+#define ggml_gemv_iq4_nl_4x4_q8_0_generic ggml_gemv_iq4_nl_4x4_q8_0
+#define ggml_gemm_q4_0_4x4_q8_0_generic ggml_gemm_q4_0_4x4_q8_0
+#define ggml_gemm_q4_0_4x8_q8_0_generic ggml_gemm_q4_0_4x8_q8_0
+#define ggml_gemm_q4_0_8x8_q8_0_generic ggml_gemm_q4_0_8x8_q8_0
+#define ggml_gemm_q4_K_8x8_q8_K_generic ggml_gemm_q4_K_8x8_q8_K
+#define ggml_gemm_iq4_nl_4x4_q8_0_generic ggml_gemm_iq4_nl_4x4_q8_0
+#endif

ggml/src/ggml-cpu/arch/arm/cpu-feats.cpp

@@ -0,0 +1,94 @@
+#include "ggml-backend-impl.h"
+
+#if defined(__aarch64__)
+
+#if defined(__linux__)
+#include <sys/auxv.h>
+#elif defined(__APPLE__)
+#include <sys/sysctl.h>
+#endif
+
+#if !defined(HWCAP2_I8MM)
+#define HWCAP2_I8MM (1 << 13)
+#endif
+
+#if !defined(HWCAP2_SME)
+#define HWCAP2_SME (1 << 23)
+#endif
+
+struct aarch64_features {
+    // has_neon not needed, aarch64 has NEON guaranteed
+    bool has_dotprod     = false;
+    bool has_fp16_va     = false;
+    bool has_sve         = false;
+    bool has_sve2        = false;
+    bool has_i8mm        = false;
+    bool has_sme         = false;
+
+    aarch64_features() {
+#if defined(__linux__)
+        uint32_t hwcap = getauxval(AT_HWCAP);
+        uint32_t hwcap2 = getauxval(AT_HWCAP2);
+
+        has_dotprod = !!(hwcap & HWCAP_ASIMDDP);
+        has_fp16_va = !!(hwcap & HWCAP_FPHP);
+        has_sve     = !!(hwcap & HWCAP_SVE);
+        has_sve2    = !!(hwcap2 & HWCAP2_SVE2);
+        has_i8mm    = !!(hwcap2 & HWCAP2_I8MM);
+        has_sme     = !!(hwcap2 & HWCAP2_SME);
+#elif defined(__APPLE__)
+        int oldp = 0;
+        size_t size = sizeof(oldp);
+
+        if (sysctlbyname("hw.optional.arm.FEAT_DotProd", &oldp, &size, NULL, 0) == 0) {
+            has_dotprod = static_cast<bool>(oldp);
+        }
+
+        if (sysctlbyname("hw.optional.arm.FEAT_I8MM", &oldp, &size, NULL, 0) == 0) {
+            has_i8mm = static_cast<bool>(oldp);
+        }
+
+        if (sysctlbyname("hw.optional.arm.FEAT_SME", &oldp, &size, NULL, 0) == 0) {
+            has_sme = static_cast<bool>(oldp);
+        }
+
+        // Apple apparently does not implement SVE yet
+#endif
+    }
+};
+
+static int ggml_backend_cpu_aarch64_score() {
+    int score = 1;
+    aarch64_features af;
+
+#ifdef GGML_USE_DOTPROD
+    if (!af.has_dotprod) { return 0; }
+    score += 1<<1;
+#endif
+#ifdef GGML_USE_FP16_VECTOR_ARITHMETIC
+    if (!af.has_fp16_va) { return 0; }
+    score += 1<<2;
+#endif
+#ifdef GGML_USE_SVE
+    if (!af.has_sve) { return 0; }
+    score += 1<<3;
+#endif
+#ifdef GGML_USE_MATMUL_INT8
+    if (!af.has_i8mm) { return 0; }
+    score += 1<<4;
+#endif
+#ifdef GGML_USE_SVE2
+    if (!af.has_sve2) { return 0; }
+    score += 1<<5;
+#endif
+#ifdef GGML_USE_SME
+    if (!af.has_sme) { return 0; }
+    score += 1<<6;
+#endif
+
+    return score;
+}
+
+GGML_BACKEND_DL_SCORE_IMPL(ggml_backend_cpu_aarch64_score)
+
+# endif // defined(__aarch64__)

ggml/src/ggml-cpu/arch/powerpc/cpu-feats.cpp

@@ -0,0 +1,82 @@
+# include "ggml-backend-impl.h"
+
+#if defined(__powerpc64__) || defined(__ppc64__) || defined(__PPC64__)
+
+#if defined(__linux__)
+#include <sys/auxv.h>
+#endif
+
+#include <string>
+
+struct powerpc_features {
+    std::string platform = "";
+    int power_version    = -1;
+
+    bool has_vsx         = false;
+
+    powerpc_features() {
+#if defined(__linux__)
+        unsigned long auxval = getauxval(AT_PLATFORM);
+        if (auxval) {
+            platform = std::string(reinterpret_cast<const char*>(auxval));
+            // TBD: Do systems exist that return this in uppercase?
+            if (platform.substr(0, 5) == "power") {
+                // Extractt a numeric suffix, if one exists
+                int vpos = -1;
+                for (int i = platform.length() - 1; i >= 0; i--) {
+                    if (std::isdigit(platform[i])) {
+                        vpos = i;
+                    } else {
+                        break;
+                    }
+                }
+                if (vpos > -1) {
+                    power_version = std::stoi(platform.substr(vpos));
+                }
+            }
+        }
+#endif
+        if (power_version >= 9) {
+            has_vsx = true;
+        }
+    }
+};
+
+static int ggml_backend_cpu_powerpc_score() {
+    int score = 1;
+    powerpc_features pf;
+
+// Platform scores
+#if defined(GGML_USE_POWER7)
+    if (pf.power_version < 7) { return 0; }
+    score += 1<<1;
+#endif
+#if defined(GGML_USE_POWER8)
+    if (pf.power_version < 8) { return 0; }
+    score += 1<<2;
+#endif
+#if defined(GGML_USE_POWER9)
+    if (pf.power_version < 9) { return 0; }
+    score += 1<<3;
+#endif
+#if defined(GGML_USE_POWER10)
+    if (pf.power_version < 10) { return 0; }
+    score += 1<<4;
+#endif
+#if defined(GGML_USE_POWER11)
+    if (pf.power_version < 11) { return 0; }
+    score += 1<<5;
+#endif
+
+// Feature scores
+#if defined(GGML_USE_VSX)
+    if (!pf.has_vsx) { return 0; }
+    score += 1<<6;
+#endif
+
+    return score;
+}
+
+GGML_BACKEND_DL_SCORE_IMPL(ggml_backend_cpu_powerpc_score)
+
+#endif // defined(__powerpc64__) || defined(__ppc64__) || defined(__PPC64__)

ggml/src/ggml-cpu/arch/riscv/repack.cpp

@@ -0,0 +1,397 @@
+#define GGML_COMMON_IMPL_CPP
+#define GGML_COMMON_DECL_CPP
+#include "ggml-common.h"
+#include "ggml-backend-impl.h"
+
+#include "ggml-impl.h"
+#include "ggml-cpu.h"
+#include "ggml-cpu-impl.h"
+#include "simd-mappings.h"
+#include "traits.h"
+
+#include <cmath>
+#include <cstring>
+#include <cassert>
+#include <cstdlib> // for qsort
+#include <cstdio>  // for GGML_ASSERT
+
+#define GGML_CPU_CLANG_WORKAROUND
+#include "../../repack.h"
+
+#if defined(__GNUC__)
+#pragma GCC diagnostic ignored "-Woverlength-strings"
+#endif
+
+#define UNUSED GGML_UNUSED
+
+void ggml_gemv_q4_0_8x8_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc) {
+    const int qk = QK8_0;
+    const int nb = n / qk;
+    const int ncols_interleaved = 8;
+    const int blocklen = 8;
+
+    assert (n % qk == 0);
+    assert (nc % ncols_interleaved == 0);
+
+    UNUSED(s);
+    UNUSED(bs);
+    UNUSED(vx);
+    UNUSED(vy);
+    UNUSED(nr);
+    UNUSED(nc);
+    UNUSED(nb);
+    UNUSED(ncols_interleaved);
+    UNUSED(blocklen);
+
+#if defined __riscv_v
+    if (__riscv_vlenb() >= QK4_0) {
+        const size_t vl = QK4_0;
+
+        const block_q8_0 * a_ptr = (const block_q8_0 *) vy;
+        for (int x = 0; x < nc / ncols_interleaved; x++) {
+            const block_q4_0x8 * b_ptr = (const block_q4_0x8 *) vx + (x * nb);
+
+            vfloat32m1_t sumf = __riscv_vfmv_v_f_f32m1(0.0, vl / 4);
+            for (int l = 0; l < nb; l++) {
+                const int64_t a0 = *(const int64_t *)&a_ptr[l].qs[0];
+                const int64_t a1 = *(const int64_t *)&a_ptr[l].qs[8];
+                const int64_t a2 = *(const int64_t *)&a_ptr[l].qs[16];
+                const int64_t a3 = *(const int64_t *)&a_ptr[l].qs[24];
+                __asm__ __volatile__("" ::: "memory"); // prevent gcc from emitting fused vlse64, violating alignment constraints
+                const vint8m2_t lhs_0_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(a0, vl / 4));
+                const vint8m2_t lhs_1_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(a1, vl / 4));
+                const vint8m2_t lhs_2_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(a2, vl / 4));
+                const vint8m2_t lhs_3_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(a3, vl / 4));
+
+                const vint8m4_t rhs_raw_vec = __riscv_vle8_v_i8m4((const int8_t *)b_ptr[l].qs, vl * 4);
+                const vint8m4_t rhs_vec_lo = __riscv_vsra_vx_i8m4(__riscv_vsll_vx_i8m4(rhs_raw_vec, 4, vl * 4), 4, vl * 4);
+                const vint8m4_t rhs_vec_hi = __riscv_vsra_vx_i8m4(rhs_raw_vec, 4, vl * 4);
+                const vint8m2_t rhs_vec_lo_0 = __riscv_vget_v_i8m4_i8m2(rhs_vec_lo, 0);
+                const vint8m2_t rhs_vec_lo_1 = __riscv_vget_v_i8m4_i8m2(rhs_vec_lo, 1);
+                const vint8m2_t rhs_vec_hi_0 = __riscv_vget_v_i8m4_i8m2(rhs_vec_hi, 0);
+                const vint8m2_t rhs_vec_hi_1 = __riscv_vget_v_i8m4_i8m2(rhs_vec_hi, 1);
+
+                const vint16m4_t sumi_lo_0 = __riscv_vwmul_vv_i16m4(rhs_vec_lo_0, lhs_0_8, vl * 2);
+                const vint16m4_t sumi_lo_1 = __riscv_vwmacc_vv_i16m4(sumi_lo_0, rhs_vec_lo_1, lhs_1_8, vl * 2);
+                const vint16m4_t sumi_hi_0 = __riscv_vwmacc_vv_i16m4(sumi_lo_1, rhs_vec_hi_0, lhs_2_8, vl * 2);
+                const vint16m4_t sumi_hi_m = __riscv_vwmacc_vv_i16m4(sumi_hi_0, rhs_vec_hi_1, lhs_3_8, vl * 2);
+
+                const vuint32m4_t sumi_i32 = __riscv_vreinterpret_v_i32m4_u32m4(__riscv_vreinterpret_v_i16m4_i32m4(sumi_hi_m));
+                const vuint16m2_t sumi_h2_0 = __riscv_vnsrl_wx_u16m2(sumi_i32, 0, vl);
+                const vuint16m2_t sumi_h2_1 = __riscv_vnsrl_wx_u16m2(sumi_i32, 16, vl);
+                const vuint16m2_t sumi_h2 = __riscv_vadd_vv_u16m2(sumi_h2_0, sumi_h2_1, vl);
+                const vuint32m2_t sumi_h2_i32 = __riscv_vreinterpret_v_u16m2_u32m2(sumi_h2);
+                const vuint16m1_t sumi_h4_0 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 0, vl / 2);
+                const vuint16m1_t sumi_h4_1 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 16, vl / 2);
+                const vuint16m1_t sumi_h4 = __riscv_vadd_vv_u16m1(sumi_h4_0, sumi_h4_1, vl / 2);
+                const vuint32m1_t sumi_h4_i32 = __riscv_vreinterpret_v_u16m1_u32m1(sumi_h4);
+                const vint16mf2_t sumi_h8_0 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 0, vl / 4));
+                const vint16mf2_t sumi_h8_1 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 16, vl / 4));
+                const vint32m1_t sumi_h8 = __riscv_vwadd_vv_i32m1(sumi_h8_0, sumi_h8_1, vl / 4);
+                const vfloat32m1_t facc = __riscv_vfcvt_f_x_v_f32m1(sumi_h8, vl / 4);
+
+                // vector version needs Zvfhmin extension
+                const float a_scale = GGML_CPU_FP16_TO_FP32(a_ptr[l].d);
+                const float b_scales[8] = {
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[0]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[1]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[2]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[3]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[4]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[5]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[6]),
+                    GGML_CPU_FP16_TO_FP32(b_ptr[l].d[7])
+                };
+                const vfloat32m1_t b_scales_vec = __riscv_vle32_v_f32m1(b_scales, vl / 4);
+                const vfloat32m1_t tmp1 = __riscv_vfmul_vf_f32m1(facc, a_scale, vl / 4);
+                sumf = __riscv_vfmacc_vv_f32m1(sumf, tmp1, b_scales_vec, vl / 4);
+            }
+            __riscv_vse32_v_f32m1(s + x * ncols_interleaved, sumf, vl / 4);
+        }
+        return;
+    }
+
+#endif
+    {
+        float sumf[8];
+        int sumi;
+
+        const block_q8_0 * a_ptr = (const block_q8_0 *) vy;
+        for (int x = 0; x < nc / ncols_interleaved; x++) {
+            const block_q4_0x8 * b_ptr = (const block_q4_0x8 *) vx + (x * nb);
+
+            for (int j = 0; j < ncols_interleaved; j++) sumf[j] = 0.0;
+            for (int l = 0; l < nb; l++) {
+                for (int k = 0; k < (qk / (2 * blocklen)); k++) {
+                    for (int j = 0; j < ncols_interleaved; j++) {
+                        sumi = 0;
+                        for (int i = 0; i < blocklen; ++i) {
+                            const int v0 = (int8_t) (b_ptr[l].qs[k * ncols_interleaved * blocklen + j * blocklen + i] << 4);
+                            const int v1 = (int8_t) (b_ptr[l].qs[k * ncols_interleaved * blocklen + j * blocklen + i] & 0xF0);
+                            sumi += ((v0 * a_ptr[l].qs[k * blocklen + i]) + (v1 * a_ptr[l].qs[k * blocklen + i + qk / 2])) >> 4;
+                        }
+                        sumf[j] += sumi * GGML_CPU_FP16_TO_FP32(b_ptr[l].d[j]) * GGML_CPU_FP16_TO_FP32(a_ptr[l].d);
+                    }
+                }
+            }
+            for (int j = 0; j < ncols_interleaved; j++) s[x * ncols_interleaved + j] = sumf[j];
+        }
+    }
+}
+
+void ggml_gemm_q4_0_8x8_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc) {
+    const int qk = QK8_0;
+    const int nb = n / qk;
+    const int ncols_interleaved = 8;
+    const int blocklen = 8;
+
+    assert (n % qk == 0);
+    assert (nr % 4 == 0);
+    assert (nc % ncols_interleaved == 0);
+
+    UNUSED(s);
+    UNUSED(bs);
+    UNUSED(vx);
+    UNUSED(vy);
+    UNUSED(nr);
+    UNUSED(nc);
+    UNUSED(nb);
+    UNUSED(ncols_interleaved);
+    UNUSED(blocklen);
+
+#if defined __riscv_v
+    if (__riscv_vlenb() >= QK4_0) {
+        const size_t vl = QK4_0;
+
+        for (int y = 0; y < nr / 4; y++) {
+            const block_q8_0x4 * a_ptr = (const block_q8_0x4 *) vy + (y * nb);
+            for (int x = 0; x < nc / ncols_interleaved; x++) {
+                const block_q4_0x8 * b_ptr = (const block_q4_0x8 *) vx + (x * nb);
+                vfloat32m1_t sumf0 = __riscv_vfmv_v_f_f32m1(0.0, vl / 4);
+                vfloat32m1_t sumf1 = __riscv_vfmv_v_f_f32m1(0.0, vl / 4);
+                vfloat32m1_t sumf2 = __riscv_vfmv_v_f_f32m1(0.0, vl / 4);
+                vfloat32m1_t sumf3 = __riscv_vfmv_v_f_f32m1(0.0, vl / 4);
+                for (int l = 0; l < nb; l++) {
+                    const vint8m4_t rhs_raw_vec = __riscv_vle8_v_i8m4((const int8_t *)b_ptr[l].qs, vl * 4);
+                    const vint8m4_t rhs_vec_lo = __riscv_vsra_vx_i8m4(__riscv_vsll_vx_i8m4(rhs_raw_vec, 4, vl * 4), 4, vl * 4);
+                    const vint8m4_t rhs_vec_hi = __riscv_vsra_vx_i8m4(rhs_raw_vec, 4, vl * 4);
+                    const vint8m2_t rhs_vec_lo_0 = __riscv_vget_v_i8m4_i8m2(rhs_vec_lo, 0);
+                    const vint8m2_t rhs_vec_lo_1 = __riscv_vget_v_i8m4_i8m2(rhs_vec_lo, 1);
+                    const vint8m2_t rhs_vec_hi_0 = __riscv_vget_v_i8m4_i8m2(rhs_vec_hi, 0);
+                    const vint8m2_t rhs_vec_hi_1 = __riscv_vget_v_i8m4_i8m2(rhs_vec_hi, 1);
+
+                    // vector version needs Zvfhmin extension
+                    const float a_scales[4] = {
+                        GGML_CPU_FP16_TO_FP32(a_ptr[l].d[0]),
+                        GGML_CPU_FP16_TO_FP32(a_ptr[l].d[1]),
+                        GGML_CPU_FP16_TO_FP32(a_ptr[l].d[2]),
+                        GGML_CPU_FP16_TO_FP32(a_ptr[l].d[3])
+                    };
+                    const float b_scales[8] = {
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[0]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[1]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[2]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[3]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[4]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[5]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[6]),
+                        GGML_CPU_FP16_TO_FP32(b_ptr[l].d[7])
+                    };
+                    const vfloat32m1_t b_scales_vec = __riscv_vle32_v_f32m1(b_scales, vl / 4);
+
+                    const int64_t A0 = *(const int64_t *)&a_ptr[l].qs[0];
+                    const int64_t A4 = *(const int64_t *)&a_ptr[l].qs[32];
+                    const int64_t A8 = *(const int64_t *)&a_ptr[l].qs[64];
+                    const int64_t Ac = *(const int64_t *)&a_ptr[l].qs[96];
+                    __asm__ __volatile__("" ::: "memory"); // prevent gcc from emitting fused vlse64, violating alignment
+                    vint16m4_t sumi_l0;
+                    {
+                        const vint8m2_t lhs_0_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A0, vl / 4));
+                        const vint8m2_t lhs_1_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A4, vl / 4));
+                        const vint8m2_t lhs_2_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A8, vl / 4));
+                        const vint8m2_t lhs_3_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(Ac, vl / 4));
+                        const vint16m4_t sumi_lo_0 = __riscv_vwmul_vv_i16m4(rhs_vec_lo_0, lhs_0_8, vl * 2);
+                        const vint16m4_t sumi_lo_1 = __riscv_vwmacc_vv_i16m4(sumi_lo_0, rhs_vec_lo_1, lhs_1_8, vl * 2);
+                        const vint16m4_t sumi_hi_0 = __riscv_vwmacc_vv_i16m4(sumi_lo_1, rhs_vec_hi_0, lhs_2_8, vl * 2);
+                        const vint16m4_t sumi_hi_m = __riscv_vwmacc_vv_i16m4(sumi_hi_0, rhs_vec_hi_1, lhs_3_8, vl * 2);
+
+                        sumi_l0 = sumi_hi_m;
+                    }
+
+                    {
+                        const vuint32m4_t sumi_i32 = __riscv_vreinterpret_v_i32m4_u32m4(__riscv_vreinterpret_v_i16m4_i32m4(sumi_l0));
+                        const vuint16m2_t sumi_h2_0 = __riscv_vnsrl_wx_u16m2(sumi_i32, 0, vl);
+                        const vuint16m2_t sumi_h2_1 = __riscv_vnsrl_wx_u16m2(sumi_i32, 16, vl);
+                        const vuint16m2_t sumi_h2 = __riscv_vadd_vv_u16m2(sumi_h2_0, sumi_h2_1, vl);
+                        const vuint32m2_t sumi_h2_i32 = __riscv_vreinterpret_v_u16m2_u32m2(sumi_h2);
+                        const vuint16m1_t sumi_h4_0 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 0, vl / 2);
+                        const vuint16m1_t sumi_h4_1 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 16, vl / 2);
+                        const vuint16m1_t sumi_h4 = __riscv_vadd_vv_u16m1(sumi_h4_0, sumi_h4_1, vl / 2);
+                        const vuint32m1_t sumi_h4_i32 = __riscv_vreinterpret_v_u16m1_u32m1(sumi_h4);
+                        const vint16mf2_t sumi_h8_0 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 0, vl / 4));
+                        const vint16mf2_t sumi_h8_1 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 16, vl / 4));
+                        const vint32m1_t sumi_h8 = __riscv_vwadd_vv_i32m1(sumi_h8_0, sumi_h8_1, vl / 4);
+                        const vfloat32m1_t facc = __riscv_vfcvt_f_x_v_f32m1(sumi_h8, vl / 4);
+
+                        const vfloat32m1_t tmp1 = __riscv_vfmul_vf_f32m1(facc, a_scales[0], vl / 4);
+                        sumf0 = __riscv_vfmacc_vv_f32m1(sumf0, tmp1, b_scales_vec, vl / 4);
+                    }
+
+                    const int64_t A1 = *(const int64_t *)&a_ptr[l].qs[8];
+                    const int64_t A5 = *(const int64_t *)&a_ptr[l].qs[40];
+                    const int64_t A9 = *(const int64_t *)&a_ptr[l].qs[72];
+                    const int64_t Ad = *(const int64_t *)&a_ptr[l].qs[104];
+                    __asm__ __volatile__("" ::: "memory"); // prevent gcc from emitting fused vlse64, violating alignment
+                    vint16m4_t sumi_l1;
+                    {
+                        const vint8m2_t lhs_0_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A1, vl / 4));
+                        const vint8m2_t lhs_1_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A5, vl / 4));
+                        const vint8m2_t lhs_2_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A9, vl / 4));
+                        const vint8m2_t lhs_3_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(Ad, vl / 4));
+                        const vint16m4_t sumi_lo_0 = __riscv_vwmul_vv_i16m4(rhs_vec_lo_0, lhs_0_8, vl * 2);
+                        const vint16m4_t sumi_lo_1 = __riscv_vwmacc_vv_i16m4(sumi_lo_0, rhs_vec_lo_1, lhs_1_8, vl * 2);
+                        const vint16m4_t sumi_hi_0 = __riscv_vwmacc_vv_i16m4(sumi_lo_1, rhs_vec_hi_0, lhs_2_8, vl * 2);
+                        const vint16m4_t sumi_hi_m = __riscv_vwmacc_vv_i16m4(sumi_hi_0, rhs_vec_hi_1, lhs_3_8, vl * 2);
+
+                        sumi_l1 = sumi_hi_m;
+                    }
+
+                    {
+                        const vuint32m4_t sumi_i32 = __riscv_vreinterpret_v_i32m4_u32m4(__riscv_vreinterpret_v_i16m4_i32m4(sumi_l1));
+                        const vuint16m2_t sumi_h2_0 = __riscv_vnsrl_wx_u16m2(sumi_i32, 0, vl);
+                        const vuint16m2_t sumi_h2_1 = __riscv_vnsrl_wx_u16m2(sumi_i32, 16, vl);
+                        const vuint16m2_t sumi_h2 = __riscv_vadd_vv_u16m2(sumi_h2_0, sumi_h2_1, vl);
+                        const vuint32m2_t sumi_h2_i32 = __riscv_vreinterpret_v_u16m2_u32m2(sumi_h2);
+                        const vuint16m1_t sumi_h4_0 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 0, vl / 2);
+                        const vuint16m1_t sumi_h4_1 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 16, vl / 2);
+                        const vuint16m1_t sumi_h4 = __riscv_vadd_vv_u16m1(sumi_h4_0, sumi_h4_1, vl / 2);
+                        const vuint32m1_t sumi_h4_i32 = __riscv_vreinterpret_v_u16m1_u32m1(sumi_h4);
+                        const vint16mf2_t sumi_h8_0 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 0, vl / 4));
+                        const vint16mf2_t sumi_h8_1 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 16, vl / 4));
+                        const vint32m1_t sumi_h8 = __riscv_vwadd_vv_i32m1(sumi_h8_0, sumi_h8_1, vl / 4);
+                        const vfloat32m1_t facc = __riscv_vfcvt_f_x_v_f32m1(sumi_h8, vl / 4);
+
+                        const vfloat32m1_t tmp1 = __riscv_vfmul_vf_f32m1(facc, a_scales[1], vl / 4);
+                        sumf1 = __riscv_vfmacc_vv_f32m1(sumf1, tmp1, b_scales_vec, vl / 4);
+                    }
+
+                    const int64_t A2 = *(const int64_t *)&a_ptr[l].qs[16];
+                    const int64_t A6 = *(const int64_t *)&a_ptr[l].qs[48];
+                    const int64_t Aa = *(const int64_t *)&a_ptr[l].qs[80];
+                    const int64_t Ae = *(const int64_t *)&a_ptr[l].qs[112];
+                    __asm__ __volatile__("" ::: "memory"); // prevent gcc from emitting fused vlse64, violating alignment
+                    vint16m4_t sumi_l2;
+                    {
+                        const vint8m2_t lhs_0_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A2, vl / 4));
+                        const vint8m2_t lhs_1_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A6, vl / 4));
+                        const vint8m2_t lhs_2_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(Aa, vl / 4));
+                        const vint8m2_t lhs_3_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(Ae, vl / 4));
+                        const vint16m4_t sumi_lo_0 = __riscv_vwmul_vv_i16m4(rhs_vec_lo_0, lhs_0_8, vl * 2);
+                        const vint16m4_t sumi_lo_1 = __riscv_vwmacc_vv_i16m4(sumi_lo_0, rhs_vec_lo_1, lhs_1_8, vl * 2);
+                        const vint16m4_t sumi_hi_0 = __riscv_vwmacc_vv_i16m4(sumi_lo_1, rhs_vec_hi_0, lhs_2_8, vl * 2);
+                        const vint16m4_t sumi_hi_m = __riscv_vwmacc_vv_i16m4(sumi_hi_0, rhs_vec_hi_1, lhs_3_8, vl * 2);
+
+                        sumi_l2 = sumi_hi_m;
+                    }
+
+                    {
+                        const vuint32m4_t sumi_i32 = __riscv_vreinterpret_v_i32m4_u32m4(__riscv_vreinterpret_v_i16m4_i32m4(sumi_l2));
+                        const vuint16m2_t sumi_h2_0 = __riscv_vnsrl_wx_u16m2(sumi_i32, 0, vl);
+                        const vuint16m2_t sumi_h2_1 = __riscv_vnsrl_wx_u16m2(sumi_i32, 16, vl);
+                        const vuint16m2_t sumi_h2 = __riscv_vadd_vv_u16m2(sumi_h2_0, sumi_h2_1, vl);
+                        const vuint32m2_t sumi_h2_i32 = __riscv_vreinterpret_v_u16m2_u32m2(sumi_h2);
+                        const vuint16m1_t sumi_h4_0 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 0, vl / 2);
+                        const vuint16m1_t sumi_h4_1 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 16, vl / 2);
+                        const vuint16m1_t sumi_h4 = __riscv_vadd_vv_u16m1(sumi_h4_0, sumi_h4_1, vl / 2);
+                        const vuint32m1_t sumi_h4_i32 = __riscv_vreinterpret_v_u16m1_u32m1(sumi_h4);
+                        const vint16mf2_t sumi_h8_0 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 0, vl / 4));
+                        const vint16mf2_t sumi_h8_1 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 16, vl / 4));
+                        const vint32m1_t sumi_h8 = __riscv_vwadd_vv_i32m1(sumi_h8_0, sumi_h8_1, vl / 4);
+                        const vfloat32m1_t facc = __riscv_vfcvt_f_x_v_f32m1(sumi_h8, vl / 4);
+
+                        const vfloat32m1_t tmp1 = __riscv_vfmul_vf_f32m1(facc, a_scales[2], vl / 4);
+                        sumf2 = __riscv_vfmacc_vv_f32m1(sumf2, tmp1, b_scales_vec, vl / 4);
+                    }
+
+                    const int64_t A3 = *(const int64_t *)&a_ptr[l].qs[24];
+                    const int64_t A7 = *(const int64_t *)&a_ptr[l].qs[56];
+                    const int64_t Ab = *(const int64_t *)&a_ptr[l].qs[88];
+                    const int64_t Af = *(const int64_t *)&a_ptr[l].qs[120];
+                    __asm__ __volatile__("" ::: "memory"); // prevent gcc from emitting fused vlse64, violating alignment
+                    vint16m4_t sumi_l3;
+                    {
+                        const vint8m2_t lhs_0_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A3, vl / 4));
+                        const vint8m2_t lhs_1_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(A7, vl / 4));
+                        const vint8m2_t lhs_2_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(Ab, vl / 4));
+                        const vint8m2_t lhs_3_8 =__riscv_vreinterpret_v_i64m2_i8m2(__riscv_vmv_v_x_i64m2(Af, vl / 4));
+                        const vint16m4_t sumi_lo_0 = __riscv_vwmul_vv_i16m4(rhs_vec_lo_0, lhs_0_8, vl * 2);
+                        const vint16m4_t sumi_lo_1 = __riscv_vwmacc_vv_i16m4(sumi_lo_0, rhs_vec_lo_1, lhs_1_8, vl * 2);
+                        const vint16m4_t sumi_hi_0 = __riscv_vwmacc_vv_i16m4(sumi_lo_1, rhs_vec_hi_0, lhs_2_8, vl * 2);
+                        const vint16m4_t sumi_hi_m = __riscv_vwmacc_vv_i16m4(sumi_hi_0, rhs_vec_hi_1, lhs_3_8, vl * 2);
+
+                        sumi_l3 = sumi_hi_m;
+                    }
+
+                    {
+                        const vuint32m4_t sumi_i32 = __riscv_vreinterpret_v_i32m4_u32m4(__riscv_vreinterpret_v_i16m4_i32m4(sumi_l3));
+                        const vuint16m2_t sumi_h2_0 = __riscv_vnsrl_wx_u16m2(sumi_i32, 0, vl);
+                        const vuint16m2_t sumi_h2_1 = __riscv_vnsrl_wx_u16m2(sumi_i32, 16, vl);
+                        const vuint16m2_t sumi_h2 = __riscv_vadd_vv_u16m2(sumi_h2_0, sumi_h2_1, vl);
+                        const vuint32m2_t sumi_h2_i32 = __riscv_vreinterpret_v_u16m2_u32m2(sumi_h2);
+                        const vuint16m1_t sumi_h4_0 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 0, vl / 2);
+                        const vuint16m1_t sumi_h4_1 = __riscv_vnsrl_wx_u16m1(sumi_h2_i32, 16, vl / 2);
+                        const vuint16m1_t sumi_h4 = __riscv_vadd_vv_u16m1(sumi_h4_0, sumi_h4_1, vl / 2);
+                        const vuint32m1_t sumi_h4_i32 = __riscv_vreinterpret_v_u16m1_u32m1(sumi_h4);
+                        const vint16mf2_t sumi_h8_0 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 0, vl / 4));
+                        const vint16mf2_t sumi_h8_1 = __riscv_vreinterpret_v_u16mf2_i16mf2(__riscv_vnsrl_wx_u16mf2(sumi_h4_i32, 16, vl / 4));
+                        const vint32m1_t sumi_h8 = __riscv_vwadd_vv_i32m1(sumi_h8_0, sumi_h8_1, vl / 4);
+                        const vfloat32m1_t facc = __riscv_vfcvt_f_x_v_f32m1(sumi_h8, vl / 4);
+
+                        const vfloat32m1_t tmp1 = __riscv_vfmul_vf_f32m1(facc, a_scales[3], vl / 4);
+                        sumf3 = __riscv_vfmacc_vv_f32m1(sumf3, tmp1, b_scales_vec, vl / 4);
+                    }
+                }
+                __riscv_vse32_v_f32m1(&s[(y * 4 + 0) * bs + x * ncols_interleaved], sumf0, vl / 4);
+                __riscv_vse32_v_f32m1(&s[(y * 4 + 1) * bs + x * ncols_interleaved], sumf1, vl / 4);
+                __riscv_vse32_v_f32m1(&s[(y * 4 + 2) * bs + x * ncols_interleaved], sumf2, vl / 4);
+                __riscv_vse32_v_f32m1(&s[(y * 4 + 3) * bs + x * ncols_interleaved], sumf3, vl / 4);
+            }
+        }
+
+        return;
+    }
+
+#endif // #if ! ((defined(_MSC_VER)) && ! defined(__clang__)) && defined(__aarch64__)
+    float sumf[4][8];
+    int sumi;
+
+    for (int y = 0; y < nr / 4; y++) {
+        const block_q8_0x4 * a_ptr = (const block_q8_0x4 *) vy + (y * nb);
+        for (int x = 0; x < nc / ncols_interleaved; x++) {
+            const block_q4_0x8 * b_ptr = (const block_q4_0x8 *) vx + (x * nb);
+            for (int m = 0; m < 4; m++) {
+                for (int j = 0; j < ncols_interleaved; j++) sumf[m][j] = 0.0;
+            }
+            for (int l = 0; l < nb; l++) {
+                for (int k = 0; k < (qk / (2 * blocklen)); k++) {
+                    for (int m = 0; m < 4; m++) {
+                        for (int j = 0; j < ncols_interleaved; j++) {
+                            sumi = 0;
+                            for (int i = 0; i < blocklen; ++i) {
+                                const int v0 = (int8_t) (b_ptr[l].qs[k * ncols_interleaved * blocklen + j * blocklen + i] << 4);
+                                const int v1 = (int8_t) (b_ptr[l].qs[k * ncols_interleaved * blocklen + j * blocklen + i] & 0xF0);
+                                sumi += ((v0 * a_ptr[l].qs[k * 4 * blocklen + m * blocklen + i]) +
+                                         (v1 * a_ptr[l].qs[k * 4 * blocklen + m * blocklen + i + qk / 2 * 4])) >> 4;
+                            }
+                            sumf[m][j] += sumi * GGML_CPU_FP16_TO_FP32(b_ptr[l].d[j]) * GGML_CPU_FP16_TO_FP32(a_ptr[l].d[m]);
+                        }
+                    }
+                }
+            }
+            for (int m = 0; m < 4; m++) {
+                for (int j = 0; j < ncols_interleaved; j++)
+                    s[(y * 4 + m) * bs + x * ncols_interleaved + j] = sumf[m][j];
+            }
+        }
+    }
+}

ggml/src/ggml-cpu/arch/x86/cpu-feats.cpp

@@ -0,0 +1,327 @@
+#include "ggml-backend-impl.h"
+
+#if defined(__x86_64__) || (defined(_MSC_VER) && defined(_M_AMD64))
+
+#ifdef _MSC_VER
+#include <intrin.h>
+#endif
+
+#include <cstring>
+#include <vector>
+#include <bitset>
+#include <array>
+#include <string>
+
+// ref: https://cdrdv2-public.intel.com/782156/325383-sdm-vol-2abcd.pdf
+struct cpuid_x86 {
+    bool SSE3(void) { return f_1_ecx[0]; }
+    bool PCLMULQDQ(void) { return f_1_ecx[1]; }
+    bool MONITOR(void) { return f_1_ecx[3]; }
+    bool SSSE3(void) { return f_1_ecx[9]; }
+    bool FMA(void) { return f_1_ecx[12]; }
+    bool CMPXCHG16B(void) { return f_1_ecx[13]; }
+    bool SSE41(void) { return f_1_ecx[19]; }
+    bool SSE42(void) { return f_1_ecx[20]; }
+    bool MOVBE(void) { return f_1_ecx[22]; }
+    bool POPCNT(void) { return f_1_ecx[23]; }
+    bool AES(void) { return f_1_ecx[25]; }
+    bool XSAVE(void) { return f_1_ecx[26]; }
+    bool OSXSAVE(void) { return f_1_ecx[27]; }
+    bool AVX(void) { return f_1_ecx[28]; }
+    bool F16C(void) { return f_1_ecx[29]; }
+    bool RDRAND(void) { return f_1_ecx[30]; }
+
+    bool MSR(void) { return f_1_edx[5]; }
+    bool CX8(void) { return f_1_edx[8]; }
+    bool SEP(void) { return f_1_edx[11]; }
+    bool CMOV(void) { return f_1_edx[15]; }
+    bool CLFSH(void) { return f_1_edx[19]; }
+    bool MMX(void) { return f_1_edx[23]; }
+    bool FXSR(void) { return f_1_edx[24]; }
+    bool SSE(void) { return f_1_edx[25]; }
+    bool SSE2(void) { return f_1_edx[26]; }
+
+    bool FSGSBASE(void) { return f_7_ebx[0]; }
+    bool BMI1(void) { return f_7_ebx[3]; }
+    bool HLE(void) { return is_intel && f_7_ebx[4]; }
+    bool AVX2(void) { return f_7_ebx[5]; }
+    bool BMI2(void) { return f_7_ebx[8]; }
+    bool ERMS(void) { return f_7_ebx[9]; }
+    bool INVPCID(void) { return f_7_ebx[10]; }
+    bool RTM(void) { return is_intel && f_7_ebx[11]; }
+    bool AVX512F(void) { return f_7_ebx[16]; }
+    bool AVX512DQ(void) { return f_7_ebx[17]; }
+    bool RDSEED(void) { return f_7_ebx[18]; }
+    bool ADX(void) { return f_7_ebx[19]; }
+    bool AVX512PF(void) { return f_7_ebx[26]; }
+    bool AVX512ER(void) { return f_7_ebx[27]; }
+    bool AVX512CD(void) { return f_7_ebx[28]; }
+    bool AVX512BW(void) { return f_7_ebx[30]; }
+    bool AVX512VL(void) { return f_7_ebx[31]; }
+
+    bool SHA(void) { return f_7_ebx[29]; }
+
+    bool PREFETCHWT1(void) { return f_7_ecx[0]; }
+
+    bool LAHF(void) { return f_81_ecx[0]; }
+    bool LZCNT(void) { return is_intel && f_81_ecx[5]; }
+    bool ABM(void) { return is_amd && f_81_ecx[5]; }
+    bool SSE4a(void) { return is_amd && f_81_ecx[6]; }
+    bool XOP(void) { return is_amd && f_81_ecx[11]; }
+    bool TBM(void) { return is_amd && f_81_ecx[21]; }
+
+    bool SYSCALL(void) { return is_intel && f_81_edx[11]; }
+    bool MMXEXT(void) { return is_amd && f_81_edx[22]; }
+    bool RDTSCP(void) { return is_intel && f_81_edx[27]; }
+    bool _3DNOWEXT(void) { return is_amd && f_81_edx[30]; }
+    bool _3DNOW(void) { return is_amd && f_81_edx[31]; }
+
+    bool AVX512_VBMI(void) { return f_7_ecx[1]; }
+    bool AVX512_VNNI(void) { return f_7_ecx[11]; }
+    bool AVX512_FP16(void) { return f_7_edx[23]; }
+    bool AVX512_BF16(void) { return f_7_1_eax[5]; }
+    bool AVX_VNNI(void) { return f_7_1_eax[4]; }
+
+    bool AMX_TILE(void) { return f_7_edx[24]; }
+    bool AMX_INT8(void) { return f_7_edx[25]; }
+    bool AMX_FP16(void) { return f_7_1_eax[21]; }
+    bool AMX_BF16(void) { return f_7_edx[22]; }
+
+#ifdef _MSC_VER
+    static void cpuid(int cpu_info[4], int eax) {
+        __cpuid(cpu_info, eax);
+    }
+    static void cpuidex(int cpu_info[4], int eax, int ecx) {
+        __cpuidex(cpu_info, eax, ecx);
+    }
+#else
+    static void cpuid(int cpu_info[4], int eax) {
+        __asm__ __volatile__(
+            "cpuid"
+            : "=a"(cpu_info[0]), "=b"(cpu_info[1]), "=c"(cpu_info[2]), "=d"(cpu_info[3])
+            : "a"(eax), "c"(0));
+    }
+    static void cpuidex(int cpu_info[4], int eax, int ecx) {
+        __asm__ __volatile__(
+            "cpuid"
+            : "=a"(cpu_info[0]), "=b"(cpu_info[1]), "=c"(cpu_info[2]), "=d"(cpu_info[3])
+            : "a"(eax), "c"(ecx));
+    }
+#endif
+
+    cpuid_x86() {
+        std::array<int, 4> cpui;
+        std::vector<std::array<int, 4>> data;
+
+        // calling __cpuid with 0x0 as the function_id argument
+        // gets the number of the highest valid function ID.
+        cpuid(cpui.data(), 0);
+        int n_ids = cpui[0];
+
+        for (int i = 0; i <= n_ids; ++i) {
+            cpuidex(cpui.data(), i, 0);
+            data.push_back(cpui);
+        }
+
+        // capture vendor string
+        char vendor[0x20] = {};
+        *reinterpret_cast<int *>(vendor)     = data[0][1];
+        *reinterpret_cast<int *>(vendor + 4) = data[0][3];
+        *reinterpret_cast<int *>(vendor + 8) = data[0][2];
+        this->vendor = vendor;
+        if (this->vendor == "GenuineIntel") {
+            is_intel = true;
+        } else if (this->vendor == "AuthenticAMD") {
+            is_amd = true;
+        }
+
+        // load bitset with flags for function 0x00000001
+        if (n_ids >= 1) {
+            f_1_ecx = data[1][2];
+            f_1_edx = data[1][3];
+        }
+
+        // load bitset with flags for function 0x00000007
+        if (n_ids >= 7) {
+            f_7_ebx = data[7][1];
+            f_7_ecx = data[7][2];
+            f_7_edx = data[7][3];
+            cpuidex(cpui.data(), 7, 1);
+            f_7_1_eax = cpui[0];
+        }
+
+        // calling __cpuid with 0x80000000 as the function_id argument
+        // gets the number of the highest valid extended ID.
+        cpuid(cpui.data(), 0x80000000);
+        unsigned int n_ex_ids = cpui[0];
+
+        std::vector<std::array<int, 4>> ext_data;
+        for (unsigned int i = 0x80000000; i <= n_ex_ids; ++i) {
+            cpuidex(cpui.data(), i, 0);
+            ext_data.push_back(cpui);
+        }
+
+        // load bitset with flags for function 0x80000001
+        if (n_ex_ids >= 0x80000001) {
+            f_81_ecx = ext_data[1][2];
+            f_81_edx = ext_data[1][3];
+        }
+
+        // interpret CPU brand string if reported
+        char brand[0x40] = {};
+        if (n_ex_ids >= 0x80000004) {
+            std::memcpy(brand, ext_data[2].data(), sizeof(cpui));
+            std::memcpy(brand + 16, ext_data[3].data(), sizeof(cpui));
+            std::memcpy(brand + 32, ext_data[4].data(), sizeof(cpui));
+            this->brand = brand;
+        }
+    }
+
+    bool is_intel = false;
+    bool is_amd = false;
+    std::string vendor;
+    std::string brand;
+    std::bitset<32> f_1_ecx;
+    std::bitset<32> f_1_edx;
+    std::bitset<32> f_7_ebx;
+    std::bitset<32> f_7_ecx;
+    std::bitset<32> f_7_edx;
+    std::bitset<32> f_7_1_eax;
+    std::bitset<32> f_81_ecx;
+    std::bitset<32> f_81_edx;
+};
+
+#if 0
+void test_x86_is() {
+    cpuid_x86 is;
+    printf("CPU Vendor: %s\n", is.vendor.c_str());
+    printf("Brand: %s\n", is.brand.c_str());
+    printf("is_intel: %d\n", is.is_intel);
+    printf("is_amd: %d\n", is.is_amd);
+    printf("sse3: %d\n", is.SSE3());
+    printf("pclmulqdq: %d\n", is.PCLMULQDQ());
+    printf("ssse3: %d\n", is.SSSE3());
+    printf("fma: %d\n", is.FMA());
+    printf("cmpxchg16b: %d\n", is.CMPXCHG16B());
+    printf("sse41: %d\n", is.SSE41());
+    printf("sse42: %d\n", is.SSE42());
+    printf("movbe: %d\n", is.MOVBE());
+    printf("popcnt: %d\n", is.POPCNT());
+    printf("aes: %d\n", is.AES());
+    printf("xsave: %d\n", is.XSAVE());
+    printf("osxsave: %d\n", is.OSXSAVE());
+    printf("avx: %d\n", is.AVX());
+    printf("f16c: %d\n", is.F16C());
+    printf("rdrand: %d\n", is.RDRAND());
+    printf("msr: %d\n", is.MSR());
+    printf("cx8: %d\n", is.CX8());
+    printf("sep: %d\n", is.SEP());
+    printf("cmov: %d\n", is.CMOV());
+    printf("clflush: %d\n", is.CLFSH());
+    printf("mmx: %d\n", is.MMX());
+    printf("fxsr: %d\n", is.FXSR());
+    printf("sse: %d\n", is.SSE());
+    printf("sse2: %d\n", is.SSE2());
+    printf("fsgsbase: %d\n", is.FSGSBASE());
+    printf("bmi1: %d\n", is.BMI1());
+    printf("hle: %d\n", is.HLE());
+    printf("avx2: %d\n", is.AVX2());
+    printf("bmi2: %d\n", is.BMI2());
+    printf("erms: %d\n", is.ERMS());
+    printf("invpcid: %d\n", is.INVPCID());
+    printf("rtm: %d\n", is.RTM());
+    printf("avx512f: %d\n", is.AVX512F());
+    printf("rdseed: %d\n", is.RDSEED());
+    printf("adx: %d\n", is.ADX());
+    printf("avx512pf: %d\n", is.AVX512PF());
+    printf("avx512er: %d\n", is.AVX512ER());
+    printf("avx512cd: %d\n", is.AVX512CD());
+    printf("sha: %d\n", is.SHA());
+    printf("prefetchwt1: %d\n", is.PREFETCHWT1());
+    printf("lahf: %d\n", is.LAHF());
+    printf("lzcnt: %d\n", is.LZCNT());
+    printf("abm: %d\n", is.ABM());
+    printf("sse4a: %d\n", is.SSE4a());
+    printf("xop: %d\n", is.XOP());
+    printf("tbm: %d\n", is.TBM());
+    printf("syscall: %d\n", is.SYSCALL());
+    printf("mmxext: %d\n", is.MMXEXT());
+    printf("rdtscp: %d\n", is.RDTSCP());
+    printf("3dnowext: %d\n", is._3DNOWEXT());
+    printf("3dnow: %d\n", is._3DNOW());
+    printf("avx512_vbmi: %d\n", is.AVX512_VBMI());
+    printf("avx512_vnni: %d\n", is.AVX512_VNNI());
+    printf("avx512_fp16: %d\n", is.AVX512_FP16());
+    printf("avx512_bf16: %d\n", is.AVX512_BF16());
+    printf("amx_tile: %d\n", is.AMX_TILE());
+    printf("amx_int8: %d\n", is.AMX_INT8());
+    printf("amx_fp16: %d\n", is.AMX_FP16());
+    printf("amx_bf16: %d\n", is.AMX_BF16());
+}
+#endif
+
+static int ggml_backend_cpu_x86_score() {
+    // FIXME: this does not check for OS support
+
+    int score = 1;
+    cpuid_x86 is;
+
+#ifdef GGML_FMA
+    if (!is.FMA()) { return 0; }
+    score += 1;
+#endif
+#ifdef GGML_F16C
+    if (!is.F16C()) { return 0; }
+    score += 1<<1;
+#endif
+#ifdef GGML_SSE42
+    if (!is.SSE42()) { return 0; }
+    score += 1<<2;
+#endif
+#ifdef GGML_BMI2
+    if (!is.BMI2()) { return 0; }
+    score += 1<<3;
+#endif
+#ifdef GGML_AVX
+    if (!is.AVX()) { return 0; }
+    score += 1<<4;
+#endif
+#ifdef GGML_AVX2
+    if (!is.AVX2()) { return 0; }
+    score += 1<<5;
+#endif
+#ifdef GGML_AVX_VNNI
+    if (!is.AVX_VNNI()) { return 0; }
+    score += 1<<6;
+#endif
+#ifdef GGML_AVX512
+    if (!is.AVX512F()) { return 0; }
+    if (!is.AVX512CD()) { return 0; }
+    if (!is.AVX512VL()) { return 0; }
+    if (!is.AVX512DQ()) { return 0; }
+    if (!is.AVX512BW()) { return 0; }
+    score += 1<<7;
+#endif
+#ifdef GGML_AVX512_VBMI
+    if (!is.AVX512_VBMI()) { return 0; }
+    score += 1<<8;
+#endif
+#ifdef GGML_AVX512_BF16
+    if (!is.AVX512_BF16()) { return 0; }
+    score += 1<<9;
+#endif
+#ifdef GGML_AVX512_VNNI
+    if (!is.AVX512_VNNI()) { return 0; }
+    score += 1<<10;
+#endif
+#ifdef GGML_AMX_INT8
+    if (!is.AMX_INT8()) { return 0; }
+    score += 1<<11;
+#endif
+
+    return score;
+}
+
+GGML_BACKEND_DL_SCORE_IMPL(ggml_backend_cpu_x86_score)
+
+#endif // defined(__x86_64__) || (defined(_MSC_VER) && defined(_M_AMD64))

ggml/src/ggml-cpu/binary-ops.cpp

@@ -0,0 +1,158 @@
+#include "binary-ops.h"
+
+#if defined(GGML_USE_ACCELERATE)
+#include <Accelerate/Accelerate.h>
+
+using vDSP_fn_t = void (*)(const float *, vDSP_Stride, const float *, vDSP_Stride, float *, vDSP_Stride, vDSP_Length);
+#endif
+
+static inline float op_add(float a, float b) {
+    return a + b;
+}
+
+static inline float op_sub(float a, float b) {
+    return a - b;
+}
+
+static inline float op_mul(float a, float b) {
+    return a * b;
+}
+
+static inline float op_div(float a, float b) {
+    return a / b;
+}
+
+template <float (*op)(float, float), typename src0_t, typename src1_t, typename dst_t>
+static inline void vec_binary_op_contiguous(const int64_t n, dst_t * z, const src0_t * x, const src1_t * y) {
+    constexpr auto src0_to_f32 = type_conversion_table<src0_t>::to_f32;
+    constexpr auto src1_to_f32 = type_conversion_table<src1_t>::to_f32;
+    constexpr auto f32_to_dst  = type_conversion_table<dst_t >::from_f32;
+
+    for (int i = 0; i < n; i++) {
+        z[i] = f32_to_dst(op(src0_to_f32(x[i]), src1_to_f32(y[i])));
+    }
+}
+
+template <float (*op)(float, float), typename src0_t, typename src1_t, typename dst_t>
+static inline void vec_binary_op_non_contiguous(const int64_t n, const int64_t ne10, const int64_t nb10, dst_t * z, const src0_t * x, const src1_t * y) {
+    constexpr auto src0_to_f32 = type_conversion_table<src0_t>::to_f32;
+    constexpr auto src1_to_f32 = type_conversion_table<src1_t>::to_f32;
+    constexpr auto f32_to_dst  = type_conversion_table<dst_t >::from_f32;
+
+    for (int i = 0; i < n; i++) {
+        int i10 = i % ne10;
+        const src1_t * y_ptr = (const src1_t *)((const char *)y + i10*nb10);
+        z[i] = f32_to_dst(op(src0_to_f32(x[i]), src1_to_f32(*y_ptr)));
+    }
+}
+
+template <float (*op)(float, float), typename src0_t, typename src1_t, typename dst_t>
+static void apply_binary_op(const ggml_compute_params * params, ggml_tensor * dst) {
+    const ggml_tensor * src0 = dst->src[0];
+    const ggml_tensor * src1 = dst->src[1];
+
+    GGML_ASSERT(ggml_can_repeat(src1, src0) && ggml_are_same_shape(src0, dst));
+
+    GGML_TENSOR_BINARY_OP_LOCALS
+
+    GGML_ASSERT( nb0 == sizeof(dst_t));
+    GGML_ASSERT(nb00 == sizeof(src0_t));
+
+    const auto [ir0, ir1] = get_thread_range(params, src0);
+    const bool is_src1_contiguous = (nb10 == sizeof(src1_t));
+
+    if (!is_src1_contiguous) { // broadcast not implemented yet for non-contiguous
+        GGML_ASSERT(ggml_are_same_shape(src0, src1));
+    }
+
+#ifdef GGML_USE_ACCELERATE
+    vDSP_fn_t vDSP_op = nullptr;
+    // TODO - avoid the f32-only check using type 'trait' lookup tables and row-based src-to-float conversion functions
+    if (src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) {
+        if (op == op_add) {
+            vDSP_op = vDSP_vadd;
+        } else if (op == op_sub) {
+            vDSP_op = vDSP_vsub;
+        } else if (op == op_mul) {
+            vDSP_op = vDSP_vmul;
+        } else if (op == op_div) {
+            vDSP_op = vDSP_vdiv;
+        }
+    }
+#endif
+
+    for (int64_t ir = ir0; ir < ir1; ++ir) {
+        const int64_t i03 = ir/(ne02*ne01);
+        const int64_t i02 = (ir - i03*ne02*ne01)/ne01;
+        const int64_t i01 = (ir - i03*ne02*ne01 - i02*ne01);
+
+        const int64_t i13 = i03 % ne13;
+        const int64_t i12 = i02 % ne12;
+        const int64_t i11 = i01 % ne11;
+
+        dst_t        * dst_ptr  = (dst_t  *)       ((char *)       dst->data  + i03*nb3  + i02*nb2  + i01*nb1 );
+        const src0_t * src0_ptr = (const src0_t *) ((const char *) src0->data + i03*nb03 + i02*nb02 + i01*nb01);
+        const src1_t * src1_ptr = (const src1_t *) ((const char *) src1->data + i13*nb13 + i12*nb12 + i11*nb11);
+
+        if (is_src1_contiguous) {
+            // src1 is broadcastable across src0 and dst in i1, i2, i3
+            const int64_t nr0 = ne00 / ne10;
+
+            for (int64_t r = 0; r < nr0; ++r) {
+#ifdef GGML_USE_ACCELERATE
+                if constexpr (std::is_same_v<src0_t, float> && std::is_same_v<src1_t, float> && std::is_same_v<dst_t, float>) {
+                    if (vDSP_op != nullptr) {
+                        vDSP_op(src1_ptr, 1, src0_ptr + r*ne10, 1, dst_ptr + r*ne10, 1, ne10);
+                        continue;
+                    }
+                }
+#endif
+                vec_binary_op_contiguous<op>(ne10, dst_ptr + r*ne10, src0_ptr + r*ne10, src1_ptr);
+            }
+        } else {
+            vec_binary_op_non_contiguous<op>(ne0, ne10, nb10, dst_ptr, src0_ptr, src1_ptr);
+        }
+    }
+}
+
+// TODO: Use the 'traits' lookup table (for type conversion fns), instead of a mass of 'if' conditions with long templates
+template <float (*op)(float, float)>
+static void binary_op(const ggml_compute_params * params, ggml_tensor * dst) {
+    const ggml_tensor * src0 = dst->src[0];
+    const ggml_tensor * src1 = dst->src[1];
+
+    /*  */ if (src0->type == GGML_TYPE_F32  && src1->type == GGML_TYPE_F32  && dst->type == GGML_TYPE_F32) { // all f32
+        apply_binary_op<op, float, float, float>(params, dst);
+    } else if (src0->type == GGML_TYPE_F16  && src1->type == GGML_TYPE_F16  && dst->type == GGML_TYPE_F16) { // all f16
+        apply_binary_op<op, ggml_fp16_t, ggml_fp16_t, ggml_fp16_t>(params, dst);
+    } else if (src0->type == GGML_TYPE_BF16 && src1->type == GGML_TYPE_BF16 && dst->type == GGML_TYPE_BF16) { // all bf16
+        apply_binary_op<op, ggml_bf16_t, ggml_bf16_t, ggml_bf16_t>(params, dst);
+    } else if (src0->type == GGML_TYPE_BF16 && src1->type == GGML_TYPE_F32  && dst->type == GGML_TYPE_BF16) {
+        apply_binary_op<op, ggml_bf16_t, float, ggml_bf16_t>(params, dst);
+    } else if (src0->type == GGML_TYPE_BF16 && src1->type == GGML_TYPE_F32  && dst->type == GGML_TYPE_F32) {
+        apply_binary_op<op, ggml_bf16_t, float, float>(params, dst);
+    } else if (src0->type == GGML_TYPE_F16  && src1->type == GGML_TYPE_F32  && dst->type == GGML_TYPE_F16) {
+        apply_binary_op<op, ggml_fp16_t, float, ggml_fp16_t>(params, dst);
+    } else if (src0->type == GGML_TYPE_F16  && src1->type == GGML_TYPE_F32  && dst->type == GGML_TYPE_F32) {
+        apply_binary_op<op, ggml_fp16_t, float, float>(params, dst);
+    } else {
+        GGML_ABORT("%s: unsupported types: dst: %s, src0: %s, src1: %s\n", __func__,
+            ggml_type_name(dst->type), ggml_type_name(src0->type), ggml_type_name(src1->type));
+    }
+}
+
+void ggml_compute_forward_add_non_quantized(const ggml_compute_params * params, ggml_tensor * dst) {
+    binary_op<op_add>(params, dst);
+}
+
+void ggml_compute_forward_sub(const ggml_compute_params * params, ggml_tensor * dst) {
+    binary_op<op_sub>(params, dst);
+}
+
+void ggml_compute_forward_mul(const ggml_compute_params * params, ggml_tensor * dst) {
+    binary_op<op_mul>(params, dst);
+}
+
+void ggml_compute_forward_div(const ggml_compute_params * params, ggml_tensor * dst) {
+    binary_op<op_div>(params, dst);
+}

ggml/src/ggml-cpu/binary-ops.h

@@ -0,0 +1,16 @@
+#pragma once
+
+#include "common.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void ggml_compute_forward_add_non_quantized(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_sub(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_mul(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_div(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+
+#ifdef __cplusplus
+}
+#endif

ggml/src/ggml-cpu/common.h

@@ -0,0 +1,73 @@
+#pragma once
+
+#include "ggml.h"
+#include "traits.h"
+#include "ggml-cpu-impl.h"
+#include "ggml-impl.h"
+#include "simd-mappings.h"
+
+#ifdef __cplusplus
+
+#include <utility>
+
+// convenience functions/macros for use in template calls
+// note: these won't be required after the 'traits' lookup table is used.
+static inline ggml_fp16_t f32_to_f16(float x) {
+    return GGML_CPU_FP32_TO_FP16(x);
+}
+
+static inline float f16_to_f32(ggml_fp16_t x) {
+    return GGML_CPU_FP16_TO_FP32(x);
+}
+
+static inline ggml_bf16_t f32_to_bf16(float x) {
+    return GGML_FP32_TO_BF16(x);
+}
+
+static inline float bf16_to_f32(ggml_bf16_t x) {
+    return GGML_BF16_TO_FP32(x);
+}
+
+static inline float f32_to_f32(float x) {
+    return x;
+}
+
+// TODO - merge this into the traits table, after using row-based conversions
+template <class T>
+struct type_conversion_table;
+
+template <>
+struct type_conversion_table<ggml_fp16_t> {
+    static constexpr float (*to_f32)(ggml_fp16_t) = f16_to_f32;
+    static constexpr ggml_fp16_t (*from_f32)(float) = f32_to_f16;
+};
+
+template <>
+struct type_conversion_table<float> {
+    static constexpr float (*to_f32)(float) = f32_to_f32;
+    static constexpr float (*from_f32)(float) = f32_to_f32;
+};
+
+template <>
+struct type_conversion_table<ggml_bf16_t> {
+    static constexpr float (*to_f32)(ggml_bf16_t) = bf16_to_f32;
+    static constexpr ggml_bf16_t (*from_f32)(float) = f32_to_bf16;
+};
+
+static std::pair<int64_t, int64_t> get_thread_range(const struct ggml_compute_params * params, const struct ggml_tensor * src0) {
+    const int64_t ith = params->ith;
+    const int64_t nth = params->nth;
+
+    const int64_t nr  = ggml_nrows(src0);
+
+    // rows per thread
+    const int64_t dr = (nr + nth - 1)/nth;
+
+    // row range for this thread
+    const int64_t ir0 = dr*ith;
+    const int64_t ir1 = MIN(ir0 + dr, nr);
+
+    return {ir0, ir1};
+}
+
+#endif

ggml/src/ggml-cpu/ggml-cpu-impl.h

@@ -0,0 +1,517 @@
+#pragma once
+
+// GGML CPU internal header
+
+#include "ggml.h"
+#include "ggml-impl.h"
+
+#include <stdlib.h> // load `stdlib.h` before other headers to work around MinGW bug: https://sourceforge.net/p/mingw-w64/bugs/192/
+//#include <stddef.h>
+#include <stdbool.h>
+#include <string.h> // memcpy
+#include <math.h>   // fabsf
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct ggml_compute_params {
+    // ith = thread index, nth = number of threads
+    int ith, nth;
+
+    // work buffer for all threads
+    size_t wsize;
+    void * wdata;
+
+    struct ggml_threadpool * threadpool;
+};
+
+
+#if defined(_MSC_VER)
+
+#define m512bh(p) p
+#define m512i(p) p
+
+#else
+
+#define m512bh(p) (__m512bh)(p)
+#define m512i(p) (__m512i)(p)
+
+#endif
+
+// __FMA__ and __F16C__ are not defined in MSVC, however they are implied with AVX2/AVX512
+#if defined(_MSC_VER) && (defined(__AVX2__) || defined(__AVX512F__))
+#ifndef __FMA__
+#define __FMA__
+#endif
+#ifndef __F16C__
+#define __F16C__
+#endif
+#endif
+
+// __SSE3__ and __SSSE3__ are not defined in MSVC, but SSE3/SSSE3 are present when AVX/AVX2/AVX512 are available
+#if defined(_MSC_VER) && (defined(__AVX__) || defined(__AVX2__) || defined(__AVX512F__))
+#ifndef __SSE3__
+#define __SSE3__
+#endif
+#ifndef __SSSE3__
+#define __SSSE3__
+#endif
+#endif
+
+#if defined(__s390x__) && defined(__VEC__)
+#ifndef __VXE__
+#define __VXE__
+#endif  // __VXE__
+#ifndef __VXE2__
+#define __VXE2__
+#endif  // __VXE2__
+#endif  // __s390x__ && __VEC__
+
+#if defined(__s390x__) && defined(GGML_NNPA)
+#ifndef __NNPA__
+#define __NNPA__
+#endif  // __NNPA__
+#endif  // __s390x__ && GGML_NNPA
+
+#if defined(__ARM_FEATURE_SVE)
+#include <sys/prctl.h>
+#endif
+
+#if defined(__ARM_NEON)
+
+// ref: https://github.com/ggml-org/llama.cpp/pull/5404
+#ifdef _MSC_VER
+#define ggml_vld1q_u32(w,x,y,z) { ((w) + ((uint64_t)(x) << 32)), ((y) + ((uint64_t)(z) << 32)) }
+#else
+#define ggml_vld1q_u32(w,x,y,z) { (w), (x), (y), (z) }
+#endif // _MSC_VER
+
+#if !defined(__aarch64__)
+
+// 32-bit ARM compatibility
+
+// vaddlvq_s16
+// vpaddq_s16
+// vpaddq_s32
+// vaddvq_s32
+// vaddvq_f32
+// vmaxvq_f32
+// vcvtnq_s32_f32
+// vzip1_u8
+// vzip2_u8
+
+inline static int32_t vaddlvq_s16(int16x8_t v) {
+    int32x4_t v0 = vreinterpretq_s32_s64(vpaddlq_s32(vpaddlq_s16(v)));
+    return vgetq_lane_s32(v0, 0) + vgetq_lane_s32(v0, 2);
+}
+
+inline static int16x8_t vpaddq_s16(int16x8_t a, int16x8_t b) {
+    int16x4_t a0 = vpadd_s16(vget_low_s16(a), vget_high_s16(a));
+    int16x4_t b0 = vpadd_s16(vget_low_s16(b), vget_high_s16(b));
+    return vcombine_s16(a0, b0);
+}
+
+inline static int32x4_t vpaddq_s32(int32x4_t a, int32x4_t b) {
+    int32x2_t a0 = vpadd_s32(vget_low_s32(a), vget_high_s32(a));
+    int32x2_t b0 = vpadd_s32(vget_low_s32(b), vget_high_s32(b));
+    return vcombine_s32(a0, b0);
+}
+
+inline static int32_t vaddvq_s32(int32x4_t v) {
+    return vgetq_lane_s32(v, 0) + vgetq_lane_s32(v, 1) + vgetq_lane_s32(v, 2) + vgetq_lane_s32(v, 3);
+}
+
+inline static float vaddvq_f32(float32x4_t v) {
+    return vgetq_lane_f32(v, 0) + vgetq_lane_f32(v, 1) + vgetq_lane_f32(v, 2) + vgetq_lane_f32(v, 3);
+}
+
+inline static float vmaxvq_f32(float32x4_t v) {
+    return
+        MAX(MAX(vgetq_lane_f32(v, 0), vgetq_lane_f32(v, 1)),
+            MAX(vgetq_lane_f32(v, 2), vgetq_lane_f32(v, 3)));
+}
+
+inline static int32x4_t vcvtnq_s32_f32(float32x4_t v) {
+    int32x4_t res;
+
+    res[0] = roundf(vgetq_lane_f32(v, 0));
+    res[1] = roundf(vgetq_lane_f32(v, 1));
+    res[2] = roundf(vgetq_lane_f32(v, 2));
+    res[3] = roundf(vgetq_lane_f32(v, 3));
+
+    return res;
+}
+
+inline static uint8x8_t vzip1_u8(uint8x8_t a, uint8x8_t b) {
+    uint8x8_t res;
+
+    res[0] = a[0]; res[1] = b[0];
+    res[2] = a[1]; res[3] = b[1];
+    res[4] = a[2]; res[5] = b[2];
+    res[6] = a[3]; res[7] = b[3];
+
+    return res;
+}
+
+inline static uint8x8_t vzip2_u8(uint8x8_t a, uint8x8_t b) {
+    uint8x8_t res;
+
+    res[0] = a[4]; res[1] = b[4];
+    res[2] = a[5]; res[3] = b[5];
+    res[4] = a[6]; res[5] = b[6];
+    res[6] = a[7]; res[7] = b[7];
+
+    return res;
+}
+
+// vld1q_s16_x2
+// vld1q_u8_x2
+// vld1q_u8_x4
+// vld1q_s8_x2
+// vld1q_s8_x4
+// TODO: double-check these work correctly
+
+typedef struct ggml_int16x8x2_t {
+    int16x8_t val[2];
+} ggml_int16x8x2_t;
+
+inline static ggml_int16x8x2_t ggml_vld1q_s16_x2(const int16_t * ptr) {
+    ggml_int16x8x2_t res;
+
+    res.val[0] = vld1q_s16(ptr + 0);
+    res.val[1] = vld1q_s16(ptr + 8);
+
+    return res;
+}
+
+typedef struct ggml_uint8x16x2_t {
+    uint8x16_t val[2];
+} ggml_uint8x16x2_t;
+
+inline static ggml_uint8x16x2_t ggml_vld1q_u8_x2(const uint8_t * ptr) {
+    ggml_uint8x16x2_t res;
+
+    res.val[0] = vld1q_u8(ptr + 0);
+    res.val[1] = vld1q_u8(ptr + 16);
+
+    return res;
+}
+
+typedef struct ggml_uint8x16x4_t {
+    uint8x16_t val[4];
+} ggml_uint8x16x4_t;
+
+inline static ggml_uint8x16x4_t ggml_vld1q_u8_x4(const uint8_t * ptr) {
+    ggml_uint8x16x4_t res;
+
+    res.val[0] = vld1q_u8(ptr + 0);
+    res.val[1] = vld1q_u8(ptr + 16);
+    res.val[2] = vld1q_u8(ptr + 32);
+    res.val[3] = vld1q_u8(ptr + 48);
+
+    return res;
+}
+
+typedef struct ggml_int8x16x2_t {
+    int8x16_t val[2];
+} ggml_int8x16x2_t;
+
+inline static ggml_int8x16x2_t ggml_vld1q_s8_x2(const int8_t * ptr) {
+    ggml_int8x16x2_t res;
+
+    res.val[0] = vld1q_s8(ptr + 0);
+    res.val[1] = vld1q_s8(ptr + 16);
+
+    return res;
+}
+
+typedef struct ggml_int8x16x4_t {
+    int8x16_t val[4];
+} ggml_int8x16x4_t;
+
+inline static ggml_int8x16x4_t ggml_vld1q_s8_x4(const int8_t * ptr) {
+    ggml_int8x16x4_t res;
+
+    res.val[0] = vld1q_s8(ptr + 0);
+    res.val[1] = vld1q_s8(ptr + 16);
+    res.val[2] = vld1q_s8(ptr + 32);
+    res.val[3] = vld1q_s8(ptr + 48);
+
+    return res;
+}
+
+// NOTE: not tested
+inline static int8x16_t ggml_vqtbl1q_s8(int8x16_t a, uint8x16_t b) {
+    int8x16_t res;
+
+    res[ 0] = a[b[ 0]];
+    res[ 1] = a[b[ 1]];
+    res[ 2] = a[b[ 2]];
+    res[ 3] = a[b[ 3]];
+    res[ 4] = a[b[ 4]];
+    res[ 5] = a[b[ 5]];
+    res[ 6] = a[b[ 6]];
+    res[ 7] = a[b[ 7]];
+    res[ 8] = a[b[ 8]];
+    res[ 9] = a[b[ 9]];
+    res[10] = a[b[10]];
+    res[11] = a[b[11]];
+    res[12] = a[b[12]];
+    res[13] = a[b[13]];
+    res[14] = a[b[14]];
+    res[15] = a[b[15]];
+
+    return res;
+}
+
+// NOTE: not tested
+inline static uint8x16_t ggml_vqtbl1q_u8(uint8x16_t a, uint8x16_t b) {
+    uint8x16_t res;
+
+    res[ 0] = a[b[ 0]];
+    res[ 1] = a[b[ 1]];
+    res[ 2] = a[b[ 2]];
+    res[ 3] = a[b[ 3]];
+    res[ 4] = a[b[ 4]];
+    res[ 5] = a[b[ 5]];
+    res[ 6] = a[b[ 6]];
+    res[ 7] = a[b[ 7]];
+    res[ 8] = a[b[ 8]];
+    res[ 9] = a[b[ 9]];
+    res[10] = a[b[10]];
+    res[11] = a[b[11]];
+    res[12] = a[b[12]];
+    res[13] = a[b[13]];
+    res[14] = a[b[14]];
+    res[15] = a[b[15]];
+
+    return res;
+}
+
+#else
+
+#define ggml_int16x8x2_t  int16x8x2_t
+#define ggml_uint8x16x2_t uint8x16x2_t
+#define ggml_uint8x16x4_t uint8x16x4_t
+#define ggml_int8x16x2_t  int8x16x2_t
+#define ggml_int8x16x4_t  int8x16x4_t
+
+#define ggml_vld1q_s16_x2 vld1q_s16_x2
+#define ggml_vld1q_u8_x2  vld1q_u8_x2
+#define ggml_vld1q_u8_x4  vld1q_u8_x4
+#define ggml_vld1q_s8_x2  vld1q_s8_x2
+#define ggml_vld1q_s8_x4  vld1q_s8_x4
+#define ggml_vqtbl1q_s8   vqtbl1q_s8
+#define ggml_vqtbl1q_u8   vqtbl1q_u8
+
+#endif // !defined(__aarch64__)
+
+#if !defined(__ARM_FEATURE_DOTPROD)
+
+inline static int32x4_t ggml_vdotq_s32(int32x4_t acc, int8x16_t a, int8x16_t b) {
+    const int16x8_t p0 = vmull_s8(vget_low_s8 (a), vget_low_s8 (b));
+    const int16x8_t p1 = vmull_s8(vget_high_s8(a), vget_high_s8(b));
+
+    return vaddq_s32(acc, vaddq_s32(vpaddlq_s16(p0), vpaddlq_s16(p1)));
+}
+
+#else
+
+#define ggml_vdotq_s32(a, b, c) vdotq_s32(a, b, c)
+
+#endif // !defined(__ARM_FEATURE_DOTPROD)
+
+#endif // defined(__ARM_NEON)
+
+#ifdef __wasm_simd128__
+#include <wasm_simd128.h>
+#endif
+
+#ifdef __POWER9_VECTOR__
+#include <altivec.h>
+#endif
+
+#if defined(_MSC_VER) || defined(__MINGW32__)
+#include <intrin.h>
+#elif defined(__AVX__) || defined(__AVX2__) || defined(__AVX512F__) || defined(__SSSE3__) || defined(__SSE3__) || defined(__SSE__)
+#include <immintrin.h>
+#endif
+
+#ifdef __riscv_v_intrinsic
+#include <riscv_vector.h>
+#endif
+
+#if defined(__loongarch64)
+#if defined(__loongarch_asx)
+#include <lasxintrin.h>
+#endif
+#if defined(__loongarch_sx)
+#include <lsxintrin.h>
+#endif
+#endif
+
+#if defined(__VXE__) || defined(__VXE2__)
+#include <vecintrin.h>
+
+#define vec_neg(a)    (-(a))                // Vector Negate
+#define vec_add(a, b) ((a) + (b))           // Vector Add
+#define vec_sub(a, b) ((a) - (b))           // Vector Subtract
+#define vec_mul(a, b) ((a) * (b))           // Vector Multiply
+#define vec_div(a, b) ((a) / (b))           // Vector Divide
+#define vec_sl(a, b)  ((a) << (b))          // Vector Shift Left
+#define vec_sra(a, b) ((a) >> (b))          // Vector Shift Right
+#define vec_sr(a, b)  ((a) >> (b))          // Vector Shift Right Algebraic
+#define vec_slo(a, b) vec_slb(a, (b) << 64) // Vector Shift Left by Octet
+#define vec_sro(a, b) vec_srb(a, (b) << 64) // Vector Shift Right by Octet
+
+#ifndef vec_and
+#define vec_and(a, b) ((a) & (b)) // Vector AND
+#endif
+
+#ifndef vec_or
+#define vec_or(a, b)  ((a) | (b)) // Vector OR
+#endif
+
+#ifndef vec_xor
+#define vec_xor(a, b) ((a) ^ (b)) // Vector XOR
+#endif
+
+typedef signed   char char8x16_t  __attribute__((vector_size(16)));
+typedef unsigned char uchar8x16_t __attribute__((vector_size(16)));
+
+typedef int8_t  int8x16_t __attribute__((vector_size(16)));
+typedef int16_t int16x8_t __attribute__((vector_size(16)));
+typedef int32_t int32x4_t __attribute__((vector_size(16)));
+
+typedef uint8_t  uint8x16_t __attribute__((vector_size(16)));
+typedef uint16_t uint16x8_t __attribute__((vector_size(16)));
+typedef uint32_t uint32x4_t __attribute__((vector_size(16)));
+
+typedef float  float32x4_t  __attribute__((vector_size(16)));
+typedef double double64x2_t __attribute__((vector_size(16)));
+
+typedef signed   long long long64x2_t  __attribute__((vector_size(16)));
+typedef unsigned long long ulong64x2_t __attribute__((vector_size(16)));
+
+typedef struct ggml_uint8x16x2_t {
+    uint8x16_t val[2];
+} ggml_uint8x16x2_t;
+
+inline static ggml_uint8x16x2_t ggml_vec_xl_u8x2(const uint8_t * ptr) {
+    ggml_uint8x16x2_t res;
+
+    res.val[0] = vec_xl( 0, ptr);
+    res.val[1] = vec_xl(16, ptr);
+
+    return res;
+}
+
+typedef struct ggml_uint8x16x4_t {
+    uint8x16_t val[4];
+} ggml_uint8x16x4_t;
+
+inline static ggml_uint8x16x4_t ggml_vec_xl_u8x4(const uint8_t * ptr) {
+    ggml_uint8x16x4_t res;
+
+    res.val[0] = vec_xl( 0, ptr);
+    res.val[1] = vec_xl(16, ptr);
+    res.val[2] = vec_xl(32, ptr);
+    res.val[3] = vec_xl(48, ptr);
+
+    return res;
+}
+
+typedef struct ggml_int8x16x4_t {
+    int8x16_t val[4];
+} ggml_int8x16x4_t;
+
+inline static ggml_int8x16x4_t ggml_vec_xl_s8x4(const int8_t * ptr) {
+    ggml_int8x16x4_t res;
+
+    res.val[0] = vec_xl( 0, ptr);
+    res.val[1] = vec_xl(16, ptr);
+    res.val[2] = vec_xl(32, ptr);
+    res.val[3] = vec_xl(48, ptr);
+
+    return res;
+}
+
+typedef struct ggml_int16x8x2_t {
+    int16x8_t val[2];
+} ggml_int16x8x2_t;
+
+inline static ggml_int16x8x2_t ggml_vec_xl_s16x2(const int16_t * ptr) {
+    ggml_int16x8x2_t res;
+
+    res.val[0] = vec_xl( 0, ptr);
+    res.val[1] = vec_xl(16, ptr);
+
+    return res;
+}
+
+/*
+    ! WARNING: Very slow. Use vec_perm if possible. Refer to iq4_xs
+    !          or iq4_nl for example implementation.
+*/
+inline static int8x16_t ggml_vec_tbl(int8x16_t a, uint8x16_t b) {
+    int8x16_t res;
+
+    res[ 0] = a[b[ 0]];
+    res[ 1] = a[b[ 1]];
+    res[ 2] = a[b[ 2]];
+    res[ 3] = a[b[ 3]];
+    res[ 4] = a[b[ 4]];
+    res[ 5] = a[b[ 5]];
+    res[ 6] = a[b[ 6]];
+    res[ 7] = a[b[ 7]];
+    res[ 8] = a[b[ 8]];
+    res[ 9] = a[b[ 9]];
+    res[10] = a[b[10]];
+    res[11] = a[b[11]];
+    res[12] = a[b[12]];
+    res[13] = a[b[13]];
+    res[14] = a[b[14]];
+    res[15] = a[b[15]];
+
+    return res;
+}
+
+inline static int16x8_t vec_padd_s16(int16x8_t a, int16x8_t b) {
+    const uchar8x16_t v_maske = {  0,  1,  4,  5,  8,  9, 12, 13,
+                                  16, 17, 20, 21, 24, 25, 28, 29 };
+
+    const int16x8_t v_abo = vec_pack((int32x4_t)a, (int32x4_t)b);
+    const int16x8_t v_abe = vec_perm(a, b, v_maske);
+    return v_abo + v_abe;
+}
+
+inline static int32x4_t ggml_vec_dot(int32x4_t acc, int8x16_t a, int8x16_t b) {
+    const int16x8_t p = vec_mule(a, b) + vec_mulo(a, b);
+    return acc + (vec_unpackh(p) + vec_unpackl(p));
+}
+
+#endif
+
+#if defined(__loongarch_asx)
+/* float type data load instructions */
+static __m128 __lsx_vreplfr2vr_s(const float val) {
+    v4f32 res = {val, val, val, val};
+    return (__m128)res;
+}
+
+static __m256 __lasx_xvreplfr2vr_s(const float val) {
+    v8f32 res = {val, val, val, val, val, val, val, val};
+    return (__m256)res;
+}
+#endif
+
+// TODO: move to ggml-threading
+void ggml_barrier(struct ggml_threadpool * tp);
+
+void ggml_threadpool_chunk_set(struct ggml_threadpool * tp, int value);
+int  ggml_threadpool_chunk_add(struct ggml_threadpool * tp, int value);
+
+#ifdef __cplusplus
+}
+#endif

ggml/src/ggml-cpu/ggml-cpu.cpp

@@ -0,0 +1,675 @@
+#include "ggml-backend.h"
+#include "ggml-backend-impl.h"
+#include "ggml-cpu.h"
+#include "repack.h"
+#include "traits.h"
+#include "ggml-impl.h"
+#include "amx/amx.h"
+
+#include <cctype>
+#include <string>
+#include <vector>
+
+#ifdef GGML_USE_CPU_HBM
+#    include "hbm.h"
+#endif
+
+#ifdef GGML_USE_CPU_KLEIDIAI
+#    include "kleidiai/kleidiai.h"
+#endif
+
+#if defined(_WIN32)
+#    define WIN32_LEAN_AND_MEAN
+#    ifndef NOMINMAX
+#        define NOMINMAX
+#    endif
+#    include <windows.h>
+#else
+#    include <unistd.h>
+#endif
+
+#if defined(__APPLE__)
+#    include <sys/sysctl.h>
+#    include <sys/types.h>
+#endif
+
+// ggml-backend interface
+
+std::vector<ggml_backend_buffer_type_t>& ggml_backend_cpu_get_extra_buffers_type() {
+    static std::vector<ggml_backend_buffer_type_t> bufts = []() {
+        std::vector<ggml_backend_buffer_type_t> bufts;
+
+#if defined(__AMX_INT8__) && defined(__AVX512VNNI__)
+        if (ggml_backend_amx_buffer_type()) {
+            bufts.push_back(ggml_backend_amx_buffer_type());
+        }
+#endif
+
+#ifdef GGML_USE_CPU_KLEIDIAI
+        if (ggml_backend_cpu_kleidiai_buffer_type()) {
+            bufts.push_back(ggml_backend_cpu_kleidiai_buffer_type());
+        }
+#endif
+
+#ifdef GGML_USE_CPU_REPACK
+        if (ggml_backend_cpu_repack_buffer_type()) {
+            bufts.push_back(ggml_backend_cpu_repack_buffer_type());
+        }
+#endif
+
+        bufts.push_back(NULL);
+
+        return bufts;
+    }();
+
+    return bufts;
+}
+
+static ggml_backend_buffer_type_t * ggml_backend_cpu_device_get_extra_buffers_type(ggml_backend_dev_t device) {
+    return ggml_backend_cpu_get_extra_buffers_type().data();
+
+    GGML_UNUSED(device);
+}
+
+static bool ggml_backend_cpu_is_extra_buffer_type(ggml_backend_buffer_type_t buft) {
+    for (auto * extra : ggml_backend_cpu_get_extra_buffers_type()) {
+        if (extra && extra == buft) {
+            return true;
+        }
+    }
+    return false;
+}
+
+// CPU backend - backend (stream)
+
+struct ggml_backend_cpu_context {
+    int                 n_threads;
+    ggml_threadpool_t   threadpool;
+
+    uint8_t *           work_data;
+    size_t              work_size;
+
+    ggml_abort_callback abort_callback;
+    void *              abort_callback_data;
+};
+
+static const char * ggml_backend_cpu_get_name(ggml_backend_t backend) {
+    return "CPU";
+
+    GGML_UNUSED(backend);
+}
+
+static void ggml_backend_cpu_free(ggml_backend_t backend) {
+    struct ggml_backend_cpu_context * cpu_ctx = (struct ggml_backend_cpu_context *)backend->context;
+    delete[] cpu_ctx->work_data;
+    delete cpu_ctx;
+    delete backend;
+}
+
+struct ggml_backend_plan_cpu {
+    struct ggml_cplan cplan;
+    struct ggml_cgraph cgraph;
+};
+
+static ggml_backend_graph_plan_t ggml_backend_cpu_graph_plan_create(ggml_backend_t backend, const struct ggml_cgraph * cgraph) {
+    struct ggml_backend_cpu_context * cpu_ctx = (struct ggml_backend_cpu_context *)backend->context;
+
+    struct ggml_backend_plan_cpu * cpu_plan = new ggml_backend_plan_cpu;
+
+    cpu_plan->cplan = ggml_graph_plan(cgraph, cpu_ctx->n_threads, cpu_ctx->threadpool);
+    cpu_plan->cgraph = *cgraph; // FIXME: deep copy
+
+    if (cpu_plan->cplan.work_size > 0) {
+        cpu_plan->cplan.work_data = new uint8_t[cpu_plan->cplan.work_size];
+        if (cpu_plan->cplan.work_data == NULL) {
+            delete cpu_plan;
+            return NULL;
+        }
+    }
+
+    cpu_plan->cplan.abort_callback      = cpu_ctx->abort_callback;
+    cpu_plan->cplan.abort_callback_data = cpu_ctx->abort_callback_data;
+
+    return cpu_plan;
+}
+
+static void ggml_backend_cpu_graph_plan_free(ggml_backend_t backend, ggml_backend_graph_plan_t plan) {
+    struct ggml_backend_plan_cpu * cpu_plan = (struct ggml_backend_plan_cpu *)plan;
+
+    delete[] cpu_plan->cplan.work_data;
+    delete cpu_plan;
+
+    GGML_UNUSED(backend);
+}
+
+static enum ggml_status ggml_backend_cpu_graph_plan_compute(ggml_backend_t backend, ggml_backend_graph_plan_t plan) {
+    struct ggml_backend_plan_cpu * cpu_plan = (struct ggml_backend_plan_cpu *)plan;
+
+    return ggml_graph_compute(&cpu_plan->cgraph, &cpu_plan->cplan);
+
+    GGML_UNUSED(backend);
+}
+
+static enum ggml_status ggml_backend_cpu_graph_compute(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
+    struct ggml_backend_cpu_context * cpu_ctx = (struct ggml_backend_cpu_context *)backend->context;
+
+    struct ggml_cplan cplan = ggml_graph_plan(cgraph, cpu_ctx->n_threads, cpu_ctx->threadpool);
+
+    if (cpu_ctx->work_size < cplan.work_size) {
+        delete[] cpu_ctx->work_data;
+        cpu_ctx->work_data = new uint8_t[cplan.work_size];
+        if (cpu_ctx->work_data == NULL) {
+            cpu_ctx->work_size = 0;
+            return GGML_STATUS_ALLOC_FAILED;
+        }
+        cpu_ctx->work_size = cplan.work_size;
+    }
+    cplan.work_data = (uint8_t *)cpu_ctx->work_data;
+
+    cplan.abort_callback      = cpu_ctx->abort_callback;
+    cplan.abort_callback_data = cpu_ctx->abort_callback_data;
+
+    return ggml_graph_compute(cgraph, &cplan);
+}
+
+static const struct ggml_backend_i ggml_backend_cpu_i = {
+    /* .get_name                = */ ggml_backend_cpu_get_name,
+    /* .free                    = */ ggml_backend_cpu_free,
+    /* .set_tensor_async        = */ NULL,
+    /* .get_tensor_async        = */ NULL,
+    /* .cpy_tensor_async        = */ NULL,
+    /* .synchronize             = */ NULL,
+    /* .graph_plan_create       = */ ggml_backend_cpu_graph_plan_create,
+    /* .graph_plan_free         = */ ggml_backend_cpu_graph_plan_free,
+    /* .graph_plan_update       = */ NULL,
+    /* .graph_plan_compute      = */ ggml_backend_cpu_graph_plan_compute,
+    /* .graph_compute           = */ ggml_backend_cpu_graph_compute,
+    /* .event_record            = */ NULL,
+    /* .event_wait              = */ NULL,
+};
+
+static ggml_guid_t ggml_backend_cpu_guid(void) {
+    static ggml_guid guid = { 0xaa, 0x67, 0xc7, 0x43, 0x96, 0xe6, 0xa3, 0x8a, 0xe3, 0xaf, 0xea, 0x92, 0x36, 0xbc, 0xfc, 0x89 };
+    return &guid;
+}
+
+ggml_backend_t ggml_backend_cpu_init(void) {
+    // initialize CPU backend now to avoid slowing the first graph computation
+    ggml_cpu_init();
+
+    struct ggml_backend_cpu_context * ctx = new ggml_backend_cpu_context;
+    if (ctx == NULL) {
+        return NULL;
+    }
+
+    ctx->n_threads           = GGML_DEFAULT_N_THREADS;
+    ctx->threadpool          = NULL;
+    ctx->work_data           = NULL;
+    ctx->work_size           = 0;
+    ctx->abort_callback      = NULL;
+    ctx->abort_callback_data = NULL;
+
+    ggml_backend_t cpu_backend = new ggml_backend {
+        /* .guid      = */ ggml_backend_cpu_guid(),
+        /* .interface = */ ggml_backend_cpu_i,
+        /* .device    = */ ggml_backend_reg_dev_get(ggml_backend_cpu_reg(), 0),
+        /* .context   = */ ctx,
+    };
+
+    if (cpu_backend == NULL) {
+        delete ctx;
+        return NULL;
+    }
+
+    return cpu_backend;
+}
+
+bool ggml_backend_is_cpu(ggml_backend_t backend) {
+    return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_cpu_guid());
+}
+
+void ggml_backend_cpu_set_n_threads(ggml_backend_t backend_cpu, int n_threads) {
+    GGML_ASSERT(ggml_backend_is_cpu(backend_cpu));
+
+    struct ggml_backend_cpu_context * ctx = (struct ggml_backend_cpu_context *)backend_cpu->context;
+    ctx->n_threads = n_threads;
+}
+
+void ggml_backend_cpu_set_threadpool(ggml_backend_t backend_cpu, ggml_threadpool_t threadpool) {
+    GGML_ASSERT(ggml_backend_is_cpu(backend_cpu));
+
+    struct ggml_backend_cpu_context * ctx = (struct ggml_backend_cpu_context *)backend_cpu->context;
+
+    if (ctx->threadpool && ctx->threadpool != threadpool) {
+        // already had a different threadpool, pause/suspend it before switching
+        ggml_threadpool_pause(ctx->threadpool);
+    }
+    ctx->threadpool = threadpool;
+}
+
+void ggml_backend_cpu_set_abort_callback(ggml_backend_t backend_cpu, ggml_abort_callback abort_callback, void * abort_callback_data) {
+    GGML_ASSERT(ggml_backend_is_cpu(backend_cpu));
+
+    struct ggml_backend_cpu_context * ctx = (struct ggml_backend_cpu_context *)backend_cpu->context;
+    ctx->abort_callback = abort_callback;
+    ctx->abort_callback_data = abort_callback_data;
+}
+
+// CPU backend - device
+
+struct ggml_backend_cpu_device_context {
+    std::string description = "CPU";
+
+    ggml_backend_cpu_device_context() {
+#ifdef __APPLE__
+        size_t len = 0;
+        if (!sysctlbyname("machdep.cpu.brand_string", NULL, &len, NULL, 0)) {
+            description.resize(len);
+            sysctlbyname("machdep.cpu.brand_string", &description[0], &len, NULL, 0); // NOLINT
+        }
+#elif defined(__linux__)
+        FILE * f = fopen("/proc/cpuinfo", "r");
+        if (f) {
+            char buf[1024];
+            while (fgets(buf, sizeof(buf), f)) {
+                if (strncmp(buf, "model name", 10) == 0) {
+                    char * p = strchr(buf, ':');
+                    if (p) {
+                        p++;
+                        while (std::isspace(*p)) {
+                            p++;
+                        }
+                        while (std::isspace(p[strlen(p) - 1])) {
+                            p[strlen(p) - 1] = '\0';
+                        }
+                        description = p;
+                        break;
+                    }
+                }
+            }
+            fclose(f);
+        }
+#elif defined(_WIN32)
+        HKEY hKey;
+        if (RegOpenKeyEx(HKEY_LOCAL_MACHINE,
+                        TEXT("HARDWARE\\DESCRIPTION\\System\\CentralProcessor\\0"),
+                        0,
+                        KEY_READ,
+                        &hKey) == ERROR_SUCCESS) {
+            DWORD cpu_brand_size = 0;
+            if (RegQueryValueExA(hKey,
+                                "ProcessorNameString",
+                                NULL,
+                                NULL,
+                                NULL,
+                                &cpu_brand_size) == ERROR_SUCCESS) {
+                description.resize(cpu_brand_size);
+                if (RegQueryValueExA(hKey,
+                                    "ProcessorNameString",
+                                    NULL,
+                                    NULL,
+                                    (LPBYTE)&description[0], // NOLINT
+                                    &cpu_brand_size) == ERROR_SUCCESS) {
+                    if (description.find('\0') != std::string::npos) {
+                        description.resize(description.find('\0'));
+                    }
+                }
+            }
+            RegCloseKey(hKey);
+        }
+#endif
+    }
+};
+
+static const char * ggml_backend_cpu_device_get_name(ggml_backend_dev_t dev) {
+    return "CPU";
+
+    GGML_UNUSED(dev);
+}
+
+static const char * ggml_backend_cpu_device_get_description(ggml_backend_dev_t dev) {
+    struct ggml_backend_cpu_device_context * ctx = (struct ggml_backend_cpu_device_context *)dev->context;
+
+    return ctx->description.c_str();
+}
+
+static void ggml_backend_cpu_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) {
+#ifdef _WIN32
+    MEMORYSTATUSEX status;
+    status.dwLength = sizeof(status);
+    GlobalMemoryStatusEx(&status);
+    *total = status.ullTotalPhys;
+    *free = status.ullAvailPhys;
+#else
+    long pages = sysconf(_SC_PHYS_PAGES);
+    long page_size = sysconf(_SC_PAGE_SIZE);
+    *total = pages * page_size;
+    *free = *total;
+#endif
+
+    GGML_UNUSED(dev);
+}
+
+static enum ggml_backend_dev_type ggml_backend_cpu_device_get_type(ggml_backend_dev_t dev) {
+    return GGML_BACKEND_DEVICE_TYPE_CPU;
+
+    GGML_UNUSED(dev);
+}
+
+static void ggml_backend_cpu_device_get_props(ggml_backend_dev_t dev, struct ggml_backend_dev_props * props) {
+    props->name        = ggml_backend_cpu_device_get_name(dev);
+    props->description = ggml_backend_cpu_device_get_description(dev);
+    props->type        = ggml_backend_cpu_device_get_type(dev);
+    ggml_backend_cpu_device_get_memory(dev, &props->memory_free, &props->memory_total);
+    props->caps = {
+        /* .async                 = */ false,
+        /* .host_buffer           = */ false,
+        /* .buffer_from_host_ptr  = */ true,
+        /* .events                = */ false,
+    };
+}
+
+static ggml_backend_t ggml_backend_cpu_device_init_backend(ggml_backend_dev_t dev, const char * params) {
+    return ggml_backend_cpu_init();
+
+    GGML_UNUSED(dev);
+    GGML_UNUSED(params);
+}
+
+static ggml_backend_buffer_type_t ggml_backend_cpu_device_get_buffer_type(ggml_backend_dev_t dev) {
+    return ggml_backend_cpu_buffer_type();
+
+    GGML_UNUSED(dev);
+}
+
+static ggml_backend_buffer_t ggml_backend_cpu_device_buffer_from_host_ptr(ggml_backend_dev_t dev, void * ptr, size_t size, size_t max_tensor_size) {
+    return ggml_backend_cpu_buffer_from_ptr(ptr, size);
+
+    GGML_UNUSED(dev);
+    GGML_UNUSED(max_tensor_size);
+}
+
+static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
+    const struct ggml_tensor * src0 = op->src[0];
+    const struct ggml_tensor * src1 = op->src[1];
+
+    if (op->op == GGML_OP_NONE || op->op == GGML_OP_RESHAPE || op->op == GGML_OP_VIEW || op->op == GGML_OP_PERMUTE || op->op == GGML_OP_TRANSPOSE) {
+        return true;
+    }
+
+    // extra_buffer_op?
+    for (auto extra : ggml_backend_cpu_get_extra_buffers_type()) {
+        if (extra) {
+            auto buf_extra = (ggml::cpu::extra_buffer_type*) extra->context;
+            if (buf_extra && buf_extra->supports_op(dev, op)) {
+                return true;
+            }
+        }
+    }
+
+    // the other case need host buffer.
+    for (int i = 0; i < GGML_MAX_SRC; i++) {
+        if (op->src[i] && op->src[i]->buffer && !ggml_backend_buft_is_host(op->src[i]->buffer->buft)) {
+            return false;
+        }
+    }
+
+    switch (op->op) {
+        case GGML_OP_CPY:
+        case GGML_OP_SET_ROWS:
+            return
+                op->type != GGML_TYPE_IQ3_XXS &&
+                op->type != GGML_TYPE_IQ3_S   &&
+                op->type != GGML_TYPE_IQ2_XXS &&
+                op->type != GGML_TYPE_IQ2_XS  &&
+                op->type != GGML_TYPE_IQ2_S   &&
+                op->type != GGML_TYPE_IQ1_S   &&
+                op->type != GGML_TYPE_IQ1_M; // missing type_traits.from_float
+        case GGML_OP_MUL_MAT:
+            return src1->type == GGML_TYPE_F32 || src1->type == ggml_get_type_traits_cpu(src0->type)->vec_dot_type;
+        case GGML_OP_SOFT_MAX_BACK: {
+            if (op->src[0]->type != GGML_TYPE_F32 || op->src[1]->type != GGML_TYPE_F32) {
+                return false;
+            }
+            float max_bias = 0.0f;
+
+            memcpy(&max_bias, (const float *) op->op_params + 1, sizeof(float));
+
+            return max_bias == 0.0f;
+        }
+        case GGML_OP_IM2COL_BACK:
+            return src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32;
+        case GGML_OP_GET_ROWS_BACK:
+            return src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16;
+        case GGML_OP_OUT_PROD:
+            return (src0->type == GGML_TYPE_F32 || (ggml_is_quantized(src0->type) && src0->ne[2] == src1->ne[2] && src0->ne[3] == src1->ne[3])) &&
+                src1->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32;
+        default:
+            return true;
+    }
+}
+
+static bool ggml_backend_cpu_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
+    return ggml_backend_buft_is_host(buft) || ggml_backend_cpu_is_extra_buffer_type(buft);
+    GGML_UNUSED(dev);
+}
+
+static const struct ggml_backend_device_i ggml_backend_cpu_device_i = {
+    /* .get_name             = */ ggml_backend_cpu_device_get_name,
+    /* .get_description      = */ ggml_backend_cpu_device_get_description,
+    /* .get_memory           = */ ggml_backend_cpu_device_get_memory,
+    /* .get_type             = */ ggml_backend_cpu_device_get_type,
+    /* .get_props            = */ ggml_backend_cpu_device_get_props,
+    /* .init_backend         = */ ggml_backend_cpu_device_init_backend,
+    /* .get_buffer_type      = */ ggml_backend_cpu_device_get_buffer_type,
+    /* .get_host_buffer_type = */ NULL,
+    /* .buffer_from_host_ptr = */ ggml_backend_cpu_device_buffer_from_host_ptr,
+    /* .supports_op          = */ ggml_backend_cpu_device_supports_op,
+    /* .supports_buft        = */ ggml_backend_cpu_device_supports_buft,
+    /* .offload_op           = */ NULL,
+    /* .event_new            = */ NULL,
+    /* .event_free           = */ NULL,
+    /* .event_synchronize    = */ NULL,
+};
+
+// CPU backend - backend (reg)
+
+static const char * ggml_backend_cpu_reg_get_name(ggml_backend_reg_t reg) {
+    return "CPU";
+
+    GGML_UNUSED(reg);
+}
+
+static size_t ggml_backend_cpu_reg_get_device_count(ggml_backend_reg_t reg) {
+    return 1;
+
+    GGML_UNUSED(reg);
+}
+
+static ggml_backend_dev_t ggml_backend_cpu_reg_get_device(ggml_backend_reg_t reg, size_t index) {
+    GGML_ASSERT(index == 0);
+
+    static ggml_backend_cpu_device_context ctx;
+    static ggml_backend_device ggml_backend_cpu_device = {
+        /* .iface   = */ ggml_backend_cpu_device_i,
+        /* .reg     = */ reg,
+        /* .context = */ &ctx,
+    };
+
+    return &ggml_backend_cpu_device;
+}
+
+// This is intended to replace the the ggml_cpu_has_* functions when loading the CPU backend dynamically,
+// and additionally to allow other backends to expose their own list of features that applications can query using the same API
+static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t reg) {
+    static std::vector<ggml_backend_feature> features = []() {
+        ggml_cpu_init();
+
+        std::vector<ggml_backend_feature> features;
+        if (ggml_cpu_has_sse3()) {
+            features.push_back({ "SSE3", "1" });
+        }
+        if (ggml_cpu_has_ssse3()) {
+            features.push_back({ "SSSE3", "1" });
+        }
+        if (ggml_cpu_has_avx()) {
+            features.push_back({ "AVX", "1" });
+        }
+        if (ggml_cpu_has_avx_vnni()) {
+            features.push_back({ "AVX_VNNI", "1" });
+        }
+        if (ggml_cpu_has_avx2()) {
+            features.push_back({ "AVX2", "1" });
+        }
+        if (ggml_cpu_has_f16c()) {
+            features.push_back({ "F16C", "1" });
+        }
+        if (ggml_cpu_has_fma()) {
+            features.push_back({ "FMA", "1" });
+        }
+        if (ggml_cpu_has_bmi2()) {
+            features.push_back({ "BMI2", "1" });
+        }
+        if (ggml_cpu_has_avx512()) {
+            features.push_back({ "AVX512", "1" });
+        }
+        if (ggml_cpu_has_avx512_vbmi()) {
+            features.push_back({ "AVX512_VBMI", "1" });
+        }
+        if (ggml_cpu_has_avx512_vnni()) {
+            features.push_back({ "AVX512_VNNI", "1" });
+        }
+        if (ggml_cpu_has_avx512_bf16()) {
+            features.push_back({ "AVX512_BF16", "1" });
+        }
+        if (ggml_cpu_has_amx_int8()) {
+            features.push_back({ "AMX_INT8", "1" });
+        }
+        if (ggml_cpu_has_neon()) {
+            features.push_back({ "NEON", "1" });
+        }
+        if (ggml_cpu_has_arm_fma()) {
+            features.push_back({ "ARM_FMA", "1" });
+        }
+        if (ggml_cpu_has_fp16_va()) {
+            features.push_back({ "FP16_VA", "1" });
+        }
+        if (ggml_cpu_has_matmul_int8()) {
+            features.push_back({ "MATMUL_INT8", "1" });
+        }
+        if (ggml_cpu_has_sve()) {
+            features.push_back({ "SVE", "1" });
+        }
+        if (ggml_cpu_has_dotprod()) {
+            features.push_back({ "DOTPROD", "1" });
+        }
+        if (ggml_cpu_get_sve_cnt() > 0) {
+            static std::string sve_cnt = std::to_string(ggml_cpu_get_sve_cnt());
+            features.push_back({ "SVE_CNT", sve_cnt.c_str() });
+        }
+        if (ggml_cpu_has_sme()) {
+            features.push_back({ "SME", "1" });
+        }
+        if (ggml_cpu_has_riscv_v()) {
+            features.push_back({ "RISCV_V", "1" });
+        }
+        if (ggml_cpu_has_vsx()) {
+            features.push_back({ "VSX", "1" });
+        }
+        if (ggml_cpu_has_vxe()) {
+            features.push_back({ "VXE", "1" });
+        }
+        if (ggml_cpu_has_nnpa()) {
+            features.push_back({ "NNPA", "1" });
+        }
+        if (ggml_cpu_has_wasm_simd()) {
+            features.push_back({ "WASM_SIMD", "1" });
+        }
+        if (ggml_cpu_has_llamafile()) {
+            features.push_back({ "LLAMAFILE", "1" });
+        }
+    #ifdef GGML_USE_ACCELERATE
+        features.push_back({ "ACCELERATE", "1" });
+    #endif
+    #ifdef GGML_USE_CPU_HBM
+        features.push_back({ "CPU_HBM", "1" });
+    #endif
+    #ifdef GGML_USE_OPENMP
+        features.push_back({ "OPENMP", "1" });
+    #endif
+    #ifdef GGML_USE_CPU_KLEIDIAI
+        features.push_back({ "KLEIDIAI", "1" });
+    #endif
+    #ifdef GGML_USE_CPU_REPACK
+        features.push_back({ "REPACK", "1" });
+    #endif
+
+        features.push_back({ nullptr, nullptr });
+
+        return features;
+    }();
+
+    return features.data();
+
+    GGML_UNUSED(reg);
+}
+
+static void * ggml_backend_cpu_get_proc_address(ggml_backend_reg_t reg, const char * name) {
+    if (strcmp(name, "ggml_backend_set_n_threads") == 0) {
+        ggml_backend_set_n_threads_t fct = ggml_backend_cpu_set_n_threads;
+        return (void *)fct;
+    }
+    if (strcmp(name, "ggml_backend_dev_get_extra_bufts") == 0) {
+        ggml_backend_dev_get_extra_bufts_t fct = ggml_backend_cpu_device_get_extra_buffers_type;
+        return (void *)fct;
+    }
+    if (strcmp(name, "ggml_backend_get_features") == 0) {
+        return (void *)ggml_backend_cpu_get_features;
+    }
+    if (strcmp(name, "ggml_backend_set_abort_callback") == 0) {
+        return (void *)ggml_backend_cpu_set_abort_callback;
+    }
+    if (strcmp(name, "ggml_backend_cpu_numa_init") == 0) {
+        return (void *)ggml_numa_init;
+    }
+    if (strcmp(name, "ggml_backend_cpu_is_numa") == 0) {
+        return (void *)ggml_is_numa;
+    }
+
+    // threadpool - TODO:  move to ggml-base
+    if (strcmp(name, "ggml_threadpool_new") == 0) {
+        return (void *)ggml_threadpool_new;
+    }
+    if (strcmp(name, "ggml_threadpool_free") == 0) {
+        return (void *)ggml_threadpool_free;
+    }
+    if (strcmp(name, "ggml_backend_cpu_set_threadpool") == 0) {
+        return (void *)ggml_backend_cpu_set_threadpool;
+    }
+
+    return NULL;
+
+    GGML_UNUSED(reg);
+}
+
+static const struct ggml_backend_reg_i ggml_backend_cpu_reg_i = {
+    /* .get_name         = */ ggml_backend_cpu_reg_get_name,
+    /* .get_device_count = */ ggml_backend_cpu_reg_get_device_count,
+    /* .get_device       = */ ggml_backend_cpu_reg_get_device,
+    /* .get_proc_address = */ ggml_backend_cpu_get_proc_address,
+};
+
+ggml_backend_reg_t ggml_backend_cpu_reg(void) {
+    // init CPU feature detection
+    ggml_cpu_init();
+
+    static struct ggml_backend_reg ggml_backend_cpu_reg = {
+        /* .api_version = */ GGML_BACKEND_API_VERSION,
+        /* .iface       = */ ggml_backend_cpu_reg_i,
+        /* .context     = */ NULL,
+    };
+
+    return &ggml_backend_cpu_reg;
+}
+
+GGML_BACKEND_DL_IMPL(ggml_backend_cpu_reg)

ggml/src/ggml-cpu/hbm.cpp

@@ -0,0 +1,55 @@
+#ifdef GGML_USE_CPU_HBM
+
+#include "ggml-backend.h"
+#include "ggml-backend-impl.h"
+#include "ggml-cpu.h"
+#include "ggml-impl.h"
+
+#include "hbm.h"
+
+// buffer type HBM
+
+#include <hbwmalloc.h>
+
+static const char * ggml_backend_cpu_hbm_buffer_type_get_name(ggml_backend_buffer_type_t buft) {
+    return "CPU_HBM";
+
+    GGML_UNUSED(buft);
+}
+
+static void ggml_backend_cpu_hbm_buffer_free_buffer(ggml_backend_buffer_t buffer) {
+    hbw_free(buffer->context);
+}
+
+static ggml_backend_buffer_t ggml_backend_cpu_hbm_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft,
+                                                                           size_t                     size) {
+    void * ptr;
+    int    result = hbw_posix_memalign(&ptr, ggml_backend_cpu_buffer_type_get_alignment(buft), size);
+    if (result != 0) {
+        GGML_LOG_ERROR("failed to allocate HBM buffer of size %zu\n", size);
+        return NULL;
+    }
+
+    ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size);
+    buffer->buft                 = buft;
+    buffer->iface.free_buffer    = ggml_backend_cpu_hbm_buffer_free_buffer;
+
+    return buffer;
+}
+
+ggml_backend_buffer_type_t ggml_backend_cpu_hbm_buffer_type(void) {
+    static struct ggml_backend_buffer_type ggml_backend_cpu_buffer_type_hbm = {
+        /* .iface    = */ {
+                           /* .get_name         = */ ggml_backend_cpu_hbm_buffer_type_get_name,
+                           /* .alloc_buffer     = */ ggml_backend_cpu_hbm_buffer_type_alloc_buffer,
+                           /* .get_alignment    = */ ggml_backend_cpu_buffer_type_get_alignment,
+                           /* .get_max_size     = */ nullptr,  // defaults to SIZE_MAX
+                           /* .get_alloc_size   = */ nullptr,  // defaults to ggml_nbytes
+                           /* .is_host          = */ ggml_backend_cpu_buffer_type_is_host,
+                           },
+        /* .context  = */ nullptr,
+    };
+
+    return &ggml_backend_cpu_buffer_type_hbm;
+}
+#endif

ggml/src/ggml-cpu/hbm.h

@@ -0,0 +1,8 @@
+#pragma once
+
+#include "ggml-backend.h"
+#include "ggml.h"
+
+// GGML CPU internal header
+
+ggml_backend_buffer_type_t ggml_backend_cpu_hbm_buffer_type(void);

ggml/src/ggml-cpu/kleidiai/kernels.cpp

@@ -0,0 +1,434 @@
+// SPDX-FileCopyrightText: Copyright 2025 Arm Limited and/or its affiliates <open-source-office@arm.com>
+// SPDX-License-Identifier: MIT
+//
+
+// KleidiAI micro-kernels
+#include "kai_matmul_clamp_f32_qsi8d32p_qsi4c32p_interface.h"
+#include "kai_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod.h"
+#include "kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod.h"
+#include "kai_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod.h"
+#include "kai_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm.h"
+#include "kai_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa.h"
+#include "kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot.h"
+#include "kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa.h"
+
+#include "kai_lhs_pack_bf16p2vlx2_f32_sme.h"
+#include "kai_lhs_quant_pack_qsi8d32p_f32.h"
+#include "kai_lhs_quant_pack_qsi8d32p_f32_neon.h"
+
+#include "kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.h"
+#include "kai_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0.h"
+#include "kai_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon.h"
+
+#include "kai_common.h"
+
+#include "simd-mappings.h"
+
+#include "kernels.h"
+
+#define NELEMS(x) sizeof(x) / sizeof(*x)
+
+static const size_t INT4_PER_BYTE = 2;
+static const size_t INT4_BITS     = 4;
+static const int Q4_0_ZERO_POINT  = 8;
+const size_t INT4_PER_UINT16      = 4;
+
+static void dequantize_row_qsi4c32pscalef16(
+    const void *packed_data,
+    int32_t row_idx,
+    int64_t nc,
+    float *out,
+    size_t nr_pack,
+    size_t packed_row_stride,
+    size_t kr,
+    size_t bl,
+    size_t num_bytes_multiplier
+) {
+    size_t group_idx = row_idx / nr_pack;
+    size_t row_in_group = row_idx % nr_pack;
+    const uint8_t *packed_group = (const uint8_t *)packed_data + group_idx * packed_row_stride;
+    size_t num_blocks = nc / bl;
+    const uint8_t *block_ptr = packed_group;
+
+    for (size_t b = 0; b < num_blocks; ++b) {
+        uint16_t scale_f16 = *((const uint16_t *)(block_ptr + row_in_group * num_bytes_multiplier));
+        float scale = GGML_CPU_FP16_TO_FP32(scale_f16);
+
+        const uint8_t *segment_ptr = block_ptr + nr_pack * num_bytes_multiplier;
+        size_t num_segments = bl / kr;
+        size_t num_bytes_per_segment = kr / INT4_PER_BYTE;
+
+        for (size_t s = 0; s < num_segments; ++s) {
+            const uint8_t *seg_base = segment_ptr + s * nr_pack * num_bytes_per_segment;
+            const uint8_t *qbytes = seg_base + row_in_group * num_bytes_per_segment;
+            for (size_t k = 0; k < num_bytes_per_segment; ++k) {
+                uint8_t byte = qbytes[k] ^ 0x88;
+                int x0 = (byte & 0x0F) - Q4_0_ZERO_POINT;
+                int x1 = (byte >> INT4_BITS) - Q4_0_ZERO_POINT;
+                out[b * bl + s * num_bytes_per_segment + k] = x0 * scale;
+                out[b * bl + s * num_bytes_per_segment + k + bl/2] = x1 * scale;
+            }
+        }
+        block_ptr += nr_pack * num_bytes_multiplier + num_segments * nr_pack * num_bytes_per_segment;
+    }
+}
+
+static void dequantize_row_qsi4c32ps1s0scalef16(
+    const void *packed_data,
+    int32_t row_idx,
+    int64_t k,
+    float *out,
+    size_t nr,
+    size_t packed_row_stride,
+    size_t kr,
+    size_t bl,
+    size_t num_bytes_multiplier
+) {
+    const size_t num_blocks = k / bl;
+    const size_t bl4 = bl / INT4_PER_UINT16;
+
+    size_t group_idx = row_idx / nr;
+    size_t row_in_group = row_idx % nr;
+
+    const uint8_t *packed_group = (const uint8_t *)packed_data + group_idx * packed_row_stride;
+    const uint16_t *qdata = (const uint16_t *)packed_group;
+    const uint16_t *scales = (const uint16_t *)(packed_group + packed_row_stride - (nr * num_blocks * num_bytes_multiplier));
+
+    for (size_t block_idx = 0; block_idx < num_blocks; ++block_idx) {
+        uint16_t scale_f16 = scales[row_in_group + block_idx * nr];
+        float scale = GGML_CPU_FP16_TO_FP32(scale_f16);
+
+        for (size_t bl4_idx = 0; bl4_idx < bl4; ++bl4_idx) {
+            uint16_t q = qdata[(block_idx * bl4 + bl4_idx) * nr + row_in_group];
+
+            for (size_t qidx = 0; qidx < INT4_PER_UINT16; ++qidx) {
+                int v = ((q >> (qidx * 4)) & 0xF) - Q4_0_ZERO_POINT;
+                out[block_idx * bl + bl4_idx * INT4_BITS + qidx] = v * scale;
+            }
+        }
+    }
+    GGML_UNUSED(kr);
+}
+
+static ggml_kleidiai_kernels gemm_gemv_kernels[] = {
+#if defined(__ARM_FEATURE_SME)
+    {
+        /* SME GEMM */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p1vlx4_qsi4c32p4vlx4_1vlx4vl_sme2_mopa,
+        },
+        /* SME GEMV */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot,
+        },
+        /* .lhs_info = */ {
+            /* .get_offset            = */ kai_get_lhs_offset_lhs_quant_pack_qsi8d32p_f32_neon,
+            /* .get_packed_offset     = */ kai_get_lhs_packed_offset_lhs_quant_pack_qsi8d32p_f32_neon,
+            /* .packed_size           = */ kai_get_lhs_packed_size_lhs_quant_pack_qsi8d32p_f32_neon,
+            /* .pack_func             = */ kai_run_lhs_quant_pack_qsi8d32p_f32_neon,
+        },
+        /* .rhs_info = */ {
+            /* .packed_size   = */ kai_get_rhs_packed_size_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon,
+            /* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon,
+            /* .pack_func     = */ kai_run_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon,
+            /* .to_float      = */ dequantize_row_qsi4c32ps1s0scalef16,
+        },
+        /* .required_cpu       = */ CPU_FEATURE_SME,
+        /* .lhs_type           = */ GGML_TYPE_F32,
+        /* .rhs_type           = */ GGML_TYPE_Q4_0,
+        /* .op_type            = */ GGML_TYPE_F32,
+    },
+    {
+        /* SME GEMM */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+        },
+        /* SME GEMV */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa,
+        },
+        /* .lhs_info = */ {
+            /* .get_offset            = */ kai_get_lhs_offset_lhs_pack_bf16p2vlx2_f32_sme,
+            /* .get_packed_offset     = */ kai_get_lhs_packed_offset_lhs_pack_bf16p2vlx2_f32_sme,
+            /* .packed_size           = */ kai_get_lhs_packed_size_lhs_pack_bf16p2vlx2_f32_sme,
+            /* .pack_func             = */ kai_run_lhs_pack_bf16p2vlx2_f32_sme,
+        },
+        /* .rhs_info = */ {
+            /* .packed_size   = */ kai_get_rhs_packed_size_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme,
+            /* .packed_stride = */ NULL,
+            /* .pack_func     = */ kai_run_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme,
+            /* .to_float      = */ NULL,
+        },
+        /* .required_cpu       = */ CPU_FEATURE_SME,
+        /* .lhs_type           = */ GGML_TYPE_F32,
+        /* .rhs_type           = */ GGML_TYPE_F16,
+        /* .op_type            = */ GGML_TYPE_F32,
+    },
+#endif
+#if defined(__APPLE__)
+#if defined(__ARM_FEATURE_DOTPROD)
+    {
+        /* DOTPROD GEMM */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+        },
+        /* DOTPROD GEMV */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+        },
+        /* .lhs_info = */ {
+            /* .get_offset            = */ kai_get_lhs_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .get_packed_offset     = */ kai_get_lhs_packed_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .packed_size           = */ kai_get_lhs_packed_size_lhs_quant_pack_qsi8d32p_f32,
+            /* .pack_func             = */ kai_run_lhs_quant_pack_qsi8d32p_f32,
+        },
+        /* .rhs_info = */ {
+            /* .packed_size   = */ kai_get_rhs_packed_size_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .pack_func     = */ kai_run_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .to_float      = */ dequantize_row_qsi4c32pscalef16,
+        },
+        /* .required_cpu       = */ CPU_FEATURE_DOTPROD,
+        /* .lhs_type           = */ GGML_TYPE_F32,
+        /* .rhs_type           = */ GGML_TYPE_Q4_0,
+        /* .op_type            = */ GGML_TYPE_F32,
+    },
+#endif
+#if defined(__ARM_FEATURE_MATMUL_INT8)
+    {
+        /* i8mm GEMM */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+        },
+        /* i8mm GEMV */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+        },
+        /* .lhs_info = */ {
+            /* .get_offset            = */ kai_get_lhs_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .get_packed_offset     = */ kai_get_lhs_packed_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .packed_size           = */ kai_get_lhs_packed_size_lhs_quant_pack_qsi8d32p_f32,
+            /* .pack_func             = */ kai_run_lhs_quant_pack_qsi8d32p_f32,
+        },
+        /* .rhs_info = */ {
+            /* .packed_size   = */ kai_get_rhs_packed_size_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .pack_func     = */ kai_run_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .to_float      = */ dequantize_row_qsi4c32pscalef16,
+        },
+        /* .required_cpu       = */ CPU_FEATURE_DOTPROD | CPU_FEATURE_I8MM,
+        /* .lhs_type           = */ GGML_TYPE_F32,
+        /* .rhs_type           = */ GGML_TYPE_Q4_0,
+        /* .op_type            = */ GGML_TYPE_F32,
+    },
+#endif
+#else
+#if defined(__ARM_FEATURE_MATMUL_INT8)
+    {
+        /* i8mm GEMM */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p4x8_16x4_neon_i8mm,
+        },
+        /* i8mm GEMV */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p4x8_1x4x32_neon_dotprod,
+        },
+        /* .lhs_info = */ {
+            /* .get_offset            = */ kai_get_lhs_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .get_packed_offset     = */ kai_get_lhs_packed_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .packed_size           = */ kai_get_lhs_packed_size_lhs_quant_pack_qsi8d32p_f32,
+            /* .pack_func             = */ kai_run_lhs_quant_pack_qsi8d32p_f32,
+        },
+        /* .rhs_info = */ {
+            /* .packed_size   = */ kai_get_rhs_packed_size_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .pack_func     = */ kai_run_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .to_float      = */ dequantize_row_qsi4c32pscalef16,
+        },
+        /* .required_cpu       = */ CPU_FEATURE_DOTPROD | CPU_FEATURE_I8MM,
+        /* .lhs_type           = */ GGML_TYPE_F32,
+        /* .rhs_type           = */ GGML_TYPE_Q4_0,
+        /* .op_type            = */ GGML_TYPE_F32,
+    },
+#endif
+#if defined(__ARM_FEATURE_DOTPROD)
+    {
+        /* DOTPROD GEMM */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p4x4_qsi4c32p4x4_16x4_neon_dotprod,
+        },
+        /* DOTPROD GEMV */
+        /* .kern_info = */ {
+            /* .get_m_step            = */ kai_get_m_step_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_n_step            = */ kai_get_n_step_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_mr                = */ kai_get_mr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_nr                = */ kai_get_nr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_kr                = */ kai_get_kr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_sr                = */ kai_get_sr_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_lhs_offset        = */ kai_get_lhs_packed_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_rhs_packed_offset = */ kai_get_rhs_packed_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_dst_offset        = */ kai_get_dst_offset_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .get_dst_size          = */ kai_get_dst_size_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+            /* .run_kernel            = */ kai_run_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4x4_1x4_neon_dotprod,
+        },
+        /* .lhs_info = */ {
+            /* .get_offset            = */ kai_get_lhs_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .get_packed_offset     = */ kai_get_lhs_packed_offset_lhs_quant_pack_qsi8d32p_f32,
+            /* .packed_size           = */ kai_get_lhs_packed_size_lhs_quant_pack_qsi8d32p_f32,
+            /* .pack_func             = */ kai_run_lhs_quant_pack_qsi8d32p_f32,
+        },
+        /* .rhs_info = */ {
+            /* .packed_size   = */ kai_get_rhs_packed_size_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .pack_func     = */ kai_run_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0,
+            /* .to_float      = */ dequantize_row_qsi4c32pscalef16,
+        },
+        /* .required_cpu       = */ CPU_FEATURE_DOTPROD,
+        /* .lhs_type           = */ GGML_TYPE_F32,
+        /* .rhs_type           = */ GGML_TYPE_Q4_0,
+        /* .op_type            = */ GGML_TYPE_F32,
+    },
+#endif
+#endif
+};
+
+ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor) {
+    ggml_kleidiai_kernels * kernel = nullptr;
+
+    if (tensor->op == GGML_OP_MUL_MAT && tensor->src[0] != nullptr && tensor->src[1] != nullptr) {
+        for (size_t i = 0; i < NELEMS(gemm_gemv_kernels); ++i) {
+            if ((cpu_features & gemm_gemv_kernels[i].required_cpu) == gemm_gemv_kernels[i].required_cpu &&
+                gemm_gemv_kernels[i].lhs_type == tensor->src[1]->type &&
+                gemm_gemv_kernels[i].rhs_type == tensor->src[0]->type &&
+                gemm_gemv_kernels[i].op_type  == tensor->type) {
+                kernel = &gemm_gemv_kernels[i];
+                break;
+            }
+        }
+    }
+
+    return kernel;
+}
+
+ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q4_0(cpu_feature features) {
+    ggml_kleidiai_kernels * kernels = nullptr;
+
+    for (size_t i = 0; i < NELEMS(gemm_gemv_kernels); ++i) {
+        if ((features & gemm_gemv_kernels[i].required_cpu) == gemm_gemv_kernels[i].required_cpu) {
+            kernels = &gemm_gemv_kernels[i];
+            break;
+        }
+    }
+
+    return kernels;
+}

ggml/src/ggml-cpu/kleidiai/kernels.h

@@ -0,0 +1,98 @@
+// SPDX-FileCopyrightText: Copyright 2025 Arm Limited and/or its affiliates <open-source-office@arm.com>
+// SPDX-License-Identifier: MIT
+//
+
+#pragma once
+
+#include <functional>
+#include <variant>
+#include "ggml.h"
+
+enum cpu_feature {
+    CPU_FEATURE_NONE    = 0,
+    CPU_FEATURE_DOTPROD = 1,
+    CPU_FEATURE_I8MM    = 2,
+    CPU_FEATURE_SVE     = 4,
+    CPU_FEATURE_SME     = 8
+};
+inline cpu_feature& operator|=(cpu_feature& lhs, cpu_feature rhs) {
+    lhs = static_cast<cpu_feature>(lhs | rhs);
+    return lhs;
+}
+inline cpu_feature operator|(cpu_feature lhs, cpu_feature rhs) {
+    return static_cast<cpu_feature>(static_cast<int>(lhs) | static_cast<int>(rhs));
+}
+
+struct kernel_info {
+    size_t (*get_m_step)(void);
+    size_t (*get_n_step)(void);
+    size_t (*get_mr)(void);
+    size_t (*get_nr)(void);
+    size_t (*get_kr)(void);
+    size_t (*get_sr)(void);
+    std::variant<
+        std::function<size_t(size_t n_idx, size_t k, size_t bl)>,
+        std::function<size_t(size_t m_idx, size_t k)>
+    > get_lhs_offset;
+    std::variant<
+        std::function<size_t(size_t n_idx, size_t k, size_t bl)>,
+        std::function<size_t(size_t n_idx, size_t k)>
+    > get_rhs_packed_offset;
+    size_t (*get_dst_offset)(size_t m_idx, size_t n_idx, size_t stride);
+    size_t (*get_dst_size)(size_t m, size_t n);
+    std::variant<
+        std::function<void(size_t m, size_t n, size_t k, size_t bl, const void* lhs_packed, const void* rhs_packed,
+            float* dst, size_t dst_stride_row, size_t dst_stride_col, float scalar_min, float scalar_max)>,
+        std::function<void(size_t m, size_t n, size_t k, const void* lhs_packed, const void* rhs_packed, void* dst, size_t dst_stride_row,
+            size_t dst_stride_col, float clamp_min, float clamp_max)>
+    > run_kernel;
+};
+
+struct lhs_packing_info {
+    size_t (*get_offset)(size_t m_idx, size_t lhs_stride);
+    std::variant<
+        std::function<size_t(size_t m_idx, size_t k, size_t bl, size_t mr, size_t kr, size_t sr)>,
+        std::function<size_t(size_t m_idx, size_t k, size_t mr, size_t kr, size_t sr)>
+    > get_packed_offset;
+    std::variant<
+        std::function<size_t(size_t m_idx, size_t k, size_t bl, size_t mr, size_t kr, size_t sr)>,
+        std::function<size_t(size_t m, size_t k, size_t mr, size_t kr, size_t sr)>
+    > packed_size;
+    std::variant<
+        std::function<void(size_t m, size_t k, size_t bl, size_t mr, size_t kr, size_t sr, size_t m_idx_start, const float* lhs,
+            size_t lhs_stride, void* lhs_packed)>,
+        std::function<void(size_t m, size_t k, size_t mr, size_t kr, size_t sr, size_t m_idx_start, const void* lhs, size_t lhs_stride,
+        void* lhs_packed)>
+    > pack_func;
+};
+
+struct rhs_packing_info {
+    std::variant<
+        std::function<size_t(size_t n, size_t k, size_t nr, size_t kr, size_t bl)>,
+        std::function<size_t(size_t n, size_t k)>
+    > packed_size;
+    size_t (*packed_stride)(size_t k, size_t nr, size_t kr, size_t bl);
+    std::variant<
+        std::function<void(size_t num_groups, size_t n, size_t k, size_t nr, size_t kr, size_t sr, size_t bl, const uint8_t* rhs,
+            const float* bias, void* rhs_packed, size_t extra_bytes, const struct kai_rhs_pack_qs4cxs1s0_param* params)>,
+        std::function<void(size_t num_groups, size_t n, size_t k, size_t nr, size_t kr, size_t sr, size_t rhs_stride, const void* rhs,
+            const void* bias, const void* scale, void* rhs_packed, size_t extra_bytes, const void* params)>
+    > pack_func;
+    void (*to_float)(const void *packed_data, int32_t row_idx, int64_t nc, float *out, size_t nr_pack, size_t packed_row_stride,
+          size_t kr, size_t bl, size_t num_bytes_multiplier);
+};
+
+struct ggml_kleidiai_kernels {
+    kernel_info gemm;
+    kernel_info gemv;
+    lhs_packing_info lhs_info;
+    rhs_packing_info rhs_info;
+
+    cpu_feature required_cpu;
+    ggml_type lhs_type;
+    ggml_type rhs_type;
+    ggml_type op_type;
+};
+
+ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor);
+ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q4_0(cpu_feature features);

ggml/src/ggml-cpu/kleidiai/kleidiai.cpp

@@ -0,0 +1,560 @@
+// SPDX-FileCopyrightText: Copyright 2025 Arm Limited and/or its affiliates <open-source-office@arm.com>
+// SPDX-License-Identifier: MIT
+//
+#include <arm_neon.h>
+#include <assert.h>
+#include <atomic>
+#include <cfloat>
+#include <stdexcept>
+#include <stdint.h>
+#include <string.h>
+#if defined(__linux__)
+#include <asm/hwcap.h>
+#include <sys/auxv.h>
+#elif defined(__APPLE__)
+#include <string_view>
+#include <sys/sysctl.h>
+#include <sys/types.h>
+#elif defined(_WIN32)
+#include <windows.h>
+#include <excpt.h>
+#endif
+
+#include "kleidiai.h"
+
+#include "ggml-cpu.h"
+#include "ggml-impl.h"
+#include "ggml-backend-impl.h"
+#include "ggml-threading.h"
+#include "traits.h"
+
+#include "kernels.h"
+
+#include "kai_common.h"
+
+#define GGML_COMMON_DECL_CPP
+#include "ggml-common.h"
+
+struct ggml_kleidiai_context {
+    cpu_feature features;
+    ggml_kleidiai_kernels * kernels;
+} static ctx = { CPU_FEATURE_NONE, NULL };
+
+static const char* cpu_feature_to_string(cpu_feature f) {
+    switch (f) {
+        case CPU_FEATURE_NONE:    return "NONE";
+        case CPU_FEATURE_DOTPROD: return "DOTPROD";
+        case CPU_FEATURE_I8MM:    return "I8MM";
+        case CPU_FEATURE_SVE:     return "SVE";
+        case CPU_FEATURE_SME:     return "SME";
+        default:                  return "UNKNOWN";
+    }
+}
+
+static void init_kleidiai_context(void) {
+
+    ggml_critical_section_start();
+    static bool initialized = false;
+
+    if (!initialized) {
+        initialized = true;
+        const char *env_var = getenv("GGML_KLEIDIAI_SME");
+        int sme_enabled = 0;
+
+        ctx.features  = (ggml_cpu_has_dotprod()     ? CPU_FEATURE_DOTPROD : CPU_FEATURE_NONE) |
+                        (ggml_cpu_has_matmul_int8() ? CPU_FEATURE_I8MM    : CPU_FEATURE_NONE) |
+                        (ggml_cpu_has_sve()         ? CPU_FEATURE_SVE     : CPU_FEATURE_NONE);
+
+        if (env_var) {
+            sme_enabled = atoi(env_var);
+        }
+
+        if (sme_enabled != 0) {
+            ctx.features |= ggml_cpu_has_sme() ? CPU_FEATURE_SME : CPU_FEATURE_NONE;
+        }
+        ctx.kernels = ggml_kleidiai_select_kernels_q4_0(ctx.features);
+#ifndef NDEBUG
+        if (ctx.kernels) {
+            GGML_LOG_DEBUG("kleidiai: using kernel with CPU feature %s\n", cpu_feature_to_string(ctx.kernels->required_cpu));
+        }
+#endif
+    }
+    ggml_critical_section_end();
+}
+
+static inline int64_t ggml_ne(const ggml_tensor * tensor, int dim) {
+    GGML_ASSERT(dim >= 0 && dim < GGML_MAX_DIMS);
+    return tensor->ne[dim];
+}
+
+template<typename Ret, typename Variant, typename... Args>
+static Ret variant_call(const Variant & var, Args&&... args) {
+    return std::visit([&](auto&& func) -> Ret {
+        if constexpr (std::is_invocable_r_v<Ret, decltype(func), Args...>) {
+            return func(std::forward<Args>(args)...);
+        } else {
+            throw std::runtime_error("Invalid function type in variant_call");
+        }
+    }, var);
+}
+
+namespace ggml::cpu::kleidiai {
+
+static size_t round_down(size_t x, size_t y) {
+    return y == 0 ? x : x - (x % y);
+}
+
+static void transpose_f32kxn_f16nxk(size_t n, size_t k, float * dst, const uint16_t * src, size_t rhs_stride) {
+    size_t src_stride = rhs_stride / sizeof(uint16_t);
+    size_t dst_stride = n;
+
+    for (size_t k_idx = 0; k_idx < k; ++k_idx) {
+        for (size_t n_idx = 0; n_idx < n; ++n_idx) {
+            uint16_t v = *(src + k_idx + n_idx * src_stride);
+            *(dst + n_idx + k_idx * dst_stride) = kai_cast_f32_f16(v);
+        }
+    }
+}
+
+class tensor_traits : public ggml::cpu::tensor_traits {
+    bool work_size(int /* n_threads */, const struct ggml_tensor * op, size_t & size) override {
+        if (op->op != GGML_OP_MUL_MAT) {
+            return false;
+        }
+        ggml_kleidiai_kernels *kernels = ggml_kleidiai_select_kernels(ctx.features, op);
+        GGML_ASSERT(kernels);
+        kernel_info * kernel = op->src[1]->ne[1] == 1 ? &kernels->gemv : &kernels->gemm;
+
+        size_t k = op->src[0]->ne[0];
+        size_t n = op->src[0]->ne[1];
+        size_t m = op->src[1]->ne[1];
+
+        size_t mr = kernel->get_mr();
+        size_t kr = kernel->get_kr();
+        size_t sr = kernel->get_sr();
+
+        if (kernels->rhs_type == GGML_TYPE_Q4_0) {
+            size = variant_call<size_t>(kernels->lhs_info.packed_size, m, k, QK4_0, mr, kr, sr);
+        } else if (kernels->rhs_type == GGML_TYPE_F16) {
+            size = variant_call<size_t>(kernels->lhs_info.packed_size, m, k, mr, kr, sr) +
+                   variant_call<size_t>(kernels->rhs_info.packed_size, n, k) +
+                   k * n * sizeof(float) + n * sizeof(float);
+        } else {
+            GGML_ASSERT(false);
+        }
+
+        return true;
+    }
+
+
+    bool compute_forward(struct ggml_compute_params * params, struct ggml_tensor * dst) override {
+        if (dst->op == GGML_OP_MUL_MAT) {
+            if (dst->src[0]->type == GGML_TYPE_Q4_0) {
+                return compute_forward_q4_0(params, dst);
+            } else if (dst->src[0]->type == GGML_TYPE_F16) {
+                return compute_forward_kv_cache(params, dst);
+            }
+        } else if (dst->op == GGML_OP_GET_ROWS) {
+            if (dst->src[0]->type == GGML_TYPE_Q4_0) {
+                return compute_forward_get_rows(params, dst);
+            }
+        }
+        return false;
+    }
+
+    bool compute_forward_kv_cache(ggml_compute_params * params, struct ggml_tensor * dst) {
+        static std::atomic_flag first_to_arrive = ATOMIC_FLAG_INIT;
+
+        const ggml_tensor * src0 = dst->src[0];
+        const ggml_tensor * src1 = dst->src[1];
+
+        GGML_TENSOR_BINARY_OP_LOCALS
+
+        ggml_kleidiai_kernels *kernels = ggml_kleidiai_select_kernels(ctx.features, dst);
+        GGML_ASSERT(kernels);
+
+        kernel_info * kernel = src1->ne[1] == 1 ? &kernels->gemv : &kernels->gemm;
+        GGML_ASSERT(kernel);
+
+        const int nth = params->nth;
+        const int ith = params->ith;
+
+        const int64_t lhs_batch_size0 = ne12;
+        const int64_t rhs_batch_size0 = ne02;
+        const int64_t batch_size      = rhs_batch_size0;
+
+        const int64_t r = lhs_batch_size0 / rhs_batch_size0;
+
+        const int64_t m = ne11 * r;
+        const int64_t n = ne01;
+        const int64_t k = ne00;
+
+        const size_t lhs_stride = src1->nb[1];
+        const size_t rhs_stride = src0->nb[1];
+        const size_t dst_stride = dst->nb[1];
+
+        const int64_t mr = static_cast<int64_t>(kernel->get_mr());
+        const int64_t nr = static_cast<int64_t>(kernel->get_nr());
+        const int64_t kr = static_cast<int64_t>(kernel->get_kr());
+        const int64_t sr = static_cast<int64_t>(kernel->get_sr());
+
+        const size_t lhs_packed_size = variant_call<size_t>(kernels->lhs_info.packed_size, m, k, mr, kr, sr);
+        const size_t rhs_packed_size = variant_call<size_t>(kernels->rhs_info.packed_size, n, k);
+        const size_t kxn_size        = k * n * sizeof(float);
+        const size_t bias_size       = n * sizeof(float);
+
+        const size_t wsize_required = lhs_packed_size + rhs_packed_size + kxn_size + bias_size;
+        GGML_ASSERT(wsize_required <= params->wsize);
+
+        uint8_t * lhs_packed = static_cast<uint8_t *>(params->wdata);
+        uint8_t * rhs_packed = lhs_packed + lhs_packed_size;
+        uint8_t * rhs_kxn    = rhs_packed + rhs_packed_size;
+        uint8_t * bias       = rhs_kxn + kxn_size;
+
+        for (int64_t batch_idx = 0; batch_idx < batch_size; ++batch_idx) {
+            const uint8_t * lhs_batch = static_cast<const uint8_t *>(src1->data) + batch_idx * m * lhs_stride;
+            const uint8_t * rhs_batch = static_cast<const uint8_t *>(src0->data) + batch_idx * n * rhs_stride;
+            uint8_t * dst_batch       = static_cast<uint8_t *>(dst->data) + batch_idx * m * dst_stride;
+
+            // LHS packing
+            {
+                const int64_t m_roundup_mr = kai_roundup(m, mr);
+                const int64_t num_threads  = KAI_MIN(m_roundup_mr / mr, nth);
+
+                if (ith < num_threads) {
+                    const int64_t num_m_per_thread0   = round_down(m_roundup_mr / num_threads, mr);
+                    const int64_t num_m_per_threadN_1 = m - (num_threads - 1) * num_m_per_thread0;
+
+                    const int64_t m_start          = ith * num_m_per_thread0;
+                    const int64_t num_m_per_thread = (ith == num_threads - 1) ? num_m_per_threadN_1 : num_m_per_thread0;
+
+                    const size_t lhs_offset        = variant_call<size_t>(kernels->gemm.get_lhs_offset, m_start, lhs_stride);
+                    const size_t lhs_packed_offset = variant_call<size_t>(kernels->lhs_info.get_packed_offset, m_start, k, mr, kr, sr);
+
+                    const void * src_ptr = static_cast<const uint8_t *>(lhs_batch) + lhs_offset;
+                    void * dst_ptr       = static_cast<uint8_t *>(lhs_packed) + lhs_packed_offset;
+
+                    variant_call<void>(kernels->lhs_info.pack_func, num_m_per_thread, k, mr, kr, sr, 0, src_ptr, lhs_stride, dst_ptr);
+                }
+            }
+
+            // RHS packing
+            if (first_to_arrive.test_and_set(std::memory_order_acquire) == false) {
+                // First thread to reach this point handles RHS packing
+                memset(bias, 0, n * sizeof(float));
+                transpose_f32kxn_f16nxk(n, k, reinterpret_cast<float *>(rhs_kxn),
+                                        reinterpret_cast<const uint16_t *>(rhs_batch), rhs_stride);
+
+                variant_call<void>(kernels->rhs_info.pack_func, 1, n, k, nr, kr, sr, n * sizeof(float),
+                             rhs_kxn, bias, nullptr, rhs_packed, 0, nullptr);
+            }
+
+            ggml_barrier(params->threadpool);
+
+            first_to_arrive.clear(std::memory_order_release);
+
+            // Perform the matmul
+            {
+                const int64_t m_to_process = m;
+                const int64_t m_start      = 0;
+
+                const int64_t n_step      = static_cast<int64_t>(kernel->get_n_step());
+                const int64_t num_threads = KAI_MIN(n / n_step, nth);
+
+                if (ith < num_threads) {
+                    const int64_t num_n_per_thread0   = round_down(n / num_threads, n_step);
+                    const int64_t num_n_per_threadN_1 = n - (num_threads - 1) * num_n_per_thread0;
+
+                    const int64_t n_start      = ith * num_n_per_thread0;
+                    const int64_t n_to_process = (ith == num_threads - 1) ? num_n_per_threadN_1 : num_n_per_thread0;
+
+                    const size_t lhs_packed_offset = variant_call<size_t>(kernel->get_lhs_offset, m_start, k);
+                    const size_t rhs_packed_offset = variant_call<size_t>(kernel->get_rhs_packed_offset, n_start, k);
+                    const size_t dst_offset        = kernel->get_dst_offset(m_start, n_start, dst_stride);
+
+                    const void * lhs_ptr = lhs_packed + lhs_packed_offset;
+                    const void * rhs_ptr = rhs_packed + rhs_packed_offset;
+                    float * dst_ptr      = reinterpret_cast<float *>(dst_batch + dst_offset);
+
+                    variant_call<void>(kernel->run_kernel, m_to_process, n_to_process, k, lhs_ptr, rhs_ptr, dst_ptr, dst_stride, sizeof(float), -FLT_MAX, FLT_MAX);
+                }
+            }
+
+            if (batch_idx != batch_size - 1) {
+                // This barrier is necessary when the batch size is larger than 1. While processing a batch,
+                // the work data buffer (params->wdata) is used as temporary storage which means that only
+                // a single batch can be processed at any given time. No barrier is needed for the last
+                // batch since GGML inserts a barrier between the execution of every operator.
+                ggml_barrier(params->threadpool);
+            }
+        }
+
+        return true;
+    }
+
+    bool compute_forward_q4_0(struct ggml_compute_params * params, struct ggml_tensor * dst) {
+        GGML_ASSERT(dst->src[0]->type == GGML_TYPE_Q4_0);
+
+        const ggml_tensor * src0 = dst->src[0];
+        const ggml_tensor * src1 = dst->src[1];
+
+        GGML_TENSOR_BINARY_OP_LOCALS
+
+        ggml_kleidiai_kernels *kernels = ggml_kleidiai_select_kernels(ctx.features, dst);
+        GGML_ASSERT(kernels);
+
+        kernel_info * kernel = src1->ne[1] == 1 ? &kernels->gemv : &kernels->gemm;
+        lhs_packing_info * lhs_info = &kernels->lhs_info;
+
+        GGML_ASSERT(kernel);
+
+        const int ith = params->ith;
+        const int nth = params->nth;
+
+        const size_t k = ne00;
+        const size_t m = ne11;
+        const size_t n = ne01;
+
+        size_t mr = kernel->get_mr();
+        size_t kr = kernel->get_kr();
+        size_t sr = kernel->get_sr();
+
+        const uint8_t * lhs        = static_cast<const uint8_t *>(src1->data);
+        uint8_t * lhs_packed       = (uint8_t*)params->wdata;
+        const uint8_t * rhs_packed = static_cast<const uint8_t *>(src0->data);
+
+        const size_t n_step = kernel->get_n_step();
+        const size_t num_n_per_thread = kai_roundup(kai_roundup(n, nth) / nth, n_step);
+        const size_t n_start = ith * num_n_per_thread;
+
+        size_t n_to_process = num_n_per_thread;
+        if ((n_start + n_to_process) > n) {
+            n_to_process = n - n_start;
+        }
+
+        // Calculate number of columns to be processed per thread
+        const size_t num_m_per_thread = kai_roundup(m, mr * nth) / nth;
+        const size_t m_start = ith * num_m_per_thread;
+        size_t m_to_process = num_m_per_thread;
+        if ((m_start + m_to_process) > m) {
+            m_to_process = m - m_start;
+        }
+
+        if (m_start < m) {
+            // Transform LHS
+            const size_t src_stride        = src1->nb[1];
+            const float * src_ptr          = reinterpret_cast<const float *>(lhs + lhs_info->get_offset(m_start, dst->src[1]->nb[1]));
+            const size_t lhs_packed_offset = variant_call<size_t>(lhs_info->get_packed_offset, m_start, k, QK4_0, mr, kr, sr);
+            void * lhs_packed_ptr          = static_cast<void *>(lhs_packed + lhs_packed_offset);
+
+            variant_call<void>(lhs_info->pack_func, m_to_process, k, QK4_0, mr, kr, sr, 0, src_ptr, src_stride, lhs_packed_ptr);
+        }
+
+        ggml_barrier(params->threadpool);
+
+        // Perform the operation
+        const size_t dst_stride        = dst->nb[1];
+        const size_t lhs_packed_offset = variant_call<size_t>(lhs_info->get_packed_offset, 0, k, QK4_0, mr, kr, sr);
+        const size_t rhs_packed_offset = variant_call<size_t>(kernel->get_rhs_packed_offset, n_start, k, QK4_0);
+        const size_t dst_offset        = kernel->get_dst_offset(0, n_start, dst_stride);
+        const void * rhs_ptr           = static_cast<const void *>(rhs_packed + rhs_packed_offset);
+        const void* lhs_ptr            = (const void*)((const char *)lhs_packed + lhs_packed_offset);
+        float *dst_ptr                 = reinterpret_cast<float *>(static_cast<uint8_t *>(dst->data) + dst_offset);
+
+        variant_call<void>(kernel->run_kernel, m, n_to_process, k, QK4_0, lhs_ptr, rhs_ptr, dst_ptr, dst_stride,
+                           sizeof(float), -FLT_MAX, FLT_MAX);
+
+        return true;
+    }
+
+    bool compute_forward_get_rows(struct ggml_compute_params * params, struct ggml_tensor * dst) {
+        GGML_ASSERT(dst->src[0]->type == GGML_TYPE_Q4_0);
+        GGML_ASSERT(ctx.kernels);
+
+        const ggml_tensor * src0 = dst->src[0];
+        const ggml_tensor * src1 = dst->src[1];
+
+        GGML_TENSOR_BINARY_OP_LOCALS
+
+        rhs_packing_info * rhs_info = &ctx.kernels->rhs_info;
+        kernel_info * kernel        = &ctx.kernels->gemm;
+
+        const int64_t nc     = ne00;
+        const int64_t nr     = ggml_nelements(src1);
+
+        const size_t block_rows = kernel->get_nr();
+        const size_t kr         = kernel->get_kr();
+
+        const size_t num_bytes_multiplier = sizeof(uint16_t);
+        const size_t packed_stride = rhs_info->packed_stride(nc, block_rows, kr, QK4_0);
+
+        const int ith = params->ith;
+        const int nth = params->nth;
+
+        const int dr = (nr + nth - 1) / nth;
+        const int ir0 = dr * ith;
+        const int ir1 = MIN(ir0 + dr, nr);
+
+        for (int64_t i = ir0; i < ir1; ++i) {
+            GGML_ASSERT(src1->type == GGML_TYPE_I32);
+            int64_t row_idx = ((const int32_t *)src1->data)[i];
+            GGML_ASSERT(row_idx >= 0 && row_idx < src0->ne[1]);
+
+            float *out = (float *)((char *)dst->data + i * nb1);
+            rhs_info->to_float(src0->data, row_idx, nc, out, block_rows, packed_stride, kr, QK4_0, num_bytes_multiplier);
+        }
+
+        return true;
+    }
+
+public:
+    int repack(struct ggml_tensor * tensor, const void * data, size_t data_size) {
+        GGML_ASSERT(tensor->type == GGML_TYPE_Q4_0);
+        GGML_ASSERT(ctx.kernels);
+        const size_t n = tensor->ne[1];
+        const size_t k = tensor->ne[0];
+        size_t nr      = ctx.kernels->gemm.get_nr();
+        size_t kr      = ctx.kernels->gemm.get_kr();
+        size_t sr      = ctx.kernels->gemm.get_sr();
+
+        struct kai_rhs_pack_qs4cxs1s0_param params;
+        params.lhs_zero_point = 1;
+        params.rhs_zero_point = 8;
+        variant_call<void>(ctx.kernels->rhs_info.pack_func, 1, n, k, nr, kr, sr, QK4_0, (const uint8_t*)data, nullptr, tensor->data, 0, &params);
+
+        return 0;
+        GGML_UNUSED(data_size);
+    }
+};
+
+static ggml::cpu::tensor_traits * get_tensor_traits(ggml_backend_buffer_t, struct ggml_tensor *) {
+    static tensor_traits traits;
+    return &traits;
+}
+}  // namespace ggml::cpu::kleidiai
+
+static enum ggml_status ggml_backend_cpu_kleidiai_buffer_init_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor) {
+    tensor->extra = (void *) ggml::cpu::kleidiai::get_tensor_traits(buffer, tensor);
+
+    return GGML_STATUS_SUCCESS;
+    GGML_UNUSED(buffer);
+}
+
+static void ggml_backend_cpu_kleidiai_buffer_set_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor,
+                                                       const void * data, size_t offset, size_t size) {
+    GGML_ASSERT(offset == 0);
+    GGML_ASSERT(size == ggml_nbytes(tensor));
+
+    auto tensor_traits = (ggml::cpu::kleidiai::tensor_traits *) tensor->extra;
+    auto OK            = tensor_traits->repack(tensor, data, size);
+
+    GGML_ASSERT(OK == 0);
+    GGML_UNUSED(buffer);
+}
+
+static const char * ggml_backend_cpu_kleidiai_buffer_type_get_name(ggml_backend_buffer_type_t buft) {
+    return "CPU_KLEIDIAI";
+
+    GGML_UNUSED(buft);
+}
+
+static ggml_backend_buffer_t ggml_backend_cpu_kleidiai_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
+    ggml_backend_buffer_t buffer = ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size);
+
+    if (buffer == nullptr) {
+        return nullptr;
+    }
+
+    buffer->buft              = buft;
+    buffer->iface.init_tensor = ggml_backend_cpu_kleidiai_buffer_init_tensor;
+    buffer->iface.set_tensor  = ggml_backend_cpu_kleidiai_buffer_set_tensor;
+    buffer->iface.get_tensor  = nullptr;
+    buffer->iface.cpy_tensor  = nullptr;
+    return buffer;
+}
+
+static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
+    return TENSOR_ALIGNMENT;
+
+    GGML_UNUSED(buft);
+}
+
+static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * tensor) {
+    GGML_ASSERT(tensor->type == GGML_TYPE_Q4_0);
+    GGML_ASSERT(ctx.kernels);
+
+    const size_t n  = tensor->ne[1];
+    const size_t k  = tensor->ne[0];
+    const size_t nr = ctx.kernels->gemm.get_nr();
+    const size_t kr = ctx.kernels->gemm.get_kr();
+
+    return variant_call<size_t>(ctx.kernels->rhs_info.packed_size, n, k, nr, kr, QK4_0);
+
+    GGML_UNUSED(buft);
+}
+
+namespace ggml::cpu::kleidiai {
+class extra_buffer_type : ggml::cpu::extra_buffer_type {
+    bool supports_op(ggml_backend_dev_t, const struct ggml_tensor * op) override {
+        if ((op->op == GGML_OP_MUL_MAT || op->op == GGML_OP_GET_ROWS) &&
+            op->src[0]->type == GGML_TYPE_Q4_0 &&
+            op->src[0]->buffer &&
+            (ggml_n_dims(op->src[0]) == 2) &&
+            op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type() && ctx.kernels) {
+            if (op->op == GGML_OP_GET_ROWS && op->src[1]->ne[0] != 8) {
+                return false;
+            }
+            if (op->src[1]->buffer && !ggml_backend_buft_is_host(op->src[1]->buffer->buft)) {
+                return false;
+            }
+            if ((op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == GGML_TYPE_I32) &&
+                ggml_ne(op->src[1], 2) == 1 && ggml_ne(op->src[1], 3) == 1) {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    ggml::cpu::tensor_traits * get_tensor_traits(const struct ggml_tensor * op) override {
+        if (op->op == GGML_OP_MUL_MAT || op->op == GGML_OP_GET_ROWS) {
+            if (op->src[0]->buffer && op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type()) {
+                return (ggml::cpu::tensor_traits *) op->src[0]->extra;
+            }
+            else if (ggml_kleidiai_select_kernels(ctx.features, op) &&
+                     op->src[0]->op == GGML_OP_VIEW &&
+                     (op->src[1]->op == GGML_OP_PERMUTE || op->src[1]->op ==  GGML_OP_SOFT_MAX) &&
+                     op->src[1]->ne[1] > 1) {
+                if ((op->src[0]->nb[0] != 2) ||
+                    (op->src[1]->nb[0] != 4) ||
+                    (op->src[0]->nb[1] * op->src[0]->ne[1] != op->src[0]->nb[2]) ||
+                    (op->src[1]->nb[1] * op->src[1]->ne[1] != op->src[1]->nb[2])) {
+                    return nullptr;
+                }
+
+                return ggml::cpu::kleidiai::get_tensor_traits(NULL, NULL);
+            }
+        }
+        return nullptr;
+    }
+};
+}  // namespace ggml::cpu::kleidiai
+
+ggml_backend_buffer_type_t ggml_backend_cpu_kleidiai_buffer_type(void) {
+    static ggml::cpu::kleidiai::extra_buffer_type ctx;
+    static struct ggml_backend_buffer_type ggml_backend_cpu_buffer_type_kleidiai = {
+        /* .iface    = */ {
+                           /* .get_name         = */ ggml_backend_cpu_kleidiai_buffer_type_get_name,
+                           /* .alloc_buffer     = */ ggml_backend_cpu_kleidiai_buffer_type_alloc_buffer,
+                           /* .get_alignment    = */ ggml_backend_cpu_kleidiai_buffer_type_get_alignment,
+                           /* .get_max_size     = */ nullptr,  // defaults to SIZE_MAX
+                           /* .get_alloc_size   = */ ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size,
+                           /* .is_host          = */ nullptr,
+                           },
+        /* .device  = */ ggml_backend_reg_dev_get(ggml_backend_cpu_reg(), 0),
+        /* .context = */ &ctx,
+    };
+
+    init_kleidiai_context();
+
+    return &ggml_backend_cpu_buffer_type_kleidiai;
+}

ggml/src/ggml-cpu/kleidiai/kleidiai.h

@@ -0,0 +1,17 @@
+// SPDX-FileCopyrightText: Copyright 2025 Arm Limited and/or its affiliates <open-source-office@arm.com>
+// SPDX-License-Identifier: MIT
+//
+
+#pragma once
+
+#include "ggml-alloc.h"
+
+#ifdef  __cplusplus
+extern "C" {
+#endif
+
+ggml_backend_buffer_type_t ggml_backend_cpu_kleidiai_buffer_type(void);
+
+#ifdef  __cplusplus
+}
+#endif

ggml/src/ggml-cpu/llamafile/sgemm.h

@@ -0,0 +1,19 @@
+#pragma once
+#include <stdint.h>
+#include <stdbool.h>
+
+#if defined(__VXE__) || defined(__VXE2__)
+#include <vecintrin.h>
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+bool llamafile_sgemm(const struct ggml_compute_params * params, int64_t, int64_t, int64_t,
+                     const void *, int64_t, const void *, int64_t, void *, int64_t,
+                     int, int, int);
+
+#ifdef __cplusplus
+}
+#endif

ggml/src/ggml-cpu/ops.h

@@ -0,0 +1,118 @@
+#pragma once
+
+#include "ggml.h"
+
+//
+// cache line
+//
+
+#if defined(__cpp_lib_hardware_interference_size)
+#define CACHE_LINE_SIZE std::hardware_destructive_interference_size
+#else
+#if defined(__POWER9_VECTOR__)
+#define CACHE_LINE_SIZE 128
+#elif defined(__VXE__) || defined(__VXE2__)
+#define CACHE_LINE_SIZE 256
+#else
+#define CACHE_LINE_SIZE 64
+#endif
+#endif
+
+static const size_t CACHE_LINE_SIZE_F32 = CACHE_LINE_SIZE/sizeof(float);
+
+// Work buffer size for im2col operations in CONV2D
+#define GGML_IM2COL_WORK_SIZE (16 * 1024 * 1024)
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+void ggml_compute_forward_dup(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_add(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_add1(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_acc(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_sum(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_sum_rows(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_mean(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_argmax(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_count_equal(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_repeat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_repeat_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_concat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_silu_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_rms_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_rms_norm_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_group_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_l2_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_out_prod(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_scale(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_set(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_cpy(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_cont(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_reshape(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_view(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_permute(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_transpose(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_get_rows(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_get_rows_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_set_rows(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_diag(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_diag_mask_inf(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_diag_mask_zero(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_soft_max(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_soft_max_ext_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_rope(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_rope_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_clamp(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_conv_transpose_1d(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_im2col(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_im2col_back_f32(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_conv_2d(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_conv_transpose_2d(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_conv_2d_dw(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_pool_1d(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_pool_2d(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_pool_2d_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_upscale(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_pad(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_pad_reflect_1d(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_roll(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_arange(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_timestep_embedding(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_argsort(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_leaky_relu(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_flash_attn_ext(
+    const struct ggml_compute_params * params,
+    const struct ggml_tensor * q,
+    const struct ggml_tensor * k,
+    const struct ggml_tensor * v,
+    const struct ggml_tensor * mask,
+    struct ggml_tensor * dst);
+void ggml_compute_forward_flash_attn_back(
+        const struct ggml_compute_params * params,
+        const bool masked,
+        struct ggml_tensor * dst);
+void ggml_compute_forward_ssm_conv(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_ssm_scan(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_win_part(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_win_unpart(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_unary(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_glu(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_get_rel_pos(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_add_rel_pos(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_rwkv_wkv6(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_rwkv_wkv7(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_gla(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_map_custom1(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_map_custom2(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_map_custom3(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_custom(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_cross_entropy_loss(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_cross_entropy_loss_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_opt_step_adamw(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+void ggml_compute_forward_mul_mat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
+
+#ifdef __cplusplus
+}
+#endif

ggml/src/ggml-cpu/quants.h

@@ -0,0 +1,89 @@
+#pragma once
+
+#define GGML_COMMON_DECL_C
+#include "ggml-common.h"
+
+#include "ggml.h"
+
+// GGML CPU internal header
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+// Quantization
+void quantize_row_q4_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q4_1(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q5_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q5_1(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q8_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q8_1(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+
+void quantize_row_q2_K(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q3_K(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q4_K(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q5_K(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q6_K(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_q8_K(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+
+void quantize_row_tq1_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_tq2_0(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+
+void quantize_row_iq4_nl (const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void quantize_row_iq4_xs (const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+
+// Dot product
+void ggml_vec_dot_q4_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q4_1_q8_1(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q5_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q5_1_q8_1(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q8_0_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+
+void ggml_vec_dot_q2_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q3_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q4_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q5_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q6_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+
+void ggml_vec_dot_tq1_0_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_tq2_0_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+
+void ggml_vec_dot_iq2_xxs_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq2_xs_q8_K (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq2_s_q8_K  (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq3_xxs_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq1_s_q8_K  (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq1_m_q8_K  (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq4_nl_q8_0 (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq4_xs_q8_K (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq3_s_q8_K  (int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+
+// Generic implementation
+void quantize_row_q8_0_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void quantize_row_q8_1_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void quantize_row_q8_K_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int64_t k);
+void ggml_vec_dot_q4_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q4_1_q8_1_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q5_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q5_1_q8_1_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q8_0_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_tq1_0_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_tq2_0_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q2_K_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q3_K_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q4_K_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_q5_K_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy,  size_t by, int nrc);
+void ggml_vec_dot_q6_K_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq2_xxs_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq2_xs_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq2_s_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq3_xxs_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq3_s_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq1_s_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq1_m_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq4_nl_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+void ggml_vec_dot_iq4_xs_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc);
+
+#ifdef __cplusplus
+}
+#endif

ggml/src/ggml-cpu/repack.h

@@ -0,0 +1,98 @@
+#pragma once
+
+#define GGML_COMMON_DECL_CPP
+#include "ggml-common.h"
+
+#include "traits.h"
+#include "ggml.h"
+
+// GGML internal header
+
+ggml_backend_buffer_type_t ggml_backend_cpu_repack_buffer_type(void);
+
+template <int K> constexpr int QK_0() {
+    if constexpr (K == 4) {
+        return QK4_0;
+    }
+    if constexpr (K == 8) {
+        return QK8_0;
+    }
+    return -1;
+}
+
+template <int K, int N> struct block {
+    ggml_half d[N];                         // deltas for N qK_0 blocks
+    int8_t    qs[(QK_0<K>() * N * K) / 8];  // quants for N qK_0 blocks
+};
+
+// control size
+static_assert(sizeof(block<4, 4>) == 4 * sizeof(ggml_half) + QK8_0 * 2, "wrong block<4,4> size/padding");
+static_assert(sizeof(block<4, 8>) == 8 * sizeof(ggml_half) + QK8_0 * 4, "wrong block<4,8> size/padding");
+static_assert(sizeof(block<8, 4>) == 4 * sizeof(ggml_half) + QK8_0 * 4, "wrong block<8,4> size/padding");
+static_assert(sizeof(block<8, 8>) == 8 * sizeof(ggml_half) + QK8_0 * 8, "wrong block<8,8> size/padding");
+
+using block_q4_0x4 = block<4, 4>;
+using block_q4_0x8 = block<4, 8>;
+using block_q8_0x4 = block<8, 4>;
+using block_q8_0x8 = block<8, 8>;
+
+struct block_q4_Kx8 {
+    ggml_half d[8];      // super-block scale for quantized scales
+    ggml_half dmin[8];   // super-block scale for quantized mins
+    uint8_t scales[96];  // scales and mins, quantized with 6 bits
+    uint8_t qs[1024];    // 4--bit quants
+};
+
+static_assert(sizeof(block_q4_Kx8) == sizeof(ggml_half) * 16 + K_SCALE_SIZE * 8 + QK_K * 4, "wrong q4_K block size/padding");
+
+struct block_q8_Kx4 {
+    float d[4];              // delta
+    int8_t qs[QK_K * 4];     // quants
+    int16_t bsums[QK_K / 4]; // sum of quants in groups of 16
+};
+
+static_assert(sizeof(block_q8_Kx4) == sizeof(float) * 4 + QK_K * 4 + (QK_K / 4) * sizeof(int16_t), "wrong q8_K block size/padding");
+
+struct block_iq4_nlx4 {
+    ggml_half d[4];            // deltas for 4 iq4_nl blocks
+    uint8_t   qs[QK4_NL * 2];  // nibbles / quants for 4 iq4_nl blocks
+};
+
+static_assert(sizeof(block_iq4_nlx4) == 4 * sizeof(ggml_half) + QK4_NL * 2, "wrong iq4_nlx4 block size/padding");
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+void ggml_quantize_mat_q8_0_4x4(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void ggml_quantize_mat_q8_0_4x8(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void ggml_quantize_mat_q8_K_4x8(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void ggml_gemv_q4_0_4x4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_q4_0_4x8_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_q4_0_8x8_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_q4_K_8x8_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_iq4_nl_4x4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_0_4x4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_0_4x8_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_0_8x8_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_K_8x8_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_iq4_nl_4x4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+
+// Native implementations
+void ggml_quantize_mat_q8_0_4x4_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void ggml_quantize_mat_q8_0_4x8_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void ggml_quantize_mat_q8_K_4x8_generic(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t k);
+void ggml_gemv_q4_0_4x4_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_q4_0_4x8_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_q4_0_8x8_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_q4_K_8x8_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemv_iq4_nl_4x4_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_0_4x4_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_0_4x8_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_0_8x8_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_q4_K_8x8_q8_K_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+void ggml_gemm_iq4_nl_4x4_q8_0_generic(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, const void * GGML_RESTRICT vy, int nr, int nc);
+
+#if defined(__cplusplus)
+} // extern "C"
+#endif