Files

T

Win Dictation Dev 1f67c07a77 v2 rebuild: GDI+ single-surface UI, self-calibrating progress, compact push-to-talk, GGML+Whisper integration

2026-06-11 15:30:19 +12:00

21 KiB

Raw Blame History

Win Dictation — Architecture Analysis & Dev Guide (Build 4)

Two parts:

Part A — Architecture analysis & recommendations. Where the app stands now (post-fixes) and what's worth hardening.
Part B — Dev guide for the two things you asked for: (1) removing the leftover Windows chrome (combo dropdown buttons, button hairlines, the line around Pinned), and (2) showing real transcription progress + a time estimate instead of a static "Transcribing…".

Context: native C++ / Win32 + SDL2 + whisper.cpp, CPU-only (i5-7th-gen, 2c/4t). Companion docs: MODERN-UI-AND-FIXES.md, FINDINGS-FIXES-TESTS.md.

Part A — Architecture Analysis & Recommendations

A1. Current architecture (it's in good shape)

┌── UI thread (message loop) ───────────────┐     ┌── Audio thread (SDL callback) ──┐
│  • window, hotkeys, tray, GDI+ paint      │     │  • append f32 samples to        │
│  • owns Transcriber                       │◀────│    m_capture (mutex)            │
│  • marshals worker results via PostMessage│     │  • update VU energy (atomic)    │
└──────────────┬────────────────────────────┘     └─────────────────────────────────┘
               │ stop_and_transcribe()
               ▼
┌── Worker thread (one per utterance) ──────┐
│  • run_inference(): whisper_full() ONCE   │
│  • PostMessage(WM_APP_RESULT, text)       │
└────────────────────────────────────────────┘

State machine: Idle → Recording → Transcribing → Idle, with the model preloaded on a detached thread at startup.

What's genuinely good now:

Clean thread boundaries. Audio capture, inference, and UI never block each other; all cross-thread hand-offs are atomics or PostMessage. This is the right shape.
Batch (record-then-transcribe). The correct model for dictation on a slow CPU — no real-time deadline, latency independent of clip length.
Core is decoupled and unit-testable (Transcriber + run_inference/transcribe_sync, text_util.h). The UI is a thin shell over it.
Robust startup (single-instance, honest g_modelOk, g_initializing guard, m_cfg_mtx) — the earlier crashes are designed out, not patched over.

The remaining issues are polish and hardening, not structural. No rewrite needed.

A2. Recommendations (ranked by impact ÷ effort)

#	Recommendation	Why	Effort
1	Transcription progress + ETA (Part B2)	The #1 UX gap: a 1:40 clip shows a static "Transcribing…" with no sign of life. whisper.cpp already exposes a progress callback — wire it up.	S
2	De-theme the stock controls (Part B1.1)	The hairlines around the buttons / Pinned are themed-edge leftovers. `SetWindowTheme(h, L"", L"")` + `WS_CLIPCHILDREN` + hide-focus removes them.	S
3	Replace the comboboxes with custom dropdowns (Part B1.2)	A `CBS_OWNERDRAWFIXED` combo cannot hide its native dropdown button — that's the "second arrow" you see. The only clean fix is to not use a combobox.	M
4	Persist settings	Mic, model, pin, auto-paste, window position all reset every launch. A tiny INI (`WritePrivateProfileStringW`) or registry blob fixes it.	S
5	Cancel/abort during transcription	Pairs with #1 — if a long clip is wrong, let the user abort via `whisper_full_params.abort_callback`.	S
6	Delete the stale `src/CMakeLists.txt`	It references removed APIs (`init`, `is_using_gpu`) and a `test-audio` target the root build ignores. It's a trap for the next contributor.	XS
7	Bound the capture buffer	`m_capture` grows ~1.9 MB/30 s with no cap. Add a soft limit (e.g. auto-stop at 10 min) so a forgotten recording can't grow unbounded.	XS
8	Configurable hotkeys + conflict check	`Ctrl+Shift+Space`/`H` are hard-coded; `RegisterHotKey` failures are silently ignored. Surface a warning and allow remap.	M
9	Lightweight logging	A single rotating log line per session (model, threads, last error) makes field issues diagnosable without a debugger.	S

Architectural note for the future: the app leans on owner-drawing stock controls (buttons, combos). That's fine at this size, but every stock control fights you on chrome (edges, focus cues, dropdown buttons). If the UI grows, the higher-leverage move is a small set of fully custom controls (a Button, a Select, a ProgressBar you paint entirely in WM_PAINT and hit-test yourself) rather than more owner-draw patches. Part B1.2's custom dropdown is the first step down that path.

Part B — Dev Guide

B1. Remove the leftover Windows chrome

There are three separate sources of "Windows-y" pixels, each with a different cause:

What you see	Cause
Thin light line around Pinned; vertical line left + horizontal line top of Copy/Paste/Clear	The visual-style themed edge drawn behind owner-draw buttons, plus focus rectangles
Two arrows / a boxy button on the mic + model selectors	A `CBS_OWNERDRAWFIXED` combo still lets the system draw the dropdown button and field border — owner-draw only covers the text/items

B1.1 Kill the button hairlines + focus rectangles (quick, high-impact)

Three changes, all small:

(a) De-theme the owner-draw controls. SetWindowTheme(h, L"", L"") strips the UxTheme styling from a control so Windows stops painting its themed border/edge behind your WM_DRAWITEM. Add the header/lib and call it on every owner-draw control right after you create it:

#include <uxtheme.h>
#pragma comment(lib, "uxtheme.lib")

// after creating each owner-draw button (Record, Pin, Copy, Paste, Clear):
SetWindowTheme(hBtn, L"", L"");      // remove themed edges
SetWindowSubclass(hBtn, BtnProc, 1, 0);

(b) Stop the parent painting under the children. Add WS_CLIPCHILDREN to the main window so its double-buffered WM_PAINT never bleeds a pixel into a child's rectangle:

hMainWnd = CreateWindowExW(
    WS_EX_TOPMOST,
    L"WhisperDictationClass", L"Dictation",
    WS_OVERLAPPEDWINDOW | WS_CLIPCHILDREN,   // <- add WS_CLIPCHILDREN
    CW_USEDEFAULT, CW_USEDEFAULT, 400, 340,
    nullptr, nullptr, hInstance, nullptr);

(c) Hide focus rectangles app-wide. The dotted/thin rect that appears on whichever control has keyboard focus (often Pinned after you click it). Tell the UI-state machine to keep focus cues hidden — once, after the controls exist:

SendMessageW(hMainWnd, WM_CHANGEUISTATE, MAKEWPARAM(UIS_SET, UISF_HIDEFOCUS), 0);

After (a)+(b)+(c), the only remaining stock chrome is the combo dropdown button, handled next.

A combobox always owns its dropdown button — you can't draw it away. Replace each combo with a flat owner-draw "select" button (you already draw exactly this look in DrawCombo) that opens a custom dark popup window. No native button, no field edge, fully on-theme.

Data model (replace g_modelComboPaths usage as needed; keep a label list + selection per selector):

std::vector<std::wstring> g_audioItems;  int g_audioSel = 0;
std::vector<std::wstring> g_modelItems;  int g_modelSel = 0;   // parallel to g_modelComboPaths

Populate these in RefreshAudioDevices / RefreshModelList instead of CB_ADDSTRING. Create ID_SEL_AUDIO / ID_SEL_MODEL as BS_OWNERDRAW buttons (not comboboxes) and draw them with a field renderer:

void DrawSelect(LPDRAWITEMSTRUCT d, const std::wstring& text) {
    Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
    Rect rc(d->rcItem.left, d->rcItem.top, d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
    SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
    Rect field = rc; field.Inflate(-1,-1);
    bool hover = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
    FillRound(g, hover ? C_SURFACEHI : C_SURFACE, field, 9);
    StrokeRound(g, C_BORDER, field, 9, 1.0f);
    Font f(d->hDC, g_fUI);
    RectF tb((REAL)field.X+10, (REAL)field.Y, (REAL)(field.Width-28), (REAL)field.Height);
    DrawTextC(g, text.c_str(), f, C_TEXT, tb, StringAlignmentNear, StringAlignmentCenter);
    int cx = field.GetRight()-16, cy = field.Y + field.Height/2;     // our chevron, the only one now
    Pen pen(C_TEXTDIM, 1.6f);
    g.DrawLine(&pen, cx-4, cy-2, cx, cy+2); g.DrawLine(&pen, cx, cy+2, cx+4, cy-2);
}
// in WM_DRAWITEM:
//   case ID_SEL_AUDIO: DrawSelect(d, g_audioItems.empty()?L"No devices":g_audioItems[g_audioSel]); return TRUE;
//   case ID_SEL_MODEL: DrawSelect(d, g_modelItems.empty()?L"—":g_modelItems[g_modelSel]); return TRUE;

The popup window — a small borderless top-level you paint yourself; closes on pick or focus loss:

struct PopupState { std::vector<std::wstring> items; int sel; int hot; HWND owner; int ctrlId; };
static PopupState g_pop;

LRESULT CALLBACK PopupProc(HWND h, UINT m, WPARAM w, LPARAM l) {
    switch (m) {
    case WM_MOUSEMOVE: {
        int row = GET_Y_LPARAM(l) / 30;
        if (row != g_pop.hot) { g_pop.hot = row; InvalidateRect(h, nullptr, FALSE); }
        TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 }; TrackMouseEvent(&t);
        return 0;
    }
    case WM_MOUSELEAVE: g_pop.hot = -1; InvalidateRect(h, nullptr, FALSE); return 0;
    case WM_LBUTTONUP: {
        int row = GET_Y_LPARAM(l) / 30;
        if (row >= 0 && row < (int)g_pop.items.size())
            PostMessageW(g_pop.owner, WM_APP_SELECT, (WPARAM)g_pop.ctrlId, (LPARAM)row);
        DestroyWindow(h); return 0;
    }
    case WM_KILLFOCUS: DestroyWindow(h); return 0;
    case WM_ERASEBKGND: return 1;
    case WM_PAINT: {
        PAINTSTRUCT ps; HDC hdc = BeginPaint(h, &ps);
        RECT rc; GetClientRect(h, &rc);
        HDC mem = CreateCompatibleDC(hdc);
        HBITMAP bmp = CreateCompatibleBitmap(hdc, rc.right, rc.bottom);
        HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
        {
            Graphics g(mem); g.SetSmoothingMode(SmoothingModeAntiAlias);
            Rect all(0,0,rc.right,rc.bottom);
            FillRound(g, C_SURFACE, all, 10); StrokeRound(g, C_BORDER, all, 10, 1.0f);
            Font f(mem, g_fUI);
            for (int i = 0; i < (int)g_pop.items.size(); ++i) {
                Rect row(3, i*30+3, rc.right-6, 28);
                if (i == g_pop.hot) FillRound(g, C_SURFACEHI, row, 7);
                RectF tb((REAL)row.X+9,(REAL)row.Y,(REAL)row.Width-12,(REAL)row.Height);
                DrawTextC(g, g_pop.items[i].c_str(), f, (i==g_pop.sel)?C_ACCENT:C_TEXT,
                          tb, StringAlignmentNear, StringAlignmentCenter);
            }
        }
        BitBlt(hdc,0,0,rc.right,rc.bottom,mem,0,0,SRCCOPY);
        SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
        EndPaint(h, &ps); return 0;
    }
    }
    return DefWindowProc(h, m, w, l);
}

void ShowSelectPopup(HWND owner, int ctrlId, const std::vector<std::wstring>& items, int sel) {
    static bool reg = false;
    if (!reg) { WNDCLASSEXW wc{ sizeof(wc) }; wc.lpfnWndProc = PopupProc; wc.hInstance = hInst;
        wc.hCursor = LoadCursor(nullptr, IDC_ARROW); wc.lpszClassName = L"DictPopup";
        RegisterClassExW(&wc); reg = true; }
    g_pop = { items, sel, -1, owner, ctrlId };
    RECT rc; GetWindowRect(GetDlgItem(owner, ctrlId), &rc);
    int h = (int)items.size()*30 + 6, wdt = rc.right - rc.left;
    HWND p = CreateWindowExW(WS_EX_TOOLWINDOW | WS_EX_TOPMOST, L"DictPopup", L"",
        WS_POPUP, rc.left, rc.bottom+2, wdt, h, owner, nullptr, hInst, nullptr);
    int corner = DWMWCP_ROUND; DwmSetWindowAttribute(p, DWMWA_WINDOW_CORNER_PREFERENCE, &corner, sizeof(corner));
    ShowWindow(p, SW_SHOWNA); SetForegroundWindow(p); SetFocus(p);
}

Wire it up: clicking a selector opens the popup; the popup posts the chosen row back:

// WM_COMMAND:
case ID_SEL_AUDIO: ShowSelectPopup(hWnd, ID_SEL_AUDIO, g_audioItems, g_audioSel); break;
case ID_SEL_MODEL: ShowSelectPopup(hWnd, ID_SEL_MODEL, g_modelItems, g_modelSel); break;

// new message handler:
case WM_APP_SELECT: {                      // #define WM_APP_SELECT (WM_USER + 5)
    int ctrlId = (int)wParam, idx = (int)lParam;
    if (ctrlId == ID_SEL_AUDIO) { g_audioSel = idx; g_config.capture_id = idx; }
    else if (ctrlId == ID_SEL_MODEL && idx < (int)g_modelComboPaths.size()) {
        g_modelSel = idx;
        g_config.model_path = exe_dir() + "\\" + g_modelComboPaths[idx];
        g_modelLoaded = false; g_modelOk = false; SetStatus(hWnd, L"Loading model…");
        std::thread([]{ bool ok=g_tx.reload(g_config); g_modelOk=ok; g_modelLoaded=true; }).detach();
    }
    InvalidateRect(GetDlgItem(hWnd, ctrlId), nullptr, FALSE);
} break;

You can now delete the CBS_*/WM_MEASUREITEM/DrawCombo combo code and the WM_CTLCOLORLISTBOX handler. The selectors are now pixel-identical to your other controls with exactly one (your) chevron.

Lighter alternative if you don't want a popup window yet: keep the combos but call SetWindowTheme(hCombo, L"", L"") — it flattens the dropdown button to a plain square. It's less boxy but the button is still there, so the popup-window route above is the real fix.

B2. Transcription progress + time estimate

You don't have to guess the timing — whisper.cpp reports real progress. whisper_full_params has a progress_callback that fires repeatedly during inference with an int 0–100 (fraction of the audio processed). Feed that to a determinate progress bar + a live ETA, so "Transcribing…" becomes "Transcribing 1:40 of audio · 45% · ~9s left".

B2.1 transcriber — expose progress + audio length

transcriber.h (public):

using ProgressCb = std::function<void(int)>;            // 0..100
void  set_progress_callback(ProgressCb cb) { m_on_progress = std::move(cb); }
float audio_seconds() const { return m_audio_seconds.load(); }
void  request_cancel() { m_abort = true; }              // optional (B2.4)

transcriber.h (private):

ProgressCb m_on_progress;
std::atomic<float> m_audio_seconds{0.0f};
std::atomic<bool>  m_abort{false};
static void s_progress(struct whisper_context*, struct whisper_state*, int p, void* ud);
static bool s_abort(void* ud);

transcriber.cpp — the static trampolines and the run_inference hook:

void Transcriber::s_progress(whisper_context*, whisper_state*, int p, void* ud) {
    auto* self = static_cast<Transcriber*>(ud);
    if (self && self->m_on_progress) self->m_on_progress(p);
}
bool Transcriber::s_abort(void* ud) {
    auto* self = static_cast<Transcriber*>(ud);
    return self && self->m_abort.load();
}

std::string Transcriber::run_inference(std::vector<float>& audio) {
    if (!m_ctx) return "";
    m_abort = false;
    if (m_cfg.trim_silence) trim_silence(audio);
    m_audio_seconds = (float)(audio.size() / (double)WHISPER_SAMPLE_RATE);   // for the UI

    whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
    wp.print_progress = false; wp.print_realtime = false; wp.print_timestamps = false;
    wp.no_timestamps  = true;  wp.translate = false;
    wp.language = m_cfg.language.c_str();
    wp.n_threads = m_cfg.n_threads;
    wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true; wp.temperature = 0.0f;
    wp.progress_callback           = &Transcriber::s_progress;   // <- live progress
    wp.progress_callback_user_data = this;
    wp.abort_callback              = &Transcriber::s_abort;      // <- optional cancel
    wp.abort_callback_user_data    = this;

    std::string out;
    if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
        int n = whisper_full_n_segments(m_ctx);
        for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
        out = clean_text(out);
    }
    return out;
}

If your whisper.h predates abort_callback, drop those two lines — progress_callback has been in whisper.cpp far longer and is what matters here.

B2.2 main.cpp — progress state + ETA

#define WM_APP_PROGRESS (WM_USER + 6)
std::atomic<int> g_progress{0};
DWORD g_busyStart = 0;

// at startup, next to set_result_callback:
g_tx.set_progress_callback([](int p){ PostMessage(hMainWnd, WM_APP_PROGRESS, (WPARAM)p, 0); });

Start the clock when transcription begins (HK_TOGGLE stop branch):

} else {                       // was recording -> stop
    g_busyStart = GetTickCount();
    g_progress  = 0;
    g_tx.stop_and_transcribe();
    SetStatus(hWnd, L"Transcribing…");
}

Receive progress and repaint the bar:

case WM_APP_PROGRESS:
    g_progress = (int)wParam;
    InvalidateRect(hWnd, &g_vuRect, FALSE);
    return 0;

ETA text (replace the is_busy() branch in UpdateStatus):

} else if (g_tx.is_busy()) {
    int   p     = g_progress.load();
    float total = g_tx.audio_seconds();
    float elapsed = (GetTickCount() - g_busyStart) / 1000.0f;
    int   mm = (int)total / 60, ss = (int)total % 60;
    if (p >= 3) {
        float est    = elapsed * 100.0f / p;          // projected total
        float remain = est - elapsed; if (remain < 0) remain = 0;
        swprintf_s(buf, L"Transcribing %d:%02d  •  %d%%  •  ~%ds left", mm, ss, p, (int)(remain + 0.5f));
    } else {
        swprintf_s(buf, L"Transcribing %d:%02d of audio…", mm, ss);
    }
    SetStatus(hwnd, buf);
}

B2.3 Reuse the VU strip as a determinate progress bar

While recording the strip shows the VU; while transcribing it shows progress. Add a renderer and branch in WM_PAINT:

void DrawProgress(Graphics& g, const RECT& r, float frac) {
    Rect track(r.left, r.top, r.right - r.left, r.bottom - r.top);
    FillRound(g, C_SURFACEHI, track, 4);
    frac = frac < 0 ? 0 : (frac > 1 ? 1 : frac);
    int w = (int)((r.right - r.left) * frac);
    if (w > 0) { Rect fill(r.left, r.top, w, r.bottom - r.top); FillRound(g, C_ACCENT, fill, 4); }
}

// in WM_PAINT, where you currently call DrawVU:
if (g_tx.is_busy())      DrawProgress(g, g_vuRect, g_progress.load() / 100.0f);
else                     DrawVU(g, g_vuRect, g_energy);

Keep it ticking between callbacks so the ETA counts down smoothly — in WM_TIMER, add:

else if (g_tx.is_busy()) {
    InvalidateRect(hWnd, &g_vuRect, FALSE);   // (UpdateStatus already runs each tick below)
}

That's it: a moving bar, a percentage, and a shrinking "~Ns left" — the user can see it's alive and roughly how long is left. The estimate self-corrects as real progress arrives (the first few percent are rougher; it tightens quickly).

B2.4 (Optional) Cancel a long transcription

You added request_cancel() + the abort callback in B2.1. Hook it so that pressing the button while busy aborts instead of being ignored, and treat the empty result as "Cancelled":

// top of the HK_TOGGLE handler:
if (g_tx.is_busy()) { g_tx.request_cancel(); SetStatus(hWnd, L"Cancelling…"); break; }

whisper_full returns promptly with no/aborted segments → your existing WM_APP_RESULT empty-string path shows "No speech detected"; change that label to "Cancelled" when a cancel was requested if you want to distinguish them.

B3. Build & test

New link deps: uxtheme.lib (added via #pragma comment in B1.1). GDI+ is already linked.
New files: none required for B2; B1.2 adds the popup window proc inside main.cpp (no new translation unit).
Rebuild: cmake --build build --config Release as usual.

Verify:

No hairline around Pinned; no left/top lines on Copy/Paste/Clear; focus no longer draws a dotted rect.
Mic + model selectors show a single (your) chevron, open a dark rounded popup, and selecting reloads the model / switches device.
Record a ~1–2 min clip → the strip fills as a progress bar, the status shows % and a shrinking ~Ns left, and it completes (no static "Transcribing…").
(If added) pressing the button mid-transcription cancels promptly.

Headless regression (extends tests/test_core.cpp from FINDINGS-FIXES-TESTS.md):

Set a progress callback that records the max value seen; assert it reaches ~100 for samples/jfk.wav, and that callbacks arrive in non-decreasing order. This locks in that progress reporting keeps working across whisper.cpp upgrades.

int last = -1, maxp = 0; bool monotonic = true;
t.set_progress_callback([&](int p){ if (p < last) monotonic = false; last = p; if (p > maxp) maxp = p; });
t.transcribe_sync(audio);
CHECK(monotonic, "progress is non-decreasing");
CHECK(maxp >= 95, "progress reaches ~100%");

Apply order: B1.1 (5 min, instant visual win) → B2 (progress, the big UX gain) → B1.2 (custom dropdowns) → optional B2.4 cancel. Each is independent and safe to ship on its own.

21 KiB Raw Blame History Unescape Escape