michael-treadgold/win-dictate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

v2 rebuild: GDI+ single-surface UI, self-calibrating progress, compact push-to-talk, GGML+Whisper integration

Browse Source
This commit is contained in:
Win Dictation Dev
2026-06-11 15:30:19 +12:00
parent 81b3d0073e
commit 1f67c07a77
28 changed files with 11856 additions and 1948 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -28,6 +28,7 @@ cmake_install.cmake
!CMakeLists.txt
!**/CMakeLists.txt
!cmake/*.cmake
!ggml/cmake/*.cmake
# Visual Studio
.vs/
Architechture-and-dev-guide.md
+411
View File
@@ -0,0 +1,411 @@
# Win Dictation — Architecture Analysis & Dev Guide (Build 4)
Two parts:
- **Part A — Architecture analysis & recommendations.** Where the app stands now (post-fixes) and what's worth hardening.
- **Part B — Dev guide** for the two things you asked for: (1) removing the leftover Windows chrome (combo dropdown buttons, button hairlines, the line around *Pinned*), and (2) showing real transcription progress + a time estimate instead of a static "Transcribing…".
Context: native C++ / Win32 + SDL2 + whisper.cpp, CPU-only (i5-7th-gen, 2c/4t). Companion docs: `MODERN-UI-AND-FIXES.md`, `FINDINGS-FIXES-TESTS.md`.
---
# Part A — Architecture Analysis & Recommendations
## A1. Current architecture (it's in good shape)
```
┌── UI thread (message loop) ───────────────┐ ┌── Audio thread (SDL callback) ──┐
│ • window, hotkeys, tray, GDI+ paint │ │ • append f32 samples to │
│ • owns Transcriber │◀────│ m_capture (mutex) │
│ • marshals worker results via PostMessage│ │ • update VU energy (atomic) │
└──────────────┬────────────────────────────┘ └─────────────────────────────────┘
│ stop_and_transcribe()
▼
┌── Worker thread (one per utterance) ──────┐
│ • run_inference(): whisper_full() ONCE │
│ • PostMessage(WM_APP_RESULT, text) │
└────────────────────────────────────────────┘
```
State machine: **Idle → Recording → Transcribing → Idle**, with the model preloaded on a detached thread at startup.
**What's genuinely good now:**
- **Clean thread boundaries.** Audio capture, inference, and UI never block each other; all cross-thread hand-offs are atomics or `PostMessage`. This is the right shape.
- **Batch (record-then-transcribe).** The correct model for dictation on a slow CPU — no real-time deadline, latency independent of clip length.
- **Core is decoupled and unit-testable** (`Transcriber` + `run_inference`/`transcribe_sync`, `text_util.h`). The UI is a thin shell over it.
- **Robust startup** (single-instance, honest `g_modelOk`, `g_initializing` guard, `m_cfg_mtx`) — the earlier crashes are designed out, not patched over.
The remaining issues are **polish and hardening**, not structural. No rewrite needed.
## A2. Recommendations (ranked by impact ÷ effort)
| # | Recommendation | Why | Effort |
|---|---|---|---|
| 1 | **Transcription progress + ETA** (Part B2) | The #1 UX gap: a 1:40 clip shows a static "Transcribing…" with no sign of life. whisper.cpp already exposes a progress callback — wire it up. | S |
| 2 | **De-theme the stock controls** (Part B1.1) | The hairlines around the buttons / *Pinned* are themed-edge leftovers. `SetWindowTheme(h, L"", L"")` + `WS_CLIPCHILDREN` + hide-focus removes them. | S |
| 3 | **Replace the comboboxes with custom dropdowns** (Part B1.2) | A `CBS_OWNERDRAWFIXED` combo *cannot* hide its native dropdown button — that's the "second arrow" you see. The only clean fix is to not use a combobox. | M |
| 4 | **Persist settings** | Mic, model, pin, auto-paste, window position all reset every launch. A tiny INI (`WritePrivateProfileStringW`) or registry blob fixes it. | S |
| 5 | **Cancel/abort during transcription** | Pairs with #1 — if a long clip is wrong, let the user abort via `whisper_full_params.abort_callback`. | S |
| 6 | **Delete the stale `src/CMakeLists.txt`** | It references removed APIs (`init`, `is_using_gpu`) and a `test-audio` target the root build ignores. It's a trap for the next contributor. | XS |
| 7 | **Bound the capture buffer** | `m_capture` grows ~1.9 MB/30 s with no cap. Add a soft limit (e.g. auto-stop at 10 min) so a forgotten recording can't grow unbounded. | XS |
| 8 | **Configurable hotkeys + conflict check** | `Ctrl+Shift+Space`/`H` are hard-coded; `RegisterHotKey` failures are silently ignored. Surface a warning and allow remap. | M |
| 9 | **Lightweight logging** | A single rotating log line per session (model, threads, last error) makes field issues diagnosable without a debugger. | S |
**Architectural note for the future:** the app leans on owner-drawing *stock* controls (buttons, combos). That's fine at this size, but every stock control fights you on chrome (edges, focus cues, dropdown buttons). If the UI grows, the higher-leverage move is a small set of **fully custom controls** (a `Button`, a `Select`, a `ProgressBar` you paint entirely in `WM_PAINT` and hit-test yourself) rather than more owner-draw patches. Part B1.2's custom dropdown is the first step down that path.
---
# Part B — Dev Guide
## B1. Remove the leftover Windows chrome
There are **three** separate sources of "Windows-y" pixels, each with a different cause:
| What you see | Cause |
|---|---|
| Thin light line around *Pinned*; vertical line left + horizontal line top of Copy/Paste/Clear | The visual-style **themed edge** drawn behind owner-draw buttons, plus **focus rectangles** |
| Two arrows / a boxy button on the mic + model selectors | A `CBS_OWNERDRAWFIXED` combo still lets the **system draw the dropdown button** and field border — owner-draw only covers the text/items |
### B1.1 Kill the button hairlines + focus rectangles (quick, high-impact)
Three changes, all small:
**(a) De-theme the owner-draw controls.** `SetWindowTheme(h, L"", L"")` strips the UxTheme styling from a control so Windows stops painting its themed border/edge behind your `WM_DRAWITEM`. Add the header/lib and call it on every owner-draw control right after you create it:
```cpp
#include <uxtheme.h>
#pragma comment(lib, "uxtheme.lib")
// after creating each owner-draw button (Record, Pin, Copy, Paste, Clear):
SetWindowTheme(hBtn, L"", L""); // remove themed edges
SetWindowSubclass(hBtn, BtnProc, 1, 0);
```
**(b) Stop the parent painting under the children.** Add `WS_CLIPCHILDREN` to the main window so its double-buffered `WM_PAINT` never bleeds a pixel into a child's rectangle:
```cpp
hMainWnd = CreateWindowExW(
WS_EX_TOPMOST,
L"WhisperDictationClass", L"Dictation",
WS_OVERLAPPEDWINDOW | WS_CLIPCHILDREN, // <- add WS_CLIPCHILDREN
CW_USEDEFAULT, CW_USEDEFAULT, 400, 340,
nullptr, nullptr, hInstance, nullptr);
```
**(c) Hide focus rectangles app-wide.** The dotted/thin rect that appears on whichever control has keyboard focus (often *Pinned* after you click it). Tell the UI-state machine to keep focus cues hidden — once, after the controls exist:
```cpp
SendMessageW(hMainWnd, WM_CHANGEUISTATE, MAKEWPARAM(UIS_SET, UISF_HIDEFOCUS), 0);
```
After (a)+(b)+(c), the only remaining stock chrome is the combo dropdown button, handled next.
### B1.2 Replace the comboboxes with a custom dropdown (removes the native button)
A combobox always owns its dropdown button — you can't draw it away. Replace each combo with a **flat owner-draw "select" button** (you already draw exactly this look in `DrawCombo`) that opens a **custom dark popup window**. No native button, no field edge, fully on-theme.
**Data model** (replace `g_modelComboPaths` usage as needed; keep a label list + selection per selector):
```cpp
std::vector<std::wstring> g_audioItems; int g_audioSel = 0;
std::vector<std::wstring> g_modelItems; int g_modelSel = 0; // parallel to g_modelComboPaths
```
Populate these in `RefreshAudioDevices` / `RefreshModelList` instead of `CB_ADDSTRING`. Create `ID_SEL_AUDIO` / `ID_SEL_MODEL` as `BS_OWNERDRAW` buttons (not comboboxes) and draw them with a field renderer:
```cpp
void DrawSelect(LPDRAWITEMSTRUCT d, const std::wstring& text) {
Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
Rect rc(d->rcItem.left, d->rcItem.top, d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
Rect field = rc; field.Inflate(-1,-1);
bool hover = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
FillRound(g, hover ? C_SURFACEHI : C_SURFACE, field, 9);
StrokeRound(g, C_BORDER, field, 9, 1.0f);
Font f(d->hDC, g_fUI);
RectF tb((REAL)field.X+10, (REAL)field.Y, (REAL)(field.Width-28), (REAL)field.Height);
DrawTextC(g, text.c_str(), f, C_TEXT, tb, StringAlignmentNear, StringAlignmentCenter);
int cx = field.GetRight()-16, cy = field.Y + field.Height/2; // our chevron, the only one now
Pen pen(C_TEXTDIM, 1.6f);
g.DrawLine(&pen, cx-4, cy-2, cx, cy+2); g.DrawLine(&pen, cx, cy+2, cx+4, cy-2);
}
// in WM_DRAWITEM:
// case ID_SEL_AUDIO: DrawSelect(d, g_audioItems.empty()?L"No devices":g_audioItems[g_audioSel]); return TRUE;
// case ID_SEL_MODEL: DrawSelect(d, g_modelItems.empty()?L"—":g_modelItems[g_modelSel]); return TRUE;
```
**The popup window** — a small borderless top-level you paint yourself; closes on pick or focus loss:
```cpp
struct PopupState { std::vector<std::wstring> items; int sel; int hot; HWND owner; int ctrlId; };
static PopupState g_pop;
LRESULT CALLBACK PopupProc(HWND h, UINT m, WPARAM w, LPARAM l) {
switch (m) {
case WM_MOUSEMOVE: {
int row = GET_Y_LPARAM(l) / 30;
if (row != g_pop.hot) { g_pop.hot = row; InvalidateRect(h, nullptr, FALSE); }
TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 }; TrackMouseEvent(&t);
return 0;
}
case WM_MOUSELEAVE: g_pop.hot = -1; InvalidateRect(h, nullptr, FALSE); return 0;
case WM_LBUTTONUP: {
int row = GET_Y_LPARAM(l) / 30;
if (row >= 0 && row < (int)g_pop.items.size())
PostMessageW(g_pop.owner, WM_APP_SELECT, (WPARAM)g_pop.ctrlId, (LPARAM)row);
DestroyWindow(h); return 0;
}
case WM_KILLFOCUS: DestroyWindow(h); return 0;
case WM_ERASEBKGND: return 1;
case WM_PAINT: {
PAINTSTRUCT ps; HDC hdc = BeginPaint(h, &ps);
RECT rc; GetClientRect(h, &rc);
HDC mem = CreateCompatibleDC(hdc);
HBITMAP bmp = CreateCompatibleBitmap(hdc, rc.right, rc.bottom);
HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
{
Graphics g(mem); g.SetSmoothingMode(SmoothingModeAntiAlias);
Rect all(0,0,rc.right,rc.bottom);
FillRound(g, C_SURFACE, all, 10); StrokeRound(g, C_BORDER, all, 10, 1.0f);
Font f(mem, g_fUI);
for (int i = 0; i < (int)g_pop.items.size(); ++i) {
Rect row(3, i*30+3, rc.right-6, 28);
if (i == g_pop.hot) FillRound(g, C_SURFACEHI, row, 7);
RectF tb((REAL)row.X+9,(REAL)row.Y,(REAL)row.Width-12,(REAL)row.Height);
DrawTextC(g, g_pop.items[i].c_str(), f, (i==g_pop.sel)?C_ACCENT:C_TEXT,
tb, StringAlignmentNear, StringAlignmentCenter);
}
}
BitBlt(hdc,0,0,rc.right,rc.bottom,mem,0,0,SRCCOPY);
SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
EndPaint(h, &ps); return 0;
}
}
return DefWindowProc(h, m, w, l);
}
void ShowSelectPopup(HWND owner, int ctrlId, const std::vector<std::wstring>& items, int sel) {
static bool reg = false;
if (!reg) { WNDCLASSEXW wc{ sizeof(wc) }; wc.lpfnWndProc = PopupProc; wc.hInstance = hInst;
wc.hCursor = LoadCursor(nullptr, IDC_ARROW); wc.lpszClassName = L"DictPopup";
RegisterClassExW(&wc); reg = true; }
g_pop = { items, sel, -1, owner, ctrlId };
RECT rc; GetWindowRect(GetDlgItem(owner, ctrlId), &rc);
int h = (int)items.size()*30 + 6, wdt = rc.right - rc.left;
HWND p = CreateWindowExW(WS_EX_TOOLWINDOW | WS_EX_TOPMOST, L"DictPopup", L"",
WS_POPUP, rc.left, rc.bottom+2, wdt, h, owner, nullptr, hInst, nullptr);
int corner = DWMWCP_ROUND; DwmSetWindowAttribute(p, DWMWA_WINDOW_CORNER_PREFERENCE, &corner, sizeof(corner));
ShowWindow(p, SW_SHOWNA); SetForegroundWindow(p); SetFocus(p);
}
```
Wire it up: clicking a selector opens the popup; the popup posts the chosen row back:
```cpp
// WM_COMMAND:
case ID_SEL_AUDIO: ShowSelectPopup(hWnd, ID_SEL_AUDIO, g_audioItems, g_audioSel); break;
case ID_SEL_MODEL: ShowSelectPopup(hWnd, ID_SEL_MODEL, g_modelItems, g_modelSel); break;
// new message handler:
case WM_APP_SELECT: { // #define WM_APP_SELECT (WM_USER + 5)
int ctrlId = (int)wParam, idx = (int)lParam;
if (ctrlId == ID_SEL_AUDIO) { g_audioSel = idx; g_config.capture_id = idx; }
else if (ctrlId == ID_SEL_MODEL && idx < (int)g_modelComboPaths.size()) {
g_modelSel = idx;
g_config.model_path = exe_dir() + "\\" + g_modelComboPaths[idx];
g_modelLoaded = false; g_modelOk = false; SetStatus(hWnd, L"Loading model…");
std::thread([]{ bool ok=g_tx.reload(g_config); g_modelOk=ok; g_modelLoaded=true; }).detach();
}
InvalidateRect(GetDlgItem(hWnd, ctrlId), nullptr, FALSE);
} break;
```
You can now delete the `CBS_*`/`WM_MEASUREITEM`/`DrawCombo` combo code and the `WM_CTLCOLORLISTBOX` handler. The selectors are now pixel-identical to your other controls with exactly one (your) chevron.
> Lighter alternative if you don't want a popup window yet: keep the combos but call `SetWindowTheme(hCombo, L"", L"")` — it flattens the dropdown button to a plain square. It's *less* boxy but the button is still there, so the popup-window route above is the real fix.
---
## B2. Transcription progress + time estimate
You don't have to guess the timing — **whisper.cpp reports real progress.** `whisper_full_params` has a `progress_callback` that fires repeatedly during inference with an `int` 0–100 (fraction of the audio processed). Feed that to a determinate progress bar + a live ETA, so "Transcribing…" becomes "Transcribing 1:40 of audio · 45% · ~9s left".
### B2.1 transcriber — expose progress + audio length
`transcriber.h` (public):
```cpp
using ProgressCb = std::function<void(int)>; // 0..100
void set_progress_callback(ProgressCb cb) { m_on_progress = std::move(cb); }
float audio_seconds() const { return m_audio_seconds.load(); }
void request_cancel() { m_abort = true; } // optional (B2.4)
```
`transcriber.h` (private):
```cpp
ProgressCb m_on_progress;
std::atomic<float> m_audio_seconds{0.0f};
std::atomic<bool> m_abort{false};
static void s_progress(struct whisper_context*, struct whisper_state*, int p, void* ud);
static bool s_abort(void* ud);
```
`transcriber.cpp` — the static trampolines and the `run_inference` hook:
```cpp
void Transcriber::s_progress(whisper_context*, whisper_state*, int p, void* ud) {
auto* self = static_cast<Transcriber*>(ud);
if (self && self->m_on_progress) self->m_on_progress(p);
}
bool Transcriber::s_abort(void* ud) {
auto* self = static_cast<Transcriber*>(ud);
return self && self->m_abort.load();
}
std::string Transcriber::run_inference(std::vector<float>& audio) {
if (!m_ctx) return "";
m_abort = false;
if (m_cfg.trim_silence) trim_silence(audio);
m_audio_seconds = (float)(audio.size() / (double)WHISPER_SAMPLE_RATE); // for the UI
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wp.print_progress = false; wp.print_realtime = false; wp.print_timestamps = false;
wp.no_timestamps = true; wp.translate = false;
wp.language = m_cfg.language.c_str();
wp.n_threads = m_cfg.n_threads;
wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true; wp.temperature = 0.0f;
wp.progress_callback = &Transcriber::s_progress; // <- live progress
wp.progress_callback_user_data = this;
wp.abort_callback = &Transcriber::s_abort; // <- optional cancel
wp.abort_callback_user_data = this;
std::string out;
if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
int n = whisper_full_n_segments(m_ctx);
for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
out = clean_text(out);
}
return out;
}
```
> If your `whisper.h` predates `abort_callback`, drop those two lines — `progress_callback` has been in whisper.cpp far longer and is what matters here.
### B2.2 main.cpp — progress state + ETA
```cpp
#define WM_APP_PROGRESS (WM_USER + 6)
std::atomic<int> g_progress{0};
DWORD g_busyStart = 0;
// at startup, next to set_result_callback:
g_tx.set_progress_callback([](int p){ PostMessage(hMainWnd, WM_APP_PROGRESS, (WPARAM)p, 0); });
```
Start the clock when transcription begins (HK_TOGGLE stop branch):
```cpp
} else { // was recording -> stop
g_busyStart = GetTickCount();
g_progress = 0;
g_tx.stop_and_transcribe();
SetStatus(hWnd, L"Transcribing…");
}
```
Receive progress and repaint the bar:
```cpp
case WM_APP_PROGRESS:
g_progress = (int)wParam;
InvalidateRect(hWnd, &g_vuRect, FALSE);
return 0;
```
ETA text (replace the `is_busy()` branch in `UpdateStatus`):
```cpp
} else if (g_tx.is_busy()) {
int p = g_progress.load();
float total = g_tx.audio_seconds();
float elapsed = (GetTickCount() - g_busyStart) / 1000.0f;
int mm = (int)total / 60, ss = (int)total % 60;
if (p >= 3) {
float est = elapsed * 100.0f / p; // projected total
float remain = est - elapsed; if (remain < 0) remain = 0;
swprintf_s(buf, L"Transcribing %d:%02d • %d%% • ~%ds left", mm, ss, p, (int)(remain + 0.5f));
} else {
swprintf_s(buf, L"Transcribing %d:%02d of audio…", mm, ss);
}
SetStatus(hwnd, buf);
}
```
### B2.3 Reuse the VU strip as a determinate progress bar
While recording the strip shows the VU; while transcribing it shows progress. Add a renderer and branch in `WM_PAINT`:
```cpp
void DrawProgress(Graphics& g, const RECT& r, float frac) {
Rect track(r.left, r.top, r.right - r.left, r.bottom - r.top);
FillRound(g, C_SURFACEHI, track, 4);
frac = frac < 0 ? 0 : (frac > 1 ? 1 : frac);
int w = (int)((r.right - r.left) * frac);
if (w > 0) { Rect fill(r.left, r.top, w, r.bottom - r.top); FillRound(g, C_ACCENT, fill, 4); }
}
```
```cpp
// in WM_PAINT, where you currently call DrawVU:
if (g_tx.is_busy()) DrawProgress(g, g_vuRect, g_progress.load() / 100.0f);
else DrawVU(g, g_vuRect, g_energy);
```
Keep it ticking between callbacks so the ETA counts down smoothly — in `WM_TIMER`, add:
```cpp
else if (g_tx.is_busy()) {
InvalidateRect(hWnd, &g_vuRect, FALSE); // (UpdateStatus already runs each tick below)
}
```
That's it: a moving bar, a percentage, and a shrinking "~Ns left" — the user can see it's alive and roughly how long is left. The estimate self-corrects as real progress arrives (the first few percent are rougher; it tightens quickly).
### B2.4 (Optional) Cancel a long transcription
You added `request_cancel()` + the abort callback in B2.1. Hook it so that pressing the button **while busy** aborts instead of being ignored, and treat the empty result as "Cancelled":
```cpp
// top of the HK_TOGGLE handler:
if (g_tx.is_busy()) { g_tx.request_cancel(); SetStatus(hWnd, L"Cancelling…"); break; }
```
`whisper_full` returns promptly with no/aborted segments → your existing `WM_APP_RESULT` empty-string path shows "No speech detected"; change that label to "Cancelled" when a cancel was requested if you want to distinguish them.
---
## B3. Build & test
- **New link deps:** `uxtheme.lib` (added via `#pragma comment` in B1.1). GDI+ is already linked.
- **New files:** none required for B2; B1.2 adds the popup window proc inside `main.cpp` (no new translation unit).
- **Rebuild:** `cmake --build build --config Release` as usual.
**Verify:**
- [ ] No hairline around *Pinned*; no left/top lines on Copy/Paste/Clear; focus no longer draws a dotted rect.
- [ ] Mic + model selectors show a single (your) chevron, open a dark rounded popup, and selecting reloads the model / switches device.
- [ ] Record a ~1–2 min clip → the strip fills as a progress bar, the status shows `%` and a shrinking `~Ns left`, and it completes (no static "Transcribing…").
- [ ] (If added) pressing the button mid-transcription cancels promptly.
**Headless regression (extends `tests/test_core.cpp` from `FINDINGS-FIXES-TESTS.md`):**
- Set a progress callback that records the max value seen; assert it reaches ~100 for `samples/jfk.wav`, and that callbacks arrive in non-decreasing order. This locks in that progress reporting keeps working across whisper.cpp upgrades.
```cpp
int last = -1, maxp = 0; bool monotonic = true;
t.set_progress_callback([&](int p){ if (p < last) monotonic = false; last = p; if (p > maxp) maxp = p; });
t.transcribe_sync(audio);
CHECK(monotonic, "progress is non-decreasing");
CHECK(maxp >= 95, "progress reaches ~100%");
```
---
*Apply order: B1.1 (5 min, instant visual win) → B2 (progress, the big UX gain) → B1.2 (custom dropdowns) → optional B2.4 cancel. Each is independent and safe to ship on its own.*
CMakeLists.txt
+60 -22
View File
@@ -18,25 +18,32 @@ option(WHISPER_NO_AVX2 "whisper: disable AVX2" OFF)
option(WHISPER_NO_FMA "whisper: disable FMA" OFF)
option(WHISPER_NO_F16C "whisper: disable F16C" OFF)
# ----------------------------
# SDL2
# ----------------------------
if(NOT SDL2_DIR)
set(SDL2_DIR "${CMAKE_CURRENT_SOURCE_DIR}/SDL2-mingw/cmake")
endif()
find_package(SDL2 REQUIRED)
string(STRIP "${SDL2_LIBRARIES}" SDL2_LIBRARIES)
include_directories(${SDL2_INCLUDE_DIRS})
# GGML
add_subdirectory(ggml)
# ----------------------------
# Whisper (ONLY dependency layer)
# ----------------------------
# IMPORTANT:
# We no longer build ggml separately.
# whisper/ must contain its own CMakeLists.txt (modern whisper.cpp layout)
add_subdirectory(whisper)
# Whisper
# We need to handle include paths for whisper.
# whisper/src/CMakeLists.txt expects ../include/whisper.h
add_subdirectory(whisper/src)
# Common Library (needed by win-dictation) - optional if files don't exist
# ----------------------------
# Common Library (optional legacy utilities)
# ----------------------------
if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/common/common.h")
set(COMMON_TARGET common)
add_library(${COMMON_TARGET} STATIC
common/common.h
common/common.cpp
@@ -47,32 +54,39 @@ if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/common/common.h")
common/grammar-parser.h
common/grammar-parser.cpp
)
target_include_directories(${COMMON_TARGET} PUBLIC
${CMAKE_CURRENT_SOURCE_DIR}/common
${CMAKE_CURRENT_SOURCE_DIR}/whisper/include
${CMAKE_CURRENT_SOURCE_DIR}/ggml/include
)
# Link against whisper target only (no ggml exposure)
target_link_libraries(${COMMON_TARGET} PRIVATE whisper)
set(COMMON_SDL_TARGET common-sdl)
add_library(${COMMON_SDL_TARGET} STATIC
common/common-sdl.h
common/common-sdl.cpp
)
target_include_directories(${COMMON_SDL_TARGET} PUBLIC
${CMAKE_CURRENT_SOURCE_DIR}/common
${SDL2_INCLUDE_DIRS}
)
target_link_libraries(${COMMON_SDL_TARGET} PRIVATE ${SDL2_LIBRARIES})
else()
# Create empty common libraries if files don't exist
# Empty fallback targets
add_library(common INTERFACE)
add_library(common-sdl INTERFACE)
target_include_directories(common INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}/whisper/include)
target_include_directories(common-sdl INTERFACE ${SDL2_INCLUDE_DIRS})
endif()
# Win Dictation Executable
# ----------------------------
# Main executable
# ----------------------------
add_executable(win-dictation WIN32
src/main.cpp
src/transcriber.cpp
@@ -90,13 +104,28 @@ target_link_libraries(win-dictation PRIVATE
target_include_directories(win-dictation PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/src
${CMAKE_CURRENT_SOURCE_DIR}/common
${CMAKE_CURRENT_SOURCE_DIR}/whisper/include
${CMAKE_CURRENT_SOURCE_DIR}/ggml/include
${SDL2_INCLUDE_DIR}
)
target_compile_definitions(win-dictation PRIVATE UNICODE _UNICODE)
# ----------------------------
# Tests
# ----------------------------
add_executable(test-core
tests/test_core.cpp
src/transcriber.cpp
src/transcriber.h
)
target_link_libraries(test-core PRIVATE whisper ${SDL2_LIBRARIES})
target_include_directories(test-core PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/src
${CMAKE_CURRENT_SOURCE_DIR}/common
)
target_compile_definitions(test-core PRIVATE UNICODE _UNICODE)
# ----------------------------
# MSVC optimisations
# ----------------------------
if(MSVC)
target_compile_options(win-dictation PRIVATE
$<$<CONFIG:Release>:/O2 /GL>
@@ -106,14 +135,23 @@ if(MSVC)
)
endif()
# Copy SDL2.dll to output directory
add_custom_command(TARGET win-dictation POST_BUILD
# ----------------------------
# Post build: runtime assets
# ----------------------------
if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/deps/SDL2-2.28.5/lib/x64/SDL2.dll")
add_custom_command(TARGET win-dictation POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different
"${CMAKE_CURRENT_SOURCE_DIR}/deps/SDL2-2.28.5/lib/x64/SDL2.dll"
$<TARGET_FILE_DIR:win-dictation>
)
elseif(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/release/WinDictation/SDL2.dll")
add_custom_command(TARGET win-dictation POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different
"${CMAKE_CURRENT_SOURCE_DIR}/release/WinDictation/SDL2.dll"
$<TARGET_FILE_DIR:win-dictation>
)
)
endif()
# Copy models directory to output
add_custom_command(TARGET win-dictation POST_BUILD
COMMAND ${CMAKE_COMMAND} -E make_directory $<TARGET_FILE_DIR:win-dictation>/models
COMMAND ${CMAKE_COMMAND} -E copy_directory
FINDINGS-FIXES-TESTS.md
+425
View File
@@ -0,0 +1,425 @@
# Win Dictation — Findings, Fixes & Test Guide
A consolidated record of everything diagnosed and changed while turning the slow,
crashing build into a fast, stable push-to-talk dictation tool. Four parts:
1. **Findings** — what was actually wrong, in the order we discovered it.
2. **Fixes required** — the concrete code changes, by file.
3. **Reasoning** — why each fix is correct (the non-obvious calls).
4. **How to build & run tests** — making the core testable, plus regression tests for every bug here.
Target machine: Dell, Intel Core i5 7th-gen (2 cores / 4 threads), 16 GB RAM, CPU-only.
Stack: native C++ / Win32 + SDL2 + whisper.cpp (`examples/win-dictation`).
---
## Part 1 — Findings
### F1. The original app was slow because of its *architecture*, not its model
The first build used whisper.cpp's live-streaming pattern: it re-ran `whisper_full` on a
rolling 5–6 second window every ~0.4 s (`length_ms` / `step_ms` in `worker_loop`). To keep
up, the CPU must transcribe ~6 s of audio in well under 1 s — i.e. **>6× real-time**.
- The README's "10–15× real-time" numbers were measured on a **24-thread** box.
- A 2-core i5-7th-gen does tiny.en at roughly **2–4× real-time**.
So each 6 s window took ~1.5–2.5 s while a new one was demanded every 0.4 s → the buffer
backed up without bound, audio was re-transcribed in overlapping chunks (duplicated/garbled
text), the CPU pegged, and 4 Whisper threads on 2 physical cores starved the UI. tiny.en was
already the right model — **the streaming design was the bottleneck.**
**Conclusion:** for dictation (not live captioning), switch to **batch / push-to-talk**:
buffer audio cheaply while recording, run `whisper_full` exactly **once** on Stop.
### F2. Build 1 crashed on Stop — null Whisper context
After the batch rewrite, the app recorded fine (VU moved) but crashed the instant you hit
Stop. Two defects combined:
1. The preload thread set `g_modelLoaded = true` **even when `preload()` returned false**
(model file not found), so the UI showed "Ready" and let you record with no model.
2. `transcribe_worker()` called `whisper_reset_timings(m_ctx)` **before** the `if (m_ctx)`
check. With `m_ctx == nullptr` that's an access violation. Recording never touches Whisper
— which is exactly why capture/VU worked and only Stop crashed.
The empty model dropdown in the first screenshot was the tell: no model files were found at
the new exe-relative path, so `preload()` failed silently.
### F3. Build 1 — window wouldn't resize, and looked boxy
- The style was `WS_POPUP | WS_CAPTION | WS_SYSMENU` — **no sizing border** — and there was
**no `WM_SIZE` handler**, so controls never reflowed. Frozen at ~360×200, Copy/Paste cut off.
- The gray 3-D system buttons + chunky title bar were the "boxy Windows" look the brief
wanted to avoid.
### F4. Build 2 — startup data race (found and fixed by the user)
`RefreshModelList()` calls `CB_SETCURSEL`, which fires `CBN_SELCHANGE`. That spawned a
`reload()` background thread that raced the initial `preload()` thread — both writing
`m_cfg` (a `std::string`) and `m_ctx` at once. The corrupted context pointer made
`whisper_full` crash ~15 s into processing. Fixed with a `g_initializing` guard (skip the
combo handler during startup) and an `m_cfg_mtx` mutex around `m_cfg`/`m_ctx` writes.
### F5. Build 3 — the "crash after transcribing" was **not a crash**
Reported: "crashes after transcribing; relaunching the exe shows the last transcription, and
a second run replaced the first instead of appending."
That behavior is **impossible for a crashed process** — a dead process can't remember the
last transcription (nothing is persisted to disk/registry), and a *fresh* launch couldn't
show it. The only explanation: **the original process never died.**
What actually happened:
1. On a successful result, `WM_APP_RESULT` ran `ShowWindow(hWnd, SW_HIDE)` in the auto-paste
branch → the window **vanished** (looked like a crash).
2. Double-clicking the `.exe` again hit the **single-instance guard**, which
`PostMessage(WM_APP_SHOW)` to the existing **hidden** window → it re-appeared, still holding
the last transcription.
3. `SetDlgItemTextW` **replaced** the edit text, so run #2 overwrote run #1.
Also: clicking the **Record button** (vs. the global hotkey) set `g_prevForeground` to the
dictation window *itself*, so it hid and "pasted" into its own read-only box — nothing landed
in a useful app.
10-second confirmation: when it "crashes," the tray icon is still present and
`win-dictation.exe` is still in Task Manager.
### F6. The test harness is stale and never built
`src/test-audio.cpp` calls `m_transcriber.init(...)` and `is_using_gpu()` — methods that the
batch rewrite **removed** (`transcriber.h` now exposes `preload`/`reload`, no `init`). So it
won't compile against the current code. And the **root** `CMakeLists.txt` (the one `build.ps1`
actually uses) defines only the `win-dictation` target — the `test-audio` target lives only in
the unused `src/CMakeLists.txt`. So there are currently **no working tests**. Part 4 fixes this.
---
## Part 2 — Fixes required
Status legend: ✅ done in current code · ⬜ still to apply.
### `transcriber.cpp` / `transcriber.h`
- ✅ **Batch architecture** — `start_recording()` buffers audio cheaply; `stop_and_transcribe()`
runs `whisper_full` once on a worker thread; result delivered via `PostMessage` (F1).
- ✅ **Null-context guard** in `transcribe_worker` — `if (!m_ctx) { m_busy=false; if(m_on_result) m_on_result(""); return; }`,
and the `whisper_reset_timings` call removed (F2).
- ✅ **`reload()`** frees + re-inits under a mutex; **`m_cfg_mtx`** protects `m_cfg`/`m_ctx`;
**`threads()`** getter (F4 + thread-count display).
- ⬜ **Refactor for testability** — extract the inference core so it can be called
synchronously from a test (see Part 4):
```cpp
// transcriber.h (public)
std::string transcribe_sync(std::vector<float> audio); // headless / tests
// transcriber.h (private)
std::string run_inference(std::vector<float>& audio);
```
```cpp
// transcriber.cpp
std::string Transcriber::run_inference(std::vector<float>& audio) {
if (!m_ctx) return "";
if (m_cfg.trim_silence) trim_silence(audio);
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wp.print_progress = wp.print_realtime = wp.print_timestamps = false;
wp.no_timestamps = true; wp.translate = false;
wp.language = m_cfg.language.c_str();
wp.n_threads = m_cfg.n_threads;
wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true;
wp.temperature = 0.0f;
std::string out;
if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
int n = whisper_full_n_segments(m_ctx);
for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
out = clean_text(out);
}
return out;
}
void Transcriber::transcribe_worker(std::vector<float> audio) {
std::string out = run_inference(audio);
m_busy = false;
if (m_on_result) m_on_result(out);
}
std::string Transcriber::transcribe_sync(std::vector<float> audio) {
return run_inference(audio); // assumes preload() already succeeded
}
```
### `main.cpp`
- ✅ **Honest load state** — separate `g_modelOk` atomic; record is gated and shows
"Model not found: …\models\ggml-tiny.en.bin" instead of crashing (F2).
- ✅ **Resizable** — `WS_OVERLAPPEDWINDOW` (has `WS_THICKFRAME`), `WM_SIZE → LayoutControls`,
`WM_GETMINMAXINFO` min size (F3).
- ✅ **Startup race guard** — `g_initializing` skips the model-combo handler during startup (F4).
- ⬜ **`WM_APP_RESULT` rewrite** — the F5 fix: **don't hide**, **append** (not replace),
clipboard = latest utterance, paste only into a *different* valid window:
```cpp
case WM_APP_RESULT:
{
std::string* res = (std::string*)wParam;
if (res && !res->empty()) {
HWND hEdit = GetDlgItem(hWnd, ID_EDIT_TEXT);
int len = GetWindowTextLengthW(hEdit);
std::wstring cur;
if (len > 0) { cur.resize(len + 1); int g = GetWindowTextW(hEdit, &cur[0], len + 1); cur.resize(g); }
std::wstring combined = cur;
if (!combined.empty()) combined += L" ";
combined += to_w(*res);
SetWindowTextW(hEdit, combined.c_str());
SendMessageW(hEdit, EM_SETSEL, (WPARAM)combined.size(), (LPARAM)combined.size());
SendMessageW(hEdit, EM_SCROLLCARET, 0, 0);
UpdatePlaceholder(hWnd);
SetClipboardTextUtf8(hWnd, *res); // clipboard = just the latest utterance
bool pasted = false;
if (g_autoPaste && g_prevForeground &&
g_prevForeground != hWnd && IsWindow(g_prevForeground)) {
PasteIntoWindow(g_prevForeground);
pasted = true;
}
SetStatus(hWnd, pasted ? L"Pasted ✓" : L"Copied ✓");
} else {
SetStatus(hWnd, L"No speech detected");
}
delete res;
}
break;
```
- ⬜ **Clear button** — `ID_BTN_CLEAR` is handled in `WM_COMMAND` but no button is created in
the modern layout, so the (now appending) transcript can't be cleared. Create it owner-drawn,
place it in `LayoutControls`, and call `UpdatePlaceholder(hWnd)` after clearing.
- ⬜ **Optional `g_autoHide`** (default `false`) — only if you want the window to tuck away after
a successful paste into another app: `if (pasted && g_autoHide) ShowWindow(hWnd, SW_HIDE);`
### Modern UI (separate doc)
The flat dark restyle (GDI+ owner-drawn buttons, rounded panel, custom VU, dark caption +
rounded corners) is fully specified in **`MODERN-UI-AND-FIXES.md`**, Part 2.
---
## Part 3 — Reasoning
**Why batch beats streaming here (F1).** Streaming only wins if you need words *as you speak*
(live captioning). Dictation doesn't: you speak a sentence, then want the text. Batch removes
the real-time deadline entirely — Whisper processes each second of audio exactly once, at its
own pace, after you stop. That means: near-zero CPU while speaking (just buffering), predictable
"a few seconds after Stop" latency that **doesn't grow with clip length**, no overlap
re-processing, and better accuracy (full context, no chunk-boundary word splits). On 2 cores
this is the difference between unusable and snappy.
**Why threads = physical cores (2), not logical (4).** Whisper's matmuls are memory-bandwidth
bound. Two extra hyperthreads share the same execution ports and cache, so they add little
throughput while stealing cycles from the UI/audio threads (janky window, laggy VU). Two
threads leaves headroom for a responsive UI.
**Why the Stop crash was a null deref (F2), and why the guard is the right fix.** Recording is
pure SDL + a `std::vector` append — it never calls into Whisper, which is why only Stop crashed.
`whisper_reset_timings(nullptr)` dereferences the context. The guard makes the worker a no-op
when no model is loaded; the separate `g_modelOk` flag makes that state *visible* (a MessageBox
with the path) instead of letting you walk into it. Defensive + diagnostic.
**Why the "crash after transcribing" was actually a hide (F5) — the logic is conclusive.** A
crashed process loses all memory. There is no on-disk persistence in this app. Therefore a
re-launched process **cannot** display the previous transcription, and a second run **cannot**
replace text in "the same box." The observed behavior (prior text reappears; second replaces
first) is only possible if it's the *same* live process — which means it hid, and the
single-instance guard re-showed it. This is why "is the process still in Task Manager?" is the
decisive test, not "did the window disappear?"
**Why clipboard = latest utterance but the box appends.** Two different jobs. The on-screen box
is your running log (you want history). The clipboard is what you paste into another app — you
want *just what you last said*, not the whole session. Splitting them gives both.
**Why paste must exclude our own window.** Auto-paste replays Ctrl+V into the foreground window
captured before the popup. If you triggered recording by clicking the Record button, that
"previous" window is the dictation app itself — pasting into its read-only box is a no-op and
(with the old hide) made the app appear to swallow the text. Restricting to
`g_prevForeground != hWnd && IsWindow(...)` makes the global-hotkey workflow the reliable path
and the button a safe "copy only."
---
## Part 4 — How to build & run tests
The goal: catch the bugs above automatically, without a human watching a window. UI behavior
(resize, hide, paste) needs a short **manual** checklist, but the **core** (model load,
transcription, null-safety, text append) can and should be tested headlessly.
### 4.1 Make the core testable
Apply the `run_inference` / `transcribe_sync` refactor from Part 2 so a test can feed a buffer
and get text synchronously. Also extract the append rule as a pure function so it can be tested
with zero Win32:
```cpp
// text_util.h (new, tiny, UI-free)
#pragma once
#include <string>
inline std::wstring append_transcript(const std::wstring& cur, const std::wstring& add) {
if (add.empty()) return cur;
if (cur.empty()) return add;
return cur + L" " + add;
}
```
Use it in `WM_APP_RESULT` (`combined = append_transcript(cur, to_w(*res));`) so the UI and the
test exercise the *same* rule.
### 4.2 The test program (`tests/test_core.cpp`)
Replaces the stale `src/test-audio.cpp`. Compiles against the **current** API and tests the real
code path (`preload` + `transcribe_sync`).
```cpp
#include "transcriber.h"
#include "text_util.h"
#include <cstdio>
#include <cstdint>
#include <fstream>
#include <vector>
#include <string>
#include <cctype>
static int g_fail = 0;
#define CHECK(cond, msg) do { if (!(cond)) { printf(" FAIL: %s\n", msg); ++g_fail; } \
else printf(" ok: %s\n", msg); } while(0)
// Minimal 16-bit PCM mono WAV loader -> float [-1,1]. Assumes 16 kHz mono (whisper.cpp samples are).
static bool load_wav(const std::string& path, std::vector<float>& out) {
std::ifstream f(path, std::ios::binary);
if (!f) return false;
char hdr[44];
f.read(hdr, 44);
if (std::string(hdr, 4) != "RIFF") return false;
std::vector<int16_t> pcm((std::istreambuf_iterator<char>(f)), {}); // crude: rest of file
out.clear(); out.reserve(pcm.size());
for (int16_t s : pcm) out.push_back(s / 32768.0f);
return !out.empty();
}
static std::string lower(std::string s) { for (char& c : s) c = (char)tolower((unsigned char)c); return s; }
int main(int argc, char** argv) {
std::string model = (argc > 1) ? argv[1] : "models/ggml-tiny.en.bin";
std::string wav = (argc > 2) ? argv[2] : "samples/jfk.wav";
// --- Pure logic: append rule (no model needed) ---
CHECK(append_transcript(L"", L"hello") == L"hello", "append: empty + a");
CHECK(append_transcript(L"hello", L"world") == L"hello world", "append: a + b spaced");
CHECK(append_transcript(L"hello", L"") == L"hello", "append: a + empty");
// --- Regression F2: bad model path must NOT crash, returns "" ---
{
Transcriber t; WhisperConfig bad; bad.model_path = "models\\does-not-exist.bin";
bool ok = t.preload(bad);
CHECK(!ok, "bad model path: preload returns false");
std::vector<float> a(16000, 0.0f); // 1 s of silence
std::string r = t.transcribe_sync(a); // must be a safe no-op
CHECK(r.empty(), "bad model path: transcribe_sync returns empty, no crash");
}
// --- Happy path: real model + known clip -> non-empty, expected words ---
{
Transcriber t; WhisperConfig cfg; cfg.model_path = model; cfg.n_threads = 4;
bool ok = t.preload(cfg);
CHECK(ok, "model loads");
if (ok) {
std::vector<float> audio;
bool loaded = load_wav(wav, audio);
CHECK(loaded, "wav loads");
if (loaded) {
std::string text = lower(t.transcribe_sync(audio));
CHECK(!text.empty(), "transcription is non-empty");
// jfk.wav: "...ask not what your country can do for you..."
CHECK(text.find("country") != std::string::npos, "transcription contains 'country'");
}
}
}
// --- Edge: sub-300ms / empty audio -> "" , no crash ---
{
Transcriber t; WhisperConfig cfg; cfg.model_path = model;
if (t.preload(cfg)) {
std::vector<float> tiny(100, 0.1f);
std::string r = t.transcribe_sync(tiny);
CHECK(true, "short audio did not crash"); // reaching here = no crash
}
}
printf("\n%s (%d failure%s)\n", g_fail ? "TESTS FAILED" : "ALL TESTS PASSED",
g_fail, g_fail == 1 ? "" : "s");
return g_fail ? 1 : 0;
}
```
> The WAV loader above is deliberately minimal (good enough for whisper.cpp's 16 kHz mono
> `samples/*.wav`). If you test arbitrary WAVs, parse the `fmt `/`data` chunks properly.
### 4.3 Build the tests (root `CMakeLists.txt`)
The active build file builds only `win-dictation`. Add a test target next to it:
```cmake
# --- tests ---
add_executable(test-core
tests/test_core.cpp
src/transcriber.cpp
src/transcriber.h
)
target_link_libraries(test-core PRIVATE whisper ${SDL2_LIBRARIES})
target_include_directories(test-core PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/src
${CMAKE_CURRENT_SOURCE_DIR}/common
)
target_compile_definitions(test-core PRIVATE UNICODE _UNICODE)
```
(`transcriber.cpp` references SDL symbols even though the test never opens a device, so the test
still links `${SDL2_LIBRARIES}`.)
Build and run:
```powershell
cmake --build build --config Release --target test-core
# run from a dir where SDL2.dll + models/ + samples/ are reachable:
cd build\bin\Release
copy ..\..\..\deps\SDL2-2.28.5\lib\x64\SDL2.dll . # if not already beside the exe
.\test-core.exe models\ggml-tiny.en.bin ..\..\..\samples\jfk.wav
```
Exit code `0` = all passed (usable in CI / a pre-commit hook).
### 4.4 What each test guards
| Test | Guards against |
|---|---|
| append rule (3 cases) | F5 regression — replace-instead-of-append |
| bad model path → false + empty | F2 — null-context crash on Stop |
| model loads / clip → non-empty, has "country" | core transcription works end to end |
| short/empty audio → no crash | the `< 0.3 s` guard + null-safety |
### 4.5 Manual UI checklist (can't be unit-tested)
- [ ] **Not a crash:** after Stop, the window **stays visible**; if it ever vanishes, the tray
icon / `win-dictation.exe` in Task Manager confirms whether it's hiding vs. truly gone.
- [ ] **Append:** two dictations in a row → second is appended after the first, view scrolls down.
- [ ] **Clipboard:** after a dictation, paste into Notepad → only the *latest* utterance appears.
- [ ] **Auto-paste:** focus Notepad, press the global hotkey, speak, press it again → text lands
in Notepad. Triggering via the Record button instead → "Copied ✓" (no self-paste).
- [ ] **Resize:** drag edges; transcript grows; min size respected.
- [ ] **Performance:** a ~10 s clip transcribes in a few seconds and the UI stays responsive;
latency does **not** grow with longer recordings.
### 4.6 Performance smoke test (optional)
`test-core` can also print timing — wrap `transcribe_sync` in `std::chrono::steady_clock` and
assert `elapsed < audio_seconds * k` for a sanity bound (e.g. `k = 1.0`, i.e. faster than
real-time on tiny.en). Useful to catch a future regression that quietly reintroduces the
streaming-style slowdown.
---
## Part 5 — Status & remaining work
**Verified fixed:** F1 (batch), F2 (null-context crash), F3 (resize), F4 (startup race).
**Apply next (⬜ in Part 2):**
1. `run_inference` / `transcribe_sync` refactor (unblocks tests).
2. `WM_APP_RESULT` rewrite (the F5 fix — no-hide + append + safe paste). **Highest priority** —
it's what's making the app *look* like it crashes.
3. Add the Clear button (needed now that text appends).
4. Add `tests/test_core.cpp` + the CMake target; run it.
**Then re-run** the Part 4 manual checklist. Expected result: the window no longer disappears,
transcriptions append, the clipboard holds the latest utterance, and `test-core.exe` exits `0`.
FINDINGS-FIXES-TESTS2.md
+425
View File
@@ -0,0 +1,425 @@
# Win Dictation — Findings, Fixes & Test Guide
A consolidated record of everything diagnosed and changed while turning the slow,
crashing build into a fast, stable push-to-talk dictation tool. Four parts:
1. **Findings** — what was actually wrong, in the order we discovered it.
2. **Fixes required** — the concrete code changes, by file.
3. **Reasoning** — why each fix is correct (the non-obvious calls).
4. **How to build & run tests** — making the core testable, plus regression tests for every bug here.
Target machine: Dell, Intel Core i5 7th-gen (2 cores / 4 threads), 16 GB RAM, CPU-only.
Stack: native C++ / Win32 + SDL2 + whisper.cpp (`examples/win-dictation`).
---
## Part 1 — Findings
### F1. The original app was slow because of its *architecture*, not its model
The first build used whisper.cpp's live-streaming pattern: it re-ran `whisper_full` on a
rolling 5–6 second window every ~0.4 s (`length_ms` / `step_ms` in `worker_loop`). To keep
up, the CPU must transcribe ~6 s of audio in well under 1 s — i.e. **>6× real-time**.
- The README's "10–15× real-time" numbers were measured on a **24-thread** box.
- A 2-core i5-7th-gen does tiny.en at roughly **2–4× real-time**.
So each 6 s window took ~1.5–2.5 s while a new one was demanded every 0.4 s → the buffer
backed up without bound, audio was re-transcribed in overlapping chunks (duplicated/garbled
text), the CPU pegged, and 4 Whisper threads on 2 physical cores starved the UI. tiny.en was
already the right model — **the streaming design was the bottleneck.**
**Conclusion:** for dictation (not live captioning), switch to **batch / push-to-talk**:
buffer audio cheaply while recording, run `whisper_full` exactly **once** on Stop.
### F2. Build 1 crashed on Stop — null Whisper context
After the batch rewrite, the app recorded fine (VU moved) but crashed the instant you hit
Stop. Two defects combined:
1. The preload thread set `g_modelLoaded = true` **even when `preload()` returned false**
(model file not found), so the UI showed "Ready" and let you record with no model.
2. `transcribe_worker()` called `whisper_reset_timings(m_ctx)` **before** the `if (m_ctx)`
check. With `m_ctx == nullptr` that's an access violation. Recording never touches Whisper
— which is exactly why capture/VU worked and only Stop crashed.
The empty model dropdown in the first screenshot was the tell: no model files were found at
the new exe-relative path, so `preload()` failed silently.
### F3. Build 1 — window wouldn't resize, and looked boxy
- The style was `WS_POPUP | WS_CAPTION | WS_SYSMENU` — **no sizing border** — and there was
**no `WM_SIZE` handler**, so controls never reflowed. Frozen at ~360×200, Copy/Paste cut off.
- The gray 3-D system buttons + chunky title bar were the "boxy Windows" look the brief
wanted to avoid.
### F4. Build 2 — startup data race (found and fixed by the user)
`RefreshModelList()` calls `CB_SETCURSEL`, which fires `CBN_SELCHANGE`. That spawned a
`reload()` background thread that raced the initial `preload()` thread — both writing
`m_cfg` (a `std::string`) and `m_ctx` at once. The corrupted context pointer made
`whisper_full` crash ~15 s into processing. Fixed with a `g_initializing` guard (skip the
combo handler during startup) and an `m_cfg_mtx` mutex around `m_cfg`/`m_ctx` writes.
### F5. Build 3 — the "crash after transcribing" was **not a crash**
Reported: "crashes after transcribing; relaunching the exe shows the last transcription, and
a second run replaced the first instead of appending."
That behavior is **impossible for a crashed process** — a dead process can't remember the
last transcription (nothing is persisted to disk/registry), and a *fresh* launch couldn't
show it. The only explanation: **the original process never died.**
What actually happened:
1. On a successful result, `WM_APP_RESULT` ran `ShowWindow(hWnd, SW_HIDE)` in the auto-paste
branch → the window **vanished** (looked like a crash).
2. Double-clicking the `.exe` again hit the **single-instance guard**, which
`PostMessage(WM_APP_SHOW)` to the existing **hidden** window → it re-appeared, still holding
the last transcription.
3. `SetDlgItemTextW` **replaced** the edit text, so run #2 overwrote run #1.
Also: clicking the **Record button** (vs. the global hotkey) set `g_prevForeground` to the
dictation window *itself*, so it hid and "pasted" into its own read-only box — nothing landed
in a useful app.
10-second confirmation: when it "crashes," the tray icon is still present and
`win-dictation.exe` is still in Task Manager.
### F6. The test harness is stale and never built
`src/test-audio.cpp` calls `m_transcriber.init(...)` and `is_using_gpu()` — methods that the
batch rewrite **removed** (`transcriber.h` now exposes `preload`/`reload`, no `init`). So it
won't compile against the current code. And the **root** `CMakeLists.txt` (the one `build.ps1`
actually uses) defines only the `win-dictation` target — the `test-audio` target lives only in
the unused `src/CMakeLists.txt`. So there are currently **no working tests**. Part 4 fixes this.
---
## Part 2 — Fixes required
Status legend: ✅ done in current code · ⬜ still to apply.
### `transcriber.cpp` / `transcriber.h`
- ✅ **Batch architecture** — `start_recording()` buffers audio cheaply; `stop_and_transcribe()`
runs `whisper_full` once on a worker thread; result delivered via `PostMessage` (F1).
- ✅ **Null-context guard** in `transcribe_worker` — `if (!m_ctx) { m_busy=false; if(m_on_result) m_on_result(""); return; }`,
and the `whisper_reset_timings` call removed (F2).
- ✅ **`reload()`** frees + re-inits under a mutex; **`m_cfg_mtx`** protects `m_cfg`/`m_ctx`;
**`threads()`** getter (F4 + thread-count display).
- ⬜ **Refactor for testability** — extract the inference core so it can be called
synchronously from a test (see Part 4):
```cpp
// transcriber.h (public)
std::string transcribe_sync(std::vector<float> audio); // headless / tests
// transcriber.h (private)
std::string run_inference(std::vector<float>& audio);
```
```cpp
// transcriber.cpp
std::string Transcriber::run_inference(std::vector<float>& audio) {
if (!m_ctx) return "";
if (m_cfg.trim_silence) trim_silence(audio);
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wp.print_progress = wp.print_realtime = wp.print_timestamps = false;
wp.no_timestamps = true; wp.translate = false;
wp.language = m_cfg.language.c_str();
wp.n_threads = m_cfg.n_threads;
wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true;
wp.temperature = 0.0f;
std::string out;
if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
int n = whisper_full_n_segments(m_ctx);
for (int i = 0; i < n; ++i) { const char* t = whisper_full_get_segment_text(m_ctx, i); if (t) out += t; }
out = clean_text(out);
}
return out;
}
void Transcriber::transcribe_worker(std::vector<float> audio) {
std::string out = run_inference(audio);
m_busy = false;
if (m_on_result) m_on_result(out);
}
std::string Transcriber::transcribe_sync(std::vector<float> audio) {
return run_inference(audio); // assumes preload() already succeeded
}
```
### `main.cpp`
- ✅ **Honest load state** — separate `g_modelOk` atomic; record is gated and shows
"Model not found: …\models\ggml-tiny.en.bin" instead of crashing (F2).
- ✅ **Resizable** — `WS_OVERLAPPEDWINDOW` (has `WS_THICKFRAME`), `WM_SIZE → LayoutControls`,
`WM_GETMINMAXINFO` min size (F3).
- ✅ **Startup race guard** — `g_initializing` skips the model-combo handler during startup (F4).
- ⬜ **`WM_APP_RESULT` rewrite** — the F5 fix: **don't hide**, **append** (not replace),
clipboard = latest utterance, paste only into a *different* valid window:
```cpp
case WM_APP_RESULT:
{
std::string* res = (std::string*)wParam;
if (res && !res->empty()) {
HWND hEdit = GetDlgItem(hWnd, ID_EDIT_TEXT);
int len = GetWindowTextLengthW(hEdit);
std::wstring cur;
if (len > 0) { cur.resize(len + 1); int g = GetWindowTextW(hEdit, &cur[0], len + 1); cur.resize(g); }
std::wstring combined = cur;
if (!combined.empty()) combined += L" ";
combined += to_w(*res);
SetWindowTextW(hEdit, combined.c_str());
SendMessageW(hEdit, EM_SETSEL, (WPARAM)combined.size(), (LPARAM)combined.size());
SendMessageW(hEdit, EM_SCROLLCARET, 0, 0);
UpdatePlaceholder(hWnd);
SetClipboardTextUtf8(hWnd, *res); // clipboard = just the latest utterance
bool pasted = false;
if (g_autoPaste && g_prevForeground &&
g_prevForeground != hWnd && IsWindow(g_prevForeground)) {
PasteIntoWindow(g_prevForeground);
pasted = true;
}
SetStatus(hWnd, pasted ? L"Pasted ✓" : L"Copied ✓");
} else {
SetStatus(hWnd, L"No speech detected");
}
delete res;
}
break;
```
- ⬜ **Clear button** — `ID_BTN_CLEAR` is handled in `WM_COMMAND` but no button is created in
the modern layout, so the (now appending) transcript can't be cleared. Create it owner-drawn,
place it in `LayoutControls`, and call `UpdatePlaceholder(hWnd)` after clearing.
- ⬜ **Optional `g_autoHide`** (default `false`) — only if you want the window to tuck away after
a successful paste into another app: `if (pasted && g_autoHide) ShowWindow(hWnd, SW_HIDE);`
### Modern UI (separate doc)
The flat dark restyle (GDI+ owner-drawn buttons, rounded panel, custom VU, dark caption +
rounded corners) is fully specified in **`MODERN-UI-AND-FIXES.md`**, Part 2.
---
## Part 3 — Reasoning
**Why batch beats streaming here (F1).** Streaming only wins if you need words *as you speak*
(live captioning). Dictation doesn't: you speak a sentence, then want the text. Batch removes
the real-time deadline entirely — Whisper processes each second of audio exactly once, at its
own pace, after you stop. That means: near-zero CPU while speaking (just buffering), predictable
"a few seconds after Stop" latency that **doesn't grow with clip length**, no overlap
re-processing, and better accuracy (full context, no chunk-boundary word splits). On 2 cores
this is the difference between unusable and snappy.
**Why threads = physical cores (2), not logical (4).** Whisper's matmuls are memory-bandwidth
bound. Two extra hyperthreads share the same execution ports and cache, so they add little
throughput while stealing cycles from the UI/audio threads (janky window, laggy VU). Two
threads leaves headroom for a responsive UI.
**Why the Stop crash was a null deref (F2), and why the guard is the right fix.** Recording is
pure SDL + a `std::vector` append — it never calls into Whisper, which is why only Stop crashed.
`whisper_reset_timings(nullptr)` dereferences the context. The guard makes the worker a no-op
when no model is loaded; the separate `g_modelOk` flag makes that state *visible* (a MessageBox
with the path) instead of letting you walk into it. Defensive + diagnostic.
**Why the "crash after transcribing" was actually a hide (F5) — the logic is conclusive.** A
crashed process loses all memory. There is no on-disk persistence in this app. Therefore a
re-launched process **cannot** display the previous transcription, and a second run **cannot**
replace text in "the same box." The observed behavior (prior text reappears; second replaces
first) is only possible if it's the *same* live process — which means it hid, and the
single-instance guard re-showed it. This is why "is the process still in Task Manager?" is the
decisive test, not "did the window disappear?"
**Why clipboard = latest utterance but the box appends.** Two different jobs. The on-screen box
is your running log (you want history). The clipboard is what you paste into another app — you
want *just what you last said*, not the whole session. Splitting them gives both.
**Why paste must exclude our own window.** Auto-paste replays Ctrl+V into the foreground window
captured before the popup. If you triggered recording by clicking the Record button, that
"previous" window is the dictation app itself — pasting into its read-only box is a no-op and
(with the old hide) made the app appear to swallow the text. Restricting to
`g_prevForeground != hWnd && IsWindow(...)` makes the global-hotkey workflow the reliable path
and the button a safe "copy only."
---
## Part 4 — How to build & run tests
The goal: catch the bugs above automatically, without a human watching a window. UI behavior
(resize, hide, paste) needs a short **manual** checklist, but the **core** (model load,
transcription, null-safety, text append) can and should be tested headlessly.
### 4.1 Make the core testable
Apply the `run_inference` / `transcribe_sync` refactor from Part 2 so a test can feed a buffer
and get text synchronously. Also extract the append rule as a pure function so it can be tested
with zero Win32:
```cpp
// text_util.h (new, tiny, UI-free)
#pragma once
#include <string>
inline std::wstring append_transcript(const std::wstring& cur, const std::wstring& add) {
if (add.empty()) return cur;
if (cur.empty()) return add;
return cur + L" " + add;
}
```
Use it in `WM_APP_RESULT` (`combined = append_transcript(cur, to_w(*res));`) so the UI and the
test exercise the *same* rule.
### 4.2 The test program (`tests/test_core.cpp`)
Replaces the stale `src/test-audio.cpp`. Compiles against the **current** API and tests the real
code path (`preload` + `transcribe_sync`).
```cpp
#include "transcriber.h"
#include "text_util.h"
#include <cstdio>
#include <cstdint>
#include <fstream>
#include <vector>
#include <string>
#include <cctype>
static int g_fail = 0;
#define CHECK(cond, msg) do { if (!(cond)) { printf(" FAIL: %s\n", msg); ++g_fail; } \
else printf(" ok: %s\n", msg); } while(0)
// Minimal 16-bit PCM mono WAV loader -> float [-1,1]. Assumes 16 kHz mono (whisper.cpp samples are).
static bool load_wav(const std::string& path, std::vector<float>& out) {
std::ifstream f(path, std::ios::binary);
if (!f) return false;
char hdr[44];
f.read(hdr, 44);
if (std::string(hdr, 4) != "RIFF") return false;
std::vector<int16_t> pcm((std::istreambuf_iterator<char>(f)), {}); // crude: rest of file
out.clear(); out.reserve(pcm.size());
for (int16_t s : pcm) out.push_back(s / 32768.0f);
return !out.empty();
}
static std::string lower(std::string s) { for (char& c : s) c = (char)tolower((unsigned char)c); return s; }
int main(int argc, char** argv) {
std::string model = (argc > 1) ? argv[1] : "models/ggml-tiny.en.bin";
std::string wav = (argc > 2) ? argv[2] : "samples/jfk.wav";
// --- Pure logic: append rule (no model needed) ---
CHECK(append_transcript(L"", L"hello") == L"hello", "append: empty + a");
CHECK(append_transcript(L"hello", L"world") == L"hello world", "append: a + b spaced");
CHECK(append_transcript(L"hello", L"") == L"hello", "append: a + empty");
// --- Regression F2: bad model path must NOT crash, returns "" ---
{
Transcriber t; WhisperConfig bad; bad.model_path = "models\\does-not-exist.bin";
bool ok = t.preload(bad);
CHECK(!ok, "bad model path: preload returns false");
std::vector<float> a(16000, 0.0f); // 1 s of silence
std::string r = t.transcribe_sync(a); // must be a safe no-op
CHECK(r.empty(), "bad model path: transcribe_sync returns empty, no crash");
}
// --- Happy path: real model + known clip -> non-empty, expected words ---
{
Transcriber t; WhisperConfig cfg; cfg.model_path = model; cfg.n_threads = 4;
bool ok = t.preload(cfg);
CHECK(ok, "model loads");
if (ok) {
std::vector<float> audio;
bool loaded = load_wav(wav, audio);
CHECK(loaded, "wav loads");
if (loaded) {
std::string text = lower(t.transcribe_sync(audio));
CHECK(!text.empty(), "transcription is non-empty");
// jfk.wav: "...ask not what your country can do for you..."
CHECK(text.find("country") != std::string::npos, "transcription contains 'country'");
}
}
}
// --- Edge: sub-300ms / empty audio -> "" , no crash ---
{
Transcriber t; WhisperConfig cfg; cfg.model_path = model;
if (t.preload(cfg)) {
std::vector<float> tiny(100, 0.1f);
std::string r = t.transcribe_sync(tiny);
CHECK(true, "short audio did not crash"); // reaching here = no crash
}
}
printf("\n%s (%d failure%s)\n", g_fail ? "TESTS FAILED" : "ALL TESTS PASSED",
g_fail, g_fail == 1 ? "" : "s");
return g_fail ? 1 : 0;
}
```
> The WAV loader above is deliberately minimal (good enough for whisper.cpp's 16 kHz mono
> `samples/*.wav`). If you test arbitrary WAVs, parse the `fmt `/`data` chunks properly.
### 4.3 Build the tests (root `CMakeLists.txt`)
The active build file builds only `win-dictation`. Add a test target next to it:
```cmake
# --- tests ---
add_executable(test-core
tests/test_core.cpp
src/transcriber.cpp
src/transcriber.h
)
target_link_libraries(test-core PRIVATE whisper ${SDL2_LIBRARIES})
target_include_directories(test-core PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/src
${CMAKE_CURRENT_SOURCE_DIR}/common
)
target_compile_definitions(test-core PRIVATE UNICODE _UNICODE)
```
(`transcriber.cpp` references SDL symbols even though the test never opens a device, so the test
still links `${SDL2_LIBRARIES}`.)
Build and run:
```powershell
cmake --build build --config Release --target test-core
# run from a dir where SDL2.dll + models/ + samples/ are reachable:
cd build\bin\Release
copy ..\..\..\deps\SDL2-2.28.5\lib\x64\SDL2.dll . # if not already beside the exe
.\test-core.exe models\ggml-tiny.en.bin ..\..\..\samples\jfk.wav
```
Exit code `0` = all passed (usable in CI / a pre-commit hook).
### 4.4 What each test guards
| Test | Guards against |
|---|---|
| append rule (3 cases) | F5 regression — replace-instead-of-append |
| bad model path → false + empty | F2 — null-context crash on Stop |
| model loads / clip → non-empty, has "country" | core transcription works end to end |
| short/empty audio → no crash | the `< 0.3 s` guard + null-safety |
### 4.5 Manual UI checklist (can't be unit-tested)
- [ ] **Not a crash:** after Stop, the window **stays visible**; if it ever vanishes, the tray
icon / `win-dictation.exe` in Task Manager confirms whether it's hiding vs. truly gone.
- [ ] **Append:** two dictations in a row → second is appended after the first, view scrolls down.
- [ ] **Clipboard:** after a dictation, paste into Notepad → only the *latest* utterance appears.
- [ ] **Auto-paste:** focus Notepad, press the global hotkey, speak, press it again → text lands
in Notepad. Triggering via the Record button instead → "Copied ✓" (no self-paste).
- [ ] **Resize:** drag edges; transcript grows; min size respected.
- [ ] **Performance:** a ~10 s clip transcribes in a few seconds and the UI stays responsive;
latency does **not** grow with longer recordings.
### 4.6 Performance smoke test (optional)
`test-core` can also print timing — wrap `transcribe_sync` in `std::chrono::steady_clock` and
assert `elapsed < audio_seconds * k` for a sanity bound (e.g. `k = 1.0`, i.e. faster than
real-time on tiny.en). Useful to catch a future regression that quietly reintroduces the
streaming-style slowdown.
---
## Part 5 — Status & remaining work
**Verified fixed:** F1 (batch), F2 (null-context crash), F3 (resize), F4 (startup race).
**Apply next (⬜ in Part 2):**
1. `run_inference` / `transcribe_sync` refactor (unblocks tests).
2. `WM_APP_RESULT` rewrite (the F5 fix — no-hide + append + safe paste). **Highest priority** —
it's what's making the app *look* like it crashes.
3. Add the Clear button (needed now that text appends).
4. Add `tests/test_core.cpp` + the CMake target; run it.
**Then re-run** the Part 4 manual checklist. Expected result: the window no longer disappears,
transcriptions append, the clipboard holds the latest utterance, and `test-core.exe` exits `0`.
README.md
+58 -238
View File
@@ -1,295 +1,115 @@
# Win Dictation - AI Voice to Text for Windows
A high-performance, real-time speech-to-text application for Windows using OpenAI's Whisper model. Convert your voice to text with GPU acceleration support and a modern, user-friendly interface.
A push-to-talk speech-to-text utility for Windows using OpenAI's Whisper model. Record, transcribe, and paste with a single hotkey.
![Win Dictation Screenshot](screenshot.png)
## 🚀 Quick Download
## Quick Download
**[Download Latest Release (WinDictation.zip)](release/WinDictation.zip)**
Simply extract the ZIP file and run `win-dictation.exe`. The release includes all required DLLs and the Whisper model.
Extract the ZIP file and run `win-dictation.exe`. The release includes all required DLLs and a Whisper model.
---
## ✨ Features
## Features
### Performance
- **Multi-Core CPU Support**: Automatically uses all available CPU cores for maximum performance
- **GPU Acceleration**: Auto-detects and uses CUDA or Vulkan when available
- **Smart Model Selection**: Automatically selects optimal model (tiny.en for CPU-only, base.en for GPU) for best performance
- **Ring Buffer Audio**: Zero audio loss with lock-free ring buffer implementation
- **Optimized Processing**: AVX2/FMA instructions for maximum performance
- **Physical-core threading**: Uses one thread per physical core for efficient batch transcription
- **CPU-only**: Optimised for the target Intel i5-7th-gen 2-core/4-thread machine
- **Smart model selection**: Auto-selects tiny.en for CPU-only, base.en when GPU is present
- **Self-calibrating progress**: Learns transcription speed per model and machine, delivering a smooth countdown
### User Interface
- **Modern Dark Theme**: Polished, professional interface
- **Real-Time Monitoring**:
- Live VU meter for audio levels
- Buffer status indicator
- GPU/CPU usage display
- **Smooth Animations**: 30 FPS UI updates for responsive experience
- **System Tray Integration**: Minimize to tray with hotkey support
- **Single-surface rendering**: No inter-window seams or hairlines — the entire UI is one painted surface
- **Dark theme**: Calm, elevated card design with hover/press feedback
- **Per-monitor DPI awareness**: Looks sharp at any display scale
- **System tray**: Minimise to tray, global hotkey to record
### Audio Processing
- **Voice Activity Detection (VAD)**: Automatically filters silence
- **Continuous Recording**: Maintains context between segments
- **Multiple Microphone Support**: Select from all available input devices
- **16kHz Sample Rate**: Optimized for Whisper model
- **Push-to-talk**: Press Ctrl+Shift+Space, speak, press again to transcribe
- **500ms auto-end**: Stops recording after 500ms of silence
- **Multiple microphones**: Select from all available input devices
- **16kHz sample rate**: Optimised for Whisper
## 🎯 Usage
## Usage
### Controls
- **Start/Stop Recording**: Click button or press `Ctrl+Shift+R`
- **Clear Text**: Click "Clear" button
- **Change Microphone**: Select from dropdown (auto-restarts recording)
- **Minimize**: Close window (minimizes to system tray)
- **Exit**: Right-click tray icon → Exit
- **Record**: Click the Record pill or press `Ctrl+Shift+Space`
- **Pin**: Keep window always-on-top
- **Copy / Paste / Clear**: Text actions
- **Model / Mic**: Select from popup menus
- **Hide**: `Ctrl+Shift+H` hides the window
### Indicators
- **Level**: Real-time audio input level
- **Buffer**: Current audio buffer usage (0-100%)
- **Status**: Shows GPU/CPU mode, recording state, thread count
- **Level**: Live audio energy during recording
- **Progress bar**: Smooth, counting-down estimate during transcription
- **Status**: Thread count at idle, elapsed time during recording
## 🔨 Building from Source
## Building from Source
### Prerequisites
- **Windows 10/11**
- **CMake** (3.5 or newer)
- **C++ Compiler** (MSVC 2019+ or MinGW)
- **PowerShell** (for build script)
- **Optional**: CUDA 12.4+ or Vulkan SDK (for GPU acceleration)
- **CMake** 3.5+
- **Visual Studio 2022/2026** with C++ workload
- **SDL2** (included in deps/)
### Build Steps
1. **Clone the repository:**
```powershell
git clone <repository-url>
cd win-dictation
```
2. **Run the build script:**
```powershell
powershell -ExecutionPolicy Bypass -File src/build.ps1
```
The build script will:
- Detect your GPU capabilities (CUDA, Vulkan)
- Download and configure SDL2 automatically
- Build the application with optimal settings
- Download both Whisper models (tiny.en for CPU, base.en for GPU)
- Deploy all required DLLs
3. **Run the application:**
```powershell
build\bin\Release\win-dictation.exe
```
### Manual Build (Alternative)
If you prefer to build manually:
```powershell
# Configure CMake
cmake -B build -DWHISPER_SDL2=ON
# For GPU support (CUDA):
cmake -B build -DWHISPER_SDL2=ON -DGGML_CUDA=ON
# For GPU support (Vulkan):
cmake -B build -DWHISPER_SDL2=ON -DGGML_VULKAN=ON
# Build
cmake -S . -B build -G "Visual Studio 18 2026" \
-DSDL2_DIR="deps/SDL2-2.28.5/cmake"
cmake --build build --config Release
# The executable will be at: build\bin\Release\win-dictation.exe
```
### SDL2 Setup
The executable will be at `build\bin\Release\win-dictation.exe`.
The build script automatically downloads SDL2. If building manually, you can:
1. Download SDL2 from: https://github.com/libsdl-org/SDL/releases
2. Extract to `SDL2-mingw/` directory
3. Set `SDL2_DIR` in CMake to point to the SDL2 cmake directory
## ⚙️ Technical Details
## Technical Details
### Architecture
#### Ring Buffer Audio Capture
- **Lock-Free Design**: Audio thread never blocks
- **30-Second Buffer**: Handles burst processing without loss
- **Atomic Operations**: Prevents race conditions
#### Processing Pipeline
```
Audio Input → Ring Buffer → VAD → Whisper Inference → Text Output
Hotkey → SDL Capture → Stop → Batch whisper_full → Text → Auto-paste
```
1. **SDL Audio Capture**: 512-sample chunks at 16kHz
2. **Ring Buffer**: Lock-free circular buffer
3. **VAD Processing**: Filters silence before inference
4. **Whisper Inference**: Multi-threaded with context overlap
5. **Text Output**: Appended to UI in real-time
### Performance Optimizations
#### CPU Mode
- All available CPU threads utilized
- AVX2/FMA SIMD instructions
- Optimized memory layout
- Minimal context switching
#### GPU Mode (When Available)
- CUDA 12.4+ or Vulkan SDK required
- Automatic offloading to GPU
- Faster inference times
- Lower CPU usage
1. **SDL audio capture**: 16kHz mono recording into memory
2. **Stop-and-transcribe**: Press stop or hit max length (600s), then one `whisper_full` call
3. **Progress estimation**: Linear model fitted per machine/model, fused with whisper's chunk progress
4. **Text output**: Appended to transcript, copied to clipboard, optionally auto-pasted
### Model
The app automatically selects the optimal model based on your system:
Place `.bin` files in `models/` next to the executable. The app auto-detects available models:
**CPU-Only Systems:**
- Uses `ggml-tiny.en.bin` (75 MB)
- **Parameters**: 39 million
- **Speed**: ~10-15x real-time on CPU
- **Accuracy**: Good for general speech
- Optimized for slower machines
| Model | Size | Params | Best for |
|-------|------|--------|----------|
| tiny.en | 75 MB | 39M | CPU-only systems |
| base.en | 140 MB | 74M | GPU-accelerated systems |
**GPU-Accelerated Systems:**
- Uses `ggml-base.en.bin` (140 MB)
- **Parameters**: 74 million
- **Speed**: >20x real-time on GPU
- **Accuracy**: Excellent for general speech
- Better accuracy with GPU acceleration
Both models are English-only (optimized). The app detects GPU availability at startup and selects the appropriate model automatically. To use a different model, place it in `models/` directory and the app will detect it.
## 📊 Performance Benchmarks
### CPU-Only (24 threads, tiny.en model)
- **Latency**: ~1-2 seconds
- **Throughput**: ~10-15x real-time
- **CPU Usage**: 40-60% during speech
- **Memory**: ~200 MB
- **Model**: Automatically selected for CPU-only systems
### CPU-Only (24 threads, base.en model - if manually selected)
- **Latency**: ~2-3 seconds
- **Throughput**: ~5x real-time
- **CPU Usage**: 60-80% during speech
- **Memory**: ~500 MB
### GPU-Accelerated (RTX 3090, base.en model)
- **Latency**: <1 second
- **Throughput**: >20x real-time
- **GPU Usage**: 20-30%
- **CPU Usage**: <10%
- **Memory**: ~1 GB (VRAM)
- **Model**: Automatically selected for GPU systems
## 🔧 Troubleshooting
### GPU Not Detected
- **CUDA**: Install CUDA Toolkit 12.4 or newer (CUDA 13.0 recommended)
- See `src/CUDA-SETUP.md` for detailed installation guide
- **Vulkan**: Install Vulkan SDK
- CPU-only mode still provides excellent performance with all cores
### Audio Not Working
- Check microphone permissions in Windows Settings
- Verify correct device selected in dropdown
- Test microphone in Windows Sound settings
### Poor Transcription Quality
- Ensure microphone is close (6-12 inches)
- Reduce background noise
- Check VU meter shows green when speaking
- Try a larger model (medium.en or large-v3-turbo)
### High CPU Usage
- Normal during active transcription
- Reduces during silence (VAD filtering)
- Consider enabling GPU acceleration
## 📁 Project Structure
## Project Structure
```
win-dictation/
├── src/ # Main application source code
│ ├── main.cpp # UI and Windows message handling
│ ├── transcriber.* # Core transcription logic
│ └── build.ps1 # Automated build script
├── src/ # Application source
│ ├── main.cpp # UI and message handling
│ ├── transcriber.* # Recording and transcription
│ ├── timing.h # Progress estimation engine
│ ├── settings.h # INI persistence
│ ├── text_util.h # Transcript helpers
│ └── logging.h # Log utilities
├── whisper/ # Whisper.cpp library
├── ggml/ # GGML tensor library
├── common/ # Shared utilities
├── models/ # Whisper model files
├── release/ # Pre-built release package
│ └── WinDictation.zip
└── CMakeLists.txt # Main build configuration
├── release/ # Pre-built package
└── CMakeLists.txt # Build configuration
```
## 🆕 Recent Improvements
## License
### v2.0 (Current)
- ✅ **Ring buffer** implementation - no more dropped audio
- ✅ **Multi-core CPU** support - uses all available threads
- ✅ **GPU auto-detection** - CUDA/Vulkan support
- ✅ **Modern UI** - dark theme, smooth animations
- ✅ **VAD integration** - skip silence for efficiency
- ✅ **Better error handling** - graceful fallbacks
- ✅ **Status indicators** - real-time monitoring
- ✅ **Build script** - automated setup and deployment
MIT — follows [whisper.cpp](https://github.com/ggerganov/whisper.cpp).
## 🔮 Future Enhancements
- [ ] Push-to-talk mode
- [ ] Multiple language support
- [ ] Punctuation model integration
- [ ] Export to file (TXT, SRT)
- [ ] Custom hotkey configuration
- [ ] Noise reduction filter
- [ ] Model switching in UI
- [ ] Real-time word highlighting
## 📝 License
This project uses the MIT license, following the same license as [whisper.cpp](https://github.com/ggerganov/whisper.cpp).
## 🤝 Contributing
Improvements welcome! The code is designed to be:
- **Readable**: Clear structure and comments
- **Maintainable**: Modular design
- **Extensible**: Easy to add features
- **Performant**: Optimized critical paths
## 📚 Resources
- [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) - Core library
- [Whisper Paper](https://arxiv.org/abs/2212.04356) - Research paper
- [Model Download](https://huggingface.co/ggerganov/whisper.cpp) - Additional models
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) - GPU acceleration
- [Vulkan SDK](https://vulkan.lunarg.com/) - Alternative GPU backend
## 💡 Tips
### For Best Results
1. Use a quality microphone
2. Position mic 6-12 inches from mouth
3. Speak clearly and naturally
4. Minimize background noise
5. Keep buffer below 50% (adjust step_ms if needed)
### For Development
- See `src/transcriber.h/cpp` for core logic
- See `src/main.cpp` for UI implementation
- Adjust parameters in `WhisperConfig` struct
- Enable logging in `whisper_full_params`
---
**Built with ❤️ using whisper.cpp**
## Resources
- [Whisper.cpp](https://github.com/ggerganov/whisper.cpp)
- [Model Download](https://huggingface.co/ggerganov/whisper.cpp)
UI-and-progress-bar-rebuild-part-2.md
+78
View File
@@ -0,0 +1,78 @@
# Win Dictation — Fix Note 01
**Every control renders as the blue Record pill**
**Applies to:** the single-surface UI rewrite described in `UI-and-Progress-Rebuild.md` (Part 1), after the first build.
**Status:** root cause confirmed, one-line fix below.
---
## Symptom
After wiring up the single-surface paint, every control — **Record**, **Pin**, the **mic** and **model** selects, and **Copy / Paste / Clear** — renders as the *same* blue accent pill with a white dot. No labels, no distinct styles. (The window chrome, card, and scrollbar are correct; only the widgets are wrong.)
## The mistake
`Widget::kind` is declared without an initializer, and the widget array has static storage duration, so the whole array is zero-initialized:
```cpp
struct Widget {
WK kind; // <-- no initializer
RectF r;
bool hover = false;
bool pressed = false;
float anim = 0.0f;
};
static Widget g_w[ (int)WK::Transcript + 1 ]; // zero-filled → every kind == 0
```
`WK::RecordHero` is the first enumerator, i.e. value `0`. So **every** slot's `kind` reads as `RecordHero`. `LayoutWidgets()` assigns each slot's *rectangle* (`g_w[(int)WK::Pin].r = …`, etc.) but never its `kind`. The paint loop dispatches on `kind`:
```cpp
for (auto& w : g_w) {
switch (w.kind) { // 0 for every widget
case WK::RecordHero: DrawHero(g, w); break; // <-- all eight land here
case WK::Copy: DrawGhost(g, w, L"Copy"); break; // unreachable
case WK::SelAudio: DrawSelectSurface(...); break; // unreachable
...
}
}
```
Result: `DrawHero` paints all eight widgets → eight identical blue pills.
## The required change
The array is **indexed by enum value** (`g_w[(int)WK::Pin]`, `g_w[(int)WK::Copy]`, …), so slot `i` must carry `kind == (WK)i`. Stamp the kinds once, at the top of `LayoutWidgets()` — it runs before the first paint and again on every resize / DPI change, so the invariant is always re-asserted:
```cpp
void LayoutWidgets(int W, int H) {
for (int i = 0; i < (int)std::size(g_w); ++i) // <-- ADD THESE TWO LINES
g_w[i].kind = (WK)i; // slot index == kind
for (auto& w : g_w) w.r = RectF(0, 0, 0, 0);
float s = g_dpiScale;
// ... unchanged ...
}
```
Equivalent alternative: an `InitWidgets()` helper called once in `wWinMain` before the first `LayoutWidgets`/paint. Doing it *inside* `LayoutWidgets` is the most robust, because nothing can paint before layout has run.
**Invariant to preserve:** `g_w[i].kind == (WK)i`. As long as rectangles keep being assigned via `g_w[(int)WK::X].r`, the slot index always equals the enum value and this holds.
After this change, `DrawGhost` (Copy/Paste/Clear), `DrawPinSurface`, and `DrawSelectSurface` (mic/model) take over their slots, and you get the intended distinct, borderless, label-bearing controls.
## Secondary issue noticed (cosmetic — not fixed here)
The transcript **placeholder** ("Your transcription will appear here…") won't appear. `PaintSurface` draws it into the transcript rect, but the real multiline `EDIT` child window covers that rect and is opaque, so it hides the parent-painted text. `EM_SETCUEBANNER` only works on *single-line* edits. Clean fix: subclass the multiline `EDIT` and draw the cue in its own `WM_PAINT` when the control is empty and unfocused. Tracked as a separate follow-up.
## Verify after rebuilding
1. **Top row:** a wide blue **Record** pill + a quiet **Pin** toggle (accent only when pinned).
2. **Bottom:** **mic** and **model** selects (label + chevron, faint at rest) and quiet **Copy / Paste / Clear** ghost buttons (text only at rest).
3. **Hover** a ghost button → a soft fill fades in; **press** → slightly darker. No resting borders, no hairlines.
4. Independent of this fix, run a transcription and confirm the progress **%** rises smoothly and the **"… s left"** value counts **down** (not up).
---
*Companion to `UI-and-Progress-Rebuild.md`. This note documents a correction to that guide's Part 1 scaffold; the guide itself is left unchanged.*
UI-and-progress-bar-rebuild-part-3.md
+214
View File
@@ -0,0 +1,214 @@
# Win Dictation — Fix Note 02
**All text missing from the interface (fills render, labels don't)**
**Applies to:** the single-surface UI after applying Fix Note 01 (`g_w[i].kind` stamping).
**Status:** root cause confirmed. One pattern to remove, used in five places. Two smaller adjacent issues documented below.
---
## Symptom
Widgets now render in their correct *shapes* — blue Record pill with white dot, two select fields with chevrons, the card, the dark scrollbar — but **no text appears anywhere**:
- Record pill has no "Record" label
- Pin is completely invisible (empty space right of the pill)
- Copy / Paste / Clear are completely invisible (empty strip at the bottom)
- Mic / model selects are empty except for the chevron
- No status line ("Ready · 2 threads" etc.)
The tell: everything drawn with Graphics *primitives* (`FillRound`, `FillEllipse`, `DrawLine`, `DrawPath`) renders; everything drawn with `DrawTextC` doesn't. Pin and the ghost buttons are text-only at rest, so they vanish entirely.
## What you missed: `Graphics::GetHDC()` locks the Graphics object
Every text call sits inside this pattern (from `DrawHero`, and repeated in the other draw functions):
```cpp
HDC hdc = g.GetHDC(); // <-- locks `g`
Font f(hdc, g_fUISemi); // (constructing the Font is fine)
RectF tb(...);
DrawTextC(g, L"Record", f, ...); // <-- call on locked `g` → fails silently
g.ReleaseHDC(hdc); // <-- unlock, too late
```
`Graphics::GetHDC()` is documented to put the Graphics object into a **locked state**: between `GetHDC()` and `ReleaseHDC()`, *any* method called on that `Graphics` fails with `Status::ObjectBusy`. GDI+ reports errors via return codes, not exceptions — so `g.DrawString(...)` inside `DrawTextC` returns an error and draws nothing, with no crash and no debugger output. The result is exactly what you see: silent, total text loss, while every primitive drawn *outside* a lock window renders fine.
This also explains two details that look confusing at first:
1. **Why the chevrons survive in `DrawSelectSurface`:** the two `g.DrawLine(...)` calls happen *after* `g.ReleaseHDC(hdc)` — outside the lock — so they render.
2. **Why the *old* owner-draw code's text worked:** it wrote `Font f(d->hDC, g_fUISemi)` using the **raw owner-draw HDC** it already had. Constructing a `Font` from an HDC does not lock anything — only `Graphics::GetHDC()` does. (Same reason the popup's `Font f(mem, g_fUI)` still works: `mem` is the raw memory HDC, not a `GetHDC()` result.)
The `GetHDC()` calls were added because the new draw functions receive only a `Graphics&` and the `Font(HDC, HFONT)` constructor needs an HDC. The intent was right; the mechanism poisons the Graphics.
### Affected call sites (all five must change)
| Function | What's invisible |
|---|---|
| `DrawHero` | "Record" / "Stop" label |
| `DrawGhost` | Copy / Paste / Clear (entire control) |
| `DrawPinSurface` | Pin / Pinned (entire control) |
| `DrawSelectSurface` | mic / model value text |
| `PaintSurface` | status line, transcript placeholder |
## The fix: cached GDI+ fonts, zero `GetHDC()` calls
Create GDI+ `Font` objects **once** from the existing HFONTs using a *screen* DC (never the Graphics), cache them, and use them everywhere. This removes every `GetHDC()`/`ReleaseHDC()` pair and is also a per-frame win — no font construction inside a 60 fps paint loop.
### 1. Add globals (near the HFONT globals)
```cpp
Gdiplus::Font* g_gpUI = nullptr; // labels (15px)
Gdiplus::Font* g_gpUISemi = nullptr; // hero label (15px semibold)
Gdiplus::Font* g_gpSmall = nullptr; // status line (12px)
Gdiplus::Font* g_gpText = nullptr; // transcript placeholder (16px)
```
### 2. Add the builder and fold it into `RecreateFonts`
```cpp
static Gdiplus::Font* GdipFontFromHFont(HFONT hf) {
HDC sdc = GetDC(nullptr); // screen DC — no Graphics involved
Gdiplus::Font* f = new Gdiplus::Font(sdc, hf);
ReleaseDC(nullptr, sdc);
if (f->GetLastStatus() != Ok) { delete f; return nullptr; }
return f;
}
static void RebuildGdipFonts() {
delete g_gpUI; delete g_gpUISemi; delete g_gpSmall; delete g_gpText;
g_gpUI = GdipFontFromHFont(g_fUI);
g_gpUISemi = GdipFontFromHFont(g_fUISemi);
g_gpSmall = GdipFontFromHFont(g_fSmall);
g_gpText = GdipFontFromHFont(g_fText);
// safety net if HFONT conversion ever fails:
if (!g_gpUI) g_gpUI = new Gdiplus::Font(L"Segoe UI", 15.0f * g_dpiScale, FontStyleRegular, UnitPixel);
if (!g_gpUISemi) g_gpUISemi = new Gdiplus::Font(L"Segoe UI", 15.0f * g_dpiScale, FontStyleBold, UnitPixel);
if (!g_gpSmall) g_gpSmall = new Gdiplus::Font(L"Segoe UI", 12.0f * g_dpiScale, FontStyleRegular, UnitPixel);
if (!g_gpText) g_gpText = new Gdiplus::Font(L"Segoe UI", 16.0f * g_dpiScale, FontStyleRegular, UnitPixel);
}
```
At the **end of `RecreateFonts(float scale)`**, add:
```cpp
RebuildGdipFonts();
```
So HFONTs and GDI+ fonts always change together (startup and `WM_DPICHANGED`).
### 3. Call it at startup — ordering matters
`RebuildGdipFonts` needs GDI+ started (it already is — `GdiplusStartup` runs first) and benefits from the real DPI. In `wWinMain`, right after the DPI is known:
```cpp
g_dpiScale = GetDpiForWindow(hMainWnd) / 96.0f;
if (g_dpiScale <= 0.0f) g_dpiScale = 1.0f;
RecreateFonts(g_dpiScale); // <-- ADD: rebuilds HFONTs at real DPI + GDI+ fonts
```
(See "Adjacent issue A" below for why `RecreateFonts` belongs here anyway.)
### 4. Shutdown ordering — GDI+ objects must die before `GdiplusShutdown`
In the shutdown block, delete the GDI+ fonts **before** `GdiplusShutdown(g_gdipToken)`:
```cpp
delete g_gpUI; delete g_gpUISemi; delete g_gpSmall; delete g_gpText;
g_gpUI = g_gpUISemi = g_gpSmall = g_gpText = nullptr;
GdiplusShutdown(g_gdipToken);
DeleteObject(g_fUI); ... // HFONTs are plain GDI; their order is fine as-is
```
(Destroying GDI+ objects after shutdown is undefined behavior — worth getting right even though it "usually" doesn't crash.)
### 5. Strip the lock pattern from all five call sites
**`DrawHero` — before:**
```cpp
HDC hdc = g.GetHDC();
Font f(hdc, g_fUISemi);
RectF tb((REAL)(pill.X + (int)(36 * g_dpiScale)), (REAL)pill.Y,
(REAL)(pill.Width - (int)(44 * g_dpiScale)), (REAL)pill.Height);
DrawTextC(g, rec ? L"Stop" : L"Record", f, Color(255, 255, 255, 255),
tb, StringAlignmentNear, StringAlignmentCenter);
g.ReleaseHDC(hdc);
```
**`DrawHero` — after:**
```cpp
RectF tb((REAL)(pill.X + (int)(36 * g_dpiScale)), (REAL)pill.Y,
(REAL)(pill.Width - (int)(44 * g_dpiScale)), (REAL)pill.Height);
DrawTextC(g, rec ? L"Stop" : L"Record", *g_gpUISemi, Color(255, 255, 255, 255),
tb, StringAlignmentNear, StringAlignmentCenter);
```
Apply the identical transformation to the rest — delete the `GetHDC`/`Font(hdc, …)`/`ReleaseHDC` lines and pass the cached font:
| Call site | Replace local `Font f(hdc, …)` with |
|---|---|
| `DrawGhost` | `*g_gpUI` |
| `DrawPinSurface` | `*g_gpUI` |
| `DrawSelectSurface` | `*g_gpUI` |
| `PaintSurface` (status line) | `*g_gpSmall` |
| `PaintSurface` (placeholder) | `*g_gpText` |
After this, there must be **zero** calls to `g.GetHDC()` anywhere in the paint path. (The popup's `Font f(mem, g_fUI)` is fine and can stay — `mem` is a raw HDC; consider migrating it to `*g_gpUI` later for consistency.)
---
## Adjacent issue A — fonts are built at the wrong DPI on startup
In `wWinMain`, the HFONTs are created with `g_dpiScale` still at its initial `1.0f` (the window doesn't exist yet), and `g_dpiScale` is only set *after* `CreateWindowExW`. Nothing recreates the fonts at startup, so on a 125%/150% display, all text is undersized until the first `WM_DPICHANGED`. The `RecreateFonts(g_dpiScale)` call added in step 3 fixes this — and because it runs *before* the child controls are created, the `EDIT` receives the correctly-scaled `g_fText` at creation.
## Adjacent issue B — transient status messages are now invisible
`SetStatus(...)` writes to the `ID_STATIC_STATUS` control — which is hidden (`SW_HIDE`). The painted status line in `PaintSurface` derives its text purely from state (recording/busy/ready/loading), so these messages can never appear: **"Copied", "Pasted", "Cancelled", "No speech detected", "Microphone error", "Hotkey in use — edit win-dictation.ini"**.
Minimal repair — route `SetStatus` into the painted surface as a transient override:
```cpp
std::wstring g_statusOverride; // shown instead of the derived idle status
DWORD g_statusOverrideUntil = 0; // GetTickCount() deadline
void SetStatus(HWND hwnd, const wchar_t* text) {
g_statusOverride = text;
g_statusOverrideUntil = GetTickCount() + 2500; // visible for 2.5s
InvalidateRect(hwnd, nullptr, FALSE);
}
```
In `PaintSurface`'s status-text branch, prefer the override when idle:
```cpp
} else if (g_modelLoaded.load()) {
if (!g_statusOverride.empty() && GetTickCount() < g_statusOverrideUntil) {
wcscpy_s(statusBuf, g_statusOverride.c_str());
} else if (!g_modelOk.load()) {
swprintf_s(statusBuf, L"Model not found — check models folder");
} else {
swprintf_s(statusBuf, L"Ready • %d threads", g_tx.threads());
}
}
```
(Recording/busy branches still win, which is correct — those states are more important than a stale "Copied".)
---
## Verify after rebuilding
1. **Labels everywhere:** "Record" on the pill; "Pinned" top-right in accent; "Copy / Paste / Clear" as dim labels; mic + model names in the selects; "Loading model…" → "Ready · 2 threads" on the status line.
2. **Hover** Copy/Paste/Clear → soft fill fades in, label brightens.
3. **Copy something** → status briefly shows "Copied" (issue B fix).
4. On a HiDPI display, text is correctly sized at first launch (issue A fix).
5. Run a transcription → smooth rising % and counting-down ETA (unrelated path, but confirm while you're there).
## Optional cleanup (safe to defer)
The legacy chrome is now dead weight: the nine hidden child windows and their `Create…`/`SetWindowTheme`/`SetWindowSubclass`/`ShowWindow(SW_HIDE)` calls, `LayoutControls`, `BtnProc`, `DrawRecordButton`, `DrawFlatButton`, `DrawSelect`, `UpdatePlaceholder`'s STATIC logic, and the `WM_DRAWITEM`/`WM_MEASUREITEM` handlers (keep the `EDIT` and everything for it). Removing them deletes ~150 lines and removes the double layout work in `WM_SIZE` (`LayoutControls` + `LayoutWidgets` both run and both `MoveWindow` the EDIT). Functionally harmless today, so treat as a tidy-up pass, not part of this fix.
---
*Companion to `UI-and-Progress-Rebuild.md` and `UI-Progress-Rebuild-Fix-01.md`. Per project convention, this note is a new document; prior documents are unchanged.*
UI-and-progress-bar-rebuild.md
+630
View File
@@ -0,0 +1,630 @@
# Win Dictation — UI & Progress Rebuild Guide
A design-led plan to (1) kill the "thin lines" problem at its architectural root rather than patching it, and (2) replace the broken progress bar with a self-calibrating, smoothly-animated estimator that learns this machine's transcription speed and fuses whisper's own progress signal.
This is implementation guidance with concrete code. You build on Windows; nothing here is compiled or tested in place.
---
## Table of contents
1. [Part 1 — The interface](#part-1--the-interface)
- [1.1 Why the lines are really there](#11-why-the-lines-are-really-there)
- [1.2 The architectural fix: one surface](#12-the-architectural-fix-one-surface)
- [1.3 Two tiers: GDI+ vs Direct2D](#13-two-tiers-gdi-vs-direct2d)
- [1.4 A real design language](#14-a-real-design-language)
- [1.5 Component specs](#15-component-specs)
- [1.6 Rendering scaffold + hit-testing (code)](#16-rendering-scaffold--hit-testing-code)
- [1.7 The transcript field & DPI](#17-the-transcript-field--dpi)
- [1.8 Migration order from today's main.cpp](#18-migration-order-from-todays-maincpp)
2. [Part 2 — The progress system](#part-2--the-progress-system)
- [2.1 Why it's broken today](#21-why-its-broken-today)
- [2.2 The plan: predict, then correct](#22-the-plan-predict-then-correct)
- [2.3 Persistent per-model timing history](#23-persistent-per-model-timing-history)
- [2.4 The live estimator (smooth countdown + fusion)](#24-the-live-estimator-smooth-countdown--fusion)
- [2.5 `timing.h` — full code](#25-timingh--full-code)
- [2.6 Wiring into main.cpp](#26-wiring-into-maincpp)
- [2.7 Tuning & edge cases](#27-tuning--edge-cases)
3. [Part 3 — Cleanup checklist](#part-3--cleanup-checklist)
4. [Part 4 — Suggested build order](#part-4--suggested-build-order)
---
# Part 1 — The interface
## 1.1 Why the lines are really there
The hairlines aren't one bug; they're an emergent property of how the window is built. Today the UI is roughly **nine separate child windows** living on top of the main window:
- `BUTTON` (owner-draw): Record, Pin, Copy, Paste, Clear
- `BUTTON` (owner-draw) used as selects: mic, model
- `EDIT` (multiline): the transcript
- `STATIC`: status + placeholder
Each child is its own HWND with its own device context, its own paint timing, and — because the parent uses `WS_CLIPCHILDREN` — its own **hard-clipped rectangle punched out of the parent's paint**. That single fact is the source of the lines:
1. **Seams at every child boundary.** The parent paints its background/panel, then Windows clips out each child rectangle and the child paints itself. The boundary between "parent pixels" and "child pixels" is a 1px hard edge. Any difference in rounding, antialiasing, or color across that edge reads as a hairline — even when both sides *intend* to be the same dark color.
2. **Theme chrome you didn't ask for.** The `EDIT` control draws its own themed 1px border and a **light-mode scrollbar** (the pale bar on the right of your screenshot). `SetWindowTheme(h, L"", L"")` on the buttons disables visual styles but doesn't make seams go away.
3. **Square corners around round shapes.** Your chips are drawn rounded, but the *child window* is rectangular, so the artifact rectangle has sharp corners that don't follow the chip — which is exactly what's visible around the Record pill and the Copy/Paste/Clear buttons.
So removing `StrokeRound(..., C_BORDER, ...)` only removes the *intentional* borders. The *structural* hairlines (items 1–3) remain. That's why it feels like a band-aid: **you can't fully remove seams while compositing many themed child windows.**
> **Root cause, one sentence:** the window is assembled from many separate themed/owner-draw child HWNDs, and the boundaries between them can never be made perfectly seamless. The fix is to stop having those boundaries.
## 1.2 The architectural fix: one surface
Render the **entire window as a single double-buffered surface**, immediate-mode:
- The parent's `WM_PAINT` draws *everything* — background, the card, every button, the selects, the status line, the VU/progress strip — onto **one off-screen bitmap**, then blits it once. (You already do this for the background and panel; we extend it to cover all chrome.)
- **There are no child windows for chrome.** "Buttons" become **painted regions** described by a small data model (a rect + a kind + interaction state). There is exactly one surface, so there are zero inter-window seams. Antialiasing, radii, spacing, shadows, and animation are all under your control.
- **Interaction** is handled in the parent: `WM_MOUSEMOVE` / `WM_LBUTTONDOWN` / `WM_LBUTTONUP` hit-test against the widget rects; you track hover/pressed/focus yourself and invalidate. (The window is tiny — invalidating the whole client area each frame is cheap.)
- **The one exception is the transcript**, which stays a real `EDIT` child because you genuinely want selection, caret, scrolling, and IME. We make it *visually chrome-less* and inset it inside the painted card so the card is the only visible frame (see [1.7](#17-the-transcript-field--dpi)).
This is the same "retained data model + immediate-mode paint" approach used by every good custom-drawn desktop UI. Separation between elements comes from **fills, spacing, and elevation — not outlines.** Once outlines stop being load-bearing, the hairline problem is gone by construction.
## 1.3 Two tiers: GDI+ vs Direct2D
You said you'll happily take more effort for a result that looks genuinely good. Here are the two honest options.
### Tier 1 — GDI+ single-surface (recommended baseline)
- Keep GDI+ (already in the project). Move all drawing into one parent paint routine that renders to a 32-bit DIB back-buffer, then `BitBlt`.
- Reuse your existing helpers (`FillRound`, `StrokeRound`, `DrawTextC`) — they're good. You're changing *what hosts them*, not the primitives.
- Add an animation clock + hover/press state.
- **Effort:** moderate. **Payoff:** the seams disappear, you get full control of spacing/elevation/motion, and it will look clean and modern. This removes 100% of the reported problem.
- **Limitations:** GDI+ has no true GPU compositing; soft drop-shadows must be faked (pre-blurred bitmap or layered alpha), and very large blurs are slow. For a 400×340 utility this is a non-issue.
### Tier 2 — Direct2D + DirectWrite (premium path)
- GPU-accelerated geometry with flawless antialiasing, real `ID2D1Effect` drop shadows / Gaussian blur, per-primitive opacity layers, and **DirectWrite** text with subpixel positioning (noticeably crisper labels, especially at fractional DPI).
- Pairs naturally with a swap-chain or a DC render target; integrates with DWM for tear-free animation at the monitor refresh rate.
- Optionally add **Windows.UI.Composition / DirectComposition** for soft shadows and an acrylic/mica backdrop — a true Windows 11 feel.
- **Effort:** higher (COM lifetimes, device-lost handling, more setup). **Payoff:** the highest visual ceiling and the best foundation if this app grows.
- You can still keep the `EDIT` child for the transcript layered above the D2D surface.
**Recommendation:** Build **Tier 1 now** — it eliminates the actual defect and looks great, and almost all of the work (the design language, the widget model, the interaction layer, the progress system in Part 2) is *identical* regardless of renderer. If you later want the extra polish, swapping the draw calls to Direct2D is a contained change because the data model and layout stay the same. The rest of this guide is written renderer-agnostic with GDI+ code samples.
## 1.4 A real design language
The current look is "many bordered boxes." The target look is **one calm, elevated card** where hierarchy comes from type, spacing, and a single light source — not lines.
### Tokens (define once)
```cpp
// ---- color tokens (ARGB) ----
const Color T_BG (255, 0x0E, 0x10, 0x14); // app backdrop (near-black)
const Color T_CARD (255, 0x16, 0x19, 0x20); // elevated card
const Color T_CARD_HI (255, 0x1E, 0x22, 0x2B); // hovered surface
const Color T_CARD_LO (255, 0x12, 0x15, 0x1B); // pressed surface / wells
const Color T_TEXT (255, 0xEC, 0xEE, 0xF2); // primary text
const Color T_DIM (255, 0x8A, 0x90, 0x9C); // secondary text
const Color T_FAINT (255, 0x5A, 0x60, 0x6C); // tertiary / icons at rest
const Color T_ACCENT (255, 0x6E, 0x8B, 0xFF); // primary action
const Color T_ACCENT_HI (255, 0x83, 0x9C, 0xFF); // accent hover
const Color T_DANGER (255, 0xFF, 0x5C, 0x5C); // recording
const Color T_GOOD (255, 0x46, 0xD3, 0x9A); // level / success
// The ONLY "edge" allowed: a low-alpha top highlight on the card,
// to read as "lit from above." Never a full gray rectangle.
const Color T_TOPLIGHT (26, 0xFF, 0xFF, 0xFF); // ~10% white
```
**Principle:** elements are distinguished by *fill* (`T_CARD` vs `T_CARD_HI`), by *space* (generous padding), and by *elevation* (the card sits on the backdrop, optionally with a soft shadow). Outlines are reserved for nothing, or at most one hairline-as-toplight on the card itself.
### Type scale (Segoe UI Variable, which you already load)
| Role | Size (logical px) | Weight | Color |
|------|------|--------|-------|
| Primary state ("Record" / "Stop" / "Transcribing") | 16 | SemiBold | white on accent / `T_TEXT` |
| Body / transcript | 16 | Regular | `T_TEXT` |
| Buttons (ghost) | 14 | Medium | `T_DIM` → `T_TEXT` on hover |
| Status caption | 12.5 | Regular | `T_DIM` |
| Micro (threads, %, ETA) | 11.5 | Regular | `T_FAINT` |
### Elevation & radius
- Card radius **16**; inner controls radius **10–11**; progress/level pill radius = half-height.
- Optional soft shadow under the card (Tier 1: a pre-rendered blurred rounded-rect bitmap at ~22% alpha, offset y+6, blur ~18; Tier 2: a D2D shadow effect). Subtle — it should read as depth, not drama.
### Motion (this is what makes it feel "good", not just look good)
- Hover/press fills cross-fade over **120–160ms**, ease-out-cubic.
- Recording state: a **1.2s sine "breathing"** on the record pill + a live waveform (see below).
- Progress: bar width and the % label are **eased**, never snapped (except the final 100%).
- Drive all of it from one animation clock (Section 1.6). Run the timer at ~16ms **only while something is animating**, and idle otherwise (don't burn CPU on a 2-core machine when nothing moves).
## 1.5 Component specs
**Record (hero).** Full-width pill, `T_ACCENT` fill, white glyph + label. States:
- *Idle:* circle glyph + "Record". Hover → `T_ACCENT_HI`. Press → ×0.9 brightness.
- *Recording:* `T_DANGER`, breathing alpha, square "stop" glyph, label "Stop", and a **live waveform** drawn across the pill or in the strip below.
- Keep it the visual anchor; everything else is quieter.
**Ghost actions (Copy / Paste / Clear).** No resting fill, no border — just a Medium-weight label in `T_DIM`. On hover, a `T_CARD_HI` rounded fill fades in and text lifts to `T_TEXT`; on press, `T_CARD_LO`. Because there's no resting border, there are no hairlines; separation is purely spacing. (Add small 16px line icons before labels for a more finished feel.)
**Pin.** An icon toggle (pin glyph), `T_ACCENT` when active, `T_FAINT` when not. No label needed.
**Selects (mic / model).** Quiet rows: small dim label on top ("Microphone"), value below in `T_TEXT`, a small chevron at the right; hover = `T_CARD_HI` fill. **Consider relocating both behind a small gear/settings affordance** — a dictation utility doesn't need model internals on the main face. If you keep them visible, give them the same fill-on-hover, no-border treatment.
**Status + progress strip (unified).** One horizontal zone under the hero that changes by state:
- *Idle:* `"Ready · 2 threads"` in `T_DIM`.
- *Recording:* live waveform + `mm:ss` timer.
- *Transcribing:* the progress bar (Part 2) with smooth % and a **counting-down** ETA.
**Level / waveform.** Replace the 14-segment VU (reads as "old") with either a smooth antialiased waveform (ring buffer of recent RMS samples drawn as a filled path) or a single breathing level pill. Color `T_GOOD`, riding on `T_CARD_LO`.
**Empty state.** Centered mic glyph + "Your transcription will appear here" in `T_DIM`, drawn *inside* the card (not as a separate STATIC) so it shares the surface.
## 1.6 Rendering scaffold + hit-testing (code)
The whole UI becomes a small list of widgets plus one paint routine and one interaction handler. Skeleton (GDI+, Tier 1):
```cpp
enum class WK { RecordHero, Pin, Copy, Paste, Clear, SelAudio, SelModel, Transcript };
struct Widget {
WK kind;
RectF r; // logical rect, filled by Layout()
bool hover = false;
bool pressed = false;
float anim = 0.0f; // 0..1 eased hover/press amount
};
static Widget g_w[ (int)WK::Transcript + 1 ];
static int g_hot = -1; // index under cursor
static int g_active = -1; // index pressed
// --- one animation clock ---
static DWORD g_lastFrame = 0;
static bool AnyAnimating(); // true if any widget anim is mid-transition, or recording, or busy
// Advance eased states; call from the render timer.
void StepAnimations(float dt) {
for (auto& w : g_w) {
float target = (g_active == (&w - g_w) ) ? 1.0f : (w.hover ? 0.6f : 0.0f);
// ease toward target
w.anim += (target - w.anim) * std::min(1.0f, dt * 12.0f);
}
}
// --- layout: compute rects from client size & DPI scale ---
void Layout(int W, int H, float s /*dpi scale*/);
// --- paint: ONE surface ---
void Paint(HWND hwnd) {
PAINTSTRUCT ps; HDC hdc = BeginPaint(hwnd, &ps);
RECT rc; GetClientRect(hwnd, &rc);
int W = rc.right, H = rc.bottom;
HDC mem = CreateCompatibleDC(hdc);
HBITMAP bmp = CreateCompatibleBitmap(hdc, W, H);
HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
{
Graphics g(mem);
g.SetSmoothingMode(SmoothingModeAntiAlias);
g.SetTextRenderingHint(TextRenderingHintClearTypeGridFit);
SolidBrush bg(T_BG); g.FillRectangle(&bg, 0, 0, W, H);
DrawCardWithShadow(g, g_cardRect, 16); // optional soft shadow + T_CARD fill + T_TOPLIGHT edge
for (auto& w : g_w) {
switch (w.kind) {
case WK::RecordHero: DrawHero(g, w); break;
case WK::Copy: DrawGhost(g, w, L"Copy"); break;
case WK::Paste: DrawGhost(g, w, L"Paste"); break;
case WK::Clear: DrawGhost(g, w, L"Clear"); break;
case WK::Pin: DrawPin(g, w); break;
case WK::SelAudio: DrawSelect(g, w, g_audioVal); break;
case WK::SelModel: DrawSelect(g, w, g_modelVal); break;
case WK::Transcript: /* the EDIT child paints itself; we just leave its inset */ break;
}
}
DrawStatusStrip(g, g_stripRect); // idle / recording waveform / progress
}
BitBlt(hdc, 0, 0, W, H, mem, 0, 0, SRCCOPY);
SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
EndPaint(hwnd, &ps);
}
// --- interaction: hit-test in the parent ---
int HitTest(POINT p) {
for (int i = 0; i < (int)std::size(g_w); ++i)
if (g_w[i].kind != WK::Transcript && g_w[i].r.Contains((REAL)p.x, (REAL)p.y)) return i;
return -1;
}
LRESULT CALLBACK WndProc(HWND h, UINT m, WPARAM w, LPARAM l) {
switch (m) {
case WM_MOUSEMOVE: {
POINT p{ GET_X_LPARAM(l), GET_Y_LPARAM(l) };
int hot = HitTest(p);
if (hot != g_hot) {
if (g_hot >= 0) g_w[g_hot].hover = false;
g_hot = hot;
if (g_hot >= 0) g_w[g_hot].hover = true;
TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 }; TrackMouseEvent(&t);
EnsureAnimating(h);
}
return 0;
}
case WM_MOUSELEAVE:
if (g_hot >= 0) { g_w[g_hot].hover = false; g_hot = -1; EnsureAnimating(h); }
return 0;
case WM_LBUTTONDOWN:
g_active = g_hot;
if (g_active >= 0) { g_w[g_active].pressed = true; SetCapture(h); EnsureAnimating(h); }
return 0;
case WM_LBUTTONUP: {
ReleaseCapture();
POINT p{ GET_X_LPARAM(l), GET_Y_LPARAM(l) };
if (g_active >= 0 && HitTest(p) == g_active) OnClick(h, g_w[g_active].kind);
if (g_active >= 0) g_w[g_active].pressed = false;
g_active = -1; EnsureAnimating(h);
return 0;
}
case WM_ERASEBKGND: return 1; // we paint everything
case WM_PAINT: Paint(h); return 0;
case WM_SIZE: Layout(LOWORD(l), HIWORD(l), g_dpiScale); InvalidateRect(h, nullptr, FALSE); return 0;
// ... WM_TIMER drives StepAnimations + InvalidateRect while AnyAnimating()
}
return DefWindowProc(h, m, w, l);
}
```
Notes:
- `DrawGhost` simply lerps its fill alpha by `w.anim` between transparent → `T_CARD_HI`, and text color between `T_DIM` → `T_TEXT`. No `StrokeRound`. That's the whole trick.
- `EnsureAnimating(h)` starts the 16ms timer if it isn't running; the timer stops itself when `AnyAnimating()` returns false to spare the CPU.
- Keyboard focus (for accessibility / Tab) can be added later by tracking a `g_focus` index and painting a soft focus ring on the focused widget only — still no native chrome.
## 1.7 The transcript field & DPI
**Transcript = the one real child window.** Keep `EDIT` (multiline, read-only) for free selection/caret/scroll/IME, but strip its chrome:
1. **No border:** create without `WS_BORDER`/`WS_EX_CLIENTEDGE` (already the case). To suppress the *themed* edit border entirely, either `SetWindowTheme(hEdit, L"", L"")` (kills the theme, gives a classic flat look) or subclass and handle `WM_NCPAINT` to no-op. Prefer the dark-mode route below so the scrollbar also matches.
2. **Dark background:** you already return `g_brSurface` from `WM_CTLCOLOREDIT`; set it to `T_CARD`/`T_CARD_LO` so the field is invisible against the card.
3. **Dark (or custom) scrollbar — this removes the pale bar in your screenshot:**
- Easiest: enable app dark mode then theme the control:
```cpp
// once, after the process starts (uxtheme, undocumented but widely used):
// AllowDarkModeForApp(true); SetPreferredAppMode(AllowDark);
SetWindowTheme(hEdit, L"DarkMode_Explorer", nullptr); // dark scrollbar
```
- Most control: hide the native scrollbar (`ShowScrollBar(hEdit, SB_VERT, FALSE)` or `WM_NCCALCSIZE`) and **paint a slim custom scrollbar on the parent surface**, driven by `EM_GETFIRSTVISIBLELINE` / line count. Best looking, more work.
4. **Inset it inside the card** by ~14–16px so the card's rounded surface is the visible frame and the EDIT contributes no edges of its own.
**DPI awareness (do this — it's part of "looks good").** Today metrics are fixed pixels; on a HiDPI panel they blur/misalign.
- Declare **Per-Monitor-V2** via the app manifest (preferred) or `SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2)` at startup.
- Compute `g_dpiScale = GetDpiForWindow(hwnd) / 96.0f`; multiply every metric (padding, radii, font sizes, widget sizes) by it.
- Recreate fonts and re-`Layout()` on `WM_DPICHANGED`, and use the suggested rect it passes for repositioning.
## 1.8 Migration order from today's main.cpp
You can do this incrementally without a rewrite:
1. **Stop creating chrome child windows.** Delete the `CreateWindow(L"BUTTON", …)` calls for Record/Pin/Copy/Paste/Clear and the two selects, and the two `STATIC`s. Keep only the `EDIT`.
2. **Add the `Widget` array + `Layout()`** computing the same rectangles your `LayoutControls` used (reuse the math; just store `RectF`s instead of `MoveWindow`-ing HWNDs).
3. **Move your existing draw functions** (`DrawRecordButton`→`DrawHero`, `DrawFlatButton`→`DrawGhost`, `DrawSelect`) to take a `Widget&` and draw into the shared `Graphics&` — and **delete every `StrokeRound(..., T_BORDER/C_BORDER, …)`**. Replace the panel border with the card + optional shadow + toplight.
4. **Route interaction** through `WndProc` hit-testing (Section 1.6). `OnClick(kind)` calls your existing handlers (toggle record, copy, paste, clear, open the popup for selects).
5. **Add the animation clock**; convert hover from per-HWND `GWLP_USERDATA` to `w.anim`.
6. **Theme the EDIT + scrollbar**, inset it, and add DPI scaling.
7. The popup list (`PopupProc`) can stay as-is — it's already a single custom-painted surface and looks consistent.
Result: one surface, zero seams, full control. The "thin lines" cannot come back because nothing draws them and there are no child boundaries to leak them.
---
# Part 2 — The progress system
## 2.1 Why it's broken today
Three separate problems, all visible in your 1:27 example.
**(a) The ETA counts *up*.** In `UpdateStatus`:
```cpp
float elapsed = (GetTickCount() - g_busyStart) / 1000.0f;
float est = elapsed * 100.0f / (float)p; // total, derived from stale p
float remain = est - elapsed; // = elapsed * (100 - p) / p
```
`remain` is recomputed every 50ms, but `p` (whisper's progress) only changes at chunk boundaries. With `p` held constant and `elapsed` rising, `remain = elapsed·(100−p)/p` **increases over time** — the ETA climbs until the next `p` update, then snaps down when `p` jumps. That's precisely "counts up, then jumps to 20s, then counts up again."
**(b) The percentage jumps in big steps.** whisper.cpp calls its progress callback at most **once per 30-second audio chunk**. 1:27 = 87s ≈ **three chunks**, so `p` arrives roughly as `0 → 33 → 67 → 100`. The 34% and 72% you saw are those chunk boundaries (off slightly due to seek rounding). The bar can't be smooth if its only input updates 3 times.
**(c) Dead air at the start.** For 87s of audio the first callback only fires after the *first* 30s chunk finishes decoding — several seconds on a 2-core CPU — so nothing moves at first (you read it as "model loading"). The model is actually already preloaded; it's first-chunk latency with no fallback signal.
**Conclusion:** whisper's callback is a *coarse, occasional measurement*, not a progress source. We need our own continuous prediction, corrected by that measurement.
## 2.2 The plan: predict, then correct
Exactly your idea, formalized:
1. **Predict** total processing time the instant recording stops, from a **history of how long this machine took** for clips of various lengths (per model). This drives a smooth bar from frame 1 — even for sub-30s clips that get *zero* whisper updates.
2. **Correct** that prediction as whisper reports progress: each callback implies a *measured* total time; we fuse it into our estimate with exponential smoothing so accuracy improves **without jumps**.
3. **Display** a strictly **counting-down** remaining time and a **smoothly rising** percent derived from the same model, ease to **95%**, and **snap to 100%** when the real result arrives.
4. **Learn:** on completion, record `(audio_seconds, actual_processing_seconds)` and persist it, so the next prediction is better.
## 2.3 Persistent per-model timing history
Processing time vs audio length is, to first order, **linear**: `proc ≈ a + b·audio`, where `b` is roughly the inverse real-time factor and `a` is fixed overhead. We fit `a, b` per model (tiny.en and base.en behave very differently) with an **online least-squares** accumulator, with a gentle decay so the model adapts to thermal throttling / machine load.
- **Key by model filename** (e.g. `ggml-tiny.en.bin`), since speed is model-dependent.
- **Cold start:** before we have ≥2 samples, use baked-in defaults (rough seeds for a 2-core i5-7th-gen; they self-correct after a run or two):
- tiny.en: `a ≈ 0.3s`, `b ≈ 0.45` (≈2.2× real-time)
- base.en: `a ≈ 0.5s`, `b ≈ 1.1` (≈0.9× real-time)
- (These are only seeds; the regression takes over quickly.)
- **Persist** alongside the existing `win-dictation.ini` using the same `WritePrivateProfileString` style you already use in `settings.h`, one section per model holding the five accumulators.
## 2.4 The live estimator (smooth countdown + fusion)
State: `T_hat` (current best total-time estimate), `disp_rem` (displayed remaining, monotonic), `t` (seconds since start).
- **begin(T_pred):** `T_hat = disp_rem = max(0.4, T_pred)`, `t = 0`.
- **on_whisper(t_now, p):** ignore `p < 5` (noisy). Else measured total `T_meas = 100·t_now / p`; fuse: `T_hat = (1−α)·T_hat + α·T_meas` with `α ≈ 0.5`. This is where whisper "adjusts our countdown" — it moves the estimate, not the displayed number directly, so there's never a visible jump.
- **tick(dt):** the smoothing rules that make it feel solid:
1. Always count down in real time: `disp_rem -= dt`.
2. Pull toward the model's `raw_rem = max(0, T_hat − t)`, but **only ever downward**, and **rate-limited**:
- `err = raw_rem − disp_rem`
- if `err < 0` (we're behind → need to speed up): `disp_rem += max(err, −maxCatchUp·dt)` (bounded extra shrink, no snap)
- if `err ≥ 0` (we have more headroom than shown): **do nothing** — never push remaining up. The bar simply keeps easing and parks near 95% if we under-predicted.
3. Clamp `disp_rem ≥ 0`.
4. Derive fraction from the same numbers: `frac = t / (t + disp_rem)`, clamp to **0.95**. Because `t` only rises and `disp_rem` only falls, `frac` only rises — smooth, monotonic, no jumps.
- **on_result:** snap `frac → 1.0`; record `(audio_seconds, t)` into the timing model and persist.
This guarantees: **ETA only counts down** (bug fixed), **% only rises smoothly** (no 34→72 jumps), whisper's coarse measurements **gently re-aim** the countdown, and there's **motion from frame 1** (no dead start). On a sub-30s clip with no whisper updates, it runs purely on the learned prediction — exactly what you asked for.
## 2.5 `timing.h` — full code
```cpp
#pragma once
#include <windows.h>
#include <string>
#include <cmath>
#include <algorithm>
// ---------------------------------------------------------------------------
// Online linear model: proc_sec ~= a + b * audio_sec, fitted per whisper model.
// Decayed least squares so it adapts to thermal / load drift over time.
// ---------------------------------------------------------------------------
struct TimingModel {
double n=0, sx=0, sy=0, sxx=0, sxy=0; // decayed accumulators
double a=0, b=0; // fitted intercept / slope
bool fitted=false;
double def_a=0.4, def_b=0.6; // cold-start seeds (set per model)
void recompute() {
if (n >= 2.0) {
double denom = n*sxx - sx*sx;
if (std::fabs(denom) > 1e-9) {
double bb = (n*sxy - sx*sy) / denom;
double aa = (sy - bb*sx) / n;
if (bb < 0.02) bb = def_b; // guard against degenerate fits
if (aa < 0.0) aa = 0.0;
a=aa; b=bb; fitted=true; return;
}
}
a=def_a; b=def_b; fitted=false;
}
double predict(double audio_sec) const {
double t = (fitted ? a : def_a) + (fitted ? b : def_b) * audio_sec;
return std::max(0.4, t);
}
void add_sample(double audio_sec, double proc_sec) {
const double decay = 0.97; // ~30-sample memory
n*=decay; sx*=decay; sy*=decay; sxx*=decay; sxy*=decay;
n+=1; sx+=audio_sec; sy+=proc_sec;
sxx+=audio_sec*audio_sec; sxy+=audio_sec*proc_sec;
recompute();
}
};
// ---------------------------------------------------------------------------
// Live estimator: smooth, monotonic countdown fused with whisper's progress.
// ---------------------------------------------------------------------------
struct ProgressEstimator {
double T_hat=1.0, disp_rem=1.0, t=0.0;
bool done=false;
void begin(double T_pred) {
T_hat = std::max(0.4, T_pred);
disp_rem = T_hat; t = 0.0; done=false;
}
void on_whisper(double t_now, int p) { // p in (0,100]
if (done || p < 5) return;
double T_meas = 100.0 * t_now / (double)p;
const double alpha = 0.5; // how much we trust the measurement
T_hat = (1.0-alpha)*T_hat + alpha*T_meas;
if (T_hat < t_now) T_hat = t_now; // never imply we're already done
}
// dt seconds since last tick. Outputs eased fraction [0,1] and remaining secs.
void tick(double dt, float& out_frac, float& out_remaining) {
if (done) { out_frac=1.0f; out_remaining=0.0f; return; }
t += dt;
disp_rem -= dt; // (1) real-time countdown
double raw_rem = std::max(0.0, T_hat - t);
const double maxCatchUp = 2.5; // cap speed-up (×realtime)
double err = raw_rem - disp_rem;
if (err < 0) disp_rem += std::max(err, -maxCatchUp*dt); // (2) shrink only
if (disp_rem < 0) disp_rem = 0; // (3)
double frac = (t + disp_rem > 1e-6) ? t/(t+disp_rem) : 0.0;
if (frac > 0.95) frac = 0.95; // (4) hold until result
out_frac = (float)frac;
out_remaining = (float)disp_rem;
}
void finish(float& out_frac, float& out_remaining) {
done=true; out_frac=1.0f; out_remaining=0.0f;
}
};
// ---------------------------------------------------------------------------
// Persistence (same ini style as settings.h). Section = model base filename.
// ---------------------------------------------------------------------------
inline std::wstring TimingIniPath() {
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
return p + L"\\win-dictation.ini";
}
inline std::wstring SectionFor(const std::string& modelPath) {
std::string base = modelPath.substr(modelPath.find_last_of("\\/")+1);
return L"timing-" + std::wstring(base.begin(), base.end());
}
inline void PutD(const std::wstring& sec, const wchar_t* k, double v) {
wchar_t b[64]; swprintf_s(b, L"%.6f", v);
WritePrivateProfileStringW(sec.c_str(), k, b, TimingIniPath().c_str());
}
inline double GetD(const std::wstring& sec, const wchar_t* k, double d) {
wchar_t b[64]; swprintf_s(b, L"%.6f", d);
wchar_t out[64];
GetPrivateProfileStringW(sec.c_str(), k, b, out, 64, TimingIniPath().c_str());
return wcstod(out, nullptr);
}
inline void LoadTiming(TimingModel& m, const std::string& modelPath) {
auto s = SectionFor(modelPath);
m.n=GetD(s,L"n",0); m.sx=GetD(s,L"sx",0); m.sy=GetD(s,L"sy",0);
m.sxx=GetD(s,L"sxx",0); m.sxy=GetD(s,L"sxy",0);
m.recompute();
}
inline void SaveTiming(const TimingModel& m, const std::string& modelPath) {
auto s = SectionFor(modelPath);
PutD(s,L"n",m.n); PutD(s,L"sx",m.sx); PutD(s,L"sy",m.sy);
PutD(s,L"sxx",m.sxx); PutD(s,L"sxy",m.sxy);
}
// Set per-model cold-start seeds when (re)loading a model.
inline void SeedDefaults(TimingModel& m, const std::string& modelPath) {
std::string p = modelPath;
auto has = [&](const char* s){ return p.find(s)!=std::string::npos; };
if (has("tiny")) { m.def_a=0.3; m.def_b=0.45; }
else if (has("base")) { m.def_a=0.5; m.def_b=1.10; }
else if (has("small")){ m.def_a=0.8; m.def_b=3.00; }
else { m.def_a=0.5; m.def_b=1.00; }
m.recompute();
}
```
## 2.6 Wiring into main.cpp
Add globals and capture the audio length **before** `stop_and_transcribe()` swaps the buffer away:
```cpp
TimingModel g_timing;
ProgressEstimator g_est;
double g_lastAudioLen = 0.0; // seconds of the clip being transcribed
DWORD g_lastTick = 0;
```
**At model load / model switch** (where you set `g_config.model_path`), seed + load history:
```cpp
SeedDefaults(g_timing, g_config.model_path);
LoadTiming(g_timing, g_config.model_path);
```
**On STOP → transcribe** (the `HK_TOGGLE` stop branch and the max-length branch):
```cpp
g_lastAudioLen = g_tx.recorded_seconds(); // BEFORE stop swaps the buffer
g_busyStart = GetTickCount();
g_lastTick = g_busyStart;
g_est.begin(g_timing.predict(g_lastAudioLen)); // bar moves from frame 1
g_progress = 0;
g_cancelRequested = false;
g_tx.stop_and_transcribe();
```
**whisper progress** (`WM_APP_PROGRESS`) becomes a *correction*, not the display source:
```cpp
case WM_APP_PROGRESS: {
double t_now = (GetTickCount() - g_busyStart) / 1000.0;
g_est.on_whisper(t_now, (int)wParam);
InvalidateRect(hWnd, &g_vuRect, FALSE);
return 0;
}
```
**The UI timer** (`WM_TIMER`, while `g_tx.is_busy()`) advances the estimator and paints:
```cpp
DWORD now = GetTickCount();
float dt = (now - g_lastTick) / 1000.0f; g_lastTick = now;
float frac, remain;
g_est.tick(dt, frac, remain);
g_progressFrac = frac; // float 0..1 used by DrawProgress
g_progressRemain = remain; // seconds, for the ETA label
InvalidateRect(hWnd, &g_vuRect, FALSE);
```
**Status text** (replaces the counts-up math entirely):
```cpp
int mm = (int)g_lastAudioLen/60, ss=(int)g_lastAudioLen%60;
int pct = (int)(g_progressFrac*100.0f + 0.5f);
int rem = (int)(g_progressRemain + 0.5f);
swprintf_s(buf, L"Transcribing %d:%02d · %d%% · %ds left", mm, ss, pct, rem);
```
**On result** (`WM_APP_RESULT`): snap, learn, persist:
```cpp
float frac, remain; g_est.finish(frac, remain);
g_progressFrac = 1.0f; g_progressRemain = 0.0f;
double actual = (GetTickCount() - g_busyStart) / 1000.0;
if (g_lastAudioLen > 0.5 && actual > 0.2 && !g_cancelRequested) {
g_timing.add_sample(g_lastAudioLen, actual);
SaveTiming(g_timing, g_config.model_path);
}
```
**`DrawProgress`** already takes a fraction — just feed `g_progressFrac` instead of `g_progress/100.0f`, and optionally add a subtle animated shimmer on the fill for life.
## 2.7 Tuning & edge cases
- **`alpha` (whisper trust)** 0.5 is a good start. Lower (0.3) = smoother but slower to correct; higher (0.7) = snappier, slightly jumpier.
- **`maxCatchUp`** (2.5× real-time) caps how fast the countdown may accelerate when we over-predicted, so a correction never looks like a snap. Raise for faster catch-up, lower for calmer motion.
- **Under-prediction** (transcription takes longer than estimated): `frac` parks at 95% and the ETA sits at a small floor until the result lands — which is the honest, expected behavior.
- **Decay `0.97`** ≈ last ~30 runs dominate. Increase toward 0.99 for steadier long-term averages, decrease for faster adaptation to a throttling machine.
- **Cancel / no-speech:** call `g_est.finish(...)`, reset `g_progressFrac=0`, and **don't** record a sample.
- **Model switch mid-history:** because history is keyed per model, switching tiny.en↔base.en uses the right curve automatically.
- **Sanity clamp:** keep `predict()`'s floor (0.4s) so ultra-short clips still show a brief, graceful sweep rather than instant 100%.
- **Optional richer model:** if you ever want better fits on very short vs long clips, swap the linear `a+b·x` for a two-segment fit (sub-30s vs ≥30s) — the accumulators and API stay the same; just keep two `TimingModel`s.
---
# Part 3 — Cleanup checklist
Smaller items that make the project cleaner and the app feel finished:
- [ ] **Kill the pale scrollbar** (dark-mode theme or custom slim scrollbar) — Section 1.7.
- [ ] **Delete all owner-draw chrome child windows**; keep only the transcript `EDIT` — Section 1.8.
- [ ] **Remove every `StrokeRound(..., C_BORDER, …)`**; rely on fills + elevation.
- [ ] **One animation clock** that idles when nothing moves (protect the 2-core CPU).
- [ ] **DPI Per-Monitor-V2** + scaled metrics + font reload on `WM_DPICHANGED`.
- [ ] **Reconcile the docs with reality.** `README.md`, `src/README.md`, and `src/CHANGES.md` still describe the *old* streaming architecture — "ring buffer", "24 threads", "VAD", "step_ms/length_ms", "<1s real-time GPU". The app is now **push-to-talk batch, CPU, physical-core threads, `whisper_full` once on stop**. Update or archive those docs so future-you isn't misled. (The `CUDA-SETUP.md` / `QUICK-REBUILD-GPU.md` RTX-3090 guides don't apply to the target Dell i5 either.)
- [ ] **Status copy:** "Ready · N threads" is good; make the idle/recording/transcribing strings come from one place.
- [ ] **Remove dead members** once streaming is gone (any leftover `step_ms`/`length_ms`/VAD config that no longer feeds `whisper_full`).
---
# Part 4 — Suggested build order
Do them in this sequence so each step is verifiable on its own:
1. **Progress system first** (Part 2). It's self-contained, low-risk, and immediately fixes the most visible "is it even working?" problem. You'll see a smooth countdown the same day.
2. **DPI + tokens** (1.4 / 1.7). Small, mechanical, and everything after looks better for it.
3. **Single-surface conversion** (1.6 / 1.8): convert one widget at a time — start with the ghost buttons (highest hairline payoff), then the hero, then the selects, then retire the STATICs.
4. **Transcript chrome + scrollbar** (1.7).
5. **Motion polish** (1.4): hover cross-fades, recording breathing, waveform, progress shimmer.
6. **Docs reconciliation** (Part 3).
7. *(Optional later)* **Direct2D/DirectWrite** (1.3) if you want the premium ceiling — the data model and Part 2 carry over unchanged.
---
*Build target reminder: native Win32 C++, MSVC Release, CPU-only on a 2-core / 4-thread i5-7th-gen. Keep the idle CPU near zero — animate only when something is actually moving.*
Win-Dictation-features-01.md
+1084
View File
File diff suppressed because it is too large Load Diff
bug-fixes-and-visual-upgrade.md
+790
View File
@@ -0,0 +1,790 @@
# Win Dictation — Crash Fixes & Modern UI (Build 2)
This document has two parts:
- **Part 1 — Critical fixes** (do these first): the Stop crash, resizing, model switching, thread count, model path.
- **Part 2 — Modern UI**: replace the boxy Win32 look with a flat, rounded, dark "web-app" interface (GDI+ custom drawing).
Notes before you start:
- All snippets target your current `main.cpp` / `transcriber.cpp` / `transcriber.h`.
- I can't compile on my side — integrate a piece at a time and build often. If a `DWMWA_*` or glyph constant is missing, see the `#define` block in §2.3.
- Part 2's `LayoutControls()` (§2.12) is the final layout and supersedes any basic one from Part 1.
---
# Part 1 — Critical Fixes
## 1.1 The Stop crash (null Whisper context)
**Symptom:** record + VU work, then it crashes the moment you hit Stop.
**Root cause:** two defects combine.
1. In `wWinMain`, the preload thread sets `g_modelLoaded = true` *even when `preload()` returns false* (model file not found). So the UI shows "Ready" and lets you record with no model loaded.
2. `transcribe_worker()` calls `whisper_reset_timings(m_ctx)` **before** the `if (m_ctx)` check. With `m_ctx == nullptr` that's an immediate access violation. Recording never touches Whisper (that's why capture + VU work) — the crash only fires when transcription starts.
The empty model dropdown in your screenshot confirms it: `RefreshModelList()` found no model files, so `preload()` failed silently.
### Fix A — make the worker null-safe (`transcriber.cpp`)
Replace the top of `transcribe_worker` and delete the `whisper_reset_timings` call entirely (it isn't needed for one-shot transcription):
```cpp
void Transcriber::transcribe_worker(std::vector<float> audio) {
// Hard guard: if the model failed to load, never touch whisper.
if (!m_ctx) {
m_busy = false;
if (m_on_result) m_on_result(""); // UI will show "No speech / not loaded"
return;
}
if (m_cfg.trim_silence) trim_silence(audio);
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wp.print_progress = false; wp.print_realtime = false; wp.print_timestamps = false;
wp.no_timestamps = true; wp.translate = false;
wp.language = m_cfg.language.c_str();
wp.n_threads = m_cfg.n_threads;
wp.no_context = true; wp.suppress_blank = true; wp.suppress_nst = true;
wp.temperature = 0.0f;
// (whisper_reset_timings removed — it was the crash site and is unnecessary)
std::string out;
if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
int n = whisper_full_n_segments(m_ctx);
for (int i = 0; i < n; ++i) {
const char* t = whisper_full_get_segment_text(m_ctx, i);
if (t) out += t;
}
out = clean_text(out);
}
m_busy = false;
if (m_on_result) m_on_result(out);
}
```
### Fix B — track load success honestly (`main.cpp`)
Add an atomic next to `g_modelLoaded`:
```cpp
std::atomic<bool> g_modelLoaded{false};
std::atomic<bool> g_modelOk{false};
```
Fix the preload lambda so failure is recorded:
```cpp
std::thread([] {
bool ok = g_tx.preload(g_config); // false if the .bin isn't found
g_modelOk = ok;
g_modelLoaded = true;
}).detach();
```
### Fix C — refuse to record without a model (`main.cpp`, `WM_HOTKEY`)
This both prevents the crash path and tells you *why* nothing happens:
```cpp
case WM_HOTKEY:
if (wParam == HK_TOGGLE) {
if (g_tx.is_busy()) break; // mid-transcription: ignore
if (!g_tx.is_recording()) {
if (!g_modelLoaded.load()) { SetStatus(hWnd, L"Loading model…"); break; }
if (!g_modelOk.load()) {
std::wstring m = L"Model not found:\n" + to_w(g_config.model_path)
+ L"\n\nPut the .bin there and restart.";
MessageBoxW(hWnd, m.c_str(), L"Dictation", MB_OK | MB_ICONWARNING);
break;
}
g_prevForeground = GetForegroundWindow();
ShowWindow(hWnd, SW_SHOWNA);
g_recordingSecs = 0;
if (g_tx.start_recording()) {
SetDlgItemText(hWnd, ID_BTN_RECORD, L"■ Stop");
SetStatus(hWnd, L"Recording…");
} else SetStatus(hWnd, L"Microphone error");
} else {
g_tx.stop_and_transcribe();
SetDlgItemText(hWnd, ID_BTN_RECORD, L"Record");
SetStatus(hWnd, L"Transcribing…");
}
} else if (wParam == HK_HIDE) {
if (g_tx.is_recording()) g_tx.cancel();
ShowWindow(hWnd, SW_HIDE);
}
break;
```
After this, a missing model shows **"Model not found: …\models\ggml-tiny.en.bin"** instead of crashing.
## 1.2 Verify the model file
The app now loads the model from a path relative to the **.exe**:
`exe_dir() + "\\models\\ggml-tiny.en.bin"`. Confirm it's actually there:
```
dir build\bin\Release\models
```
You should see `ggml-tiny.en.bin`. If it's missing, re-run `build.ps1`, or drop the `.bin` into that `models\` folder manually. (Your CMake post-build step copies `models/` next to the exe — if the source `models/` had no `.bin` at build time, nothing got copied.)
## 1.3 Resizable window
Two reasons it's frozen: the window style has no sizing border, and there's no `WM_SIZE` handler so controls never reflow.
### Window style (`main.cpp`, `CreateWindowExW`)
> If you do Part 2 Option A (§2.3), keep a normal frame as shown here. If you do the borderless Option B (appendix), that section replaces this.
```cpp
hMainWnd = CreateWindowExW(
WS_EX_TOPMOST,
L"WhisperDictationClass", L"Dictation",
WS_OVERLAPPEDWINDOW, // caption + sysmenu + THICKFRAME + min/max = resizable
CW_USEDEFAULT, CW_USEDEFAULT, 400, 340,
nullptr, nullptr, hInstance, nullptr);
```
### Handlers (`main.cpp`, `WndProc`)
```cpp
case WM_SIZE:
LayoutControls(hWnd, LOWORD(lParam), HIWORD(lParam)); // defined in §2.12
return 0;
case WM_GETMINMAXINFO:
((MINMAXINFO*)lParam)->ptMinTrackSize.x = 340;
((MINMAXINFO*)lParam)->ptMinTrackSize.y = 280;
return 0;
```
Control positions in `InitializeUI` no longer matter — `LayoutControls` owns geometry. Add one explicit call right after `ShowWindow` so the first frame is laid out:
```cpp
RECT rc; GetClientRect(hMainWnd, &rc);
LayoutControls(hMainWnd, rc.right, rc.bottom);
```
## 1.4 Model switching + correct thread count
Two real bugs you'll hit:
- The model dropdown only lists files that **exist**, so its selected index does **not** map to `kModelFiles[]`.
- `preload()` early-returns when `m_ctx` is already set, so switching models **never reloads**.
- The status bar shows `hardware_concurrency()` (4) instead of the threads actually used (2).
### transcriber.h — add a reload + a threads getter
```cpp
bool reload(const WhisperConfig& cfg); // free + re-init with a new model
int threads() const { return m_cfg.n_threads; }
```
### transcriber.cpp — implement reload
```cpp
bool Transcriber::reload(const WhisperConfig& cfg) {
if (m_recording.load() || m_busy.load()) return false; // not mid-use
if (m_worker.joinable()) m_worker.join();
if (m_ctx) { whisper_free(m_ctx); m_ctx = nullptr; }
return preload(cfg);
}
```
### main.cpp — track the real file paths the combo shows
Replace the static arrays + `RefreshModelList` + the `ID_COMBO_MODEL` handler:
```cpp
static const wchar_t* kModelNames[] = { L"tiny.en", L"tiny.en-q8_0", L"base.en-q5_1", L"base.en" };
static const char* kModelFiles[] = {
"models\\ggml-tiny.en.bin", "models\\ggml-tiny.en-q8_0.bin",
"models\\ggml-base.en-q5_1.bin", "models\\ggml-base.en.bin",
};
std::vector<std::string> g_modelComboPaths; // parallel to combo entries
void RefreshModelList(HWND hwnd) {
HWND hCombo = GetDlgItem(hwnd, ID_COMBO_MODEL);
SendMessage(hCombo, CB_RESETCONTENT, 0, 0);
g_modelComboPaths.clear();
std::string dir = exe_dir();
for (int i = 0; i < 4; ++i) {
std::string full = dir + "\\" + kModelFiles[i];
if (GetFileAttributesA(full.c_str()) != INVALID_FILE_ATTRIBUTES) {
SendMessage(hCombo, CB_ADDSTRING, 0, (LPARAM)kModelNames[i]);
g_modelComboPaths.push_back(kModelFiles[i]); // remember the real file
}
}
if (!g_modelComboPaths.empty()) SendMessage(hCombo, CB_SETCURSEL, 0, 0);
}
```
```cpp
case ID_COMBO_MODEL:
if (HIWORD(wParam) == CBN_SELCHANGE) {
int idx = (int)SendMessage(GetDlgItem(hWnd, ID_COMBO_MODEL), CB_GETCURSEL, 0, 0);
if (idx >= 0 && idx < (int)g_modelComboPaths.size()) {
g_config.model_path = exe_dir() + "\\" + g_modelComboPaths[idx];
g_modelLoaded = false; g_modelOk = false;
SetStatus(hWnd, L"Loading model…");
std::thread([] {
bool ok = g_tx.reload(g_config); // <- reload, not preload
g_modelOk = ok; g_modelLoaded = true;
}).detach();
}
}
break;
```
### Correct thread count in the status (`UpdateStatus`)
```cpp
} else if (g_modelLoaded.load()) {
if (!g_modelOk.load()) { SetStatus(hwnd, L"Model not found — check models folder"); return; }
swprintf_s(buf, L"Ready • %d threads", g_tx.threads());
SetStatus(hwnd, buf);
}
```
---
# Part 2 — Modern UI (flat, rounded, dark)
## 2.1 Design language
The "boxy" feeling comes from three things: the gray 3-D system buttons, the chunky title bar, and zero breathing room. We fix all three.
- **Palette** — near-black canvas, one elevated surface tone, a single indigo accent, hairline borders. (Linear/Raycast register.)
- **Shape** — everything rounded: 10px buttons/fields, 14px panels, rounded window corners.
- **Type** — Segoe UI Variable / Segoe UI, a clear size hierarchy, no monospace for the transcript (sans reads more "app").
- **Space** — 16px outer margin, 10–12px gaps, the transcript is the hero and grows with the window.
- **Motion** — a soft pulse on the record button while recording; subtle hover/press on buttons.
| Token | Hex | Use |
|---|---|---|
| Bg | `#0F1115` | window canvas |
| Surface | `#181B22` | fields, transcript panel |
| SurfaceHi | `#20242D` | hover |
| Border | `#262B36` | 1px hairlines |
| Text | `#E7E9EE` | primary text |
| TextDim | `#9AA0AB` | status, placeholders |
| Accent | `#6E8BFF` | record idle, focus |
| AccentHi | `#5B7BFF` | accent hover |
| Danger | `#FF5C5C` | recording / stop |
| Good | `#46D39A` | VU meter |
## 2.2 How it's built
- **GDI+** does the drawing — it anti-aliases rounded rectangles and supports alpha, so we get smooth corners without Direct2D.
- **Owner-drawn child buttons** for Record / Copy / Paste / Pin (we paint them; Windows still gives us click + focus).
- **Parent-painted** background, transcript panel frame, and the VU meter (in `WM_PAINT`).
- **Native EDIT** stays for the transcript (so selection/scroll work) but flat-themed with padding; a STATIC shows placeholder text when empty.
- **Owner-drawn comboboxes** for mic/model so they match the dark theme.
- A tiny **button subclass** gives reliable hover.
This keeps real controls (accessibility, IME, scrolling) while looking custom.
## 2.3 Setup: GDI+ and a dark, rounded window (Option A — recommended)
Option A keeps the native frame but recolors it dark and rounds the corners — low-risk and modern. (Option B, fully borderless custom title bar, is in the appendix.)
At the top of `main.cpp`:
```cpp
#include <gdiplus.h>
#pragma comment(lib, "gdiplus.lib")
using namespace Gdiplus;
// DWM attributes (define in case your SDK headers are older)
#ifndef DWMWA_USE_IMMERSIVE_DARK_MODE
#define DWMWA_USE_IMMERSIVE_DARK_MODE 20
#endif
#ifndef DWMWA_BORDER_COLOR
#define DWMWA_BORDER_COLOR 34
#endif
#ifndef DWMWA_CAPTION_COLOR
#define DWMWA_CAPTION_COLOR 35
#endif
#ifndef DWMWA_TEXT_COLOR
#define DWMWA_TEXT_COLOR 36
#endif
#ifndef DWMWA_WINDOW_CORNER_PREFERENCE
#define DWMWA_WINDOW_CORNER_PREFERENCE 33
#endif
#ifndef DWMWCP_ROUND
#define DWMWCP_ROUND 2
#endif
```
Start/stop GDI+ in `wWinMain`:
```cpp
ULONG_PTR g_gdipToken = 0;
// near the top of wWinMain, before creating the window:
GdiplusStartupInput gdipIn;
GdiplusStartup(&g_gdipToken, &gdipIn, nullptr);
// ... after the message loop, before return:
GdiplusShutdown(g_gdipToken);
```
After the window is created, theme the frame (Win11 honors all of these; Win10 ignores caption/border color but keeps dark mode):
```cpp
BOOL dark = TRUE;
DwmSetWindowAttribute(hMainWnd, DWMWA_USE_IMMERSIVE_DARK_MODE, &dark, sizeof(dark));
COLORREF cap = RGB(0x0F,0x11,0x15), bord = RGB(0x26,0x2B,0x36), txt = RGB(0xE7,0xE9,0xEE);
DwmSetWindowAttribute(hMainWnd, DWMWA_CAPTION_COLOR, &cap, sizeof(cap));
DwmSetWindowAttribute(hMainWnd, DWMWA_BORDER_COLOR, &bord, sizeof(bord));
DwmSetWindowAttribute(hMainWnd, DWMWA_TEXT_COLOR, &txt, sizeof(txt));
int corner = DWMWCP_ROUND;
DwmSetWindowAttribute(hMainWnd, DWMWA_WINDOW_CORNER_PREFERENCE, &corner, sizeof(corner));
```
## 2.4 Theme constants + fonts (`main.cpp`)
```cpp
// GDI+ colors
static const Color C_BG (255,0x0F,0x11,0x15);
static const Color C_SURFACE (255,0x18,0x1B,0x22);
static const Color C_SURFACEHI(255,0x20,0x24,0x2D);
static const Color C_BORDER (255,0x26,0x2B,0x36);
static const Color C_TEXT (255,0xE7,0xE9,0xEE);
static const Color C_TEXTDIM (255,0x9A,0xA0,0xAB);
static const Color C_ACCENT (255,0x6E,0x8B,0xFF);
static const Color C_ACCENTHI (255,0x5B,0x7B,0xFF);
static const Color C_DANGER (255,0xFF,0x5C,0x5C);
static const Color C_GOOD (255,0x46,0xD3,0x9A);
// COLORREF mirrors for GDI (WM_CTLCOLOR*)
#define CR_BG RGB(0x0F,0x11,0x15)
#define CR_SURFACE RGB(0x18,0x1B,0x22)
#define CR_TEXT RGB(0xE7,0xE9,0xEE)
HFONT g_fUI=nullptr, g_fUISemi=nullptr, g_fSmall=nullptr, g_fText=nullptr;
HBRUSH g_brBg=nullptr, g_brSurface=nullptr;
static HFONT MakeFont(int px, int weight) {
return CreateFontW(-px,0,0,0,weight,FALSE,FALSE,FALSE,DEFAULT_CHARSET,
OUT_DEFAULT_PRECIS,CLIP_DEFAULT_PRECIS,CLEARTYPE_QUALITY,
DEFAULT_PITCH|FF_DONTCARE,L"Segoe UI Variable Display");
}
// build once in wWinMain:
// g_fUI = MakeFont(15, FW_NORMAL); g_fUISemi = MakeFont(15, FW_SEMIBOLD);
// g_fSmall = MakeFont(12, FW_NORMAL); g_fText = MakeFont(16, FW_NORMAL);
// g_brBg = CreateSolidBrush(CR_BG); g_brSurface = CreateSolidBrush(CR_SURFACE);
// (If "Segoe UI Variable Display" is unavailable it falls back to Segoe UI.)
```
## 2.5 GDI+ helpers
```cpp
static void RoundPath(GraphicsPath& p, const Rect& r, int rad) {
int d = rad * 2;
p.AddArc(r.X, r.Y, d, d, 180, 90);
p.AddArc(r.GetRight()-d, r.Y, d, d, 270, 90);
p.AddArc(r.GetRight()-d, r.GetBottom()-d, d, d, 0, 90);
p.AddArc(r.X, r.GetBottom()-d, d, d, 90, 90);
p.CloseFigure();
}
static void FillRound(Graphics& g, const Color& c, const Rect& r, int rad) {
GraphicsPath p; RoundPath(p, r, rad); SolidBrush b(c); g.FillPath(&b, &p);
}
static void StrokeRound(Graphics& g, const Color& c, const Rect& r, int rad, REAL w=1.0f) {
GraphicsPath p; RoundPath(p, r, rad); Pen pen(c, w); g.DrawPath(&pen, &p);
}
static void DrawTextC(Graphics& g, const wchar_t* s, const Font& f, const Color& c,
const RectF& box, StringAlignment h, StringAlignment v) {
StringFormat sf; sf.SetAlignment(h); sf.SetLineAlignment(v);
sf.SetTrimming(StringTrimmingEllipsisCharacter);
SolidBrush b(c); g.DrawString(s, -1, &f, box, &sf, &b);
}
```
## 2.6 Reliable hover (button subclass)
Owner-draw buttons don't reliably get hover. This subclass tracks it and stores a 0/1 hover flag in the window's user data, repainting on enter/leave.
```cpp
LRESULT CALLBACK BtnProc(HWND h, UINT m, WPARAM w, LPARAM l, UINT_PTR, DWORD_PTR) {
if (m == WM_MOUSEMOVE) {
if (!GetWindowLongPtr(h, GWLP_USERDATA)) {
SetWindowLongPtr(h, GWLP_USERDATA, 1);
TRACKMOUSEEVENT t{ sizeof(t), TME_LEAVE, h, 0 };
TrackMouseEvent(&t);
InvalidateRect(h, nullptr, FALSE);
}
} else if (m == WM_MOUSELEAVE) {
SetWindowLongPtr(h, GWLP_USERDATA, 0);
InvalidateRect(h, nullptr, FALSE);
}
return DefSubclassProc(h, m, w, l); // needs <commctrl.h> (already included)
}
// after creating each owner-draw button:
// SetWindowSubclass(hBtn, BtnProc, 1, 0);
```
(Add `#pragma comment(lib, "comctl32.lib")` is already present; `SetWindowSubclass` lives in `<commctrl.h>`.)
## 2.7 The record button (hero, pulsing)
`ID_BTN_RECORD` is an owner-draw button drawn as a rounded pill: accent "● Record" when idle, danger "■ Stop" with a soft brightness pulse while recording. The dot/square are drawn shapes (no icon-font dependency).
```cpp
void DrawRecordButton(LPDRAWITEMSTRUCT d) {
Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
Rect rc(d->rcItem.left, d->rcItem.top,
d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
// paint the parent canvas in the corners first
SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
bool rec = g_tx.is_recording();
bool hover = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
bool pressed = (d->itemState & ODS_SELECTED) != 0;
Color fill = rec ? C_DANGER : (hover ? C_ACCENTHI : C_ACCENT);
if (rec) { // gentle pulse
double ph = (GetTickCount() % 1400) / 1400.0;
int add = (int)(18 * (0.5 + 0.5 * sin(ph * 6.2831853)));
fill = Color(255, min(255,0xFF), min(255,0x5C+add), min(255,0x5C+add));
}
if (pressed) fill = Color(255, GetRValue(0)+ (BYTE)(fill.GetR()*0.85),
(BYTE)(fill.GetG()*0.85), (BYTE)(fill.GetB()*0.85));
Rect pill = rc; pill.Inflate(-1,-1);
FillRound(g, fill, pill, pill.Height/2); // full pill
// icon: filled circle (idle) or rounded square (recording)
int cx = pill.X + 22, cy = pill.Y + pill.Height/2;
SolidBrush white(Color(255,255,255,255));
if (rec) { Rect sq(cx-7, cy-7, 14, 14); FillRound(g, Color(255,255,255,255), sq, 3); }
else { g.FillEllipse(&white, cx-7, cy-7, 14, 14); }
Font f(d->hDC, g_fUISemi);
RectF tb((REAL)(pill.X+36), (REAL)pill.Y, (REAL)(pill.Width-44), (REAL)pill.Height);
DrawTextC(g, rec ? L"Stop" : L"Record", f, Color(255,255,255,255),
tb, StringAlignmentNear, StringAlignmentCenter);
}
```
While recording, repaint it each tick so the pulse animates — in `WM_TIMER`:
```cpp
if (g_tx.is_recording()) InvalidateRect(GetDlgItem(hWnd, ID_BTN_RECORD), nullptr, FALSE);
```
## 2.8 Flat buttons (Copy / Paste / Pin)
Generic owner-draw for secondary buttons — a flat surface chip that lifts on hover, accent text. Pin shows a filled dot when active.
```cpp
void DrawFlatButton(LPDRAWITEMSTRUCT d, const wchar_t* label, bool active=false) {
Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
Rect rc(d->rcItem.left, d->rcItem.top,
d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
SolidBrush bg(C_BG); g.FillRectangle(&bg, rc);
bool hover = GetWindowLongPtr(d->hwndItem, GWLP_USERDATA) != 0;
bool pressed = (d->itemState & ODS_SELECTED) != 0;
Rect chip = rc; chip.Inflate(-1,-1);
FillRound(g, pressed ? C_SURFACE : (hover ? C_SURFACEHI : C_SURFACE), chip, 10);
StrokeRound(g, C_BORDER, chip, 10, 1.0f);
Font f(d->hDC, g_fUI);
Color tc = active ? C_ACCENT : C_TEXT;
RectF tb((REAL)chip.X, (REAL)chip.Y, (REAL)chip.Width, (REAL)chip.Height);
DrawTextC(g, label, f, tc, tb, StringAlignmentCenter, StringAlignmentCenter);
}
```
In `WM_DRAWITEM`, route each control:
```cpp
case WM_DRAWITEM: {
LPDRAWITEMSTRUCT d = (LPDRAWITEMSTRUCT)lParam;
switch (d->CtlID) {
case ID_BTN_RECORD: DrawRecordButton(d); return TRUE;
case ID_BTN_COPY: DrawFlatButton(d, L"Copy"); return TRUE;
case ID_BTN_PASTE: DrawFlatButton(d, L"Paste"); return TRUE;
case ID_BTN_PIN: DrawFlatButton(d, g_pinned ? L"Pinned" : L"Pin", g_pinned); return TRUE;
case ID_COMBO_AUDIO:
case ID_COMBO_MODEL: DrawCombo(d); return TRUE; // §2.9
}
break;
}
```
Make Copy/Paste/Pin owner-draw too (add `BS_OWNERDRAW`) and subclass them for hover:
```cpp
// in InitializeUI, create with BS_OWNERDRAW, then:
SetWindowSubclass(hCopy, BtnProc, 1, 0);
SetWindowSubclass(hPaste, BtnProc, 1, 0);
SetWindowSubclass(hPin, BtnProc, 1, 0);
SetWindowSubclass(GetDlgItem(hwnd, ID_BTN_RECORD), BtnProc, 1, 0);
```
## 2.9 Mic & model selectors (dark combos)
Make both combos `CBS_DROPDOWNLIST | CBS_OWNERDRAWFIXED | CBS_HASSTRINGS`. Set row height once, then draw field + list items dark with a drawn chevron.
```cpp
case WM_MEASUREITEM: {
auto* mi = (LPMEASUREITEMSTRUCT)lParam;
if (mi->CtlType == ODT_COMBOBOX) mi->itemHeight = 26;
return TRUE;
}
```
```cpp
void DrawCombo(LPDRAWITEMSTRUCT d) {
Graphics g(d->hDC); g.SetSmoothingMode(SmoothingModeAntiAlias);
Rect rc(d->rcItem.left, d->rcItem.top,
d->rcItem.right-d->rcItem.left, d->rcItem.bottom-d->rcItem.top);
bool inField = (d->itemState & ODS_COMBOBOXEDIT) != 0; // the closed field
bool sel = (d->itemState & ODS_SELECTED) != 0;
SolidBrush bg(inField ? C_SURFACE : (sel ? C_SURFACEHI : C_SURFACE));
g.FillRectangle(&bg, rc);
if (inField) { Rect b=rc; b.Inflate(-1,-1); StrokeRound(g, C_BORDER, b, 9); }
wchar_t txt[256] = L"";
if ((int)d->itemID >= 0)
SendMessageW(d->hwndItem, CB_GETLBTEXT, d->itemID, (LPARAM)txt);
Font f(d->hDC, g_fUI);
RectF tb((REAL)rc.X+10, (REAL)rc.Y, (REAL)(rc.Width-28), (REAL)rc.Height);
DrawTextC(g, txt, f, C_TEXT, tb, StringAlignmentNear, StringAlignmentCenter);
if (inField) { // chevron
int cx = rc.X + rc.Width - 16, cy = rc.Y + rc.Height/2;
Pen pen(C_TEXTDIM, 1.6f);
g.DrawLine(&pen, cx-4, cy-2, cx, cy+2);
g.DrawLine(&pen, cx, cy+2, cx+4, cy-2);
}
}
```
Keep your existing `WM_CTLCOLORLISTBOX` returning the surface brush so the dropdown list background is dark too.
> Minimal-by-default option: hide both combos and reveal them only when a small "settings" toggle in the header is clicked — so the resting UI is just record + transcript. Skip if you'd rather keep them always visible.
## 2.10 Transcript field (flat panel + placeholder)
Keep the native multiline EDIT but make it flat: no client edge, surface background, light text, inner padding. Draw a rounded surface panel behind it in `WM_PAINT`; inset the EDIT a few px inside that panel so the rounded corners show.
Create it without `WS_EX_CLIENTEDGE` (flat) and give it inner margins:
```cpp
HWND hEdit = CreateWindowExW(0, L"EDIT", L"",
WS_CHILD | WS_VISIBLE | WS_VSCROLL | ES_MULTILINE | ES_AUTOVSCROLL | ES_READONLY,
0,0,0,0, hwnd, (HMENU)ID_EDIT_TEXT, hInst, nullptr);
SendMessage(hEdit, WM_SETFONT, (WPARAM)g_fText, TRUE);
SendMessage(hEdit, EM_SETMARGINS, EC_LEFTMARGIN|EC_RIGHTMARGIN, MAKELONG(10,10));
```
Theme it (you already have `WM_CTLCOLOREDIT` — point it at the surface):
```cpp
case WM_CTLCOLOREDIT: {
HDC hdc=(HDC)wParam; SetTextColor(hdc, CR_TEXT); SetBkColor(hdc, CR_SURFACE);
return (LRESULT)g_brSurface;
}
```
Placeholder via a STATIC shown only when empty (`ID_STATIC_PLACEHOLDER`), created after the edit so it sits on top:
```cpp
CreateWindowW(L"STATIC", L"Your transcription will appear here…",
WS_CHILD | WS_VISIBLE | SS_LEFT, 0,0,0,0, hwnd,
(HMENU)ID_STATIC_PLACEHOLDER, hInst, nullptr);
// font g_fText; color via WM_CTLCOLORSTATIC -> C_TEXTDIM on C_SURFACE
// toggle it:
void UpdatePlaceholder(HWND hwnd) {
bool empty = GetWindowTextLengthW(GetDlgItem(hwnd, ID_EDIT_TEXT)) == 0;
ShowWindow(GetDlgItem(hwnd, ID_STATIC_PLACEHOLDER), empty ? SW_SHOW : SW_HIDE);
}
// call after setting/clearing transcript text (WM_APP_RESULT, Clear).
```
## 2.11 Custom VU meter
Drop the `PROGRESS_CLASS` control; draw a row of rounded segments in `WM_PAINT` that light up green with the input level. Store its rect in a global from `LayoutControls`.
```cpp
RECT g_vuRect = {0,0,0,0};
float g_energy = 0.0f; // updated in WM_TIMER from g_tx.get_audio_energy()
void DrawVU(Graphics& g, const RECT& r, float level) {
const int N = 14, gap = 3;
int w = (r.right-r.left); int segW = (w - gap*(N-1)) / N;
int h = r.bottom - r.top;
int lit = (int)(level * N + 0.5f);
for (int i = 0; i < N; ++i) {
int x = r.left + i*(segW+gap);
Rect seg(x, r.top, segW, h);
Color c = (i < lit) ? C_GOOD : C_SURFACEHI;
FillRound(g, c, seg, 2);
}
}
```
In `WM_TIMER`, refresh the level and invalidate just the VU rect:
```cpp
g_energy = g_tx.is_recording() ? g_tx.get_audio_energy() : 0.0f;
InvalidateRect(hWnd, &g_vuRect, FALSE);
```
## 2.12 Responsive layout (final)
This is the single source of truth for geometry — referenced by §1.3.
```cpp
void LayoutControls(HWND h, int W, int H) {
const int M = 16, row = 36, gap = 10;
int x = M, y = M, innerW = W - 2*M;
// Row 1: Record (left, prominent) + Pin (right)
int pinW = 78, recW = innerW - pinW - gap;
MoveWindow(GetDlgItem(h, ID_BTN_RECORD), x, y, recW, row, TRUE);
MoveWindow(GetDlgItem(h, ID_BTN_PIN), x + recW + gap, y, pinW, row, TRUE);
// Row 2: VU meter (custom-painted) — reserve a slim strip
y += row + gap;
g_vuRect = { x, y, x + innerW, y + 8 };
// Row 3: status line
y += 8 + gap;
MoveWindow(GetDlgItem(h, ID_STATIC_STATUS), x, y, innerW, 18, TRUE);
// Row 4: transcript panel (hero) — fills the middle
y += 18 + gap;
int bottom = (row + gap) * 2; // selectors row + copy/paste row
int panelTop = y;
int textH = H - y - M - bottom;
if (textH < 70) textH = 70;
// EDIT inset 10px inside the rounded panel drawn in WM_PAINT
MoveWindow(GetDlgItem(h, ID_EDIT_TEXT), x+10, panelTop+10, innerW-20, textH-20, TRUE);
MoveWindow(GetDlgItem(h, ID_STATIC_PLACEHOLDER), x+16, panelTop+16, innerW-32, 22, TRUE);
// Row 5: mic + model selectors
y += textH + gap;
int halfW = (innerW - gap) / 2;
MoveWindow(GetDlgItem(h, ID_COMBO_AUDIO), x, y, halfW, row, TRUE);
MoveWindow(GetDlgItem(h, ID_COMBO_MODEL), x + halfW + gap, y, halfW, row, TRUE);
// Row 6: Copy + Paste
y += row + gap;
MoveWindow(GetDlgItem(h, ID_BTN_COPY), x, y, halfW, row, TRUE);
MoveWindow(GetDlgItem(h, ID_BTN_PASTE), x + halfW + gap, y, halfW, row, TRUE);
// remember the transcript panel rect for WM_PAINT
extern RECT g_panelRect; g_panelRect = { x, panelTop, x+innerW, panelTop+textH };
InvalidateRect(h, nullptr, FALSE);
}
```
## 2.13 Putting it together (paint + flicker-free)
Flicker-free background and the painted bits (canvas, transcript panel frame, VU):
```cpp
RECT g_panelRect = {0,0,0,0};
case WM_ERASEBKGND: return 1; // we paint everything in WM_PAINT
case WM_PAINT: {
PAINTSTRUCT ps; HDC hdc = BeginPaint(hWnd, &ps);
RECT rc; GetClientRect(hWnd, &rc);
// double buffer
HDC mem = CreateCompatibleDC(hdc);
HBITMAP bmp = CreateCompatibleBitmap(hdc, rc.right, rc.bottom);
HBITMAP old = (HBITMAP)SelectObject(mem, bmp);
{
Graphics g(mem); g.SetSmoothingMode(SmoothingModeAntiAlias);
SolidBrush bg(C_BG); g.FillRectangle(&bg, 0,0,rc.right,rc.bottom);
// transcript panel
Rect panel(g_panelRect.left, g_panelRect.top,
g_panelRect.right-g_panelRect.left, g_panelRect.bottom-g_panelRect.top);
FillRound(g, C_SURFACE, panel, 14);
StrokeRound(g, C_BORDER, panel, 14, 1.0f);
// VU
DrawVU(g, g_vuRect, g_energy);
}
BitBlt(hdc, 0,0, rc.right, rc.bottom, mem, 0,0, SRCCOPY);
SelectObject(mem, old); DeleteObject(bmp); DeleteDC(mem);
EndPaint(hWnd, &ps);
return 0;
}
```
> Child controls (EDIT, combos, buttons) paint themselves on top of this — they're opaque where they sit, so the panel fill shows only in its 10px inset border. Because `WM_ERASEBKGND` returns 1 and we double-buffer, there's no flicker on resize.
`WM_CTLCOLORSTATIC` should return the bg brush for the status line and the surface brush for the placeholder:
```cpp
case WM_CTLCOLORSTATIC: {
HDC hdc=(HDC)wParam;
if (GetDlgCtrlID((HWND)lParam) == ID_STATIC_PLACEHOLDER) {
SetTextColor(hdc, RGB(0x9A,0xA0,0xAB)); SetBkColor(hdc, CR_SURFACE);
return (LRESULT)g_brSurface;
}
SetTextColor(hdc, CR_TEXT); SetBkColor(hdc, CR_BG);
return (LRESULT)g_brBg;
}
```
Define `#define ID_STATIC_PLACEHOLDER 1014` with your other IDs, and remember to call `UpdatePlaceholder(hWnd)` after `WM_APP_RESULT` sets text and after Clear.
---
# Part 3 — Build & checklist
- **Link GDI+:** the `#pragma comment(lib,"gdiplus.lib")` covers MSVC; no CMake change needed. (`comctl32` for `SetWindowSubclass` is already linked.)
- **Build:** `cmake --build build --config Release` as before.
- **Verify, in order:**
- [ ] Launch → window has rounded corners + dark title bar, no gray 3-D buttons.
- [ ] Record pill fills accent; hover lightens it; recording turns it red and it pulses; VU segments track your voice.
- [ ] Stop → transcript appears in the flat panel; placeholder hides; "Copied/Pasted".
- [ ] **No crash on Stop.** If you see "Model not found", fix the path (§1.2).
- [ ] Drag the window edges → it resizes and the transcript grows; min size respected.
- [ ] Switch model in the dropdown → status shows "Loading…" then "Ready • 2 threads"; next transcription uses it.
- **OS notes:** caption color / rounded corners need Windows 11 (build 22000+). On Windows 10 you still get dark mode (immersive) but square corners and the default caption color — everything else is identical since it's all custom-drawn.
---
# Appendix — Option B: fully borderless custom title bar (advanced)
If you want **zero** system chrome (no native title bar at all), use the "extend client over the caption" technique instead of Option A's recoloring. Keep `WS_OVERLAPPEDWINDOW` (so resize/snap/shadow/rounding stay native) and reclaim the caption area:
```cpp
case WM_NCCALCSIZE:
if (wParam) {
NCCALCSIZE_PARAMS* p = (NCCALCSIZE_PARAMS*)lParam;
int top = p->rgrc[0].top;
LRESULT r = DefWindowProc(hWnd, message, wParam, lParam); // default frame calc
p->rgrc[0].top = top; // give the caption height back to the client area
return r;
}
break;
case WM_NCHITTEST: {
LRESULT ht = DefWindowProc(hWnd, message, wParam, lParam); // handles resize edges
if (ht != HTCLIENT) return ht;
POINT pt{ GET_X_LPARAM(lParam), GET_Y_LPARAM(lParam) };
ScreenToClient(hWnd, &pt);
const int HEADER = 40;
if (pt.y < HEADER) {
// exclude your custom min/close/pin hit-rects here, else:
return HTCAPTION; // drag region
}
return HTCLIENT;
}
```
Then draw your own header (title text on the left, custom min/close buttons on the right) in `WM_PAINT`, and shift `LayoutControls`' starting `y` down by `HEADER`. This is more finicky across DPI/OS — only take it on if Option A's dark caption isn't minimal enough for you.
dev-task-list.md
+321
View File
@@ -0,0 +1,321 @@
# Win Dictation — Developer Task List
A prioritised, junior-friendly backlog to finish polishing the app. Each task says **what**, **which files**, **steps**, **the code** (inline, or a pointer to the verbatim block in a companion doc), and **how to know it's done**.
**Companion docs (full code lives here — don't retype, copy from them):**
- `ARCHITECTURE-AND-DEVGUIDE.md` — Part B1 (chrome removal), Part B2 (progress + ETA). Verbatim code.
- `MODERN-UI-AND-FIXES.md` — the modern UI design + GDI+ helpers.
- `FINDINGS-FIXES-TESTS.md` — the test harness (`tests/test_core.cpp`).
**Baseline (already done — do NOT redo):** batch record-then-transcribe; crash fixes (null-context guard, `g_modelOk`, `g_initializing`, `m_cfg_mtx`); append behaviour (`text_util.h`); no-hide + safe auto-paste; Clear button; `run_inference`/`transcribe_sync` refactor; modern dark UI (GDI+, rounded panel, owner-draw buttons, custom VU, dark caption); resizable window; `test-core` CMake target.
**Conventions:** Effort = XS (<30 min) · S (≤2 h) · M (half-day) · L (1–2 days). Do phases in order; tasks within a phase are mostly independent unless "Depends on" says otherwise. After every task: `cmake --build build --config Release` must succeed with **zero new warnings**, and the app must still launch.
**Color tokens (already in `main.cpp`, reuse — never hard-code hex elsewhere):** `C_BG #0F1115`, `C_SURFACE #181B22`, `C_SURFACEHI #20242D`, `C_BORDER #262B36`, `C_TEXT #E7E9EE`, `C_TEXTDIM #9AA0AB`, `C_ACCENT #6E8BFF`, `C_DANGER #FF5C5C`, `C_GOOD #46D39A`.
---
## Phase 0 — Repo hygiene (do first; clears traps)
### Task 0.1 — Delete the stale `src/CMakeLists.txt`
- **Goal:** Remove a build file that references **removed** APIs (`init()`, `is_using_gpu()`) and a `test-audio` target the real build ignores. The authoritative build is the **root** `CMakeLists.txt`.
- **Files:** `src/CMakeLists.txt` (delete).
- **Steps:** Confirm `build.ps1` configures from repo root (`-S $RepoRoot`). It does. Delete `src/CMakeLists.txt`.
- **Done when:** Clean build from root still works; no other file `add_subdirectory(src)`.
- **Effort:** XS
### Task 0.2 — Remove the broken `src/test-audio.cpp` (replaced in Phase 4)
- **Goal:** `src/test-audio.cpp` calls `m_transcriber.init(...)` / `is_using_gpu()` which no longer exist — it cannot compile. It's superseded by `tests/test_core.cpp` (Task 4.1).
- **Files:** `src/test-audio.cpp` (delete), `src/record-test-audio.ps1` (keep — still useful for capturing WAVs), `src/TESTING.md` (mark superseded in Task 4.4).
- **Done when:** No target references `test-audio.cpp`.
- **Effort:** XS
### Task 0.3 — Create the `tests/` folder + placeholder
- **Goal:** The root `CMakeLists.txt` already declares `add_executable(test-core tests/test_core.cpp ...)`, but the file doesn't exist yet → configure fails if anyone builds `test-core`.
- **Steps:** Create `tests/` and add `tests/test_core.cpp` (full content in Task 4.1). Until then, the `test-core` target can stay; just don't build it.
- **Done when:** `tests/test_core.cpp` exists and `cmake --build build --target test-core` compiles (after Task 4.1).
- **Effort:** XS · **Depends on:** 4.1 for real content
---
## Phase 1 — Visual polish & assets
### Task 1.1 — App icon asset (custom)
- **Goal:** Replace the placeholder icon with a clean, modern app icon used for the window, taskbar, and tray.
- **Files:** `assets/icon-source.png` (new, 1024×1024), `src/icon.ico` (generated), `src/convert_icon.py` (fix paths), `src/win-dictation.rc` (already references `101 ICON "icon.ico"`).
- **Design spec:** Flat, minimal. A single rounded **microphone** glyph, centered, on a dark charcoal rounded-square (`#15171C`). Mic filled with the indigo accent (`#6E8BFF`), subtle top-down gradient to `#5B7BFF`. No text. Must read clearly at **16×16**. Keep ~12% padding around the glyph.
- Ready-to-use generation prompt (AI image tool): *"Minimalist modern app icon, a single simple microphone glyph centered on a dark charcoal rounded square, microphone filled indigo #6E8BFF with a soft vertical gradient, flat design, crisp clean edges, no text, high contrast, legible at small sizes, 1024×1024."*
- Or design in Figma/Inkscape and export 1024×1024 PNG.
- **Steps:**
1. Put the source PNG at `assets/icon-source.png`.
2. Fix `convert_icon.py` to use real paths and multi-size output:
```python
from PIL import Image
img = Image.open("assets/icon-source.png").convert("RGBA")
img.save("src/icon.ico", format="ICO",
sizes=[(256,256),(64,64),(48,48),(32,32),(16,16)])
print("wrote src/icon.ico")
```
3. Run it (`python src/convert_icon.py` from repo root). Confirm `src/icon.ico` exists.
4. Rebuild; the resource compiler picks up `src/icon.ico` via the `.rc`.
- **Done when:** The new icon shows on the title bar, taskbar, Alt-Tab, and tray — sharp at all sizes.
- **Effort:** S
### Task 1.2 — Tray icon reflects recording state (optional but nice)
- **Goal:** When recording (window may be hidden), the **tray** icon turns red so state is visible at a glance.
- **Files:** `assets/icon-rec-source.png` (new), `src/icon-rec.ico`, `src/win-dictation.rc` (add `102 ICON "icon-rec.ico"`), `src/main.cpp`.
- **Steps:**
1. Create a red variant of the icon (mic in `#FF5C5C`). Convert to `src/icon-rec.ico` (same sizes), add `102 ICON "icon-rec.ico"` to the `.rc`.
2. In `main.cpp`, load both icons once: `HICON g_icoIdle, g_icoRec;` via `LoadIcon(hInst, MAKEINTRESOURCE(101/102))`.
3. Add a helper `void SetTrayIcon(bool rec){ nid.uFlags = NIF_ICON; nid.hIcon = rec?g_icoRec:g_icoIdle; Shell_NotifyIcon(NIM_MODIFY,&nid); }`.
4. Call `SetTrayIcon(true)` when recording starts, `SetTrayIcon(false)` on stop/cancel/result.
- **Done when:** Start recording, hide the window — the tray icon is red; after transcription it returns to normal.
- **Effort:** S · **Depends on:** 1.1
### Task 1.3 — Kill the button hairlines + focus rectangles (B1.1)
- **Goal:** Remove the thin light line around *Pinned* and the left/top lines on Copy/Paste/Clear.
- **Files:** `src/main.cpp`; link `uxtheme.lib`.
- **Steps (full code in `ARCHITECTURE-AND-DEVGUIDE.md` §B1.1):**
1. Add `#include <uxtheme.h>` and `#pragma comment(lib, "uxtheme.lib")`.
2. After creating each owner-draw button (Record, Pin, Copy, Paste, Clear), call `SetWindowTheme(hBtn, L"", L"");` (before/after `SetWindowSubclass` is fine).
3. Add `WS_CLIPCHILDREN` to the main window style in `CreateWindowExW`.
4. After all controls are created (end of the create block), call once:
`SendMessageW(hMainWnd, WM_CHANGEUISTATE, MAKEWPARAM(UIS_SET, UISF_HIDEFOCUS), 0);`
- **Done when:** No hairline around any button; tabbing between controls draws no dotted focus rect.
- **Effort:** S
### Task 1.4 — Replace comboboxes with custom dropdowns (B1.2)
- **Goal:** Remove the native Windows dropdown button (the "second arrow") on the mic + model selectors.
- **Files:** `src/main.cpp`.
- **Steps (full code — `DrawSelect`, `PopupProc`, `ShowSelectPopup`, `WM_APP_SELECT` — in `ARCHITECTURE-AND-DEVGUIDE.md` §B1.2):**
1. Add label/selection state: `std::vector<std::wstring> g_audioItems; int g_audioSel=0;` and `g_modelItems`/`g_modelSel`. Populate them in `RefreshAudioDevices`/`RefreshModelList` (keep populating `g_modelComboPaths` in parallel).
2. Replace the two `COMBOBOX` creations with `BS_OWNERDRAW` buttons `ID_SEL_AUDIO`/`ID_SEL_MODEL`; subclass with `BtnProc`; `SetWindowTheme(.., L"", L"")`.
3. Add `#define WM_APP_SELECT (WM_USER + 5)` and `#define ID_SEL_AUDIO/ID_SEL_MODEL`.
4. Paste `DrawSelect`, route both IDs in `WM_DRAWITEM`. Paste `PopupProc` + `ShowSelectPopup`. On `WM_COMMAND` for the two IDs call `ShowSelectPopup(...)`. Handle `WM_APP_SELECT` to apply the choice (set `capture_id` / reload model).
5. Delete the now-unused `DrawCombo`, `WM_MEASUREITEM` combo branch, and `WM_CTLCOLORLISTBOX`.
6. Update `LayoutControls` to position `ID_SEL_AUDIO`/`ID_SEL_MODEL` where the combos were.
- **Done when:** Each selector shows exactly one (our) chevron, opens a dark rounded popup, hover highlights rows, selecting reloads the model / switches mic, and clicking elsewhere dismisses it.
- **Effort:** M · **Depends on:** 1.3 (shared `SetWindowTheme`)
---
## Phase 2 — Progress feedback & control
### Task 2.1 — Transcription progress + ETA (B2)
- **Goal:** Replace the static "Transcribing…" with a moving progress bar + live status like **"Transcribing 1:40 · 45% · ~9s left"**.
- **Files:** `src/transcriber.h`, `src/transcriber.cpp`, `src/main.cpp`.
- **Steps (full code in `ARCHITECTURE-AND-DEVGUIDE.md` §B2.1–§B2.3):**
1. `transcriber.h`: add `set_progress_callback`, `audio_seconds()`, private `m_on_progress`, `m_audio_seconds`, and the static `s_progress` trampoline.
2. `transcriber.cpp`: implement `s_progress`; in `run_inference` set `m_audio_seconds`, `wp.progress_callback = &Transcriber::s_progress; wp.progress_callback_user_data = this;`.
3. `main.cpp`: add `#define WM_APP_PROGRESS (WM_USER + 6)`, `std::atomic<int> g_progress{0};`, `DWORD g_busyStart;`. Register `set_progress_callback([](int p){ PostMessage(hMainWnd, WM_APP_PROGRESS, p, 0); })`.
4. Set `g_busyStart = GetTickCount(); g_progress = 0;` right before `stop_and_transcribe()`.
5. Handle `WM_APP_PROGRESS` (store + invalidate `g_vuRect`). Replace the `is_busy()` branch of `UpdateStatus` with the ETA formatter. Add `DrawProgress` and, in `WM_PAINT`, draw the progress bar in `g_vuRect` while busy (VU otherwise). In `WM_TIMER`, also invalidate `g_vuRect` while busy so the bar/ETA tick.
- **Edge cases:** `progress < 3%` → show "Transcribing m:ss of audio…" (ETA not stable yet). Clamp `remain >= 0`.
- **Done when:** A 1–2 min clip shows a filling bar + percentage + shrinking ETA and completes; short clips still feel instant.
- **Effort:** M
### Task 2.2 — Cancel a running transcription (B2.4)
- **Goal:** Let the user abort a long/incorrect transcription instead of waiting it out.
- **Files:** `src/transcriber.h/.cpp`, `src/main.cpp`.
- **Steps:**
1. `transcriber.h`: add `void request_cancel(){ m_abort = true; }`, private `std::atomic<bool> m_abort{false};`, static `s_abort`.
2. `transcriber.cpp`: in `run_inference`, `m_abort = false;` at the top, and set `wp.abort_callback = &Transcriber::s_abort; wp.abort_callback_user_data = this;` (skip if your `whisper.h` lacks `abort_callback`).
3. `main.cpp`: at the very top of the `HK_TOGGLE` handler add `if (g_tx.is_busy()) { g_tx.request_cancel(); SetStatus(hWnd, L"Cancelling…"); break; }` — so the Record button becomes "cancel" while busy.
4. In `WM_APP_RESULT`, when the result is empty *and* a cancel was requested, show "Cancelled" instead of "No speech detected" (track a `g_cancelRequested` flag, reset each Stop).
- **Done when:** Pressing the button (or hotkey) mid-transcription stops it within ~1 s and the status reads "Cancelled".
- **Effort:** S · **Depends on:** 2.1
---
## Phase 3 — Robustness & persistence
### Task 3.1 — Persist settings between launches
- **Goal:** Remember mic, model, pin state, auto-paste, auto-hide, and window position. Today everything resets each launch.
- **Files:** `src/settings.h` (new, header-only), `src/main.cpp`.
- **Code (`src/settings.h`):**
```cpp
#pragma once
#include <windows.h>
#include <string>
struct AppSettings {
int captureId = 0;
std::wstring modelFile; // e.g. L"models\\ggml-tiny.en.bin" ("" = auto)
bool pinned = true, autoPaste = true, autoHide = false;
int winX = CW_USEDEFAULT, winY = CW_USEDEFAULT, winW = 400, winH = 340;
};
inline std::wstring SettingsPath() {
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
return p + L"\\win-dictation.ini";
}
inline int GetIni(const wchar_t* k, int d){ return GetPrivateProfileIntW(L"app", k, d, SettingsPath().c_str()); }
inline void PutIni(const wchar_t* k, int v){ wchar_t b[32]; wsprintfW(b, L"%d", v); WritePrivateProfileStringW(L"app", k, b, SettingsPath().c_str()); }
inline void LoadSettings(AppSettings& s){
s.captureId = GetIni(L"captureId", s.captureId);
s.pinned = GetIni(L"pinned", s.pinned) != 0;
s.autoPaste = GetIni(L"autoPaste", s.autoPaste) != 0;
s.autoHide = GetIni(L"autoHide", s.autoHide) != 0;
s.winX = GetIni(L"winX", s.winX); s.winY = GetIni(L"winY", s.winY);
s.winW = GetIni(L"winW", s.winW); s.winH = GetIni(L"winH", s.winH);
wchar_t m[MAX_PATH]; GetPrivateProfileStringW(L"app", L"modelFile", L"", m, MAX_PATH, SettingsPath().c_str());
s.modelFile = m;
}
inline void SaveSettings(const AppSettings& s){
PutIni(L"captureId", s.captureId); PutIni(L"pinned", s.pinned);
PutIni(L"autoPaste", s.autoPaste); PutIni(L"autoHide", s.autoHide);
PutIni(L"winX", s.winX); PutIni(L"winY", s.winY); PutIni(L"winW", s.winW); PutIni(L"winH", s.winH);
WritePrivateProfileStringW(L"app", L"modelFile", s.modelFile.c_str(), SettingsPath().c_str());
}
```
- **Wiring (`main.cpp`):**
1. Add a global `AppSettings g_set;` Call `LoadSettings(g_set);` **before** creating the window.
2. Apply: use `g_set.winX/Y/W/H` in `CreateWindowExW` (validate on-screen; fall back to `CW_USEDEFAULT` if off all monitors). Set `g_pinned = g_set.pinned`, `g_autoPaste = g_set.autoPaste`, `g_autoHide = g_set.autoHide`, `g_config.capture_id = g_set.captureId`. If `g_set.modelFile` is non-empty and the file exists, use it instead of `SelectOptimalModel`.
3. Save on change: after toggling pin/auto-paste/auto-hide, after a model/mic change, and on `WM_EXITSIZEMOVE` (window moved/resized → store rect) and `WM_DESTROY` (final save). A `void PersistNow()` that copies the live globals into `g_set` then `SaveSettings(g_set)` keeps it DRY.
- **Done when:** Change mic/model, move/resize the window, toggle pin, quit, relaunch → everything is restored. Deleting the `.ini` restores defaults.
- **Effort:** M
### Task 3.2 — Quick toggles in the tray menu
- **Goal:** Expose Auto-paste, Always-on-top, and Auto-hide without building a settings panel.
- **Files:** `src/main.cpp` (`ShowContextMenu`, `WM_COMMAND`).
- **Steps:** Add checkable items to the tray popup (`MF_STRING | (flag?MF_CHECKED:0)`) with new IDs (`ID_TRAY_AUTOPASTE`, `ID_TRAY_TOPMOST`, `ID_TRAY_AUTOHIDE`). In `WM_COMMAND`, flip the matching global, apply (for top-most call `SetWindowPos(... HWND_TOPMOST/NOTOPMOST ...)`), then `PersistNow()`.
- **Done when:** Right-click tray → toggles show check state, take effect immediately, and survive a relaunch.
- **Effort:** S · **Depends on:** 3.1
### Task 3.3 — Bound the recording length
- **Goal:** A forgotten recording shouldn't grow memory without limit (~1.9 MB/30 s today, uncapped).
- **Files:** `src/transcriber.h/.cpp`, `src/main.cpp`.
- **Steps:**
1. `transcriber.h`: add `float recorded_seconds() const;` returning `m_capture` size / `WHISPER_SAMPLE_RATE` under `m_capture_mtx` (or maintain an atomic sample counter incremented in `on_audio`).
2. `main.cpp` `WM_TIMER` (recording branch): if `g_tx.recorded_seconds() >= kMaxRecordSeconds` (e.g. 600), auto-stop by posting the same path as a manual Stop, and set status "Max length reached — transcribing".
- **Done when:** Recording auto-stops at the cap and transcribes what was captured; normal short clips unaffected.
- **Effort:** S
### Task 3.4 — Surface hotkey-registration failures + make hotkeys configurable
- **Goal:** Today `RegisterHotKey` return values are ignored — if another app owns `Ctrl+Shift+Space`, the hotkey silently dies. Also allow remapping.
- **Files:** `src/main.cpp`, `src/settings.h`.
- **Steps:**
1. Capture the return of both `RegisterHotKey` calls. If either fails, show a non-blocking status ("Hotkey in use — set another in win-dictation.ini") and still allow the on-screen Record button to work.
2. Read modifiers + key from the INI (`hkMods`, `hkVk`, defaulting to `MOD_CONTROL|MOD_SHIFT` + `VK_SPACE` / `'H'`); register those. (Full remap UI is a Phase 5 stretch — INI is enough now.)
- **Done when:** With a conflicting global hotkey registered by another app, the app launches, warns, and the button still records; editing the INI changes the hotkey.
- **Effort:** S · **Depends on:** 3.1
### Task 3.5 — Lightweight logging
- **Goal:** One small log file so field issues are diagnosable without a debugger.
- **Files:** `src/logging.h` (new), `src/main.cpp`, `src/transcriber.cpp`.
- **Code (`src/logging.h`):**
```cpp
#pragma once
#include <windows.h>
#include <cstdio>
#include <string>
inline void LogLine(const char* msg) {
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/")) + L"\\win-dictation.log";
FILE* f = _wfopen(p.c_str(), L"a"); if (!f) return;
SYSTEMTIME t; GetLocalTime(&t);
fprintf(f, "%04d-%02d-%02d %02d:%02d:%02d %s\n", t.wYear,t.wMonth,t.wDay,t.wHour,t.wMinute,t.wSecond, msg);
fclose(f);
}
```
- **Log at minimum:** startup (model path, threads, GPU on/off), model load success/failure, mic-open failure, and each transcription (audio seconds, elapsed ms, output char count). Keep messages one line, ASCII.
- **Optional:** if the file exceeds ~1 MB at startup, rename to `.log.1` (simple 1-file rotation).
- **Done when:** `win-dictation.log` appears next to the exe and records a startup line + one line per transcription.
- **Effort:** S
---
## Phase 4 — Tests, docs & packaging
### Task 4.1 — Create `tests/test_core.cpp`
- **Goal:** Headless regression tests that catch the bugs we already fixed and lock in progress reporting.
- **Files:** `tests/test_core.cpp` (new — **full content in `FINDINGS-FIXES-TESTS.md` §4.2**).
- **Steps:**
1. Copy the test program from `FINDINGS-FIXES-TESTS.md` §4.2 into `tests/test_core.cpp`.
2. Add the progress assertion from `ARCHITECTURE-AND-DEVGUIDE.md` §B3 (max progress ≥95, monotonic).
3. Build + run:
```powershell
cmake --build build --config Release --target test-core
cd build\bin\Release
copy ..\..\..\deps\SDL2-2.28.5\lib\x64\SDL2.dll . # if missing
.\test-core.exe models\ggml-tiny.en.bin ..\..\..\samples\jfk.wav
```
- **Tests covered:** append rule; bad model path → no crash + empty; real clip → contains "country"; short audio → no crash; progress reaches ~100% and is non-decreasing.
- **Done when:** `test-core.exe` prints "ALL TESTS PASSED" and exits 0.
- **Effort:** S · **Depends on:** 2.1 (for the progress test)
### Task 4.2 — (Optional) CI workflow
- **Goal:** Run the build + `test-core` on every push.
- **Files:** `.github/workflows/build.yml` (new).
- **Steps:** Windows runner → configure CMake (CPU-only) → build `win-dictation` + `test-core` → download `ggml-tiny.en.bin` → run `test-core` with `jfk.wav`. Fail the job on non-zero exit.
- **Done when:** A pushed branch shows a green check that actually ran the tests.
- **Effort:** M · **Depends on:** 4.1
### Task 4.3 — Desktop / Start-Menu shortcut on install
- **Goal:** One-click launch (the user already pins to taskbar; a shortcut makes first run easy).
- **Files:** `src/package.ps1` (extend) or a new `install-shortcut.ps1`.
- **Steps:** After packaging, create a `.lnk` via `WScript.Shell` with `TargetPath` = the exe and `WorkingDirectory` = its folder (so `models\` resolves even though we also use `exe_dir()`), `IconLocation` = the exe. (Snippet in `MODERN-UI-AND-FIXES.md` §9 / spec doc §9.)
- **Done when:** Running the script creates a working Desktop shortcut that launches the app with the icon.
- **Effort:** S
### Task 4.4 — Fix the documentation (it describes the OLD app)
- **Goal:** `README.md`, `src/README.md`, `src/CHANGES.md`, `src/CUDA-SETUP.md`, `src/QUICK-REBUILD-GPU.md`, `src/TESTING.md`, `src/DESIGN.md`, `src/FIXES-APPLIED.md` still describe the **streaming / ring-buffer / 24-thread / Ctrl+Shift+R** design and RTX-3090 benchmarks — all now wrong/misleading.
- **Steps:**
1. Rewrite the top-level `README.md` to describe the **current** app: push-to-talk batch transcription, `Ctrl+Shift+Space` to record/stop, `Ctrl+Shift+H` to hide, tray, always-on-top, copy + auto-paste, model/mic selectors, CPU-tuned (physical-core threads, tiny.en default). Remove ring-buffer/VAD/24-thread/streaming claims and the RTX benchmarks (or move GPU notes to an "optional" aside).
2. Update `build.ps1` end-of-run messages ("Hotkey: Ctrl+Shift+R", "Model: base.en") to match reality.
3. Mark `CHANGES.md`, `FIXES-APPLIED.md`, `TESTING.md`, `QUICK-REBUILD-GPU.md`, `CUDA-SETUP.md` as **historical/superseded** (a one-line banner at top), or fold the still-true bits into the README and delete the rest. Keep `DESIGN.md` only if updated to the batch architecture.
- **Done when:** A new reader following `README.md` gets accurate hotkeys, model behaviour, and build steps; no doc claims a ring buffer or 24 threads.
- **Effort:** M
---
## Phase 5 — Stretch features (nice-to-have)
### Task 5.1 — Hold-to-talk mode
- **Goal:** Option to record only while a key is held (vs. toggle).
- **Files:** `src/main.cpp`, `src/settings.h`.
- **Steps:** Add a low-level keyboard hook (`SetWindowsHookEx(WH_KEYBOARD_LL, ...)`); on key-down of the chosen key start recording, on key-up `stop_and_transcribe`; debounce auto-repeat with a flag. Gate behind a `holdToTalk` INI setting; keep toggle as default. (Outline in the original spec doc §7.)
- **Done when:** With the setting on, holding the key records and releasing transcribes; toggle mode still available.
- **Effort:** M
### Task 5.2 — Settings panel (graduate from tray toggles + INI)
- **Goal:** A small in-app settings popup (reuse the custom popup window from Task 1.4) for mic, model, auto-paste, auto-hide, hold-to-talk, and hotkey capture.
- **Done when:** All settings are editable in-app and persist (Task 3.1).
- **Effort:** L · **Depends on:** 1.4, 3.1
### Task 5.3 — Export / save transcript
- **Goal:** Save the transcript box to a `.txt` (and timestamped filename) from the tray menu or a button.
- **Effort:** S
### Task 5.4 — Multi-language support
- **Goal:** Allow non-English models + a language selector (currently hard-wired `en`). Swap to a multilingual model (`ggml-base.bin`) and set `WhisperConfig.language` from a selector.
- **Effort:** M · **Depends on:** 1.4 (selector), 3.1
---
## Definition of Done (per task)
- [ ] Builds clean (`cmake --build build --config Release`), **zero new warnings**.
- [ ] App launches, records, transcribes, appends, copies/pastes — no regressions.
- [ ] `test-core.exe` exits 0 (after Phase 4).
- [ ] Any new setting persists across relaunch (after Phase 3).
- [ ] Change is reflected in `README.md` if user-facing.
## Suggested order (fastest path to "feels finished")
1. **0.1–0.3** (hygiene) → **1.3** (hairlines, 5-min win) → **2.1** (progress — biggest UX gain).
2. **1.1** (icon) → **1.4** (custom dropdowns) → **1.2** (tray state) → **2.2** (cancel).
3. **3.1** (persistence) → **3.2** (tray toggles) → **3.4** (hotkey safety) → **3.3** (length cap) → **3.5** (logging).
4. **4.1** (tests) → **4.4** (docs) → **4.3** (shortcut) → **4.2** (CI).
5. Stretch (**5.x**) as desired.
## Verification matrix (final smoke test)
| Area | Check |
|---|---|
| Crash-free | Record/stop 10× incl. a 2-min clip; window never vanishes; process stable |
| Progress | 2-min clip shows filling bar + % + shrinking ETA; cancel works |
| Chrome | No hairlines/focus rects; selectors have one chevron + dark popup |
| Paste | Hotkey-from-another-app pastes the latest utterance; button = copy only |
| Persistence | mic/model/pin/auto-paste/window pos restored after relaunch |
| Assets | New icon crisp in title bar, taskbar, Alt-Tab, tray; red tray icon while recording |
| Robustness | Missing model → clear message (no crash); hotkey conflict → warned; long record auto-stops |
| Tests/docs | `test-core` green; README matches actual hotkeys/behaviour |
ggml/cmake/common.cmake
+25
View File
@@ -0,0 +1,25 @@
include_guard(GLOBAL)
function(add_library name)
_add_library(${name} ${ARGN})
set(TARGET ${name} PARENT_SCOPE)
endfunction()
function(ggml_get_system_arch)
set(options)
set(oneValueArgs RESULT)
set(multiValueArgs)
cmake_parse_arguments(ARG "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
if(CMAKE_OSX_ARCHITECTURES STREQUAL "arm64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64")
set(${ARG_RESULT} "arm64" PARENT_SCOPE)
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64|AMD64")
set(${ARG_RESULT} "x86" PARENT_SCOPE)
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "i[3-6]86")
set(${ARG_RESULT} "x86" PARENT_SCOPE)
elseif(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(${ARG_RESULT} "x86" PARENT_SCOPE)
else()
set(${ARG_RESULT} "x86" PARENT_SCOPE)
endif()
endfunction()
snapshot.txt
+4830
View File
File diff suppressed because it is too large Load Diff
src/CHANGES.md
+26 -295
View File
@@ -1,308 +1,39 @@
# Whisper Dictation v2.0 - Changelog
# Win Dictation — Changelog
## Overview
Complete rewrite of win-dictation with focus on performance, reliability, and user experience.
## v3.0 — Architecture & UI Rebuild (current)
## Major Changes
### Single-surface UI
### 1. Audio Capture System (Zero Loss)
- Replaced 9 separate child windows with one immediate-mode painted surface
- Eliminated inter-window hairline seams at the root
- Widget data model with hover/press animation blending
- Per-monitor DPI scaling (V2)
- Dark mode design tokens
- Dark scrollbar on the transcript edit control
**Problem:** Audio chunks were being dropped between recording and processing.
### Progress system
**Solution:** Implemented lock-free ring buffer system
- 30-second circular buffer (480K samples)
- Atomic read/write positions
- No mutex in audio callback
- Handles burst processing gracefully
- Self-calibrating linear estimator: `proc = a + b·audio`, fitted per model with decayed online least-squares
- Persisted per-model timing history in `win-dictation.ini`
- Strictly monotonic countdown (never counts up)
- Smooth percent that eases, never snaps (except final 100%)
- Motion from frame 1 (predicts before whisper reports)
**Files Changed:**
- `transcriber.h`: Added ring buffer members
- `transcriber.cpp`: Rewrote audio_callback() and worker_loop()
### Push-to-talk batch mode
### 2. Multi-Core CPU Support
- Removed streaming/VAD/ring-buffer architecture
- Record `→` stop `→` single `whisper_full` call
- 500ms silence auto-end timer
- Physical-core threading (matches the i5 target)
**Problem:** Only using 1-2 CPU cores despite having 24 available.
### Files
**Solution:** Proper thread configuration
- Uses `std::thread::hardware_concurrency()` (24 threads)
- OpenMP support enabled in build
- Optimized work distribution
**Configuration:**
```cpp
WhisperConfig::n_threads = std::thread::hardware_concurrency(); // 24
```
### 3. GPU Acceleration
**Problem:** No GPU utilization despite CUDA installation.
**Solution:** GPU detection and backend selection
- Auto-detects CUDA/Vulkan/Metal
- Graceful fallback to CPU
- Version compatibility checking
- Status display in UI
**Build Script:**
- Detects GPU capabilities
- Checks CUDA version compatibility (11.7 vs 12.4 requirement)
- Falls back to optimized CPU build
### 4. Modern User Interface
**Problem:** Slow, glitchy interface with poor visual feedback.
**Solution:** Complete UI overhaul
- Dark theme with modern colors
- 30 FPS update timer (was 50ms/20 FPS)
- Custom button drawing
- Real-time status indicators
- Smooth animations
**UI Features:**
- VU meter (real-time audio level)
- Buffer indicator (queue status)
- GPU/CPU status
- Thread count display
- Responsive layout
**Colors:**
```cpp
#define COLOR_BG RGB(32, 33, 36)
#define COLOR_PRIMARY RGB(138, 180, 248)
#define COLOR_SUCCESS RGB(129, 201, 149)
```
### 5. Processing Optimizations
**Changes:**
- Reduced step_ms: 3000ms → 1500ms (faster response)
- Reduced length_ms: 10000ms → 8000ms (better streaming)
- Added VAD filtering (skip silence)
- Improved context handling
- Better memory management
### 6. Build System
**New Features:**
- Automated GPU detection
- Version compatibility checking
- One-command build and deploy
- Automatic model download
- DLL deployment
**Script:** `build.ps1`
```powershell
# Detects:
- NVIDIA GPU + CUDA version
- AMD GPU + ROCm
- Vulkan SDK
- CPU capabilities (AVX2/FMA)
```
## File-by-File Changes
### transcriber.h
```diff
+ Ring buffer implementation (RING_BUFFER_SIZE = 480K)
+ Atomic position tracking
+ get_buffer_fullness() method
+ is_using_gpu() const correctness
+ Processing buffer for context
+ GPU active flag
- Simple deque queue
- Blocking mutex in callback
```
### transcriber.cpp
```diff
+ Lock-free ring buffer audio_callback()
+ Atomic read/write operations
+ VAD integration (skip silence)
+ GPU detection in init()
+ Improved error handling
+ Context-aware processing
+ Smaller SDL buffer (512 vs 1024)
- Blocking queue operations
- No VAD filtering
- Poor error messages
```
### main.cpp
```diff
+ Modern dark theme
+ Custom button drawing (owner-draw)
+ 30 FPS timer (was 20 FPS)
+ Status text with GPU/CPU/threads
+ DWM dark mode titlebar
+ Improved layout handling
+ Better font selection
- Basic Windows theme
- Standard buttons
- Slow updates
- Minimal status info
```
### CMakeLists.txt
```diff
+ dwmapi library link
+ Optimization flags (/O2 /GL /LTCG)
+ Better include paths
+ Separate Debug/Release outputs
- Basic configuration
```
### build.ps1
```diff
+ Complete rewrite
+ GPU detection logic
+ CUDA version checking
+ Automatic DLL deployment
+ Model download automation
+ Colored output
+ Error handling
- CPU-only hardcoded
- Manual deployment
- No GPU support
```
## Performance Impact
### Before
- **Audio Loss**: Frequent dropped chunks
- **CPU Usage**: 1-2 cores (~8%)
- **GPU Usage**: 0%
- **Latency**: 3-5 seconds
- **UI FPS**: ~10-15 (choppy)
- **Buffer Issues**: Frequent overruns
### After (CPU-Only)
- **Audio Loss**: Zero (ring buffer)
- **CPU Usage**: 24 cores (~60-80% during speech)
- **GPU Usage**: 0% (incompatible CUDA version)
- **Latency**: 2-3 seconds
- **UI FPS**: 30 (smooth)
- **Buffer Management**: Handles 30s bursts
### Potential (With GPU)
- **Audio Loss**: Zero
- **CPU Usage**: <10%
- **GPU Usage**: 20-30%
- **Latency**: <1 second
- **Throughput**: >20x real-time
## Bug Fixes
1. ✅ Fixed audio chunk loss (ring buffer)
2. ✅ Fixed CPU underutilization (thread count)
3. ✅ Fixed UI glitches (proper timing)
4. ✅ Fixed missing GPU support (detection)
5. ✅ Fixed model loading errors (better paths)
6. ✅ Fixed memory leaks (proper cleanup)
7. ✅ Fixed race conditions (atomics)
8. ✅ Fixed build issues (explicit GPU disable)
## Code Quality Improvements
- **Better error handling**: Graceful fallbacks
- **More comments**: Explain complex logic
- **Type safety**: size_t for sizes, proper casts
- **Memory safety**: RAII, smart pointers ready
- **Threading**: Atomic operations, no races
- **Modularity**: Clear separation of concerns
## Configuration Changes
### WhisperConfig
```cpp
struct WhisperConfig {
std::string model_path;
std::string language = "en";
int n_threads = std::thread::hardware_concurrency(); // NEW: 24
int step_ms = 1500; // NEW: was 3000
int length_ms = 8000; // NEW: was 10000
bool use_gpu = true; // NEW: auto-detect
int capture_id = 0;
int n_gpu_layers = -1; // NEW: auto
};
```
## Testing Results
### Build
- ✅ Clean compile on MSVC 2022
- ✅ No linter errors
- ✅ All warnings addressed
- ✅ Proper DLL deployment
### Runtime (Expected)
- ✅ Window opens correctly
- ✅ Modern UI renders
- ✅ Audio devices detected
- ✅ Recording works
- ✅ Transcription functions
- ✅ No crashes
- ✅ System tray works
- ✅ Hotkey functions
## Known Limitations
1. **CUDA 11.7 Incompatible**: User has CUDA 11.7 but MSVC 2022 requires CUDA 12.4+
- **Workaround**: Using optimized CPU-only build
- **Solution**: Upgrade to CUDA 12.4+ for GPU support
2. **Single Language**: Currently English-only (base.en model)
- **Workaround**: Use multilingual model (ggml-base.bin)
3. **Model in Binary**: Model path hardcoded in source
- **Future**: UI-based model selection
## Upgrade Path
### To Enable GPU (CUDA)
1. Download and install CUDA Toolkit 12.4+
2. Clean build directory
3. Run build script (will auto-detect new CUDA)
4. Rebuild application
### To Enable GPU (Vulkan - Alternative)
1. Download and install Vulkan SDK
2. Set VULKAN_SDK environment variable
3. Clean and rebuild
### To Use Different Model
1. Download model from HuggingFace
2. Place in `build/bin/Release/models/`
3. Update `g_config.model_path` in main.cpp
4. Rebuild
## Documentation Added
1. **README.md**: Complete user guide
2. **CHANGES.md**: This detailed changelog
3. **Code Comments**: Inline documentation
4. **Build Output**: Informative messages
## Migration Notes
This is a **breaking change** from v1.0:
- API compatible but implementation completely different
- Rebuild required (not drop-in replacement)
- Configuration values changed
- UI completely redesigned
## Acknowledgments
- whisper.cpp team for the excellent base library
- SDL2 for cross-platform audio
- User feedback on performance issues
- Added `src/timing.h` — timing model + progress estimator + persistence
- Rewrote `src/main.cpp` — single-surface paint, widget model, animation clock, DPI
- Rewrote `README.md`, `src/README.md`, `src/CHANGES.md`
---
**Version**: 2.0
**Date**: November 26, 2025
**Status**: Production Ready (CPU-only), GPU Ready (pending CUDA upgrade)
## v2.0 — Previous architecture (deprecated)
Used streaming with ring buffer, VAD, owner-draw child windows. Described in earlier versions of the docs.
src/CMakeLists.txt
-77
View File
@@ -1,77 +0,0 @@
project(win-dictation)
if (WIN32)
# Main application
add_executable(win-dictation WIN32
main.cpp
transcriber.cpp
transcriber.h
win-dictation.rc
)
# Link dependencies
target_link_libraries(win-dictation PRIVATE
whisper
common
common-sdl
${SDL2_LIBRARY}
comctl32
dwmapi
)
# Include directories
target_include_directories(win-dictation PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/../..
${CMAKE_CURRENT_SOURCE_DIR}/../../include
${CMAKE_CURRENT_SOURCE_DIR}/../
${SDL2_INCLUDE_DIR}
)
# Use Unicode and enable optimizations
target_compile_definitions(win-dictation PRIVATE UNICODE _UNICODE)
# Enable /O2 optimization in Release
if(MSVC)
target_compile_options(win-dictation PRIVATE
$<$<CONFIG:Release>:/O2 /GL>
)
target_link_options(win-dictation PRIVATE
$<$<CONFIG:Release>:/LTCG>
)
endif()
# Set properties
set_target_properties(win-dictation PROPERTIES
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
RUNTIME_OUTPUT_DIRECTORY_RELEASE "${CMAKE_BINARY_DIR}/bin/Release"
RUNTIME_OUTPUT_DIRECTORY_DEBUG "${CMAKE_BINARY_DIR}/bin/Debug"
)
# Test executable
add_executable(test-audio
test-audio.cpp
transcriber.cpp
transcriber.h
)
target_link_libraries(test-audio PRIVATE
whisper
common
common-sdl
${SDL2_LIBRARY}
)
target_include_directories(test-audio PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/../..
${CMAKE_CURRENT_SOURCE_DIR}/../../include
${CMAKE_CURRENT_SOURCE_DIR}/../
${SDL2_INCLUDE_DIR}
)
set_target_properties(test-audio PROPERTIES
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
RUNTIME_OUTPUT_DIRECTORY_RELEASE "${CMAKE_BINARY_DIR}/bin/Release"
RUNTIME_OUTPUT_DIRECTORY_DEBUG "${CMAKE_BINARY_DIR}/bin/Debug"
)
endif()
src/README.md
+32 -228
View File
@@ -1,244 +1,48 @@
# Whisper Dictation - AI Voice to Text for Windows
# Win Dictation - Source Notes
A high-performance, real-time speech-to-text application for Windows using OpenAI's Whisper model.
## Architecture (current)
## ✨ Features
**Push-to-talk, batch mode.** Press record, speak, press again. The full audio clip is passed to `whisper_full` once on stop. No streaming, no VAD, no ring buffer.
### Performance
- **Multi-Core CPU Support**: Automatically uses all available CPU cores (24 threads detected)
- **GPU Acceleration**: Auto-detects and uses CUDA, Vulkan, or Metal when available
- **Ring Buffer Audio**: Zero audio loss with lock-free ring buffer implementation
- **Optimized Processing**: AVX2/FMA instructions for maximum performance
- Audio: SDL2 capture at 16kHz mono
- Inference: `whisper_full` with physical-core threads
- UI: Single-surface GDI+ immediate-mode painting (no child-window chrome)
- Progress: Self-calibrating linear estimator fused with whisper's chunk-boundary callbacks
### User Interface
- **Modern Dark Theme**: Polished, professional interface
- **Real-Time Monitoring**:
- Live VU meter for audio levels
- Buffer status indicator
- GPU/CPU usage display
- **Smooth Animations**: 30 FPS UI updates for responsive experience
- **System Tray Integration**: Minimize to tray with hotkey support
## Key files
### Audio Processing
- **Voice Activity Detection (VAD)**: Automatically filters silence
- **Continuous Recording**: Maintains context between segments
- **Multiple Microphone Support**: Select from all available input devices
- **16kHz Sample Rate**: Optimized for Whisper model
| File | Purpose |
|------|---------|
| `main.cpp` | Window, painting, interaction, settings, clipboard |
| `transcriber.h` / `transcriber.cpp` | Audio capture, whisper preload/inference, callbacks |
| `timing.h` | Per-model online least-squares timing model + live progress estimator |
| `settings.h` | INI file read/write for persistent settings |
| `text_util.h` | Transcript concatenation |
| `logging.h` | Timestamped log to `win-dictation.log` |
## 🚀 Quick Start
### Build
## Building
```powershell
powershell -ExecutionPolicy Bypass -File examples/win-dictation/build.ps1
cmake -S . -B build -G "Visual Studio 18 2026" -DSDL2_DIR="deps/SDL2-2.28.5/cmake"
cmake --build build --config Release
```
The build script will:
1. Detect your GPU capabilities (CUDA, Vulkan)
2. Download and configure SDL2
3. Build the application with optimal settings
4. Download the Whisper model (base.en - 140MB)
5. Deploy all required DLLs
Output: `build\bin\Release\win-dictation.exe`
### Run
Target machine: 2-core / 4-thread Intel i5-7th-gen, GPU CUDA disabled.
```
build/bin/Release/win-dictation.exe
```
## Model placement
Or double-click the exe in the build output directory.
Drop `.bin` files in `models/` next to the executable. The app scans for:
- `ggml-tiny.en.bin`
- `ggml-tiny.en-q8_0.bin`
- `ggml-base.en-q5_1.bin`
- `ggml-base.en.bin`
## 🎯 Usage
First available is used. Toggle in the model popup.
### Controls
- **Start/Stop Recording**: Click button or press `Ctrl+Shift+R`
- **Clear Text**: Click "Clear" button
- **Change Microphone**: Select from dropdown (auto-restarts recording)
- **Minimize**: Close window (minimizes to system tray)
- **Exit**: Right-click tray icon → Exit
## Settings
### Indicators
- **Level**: Real-time audio input level
- **Buffer**: Current audio buffer usage (0-100%)
- **Status**: Shows GPU/CPU mode, recording state, thread count
## ⚙️ Technical Details
### Architecture
#### Ring Buffer Audio Capture
- **Lock-Free Design**: Audio thread never blocks
- **30-Second Buffer**: Handles burst processing without loss
- **Atomic Operations**: Prevents race conditions
#### Processing Pipeline
```
Audio Input → Ring Buffer → VAD → Whisper Inference → Text Output
```
1. **SDL Audio Capture**: 512-sample chunks at 16kHz
2. **Ring Buffer**: Lock-free circular buffer
3. **VAD Processing**: Filters silence before inference
4. **Whisper Inference**: Multi-threaded with context overlap
5. **Text Output**: Appended to UI in real-time
### Performance Optimizations
#### CPU Mode (Current Build)
- All 24 CPU threads utilized
- AVX2/FMA SIMD instructions
- Optimized memory layout
- Minimal context switching
#### GPU Mode (When Available)
- CUDA 12.4+ or Vulkan SDK required
- Automatic offloading to GPU
- Faster inference times
- Lower CPU usage
### Model
Currently using `ggml-base.en.bin`:
- **Size**: 140 MB
- **Parameters**: 74 million
- **Languages**: English only (optimized)
- **Speed**: ~5x real-time on CPU, >20x on GPU
- **Accuracy**: Excellent for general speech
To use a different model, place it in `build/bin/Release/models/` and update the config in `main.cpp`.
## 🔧 Troubleshooting
### GPU Not Detected
- **CUDA**: Install CUDA Toolkit 12.4 or newer (CUDA 13.0 recommended)
- **See [CUDA-SETUP.md](CUDA-SETUP.md) for detailed installation guide**
- **Vulkan**: Install Vulkan SDK
- CPU-only mode still provides excellent performance with all cores
### After Installing CUDA 13.0
See **[CUDA-SETUP.md](CUDA-SETUP.md)** for complete setup instructions including:
- Verification steps
- Clean rebuild process
- Performance benchmarking
- Troubleshooting GPU issues
### Audio Not Working
- Check microphone permissions in Windows Settings
- Verify correct device selected in dropdown
- Test microphone in Windows Sound settings
### Poor Transcription Quality
- Ensure microphone is close (6-12 inches)
- Reduce background noise
- Check VU meter shows green when speaking
- Try a larger model (medium.en or large-v3-turbo)
### High CPU Usage
- Normal during active transcription
- Reduces during silence (VAD filtering)
- Consider enabling GPU acceleration
## 📊 Performance Benchmarks
### CPU-Only (24 threads, base.en model)
- **Latency**: ~2-3 seconds
- **Throughput**: ~5x real-time
- **CPU Usage**: 60-80% during speech
- **Memory**: ~500 MB
### GPU-Accelerated (RTX 3090, base.en model)
- **Latency**: <1 second
- **Throughput**: >20x real-time
- **GPU Usage**: 20-30%
- **CPU Usage**: <10%
- **Memory**: ~1 GB (VRAM)
## 🆕 Recent Improvements
### v2.0 (Current)
- ✅ **Ring buffer** implementation - no more dropped audio
- ✅ **Multi-core CPU** support - uses all available threads
- ✅ **GPU auto-detection** - CUDA/Vulkan support
- ✅ **Modern UI** - dark theme, smooth animations
- ✅ **VAD integration** - skip silence for efficiency
- ✅ **Better error handling** - graceful fallbacks
- ✅ **Status indicators** - real-time monitoring
- ✅ **Build script** - automated setup and deployment
### Previous Issues (Fixed)
- ❌ Audio chunks lost between recording and processing
- ❌ No GPU utilization
- ❌ Only used 1-2 CPU cores
- ❌ Slow, glitchy interface
- ❌ No real-time feedback
- ❌ Poor error messages
## 🎨 UI Features
### Modern Dark Theme
- Background: `#202124`
- Surface: `#292A2D`
- Primary: `#8AB4F8` (Blue)
- Success: `#81C995` (Green)
- Text: `#E8EAED`
### Responsive Layout
- Auto-resizes with window
- Maintains proper spacing
- Smooth transitions
### Visual Feedback
- VU meter with color coding
- Buffer status bar
- GPU/CPU indicator
- Thread count display
## 🔮 Future Enhancements
- [ ] Push-to-talk mode
- [ ] Multiple language support
- [ ] Punctuation model integration
- [ ] Export to file (TXT, SRT)
- [ ] Custom hotkey configuration
- [ ] Noise reduction filter
- [ ] Model switching in UI
- [ ] Real-time word highlighting
## 📝 License
This example is part of the whisper.cpp project and follows the same license (MIT).
## 🤝 Contributing
Improvements welcome! The code is designed to be:
- **Readable**: Clear structure and comments
- **Maintainable**: Modular design
- **Extensible**: Easy to add features
- **Performant**: Optimized critical paths
## 💡 Tips
### For Best Results
1. Use a quality microphone
2. Position mic 6-12 inches from mouth
3. Speak clearly and naturally
4. Minimize background noise
5. Keep buffer below 50% (adjust step_ms if needed)
### For Development
- See `transcriber.h/cpp` for core logic
- See `main.cpp` for UI implementation
- Adjust parameters in `WhisperConfig` struct
- Enable logging in `whisper_full_params`
## 📚 Resources
- [Whisper.cpp](https://github.com/ggerganov/whisper.cpp)
- [Whisper Paper](https://arxiv.org/abs/2212.04356)
- [Model Download](https://huggingface.co/ggerganov/whisper.cpp)
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads)
- [Vulkan SDK](https://vulkan.lunarg.com/)
---
**Built with ❤️ using whisper.cpp**
Stored in `win-dictation.ini` next to the executable. Sections:
- `[app]`: window position, hotkey, model, capture device, pinned/autopaste/autohide flags
- `[timing-ggml-*.bin]`: per-model timing accumulators (learned transcription speed)
src/logging.h
+16
View File
@@ -0,0 +1,16 @@
#pragma once
#include <windows.h>
#include <cstdio>
#include <string>
inline void LogLine(const char* msg) {
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/")) + L"\\win-dictation.log";
FILE* f = nullptr;
_wfopen_s(&f, p.c_str(), L"a");
if (!f) return;
SYSTEMTIME t; GetLocalTime(&t);
fprintf(f, "%04d-%02d-%02d %02d:%02d:%02d %s\n",
t.wYear, t.wMonth, t.wDay, t.wHour, t.wMinute, t.wSecond, msg);
fclose(f);
}
src/main.cpp
+1300 -371
View File
File diff suppressed because it is too large Load Diff
src/settings.h
+42
View File
@@ -0,0 +1,42 @@
#pragma once
#include <windows.h>
#include <string>
struct AppSettings {
int captureId = 0;
std::wstring modelFile;
bool pinned = true, autoPaste = true, autoHide = false;
int winX = CW_USEDEFAULT, winY = CW_USEDEFAULT, winW = 400, winH = 340;
int hkMods = MOD_CONTROL | MOD_SHIFT;
int hkVk = VK_SPACE;
};
inline std::wstring SettingsPath() {
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
return p + L"\\win-dictation.ini";
}
inline int GetIni(const wchar_t* k, int d) { return GetPrivateProfileIntW(L"app", k, d, SettingsPath().c_str()); }
inline void PutIni(const wchar_t* k, int v) { wchar_t b[32]; wsprintfW(b, L"%d", v); WritePrivateProfileStringW(L"app", k, b, SettingsPath().c_str()); }
inline void LoadSettings(AppSettings& s) {
s.captureId = GetIni(L"captureId", s.captureId);
s.pinned = GetIni(L"pinned", s.pinned) != 0;
s.autoPaste = GetIni(L"autoPaste", s.autoPaste) != 0;
s.autoHide = GetIni(L"autoHide", s.autoHide) != 0;
s.winX = GetIni(L"winX", s.winX); s.winY = GetIni(L"winY", s.winY);
s.winW = GetIni(L"winW", s.winW); s.winH = GetIni(L"winH", s.winH);
s.hkMods = GetIni(L"hkMods", s.hkMods);
s.hkVk = GetIni(L"hkVk", s.hkVk);
wchar_t m[MAX_PATH]; GetPrivateProfileStringW(L"app", L"modelFile", L"", m, MAX_PATH, SettingsPath().c_str());
s.modelFile = m;
}
inline void SaveSettings(const AppSettings& s) {
PutIni(L"captureId", s.captureId); PutIni(L"pinned", s.pinned);
PutIni(L"autoPaste", s.autoPaste); PutIni(L"autoHide", s.autoHide);
PutIni(L"winX", s.winX); PutIni(L"winY", s.winY); PutIni(L"winW", s.winW); PutIni(L"winH", s.winH);
PutIni(L"hkMods", s.hkMods); PutIni(L"hkVk", s.hkVk);
WritePrivateProfileStringW(L"app", L"modelFile", s.modelFile.c_str(), SettingsPath().c_str());
}
src/test-audio.cpp
-254
View File
@@ -1,254 +0,0 @@
// Test program for win-dictation with audio files
#include "whisper.h"
#include "transcriber.h"
#include "common.h"
#include <iostream>
#include <fstream>
#include <vector>
#include <string>
#include <chrono>
// Simple file exists check without filesystem
bool file_exists(const std::string& name) {
std::ifstream f(name.c_str());
return f.good();
}
// WAV file header structure
struct WAVHeader {
char riff[4]; // "RIFF"
uint32_t fileSize;
char wave[4]; // "WAVE"
char fmt[4]; // "fmt "
uint32_t fmtSize;
uint16_t audioFormat;
uint16_t numChannels;
uint32_t sampleRate;
uint32_t byteRate;
uint16_t blockAlign;
uint16_t bitsPerSample;
char data[4]; // "data"
uint32_t dataSize;
};
// Load WAV file and convert to float32 mono 16kHz
bool load_wav_file(const std::string& filename, std::vector<float>& audio_data) {
std::ifstream file(filename, std::ios::binary);
if (!file) {
std::cerr << "Failed to open: " << filename << std::endl;
return false;
}
WAVHeader header;
file.read(reinterpret_cast<char*>(&header), sizeof(WAVHeader));
// Verify WAV format
if (std::string(header.riff, 4) != "RIFF" || std::string(header.wave, 4) != "WAVE") {
std::cerr << "Invalid WAV file" << std::endl;
return false;
}
// Read audio data
std::vector<int16_t> raw_data(header.dataSize / sizeof(int16_t));
file.read(reinterpret_cast<char*>(raw_data.data()), header.dataSize);
// Convert to float and resample if needed
audio_data.clear();
audio_data.reserve(raw_data.size());
for (int16_t sample : raw_data) {
audio_data.push_back(sample / 32768.0f);
}
std::cout << "Loaded: " << filename << std::endl;
std::cout << " Sample rate: " << header.sampleRate << " Hz" << std::endl;
std::cout << " Channels: " << header.numChannels << std::endl;
std::cout << " Duration: " << (audio_data.size() / (float)header.sampleRate) << " seconds" << std::endl;
return true;
}
// Test case structure
struct TestCase {
std::string name;
std::string audio_file;
std::string expected_text;
bool passed = false;
std::string actual_text;
float duration_ms = 0.0f;
};
// Test runner
class AudioTester {
public:
AudioTester(const std::string& model_path) {
m_config.model_path = model_path;
m_config.language = "en";
m_config.n_threads = std::thread::hardware_concurrency();
m_config.use_gpu = true;
// Initialize transcriber
if (!m_transcriber.init(m_config)) {
std::cerr << "Failed to initialize transcriber!" << std::endl;
exit(1);
}
std::cout << "Transcriber initialized" << std::endl;
std::cout << " GPU: " << (m_transcriber.is_using_gpu() ? "ON" : "OFF") << std::endl;
std::cout << " Threads: " << m_config.n_threads << std::endl;
}
bool run_test(TestCase& test) {
std::cout << "\n=== Test: " << test.name << " ===" << std::endl;
// Load audio file
std::vector<float> audio_data;
if (!load_wav_file(test.audio_file, audio_data)) {
test.passed = false;
return false;
}
// Process audio
m_result_text.clear();
auto start = std::chrono::high_resolution_clock::now();
whisper_context* ctx = whisper_init_from_file_with_params(
m_config.model_path.c_str(),
whisper_context_default_params()
);
if (!ctx) {
std::cerr << "Failed to load model!" << std::endl;
return false;
}
whisper_full_params wparams = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wparams.language = "en";
wparams.n_threads = m_config.n_threads;
wparams.print_progress = false;
wparams.print_realtime = false;
int result = whisper_full(ctx, wparams, audio_data.data(), (int)audio_data.size());
if (result == 0) {
const int n_segments = whisper_full_n_segments(ctx);
for (int i = 0; i < n_segments; ++i) {
const char* text = whisper_full_get_segment_text(ctx, i);
if (text) {
m_result_text += text;
}
}
}
whisper_free(ctx);
auto end = std::chrono::high_resolution_clock::now();
test.duration_ms = std::chrono::duration<float, std::milli>(end - start).count();
// Store result
test.actual_text = m_result_text;
// Trim and compare
std::string actual_trimmed = trim(m_result_text);
std::string expected_trimmed = trim(test.expected_text);
// Case-insensitive comparison
std::transform(actual_trimmed.begin(), actual_trimmed.end(), actual_trimmed.begin(), ::tolower);
std::transform(expected_trimmed.begin(), expected_trimmed.end(), expected_trimmed.begin(), ::tolower);
test.passed = (actual_trimmed.find(expected_trimmed) != std::string::npos);
// Print results
std::cout << "Expected: \"" << test.expected_text << "\"" << std::endl;
std::cout << "Actual: \"" << test.actual_text << "\"" << std::endl;
std::cout << "Duration: " << test.duration_ms << " ms" << std::endl;
std::cout << "Result: " << (test.passed ? "✓ PASS" : "✗ FAIL") << std::endl;
return test.passed;
}
private:
WhisperConfig m_config;
Transcriber m_transcriber;
std::string m_result_text;
std::string trim(const std::string& str) {
size_t start = str.find_first_not_of(" \t\n\r");
size_t end = str.find_last_not_of(" \t\n\r");
if (start == std::string::npos || end == std::string::npos) {
return "";
}
return str.substr(start, end - start + 1);
}
};
int main(int argc, char** argv) {
std::cout << "=== Whisper Dictation Audio Tests ===" << std::endl;
// Determine model path
std::string model_path = "models/ggml-base.en.bin";
if (argc > 1) {
model_path = argv[1];
}
std::cout << "Using model: " << model_path << std::endl;
// Create tester
AudioTester tester(model_path);
// Define test cases
std::vector<TestCase> tests = {
{"Short sentence", "test-audio/test1.wav", "hello world"},
{"Numbers", "test-audio/test2.wav", "one two three four five"},
{"Long sentence", "test-audio/test3.wav", "the quick brown fox jumps over the lazy dog"},
};
// Check if test audio directory exists
if (!file_exists("test-audio/test1.wav")) {
std::cout << "\nNo test-audio directory found. Creating example..." << std::endl;
std::cout << "Please add your test WAV files (16kHz, mono) to test-audio/" << std::endl;
std::cout << "\nYou can record test audio with:" << std::endl;
std::cout << " ffmpeg -f dshow -i audio=\"Your Microphone\" -t 5 -ar 16000 -ac 1 test-audio/test1.wav" << std::endl;
std::cout << "\nOr use the recording script:" << std::endl;
std::cout << " powershell -ExecutionPolicy Bypass -File record-test-audio.ps1" << std::endl;
// Try to find user-provided test files
if (argc > 2) {
std::cout << "\nRunning with user-provided files..." << std::endl;
tests.clear();
for (int i = 2; i < argc; i += 2) {
if (i + 1 < argc) {
tests.push_back({
argv[i],
argv[i],
argv[i + 1]
});
}
}
} else {
return 1;
}
}
// Run tests
int passed = 0;
int failed = 0;
for (auto& test : tests) {
if (tester.run_test(test)) {
passed++;
} else {
failed++;
}
}
// Summary
std::cout << "\n=== Test Summary ===" << std::endl;
std::cout << "Total: " << (passed + failed) << std::endl;
std::cout << "Passed: " << passed << std::endl;
std::cout << "Failed: " << failed << std::endl;
std::cout << "Success rate: " << (passed * 100.0f / (passed + failed)) << "%" << std::endl;
return (failed == 0) ? 0 : 1;
}
src/text_util.h
+8
View File
@@ -0,0 +1,8 @@
#pragma once
#include <string>
inline std::wstring append_transcript(const std::wstring& cur, const std::wstring& add) {
if (add.empty()) return cur;
if (cur.empty()) return add;
return cur + L" " + add;
}
src/timing.h
+121
View File
@@ -0,0 +1,121 @@
#pragma once
#include <windows.h>
#include <string>
#include <cmath>
#include <algorithm>
struct TimingModel {
double n=0, sx=0, sy=0, sxx=0, sxy=0;
double a=0, b=0;
bool fitted=false;
double def_a=0.4, def_b=0.6;
void recompute() {
if (n >= 2.0) {
double denom = n*sxx - sx*sx;
if (std::fabs(denom) > 1e-9) {
double bb = (n*sxy - sx*sy) / denom;
double aa = (sy - bb*sx) / n;
if (bb < 0.02) bb = def_b;
if (aa < 0.0) aa = 0.0;
a=aa; b=bb; fitted=true; return;
}
}
a=def_a; b=def_b; fitted=false;
}
double predict(double audio_sec) const {
double t = (fitted ? a : def_a) + (fitted ? b : def_b) * audio_sec;
return std::max(0.4, t);
}
void add_sample(double audio_sec, double proc_sec) {
const double decay = 0.97;
n*=decay; sx*=decay; sy*=decay; sxx*=decay; sxy*=decay;
n+=1; sx+=audio_sec; sy+=proc_sec;
sxx+=audio_sec*audio_sec; sxy+=audio_sec*proc_sec;
recompute();
}
};
struct ProgressEstimator {
double T_hat=1.0, disp_rem=1.0, t=0.0;
bool done=false;
void begin(double T_pred) {
T_hat = std::max(0.4, T_pred);
disp_rem = T_hat; t = 0.0; done=false;
}
void on_whisper(double t_now, int p) {
if (done || p < 5) return;
double T_meas = 100.0 * t_now / (double)p;
const double alpha = 0.5;
T_hat = (1.0-alpha)*T_hat + alpha*T_meas;
if (T_hat < t_now) T_hat = t_now;
}
void tick(double dt, float& out_frac, float& out_remaining) {
if (done) { out_frac=1.0f; out_remaining=0.0f; return; }
t += dt;
disp_rem -= dt;
double raw_rem = std::max(0.0, T_hat - t);
const double maxCatchUp = 2.5;
double err = raw_rem - disp_rem;
if (err < 0) disp_rem += std::max(err, -maxCatchUp*dt);
if (disp_rem < 0) disp_rem = 0;
double frac = (t + disp_rem > 1e-6) ? t/(t+disp_rem) : 0.0;
if (frac > 0.95) frac = 0.95;
out_frac = (float)frac;
out_remaining = (float)disp_rem;
}
void finish(float& out_frac, float& out_remaining) {
done=true; out_frac=1.0f; out_remaining=0.0f;
}
void reset_busy() {
T_hat=1.0; disp_rem=1.0; t=0.0; done=false;
}
};
inline std::wstring TimingIniPath() {
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
return p + L"\\win-dictation.ini";
}
inline std::wstring SectionFor(const std::string& modelPath) {
std::string base = modelPath.substr(modelPath.find_last_of("\\/")+1);
return L"timing-" + std::wstring(base.begin(), base.end());
}
inline void PutD(const std::wstring& sec, const wchar_t* k, double v) {
wchar_t b[64]; swprintf_s(b, L"%.6f", v);
WritePrivateProfileStringW(sec.c_str(), k, b, TimingIniPath().c_str());
}
inline double GetD(const std::wstring& sec, const wchar_t* k, double d) {
wchar_t b[64]; swprintf_s(b, L"%.6f", d);
wchar_t out[64];
GetPrivateProfileStringW(sec.c_str(), k, b, out, 64, TimingIniPath().c_str());
return wcstod(out, nullptr);
}
inline void LoadTiming(TimingModel& m, const std::string& modelPath) {
auto s = SectionFor(modelPath);
m.n=GetD(s,L"n",0); m.sx=GetD(s,L"sx",0); m.sy=GetD(s,L"sy",0);
m.sxx=GetD(s,L"sxx",0); m.sxy=GetD(s,L"sxy",0);
m.recompute();
}
inline void SaveTiming(const TimingModel& m, const std::string& modelPath) {
auto s = SectionFor(modelPath);
PutD(s,L"n",m.n); PutD(s,L"sx",m.sx); PutD(s,L"sy",m.sy);
PutD(s,L"sxx",m.sxx); PutD(s,L"sxy",m.sxy);
}
inline void SeedDefaults(TimingModel& m, const std::string& modelPath) {
std::string p = modelPath;
auto has = [&](const char* s){ return p.find(s)!=std::string::npos; };
if (has("tiny")) { m.def_a=0.3; m.def_b=0.45; }
else if (has("base")) { m.def_a=0.5; m.def_b=1.10; }
else if (has("small")){ m.def_a=0.8; m.def_b=3.00; }
else { m.def_a=0.5; m.def_b=1.00; }
m.recompute();
}
src/transcriber.cpp
+185 -364
View File
@@ -1,383 +1,204 @@
#include "transcriber.h"
#include "whisper.h"
// Note: WHISPER_SAMPLE_RATE is defined in whisper.h, so common.h is not needed
#include <SDL.h>
#include <SDL_audio.h>
#include <iostream>
#include <chrono>
#include <cmath>
#include <numeric>
#include <algorithm>
#include <cmath>
#include <cstring>
Transcriber::Transcriber() {
m_ring_buffer.resize(RING_BUFFER_SIZE, 0.0f);
Transcriber::~Transcriber() {
cancel();
if (m_worker.joinable()) m_worker.join();
if (m_ctx) whisper_free(m_ctx);
}
Transcriber::~Transcriber() {
stop();
free_model();
int Transcriber::default_threads() {
unsigned hc = std::thread::hardware_concurrency();
if (hc <= 2) return (int)std::max(1u, hc);
return (int)(hc / 2);
}
float Transcriber::recorded_seconds() const {
std::lock_guard<std::mutex> lk(m_capture_mtx);
return (float)(m_capture.size() / (double)WHISPER_SAMPLE_RATE);
}
void Transcriber::s_progress(whisper_context*, whisper_state*, int p, void* ud) {
auto* self = static_cast<Transcriber*>(ud);
if (self && self->m_on_progress) self->m_on_progress(p);
}
bool Transcriber::s_abort(void* ud) {
auto* self = static_cast<Transcriber*>(ud);
return self && self->m_abort.load();
}
bool Transcriber::preload(const WhisperConfig& cfg) {
std::lock_guard<std::mutex> lk(m_cfg_mtx);
m_cfg = cfg;
if (m_cfg.n_threads <= 0) m_cfg.n_threads = default_threads();
if (m_ctx) return true;
whisper_context_params cp = whisper_context_default_params();
cp.use_gpu = m_cfg.use_gpu;
m_ctx = whisper_init_from_file_with_params(m_cfg.model_path.c_str(), cp);
return m_ctx != nullptr;
}
static void sdl_capture_cb(void* user, Uint8* stream, int len) {
auto* self = static_cast<Transcriber*>(user);
self->on_audio(reinterpret_cast<float*>(stream), len / (int)sizeof(float));
}
bool Transcriber::start_recording() {
if (m_recording.load() || m_busy.load()) return false;
{
std::lock_guard<std::mutex> lk(m_capture_mtx);
m_capture.clear();
m_capture.reserve(WHISPER_SAMPLE_RATE * 30);
}
SDL_AudioSpec want{}, have{};
want.freq = WHISPER_SAMPLE_RATE;
want.format = AUDIO_F32;
want.channels = 1;
want.samples = 1024;
want.callback = sdl_capture_cb;
want.userdata = this;
const char* dev = SDL_GetAudioDeviceName(m_cfg.capture_id, SDL_TRUE);
m_dev = SDL_OpenAudioDevice(dev, SDL_TRUE, &want, &have, 0);
if (!m_dev) return false;
m_energy = 0.0f;
m_recording = true;
SDL_PauseAudioDevice(m_dev, 0);
return true;
}
void Transcriber::on_audio(const float* s, int n) {
if (n <= 0 || !m_recording.load()) return;
double sq = 0.0;
for (int i = 0; i < n; ++i) sq += (double)s[i] * s[i];
float rms = (float)std::sqrt(sq / n);
float e = m_energy.load();
m_energy = std::min(1.0f, e * 0.6f + (rms * 4.0f) * 0.4f);
std::lock_guard<std::mutex> lk(m_capture_mtx);
m_capture.insert(m_capture.end(), s, s + n);
}
void Transcriber::cancel() {
if (!m_recording.load()) return;
m_recording = false;
if (m_dev) { SDL_PauseAudioDevice(m_dev, 1); SDL_CloseAudioDevice(m_dev); m_dev = 0; }
std::lock_guard<std::mutex> lk(m_capture_mtx);
m_capture.clear();
m_energy = 0.0f;
}
void Transcriber::stop_and_transcribe() {
if (!m_recording.load()) return;
m_recording = false;
if (m_dev) { SDL_PauseAudioDevice(m_dev, 1); SDL_CloseAudioDevice(m_dev); m_dev = 0; }
m_energy = 0.0f;
std::vector<float> audio;
{ std::lock_guard<std::mutex> lk(m_capture_mtx); audio.swap(m_capture); }
if (audio.size() < (size_t)(WHISPER_SAMPLE_RATE * 0.3)) {
if (m_on_result) m_on_result("");
return;
}
if (m_worker.joinable()) m_worker.join();
m_busy = true;
m_worker = std::thread(&Transcriber::transcribe_worker, this, std::move(audio));
}
static void trim_silence(std::vector<float>& a, float thresh = 0.01f) {
const size_t win = 1600;
auto loud = [&](size_t i) {
float m = 0.f;
for (size_t k = i; k < std::min(a.size(), i + win); ++k)
m = std::max(m, std::fabs(a[k]));
return m > thresh;
};
size_t s = 0, e = a.size();
while (s + win < a.size() && !loud(s)) s += win;
while (e > win && !loud(e - win)) e -= win;
if (s + win <= e)
a.assign(a.begin() + (s > win ? s - win : 0), a.begin() + e);
}
static std::string clean_text(std::string s) {
const char* junk[] = {"[BLANK_AUDIO]", "[NOISE]", "(blank)", "(noise)", "[ Silence ]"};
for (auto j : junk) {
size_t p;
while ((p = s.find(j)) != std::string::npos) s.erase(p, strlen(j));
}
size_t b = s.find_first_not_of(" \t\r\n");
size_t e = s.find_last_not_of(" \t\r\n");
return (b == std::string::npos) ? "" : s.substr(b, e - b + 1);
}
std::string Transcriber::run_inference(std::vector<float>& audio) {
if (!m_ctx) return "";
m_abort = false;
if (m_cfg.trim_silence) trim_silence(audio);
m_audio_seconds = (float)(audio.size() / (double)WHISPER_SAMPLE_RATE);
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wp.print_progress = false;
wp.print_realtime = false;
wp.print_timestamps = false;
wp.no_timestamps = true;
wp.translate = false;
wp.language = m_cfg.language.c_str();
wp.n_threads = m_cfg.n_threads;
wp.no_context = true;
wp.suppress_blank = true;
wp.suppress_nst = true;
wp.temperature = 0.0f;
wp.progress_callback = &Transcriber::s_progress;
wp.progress_callback_user_data = this;
wp.abort_callback = &Transcriber::s_abort;
wp.abort_callback_user_data = this;
std::string out;
if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
int n = whisper_full_n_segments(m_ctx);
for (int i = 0; i < n; ++i) {
const char* t = whisper_full_get_segment_text(m_ctx, i);
if (t) out += t;
}
out = clean_text(out);
}
return out;
}
void Transcriber::transcribe_worker(std::vector<float> audio) {
std::string out = run_inference(audio);
m_busy = false;
if (m_on_result) m_on_result(out);
}
std::string Transcriber::transcribe_sync(std::vector<float> audio) {
return run_inference(audio);
}
bool Transcriber::reload(const WhisperConfig& cfg) {
if (m_recording.load() || m_busy.load()) return false;
if (m_worker.joinable()) m_worker.join();
{
std::lock_guard<std::mutex> lk(m_cfg_mtx);
if (m_ctx) { whisper_free(m_ctx); m_ctx = nullptr; }
}
return preload(cfg);
}
std::vector<std::string> Transcriber::get_audio_devices() {
std::vector<std::string> devices;
if (SDL_Init(SDL_INIT_AUDIO) < 0) {
return devices;
}
if (SDL_Init(SDL_INIT_AUDIO) < 0) return devices;
int nDevices = SDL_GetNumAudioDevices(SDL_TRUE);
for (int i = 0; i < nDevices; ++i) {
const char* name = SDL_GetAudioDeviceName(i, SDL_TRUE);
if (name) {
devices.push_back(name);
}
if (name) devices.push_back(name);
}
return devices;
}
bool Transcriber::init(const WhisperConfig& config) {
m_config = config;
// Load model immediately to check GPU availability
std::lock_guard<std::mutex> lock(m_mutex);
if (!m_ctx) {
struct whisper_context_params cparams = whisper_context_default_params();
cparams.use_gpu = m_config.use_gpu;
m_ctx = whisper_init_from_file_with_params(m_config.model_path.c_str(), cparams);
if (!m_ctx) {
return false;
}
// Check if GPU is actually active
const char* info = whisper_print_system_info();
m_gpu_active = (info && (strstr(info, "CUDA") != nullptr ||
strstr(info, "Metal") != nullptr ||
strstr(info, "HIP") != nullptr ||
strstr(info, "Vulkan") != nullptr));
}
return true;
}
void Transcriber::free_model() {
std::lock_guard<std::mutex> lock(m_mutex);
if (m_ctx) {
whisper_free(m_ctx);
m_ctx = nullptr;
}
}
void Transcriber::start() {
if (m_running) return;
m_should_stop = false;
// Clear ring buffer completely
m_ring_write_pos = 0;
m_ring_read_pos = 0;
// Clear processing buffer
m_processing_buffer.clear();
m_last_process_time = std::chrono::steady_clock::now();
m_worker = std::thread(&Transcriber::worker_loop, this);
m_running = true;
}
void Transcriber::stop() {
if (!m_running) return;
m_should_stop = true;
m_ring_cv.notify_all();
// Wait for worker thread to complete
if (m_worker.joinable()) {
m_worker.join();
}
// Clear ALL state to prevent contamination
{
std::lock_guard<std::mutex> lock(m_ring_mutex);
// Reset ring buffer positions
m_ring_write_pos = 0;
m_ring_read_pos = 0;
// Clear ring buffer data
std::fill(m_ring_buffer.begin(), m_ring_buffer.end(), 0.0f);
}
// Clear processing buffer
m_processing_buffer.clear();
m_running = false;
m_audio_energy = 0.0f;
}
void Transcriber::set_callback(Callback cb) {
std::lock_guard<std::mutex> lock(m_mutex);
m_callback = cb;
}
float Transcriber::get_audio_energy() {
return m_audio_energy;
}
size_t Transcriber::get_queue_size() {
size_t write_pos = m_ring_write_pos.load();
size_t read_pos = m_ring_read_pos.load();
if (write_pos >= read_pos) {
return write_pos - read_pos;
} else {
return RING_BUFFER_SIZE - read_pos + write_pos;
}
}
float Transcriber::get_buffer_fullness() {
return (float)get_queue_size() / (float)RING_BUFFER_SIZE;
}
bool Transcriber::is_using_gpu() const {
return m_gpu_active;
}
// Ring buffer audio callback - NO DATA LOSS
void Transcriber::audio_callback(const float* samples, int n_samples) {
if (n_samples <= 0) return;
// Calculate RMS for VU meter (with smoothing)
double sum_sq = 0.0;
for (int i = 0; i < n_samples; i++) {
sum_sq += samples[i] * samples[i];
}
float rms = (float)std::sqrt(sum_sq / n_samples);
// Smooth the energy reading for better visual effect
float current_energy = m_audio_energy.load();
float new_energy = current_energy * 0.7f + (rms * 5.0f) * 0.3f;
m_audio_energy = std::min(1.0f, new_energy);
// Write to ring buffer (lock-free for audio thread)
size_t write_pos = m_ring_write_pos.load(std::memory_order_acquire);
for (int i = 0; i < n_samples; i++) {
size_t next_pos = (write_pos + 1) % RING_BUFFER_SIZE;
// Check if buffer is full (would overwrite unread data)
if (next_pos == m_ring_read_pos.load(std::memory_order_acquire)) {
// Buffer full - drop oldest samples (shouldn't happen with 30s buffer)
m_ring_read_pos.store((m_ring_read_pos.load() + 1) % RING_BUFFER_SIZE, std::memory_order_release);
}
m_ring_buffer[write_pos] = samples[i];
write_pos = next_pos;
}
m_ring_write_pos.store(write_pos, std::memory_order_release);
m_ring_cv.notify_one();
}
// SDL callback wrapper
static void sdl_audio_callback(void* userdata, Uint8* stream, int len) {
Transcriber* self = (Transcriber*)userdata;
int n_samples = len / sizeof(float);
float* samples = (float*)stream;
self->audio_callback(samples, n_samples);
}
void Transcriber::process_audio_chunk(const std::vector<float>& audio_data) {
if (audio_data.empty()) return;
// Minimum audio length check (at least 1 second for reliable transcription)
const size_t min_samples = WHISPER_SAMPLE_RATE; // 1 second
if (audio_data.size() < min_samples) {
return; // Need more audio data
}
// Basic energy check - skip completely silent audio
float max_energy = 0.0f;
for (float sample : audio_data) {
max_energy = std::max(max_energy, std::abs(sample));
}
if (max_energy < 0.001f) { // Essentially silent
return;
}
// Run Whisper inference
whisper_full_params wparams = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wparams.print_progress = false;
wparams.print_realtime = false;
wparams.print_timestamps = false;
wparams.language = m_config.language.c_str();
wparams.n_threads = m_config.n_threads;
wparams.no_context = true; // CRITICAL: Don't reuse previous text as context!
wparams.single_segment = false;
wparams.suppress_blank = true; // Suppress blank outputs
// Reset the context state before each inference to prevent contamination
whisper_reset_timings(m_ctx);
int result = whisper_full(m_ctx, wparams, audio_data.data(), (int)audio_data.size());
if (result != 0) {
return; // Skip this chunk on error
}
// Get transcribed text and filter blanks
const int n_segments = whisper_full_n_segments(m_ctx);
std::string segment_text;
for (int i = 0; i < n_segments; ++i) {
const char* text = whisper_full_get_segment_text(m_ctx, i);
if (text && strlen(text) > 0) {
std::string seg(text);
// Filter out blank/noise tokens
if (seg.find("[BLANK_AUDIO]") == std::string::npos &&
seg.find("[NOISE]") == std::string::npos &&
seg.find("(blank)") == std::string::npos &&
seg.find("(noise)") == std::string::npos &&
seg != " " && seg != " ") {
segment_text += seg;
}
}
}
// Send callback only if we have real content
if (!segment_text.empty()) {
// Trim whitespace
size_t start = segment_text.find_first_not_of(" \t\n\r");
size_t end = segment_text.find_last_not_of(" \t\n\r");
if (start != std::string::npos && end != std::string::npos) {
segment_text = segment_text.substr(start, end - start + 1);
// Only send if meaningful content (at least 2 characters)
if (segment_text.length() >= 2) {
std::lock_guard<std::mutex> lock(m_mutex);
if (m_callback) {
m_callback(segment_text);
}
}
}
}
}
void Transcriber::worker_loop() {
// Model should already be loaded from init()
if (!m_ctx) {
std::lock_guard<std::mutex> lock(m_mutex);
if (m_callback) m_callback("[Error: Model not loaded]\n");
return;
}
// Initialize SDL Audio
if (SDL_Init(SDL_INIT_AUDIO) < 0) {
std::lock_guard<std::mutex> lock(m_mutex);
if (m_callback) m_callback("[Error: SDL Init failed]\n");
return;
}
SDL_AudioSpec capture_spec_requested;
SDL_AudioSpec capture_spec_obtained;
SDL_zero(capture_spec_requested);
SDL_zero(capture_spec_obtained);
capture_spec_requested.freq = WHISPER_SAMPLE_RATE;
capture_spec_requested.format = AUDIO_F32;
capture_spec_requested.channels = 1;
capture_spec_requested.samples = 512; // Smaller buffer for lower latency
capture_spec_requested.callback = sdl_audio_callback;
capture_spec_requested.userdata = this;
const char* device_name = SDL_GetAudioDeviceName(m_config.capture_id, SDL_TRUE);
m_dev_id_in = SDL_OpenAudioDevice(
device_name,
SDL_TRUE,
&capture_spec_requested,
&capture_spec_obtained,
0
);
if (!m_dev_id_in) {
std::lock_guard<std::mutex> lock(m_mutex);
if (m_callback) m_callback("[Error: Failed to open audio device]\n");
return;
}
SDL_PauseAudioDevice(m_dev_id_in, 0); // Start capturing
// Processing parameters
const size_t n_samples_step = (size_t)((1e-3 * m_config.step_ms) * WHISPER_SAMPLE_RATE);
const size_t n_samples_len = (size_t)((1e-3 * m_config.length_ms) * WHISPER_SAMPLE_RATE);
const size_t n_samples_keep = (size_t)((1e-3 * 200) * WHISPER_SAMPLE_RATE); // Keep 200ms overlap
m_processing_buffer.clear();
m_processing_buffer.reserve(n_samples_len * 2);
while (!m_should_stop) {
// Wait for audio data with shorter timeout for responsiveness
{
std::unique_lock<std::mutex> lock(m_ring_mutex);
m_ring_cv.wait_for(lock, std::chrono::milliseconds(50), [&]{
return get_queue_size() >= n_samples_step || m_should_stop;
});
}
if (m_should_stop) break;
// Read from ring buffer - need enough data for reliable transcription
size_t available = get_queue_size();
if (available < n_samples_step) { // Need at least full threshold
continue;
}
// Read samples from ring buffer
std::vector<float> new_samples;
new_samples.reserve(available);
size_t read_pos = m_ring_read_pos.load(std::memory_order_acquire);
size_t write_pos = m_ring_write_pos.load(std::memory_order_acquire);
while (read_pos != write_pos) {
new_samples.push_back(m_ring_buffer[read_pos]);
read_pos = (read_pos + 1) % RING_BUFFER_SIZE;
}
m_ring_read_pos.store(read_pos, std::memory_order_release);
// Append new samples to processing buffer
m_processing_buffer.insert(m_processing_buffer.end(), new_samples.begin(), new_samples.end());
// Process when we have enough data
if (m_processing_buffer.size() >= n_samples_len) {
// Take exactly n_samples_len for processing
std::vector<float> chunk(
m_processing_buffer.end() - n_samples_len,
m_processing_buffer.end()
);
// Process this chunk
process_audio_chunk(chunk);
// CRITICAL: Remove processed audio, keep only overlap for continuity
// This prevents re-processing the same audio repeatedly!
size_t samples_to_remove = m_processing_buffer.size() - n_samples_keep;
if (samples_to_remove > 0) {
m_processing_buffer.erase(
m_processing_buffer.begin(),
m_processing_buffer.begin() + samples_to_remove
);
}
}
}
// Process any remaining audio
if (!m_processing_buffer.empty()) {
process_audio_chunk(m_processing_buffer);
}
SDL_CloseAudioDevice(m_dev_id_in);
SDL_Quit();
}
src/transcriber.h
+50 -57
View File
@@ -2,83 +2,76 @@
#include <string>
#include <vector>
#include <deque>
#include <thread>
#include <mutex>
#include <atomic>
#include <functional>
#include <condition_variable>
#include <memory>
struct whisper_context;
struct WhisperConfig {
std::string model_path;
std::string model_path = "models/ggml-tiny.en.bin";
std::string language = "en";
int n_threads = std::thread::hardware_concurrency(); // Use all available threads
int step_ms = 1000; // Process every 1s (reliable transcription)
int length_ms = 6000; // 6s context window (good balance)
bool use_gpu = true; // Auto-detect and use if available
int capture_id = 0; // Default to first device
int n_gpu_layers = -1; // -1 = auto (all layers if GPU available)
int n_threads = 0; // 0 = auto (physical cores)
bool use_gpu = false;
int capture_id = 0;
bool trim_silence = true;
};
class Transcriber {
public:
using Callback = std::function<void(const std::string&)>;
using ResultCb = std::function<void(const std::string&)>;
using ProgressCb = std::function<void(int)>;
Transcriber();
Transcriber() = default;
~Transcriber();
bool init(const WhisperConfig& config);
void start();
void stop();
void set_callback(Callback cb);
bool is_running() const { return m_running; }
bool preload(const WhisperConfig& cfg);
bool reload(const WhisperConfig& cfg);
bool is_loaded() const { return m_ctx != nullptr; }
int threads() const { return m_cfg.n_threads; }
// Audio device management
bool start_recording();
void stop_and_transcribe();
void cancel();
bool is_recording() const { return m_recording.load(); }
bool is_busy() const { return m_busy.load(); }
float get_audio_energy() const { return m_energy.load(); }
float audio_seconds() const { return m_audio_seconds.load(); }
float recorded_seconds() const;
void set_result_callback(ResultCb cb) { m_on_result = std::move(cb); }
void set_progress_callback(ProgressCb cb) { m_on_progress = std::move(cb); }
void request_cancel() { m_abort = true; }
void on_audio(const float* samples, int n);
std::string transcribe_sync(std::vector<float> audio);
static std::vector<std::string> get_audio_devices();
float get_audio_energy(); // 0.0 to 1.0 (normalized)
// Status
bool is_using_gpu() const;
size_t get_queue_size();
float get_buffer_fullness(); // 0.0 to 1.0
// Resource management
void free_model();
// Internal audio callback (public so C callback can reach it)
void audio_callback(const float* samples, int n_samples);
private:
void worker_loop();
void process_audio_chunk(const std::vector<float>& audio_data);
void transcribe_worker(std::vector<float> audio);
std::string run_inference(std::vector<float>& audio);
static int default_threads();
WhisperConfig m_cfg;
std::mutex m_cfg_mtx;
whisper_context* m_ctx = nullptr;
unsigned int m_dev = 0;
std::vector<float> m_capture;
mutable std::mutex m_capture_mtx;
std::atomic<bool> m_recording{false};
std::atomic<bool> m_busy{false};
std::atomic<float> m_energy{0.0f};
std::atomic<float> m_audio_seconds{0.0f};
std::atomic<bool> m_abort{false};
WhisperConfig m_config;
std::atomic<bool> m_running{false};
std::atomic<bool> m_should_stop{false};
std::thread m_worker;
std::mutex m_mutex;
Callback m_callback;
ResultCb m_on_result;
ProgressCb m_on_progress;
// Shared audio energy level (smoothed)
std::atomic<float> m_audio_energy{0.0f};
std::atomic<bool> m_gpu_active{false};
// Audio Capture State
uint32_t m_dev_id_in = 0;
// Ring buffer for audio - prevents any loss
static constexpr size_t RING_BUFFER_SIZE = 16000 * 30; // 30 seconds max buffer
std::vector<float> m_ring_buffer;
std::atomic<size_t> m_ring_write_pos{0};
std::atomic<size_t> m_ring_read_pos{0};
std::mutex m_ring_mutex;
std::condition_variable m_ring_cv;
struct whisper_context* m_ctx = nullptr;
// Processing buffer to maintain context
std::vector<float> m_processing_buffer;
std::chrono::steady_clock::time_point m_last_process_time;
static void s_progress(struct whisper_context*, struct whisper_state*, int p, void* ud);
static bool s_abort(void* ud);
};
tests/test_core.cpp
+81
View File
@@ -0,0 +1,81 @@
#include "transcriber.h"
#include "text_util.h"
#include <cstdio>
#include <cstdint>
#include <fstream>
#include <vector>
#include <string>
#include <cctype>
static int g_fail = 0;
#define CHECK(cond, msg) do { if (!(cond)) { printf(" FAIL: %s\n", msg); ++g_fail; } \
else printf(" ok: %s\n", msg); } while(0)
static bool load_wav(const std::string& path, std::vector<float>& out) {
std::ifstream f(path, std::ios::binary);
if (!f) return false;
char hdr[44];
f.read(hdr, 44);
if (std::string(hdr, 4) != "RIFF") return false;
std::vector<int16_t> pcm((std::istreambuf_iterator<char>(f)), {});
out.clear(); out.reserve(pcm.size());
for (int16_t s : pcm) out.push_back(s / 32768.0f);
return !out.empty();
}
static std::string lower(std::string s) { for (char& c : s) c = (char)tolower((unsigned char)c); return s; }
int main(int argc, char** argv) {
std::string model = (argc > 1) ? argv[1] : "models/ggml-tiny.en.bin";
std::string wav = (argc > 2) ? argv[2] : "samples/jfk.wav";
CHECK(append_transcript(L"", L"hello") == L"hello", "append: empty + a");
CHECK(append_transcript(L"hello", L"world") == L"hello world", "append: a + b spaced");
CHECK(append_transcript(L"hello", L"") == L"hello", "append: a + empty");
{
Transcriber t; WhisperConfig bad; bad.model_path = "models\\does-not-exist.bin";
bool ok = t.preload(bad);
CHECK(!ok, "bad model path: preload returns false");
std::vector<float> a(16000, 0.0f);
std::string r = t.transcribe_sync(a);
CHECK(r.empty(), "bad model path: transcribe_sync returns empty, no crash");
}
{
Transcriber t; WhisperConfig cfg; cfg.model_path = model; cfg.n_threads = 4;
bool ok = t.preload(cfg);
CHECK(ok, "model loads");
if (ok) {
int last = -1, maxp = 0; bool monotonic = true;
t.set_progress_callback([&](int p) {
if (p < last) monotonic = false;
last = p; if (p > maxp) maxp = p;
});
std::vector<float> audio;
bool loaded = load_wav(wav, audio);
CHECK(loaded, "wav loads");
if (loaded) {
std::string text = lower(t.transcribe_sync(audio));
CHECK(!text.empty(), "transcription is non-empty");
CHECK(text.find("country") != std::string::npos, "transcription contains 'country'");
CHECK(monotonic, "progress is non-decreasing");
CHECK(maxp >= 95, "progress reaches ~100%");
}
}
}
{
Transcriber t; WhisperConfig cfg; cfg.model_path = model;
if (t.preload(cfg)) {
std::vector<float> tiny(100, 0.1f);
std::string r = t.transcribe_sync(tiny);
CHECK(true, "short audio did not crash");
}
}
printf("\n%s (%d failure%s)\n", g_fail ? "TESTS FAILED" : "ALL TESTS PASSED",
g_fail, g_fail == 1 ? "" : "s");
return g_fail ? 1 : 0;
}
whisper/CMakeLists.txt
+8
View File
@@ -0,0 +1,8 @@
# Whisper library root CMakeLists
# This falls through to whisper/src which contains the actual build logic
if(NOT TARGET ggml)
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../ggml ${CMAKE_CURRENT_BINARY_DIR}/ggml)
endif()
add_subdirectory(src)
win-dictation-2-0-rebuild-spec-fast-compact-push-to-talk.md
+593
View File
@@ -0,0 +1,593 @@
# Win Dictation 2.0 — Rebuild Spec (fast, compact, push-to-talk)
## 1. Summary & Assumptions
This spec rebuilds the existing `whisper.cpp/examples/win-dictation` C++/Win32 app into a **fast, compact, push-to-talk dictation tool**. The headline change is architectural, not cosmetic: stop doing live streaming transcription and instead **record audio cheaply, then run Whisper exactly once when you stop**. That single change is what makes it usable on a 2-core CPU.
**Assumptions made** (you didn't pick on the two questions — flip any of these freely):
- **Transcription model: batch / record-then-transcribe.** You hit record, speak, hit stop; ~1–2s later the text appears. No live word-by-word feed. This is the big performance win and is also more accurate.
- **Output: copy to clipboard + auto-paste into the app you were last in.** A toggle lets you fall back to copy-only.
- **English-only**, CPU-only, model `ggml-tiny.en.bin` by default (with an easy switch to `base.en` / quantized).
- **Toggle hotkey** (press once to start, again to stop) rather than hold-to-talk. Hold-to-talk is included as an optional add-on in §7.
**What changes, at a glance:**
| Area | Today | 2.0 |
|---|---|---|
| Inference | Rolling 5–6s window every ~0.4s (needs many cores) | One `whisper_full` call per utterance |
| CPU while speaking | Pegged (continuous inference) | ~0% (just buffering audio) |
| Threads | 4 on 2 physical cores (UI starves) | = physical cores (default 2), UI stays responsive |
| Model load | On first record (blocks UI) | Preloaded in background at startup |
| Window | 720×600, not on top | Compact ~360×180, always-on-top, pin toggle |
| Get text out | Manually select + Ctrl+C | Auto-copied; optional auto-paste into last app |
| Launch | exe | exe + desktop shortcut, single-instance, start-to-tray |
**I cannot build or test Windows binaries in my environment** — every snippet below is written against the whisper.cpp API your code already uses (`whisper_init_from_file_with_params`, `whisper_full`, …) and standard Win32/SDL2. Build on your machine and send me any compiler errors; I'll fix them.
## 2. Root Cause — why it's slow today
Your `transcriber.cpp` `worker_loop` implements the classic whisper.cpp *stream* pattern:
- `length_ms` ≈ 5000–6000 → every inference transcribes a **5–6 second** window.
- `step_ms` ≈ 400–1000 → it tries to do that **every 0.4–1s**, keeping a 200ms overlap.
For this to keep up, your CPU must transcribe 6s of audio in well under 1s — i.e. **>6× real-time**. The README's reference numbers (10–15× real-time) were measured on a **24-thread** machine. Your i5-7th-gen is **2 cores / 4 threads** and will do tiny.en at roughly **2–4× real-time** at best. Consequences:
1. **Unbounded backlog.** Each 6s window takes ~1.5–2.5s to process, but a new one is requested every 0.4s. The ring buffer fills, latency grows the longer you talk, and `get_buffer_fullness()` climbs toward 100%.
2. **Wasted re-work.** The sliding window + overlap re-transcribes much of the same audio repeatedly, and chunk boundaries split words → duplicated/garbled output.
3. **UI starvation.** `n_threads = hardware_concurrency()` = 4 Whisper threads on 2 physical cores. Whisper's matmuls are memory-bandwidth bound, so the hyperthreads add little throughput but do steal cycles from the UI/audio threads → janky window, laggy VU meter.
**Key insight:** for *dictation* (as opposed to live captioning) you never needed streaming. Record the whole utterance, transcribe once. Whisper then runs at its own pace with **no deadline**, processes each second of audio exactly once, and produces cleaner text. A 10s utterance at 3× real-time = ~3.3s of processing **after** you stop talking — predictable and fine. While you're *speaking*, CPU is near-idle because you're only copying samples into a buffer.
## 3. Target architecture
Three threads, a simple state machine, and one-shot inference.
```
┌─ UI thread (Win32 message loop) ──────────────┐
│ • owns the window, hotkeys, tray, buttons │
│ • owns Transcriber │
│ • receives result via PostMessage │
└───────────────────────────────────────────────┘
│ start_recording() ▲ WM_APP_RESULT (text)
▼ │
┌─ Audio thread (SDL callback) ─┐ │
│ • appends f32 samples to │ │
│ m_capture (mutex) │ │
│ • updates VU energy (atomic) │ │
└───────────────────────────────┘ │
│ stop_and_transcribe() │
▼ │
┌─ Worker thread (spawned on stop) ─────────────┐
│ • optional silence trim │
│ • whisper_full(...) ONCE │
│ • clean text → PostMessage to UI ────────────┘
└────────────────────────────────────────────────
```
**State machine:**
```
Idle ──(hotkey/Record)──▶ Recording ──(hotkey/Stop)──▶ Transcribing ──(result)──▶ Idle
▲ │
└────────────────────────────(cancel / Esc)─────────────────────────────────────┘
```
Guards: ignore Start while `Transcribing`; `stop_and_transcribe` swaps the capture buffer out under the mutex and hands it to the worker by value, so the audio thread can't race the reader. The whole ring-buffer / overlap machinery from the current `transcriber.cpp` is **deleted**.
## 4. transcriber.h (new)
Drop the ring buffer, `step_ms`/`length_ms`, buffer-fullness, etc. New surface:
```cpp
#pragma once
#include <string>
#include <vector>
#include <thread>
#include <mutex>
#include <atomic>
#include <functional>
struct whisper_context;
struct WhisperConfig {
std::string model_path = "models/ggml-tiny.en.bin";
std::string language = "en";
int n_threads = 0; // 0 = auto (physical cores)
bool use_gpu = false; // CPU on this machine
int capture_id = 0; // SDL capture device index
bool trim_silence = true; // cheap VAD on the captured clip
};
class Transcriber {
public:
using ResultCb = std::function<void(const std::string&)>;
Transcriber() = default;
~Transcriber();
bool preload(const WhisperConfig& cfg); // load model off the UI thread
bool is_loaded() const { return m_ctx != nullptr; }
bool start_recording(); // open mic, begin capture (cheap)
void stop_and_transcribe(); // stop mic, kick ONE transcription
void cancel(); // abort recording, no transcription
bool is_recording() const { return m_recording.load(); }
bool is_busy() const { return m_busy.load(); } // transcribing
float get_audio_energy() const { return m_energy.load(); }// 0..1 VU
void set_result_callback(ResultCb cb) { m_on_result = std::move(cb); }
void on_audio(const float* samples, int n); // called by SDL C shim
static std::vector<std::string> get_audio_devices();
private:
void transcribe_worker(std::vector<float> audio);
static int default_threads();
WhisperConfig m_cfg;
whisper_context* m_ctx = nullptr;
unsigned int m_dev = 0;
std::vector<float> m_capture; // grows while recording
std::mutex m_capture_mtx;
std::atomic<bool> m_recording{false};
std::atomic<bool> m_busy{false};
std::atomic<float> m_energy{0.0f};
std::thread m_worker;
ResultCb m_on_result;
};
```
Memory note: 16kHz × 4 bytes = 64 KB/s, so a 2-minute clip ≈ 7.6 MB. Reserve ~30s up front; optionally cap recording length (e.g. 5 min) to bound memory.
## 5. transcriber.cpp — capture & one-shot transcription
```cpp
#include "transcriber.h"
#include "whisper.h"
#include <SDL.h>
#include <SDL_audio.h>
#include <algorithm>
#include <cmath>
#include <cstring>
Transcriber::~Transcriber() {
cancel();
if (m_worker.joinable()) m_worker.join();
if (m_ctx) whisper_free(m_ctx);
}
int Transcriber::default_threads() {
unsigned hc = std::thread::hardware_concurrency(); // 4 on 2c/4t
if (hc <= 2) return (int)std::max(1u, hc);
return (int)(hc / 2); // 4 logical -> 2 physical
}
bool Transcriber::preload(const WhisperConfig& cfg) {
m_cfg = cfg;
if (m_cfg.n_threads <= 0) m_cfg.n_threads = default_threads();
if (m_ctx) return true;
whisper_context_params cp = whisper_context_default_params();
cp.use_gpu = m_cfg.use_gpu;
m_ctx = whisper_init_from_file_with_params(m_cfg.model_path.c_str(), cp);
return m_ctx != nullptr;
}
static void sdl_capture_cb(void* user, Uint8* stream, int len) {
auto* self = static_cast<Transcriber*>(user);
self->on_audio(reinterpret_cast<float*>(stream), len / (int)sizeof(float));
}
bool Transcriber::start_recording() {
if (m_recording.load() || m_busy.load()) return false;
{ std::lock_guard<std::mutex> lk(m_capture_mtx);
m_capture.clear(); m_capture.reserve(WHISPER_SAMPLE_RATE * 30); }
SDL_AudioSpec want{}, have{};
want.freq = WHISPER_SAMPLE_RATE; // 16000
want.format = AUDIO_F32SYS;
want.channels = 1;
want.samples = 1024;
want.callback = sdl_capture_cb;
want.userdata = this;
const char* dev = SDL_GetAudioDeviceName(m_cfg.capture_id, SDL_TRUE);
m_dev = SDL_OpenAudioDevice(dev, SDL_TRUE, &want, &have, 0);
if (!m_dev) return false;
m_energy = 0.0f;
m_recording = true;
SDL_PauseAudioDevice(m_dev, 0); // start capturing
return true;
}
void Transcriber::on_audio(const float* s, int n) {
if (n <= 0 || !m_recording.load()) return;
double sq = 0.0;
for (int i = 0; i < n; ++i) sq += (double)s[i] * s[i];
float rms = (float)std::sqrt(sq / n);
float e = m_energy.load();
m_energy = std::min(1.0f, e * 0.6f + (rms * 4.0f) * 0.4f); // smoothed
std::lock_guard<std::mutex> lk(m_capture_mtx);
m_capture.insert(m_capture.end(), s, s + n);
}
void Transcriber::cancel() {
if (!m_recording.load()) return;
m_recording = false;
if (m_dev) { SDL_PauseAudioDevice(m_dev, 1); SDL_CloseAudioDevice(m_dev); m_dev = 0; }
std::lock_guard<std::mutex> lk(m_capture_mtx);
m_capture.clear();
m_energy = 0.0f;
}
void Transcriber::stop_and_transcribe() {
if (!m_recording.load()) return;
m_recording = false;
if (m_dev) { SDL_PauseAudioDevice(m_dev, 1); SDL_CloseAudioDevice(m_dev); m_dev = 0; }
m_energy = 0.0f;
std::vector<float> audio;
{ std::lock_guard<std::mutex> lk(m_capture_mtx); audio.swap(m_capture); }
if (audio.size() < (size_t)(WHISPER_SAMPLE_RATE * 0.3)) { // <300ms
if (m_on_result) m_on_result("");
return;
}
if (m_worker.joinable()) m_worker.join();
m_busy = true;
m_worker = std::thread(&Transcriber::transcribe_worker, this, std::move(audio));
}
```
**Silence trim + text cleanup helpers** (file-local statics):
```cpp
static void trim_silence(std::vector<float>& a, float thresh = 0.01f) {
const size_t win = 1600; // 100ms
auto loud = [&](size_t i){
float m = 0.f;
for (size_t k=i; k<std::min(a.size(), i+win); ++k) m = std::max(m, std::fabs(a[k]));
return m > thresh;
};
size_t s = 0, e = a.size();
while (s + win < a.size() && !loud(s)) s += win;
while (e > win && !loud(e - win)) e -= win;
if (s + win <= e) a.assign(a.begin()+ (s>win? s-win:0), a.begin()+e); // keep 100ms pad
}
static std::string clean_text(std::string s) {
const char* junk[] = {"[BLANK_AUDIO]","[NOISE]","(blank)","(noise)","[ Silence ]"};
for (auto j : junk) { size_t p; while ((p=s.find(j))!=std::string::npos) s.erase(p, strlen(j)); }
size_t b = s.find_first_not_of(" \t\r\n");
size_t e = s.find_last_not_of(" \t\r\n");
return (b==std::string::npos) ? "" : s.substr(b, e-b+1);
}
```
**The one-shot worker:**
```cpp
void Transcriber::transcribe_worker(std::vector<float> audio) {
if (m_cfg.trim_silence) trim_silence(audio);
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
wp.print_progress = false;
wp.print_realtime = false;
wp.print_timestamps = false;
wp.no_timestamps = true;
wp.translate = false;
wp.language = m_cfg.language.c_str();
wp.n_threads = m_cfg.n_threads;
wp.no_context = true;
wp.suppress_blank = true;
wp.temperature = 0.0f;
// greedy + temperature 0 = fastest, deterministic. (Optionally set
// wp.suppress_nst = true on newer whisper.cpp to drop non-speech tokens.)
std::string out;
if (m_ctx && whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
int n = whisper_full_n_segments(m_ctx);
for (int i = 0; i < n; ++i) {
const char* t = whisper_full_get_segment_text(m_ctx, i);
if (t) out += t;
}
out = clean_text(out);
}
m_busy = false;
if (m_on_result) m_on_result(out); // runs on worker thread -> PostMessage in UI
}
```
`get_audio_devices()` stays as-is from your current file (it already enumerates SDL capture devices). Call `SDL_Init(SDL_INIT_AUDIO)` once at app startup (and `SDL_Quit()` at exit) rather than per-call.
## 6. Clipboard & auto-paste
This is the feature that makes it actually useful: text lands on the clipboard automatically, and (optionally) gets pasted straight into whatever app you were in before the popup.
**UTF-8 → UTF-16 + set clipboard:**
```cpp
static std::wstring to_w(const std::string& s) {
if (s.empty()) return L"";
int n = MultiByteToWideChar(CP_UTF8, 0, s.c_str(), -1, nullptr, 0);
std::wstring w(n ? n-1 : 0, L'\0');
if (n) MultiByteToWideChar(CP_UTF8, 0, s.c_str(), -1, &w[0], n);
return w;
}
bool SetClipboardTextUtf8(HWND owner, const std::string& utf8) {
std::wstring w = to_w(utf8);
if (!OpenClipboard(owner)) return false;
EmptyClipboard();
size_t bytes = (w.size() + 1) * sizeof(wchar_t);
HGLOBAL h = GlobalAlloc(GMEM_MOVEABLE, bytes);
if (h) {
void* p = GlobalLock(h);
memcpy(p, w.c_str(), bytes);
GlobalUnlock(h);
SetClipboardData(CF_UNICODETEXT, h); // clipboard now owns h; don't free
}
CloseClipboard();
return h != nullptr;
}
```
**Auto-paste into the previously focused window.** Capture the target HWND at the moment your hotkey fires (before you steal focus — see §7), then:
```cpp
static void send_ctrl_v() {
INPUT in[4] = {};
in[0].type = INPUT_KEYBOARD; in[0].ki.wVk = VK_CONTROL;
in[1].type = INPUT_KEYBOARD; in[1].ki.wVk = 'V';
in[2].type = INPUT_KEYBOARD; in[2].ki.wVk = 'V'; in[2].ki.dwFlags = KEYEVENTF_KEYUP;
in[3].type = INPUT_KEYBOARD; in[3].ki.wVk = VK_CONTROL; in[3].ki.dwFlags = KEYEVENTF_KEYUP;
SendInput(4, in, sizeof(INPUT));
}
void PasteIntoWindow(HWND target) {
if (!target || !IsWindow(target)) return;
DWORD me = GetCurrentThreadId();
DWORD other = GetWindowThreadProcessId(target, nullptr);
AttachThreadInput(me, other, TRUE); // bypass foreground-lock
SetForegroundWindow(target);
SetFocus(target);
AttachThreadInput(me, other, FALSE);
Sleep(40); // let focus settle
send_ctrl_v();
}
```
**Alternative — type the text directly** (no clipboard touched; works in apps with quirky paste handling). Surrogate pairs are handled because each UTF-16 code unit is sent as its own scan code:
```cpp
void TypeUnicode(const std::wstring& text) {
std::vector<INPUT> in; in.reserve(text.size()*2);
for (wchar_t c : text) {
INPUT d{}; d.type = INPUT_KEYBOARD; d.ki.wScan = c; d.ki.dwFlags = KEYEVENTF_UNICODE;
INPUT u = d; u.ki.dwFlags |= KEYEVENTF_KEYUP;
in.push_back(d); in.push_back(u);
}
if (!in.empty()) SendInput((UINT)in.size(), in.data(), sizeof(INPUT));
}
```
**Caveats to bake in:**
- **Elevation/UIPI:** a non-elevated app cannot `SendInput` into an elevated (Run-as-Admin) window. If you dictate into elevated apps, ship an elevation manifest — otherwise leave it un-elevated (recommended) and it'll just work for normal apps.
- Recommend **clipboard + Ctrl+V** as the default (fast, preserves formatting-free text); offer **TypeUnicode** as a fallback toggle for stubborn targets.
- Consider saving/restoring the user's previous clipboard contents if you want to be polite (optional).
## 7. main.cpp — window, hotkeys, tray, single-instance
Targeted changes to your existing `main.cpp`. New message IDs:
```cpp
#define WM_APP_RESULT (WM_APP + 1) // worker -> UI: transcription text
#define WM_APP_SHOW (WM_APP + 2) // 2nd instance -> existing window
#define HK_TOGGLE 1
#define HK_HIDE 2
static Transcriber g_tx;
static WhisperConfig g_config;
static HWND g_prevForeground = nullptr; // app to paste back into
static bool g_autoPaste = true;
```
**Single instance** (very top of `wWinMain`, before creating the window):
```cpp
HANDLE hMutex = CreateMutexW(nullptr, TRUE, L"WhisperDictation_SingleInstance");
if (GetLastError() == ERROR_ALREADY_EXISTS) {
HWND existing = FindWindowW(L"WhisperDictationClass", nullptr);
if (existing) PostMessage(existing, WM_APP_SHOW, 0, 0);
return 0; // a copy is already running (and owns the hotkeys)
}
```
**Compact, always-on-top window** (replace the big `CreateWindowEx`):
```cpp
hMainWnd = CreateWindowExW(
WS_EX_TOPMOST | WS_EX_TOOLWINDOW, // on top, no taskbar button
L"WhisperDictationClass", L"Dictation",
WS_POPUP | WS_CAPTION | WS_SYSMENU, // small, draggable by caption
CW_USEDEFAULT, CW_USEDEFAULT, 360, 180,
nullptr, nullptr, hInstance, nullptr);
void SetAlwaysOnTop(HWND h, bool on) {
SetWindowPos(h, on ? HWND_TOPMOST : HWND_NOTOPMOST, 0,0,0,0, SWP_NOMOVE|SWP_NOSIZE);
}
```
Suggested compact layout (3 rows): **[ ● Record / ■ Stop ] [ 📌 pin ]** · status line ("Ready • tiny.en • 2 threads" / "Recording 0:04" / "Transcribing…" / "Copied ✓") · a small read-only multiline EDIT showing the last result + a **Copy** and **Paste** button. Keep the VU meter — it's cheap and reassures you the mic is live.
**Preload the model in the background** (after the window exists, so first record is instant):
```cpp
std::thread([]{ g_tx.preload(g_config); }).detach();
g_tx.set_result_callback([](const std::string& t){
PostMessage(hMainWnd, WM_APP_RESULT, (WPARAM)new std::string(t), 0);
});
```
**Global hotkeys** (after window creation):
```cpp
RegisterHotKey(hMainWnd, HK_TOGGLE, MOD_CONTROL | MOD_SHIFT, VK_SPACE); // show + record/stop
RegisterHotKey(hMainWnd, HK_HIDE, MOD_CONTROL | MOD_SHIFT, 'H'); // hide to tray
```
**Message handling:**
```cpp
case WM_HOTKEY:
if (wParam == HK_TOGGLE) {
if (!g_tx.is_recording() && !g_tx.is_busy()) {
g_prevForeground = GetForegroundWindow(); // capture BEFORE we steal focus
ShowWindow(hWnd, SW_SHOWNA); // show without stealing focus
if (g_tx.start_recording()) SetStatus(hWnd, L"Recording…");
} else if (g_tx.is_recording()) {
g_tx.stop_and_transcribe();
SetStatus(hWnd, L"Transcribing…");
}
} else if (wParam == HK_HIDE) {
if (g_tx.is_recording()) g_tx.cancel();
ShowWindow(hWnd, SW_HIDE);
}
break;
case WM_APP_SHOW:
ShowWindow(hWnd, SW_SHOW); SetForegroundWindow(hWnd);
break;
case WM_APP_RESULT: {
std::string* res = (std::string*)wParam;
if (res && !res->empty()) {
SetDlgItemTextW(hWnd, ID_EDIT_TEXT, to_w(*res).c_str());
SetClipboardTextUtf8(hWnd, *res);
if (g_autoPaste && g_prevForeground) {
ShowWindow(hWnd, SW_HIDE); // get out of the way first
PasteIntoWindow(g_prevForeground);
}
SetStatus(hWnd, g_autoPaste ? L"Pasted ✓" : L"Copied ✓");
} else {
SetStatus(hWnd, L"No speech detected");
}
delete res;
} break;
```
**Recording timer / VU:** keep a lightweight `WM_TIMER` (e.g. 50ms) that, while `is_recording()`, updates the VU meter from `get_audio_energy()` and shows elapsed seconds. Drop the buffer-fullness bar (no longer meaningful).
**Tray:** keep your existing tray setup; `WM_CLOSE` hides to tray (as today). Add a "Start in tray" option: `ShowWindow(hMainWnd, startHidden ? SW_HIDE : nCmdShow);`. On `WM_DESTROY`, also `ReleaseMutex(hMutex)`.
**Optional — hold-to-talk** (instead of toggle): `RegisterHotKey` only fires on key-down, so true push-and-hold needs a low-level keyboard hook:
```cpp
// SetWindowsHookEx(WH_KEYBOARD_LL, LowLevelKbProc, hInst, 0);
// In the proc: on your chosen key WM_KEYDOWN -> start_recording (once),
// on WM_KEYUP -> stop_and_transcribe. Debounce auto-repeat with a flag.
```
Keep it behind a setting; the toggle hotkey matches your "hit record" description and is simpler/robust.
## 8. WhisperConfig tuning for the i5-7th-gen
**Threads.** Default to physical cores (2). Whisper's matmuls are memory-bandwidth bound, so 4 threads on 2 cores buys little throughput and steals from the UI/audio threads. Try 2 (default) vs 3 and keep whichever feels best — expose it in settings.
**Model choice (CPU, English).** All from `download-ggml-model`:
| Model | Size | Rel. speed on 2c | Accuracy | Use when |
|---|---|---|---|---|
| `tiny.en` | 75 MB | ★★★★★ fastest | ok | **default** — snappy dictation |
| `tiny.en-q8_0` | ~42 MB | ★★★★★ | ≈ tiny | low RAM / similar speed |
| `base.en-q5_1` | ~57 MB | ★★★☆ | better | want more accuracy, can wait ~2× |
| `base.en` | 142 MB | ★★★ | better | accuracy over latency |
| `small.en` | 466 MB | ★★ slow | best | only for short clips / patience |
Quantized (`q5_1`/`q8_0`) models are smaller and can be a touch faster on a bandwidth-limited CPU for a small accuracy cost — worth A/B testing `tiny.en` vs `tiny.en-q8_0` and `base.en-q5_1`. Add a model dropdown in the UI that re-runs `preload()` on a background thread.
**Whisper params** (already in §5): greedy sampling, `temperature = 0`, `no_context = true`, `no_timestamps = true`. These are the fastest, most deterministic settings. Avoid beam search.
**Build flags.** Your CMake already enables AVX2/FMA/F16C (the `WHISPER_NO_*` options default OFF) and MSVC `/O2 /GL` + `/LTCG`. That's correct for Kaby Lake — keep it. Confirm you're building **Release**, not Debug (Debug whisper is multiples slower). Optional extras, in rough order of effort/value:
- **OpenBLAS** (`-DGGML_BLAS=ON` with a BLAS vendor) — sometimes helps CPU matmul; measure, it's not always a win for tiny.
- **Vulkan on the iGPU** (`-DGGML_VULKAN=ON`) — your HD/UHD 620 *can* run it, but for tiny.en it's often no faster than CPU and adds driver/DLL complexity. Low priority; the batch redesign already solves the felt problem.
**Path robustness.** `model_path` is relative (`models/...`), so the app only works when the working directory is the exe folder. Resolve it from the exe location so a desktop shortcut always works:
```cpp
std::string exe_dir() {
char buf[MAX_PATH]; GetModuleFileNameA(nullptr, buf, MAX_PATH);
std::string p(buf); return p.substr(0, p.find_last_of("\\/"));
}
// g_config.model_path = exe_dir() + "\\models\\ggml-tiny.en.bin";
```
## 9. Build, paths & desktop shortcut
**Files touched:** `src/transcriber.h`, `src/transcriber.cpp`, `src/main.cpp`. The clipboard/paste helpers can live inside `main.cpp` (no new translation unit needed). If you split them into `src/clipboard.cpp`, add it to both `add_executable(...)` lists in `CMakeLists.txt` and `src/CMakeLists.txt`. No new third-party dependencies.
**Build (unchanged):**
```powershell
cmake -B build -DWHISPER_SDL2=ON
cmake --build build --config Release
# build\bin\Release\win-dictation.exe
```
Your `build.ps1` already downloads SDL2 + models and deploys DLLs; keep using it.
**Desktop shortcut** (double-click to launch). The `WorkingDirectory` must be the exe folder so `models/` resolves — unless you adopt the `exe_dir()` fix in §8, in which case it doesn't matter:
```powershell
$exe = "C:\code\whisper.cpp\examples\win-dictation\build\bin\Release\win-dictation.exe"
$ws = New-Object -ComObject WScript.Shell
$sc = $ws.CreateShortcut("$env:USERPROFILE\Desktop\Dictation.lnk")
$sc.TargetPath = $exe
$sc.WorkingDirectory = Split-Path $exe
$sc.IconLocation = "$exe,0"
$sc.Save()
```
**Start with Windows** (optional): drop that same `.lnk` into `shell:startup`, or add a `Run` registry value. Combined with start-to-tray, it's always one hotkey away.
**Icon:** you already have `win-dictation.rc` / `IDI_ICON1`, so the exe and tray icon are covered.
## 10. Testing & acceptance checklist
Since I can't run it, here's what to verify on the laptop:
**Performance (the point of all this):**
- [ ] While recording, Task Manager shows the app near-idle on CPU (you're only buffering).
- [ ] After Stop, a ~10s utterance transcribes in a few seconds and the window stays responsive throughout.
- [ ] Latency does **not** grow with longer recordings (the old unbounded-backlog bug is gone).
- [ ] First recording after launch is instant (model preloaded) — no "Loading model…" stall.
**Workflow:**
- [ ] `Ctrl+Shift+Space` shows the mini window and starts recording; pressing it again stops and produces text.
- [ ] Text is on the clipboard automatically; with auto-paste on, it lands in the app you were in before the hotkey.
- [ ] `Ctrl+Shift+H` hides to tray; double-clicking the tray icon restores.
- [ ] Launching a second copy focuses the existing one instead of starting a rival (single-instance + hotkey ownership).
- [ ] Pin toggle keeps it above other windows; window is draggable and compact.
**Robustness:**
- [ ] Recording <0.3s or pure silence → "No speech detected", no crash.
- [ ] Mic selection change takes effect on the next recording.
- [ ] Paste into Notepad, a browser field, and your editor all work; note any app where Ctrl+V fails (use TypeUnicode fallback there).
- [ ] Unicode / punctuation comes through intact (UTF-8↔UTF-16 path).
## 11. Suggested implementation order
Build it incrementally so you can feel the win early and isolate any breakage:
1. **Batch core first (biggest payoff).** Rewrite `transcriber.h/.cpp` per §4–§5. Temporarily wire your *existing* big window's Record button to `start_recording()` / `stop_and_transcribe()` and dump the result into the text box. At this point the performance problem should already be gone. Verify §10 "Performance".
2. **Clipboard + auto-paste** (§6). Add `SetClipboardTextUtf8` and capture `g_prevForeground` on the button press; confirm copy works, then add `PasteIntoWindow`.
3. **Global hotkeys + focus capture** (§7). Switch to driving everything from `Ctrl+Shift+Space`; make sure `g_prevForeground` is grabbed *before* showing the window.
4. **Compact always-on-top UI + pin** (§7). Shrink the window, add `WS_EX_TOPMOST`, trim the layout, drop the buffer bar.
5. **Single-instance + start-to-tray + background preload** (§7).
6. **Tuning pass** (§8): set threads = 2, try `tiny.en` vs `tiny.en-q8_0` vs `base.en-q5_1`, add the model dropdown, apply the `exe_dir()` path fix.
7. **Desktop shortcut** (§9) and optional hold-to-talk.
Each step compiles and runs on its own. If you want, I can generate the **complete** rewritten `main.cpp`, `transcriber.cpp`, and `transcriber.h` (not just snippets) for step 1 so you have a drop-in starting point.