v2 rebuild: GDI+ single-surface UI, self-calibrating progress, compact push-to-talk, GGML+Whisper integration
This commit is contained in:
Win Dictation Dev
+26
-295
@@ -1,308 +1,39 @@
|
||||
# Whisper Dictation v2.0 - Changelog
|
||||
# Win Dictation — Changelog
|
||||
|
||||
## Overview
|
||||
Complete rewrite of win-dictation with focus on performance, reliability, and user experience.
|
||||
## v3.0 — Architecture & UI Rebuild (current)
|
||||
|
||||
## Major Changes
|
||||
### Single-surface UI
|
||||
|
||||
### 1. Audio Capture System (Zero Loss)
|
||||
- Replaced 9 separate child windows with one immediate-mode painted surface
|
||||
- Eliminated inter-window hairline seams at the root
|
||||
- Widget data model with hover/press animation blending
|
||||
- Per-monitor DPI scaling (V2)
|
||||
- Dark mode design tokens
|
||||
- Dark scrollbar on the transcript edit control
|
||||
|
||||
**Problem:** Audio chunks were being dropped between recording and processing.
|
||||
### Progress system
|
||||
|
||||
**Solution:** Implemented lock-free ring buffer system
|
||||
- 30-second circular buffer (480K samples)
|
||||
- Atomic read/write positions
|
||||
- No mutex in audio callback
|
||||
- Handles burst processing gracefully
|
||||
- Self-calibrating linear estimator: `proc = a + b·audio`, fitted per model with decayed online least-squares
|
||||
- Persisted per-model timing history in `win-dictation.ini`
|
||||
- Strictly monotonic countdown (never counts up)
|
||||
- Smooth percent that eases, never snaps (except final 100%)
|
||||
- Motion from frame 1 (predicts before whisper reports)
|
||||
|
||||
**Files Changed:**
|
||||
- `transcriber.h`: Added ring buffer members
|
||||
- `transcriber.cpp`: Rewrote audio_callback() and worker_loop()
|
||||
### Push-to-talk batch mode
|
||||
|
||||
### 2. Multi-Core CPU Support
|
||||
- Removed streaming/VAD/ring-buffer architecture
|
||||
- Record `→` stop `→` single `whisper_full` call
|
||||
- 500ms silence auto-end timer
|
||||
- Physical-core threading (matches the i5 target)
|
||||
|
||||
**Problem:** Only using 1-2 CPU cores despite having 24 available.
|
||||
### Files
|
||||
|
||||
**Solution:** Proper thread configuration
|
||||
- Uses `std::thread::hardware_concurrency()` (24 threads)
|
||||
- OpenMP support enabled in build
|
||||
- Optimized work distribution
|
||||
|
||||
**Configuration:**
|
||||
```cpp
|
||||
WhisperConfig::n_threads = std::thread::hardware_concurrency(); // 24
|
||||
```
|
||||
|
||||
### 3. GPU Acceleration
|
||||
|
||||
**Problem:** No GPU utilization despite CUDA installation.
|
||||
|
||||
**Solution:** GPU detection and backend selection
|
||||
- Auto-detects CUDA/Vulkan/Metal
|
||||
- Graceful fallback to CPU
|
||||
- Version compatibility checking
|
||||
- Status display in UI
|
||||
|
||||
**Build Script:**
|
||||
- Detects GPU capabilities
|
||||
- Checks CUDA version compatibility (11.7 vs 12.4 requirement)
|
||||
- Falls back to optimized CPU build
|
||||
|
||||
### 4. Modern User Interface
|
||||
|
||||
**Problem:** Slow, glitchy interface with poor visual feedback.
|
||||
|
||||
**Solution:** Complete UI overhaul
|
||||
- Dark theme with modern colors
|
||||
- 30 FPS update timer (was 50ms/20 FPS)
|
||||
- Custom button drawing
|
||||
- Real-time status indicators
|
||||
- Smooth animations
|
||||
|
||||
**UI Features:**
|
||||
- VU meter (real-time audio level)
|
||||
- Buffer indicator (queue status)
|
||||
- GPU/CPU status
|
||||
- Thread count display
|
||||
- Responsive layout
|
||||
|
||||
**Colors:**
|
||||
```cpp
|
||||
#define COLOR_BG RGB(32, 33, 36)
|
||||
#define COLOR_PRIMARY RGB(138, 180, 248)
|
||||
#define COLOR_SUCCESS RGB(129, 201, 149)
|
||||
```
|
||||
|
||||
### 5. Processing Optimizations
|
||||
|
||||
**Changes:**
|
||||
- Reduced step_ms: 3000ms → 1500ms (faster response)
|
||||
- Reduced length_ms: 10000ms → 8000ms (better streaming)
|
||||
- Added VAD filtering (skip silence)
|
||||
- Improved context handling
|
||||
- Better memory management
|
||||
|
||||
### 6. Build System
|
||||
|
||||
**New Features:**
|
||||
- Automated GPU detection
|
||||
- Version compatibility checking
|
||||
- One-command build and deploy
|
||||
- Automatic model download
|
||||
- DLL deployment
|
||||
|
||||
**Script:** `build.ps1`
|
||||
```powershell
|
||||
# Detects:
|
||||
- NVIDIA GPU + CUDA version
|
||||
- AMD GPU + ROCm
|
||||
- Vulkan SDK
|
||||
- CPU capabilities (AVX2/FMA)
|
||||
```
|
||||
|
||||
## File-by-File Changes
|
||||
|
||||
### transcriber.h
|
||||
```diff
|
||||
+ Ring buffer implementation (RING_BUFFER_SIZE = 480K)
|
||||
+ Atomic position tracking
|
||||
+ get_buffer_fullness() method
|
||||
+ is_using_gpu() const correctness
|
||||
+ Processing buffer for context
|
||||
+ GPU active flag
|
||||
- Simple deque queue
|
||||
- Blocking mutex in callback
|
||||
```
|
||||
|
||||
### transcriber.cpp
|
||||
```diff
|
||||
+ Lock-free ring buffer audio_callback()
|
||||
+ Atomic read/write operations
|
||||
+ VAD integration (skip silence)
|
||||
+ GPU detection in init()
|
||||
+ Improved error handling
|
||||
+ Context-aware processing
|
||||
+ Smaller SDL buffer (512 vs 1024)
|
||||
- Blocking queue operations
|
||||
- No VAD filtering
|
||||
- Poor error messages
|
||||
```
|
||||
|
||||
### main.cpp
|
||||
```diff
|
||||
+ Modern dark theme
|
||||
+ Custom button drawing (owner-draw)
|
||||
+ 30 FPS timer (was 20 FPS)
|
||||
+ Status text with GPU/CPU/threads
|
||||
+ DWM dark mode titlebar
|
||||
+ Improved layout handling
|
||||
+ Better font selection
|
||||
- Basic Windows theme
|
||||
- Standard buttons
|
||||
- Slow updates
|
||||
- Minimal status info
|
||||
```
|
||||
|
||||
### CMakeLists.txt
|
||||
```diff
|
||||
+ dwmapi library link
|
||||
+ Optimization flags (/O2 /GL /LTCG)
|
||||
+ Better include paths
|
||||
+ Separate Debug/Release outputs
|
||||
- Basic configuration
|
||||
```
|
||||
|
||||
### build.ps1
|
||||
```diff
|
||||
+ Complete rewrite
|
||||
+ GPU detection logic
|
||||
+ CUDA version checking
|
||||
+ Automatic DLL deployment
|
||||
+ Model download automation
|
||||
+ Colored output
|
||||
+ Error handling
|
||||
- CPU-only hardcoded
|
||||
- Manual deployment
|
||||
- No GPU support
|
||||
```
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### Before
|
||||
- **Audio Loss**: Frequent dropped chunks
|
||||
- **CPU Usage**: 1-2 cores (~8%)
|
||||
- **GPU Usage**: 0%
|
||||
- **Latency**: 3-5 seconds
|
||||
- **UI FPS**: ~10-15 (choppy)
|
||||
- **Buffer Issues**: Frequent overruns
|
||||
|
||||
### After (CPU-Only)
|
||||
- **Audio Loss**: Zero (ring buffer)
|
||||
- **CPU Usage**: 24 cores (~60-80% during speech)
|
||||
- **GPU Usage**: 0% (incompatible CUDA version)
|
||||
- **Latency**: 2-3 seconds
|
||||
- **UI FPS**: 30 (smooth)
|
||||
- **Buffer Management**: Handles 30s bursts
|
||||
|
||||
### Potential (With GPU)
|
||||
- **Audio Loss**: Zero
|
||||
- **CPU Usage**: <10%
|
||||
- **GPU Usage**: 20-30%
|
||||
- **Latency**: <1 second
|
||||
- **Throughput**: >20x real-time
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
1. ✅ Fixed audio chunk loss (ring buffer)
|
||||
2. ✅ Fixed CPU underutilization (thread count)
|
||||
3. ✅ Fixed UI glitches (proper timing)
|
||||
4. ✅ Fixed missing GPU support (detection)
|
||||
5. ✅ Fixed model loading errors (better paths)
|
||||
6. ✅ Fixed memory leaks (proper cleanup)
|
||||
7. ✅ Fixed race conditions (atomics)
|
||||
8. ✅ Fixed build issues (explicit GPU disable)
|
||||
|
||||
## Code Quality Improvements
|
||||
|
||||
- **Better error handling**: Graceful fallbacks
|
||||
- **More comments**: Explain complex logic
|
||||
- **Type safety**: size_t for sizes, proper casts
|
||||
- **Memory safety**: RAII, smart pointers ready
|
||||
- **Threading**: Atomic operations, no races
|
||||
- **Modularity**: Clear separation of concerns
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
### WhisperConfig
|
||||
```cpp
|
||||
struct WhisperConfig {
|
||||
std::string model_path;
|
||||
std::string language = "en";
|
||||
int n_threads = std::thread::hardware_concurrency(); // NEW: 24
|
||||
int step_ms = 1500; // NEW: was 3000
|
||||
int length_ms = 8000; // NEW: was 10000
|
||||
bool use_gpu = true; // NEW: auto-detect
|
||||
int capture_id = 0;
|
||||
int n_gpu_layers = -1; // NEW: auto
|
||||
};
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
### Build
|
||||
- ✅ Clean compile on MSVC 2022
|
||||
- ✅ No linter errors
|
||||
- ✅ All warnings addressed
|
||||
- ✅ Proper DLL deployment
|
||||
|
||||
### Runtime (Expected)
|
||||
- ✅ Window opens correctly
|
||||
- ✅ Modern UI renders
|
||||
- ✅ Audio devices detected
|
||||
- ✅ Recording works
|
||||
- ✅ Transcription functions
|
||||
- ✅ No crashes
|
||||
- ✅ System tray works
|
||||
- ✅ Hotkey functions
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **CUDA 11.7 Incompatible**: User has CUDA 11.7 but MSVC 2022 requires CUDA 12.4+
|
||||
- **Workaround**: Using optimized CPU-only build
|
||||
- **Solution**: Upgrade to CUDA 12.4+ for GPU support
|
||||
|
||||
2. **Single Language**: Currently English-only (base.en model)
|
||||
- **Workaround**: Use multilingual model (ggml-base.bin)
|
||||
|
||||
3. **Model in Binary**: Model path hardcoded in source
|
||||
- **Future**: UI-based model selection
|
||||
|
||||
## Upgrade Path
|
||||
|
||||
### To Enable GPU (CUDA)
|
||||
1. Download and install CUDA Toolkit 12.4+
|
||||
2. Clean build directory
|
||||
3. Run build script (will auto-detect new CUDA)
|
||||
4. Rebuild application
|
||||
|
||||
### To Enable GPU (Vulkan - Alternative)
|
||||
1. Download and install Vulkan SDK
|
||||
2. Set VULKAN_SDK environment variable
|
||||
3. Clean and rebuild
|
||||
|
||||
### To Use Different Model
|
||||
1. Download model from HuggingFace
|
||||
2. Place in `build/bin/Release/models/`
|
||||
3. Update `g_config.model_path` in main.cpp
|
||||
4. Rebuild
|
||||
|
||||
## Documentation Added
|
||||
|
||||
1. **README.md**: Complete user guide
|
||||
2. **CHANGES.md**: This detailed changelog
|
||||
3. **Code Comments**: Inline documentation
|
||||
4. **Build Output**: Informative messages
|
||||
|
||||
## Migration Notes
|
||||
|
||||
This is a **breaking change** from v1.0:
|
||||
- API compatible but implementation completely different
|
||||
- Rebuild required (not drop-in replacement)
|
||||
- Configuration values changed
|
||||
- UI completely redesigned
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- whisper.cpp team for the excellent base library
|
||||
- SDL2 for cross-platform audio
|
||||
- User feedback on performance issues
|
||||
- Added `src/timing.h` — timing model + progress estimator + persistence
|
||||
- Rewrote `src/main.cpp` — single-surface paint, widget model, animation clock, DPI
|
||||
- Rewrote `README.md`, `src/README.md`, `src/CHANGES.md`
|
||||
|
||||
---
|
||||
|
||||
**Version**: 2.0
|
||||
**Date**: November 26, 2025
|
||||
**Status**: Production Ready (CPU-only), GPU Ready (pending CUDA upgrade)
|
||||
|
||||
|
||||
|
||||
## v2.0 — Previous architecture (deprecated)
|
||||
|
||||
Used streaming with ring buffer, VAD, owner-draw child windows. Described in earlier versions of the docs.
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
project(win-dictation)
|
||||
|
||||
if (WIN32)
|
||||
# Main application
|
||||
add_executable(win-dictation WIN32
|
||||
main.cpp
|
||||
transcriber.cpp
|
||||
transcriber.h
|
||||
win-dictation.rc
|
||||
)
|
||||
|
||||
# Link dependencies
|
||||
target_link_libraries(win-dictation PRIVATE
|
||||
whisper
|
||||
common
|
||||
common-sdl
|
||||
${SDL2_LIBRARY}
|
||||
comctl32
|
||||
dwmapi
|
||||
)
|
||||
|
||||
# Include directories
|
||||
target_include_directories(win-dictation PRIVATE
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../..
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../include
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../
|
||||
${SDL2_INCLUDE_DIR}
|
||||
)
|
||||
|
||||
# Use Unicode and enable optimizations
|
||||
target_compile_definitions(win-dictation PRIVATE UNICODE _UNICODE)
|
||||
|
||||
# Enable /O2 optimization in Release
|
||||
if(MSVC)
|
||||
target_compile_options(win-dictation PRIVATE
|
||||
$<$<CONFIG:Release>:/O2 /GL>
|
||||
)
|
||||
target_link_options(win-dictation PRIVATE
|
||||
$<$<CONFIG:Release>:/LTCG>
|
||||
)
|
||||
endif()
|
||||
|
||||
# Set properties
|
||||
set_target_properties(win-dictation PROPERTIES
|
||||
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
|
||||
RUNTIME_OUTPUT_DIRECTORY_RELEASE "${CMAKE_BINARY_DIR}/bin/Release"
|
||||
RUNTIME_OUTPUT_DIRECTORY_DEBUG "${CMAKE_BINARY_DIR}/bin/Debug"
|
||||
)
|
||||
|
||||
# Test executable
|
||||
add_executable(test-audio
|
||||
test-audio.cpp
|
||||
transcriber.cpp
|
||||
transcriber.h
|
||||
)
|
||||
|
||||
target_link_libraries(test-audio PRIVATE
|
||||
whisper
|
||||
common
|
||||
common-sdl
|
||||
${SDL2_LIBRARY}
|
||||
)
|
||||
|
||||
target_include_directories(test-audio PRIVATE
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../..
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../include
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../
|
||||
${SDL2_INCLUDE_DIR}
|
||||
)
|
||||
|
||||
set_target_properties(test-audio PROPERTIES
|
||||
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
|
||||
RUNTIME_OUTPUT_DIRECTORY_RELEASE "${CMAKE_BINARY_DIR}/bin/Release"
|
||||
RUNTIME_OUTPUT_DIRECTORY_DEBUG "${CMAKE_BINARY_DIR}/bin/Debug"
|
||||
)
|
||||
endif()
|
||||
|
||||
+32
-228
@@ -1,244 +1,48 @@
|
||||
# Whisper Dictation - AI Voice to Text for Windows
|
||||
# Win Dictation - Source Notes
|
||||
|
||||
A high-performance, real-time speech-to-text application for Windows using OpenAI's Whisper model.
|
||||
## Architecture (current)
|
||||
|
||||
## ✨ Features
|
||||
**Push-to-talk, batch mode.** Press record, speak, press again. The full audio clip is passed to `whisper_full` once on stop. No streaming, no VAD, no ring buffer.
|
||||
|
||||
### Performance
|
||||
- **Multi-Core CPU Support**: Automatically uses all available CPU cores (24 threads detected)
|
||||
- **GPU Acceleration**: Auto-detects and uses CUDA, Vulkan, or Metal when available
|
||||
- **Ring Buffer Audio**: Zero audio loss with lock-free ring buffer implementation
|
||||
- **Optimized Processing**: AVX2/FMA instructions for maximum performance
|
||||
- Audio: SDL2 capture at 16kHz mono
|
||||
- Inference: `whisper_full` with physical-core threads
|
||||
- UI: Single-surface GDI+ immediate-mode painting (no child-window chrome)
|
||||
- Progress: Self-calibrating linear estimator fused with whisper's chunk-boundary callbacks
|
||||
|
||||
### User Interface
|
||||
- **Modern Dark Theme**: Polished, professional interface
|
||||
- **Real-Time Monitoring**:
|
||||
- Live VU meter for audio levels
|
||||
- Buffer status indicator
|
||||
- GPU/CPU usage display
|
||||
- **Smooth Animations**: 30 FPS UI updates for responsive experience
|
||||
- **System Tray Integration**: Minimize to tray with hotkey support
|
||||
## Key files
|
||||
|
||||
### Audio Processing
|
||||
- **Voice Activity Detection (VAD)**: Automatically filters silence
|
||||
- **Continuous Recording**: Maintains context between segments
|
||||
- **Multiple Microphone Support**: Select from all available input devices
|
||||
- **16kHz Sample Rate**: Optimized for Whisper model
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `main.cpp` | Window, painting, interaction, settings, clipboard |
|
||||
| `transcriber.h` / `transcriber.cpp` | Audio capture, whisper preload/inference, callbacks |
|
||||
| `timing.h` | Per-model online least-squares timing model + live progress estimator |
|
||||
| `settings.h` | INI file read/write for persistent settings |
|
||||
| `text_util.h` | Transcript concatenation |
|
||||
| `logging.h` | Timestamped log to `win-dictation.log` |
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Build
|
||||
## Building
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File examples/win-dictation/build.ps1
|
||||
cmake -S . -B build -G "Visual Studio 18 2026" -DSDL2_DIR="deps/SDL2-2.28.5/cmake"
|
||||
cmake --build build --config Release
|
||||
```
|
||||
|
||||
The build script will:
|
||||
1. Detect your GPU capabilities (CUDA, Vulkan)
|
||||
2. Download and configure SDL2
|
||||
3. Build the application with optimal settings
|
||||
4. Download the Whisper model (base.en - 140MB)
|
||||
5. Deploy all required DLLs
|
||||
Output: `build\bin\Release\win-dictation.exe`
|
||||
|
||||
### Run
|
||||
Target machine: 2-core / 4-thread Intel i5-7th-gen, GPU CUDA disabled.
|
||||
|
||||
```
|
||||
build/bin/Release/win-dictation.exe
|
||||
```
|
||||
## Model placement
|
||||
|
||||
Or double-click the exe in the build output directory.
|
||||
Drop `.bin` files in `models/` next to the executable. The app scans for:
|
||||
- `ggml-tiny.en.bin`
|
||||
- `ggml-tiny.en-q8_0.bin`
|
||||
- `ggml-base.en-q5_1.bin`
|
||||
- `ggml-base.en.bin`
|
||||
|
||||
## 🎯 Usage
|
||||
First available is used. Toggle in the model popup.
|
||||
|
||||
### Controls
|
||||
- **Start/Stop Recording**: Click button or press `Ctrl+Shift+R`
|
||||
- **Clear Text**: Click "Clear" button
|
||||
- **Change Microphone**: Select from dropdown (auto-restarts recording)
|
||||
- **Minimize**: Close window (minimizes to system tray)
|
||||
- **Exit**: Right-click tray icon → Exit
|
||||
## Settings
|
||||
|
||||
### Indicators
|
||||
- **Level**: Real-time audio input level
|
||||
- **Buffer**: Current audio buffer usage (0-100%)
|
||||
- **Status**: Shows GPU/CPU mode, recording state, thread count
|
||||
|
||||
## ⚙️ Technical Details
|
||||
|
||||
### Architecture
|
||||
|
||||
#### Ring Buffer Audio Capture
|
||||
- **Lock-Free Design**: Audio thread never blocks
|
||||
- **30-Second Buffer**: Handles burst processing without loss
|
||||
- **Atomic Operations**: Prevents race conditions
|
||||
|
||||
#### Processing Pipeline
|
||||
```
|
||||
Audio Input → Ring Buffer → VAD → Whisper Inference → Text Output
|
||||
```
|
||||
|
||||
1. **SDL Audio Capture**: 512-sample chunks at 16kHz
|
||||
2. **Ring Buffer**: Lock-free circular buffer
|
||||
3. **VAD Processing**: Filters silence before inference
|
||||
4. **Whisper Inference**: Multi-threaded with context overlap
|
||||
5. **Text Output**: Appended to UI in real-time
|
||||
|
||||
### Performance Optimizations
|
||||
|
||||
#### CPU Mode (Current Build)
|
||||
- All 24 CPU threads utilized
|
||||
- AVX2/FMA SIMD instructions
|
||||
- Optimized memory layout
|
||||
- Minimal context switching
|
||||
|
||||
#### GPU Mode (When Available)
|
||||
- CUDA 12.4+ or Vulkan SDK required
|
||||
- Automatic offloading to GPU
|
||||
- Faster inference times
|
||||
- Lower CPU usage
|
||||
|
||||
### Model
|
||||
|
||||
Currently using `ggml-base.en.bin`:
|
||||
- **Size**: 140 MB
|
||||
- **Parameters**: 74 million
|
||||
- **Languages**: English only (optimized)
|
||||
- **Speed**: ~5x real-time on CPU, >20x on GPU
|
||||
- **Accuracy**: Excellent for general speech
|
||||
|
||||
To use a different model, place it in `build/bin/Release/models/` and update the config in `main.cpp`.
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### GPU Not Detected
|
||||
- **CUDA**: Install CUDA Toolkit 12.4 or newer (CUDA 13.0 recommended)
|
||||
- **See [CUDA-SETUP.md](CUDA-SETUP.md) for detailed installation guide**
|
||||
- **Vulkan**: Install Vulkan SDK
|
||||
- CPU-only mode still provides excellent performance with all cores
|
||||
|
||||
### After Installing CUDA 13.0
|
||||
See **[CUDA-SETUP.md](CUDA-SETUP.md)** for complete setup instructions including:
|
||||
- Verification steps
|
||||
- Clean rebuild process
|
||||
- Performance benchmarking
|
||||
- Troubleshooting GPU issues
|
||||
|
||||
### Audio Not Working
|
||||
- Check microphone permissions in Windows Settings
|
||||
- Verify correct device selected in dropdown
|
||||
- Test microphone in Windows Sound settings
|
||||
|
||||
### Poor Transcription Quality
|
||||
- Ensure microphone is close (6-12 inches)
|
||||
- Reduce background noise
|
||||
- Check VU meter shows green when speaking
|
||||
- Try a larger model (medium.en or large-v3-turbo)
|
||||
|
||||
### High CPU Usage
|
||||
- Normal during active transcription
|
||||
- Reduces during silence (VAD filtering)
|
||||
- Consider enabling GPU acceleration
|
||||
|
||||
## 📊 Performance Benchmarks
|
||||
|
||||
### CPU-Only (24 threads, base.en model)
|
||||
- **Latency**: ~2-3 seconds
|
||||
- **Throughput**: ~5x real-time
|
||||
- **CPU Usage**: 60-80% during speech
|
||||
- **Memory**: ~500 MB
|
||||
|
||||
### GPU-Accelerated (RTX 3090, base.en model)
|
||||
- **Latency**: <1 second
|
||||
- **Throughput**: >20x real-time
|
||||
- **GPU Usage**: 20-30%
|
||||
- **CPU Usage**: <10%
|
||||
- **Memory**: ~1 GB (VRAM)
|
||||
|
||||
## 🆕 Recent Improvements
|
||||
|
||||
### v2.0 (Current)
|
||||
- ✅ **Ring buffer** implementation - no more dropped audio
|
||||
- ✅ **Multi-core CPU** support - uses all available threads
|
||||
- ✅ **GPU auto-detection** - CUDA/Vulkan support
|
||||
- ✅ **Modern UI** - dark theme, smooth animations
|
||||
- ✅ **VAD integration** - skip silence for efficiency
|
||||
- ✅ **Better error handling** - graceful fallbacks
|
||||
- ✅ **Status indicators** - real-time monitoring
|
||||
- ✅ **Build script** - automated setup and deployment
|
||||
|
||||
### Previous Issues (Fixed)
|
||||
- ❌ Audio chunks lost between recording and processing
|
||||
- ❌ No GPU utilization
|
||||
- ❌ Only used 1-2 CPU cores
|
||||
- ❌ Slow, glitchy interface
|
||||
- ❌ No real-time feedback
|
||||
- ❌ Poor error messages
|
||||
|
||||
## 🎨 UI Features
|
||||
|
||||
### Modern Dark Theme
|
||||
- Background: `#202124`
|
||||
- Surface: `#292A2D`
|
||||
- Primary: `#8AB4F8` (Blue)
|
||||
- Success: `#81C995` (Green)
|
||||
- Text: `#E8EAED`
|
||||
|
||||
### Responsive Layout
|
||||
- Auto-resizes with window
|
||||
- Maintains proper spacing
|
||||
- Smooth transitions
|
||||
|
||||
### Visual Feedback
|
||||
- VU meter with color coding
|
||||
- Buffer status bar
|
||||
- GPU/CPU indicator
|
||||
- Thread count display
|
||||
|
||||
## 🔮 Future Enhancements
|
||||
|
||||
- [ ] Push-to-talk mode
|
||||
- [ ] Multiple language support
|
||||
- [ ] Punctuation model integration
|
||||
- [ ] Export to file (TXT, SRT)
|
||||
- [ ] Custom hotkey configuration
|
||||
- [ ] Noise reduction filter
|
||||
- [ ] Model switching in UI
|
||||
- [ ] Real-time word highlighting
|
||||
|
||||
## 📝 License
|
||||
|
||||
This example is part of the whisper.cpp project and follows the same license (MIT).
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Improvements welcome! The code is designed to be:
|
||||
- **Readable**: Clear structure and comments
|
||||
- **Maintainable**: Modular design
|
||||
- **Extensible**: Easy to add features
|
||||
- **Performant**: Optimized critical paths
|
||||
|
||||
## 💡 Tips
|
||||
|
||||
### For Best Results
|
||||
1. Use a quality microphone
|
||||
2. Position mic 6-12 inches from mouth
|
||||
3. Speak clearly and naturally
|
||||
4. Minimize background noise
|
||||
5. Keep buffer below 50% (adjust step_ms if needed)
|
||||
|
||||
### For Development
|
||||
- See `transcriber.h/cpp` for core logic
|
||||
- See `main.cpp` for UI implementation
|
||||
- Adjust parameters in `WhisperConfig` struct
|
||||
- Enable logging in `whisper_full_params`
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [Whisper.cpp](https://github.com/ggerganov/whisper.cpp)
|
||||
- [Whisper Paper](https://arxiv.org/abs/2212.04356)
|
||||
- [Model Download](https://huggingface.co/ggerganov/whisper.cpp)
|
||||
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads)
|
||||
- [Vulkan SDK](https://vulkan.lunarg.com/)
|
||||
|
||||
---
|
||||
|
||||
**Built with ❤️ using whisper.cpp**
|
||||
Stored in `win-dictation.ini` next to the executable. Sections:
|
||||
- `[app]`: window position, hotkey, model, capture device, pinned/autopaste/autohide flags
|
||||
- `[timing-ggml-*.bin]`: per-model timing accumulators (learned transcription speed)
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
#pragma once
|
||||
#include <windows.h>
|
||||
#include <cstdio>
|
||||
#include <string>
|
||||
|
||||
inline void LogLine(const char* msg) {
|
||||
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
|
||||
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/")) + L"\\win-dictation.log";
|
||||
FILE* f = nullptr;
|
||||
_wfopen_s(&f, p.c_str(), L"a");
|
||||
if (!f) return;
|
||||
SYSTEMTIME t; GetLocalTime(&t);
|
||||
fprintf(f, "%04d-%02d-%02d %02d:%02d:%02d %s\n",
|
||||
t.wYear, t.wMonth, t.wDay, t.wHour, t.wMinute, t.wSecond, msg);
|
||||
fclose(f);
|
||||
}
|
||||
+1336
-407
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,42 @@
|
||||
#pragma once
|
||||
#include <windows.h>
|
||||
#include <string>
|
||||
|
||||
struct AppSettings {
|
||||
int captureId = 0;
|
||||
std::wstring modelFile;
|
||||
bool pinned = true, autoPaste = true, autoHide = false;
|
||||
int winX = CW_USEDEFAULT, winY = CW_USEDEFAULT, winW = 400, winH = 340;
|
||||
int hkMods = MOD_CONTROL | MOD_SHIFT;
|
||||
int hkVk = VK_SPACE;
|
||||
};
|
||||
|
||||
inline std::wstring SettingsPath() {
|
||||
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
|
||||
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
|
||||
return p + L"\\win-dictation.ini";
|
||||
}
|
||||
|
||||
inline int GetIni(const wchar_t* k, int d) { return GetPrivateProfileIntW(L"app", k, d, SettingsPath().c_str()); }
|
||||
inline void PutIni(const wchar_t* k, int v) { wchar_t b[32]; wsprintfW(b, L"%d", v); WritePrivateProfileStringW(L"app", k, b, SettingsPath().c_str()); }
|
||||
|
||||
inline void LoadSettings(AppSettings& s) {
|
||||
s.captureId = GetIni(L"captureId", s.captureId);
|
||||
s.pinned = GetIni(L"pinned", s.pinned) != 0;
|
||||
s.autoPaste = GetIni(L"autoPaste", s.autoPaste) != 0;
|
||||
s.autoHide = GetIni(L"autoHide", s.autoHide) != 0;
|
||||
s.winX = GetIni(L"winX", s.winX); s.winY = GetIni(L"winY", s.winY);
|
||||
s.winW = GetIni(L"winW", s.winW); s.winH = GetIni(L"winH", s.winH);
|
||||
s.hkMods = GetIni(L"hkMods", s.hkMods);
|
||||
s.hkVk = GetIni(L"hkVk", s.hkVk);
|
||||
wchar_t m[MAX_PATH]; GetPrivateProfileStringW(L"app", L"modelFile", L"", m, MAX_PATH, SettingsPath().c_str());
|
||||
s.modelFile = m;
|
||||
}
|
||||
|
||||
inline void SaveSettings(const AppSettings& s) {
|
||||
PutIni(L"captureId", s.captureId); PutIni(L"pinned", s.pinned);
|
||||
PutIni(L"autoPaste", s.autoPaste); PutIni(L"autoHide", s.autoHide);
|
||||
PutIni(L"winX", s.winX); PutIni(L"winY", s.winY); PutIni(L"winW", s.winW); PutIni(L"winH", s.winH);
|
||||
PutIni(L"hkMods", s.hkMods); PutIni(L"hkVk", s.hkVk);
|
||||
WritePrivateProfileStringW(L"app", L"modelFile", s.modelFile.c_str(), SettingsPath().c_str());
|
||||
}
|
||||
@@ -1,254 +0,0 @@
|
||||
// Test program for win-dictation with audio files
|
||||
#include "whisper.h"
|
||||
#include "transcriber.h"
|
||||
#include "common.h"
|
||||
#include <iostream>
|
||||
#include <fstream>
|
||||
#include <vector>
|
||||
#include <string>
|
||||
#include <chrono>
|
||||
|
||||
// Simple file exists check without filesystem
|
||||
bool file_exists(const std::string& name) {
|
||||
std::ifstream f(name.c_str());
|
||||
return f.good();
|
||||
}
|
||||
|
||||
// WAV file header structure
|
||||
struct WAVHeader {
|
||||
char riff[4]; // "RIFF"
|
||||
uint32_t fileSize;
|
||||
char wave[4]; // "WAVE"
|
||||
char fmt[4]; // "fmt "
|
||||
uint32_t fmtSize;
|
||||
uint16_t audioFormat;
|
||||
uint16_t numChannels;
|
||||
uint32_t sampleRate;
|
||||
uint32_t byteRate;
|
||||
uint16_t blockAlign;
|
||||
uint16_t bitsPerSample;
|
||||
char data[4]; // "data"
|
||||
uint32_t dataSize;
|
||||
};
|
||||
|
||||
// Load WAV file and convert to float32 mono 16kHz
|
||||
bool load_wav_file(const std::string& filename, std::vector<float>& audio_data) {
|
||||
std::ifstream file(filename, std::ios::binary);
|
||||
if (!file) {
|
||||
std::cerr << "Failed to open: " << filename << std::endl;
|
||||
return false;
|
||||
}
|
||||
|
||||
WAVHeader header;
|
||||
file.read(reinterpret_cast<char*>(&header), sizeof(WAVHeader));
|
||||
|
||||
// Verify WAV format
|
||||
if (std::string(header.riff, 4) != "RIFF" || std::string(header.wave, 4) != "WAVE") {
|
||||
std::cerr << "Invalid WAV file" << std::endl;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Read audio data
|
||||
std::vector<int16_t> raw_data(header.dataSize / sizeof(int16_t));
|
||||
file.read(reinterpret_cast<char*>(raw_data.data()), header.dataSize);
|
||||
|
||||
// Convert to float and resample if needed
|
||||
audio_data.clear();
|
||||
audio_data.reserve(raw_data.size());
|
||||
|
||||
for (int16_t sample : raw_data) {
|
||||
audio_data.push_back(sample / 32768.0f);
|
||||
}
|
||||
|
||||
std::cout << "Loaded: " << filename << std::endl;
|
||||
std::cout << " Sample rate: " << header.sampleRate << " Hz" << std::endl;
|
||||
std::cout << " Channels: " << header.numChannels << std::endl;
|
||||
std::cout << " Duration: " << (audio_data.size() / (float)header.sampleRate) << " seconds" << std::endl;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
// Test case structure
|
||||
struct TestCase {
|
||||
std::string name;
|
||||
std::string audio_file;
|
||||
std::string expected_text;
|
||||
bool passed = false;
|
||||
std::string actual_text;
|
||||
float duration_ms = 0.0f;
|
||||
};
|
||||
|
||||
// Test runner
|
||||
class AudioTester {
|
||||
public:
|
||||
AudioTester(const std::string& model_path) {
|
||||
m_config.model_path = model_path;
|
||||
m_config.language = "en";
|
||||
m_config.n_threads = std::thread::hardware_concurrency();
|
||||
m_config.use_gpu = true;
|
||||
|
||||
// Initialize transcriber
|
||||
if (!m_transcriber.init(m_config)) {
|
||||
std::cerr << "Failed to initialize transcriber!" << std::endl;
|
||||
exit(1);
|
||||
}
|
||||
|
||||
std::cout << "Transcriber initialized" << std::endl;
|
||||
std::cout << " GPU: " << (m_transcriber.is_using_gpu() ? "ON" : "OFF") << std::endl;
|
||||
std::cout << " Threads: " << m_config.n_threads << std::endl;
|
||||
}
|
||||
|
||||
bool run_test(TestCase& test) {
|
||||
std::cout << "\n=== Test: " << test.name << " ===" << std::endl;
|
||||
|
||||
// Load audio file
|
||||
std::vector<float> audio_data;
|
||||
if (!load_wav_file(test.audio_file, audio_data)) {
|
||||
test.passed = false;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Process audio
|
||||
m_result_text.clear();
|
||||
auto start = std::chrono::high_resolution_clock::now();
|
||||
|
||||
whisper_context* ctx = whisper_init_from_file_with_params(
|
||||
m_config.model_path.c_str(),
|
||||
whisper_context_default_params()
|
||||
);
|
||||
|
||||
if (!ctx) {
|
||||
std::cerr << "Failed to load model!" << std::endl;
|
||||
return false;
|
||||
}
|
||||
|
||||
whisper_full_params wparams = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
|
||||
wparams.language = "en";
|
||||
wparams.n_threads = m_config.n_threads;
|
||||
wparams.print_progress = false;
|
||||
wparams.print_realtime = false;
|
||||
|
||||
int result = whisper_full(ctx, wparams, audio_data.data(), (int)audio_data.size());
|
||||
|
||||
if (result == 0) {
|
||||
const int n_segments = whisper_full_n_segments(ctx);
|
||||
for (int i = 0; i < n_segments; ++i) {
|
||||
const char* text = whisper_full_get_segment_text(ctx, i);
|
||||
if (text) {
|
||||
m_result_text += text;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
whisper_free(ctx);
|
||||
|
||||
auto end = std::chrono::high_resolution_clock::now();
|
||||
test.duration_ms = std::chrono::duration<float, std::milli>(end - start).count();
|
||||
|
||||
// Store result
|
||||
test.actual_text = m_result_text;
|
||||
|
||||
// Trim and compare
|
||||
std::string actual_trimmed = trim(m_result_text);
|
||||
std::string expected_trimmed = trim(test.expected_text);
|
||||
|
||||
// Case-insensitive comparison
|
||||
std::transform(actual_trimmed.begin(), actual_trimmed.end(), actual_trimmed.begin(), ::tolower);
|
||||
std::transform(expected_trimmed.begin(), expected_trimmed.end(), expected_trimmed.begin(), ::tolower);
|
||||
|
||||
test.passed = (actual_trimmed.find(expected_trimmed) != std::string::npos);
|
||||
|
||||
// Print results
|
||||
std::cout << "Expected: \"" << test.expected_text << "\"" << std::endl;
|
||||
std::cout << "Actual: \"" << test.actual_text << "\"" << std::endl;
|
||||
std::cout << "Duration: " << test.duration_ms << " ms" << std::endl;
|
||||
std::cout << "Result: " << (test.passed ? "✓ PASS" : "✗ FAIL") << std::endl;
|
||||
|
||||
return test.passed;
|
||||
}
|
||||
|
||||
private:
|
||||
WhisperConfig m_config;
|
||||
Transcriber m_transcriber;
|
||||
std::string m_result_text;
|
||||
|
||||
std::string trim(const std::string& str) {
|
||||
size_t start = str.find_first_not_of(" \t\n\r");
|
||||
size_t end = str.find_last_not_of(" \t\n\r");
|
||||
if (start == std::string::npos || end == std::string::npos) {
|
||||
return "";
|
||||
}
|
||||
return str.substr(start, end - start + 1);
|
||||
}
|
||||
};
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
std::cout << "=== Whisper Dictation Audio Tests ===" << std::endl;
|
||||
|
||||
// Determine model path
|
||||
std::string model_path = "models/ggml-base.en.bin";
|
||||
if (argc > 1) {
|
||||
model_path = argv[1];
|
||||
}
|
||||
|
||||
std::cout << "Using model: " << model_path << std::endl;
|
||||
|
||||
// Create tester
|
||||
AudioTester tester(model_path);
|
||||
|
||||
// Define test cases
|
||||
std::vector<TestCase> tests = {
|
||||
{"Short sentence", "test-audio/test1.wav", "hello world"},
|
||||
{"Numbers", "test-audio/test2.wav", "one two three four five"},
|
||||
{"Long sentence", "test-audio/test3.wav", "the quick brown fox jumps over the lazy dog"},
|
||||
};
|
||||
|
||||
// Check if test audio directory exists
|
||||
if (!file_exists("test-audio/test1.wav")) {
|
||||
std::cout << "\nNo test-audio directory found. Creating example..." << std::endl;
|
||||
std::cout << "Please add your test WAV files (16kHz, mono) to test-audio/" << std::endl;
|
||||
std::cout << "\nYou can record test audio with:" << std::endl;
|
||||
std::cout << " ffmpeg -f dshow -i audio=\"Your Microphone\" -t 5 -ar 16000 -ac 1 test-audio/test1.wav" << std::endl;
|
||||
std::cout << "\nOr use the recording script:" << std::endl;
|
||||
std::cout << " powershell -ExecutionPolicy Bypass -File record-test-audio.ps1" << std::endl;
|
||||
|
||||
// Try to find user-provided test files
|
||||
if (argc > 2) {
|
||||
std::cout << "\nRunning with user-provided files..." << std::endl;
|
||||
tests.clear();
|
||||
for (int i = 2; i < argc; i += 2) {
|
||||
if (i + 1 < argc) {
|
||||
tests.push_back({
|
||||
argv[i],
|
||||
argv[i],
|
||||
argv[i + 1]
|
||||
});
|
||||
}
|
||||
}
|
||||
} else {
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Run tests
|
||||
int passed = 0;
|
||||
int failed = 0;
|
||||
|
||||
for (auto& test : tests) {
|
||||
if (tester.run_test(test)) {
|
||||
passed++;
|
||||
} else {
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
|
||||
// Summary
|
||||
std::cout << "\n=== Test Summary ===" << std::endl;
|
||||
std::cout << "Total: " << (passed + failed) << std::endl;
|
||||
std::cout << "Passed: " << passed << std::endl;
|
||||
std::cout << "Failed: " << failed << std::endl;
|
||||
std::cout << "Success rate: " << (passed * 100.0f / (passed + failed)) << "%" << std::endl;
|
||||
|
||||
return (failed == 0) ? 0 : 1;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
#pragma once
|
||||
#include <string>
|
||||
|
||||
inline std::wstring append_transcript(const std::wstring& cur, const std::wstring& add) {
|
||||
if (add.empty()) return cur;
|
||||
if (cur.empty()) return add;
|
||||
return cur + L" " + add;
|
||||
}
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
#pragma once
|
||||
#include <windows.h>
|
||||
#include <string>
|
||||
#include <cmath>
|
||||
#include <algorithm>
|
||||
|
||||
struct TimingModel {
|
||||
double n=0, sx=0, sy=0, sxx=0, sxy=0;
|
||||
double a=0, b=0;
|
||||
bool fitted=false;
|
||||
double def_a=0.4, def_b=0.6;
|
||||
|
||||
void recompute() {
|
||||
if (n >= 2.0) {
|
||||
double denom = n*sxx - sx*sx;
|
||||
if (std::fabs(denom) > 1e-9) {
|
||||
double bb = (n*sxy - sx*sy) / denom;
|
||||
double aa = (sy - bb*sx) / n;
|
||||
if (bb < 0.02) bb = def_b;
|
||||
if (aa < 0.0) aa = 0.0;
|
||||
a=aa; b=bb; fitted=true; return;
|
||||
}
|
||||
}
|
||||
a=def_a; b=def_b; fitted=false;
|
||||
}
|
||||
|
||||
double predict(double audio_sec) const {
|
||||
double t = (fitted ? a : def_a) + (fitted ? b : def_b) * audio_sec;
|
||||
return std::max(0.4, t);
|
||||
}
|
||||
|
||||
void add_sample(double audio_sec, double proc_sec) {
|
||||
const double decay = 0.97;
|
||||
n*=decay; sx*=decay; sy*=decay; sxx*=decay; sxy*=decay;
|
||||
n+=1; sx+=audio_sec; sy+=proc_sec;
|
||||
sxx+=audio_sec*audio_sec; sxy+=audio_sec*proc_sec;
|
||||
recompute();
|
||||
}
|
||||
};
|
||||
|
||||
struct ProgressEstimator {
|
||||
double T_hat=1.0, disp_rem=1.0, t=0.0;
|
||||
bool done=false;
|
||||
|
||||
void begin(double T_pred) {
|
||||
T_hat = std::max(0.4, T_pred);
|
||||
disp_rem = T_hat; t = 0.0; done=false;
|
||||
}
|
||||
|
||||
void on_whisper(double t_now, int p) {
|
||||
if (done || p < 5) return;
|
||||
double T_meas = 100.0 * t_now / (double)p;
|
||||
const double alpha = 0.5;
|
||||
T_hat = (1.0-alpha)*T_hat + alpha*T_meas;
|
||||
if (T_hat < t_now) T_hat = t_now;
|
||||
}
|
||||
|
||||
void tick(double dt, float& out_frac, float& out_remaining) {
|
||||
if (done) { out_frac=1.0f; out_remaining=0.0f; return; }
|
||||
t += dt;
|
||||
disp_rem -= dt;
|
||||
double raw_rem = std::max(0.0, T_hat - t);
|
||||
const double maxCatchUp = 2.5;
|
||||
double err = raw_rem - disp_rem;
|
||||
if (err < 0) disp_rem += std::max(err, -maxCatchUp*dt);
|
||||
if (disp_rem < 0) disp_rem = 0;
|
||||
double frac = (t + disp_rem > 1e-6) ? t/(t+disp_rem) : 0.0;
|
||||
if (frac > 0.95) frac = 0.95;
|
||||
out_frac = (float)frac;
|
||||
out_remaining = (float)disp_rem;
|
||||
}
|
||||
|
||||
void finish(float& out_frac, float& out_remaining) {
|
||||
done=true; out_frac=1.0f; out_remaining=0.0f;
|
||||
}
|
||||
|
||||
void reset_busy() {
|
||||
T_hat=1.0; disp_rem=1.0; t=0.0; done=false;
|
||||
}
|
||||
};
|
||||
|
||||
inline std::wstring TimingIniPath() {
|
||||
wchar_t buf[MAX_PATH]; GetModuleFileNameW(nullptr, buf, MAX_PATH);
|
||||
std::wstring p(buf); p = p.substr(0, p.find_last_of(L"\\/"));
|
||||
return p + L"\\win-dictation.ini";
|
||||
}
|
||||
inline std::wstring SectionFor(const std::string& modelPath) {
|
||||
std::string base = modelPath.substr(modelPath.find_last_of("\\/")+1);
|
||||
return L"timing-" + std::wstring(base.begin(), base.end());
|
||||
}
|
||||
inline void PutD(const std::wstring& sec, const wchar_t* k, double v) {
|
||||
wchar_t b[64]; swprintf_s(b, L"%.6f", v);
|
||||
WritePrivateProfileStringW(sec.c_str(), k, b, TimingIniPath().c_str());
|
||||
}
|
||||
inline double GetD(const std::wstring& sec, const wchar_t* k, double d) {
|
||||
wchar_t b[64]; swprintf_s(b, L"%.6f", d);
|
||||
wchar_t out[64];
|
||||
GetPrivateProfileStringW(sec.c_str(), k, b, out, 64, TimingIniPath().c_str());
|
||||
return wcstod(out, nullptr);
|
||||
}
|
||||
inline void LoadTiming(TimingModel& m, const std::string& modelPath) {
|
||||
auto s = SectionFor(modelPath);
|
||||
m.n=GetD(s,L"n",0); m.sx=GetD(s,L"sx",0); m.sy=GetD(s,L"sy",0);
|
||||
m.sxx=GetD(s,L"sxx",0); m.sxy=GetD(s,L"sxy",0);
|
||||
m.recompute();
|
||||
}
|
||||
inline void SaveTiming(const TimingModel& m, const std::string& modelPath) {
|
||||
auto s = SectionFor(modelPath);
|
||||
PutD(s,L"n",m.n); PutD(s,L"sx",m.sx); PutD(s,L"sy",m.sy);
|
||||
PutD(s,L"sxx",m.sxx); PutD(s,L"sxy",m.sxy);
|
||||
}
|
||||
|
||||
inline void SeedDefaults(TimingModel& m, const std::string& modelPath) {
|
||||
std::string p = modelPath;
|
||||
auto has = [&](const char* s){ return p.find(s)!=std::string::npos; };
|
||||
if (has("tiny")) { m.def_a=0.3; m.def_b=0.45; }
|
||||
else if (has("base")) { m.def_a=0.5; m.def_b=1.10; }
|
||||
else if (has("small")){ m.def_a=0.8; m.def_b=3.00; }
|
||||
else { m.def_a=0.5; m.def_b=1.00; }
|
||||
m.recompute();
|
||||
}
|
||||
+185
-364
@@ -1,383 +1,204 @@
|
||||
#include "transcriber.h"
|
||||
#include "whisper.h"
|
||||
// Note: WHISPER_SAMPLE_RATE is defined in whisper.h, so common.h is not needed
|
||||
|
||||
#include <SDL.h>
|
||||
#include <SDL_audio.h>
|
||||
|
||||
#include <iostream>
|
||||
#include <chrono>
|
||||
#include <cmath>
|
||||
#include <numeric>
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cstring>
|
||||
|
||||
Transcriber::Transcriber() {
|
||||
m_ring_buffer.resize(RING_BUFFER_SIZE, 0.0f);
|
||||
Transcriber::~Transcriber() {
|
||||
cancel();
|
||||
if (m_worker.joinable()) m_worker.join();
|
||||
if (m_ctx) whisper_free(m_ctx);
|
||||
}
|
||||
|
||||
Transcriber::~Transcriber() {
|
||||
stop();
|
||||
free_model();
|
||||
int Transcriber::default_threads() {
|
||||
unsigned hc = std::thread::hardware_concurrency();
|
||||
if (hc <= 2) return (int)std::max(1u, hc);
|
||||
return (int)(hc / 2);
|
||||
}
|
||||
|
||||
float Transcriber::recorded_seconds() const {
|
||||
std::lock_guard<std::mutex> lk(m_capture_mtx);
|
||||
return (float)(m_capture.size() / (double)WHISPER_SAMPLE_RATE);
|
||||
}
|
||||
|
||||
void Transcriber::s_progress(whisper_context*, whisper_state*, int p, void* ud) {
|
||||
auto* self = static_cast<Transcriber*>(ud);
|
||||
if (self && self->m_on_progress) self->m_on_progress(p);
|
||||
}
|
||||
|
||||
bool Transcriber::s_abort(void* ud) {
|
||||
auto* self = static_cast<Transcriber*>(ud);
|
||||
return self && self->m_abort.load();
|
||||
}
|
||||
|
||||
bool Transcriber::preload(const WhisperConfig& cfg) {
|
||||
std::lock_guard<std::mutex> lk(m_cfg_mtx);
|
||||
m_cfg = cfg;
|
||||
if (m_cfg.n_threads <= 0) m_cfg.n_threads = default_threads();
|
||||
if (m_ctx) return true;
|
||||
whisper_context_params cp = whisper_context_default_params();
|
||||
cp.use_gpu = m_cfg.use_gpu;
|
||||
m_ctx = whisper_init_from_file_with_params(m_cfg.model_path.c_str(), cp);
|
||||
return m_ctx != nullptr;
|
||||
}
|
||||
|
||||
static void sdl_capture_cb(void* user, Uint8* stream, int len) {
|
||||
auto* self = static_cast<Transcriber*>(user);
|
||||
self->on_audio(reinterpret_cast<float*>(stream), len / (int)sizeof(float));
|
||||
}
|
||||
|
||||
bool Transcriber::start_recording() {
|
||||
if (m_recording.load() || m_busy.load()) return false;
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(m_capture_mtx);
|
||||
m_capture.clear();
|
||||
m_capture.reserve(WHISPER_SAMPLE_RATE * 30);
|
||||
}
|
||||
|
||||
SDL_AudioSpec want{}, have{};
|
||||
want.freq = WHISPER_SAMPLE_RATE;
|
||||
want.format = AUDIO_F32;
|
||||
want.channels = 1;
|
||||
want.samples = 1024;
|
||||
want.callback = sdl_capture_cb;
|
||||
want.userdata = this;
|
||||
|
||||
const char* dev = SDL_GetAudioDeviceName(m_cfg.capture_id, SDL_TRUE);
|
||||
m_dev = SDL_OpenAudioDevice(dev, SDL_TRUE, &want, &have, 0);
|
||||
if (!m_dev) return false;
|
||||
|
||||
m_energy = 0.0f;
|
||||
m_recording = true;
|
||||
SDL_PauseAudioDevice(m_dev, 0);
|
||||
return true;
|
||||
}
|
||||
|
||||
void Transcriber::on_audio(const float* s, int n) {
|
||||
if (n <= 0 || !m_recording.load()) return;
|
||||
double sq = 0.0;
|
||||
for (int i = 0; i < n; ++i) sq += (double)s[i] * s[i];
|
||||
float rms = (float)std::sqrt(sq / n);
|
||||
float e = m_energy.load();
|
||||
m_energy = std::min(1.0f, e * 0.6f + (rms * 4.0f) * 0.4f);
|
||||
std::lock_guard<std::mutex> lk(m_capture_mtx);
|
||||
m_capture.insert(m_capture.end(), s, s + n);
|
||||
}
|
||||
|
||||
void Transcriber::cancel() {
|
||||
if (!m_recording.load()) return;
|
||||
m_recording = false;
|
||||
if (m_dev) { SDL_PauseAudioDevice(m_dev, 1); SDL_CloseAudioDevice(m_dev); m_dev = 0; }
|
||||
std::lock_guard<std::mutex> lk(m_capture_mtx);
|
||||
m_capture.clear();
|
||||
m_energy = 0.0f;
|
||||
}
|
||||
|
||||
void Transcriber::stop_and_transcribe() {
|
||||
if (!m_recording.load()) return;
|
||||
m_recording = false;
|
||||
if (m_dev) { SDL_PauseAudioDevice(m_dev, 1); SDL_CloseAudioDevice(m_dev); m_dev = 0; }
|
||||
m_energy = 0.0f;
|
||||
|
||||
std::vector<float> audio;
|
||||
{ std::lock_guard<std::mutex> lk(m_capture_mtx); audio.swap(m_capture); }
|
||||
|
||||
if (audio.size() < (size_t)(WHISPER_SAMPLE_RATE * 0.3)) {
|
||||
if (m_on_result) m_on_result("");
|
||||
return;
|
||||
}
|
||||
if (m_worker.joinable()) m_worker.join();
|
||||
m_busy = true;
|
||||
m_worker = std::thread(&Transcriber::transcribe_worker, this, std::move(audio));
|
||||
}
|
||||
|
||||
static void trim_silence(std::vector<float>& a, float thresh = 0.01f) {
|
||||
const size_t win = 1600;
|
||||
auto loud = [&](size_t i) {
|
||||
float m = 0.f;
|
||||
for (size_t k = i; k < std::min(a.size(), i + win); ++k)
|
||||
m = std::max(m, std::fabs(a[k]));
|
||||
return m > thresh;
|
||||
};
|
||||
size_t s = 0, e = a.size();
|
||||
while (s + win < a.size() && !loud(s)) s += win;
|
||||
while (e > win && !loud(e - win)) e -= win;
|
||||
if (s + win <= e)
|
||||
a.assign(a.begin() + (s > win ? s - win : 0), a.begin() + e);
|
||||
}
|
||||
|
||||
static std::string clean_text(std::string s) {
|
||||
const char* junk[] = {"[BLANK_AUDIO]", "[NOISE]", "(blank)", "(noise)", "[ Silence ]"};
|
||||
for (auto j : junk) {
|
||||
size_t p;
|
||||
while ((p = s.find(j)) != std::string::npos) s.erase(p, strlen(j));
|
||||
}
|
||||
size_t b = s.find_first_not_of(" \t\r\n");
|
||||
size_t e = s.find_last_not_of(" \t\r\n");
|
||||
return (b == std::string::npos) ? "" : s.substr(b, e - b + 1);
|
||||
}
|
||||
|
||||
std::string Transcriber::run_inference(std::vector<float>& audio) {
|
||||
if (!m_ctx) return "";
|
||||
m_abort = false;
|
||||
if (m_cfg.trim_silence) trim_silence(audio);
|
||||
m_audio_seconds = (float)(audio.size() / (double)WHISPER_SAMPLE_RATE);
|
||||
whisper_full_params wp = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
|
||||
wp.print_progress = false;
|
||||
wp.print_realtime = false;
|
||||
wp.print_timestamps = false;
|
||||
wp.no_timestamps = true;
|
||||
wp.translate = false;
|
||||
wp.language = m_cfg.language.c_str();
|
||||
wp.n_threads = m_cfg.n_threads;
|
||||
wp.no_context = true;
|
||||
wp.suppress_blank = true;
|
||||
wp.suppress_nst = true;
|
||||
wp.temperature = 0.0f;
|
||||
wp.progress_callback = &Transcriber::s_progress;
|
||||
wp.progress_callback_user_data = this;
|
||||
wp.abort_callback = &Transcriber::s_abort;
|
||||
wp.abort_callback_user_data = this;
|
||||
std::string out;
|
||||
if (whisper_full(m_ctx, wp, audio.data(), (int)audio.size()) == 0) {
|
||||
int n = whisper_full_n_segments(m_ctx);
|
||||
for (int i = 0; i < n; ++i) {
|
||||
const char* t = whisper_full_get_segment_text(m_ctx, i);
|
||||
if (t) out += t;
|
||||
}
|
||||
out = clean_text(out);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void Transcriber::transcribe_worker(std::vector<float> audio) {
|
||||
std::string out = run_inference(audio);
|
||||
m_busy = false;
|
||||
if (m_on_result) m_on_result(out);
|
||||
}
|
||||
|
||||
std::string Transcriber::transcribe_sync(std::vector<float> audio) {
|
||||
return run_inference(audio);
|
||||
}
|
||||
|
||||
bool Transcriber::reload(const WhisperConfig& cfg) {
|
||||
if (m_recording.load() || m_busy.load()) return false;
|
||||
if (m_worker.joinable()) m_worker.join();
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(m_cfg_mtx);
|
||||
if (m_ctx) { whisper_free(m_ctx); m_ctx = nullptr; }
|
||||
}
|
||||
return preload(cfg);
|
||||
}
|
||||
|
||||
std::vector<std::string> Transcriber::get_audio_devices() {
|
||||
std::vector<std::string> devices;
|
||||
|
||||
if (SDL_Init(SDL_INIT_AUDIO) < 0) {
|
||||
return devices;
|
||||
}
|
||||
|
||||
if (SDL_Init(SDL_INIT_AUDIO) < 0) return devices;
|
||||
int nDevices = SDL_GetNumAudioDevices(SDL_TRUE);
|
||||
for (int i = 0; i < nDevices; ++i) {
|
||||
const char* name = SDL_GetAudioDeviceName(i, SDL_TRUE);
|
||||
if (name) {
|
||||
devices.push_back(name);
|
||||
}
|
||||
if (name) devices.push_back(name);
|
||||
}
|
||||
return devices;
|
||||
}
|
||||
|
||||
bool Transcriber::init(const WhisperConfig& config) {
|
||||
m_config = config;
|
||||
|
||||
// Load model immediately to check GPU availability
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
if (!m_ctx) {
|
||||
struct whisper_context_params cparams = whisper_context_default_params();
|
||||
cparams.use_gpu = m_config.use_gpu;
|
||||
|
||||
m_ctx = whisper_init_from_file_with_params(m_config.model_path.c_str(), cparams);
|
||||
|
||||
if (!m_ctx) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Check if GPU is actually active
|
||||
const char* info = whisper_print_system_info();
|
||||
m_gpu_active = (info && (strstr(info, "CUDA") != nullptr ||
|
||||
strstr(info, "Metal") != nullptr ||
|
||||
strstr(info, "HIP") != nullptr ||
|
||||
strstr(info, "Vulkan") != nullptr));
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void Transcriber::free_model() {
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
if (m_ctx) {
|
||||
whisper_free(m_ctx);
|
||||
m_ctx = nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
void Transcriber::start() {
|
||||
if (m_running) return;
|
||||
|
||||
m_should_stop = false;
|
||||
|
||||
// Clear ring buffer completely
|
||||
m_ring_write_pos = 0;
|
||||
m_ring_read_pos = 0;
|
||||
|
||||
// Clear processing buffer
|
||||
m_processing_buffer.clear();
|
||||
m_last_process_time = std::chrono::steady_clock::now();
|
||||
|
||||
m_worker = std::thread(&Transcriber::worker_loop, this);
|
||||
m_running = true;
|
||||
}
|
||||
|
||||
void Transcriber::stop() {
|
||||
if (!m_running) return;
|
||||
|
||||
m_should_stop = true;
|
||||
m_ring_cv.notify_all();
|
||||
|
||||
// Wait for worker thread to complete
|
||||
if (m_worker.joinable()) {
|
||||
m_worker.join();
|
||||
}
|
||||
|
||||
// Clear ALL state to prevent contamination
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(m_ring_mutex);
|
||||
// Reset ring buffer positions
|
||||
m_ring_write_pos = 0;
|
||||
m_ring_read_pos = 0;
|
||||
// Clear ring buffer data
|
||||
std::fill(m_ring_buffer.begin(), m_ring_buffer.end(), 0.0f);
|
||||
}
|
||||
|
||||
// Clear processing buffer
|
||||
m_processing_buffer.clear();
|
||||
|
||||
m_running = false;
|
||||
m_audio_energy = 0.0f;
|
||||
}
|
||||
|
||||
void Transcriber::set_callback(Callback cb) {
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
m_callback = cb;
|
||||
}
|
||||
|
||||
float Transcriber::get_audio_energy() {
|
||||
return m_audio_energy;
|
||||
}
|
||||
|
||||
size_t Transcriber::get_queue_size() {
|
||||
size_t write_pos = m_ring_write_pos.load();
|
||||
size_t read_pos = m_ring_read_pos.load();
|
||||
|
||||
if (write_pos >= read_pos) {
|
||||
return write_pos - read_pos;
|
||||
} else {
|
||||
return RING_BUFFER_SIZE - read_pos + write_pos;
|
||||
}
|
||||
}
|
||||
|
||||
float Transcriber::get_buffer_fullness() {
|
||||
return (float)get_queue_size() / (float)RING_BUFFER_SIZE;
|
||||
}
|
||||
|
||||
bool Transcriber::is_using_gpu() const {
|
||||
return m_gpu_active;
|
||||
}
|
||||
|
||||
// Ring buffer audio callback - NO DATA LOSS
|
||||
void Transcriber::audio_callback(const float* samples, int n_samples) {
|
||||
if (n_samples <= 0) return;
|
||||
|
||||
// Calculate RMS for VU meter (with smoothing)
|
||||
double sum_sq = 0.0;
|
||||
for (int i = 0; i < n_samples; i++) {
|
||||
sum_sq += samples[i] * samples[i];
|
||||
}
|
||||
float rms = (float)std::sqrt(sum_sq / n_samples);
|
||||
|
||||
// Smooth the energy reading for better visual effect
|
||||
float current_energy = m_audio_energy.load();
|
||||
float new_energy = current_energy * 0.7f + (rms * 5.0f) * 0.3f;
|
||||
m_audio_energy = std::min(1.0f, new_energy);
|
||||
|
||||
// Write to ring buffer (lock-free for audio thread)
|
||||
size_t write_pos = m_ring_write_pos.load(std::memory_order_acquire);
|
||||
|
||||
for (int i = 0; i < n_samples; i++) {
|
||||
size_t next_pos = (write_pos + 1) % RING_BUFFER_SIZE;
|
||||
|
||||
// Check if buffer is full (would overwrite unread data)
|
||||
if (next_pos == m_ring_read_pos.load(std::memory_order_acquire)) {
|
||||
// Buffer full - drop oldest samples (shouldn't happen with 30s buffer)
|
||||
m_ring_read_pos.store((m_ring_read_pos.load() + 1) % RING_BUFFER_SIZE, std::memory_order_release);
|
||||
}
|
||||
|
||||
m_ring_buffer[write_pos] = samples[i];
|
||||
write_pos = next_pos;
|
||||
}
|
||||
|
||||
m_ring_write_pos.store(write_pos, std::memory_order_release);
|
||||
m_ring_cv.notify_one();
|
||||
}
|
||||
|
||||
// SDL callback wrapper
|
||||
static void sdl_audio_callback(void* userdata, Uint8* stream, int len) {
|
||||
Transcriber* self = (Transcriber*)userdata;
|
||||
int n_samples = len / sizeof(float);
|
||||
float* samples = (float*)stream;
|
||||
self->audio_callback(samples, n_samples);
|
||||
}
|
||||
|
||||
void Transcriber::process_audio_chunk(const std::vector<float>& audio_data) {
|
||||
if (audio_data.empty()) return;
|
||||
|
||||
// Minimum audio length check (at least 1 second for reliable transcription)
|
||||
const size_t min_samples = WHISPER_SAMPLE_RATE; // 1 second
|
||||
if (audio_data.size() < min_samples) {
|
||||
return; // Need more audio data
|
||||
}
|
||||
|
||||
// Basic energy check - skip completely silent audio
|
||||
float max_energy = 0.0f;
|
||||
for (float sample : audio_data) {
|
||||
max_energy = std::max(max_energy, std::abs(sample));
|
||||
}
|
||||
if (max_energy < 0.001f) { // Essentially silent
|
||||
return;
|
||||
}
|
||||
|
||||
// Run Whisper inference
|
||||
whisper_full_params wparams = whisper_full_default_params(WHISPER_SAMPLING_GREEDY);
|
||||
wparams.print_progress = false;
|
||||
wparams.print_realtime = false;
|
||||
wparams.print_timestamps = false;
|
||||
wparams.language = m_config.language.c_str();
|
||||
wparams.n_threads = m_config.n_threads;
|
||||
wparams.no_context = true; // CRITICAL: Don't reuse previous text as context!
|
||||
wparams.single_segment = false;
|
||||
wparams.suppress_blank = true; // Suppress blank outputs
|
||||
|
||||
// Reset the context state before each inference to prevent contamination
|
||||
whisper_reset_timings(m_ctx);
|
||||
|
||||
int result = whisper_full(m_ctx, wparams, audio_data.data(), (int)audio_data.size());
|
||||
|
||||
if (result != 0) {
|
||||
return; // Skip this chunk on error
|
||||
}
|
||||
|
||||
// Get transcribed text and filter blanks
|
||||
const int n_segments = whisper_full_n_segments(m_ctx);
|
||||
std::string segment_text;
|
||||
for (int i = 0; i < n_segments; ++i) {
|
||||
const char* text = whisper_full_get_segment_text(m_ctx, i);
|
||||
if (text && strlen(text) > 0) {
|
||||
std::string seg(text);
|
||||
|
||||
// Filter out blank/noise tokens
|
||||
if (seg.find("[BLANK_AUDIO]") == std::string::npos &&
|
||||
seg.find("[NOISE]") == std::string::npos &&
|
||||
seg.find("(blank)") == std::string::npos &&
|
||||
seg.find("(noise)") == std::string::npos &&
|
||||
seg != " " && seg != " ") {
|
||||
segment_text += seg;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Send callback only if we have real content
|
||||
if (!segment_text.empty()) {
|
||||
// Trim whitespace
|
||||
size_t start = segment_text.find_first_not_of(" \t\n\r");
|
||||
size_t end = segment_text.find_last_not_of(" \t\n\r");
|
||||
if (start != std::string::npos && end != std::string::npos) {
|
||||
segment_text = segment_text.substr(start, end - start + 1);
|
||||
|
||||
// Only send if meaningful content (at least 2 characters)
|
||||
if (segment_text.length() >= 2) {
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
if (m_callback) {
|
||||
m_callback(segment_text);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void Transcriber::worker_loop() {
|
||||
// Model should already be loaded from init()
|
||||
if (!m_ctx) {
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
if (m_callback) m_callback("[Error: Model not loaded]\n");
|
||||
return;
|
||||
}
|
||||
|
||||
// Initialize SDL Audio
|
||||
if (SDL_Init(SDL_INIT_AUDIO) < 0) {
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
if (m_callback) m_callback("[Error: SDL Init failed]\n");
|
||||
return;
|
||||
}
|
||||
|
||||
SDL_AudioSpec capture_spec_requested;
|
||||
SDL_AudioSpec capture_spec_obtained;
|
||||
SDL_zero(capture_spec_requested);
|
||||
SDL_zero(capture_spec_obtained);
|
||||
|
||||
capture_spec_requested.freq = WHISPER_SAMPLE_RATE;
|
||||
capture_spec_requested.format = AUDIO_F32;
|
||||
capture_spec_requested.channels = 1;
|
||||
capture_spec_requested.samples = 512; // Smaller buffer for lower latency
|
||||
capture_spec_requested.callback = sdl_audio_callback;
|
||||
capture_spec_requested.userdata = this;
|
||||
|
||||
const char* device_name = SDL_GetAudioDeviceName(m_config.capture_id, SDL_TRUE);
|
||||
m_dev_id_in = SDL_OpenAudioDevice(
|
||||
device_name,
|
||||
SDL_TRUE,
|
||||
&capture_spec_requested,
|
||||
&capture_spec_obtained,
|
||||
0
|
||||
);
|
||||
|
||||
if (!m_dev_id_in) {
|
||||
std::lock_guard<std::mutex> lock(m_mutex);
|
||||
if (m_callback) m_callback("[Error: Failed to open audio device]\n");
|
||||
return;
|
||||
}
|
||||
|
||||
SDL_PauseAudioDevice(m_dev_id_in, 0); // Start capturing
|
||||
|
||||
// Processing parameters
|
||||
const size_t n_samples_step = (size_t)((1e-3 * m_config.step_ms) * WHISPER_SAMPLE_RATE);
|
||||
const size_t n_samples_len = (size_t)((1e-3 * m_config.length_ms) * WHISPER_SAMPLE_RATE);
|
||||
const size_t n_samples_keep = (size_t)((1e-3 * 200) * WHISPER_SAMPLE_RATE); // Keep 200ms overlap
|
||||
|
||||
m_processing_buffer.clear();
|
||||
m_processing_buffer.reserve(n_samples_len * 2);
|
||||
|
||||
while (!m_should_stop) {
|
||||
// Wait for audio data with shorter timeout for responsiveness
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(m_ring_mutex);
|
||||
m_ring_cv.wait_for(lock, std::chrono::milliseconds(50), [&]{
|
||||
return get_queue_size() >= n_samples_step || m_should_stop;
|
||||
});
|
||||
}
|
||||
|
||||
if (m_should_stop) break;
|
||||
|
||||
// Read from ring buffer - need enough data for reliable transcription
|
||||
size_t available = get_queue_size();
|
||||
if (available < n_samples_step) { // Need at least full threshold
|
||||
continue;
|
||||
}
|
||||
|
||||
// Read samples from ring buffer
|
||||
std::vector<float> new_samples;
|
||||
new_samples.reserve(available);
|
||||
|
||||
size_t read_pos = m_ring_read_pos.load(std::memory_order_acquire);
|
||||
size_t write_pos = m_ring_write_pos.load(std::memory_order_acquire);
|
||||
|
||||
while (read_pos != write_pos) {
|
||||
new_samples.push_back(m_ring_buffer[read_pos]);
|
||||
read_pos = (read_pos + 1) % RING_BUFFER_SIZE;
|
||||
}
|
||||
|
||||
m_ring_read_pos.store(read_pos, std::memory_order_release);
|
||||
|
||||
// Append new samples to processing buffer
|
||||
m_processing_buffer.insert(m_processing_buffer.end(), new_samples.begin(), new_samples.end());
|
||||
|
||||
// Process when we have enough data
|
||||
if (m_processing_buffer.size() >= n_samples_len) {
|
||||
// Take exactly n_samples_len for processing
|
||||
std::vector<float> chunk(
|
||||
m_processing_buffer.end() - n_samples_len,
|
||||
m_processing_buffer.end()
|
||||
);
|
||||
|
||||
// Process this chunk
|
||||
process_audio_chunk(chunk);
|
||||
|
||||
// CRITICAL: Remove processed audio, keep only overlap for continuity
|
||||
// This prevents re-processing the same audio repeatedly!
|
||||
size_t samples_to_remove = m_processing_buffer.size() - n_samples_keep;
|
||||
if (samples_to_remove > 0) {
|
||||
m_processing_buffer.erase(
|
||||
m_processing_buffer.begin(),
|
||||
m_processing_buffer.begin() + samples_to_remove
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Process any remaining audio
|
||||
if (!m_processing_buffer.empty()) {
|
||||
process_audio_chunk(m_processing_buffer);
|
||||
}
|
||||
|
||||
SDL_CloseAudioDevice(m_dev_id_in);
|
||||
SDL_Quit();
|
||||
}
|
||||
|
||||
+52
-59
@@ -2,83 +2,76 @@
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <deque>
|
||||
#include <thread>
|
||||
#include <mutex>
|
||||
#include <atomic>
|
||||
#include <functional>
|
||||
#include <condition_variable>
|
||||
#include <memory>
|
||||
|
||||
struct whisper_context;
|
||||
|
||||
struct WhisperConfig {
|
||||
std::string model_path;
|
||||
std::string language = "en";
|
||||
int n_threads = std::thread::hardware_concurrency(); // Use all available threads
|
||||
int step_ms = 1000; // Process every 1s (reliable transcription)
|
||||
int length_ms = 6000; // 6s context window (good balance)
|
||||
bool use_gpu = true; // Auto-detect and use if available
|
||||
int capture_id = 0; // Default to first device
|
||||
int n_gpu_layers = -1; // -1 = auto (all layers if GPU available)
|
||||
std::string model_path = "models/ggml-tiny.en.bin";
|
||||
std::string language = "en";
|
||||
int n_threads = 0; // 0 = auto (physical cores)
|
||||
bool use_gpu = false;
|
||||
int capture_id = 0;
|
||||
bool trim_silence = true;
|
||||
};
|
||||
|
||||
class Transcriber {
|
||||
public:
|
||||
using Callback = std::function<void(const std::string&)>;
|
||||
using ResultCb = std::function<void(const std::string&)>;
|
||||
using ProgressCb = std::function<void(int)>;
|
||||
|
||||
Transcriber();
|
||||
Transcriber() = default;
|
||||
~Transcriber();
|
||||
|
||||
bool init(const WhisperConfig& config);
|
||||
void start();
|
||||
void stop();
|
||||
void set_callback(Callback cb);
|
||||
bool is_running() const { return m_running; }
|
||||
bool preload(const WhisperConfig& cfg);
|
||||
bool reload(const WhisperConfig& cfg);
|
||||
bool is_loaded() const { return m_ctx != nullptr; }
|
||||
|
||||
// Audio device management
|
||||
int threads() const { return m_cfg.n_threads; }
|
||||
|
||||
bool start_recording();
|
||||
void stop_and_transcribe();
|
||||
void cancel();
|
||||
|
||||
bool is_recording() const { return m_recording.load(); }
|
||||
bool is_busy() const { return m_busy.load(); }
|
||||
float get_audio_energy() const { return m_energy.load(); }
|
||||
float audio_seconds() const { return m_audio_seconds.load(); }
|
||||
float recorded_seconds() const;
|
||||
|
||||
void set_result_callback(ResultCb cb) { m_on_result = std::move(cb); }
|
||||
void set_progress_callback(ProgressCb cb) { m_on_progress = std::move(cb); }
|
||||
void request_cancel() { m_abort = true; }
|
||||
|
||||
void on_audio(const float* samples, int n);
|
||||
std::string transcribe_sync(std::vector<float> audio);
|
||||
static std::vector<std::string> get_audio_devices();
|
||||
float get_audio_energy(); // 0.0 to 1.0 (normalized)
|
||||
|
||||
// Status
|
||||
bool is_using_gpu() const;
|
||||
size_t get_queue_size();
|
||||
float get_buffer_fullness(); // 0.0 to 1.0
|
||||
|
||||
// Resource management
|
||||
void free_model();
|
||||
|
||||
// Internal audio callback (public so C callback can reach it)
|
||||
void audio_callback(const float* samples, int n_samples);
|
||||
|
||||
private:
|
||||
void worker_loop();
|
||||
void process_audio_chunk(const std::vector<float>& audio_data);
|
||||
void transcribe_worker(std::vector<float> audio);
|
||||
std::string run_inference(std::vector<float>& audio);
|
||||
static int default_threads();
|
||||
|
||||
WhisperConfig m_cfg;
|
||||
std::mutex m_cfg_mtx;
|
||||
whisper_context* m_ctx = nullptr;
|
||||
|
||||
unsigned int m_dev = 0;
|
||||
std::vector<float> m_capture;
|
||||
mutable std::mutex m_capture_mtx;
|
||||
|
||||
std::atomic<bool> m_recording{false};
|
||||
std::atomic<bool> m_busy{false};
|
||||
std::atomic<float> m_energy{0.0f};
|
||||
std::atomic<float> m_audio_seconds{0.0f};
|
||||
std::atomic<bool> m_abort{false};
|
||||
|
||||
WhisperConfig m_config;
|
||||
std::atomic<bool> m_running{false};
|
||||
std::atomic<bool> m_should_stop{false};
|
||||
std::thread m_worker;
|
||||
std::mutex m_mutex;
|
||||
Callback m_callback;
|
||||
|
||||
// Shared audio energy level (smoothed)
|
||||
std::atomic<float> m_audio_energy{0.0f};
|
||||
std::atomic<bool> m_gpu_active{false};
|
||||
ResultCb m_on_result;
|
||||
ProgressCb m_on_progress;
|
||||
|
||||
// Audio Capture State
|
||||
uint32_t m_dev_id_in = 0;
|
||||
|
||||
// Ring buffer for audio - prevents any loss
|
||||
static constexpr size_t RING_BUFFER_SIZE = 16000 * 30; // 30 seconds max buffer
|
||||
std::vector<float> m_ring_buffer;
|
||||
std::atomic<size_t> m_ring_write_pos{0};
|
||||
std::atomic<size_t> m_ring_read_pos{0};
|
||||
std::mutex m_ring_mutex;
|
||||
std::condition_variable m_ring_cv;
|
||||
|
||||
struct whisper_context* m_ctx = nullptr;
|
||||
|
||||
// Processing buffer to maintain context
|
||||
std::vector<float> m_processing_buffer;
|
||||
std::chrono::steady_clock::time_point m_last_process_time;
|
||||
static void s_progress(struct whisper_context*, struct whisper_state*, int p, void* ud);
|
||||
static bool s_abort(void* ud);
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user