# Testing Guide for win-dictation

## Overview

This document describes how to test win-dictation with known audio samples to ensure end-to-end functionality.

## Test Framework

### Components

1. **test-audio.exe** - Automated test runner that processes WAV files
2. **record-test-audio.ps1** - Script to record test audio with known transcriptions
3. **test-audio/** - Directory containing test WAV files and expected transcriptions

## Quick Start

### 1. Record Test Audio

```powershell
powershell -ExecutionPolicy Bypass -File examples/win-dictation/record-test-audio.ps1
```

This will:
- List your available microphones
- Guide you through recording 5 test clips
- Save WAV files (16kHz, mono) with expected transcriptions
- Create test-audio/*.wav and test-audio/*.wav.txt files

### 2. Build Test Program

```powershell
cmake --build build --config Release --target test-audio
```

### 3. Run Tests

```powershell
cd build/bin/Release
.\test-audio.exe
```

Or with custom model:

```powershell
.\test-audio.exe models/ggml-medium.en.bin
```

## Test Output

```
=== Test: test1 ===
Loaded: test-audio/test1.wav
  Sample rate: 16000 Hz
  Channels: 1
  Duration: 5.0 seconds
Expected: "hello world"
Actual:   " Hello world."
Duration: 1234.5 ms
Result:   ✓ PASS

=== Test Summary ===
Total: 5
Passed: 5
Failed: 0
Success rate: 100%
```

## Manual Testing with Provided Audio

If you have pre-recorded test files:

```powershell
.\test-audio.exe models/ggml-base.en.bin `
    test-audio/custom1.wav "expected text here" `
    test-audio/custom2.wav "another expected text"
```

## Test Cases

### Default Test Suite

| Test | Audio File | Expected Text | Purpose |
|------|------------|---------------|---------|
| test1 | test1.wav | "Hello world" | Basic functionality |
| test2 | test2.wav | "One two three four five" | Number recognition |
| test3 | test3.wav | "The quick brown fox..." | Long sentence |
| test4 | test4.wav | "Testing microphone switching" | Device switching |
| test5 | test5.wav | "This is a longer sentence..." | Real-time feel |

### Creating Custom Tests

1. Record audio at 16kHz, mono, WAV format:

```powershell
ffmpeg -f dshow -i audio="Your Microphone" -t 5 -ar 16000 -ac 1 test-audio/mycustom.wav
```

2. Create expected transcription file:

```powershell
"my expected transcription" | Out-File test-audio/mycustom.wav.txt
```

3. Run test:

```powershell
.\test-audio.exe models/ggml-base.en.bin test-audio/mycustom.wav "my expected transcription"
```

## Bug Reproduction Tests

### Test Microphone Switching

1. Start win-dictation
2. Record a segment with microphone A
3. Switch to microphone B while recording
4. Verify:
   - No repeated text
   - No missed segments
   - Clean transition

### Test Stop/Start Cycles

1. Record segment
2. Stop
3. Start again (without clear)
4. Record another segment
5. Verify:
   - Text appends correctly
   - No old data contamination
   - No missing audio

### Test Real-Time Response

1. Record continuous speech
2. Measure time from speaking to text appearing
3. Expected: < 1 second
4. Verify: No "looping" or repeated text

## Performance Benchmarks

### Metrics to Track

- **Latency**: Time from audio input to text output
- **Throughput**: Audio processed per second (should be >1x real-time)
- **Accuracy**: WER (Word Error Rate) on known transcriptions
- **Memory**: Peak memory usage during recording
- **CPU/GPU**: Resource utilization

### Running Benchmarks

```powershell
# Measure processing time
Measure-Command { .\test-audio.exe }

# Check accuracy
.\test-audio.exe | Select-String "Success rate"
```

## Troubleshooting Tests

### No test-audio directory

Create it manually:

```powershell
mkdir test-audio
```

Then record audio or copy pre-recorded WAV files.

### ffmpeg not found

Install ffmpeg:
- Download from: https://ffmpeg.org/download.html
- Add to PATH
- Or use pre-recorded audio files

### Test failures

Check:
1. Audio format (16kHz, mono, WAV)
2. Model path is correct
3. Expected text matches reasonably (case-insensitive, fuzzy match)
4. Audio quality is good

### Model loading errors

Ensure model file exists:

```powershell
Test-Path models/ggml-base.en.bin
```

Download if missing:

```powershell
cd models
.\download-ggml-model.sh base.en  # or use .cmd on Windows
```

## CI/CD Integration

### GitHub Actions Example

```yaml
- name: Run audio tests
  run: |
    cd build/bin/Release
    ./test-audio.exe ../../models/ggml-base.en.bin
```

### Pre-commit Hook

```bash
#!/bin/bash
cd build/bin/Release
./test-audio.exe || exit 1
```

## Test Coverage

Current tests cover:
- ✅ Basic transcription
- ✅ Number recognition
- ✅ Long sentences
- ✅ Performance measurement
- ⬜ Multiple languages (TODO)
- ⬜ Noise robustness (TODO)
- ⬜ Different accents (TODO)

## Contributing Tests

To add new test cases:

1. Record audio with `record-test-audio.ps1`
2. Verify transcription quality
3. Add to default test suite in `test-audio.cpp`
4. Document in this file
5. Submit PR with test files and updates

## Known Limitations

- Test audio must be 16kHz, mono, WAV format
- Fuzzy matching may accept slightly incorrect transcriptions
- Performance varies by CPU/GPU and model size
- Some tests may be environment-specific

---

**Happy Testing!** 🧪







