Local ASR for speech input
WHY Speech recognition was kept close to the application and hardware available in 2024.
TRADE-OFF / EFFECT CPU-local Faster Whisper preserved a clear local boundary while keeping transcription turn-based.
HISTORICAL 2024 PROTOTYPE
Voice-Operated AI & RAG Investment Assistant
A 2024 voice-operated AI prototype combining local speech recognition, retrieval, financial data, language-model reasoning and streamed speech output in an end-to-end interaction loop.
01
CONTEXT
Problem Class
How can a user ask investment-related questions by voice and receive contextual responses without manually navigating data sources and interfaces?
Engineering Challenge
The system had to coordinate microphone capture, local speech recognition, request handling, retrieval, financial-data access, model reasoning, response generation and speech synthesis across a web application.
02
STRUCTURE
Architecture
The 2024 implementation separated voice input, local transcription, retrieval and provider-backed reasoning into explicit stages. Speech recognition was turn-based; response audio used streaming text-to-speech.
01VOICE INPUTmicrophone capture
↓02LOCAL SPEECH RECOGNITIONFaster Whisper base on CPU
↓03REQUEST HANDLINGFlask application routing
↓04RETRIEVAL & FINANCIAL CONTEXTembeddings, FAISS, SQLite and data access
↓05LANGUAGE-MODEL REASONINGexplicit external provider route
↓06RESPONSE CONSTRUCTIONanswer assembly for the voice loop
↓07SPEECH OUTPUTstreaming OpenAI TTS response
03
OWNERSHIP
My Responsibility
Hands-on implementation across the end-to-end voice, retrieval and application coordination layers.
- 01ArchitectureDesigned the separation between voice capture, local ASR, retrieval, reasoning and speech response.
- 02ImplementationImplemented the application flow, Flask coordination and provider-facing request and response handling.
- 03Voice PipelineIntegrated microphone capture, local Faster Whisper transcription and the spoken response path.
- 04Retrieval / Data IntegrationImplemented embedding, FAISS, SQLite metadata and financial-data integration paths for contextual responses.
- 05System IntegrationConnected model reasoning, retrieval and streaming TTS into a working voice interaction prototype.
04
JUDGMENT
Critical Decisions
Retrieval outside the language model
WHY Financial and domain context should be assembled through an explicit retrieval path.
TRADE-OFF / EFFECT Embeddings, FAISS and SQLite made context access a separate system responsibility rather than a memory claim.
Separate speech pipeline stages
WHY Capture, transcription, retrieval, reasoning and synthesis had different constraints and failure modes.
TRADE-OFF / EFFECT The stages were easier to inspect and replace, but end-to-end latency accumulated across them.
External reasoning with local-first components
WHY The preserved implementation used an external provider for response generation while other components remained local or application-owned.
TRADE-OFF / EFFECT This was not a fully local system, but provider routing stayed explicit.
Modular component boundaries
WHY Retrieval, model and speech components were evolving during the historical prototype.
TRADE-OFF / EFFECT Modular boundaries supported experimentation without presenting every branch as one simultaneous production architecture.
05
AUTHORITY
Trust / Safety Boundaries
The system connects voice and financial context, but model output remains an interpretation layer rather than authoritative financial state.
Interpret transcribed requests and generate advisory responses from retrieved context.
Audio handling, request routing, retrieval operations, data integration and response pipeline coordination.
The user remains responsible for interpreting information and making any consequential decision.
Autonomous trading, capital deployment, brokerage authority and formal financial-advice compliance claims.
06
RESPONSE
Failure / Constraint → Engineering Response
Turn-based transcription
The main ASR path captured and wrote a complete spoken segment before transcription.
Bounded voice stage
The historical system kept transcription explicit. A modern rebuild would evaluate VAD and incremental ASR separately rather than rewriting the 2024 claim.
Heterogeneous retrieval experiments
Multiple embedding and vector-store paths existed across the preserved project history.
Separate core path from experiments
The public case study distinguishes the evidenced FAISS and SQLite core path from optional or experimental alternatives preserved in the project history.
No formal WER benchmark
Repeated live end-to-end conversations were preserved, but quantified WER results were not collected.
Qualify the evidence
The case study describes the implemented ASR path without claiming measured accuracy or high-volume production behavior.
07
EVIDENCE
Technology / Versions
Technology at time of build — 2024. The table distinguishes evidenced core implementation from historical context and qualifies versions only where the evidence requires it.
| Technology | Version / Model | Role | Evidence basis |
|---|---|---|---|
| Faster Whisper | 1.0.2 | Local ASR runtime | Evidenced core |
| Whisper | base | Primary speech model | Evidenced core |
| PyAudio | 0.2.14 | Microphone and audio I/O | Evidenced core |
| SpeechRecognition | 3.10.1 | Speech-input integration | Evidenced core |
| OpenAI Python | 1.30.1 | External provider integration | Evidenced core |
| OpenAI TTS | tts-1 | Streaming speech response | Evidenced core |
| Groq | llama3-70b-8192 | Response generation route | Evidenced core |
| Sentence Transformers | 3.3.1 | Embedding runtime | Evidenced core |
| all-MiniLM-L6-v2 | Model ID | Semantic embeddings | Evidenced core |
| FAISS CPU | 1.9.0.post1 | Vector similarity search | Evidenced core |
| SQLite | Exact version not captured | Retrieval metadata | Version qualified |
| Flask | 2.3.3 | Application backend | Evidenced core |
| flask-cors | 3.0.10 | Frontend/backend integration | Evidenced core |
| torch | 2.5.1 | Local ML runtime | Evidenced core |
| transformers | 4.47.0 | Transformer tooling | Evidenced core |
Historical experiments such as nomic-embed-text-v1.5, optional CLIP, Chroma, OpenAI embeddings, ChatOpenAI, ChatOllama and pyttsx3 are kept as architectural context rather than presented as one simultaneous core stack.
08
PROOF
Evidence
This public case study preserves the approved architecture, technology history, system boundaries, implementation decisions and historical constraints without linking the legacy source tree.
09
BOUNDARY
Scope Boundaries
End-to-end voice architecture, local ASR role, retrieval architecture, evidenced technologies and versions, provider separation and historical constraints.
Credentials, provider keys, private source paths, internal configuration, account details, private financial data and proprietary prompts or investment logic.
Formal WER, fine-tuning, custom model training, streaming ASR, telephony, production-scale concurrency, fully local operation, autonomous trading and air-gapped operation.