Field Notes: Patient Virtuel

The Real Person Behind the App

My wife is a dentist. She practiced for years in our home country, then we moved to Switzerland. When our son was born, she stepped back from her career to take care of him. Now that he is older, she is ready to return to the workforce. But in a new country and a new language.

Her French is intermediate. She learned it in high school and can hold a conversation, but she never learned the specific vocabulary of a dental hygiene consultation. In Switzerland, dental hygienists must communicate clearly, empathetically, and precisely with patients: explaining procedures, giving instructions, asking about sensitivity. Getting that language wrong is not just embarrassing. It affects patient trust.

The traditional approach would be a textbook or a tutor. A tutor is expensive and inflexible. A textbook cannot react to what you say. What she needed was a patient who could respond to her naturally, in real time, with the vocabulary and flow of a real Swiss dental appointment.

Why an LLM?

A scripted chatbot would not work. Every session needs to be different, or she would memorize the lines and lose the challenge of thinking on her feet. An LLM provides:

• Variability. The patient answers differently each time, within the same clinical scenario.
• Realism. The patient can react to sensitivity, ask questions about cost, or make small talk.
• Feedback. At the end of the session, the same model shifts from patient to tutor and produces a structured recap with grammar corrections, vocabulary, and scores.

One model, two roles. No second service needed.

Architecture

┌─────────────────────────────────────────────────┐
│  Browser (custom HTML/CSS/JS)                   │
│  ← @gradio/client ↔ gr.Server (FastAPI) →      │
├─────────────────────────────────────────────────┤
│  HF Space (CPU only)                            │
│                                                  │
│  src/server_app.py  ← API endpoints             │
│  src/core.py        ← session logic             │
│  src/tts_engine.py  ← piper-tts (local, fr_FR)  │
│  src/prompts.py     ← system prompt + phase     │
│  src/parse_feedback.py ← structured recap       │
│                                                  │
├──────────────────┬──────────────────────────────┤
│  Modal app 1     │  Modal app 2                  │
│  L4 GPU          │  T4 GPU                       │
│  llama.cpp       │  faster-whisper               │
│  Gemma 4 26B     │  large-v3-turbo               │
│  /warmup + /chat │  / (STT endpoint)             │
└──────────────────┴──────────────────────────────┘

Request flow (one turn)

1. User presses mic, browser records WAV, sends to gr.Server API.
2. process_turn() in core.py: a. Sends audio to Modal Whisper endpoint, returns transcript. b. Appends user message to chat history, sends to Modal Gemma 4 endpoint, returns reply. c. Strips stray markdown from LLM output, feeds to local piper-tts, returns WAV bytes.
3. Browser receives {chat, audio_url, state}, renders bubble, plays audio.

The split between Modal (GPU) and HF Space (CPU) comes from the free-tier constraints of HF Spaces: no GPU, limited RAM. Heavy compute (LLM inference, Whisper transcription) runs on Modal's serverless GPU tier. Piper TTS runs locally because it is lightweight (~50 MB model) and eliminates one network hop for audio latency.

Cost Control on Modal

I run two Modal apps: one with an L4 GPU for the LLM and one with a T4 GPU for Whisper. Both use Modal's serverless containers with a scaledown_window of 90 seconds. This means containers scale to zero when idle and I only pay for actual compute time.

Key cost decisions:

• container_idle_timeout=90. Containers shut down 90 seconds after the last request. No idle GPU cost.
• No always-on provisioned concurrency. The 25 second cold start on first request is acceptable. Paying for 24/7 GPU time would defeat the purpose of a hobby project.
• L4 vs. T4. The L4 ($0.30/hr on Modal) handles the 26B MoE LLM comfortably with 24 GB VRAM. The T4 ($0.20/hr) is overkill for Whisper but Whisper needs CUDA and the T4 is the cheapest Modal option with CUDA.
• Warmup strategy. Both endpoints have lightweight warmup handlers triggered on page load. This moves the cold start cost from the user's first turn to page load, which feels faster. The warmup itself costs a few seconds of GPU time per session.

Total Modal cost for development and testing: roughly $15 over two months. For production use (a few sessions per week), I expect under $5/month.

Component Choices

LLM: Gemma 4 26B-A4B via llama.cpp on Modal L4

• Why Gemma 4. Strong French performance out of the box, Apache 2.0 license, and the 26B-A4B MoE variant keeps inference fast enough for interactive use on a single L4.
• Why llama.cpp. The llama-cpp-python bindings with flash attention and CUDA offloading give excellent throughput on the L4's 24 GB VRAM. The MoE architecture loads ~13B active parameters, leaving headroom for the KV cache.
• Key parameters. n_batch=4096, n_ubatch=1024, temperature=0.85, repeat_penalty=1.1. Tuned for creative but coherent roleplay.
• Cold start. First request after idle spins up the container (~10s) and loads the model into VRAM (~15s). A /warmup endpoint triggers this on page load so the user does not wait.

STT: faster-whisper-large-v3-turbo on Modal T4

The turbo variant of Whisper large-v3 offers near-parity accuracy at roughly half the compute. Running on a T4, it transcribes a 10-second audio clip in under 2 seconds. Dental French vocabulary (détartrage, aéropolissage, bavette, etc.) is handled accurately by Whisper's multilingual training.

TTS: piper-tts (local, fr_FR-siwis-medium)

The biggest pivot in the project. I originally used Microsoft's edge-tts cloud API on the free tier. I switched to piper-tts to claim the Off the Grid badge with zero third-party cloud APIs.

Piper TTS runs entirely on the HF Space CPU. The fr_FR-siwis-medium voice (Sie bavarde, Sie ne tait) is a neural model that produces natural French prosody for ~50 MB.

One gotcha: Piper v1.x changed synthesize() to return a generator of AudioChunk objects instead of (audio_bytes, sample_rate). You access the audio via chunk.audio_int16_bytes in a loop.

Frontend: gr.Server with custom HTML/CSS/JS

Gradio's default UI components are great for demos but inflexible for conversational UIs. I wanted a chat interface that looks like a messaging app, not a Jupyter notebook.

gr.Server (formerly gradio.Server) is a thin wrapper around FastAPI that registers Gradio-compatible API routes. You get the full @gradio/client JS library for calling those routes, plus total freedom over the HTML.

The frontend is vanilla: no React, no build step. The custom_index.html is served directly by FastAPI and uses import { Client } from "https://cdn.jsdelivr.net/npm/@gradio/client/+esm" for API calls.

Lessons Learned

1. gr.Server is Gradio's secret weapon

Most people know Gradio for gr.Interface and gr.Blocks. Those auto-generate UI. gr.Server strips away all UI and exposes Gradio as a pure API framework with FastAPI underneath. You can register endpoints with @app.api(name="..."), serve custom HTML with @app.get("/"), and call everything from JavaScript using the same @gradio/client library.

This was the single biggest discovery of the project. You get Gradio's easy API serialization, HF Spaces auto-deployment, and @gradio/client streaming without being locked into the default UI.

2. Piper TTS v1.x generator API

The tts-engine rewrite from edge-tts to piper-tts hit an API change between versions. The docs showed voice.synthesize(text)[0], but at v1.2 synthesize() returns a generator. Debugging this required reading the actual source:

for chunk in voice.synthesize(text):
    frames.extend(chunk.audio_int16_bytes)

Lesson: always check the return type of the version you actually have installed.

3. Modal cold starts need a warmup strategy

Modal containers scale to zero after a configurable scaledown_window. I use 90 seconds. The first request after idle triggers a cold start: container boot plus model load. For a 26B model, that is roughly 25 seconds of wait time.

My solution: fire a warmup request on page load. Both the STT and LLM endpoints have lightweight warmup handlers that trigger model loading without doing meaningful work.

• STT warmup: sends 44 bytes of silence (minimal WAV header + 1 sample), Whisper processes and returns.
• LLM warmup: calls POST /warmup, validates token, calls _load_llm(), returns {"status": "ready"}.

The warmups are fire-and-forget (daemon threads) so they do not block the frontend. By the time the user presses the mic for the first turn, both containers are warm.

4. Prompt engineering for dual roleplay + feedback

The same model serves two roles: patient during the session and French coach after. The trick is the phase switch.

During roleplay, the model stays in character because the system prompt tells it to. When the user says "Fin de la séance" (detected by a regex), I append a PHASE_SWITCH_REMINDER message:

Reminder: you are now a French coach. Do NOT continue the patient roleplay.
Respond with exactly 2 spoken sentences in French (encouragement + priority),
then output --- on its own line, then provide the written recap with
Points forts, Citation/Correction/Pourquoi, Vocabulaire dentaire utile,
Priorité, and Bilan scores. Use plain text only.

Without this reminder, the model often stayed in character and produced a patient-voice recap. That was both confusing and unusable for the learner.

4b. Making the patient feel real

Early versions of the prompt produced a patient who agreed with everything and never initiated. Every response was "D'accord" — technically correct but useless for practicing patient communication. Real patients ask questions, express concerns, and react to sensations.

The fix was behavioral directives with probabilistic targets: - Patient must have mild dental anxiety and opinions about cost/procedures - Patient must ask questions, express concerns, or react unprompted in roughly 1 in 3 turns - Response length must vary: 1 word during active work, 2-3 sentences when initiating - Few-shot examples in the prompt show the desired style

The key insight: listing behaviors ("ask questions") is not enough. The model needs concrete examples and probability targets to avoid collapsing into a single repetitive pattern. Adding a safety valve instruction ("if the conversation stalls, de-escalate and let the hygienist lead") prevents the patient from becoming uncooperative.

5. Edge cases that matter

• STT returns empty or too-short text (< 2 chars). Show "Parlez plus fort ou plus longtemps" instead of sending garbage to the LLM.
• LLM outputs markdown despite system instruction to use plain text. strip_markdown() strips bold, bullets, and code fences defensively.
• Termination detection. Regex matches "fin de la séance", "session terminée", "on a terminé", "c'est fini" (and variants with missing accents).
• Context window overflow. _trim() in llm_engine.py truncates early turns to keep the conversation within the model's context limit.

6. What did not work

• MTP speculative decoding. I tried using a draft model (a smaller Gemma variant) with speculative decoding to reduce latency. The integration with llama.cpp's MTP path was unstable and produced inconsistent outputs. Removed entirely.
• edge-tts as initial TTS choice. Worked well but depended on Microsoft's cloud API. The voices were good (fr-CH-ArianeNeural), but using a cloud TTS meant the app could not claim the Off the Grid badge. Piper delivered equivalent quality with full local control.
• React/TypeScript frontend. The first frontend was built with React, Vite, and the Gemini Live API. It was overengineered for what is essentially a single-page chat app. Switching to vanilla HTML/CSS/JS eliminated the build step, reduced the payload, and made the frontend trivially deployable as a single file served by FastAPI.

7. Previous llama.cpp experience made Modal adoption easier

This was my first time working with a service like Modal. What helped me get started quickly was the hours I already spent tuning llama.cpp at home on my old Quadro RTX 3000 with 6 GB VRAM. I had already gone through the pain of getting decent versions of Qwen 3.6 and Gemma 4 to run within that constraint: quantizing, tweaking batch sizes, managing KV cache, understanding context limits. By the time I moved to Modal with an L4, I knew exactly what parameters to adjust and what to expect from the runtime. The home setup taught me the fundamentals, Modal just gave me more VRAM.

Quantitative Results

Measured from production logs (HF Space CPU + Modal L4/T4):

Cold start adds 14–20 s on first turn (container boot + model load into VRAM). Warmup fires on page load, reducing first-turn latency to match warm times. After warmup, subsequent turns average under 7 s end to end.

Result

The app is used by my wife frequently. She reports that the variability in patient responses keeps each session fresh. The structured feedback at the end helps her focus on specific grammar patterns she struggles with, like subjonctif after "il faut que" and preposition usage with "a" versus "de" for dental instruments.

Most importantly, she feels more confident walking into a real consultation. I consider this as the real achievement.

Technical Stack Summary