# AI Conversation Implementation Tutorial

> Two working approaches to AI voice conversation compared side by side: sequential record-then-respond, and the OpenAI Realtime API with streaming. Covers the architecture of each, a comparison matrix, and setup guides for both.

Source: https://chanmeng.org/blog/ai-conversation-implementation
Author: Chan Meng — https://chanmeng.org
Published: 2025-11-24
Tags: AI Engineering

---

*Originally published on LinkedIn, 24 November 2025. Republished here with light editing.*

> A practical guide for implementing two different AI conversation approaches in ESOL learning platforms

## Overview

This project implements two distinct approaches for AI-powered voice conversations:

Sequential Recording (/practice/nzcel/conversation)

- Turn-based conversation with manual recording control
- Uses separate API calls for transcription, response generation, and TTS
- Best for structured practice scenarios

Realtime Streaming (/speaking)

- Natural, free-flowing conversation with automatic turn detection
- Uses OpenAI Realtime API with WebRTC for bidirectional streaming
- Best for realistic speaking practice

---

## Implementation #1: Sequential Voice Recording

## Architecture

```
User Speaks → Record Audio → Stop Recording → Transcribe (Whisper)
    ↓
Display Transcript → Generate Response (GPT-4) → Convert to Speech (TTS)
    ↓
Play Audio → Wait for User → Repeat
```

## Core Files

```
src/
├── app/(main)/practice/nzcel/conversation/page.tsx
├── components/conversation/realtime-conversation.tsx
├── hooks/use-voice-recorder.ts
└── app/api/openai/
    ├── transcribe/route.ts    # Whisper API
    ├── conversation/route.ts  # GPT-4 Chat
    └── tts/route.ts          # Text-to-Speech
```

## Key Technologies

- Audio Recording: MediaRecorder API (audio/webm)
- Transcription: OpenAI Whisper (whisper-1)
- AI Response: GPT-4 Turbo (gpt-4-turbo-preview)
- Text-to-Speech: OpenAI TTS (tts-1)

## Implementation Pattern

1\. Voice Recording Hook (use-voice-recorder.ts)

```
export function useVoiceRecorder() {
  const [state, setState] = useState<VoiceRecordingState>({
    isRecording: false,
    audioBlob: null,
    transcription: null,
  });

  const startRecording = async () => {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    const mediaRecorder = new MediaRecorder(stream, { mimeType: "audio/webm" });

    mediaRecorder.ondataavailable = (event) => {
      chunks.push(event.data);
    };

    mediaRecorder.onstop = () => {
      const blob = new Blob(chunks, { type: "audio/webm" });
      setState(prev => ({ ...prev, audioBlob: blob }));
    };

    mediaRecorder.start();
  };

  return { state, startRecording, stopRecording, transcribe, assess };
}
```

2\. Conversation Flow (realtime-conversation.tsx)

```js
const processUserSpeech = async () => {
  // 1. Transcribe audio
  const formData = new FormData();
  formData.append("audio", new File([audioBlob], "recording.webm"));

  const transcribeResponse = await fetch("/api/openai/transcribe", {
    method: "POST",
    body: formData,
  });
  const { text } = await transcribeResponse.json();

  // 2. Generate AI response
  const aiResponse = await fetch("/api/openai/conversation", {
    method: "POST",
    body: JSON.stringify({
      messages: [
        { role: "system", content: scenarioContext },
        ...conversationHistory,
        { role: "user", content: text }
      ],
    }),
  });
  const { response } = await aiResponse.json();

  // 3. Convert to speech
  const ttsResponse = await fetch("/api/openai/tts", {
    method: "POST",
    body: JSON.stringify({ text: response }),
  });
  const audioBlob = await ttsResponse.blob();

  // 4. Play audio
  const audio = new Audio(URL.createObjectURL(audioBlob));
  await audio.play();
};
```

3\. API Routes

```js
// Transcription (transcribe/route.ts)
export async function POST(request: NextRequest) {
  const formData = await request.formData();
  const audioFile = formData.get("audio") as File;

  const transcription = await openai.audio.transcriptions.create({
    file: audioFile,
    model: "whisper-1",
    language: "en",
  });

  return NextResponse.json({ text: transcription.text });
}

// Conversation (conversation/route.ts)
export async function POST(request: NextRequest) {
  const { messages } = await request.json();

  const completion = await openai.chat.completions.create({
    model: "gpt-4-turbo-preview",
    messages,
    temperature: 0.8,
    max_tokens: 300,
  });

  return NextResponse.json({ response: completion.choices[0].message.content });
}
```

---

## Implementation #2: OpenAI Realtime API

## Architecture

```
User Speaks (continuously) ←→ WebRTC Connection ←→ OpenAI Realtime API
         ↓                           ↓                        ↓
  VAD Detection              Auto-transcription        AI Response (voice)
         ↓                           ↓                        ↓
    Turn Detection  →  Save to Database  ←  Stream Audio Back
```

## Core Files

```
src/
├── app/(main)/speaking/page.tsx
├── components/speaking/ai-coach.tsx
└── app/api/openai/realtime-client-secret/route.ts
```

## Key Technologies

- SDK: @openai/agents/realtime
- Transport: WebRTC with bidirectional streaming
- Voice Activity Detection: Server-side VAD (threshold: 0.5, silence: 500ms)
- Transcription: Built-in (gpt-4o-mini-transcribe)
- Response: Real-time audio streaming

## Implementation Pattern

1\. Session Initialization

```js
import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const startSession = async () => {
  // Get ephemeral client secret
  const response = await fetch("/api/openai/realtime-client-secret", {
    method: "POST",
    body: JSON.stringify({
      voice: "verse",
      instructions: ESOL_COACH_INSTRUCTIONS,
    }),
  });
  const { clientSecret } = await response.json();

  // Create agent and session
  const agent = new RealtimeAgent({
    name: "ESOL Coach",
    instructions: ESOL_COACH_INSTRUCTIONS,
  });

  const session = new RealtimeSession(agent, {
    transport: "webrtc",
    config: {
      audio: {
        input: {
          transcription: { model: "gpt-4o-mini-transcribe" },
        },
      },
      turn_detection: {
        type: "server_vad",
        threshold: 0.5,
        silence_duration_ms: 500,
      },
    },
  });

  // Connect to OpenAI
  await session.connect({ apiKey: clientSecret });

  // Setup event handlers
  setupEventHandlers(session);
};
```

2\. Event Handlers

```js
const setupEventHandlers = (session: RealtimeSession) => {
  // Handle message history updates
  session.on("history_updated", (history) => {
    const messages = history
      .filter(item => item.type === "message")
      .map(item => ({
        role: item.role === "user" ? "user" : "assistant",
        content: extractTranscript(item.content),
        timestamp: new Date(),
      }));

    setMessages(messages);
    saveToDatabase(messages);
  });

  // Handle voice activity detection
  session.on("transport_event", (event) => {
    if (event.type === "input_audio_buffer.speech_started") {
      setIsSpeaking(true);
      startRecordingUserAudio();
    } else if (event.type === "input_audio_buffer.speech_stopped") {
      setIsSpeaking(false);
      stopRecordingUserAudio();
    }
  });

  // Handle errors
  session.on("error", (error) => {
    console.error("Session error:", error);
    toast.error("Conversation error occurred");
  });
};
```

3\. Client Secret Generation

```js
// realtime-client-secret/route.ts
export async function POST(request: NextRequest) {
  const { voice, instructions } = await request.json();

  const sessionConfig = {
    session: {
      type: "realtime" as const,
      model: "gpt-realtime",
      audio: { output: { voice } },
      instructions,
    },
  };

  const response = await fetch(
    "https://api.openai.com/v1/realtime/client_secrets",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(sessionConfig),
    }
  );

  const data = await response.json();
  return NextResponse.json({
    clientSecret: data.value,
    expiresAt: data.expires_at,
  });
}
```

4\. Audio Recording Synchronization

```js
// Track user audio for database persistence
const mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });
const userAudioChunks = {};
const recordingToMessageMap = new Map();

session.on("transport_event", (event) => {
  if (event.type === "input_audio_buffer.speech_started") {
    const recordingIndex = userRecordingCounter++;
    const currentUserMessageCount = session.history
      .filter(item => item.type === "message" && item.role === "user").length;

    recordingToMessageMap.set(recordingIndex, currentUserMessageCount);

    mediaRecorder.ondataavailable = (e) => {
      if (!userAudioChunks[recordingIndex]) {
        userAudioChunks[recordingIndex] = [];
      }
      userAudioChunks[recordingIndex].push(e.data);
    };

    mediaRecorder.start();
  } else if (event.type === "input_audio_buffer.speech_stopped") {
    mediaRecorder.stop();
  }
});
```

---

## Comparison Matrix

| Feature                   | Sequential Recording            | Realtime Streaming                                 |
| ------------------------- | ------------------------------- | -------------------------------------------------- |
| Latency                   | 3-5 seconds per turn            | <1 second                                          |
| User Experience           | Manual button clicks            | Natural conversation flow                          |
| Implementation Complexity | Medium (3 API calls)            | High (WebRTC + events)                             |
| API Usage                 | 3 separate calls per turn       | Single persistent connection                       |
| Cost Model                | Pay per API call                | Pay per minute ($0.06/min input, $0.24/min output) |
| Browser Support           | Wide (MediaRecorder API)        | Modern browsers (WebRTC)                           |
| Turn Detection            | Manual (user clicks stop)       | Automatic (Voice Activity Detection)               |
| Audio Format              | WebM → uploaded                 | PCM streaming                                      |
| Conversation Style        | Turn-based (structured)         | Free-flowing (natural)                             |
| Setup Time                | \~1 hour                        | \~3-4 hours                                        |
| Debugging                 | Easy (inspect each step)        | Moderate (event-driven)                            |
| Interruption Handling     | Not supported                   | Built-in (barge-in)                                |
| Best For                  | Practice exercises, assessments | Conversational fluency, immersion                  |

---

## Quick Implementation Guide

## Prerequisites

```bash
npm install openai @openai/agents sonner
```

## Environment Variables

```
OPENAI_API_KEY=sk-...
```

## Sequential Recording Setup (30 mins)

1. Create voice recorder hook
2. Create API routes
3. Build conversation component
4. Test workflow

## Realtime Streaming Setup (2 hours)

1. Install Realtime SDK
2. Create client secret endpoint
3. Build coach component
4. Implement event handlers
5. Add audio recording (optional for persistence)
6. Test real-time flow

---

## When to Use Each Approach

## Choose Sequential Recording When:

✅ Budget is a primary concern

- Pay only for API calls used
- Predictable costs per conversation

✅ You need full control

- Explicit start/stop recording
- Validate each step separately
- Custom processing between steps

✅ Implementing structured practice

- Question-answer format
- Assessment-focused exercises
- Clear turn boundaries

✅ Supporting older browsers

- Requires only MediaRecorder API
- Broader compatibility

✅ Debugging is critical

- Easy to inspect each pipeline stage
- Clear error boundaries

## Choose Realtime Streaming When:

✅ User experience is paramount

- Natural, free-flowing conversations
- Minimal latency (<1 second)
- Interrupt and respond (barge-in)

✅ Simulating real conversations

- Speaking fluency practice
- Interview preparation
- Casual dialogue

✅ Users have modern browsers

- WebRTC support required
- Chrome, Edge, Safari (recent versions)

✅ Volume justifies setup time

- Higher upfront complexity
- Better UX for frequent use

✅ You want built-in features

- Automatic transcription
- Voice activity detection
- Turn management included

---

## Common Pitfalls amp; Solutions

## Sequential Recording

Issue: Audio blob is null after stopping recording

```
// ❌ Wrong: Process immediately
stopRecording();
processAudio(state.audioBlob); // blob is still null!

// ✅ Correct: Use effect to wait for blob
useEffect(() => {
  if (!state.isRecording && state.audioBlob) {
    processAudio(state.audioBlob);
  }
}, [state.audioBlob, state.isRecording]);
```

Issue: Microphone stays active after component unmounts

```
// ✅ Always cleanup
useEffect(() => {
  return () => {
    if (streamRef.current) {
      streamRef.current.getTracks().forEach(track => track.stop());
    }
  };
}, []);
```

## Realtime Streaming

Issue: Message and audio recording out of sync

```js
// ✅ Map recording index to expected message index
const recordingToMessageMap = new Map();

session.on("transport_event", (event) => {
  if (event.type === "input_audio_buffer.speech_started") {
    const recordingIndex = userRecordingCounter++;
    const expectedMessageIndex = getCurrentUserMessageCount();
    recordingToMessageMap.set(recordingIndex, expectedMessageIndex);
  }
});
```

Issue: Session not cleaning up properly

```js
// ✅ Comprehensive cleanup
const stopSession = async () => {
  // Stop all recordings
  if (mediaRecorder?.state === "recording") {
    mediaRecorder.stop();
  }

  // Stop media streams
  mediaRecorder?.stream?.getTracks().forEach(track => track.stop());

  // Close session
  session?.close();

  // Clear refs
  sessionRef.current = null;
  mediaRecorderRef.current = null;
};
```

---

## Best Practices

## Security

```
// ✅ Never expose API keys in client code
// Use API routes to proxy requests
const response = await fetch("/api/openai/realtime-client-secret", {
  method: "POST"
});

// ❌ Never do this
const session = new RealtimeSession(agent, {
  apiKey: process.env.OPENAI_API_KEY // Exposed to client!
});
```

## Performance

```
// ✅ Cache TTS audio for repeated questions
const getQuestionAudio = async (questionId: string, text: string) => {
  // Check cache first
  const cached = await db.query.questionAudioCache.findFirst({
    where: eq(schema.questionAudioCache.questionId, questionId)
  });

  if (cached) return cached.audioUrl; // 90% cost reduction

  // Generate and cache
  const audio = await generateTTS(text);
  await cacheAudio(questionId, audio);
  return audio;
};
```

## User Experience

```
// ✅ Provide clear feedback at each stage
toast.info("Transcribing your response...");
toast.info("AI is thinking...");
toast.success("Ready for your next response!");

// ✅ Show recording indicator
{state.isRecording && (
  <div className="flex items-center gap-2">
    <div className="h-3 w-3 rounded-full bg-red-500 animate-pulse" />
    <span>Recording... ({state.recordingDuration}s)</span>
  </div>
)}
```

## Error Handling

```
// ✅ Graceful degradation
try {
  const transcription = await transcribe();
  const response = await generateResponse(transcription);
  await speakResponse(response);
} catch (error) {
  if (error.message.includes("microphone")) {
    toast.error("Microphone access denied. Please check permissions.");
  } else if (error.message.includes("API")) {
    toast.error("Service temporarily unavailable. Please try again.");
  } else {
    toast.error("An error occurred. Please refresh the page.");
  }
}
```

---

## Advanced Topics

## Custom AI Instructions

```
const INSTRUCTIONS = `You are an ESOL speaking coach.

- Adapt to CEFR level: ${userLevel}
- Focus areas: ${focusSkills.join(", ")}
- Provide feedback on: fluency, accuracy, coherence
- Keep responses under 20 seconds
- Encourage learner after each turn`;
```

## Audio Persistence Strategy

```
// Sequential: Simple - save after each turn
await saveUserRecording({
  sessionId,
  audioBlob,
  transcription,
  timestamp: new Date(),
});

// Realtime: Complex - sync with message history
// Map recording index → message index → upload when message saved
```

## Session Analytics

```ts
interface SessionMetrics {
  duration: number;           // Total session time
  turnCount: number;          // Number of exchanges
  userSpeakingTime: number;   // Active speaking time
  avgResponseLatency: number; // Sequential only
  completionRate: number;     // % of target turns reached
}
```

---

## Conclusion

Both approaches have their place in ESOL learning platforms:

Sequential Recording: Reliable, cost-effective, and easier to implement. Perfect for structured practice, assessments, and budget-conscious applications.

Realtime Streaming: Premium experience with natural conversation flow. Ideal for advanced learners, fluency practice, and applications where UX justifies the complexity.

Consider implementing both: use Sequential Recording for structured exercises and Realtime Streaming for conversational practice. This hybrid approach provides flexibility for different learning contexts and user needs.

---

## Resources

## Project Files

Sequential Implementation:

- src/components/conversation/realtime-conversation.tsx - Main component
- src/hooks/use-voice-recorder.ts - Recording logic
- src/app/api/openai/transcribe/route.ts - Whisper API
- src/app/api/openai/conversation/route.ts - GPT-4 API
- src/app/api/openai/tts/route.ts - Text-to-speech API

Realtime Implementation:

- src/components/speaking/ai-coach.tsx - Main component
- src/app/api/openai/realtime-client-secret/route.ts - Authentication

## External Documentation

- [OpenAI Realtime API Guide](https://platform.openai.com/docs/guides/realtime)
- [OpenAI Whisper API](https://platform.openai.com/docs/guides/speech-to-text)
- [MediaRecorder API (MDN)](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder)
- [WebRTC API (MDN)](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API)