AI Conversation Implementation Tutorial

Originally published on LinkedIn, 24 November 2025. Republished here with light editing.

A practical guide for implementing two different AI conversation approaches in ESOL learning platforms

Table of Contents#

1. Overview
2. Implementation #1: Sequential Voice Recording
3. Implementation #2: OpenAI Realtime API
4. Comparison Matrix
5. Quick Implementation Guide
6. When to Use Each Approach

Overview#

This project implements two distinct approaches for AI-powered voice conversations:

Sequential Recording (/practice/nzcel/conversation)

• Turn-based conversation with manual recording control
• Uses separate API calls for transcription, response generation, and TTS
• Best for structured practice scenarios

Realtime Streaming (/speaking)

• Natural, free-flowing conversation with automatic turn detection
• Uses OpenAI Realtime API with WebRTC for bidirectional streaming
• Best for realistic speaking practice

Implementation #1: Sequential Voice Recording#

Architecture#

User Speaks → Record Audio → Stop Recording → Transcribe (Whisper)
    ↓
Display Transcript → Generate Response (GPT-4) → Convert to Speech (TTS)
    ↓
Play Audio → Wait for User → Repeat

Core Files#

src/
├── app/(main)/practice/nzcel/conversation/page.tsx
├── components/conversation/realtime-conversation.tsx
├── hooks/use-voice-recorder.ts
└── app/api/openai/
    ├── transcribe/route.ts    # Whisper API
    ├── conversation/route.ts  # GPT-4 Chat
    └── tts/route.ts          # Text-to-Speech

Key Technologies#

• Audio Recording: MediaRecorder API (audio/webm)
• Transcription: OpenAI Whisper (whisper-1)
• AI Response: GPT-4 Turbo (gpt-4-turbo-preview)
• Text-to-Speech: OpenAI TTS (tts-1)

Implementation Pattern#

1. Voice Recording Hook (use-voice-recorder.ts)

export function useVoiceRecorder() {
  const [state, setState] = useState<VoiceRecordingState>({
    isRecording: false,
    audioBlob: null,
    transcription: null,
  });

  const startRecording = async () => {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    const mediaRecorder = new MediaRecorder(stream, { mimeType: "audio/webm" });

    mediaRecorder.ondataavailable = (event) => {
      chunks.push(event.data);
    };

    mediaRecorder.onstop = () => {
      const blob = new Blob(chunks, { type: "audio/webm" });
      setState(prev => ({ ...prev, audioBlob: blob }));
    };

    mediaRecorder.start();
  };

  return { state, startRecording, stopRecording, transcribe, assess };
}

2. Conversation Flow (realtime-conversation.tsx)

const processUserSpeech = async () => {
  // 1. Transcribe audio
  const formData = new FormData();
  formData.append("audio", new File([audioBlob], "recording.webm"));
 
  const transcribeResponse = await fetch("/api/openai/transcribe", {
    method: "POST",
    body: formData,
  });
  const { text } = await transcribeResponse.json();
 
  // 2. Generate AI response
  const aiResponse = await fetch("/api/openai/conversation", {
    method: "POST",
    body: JSON.stringify({
      messages: [
        { role: "system", content: scenarioContext },
        ...conversationHistory,
        { role: "user", content: text }
      ],
    }),
  });
  const { response } = await aiResponse.json();
 
  // 3. Convert to speech
  const ttsResponse = await fetch("/api/openai/tts", {
    method: "POST",
    body: JSON.stringify({ text: response }),
  });
  const audioBlob = await ttsResponse.blob();
 
  // 4. Play audio
  const audio = new Audio(URL.createObjectURL(audioBlob));
  await audio.play();
};

3. API Routes

// Transcription (transcribe/route.ts)
export async function POST(request: NextRequest) {
  const formData = await request.formData();
  const audioFile = formData.get("audio") as File;
 
  const transcription = await openai.audio.transcriptions.create({
    file: audioFile,
    model: "whisper-1",
    language: "en",
  });
 
  return NextResponse.json({ text: transcription.text });
}
 
// Conversation (conversation/route.ts)
export async function POST(request: NextRequest) {
  const { messages } = await request.json();
 
  const completion = await openai.chat.completions.create({
    model: "gpt-4-turbo-preview",
    messages,
    temperature: 0.8,
    max_tokens: 300,
  });
 
  return NextResponse.json({ response: completion.choices[0].message.content });
}

Implementation #2: OpenAI Realtime API#

Architecture#

User Speaks (continuously) ←→ WebRTC Connection ←→ OpenAI Realtime API
         ↓                           ↓                        ↓
  VAD Detection              Auto-transcription        AI Response (voice)
         ↓                           ↓                        ↓
    Turn Detection  →  Save to Database  ←  Stream Audio Back

Core Files#

src/
├── app/(main)/speaking/page.tsx
├── components/speaking/ai-coach.tsx
└── app/api/openai/realtime-client-secret/route.ts

Key Technologies#

• SDK: @openai/agents/realtime
• Transport: WebRTC with bidirectional streaming
• Voice Activity Detection: Server-side VAD (threshold: 0.5, silence: 500ms)
• Transcription: Built-in (gpt-4o-mini-transcribe)
• Response: Real-time audio streaming

Implementation Pattern#

1. Session Initialization

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";
 
const startSession = async () => {
  // Get ephemeral client secret
  const response = await fetch("/api/openai/realtime-client-secret", {
    method: "POST",
    body: JSON.stringify({
      voice: "verse",
      instructions: ESOL_COACH_INSTRUCTIONS,
    }),
  });
  const { clientSecret } = await response.json();
 
  // Create agent and session
  const agent = new RealtimeAgent({
    name: "ESOL Coach",
    instructions: ESOL_COACH_INSTRUCTIONS,
  });
 
  const session = new RealtimeSession(agent, {
    transport: "webrtc",
    config: {
      audio: {
        input: {
          transcription: { model: "gpt-4o-mini-transcribe" },
        },
      },
      turn_detection: {
        type: "server_vad",
        threshold: 0.5,
        silence_duration_ms: 500,
      },
    },
  });
 
  // Connect to OpenAI
  await session.connect({ apiKey: clientSecret });
 
  // Setup event handlers
  setupEventHandlers(session);
};

2. Event Handlers

const setupEventHandlers = (session: RealtimeSession) => {
  // Handle message history updates
  session.on("history_updated", (history) => {
    const messages = history
      .filter(item => item.type === "message")
      .map(item => ({
        role: item.role === "user" ? "user" : "assistant",
        content: extractTranscript(item.content),
        timestamp: new Date(),
      }));
 
    setMessages(messages);
    saveToDatabase(messages);
  });
 
  // Handle voice activity detection
  session.on("transport_event", (event) => {
    if (event.type === "input_audio_buffer.speech_started") {
      setIsSpeaking(true);
      startRecordingUserAudio();
    } else if (event.type === "input_audio_buffer.speech_stopped") {
      setIsSpeaking(false);
      stopRecordingUserAudio();
    }
  });
 
  // Handle errors
  session.on("error", (error) => {
    console.error("Session error:", error);
    toast.error("Conversation error occurred");
  });
};

3. Client Secret Generation

// realtime-client-secret/route.ts
export async function POST(request: NextRequest) {
  const { voice, instructions } = await request.json();
 
  const sessionConfig = {
    session: {
      type: "realtime" as const,
      model: "gpt-realtime",
      audio: { output: { voice } },
      instructions,
    },
  };
 
  const response = await fetch(
    "https://api.openai.com/v1/realtime/client_secrets",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(sessionConfig),
    }
  );
 
  const data = await response.json();
  return NextResponse.json({
    clientSecret: data.value,
    expiresAt: data.expires_at,
  });
}

4. Audio Recording Synchronization

// Track user audio for database persistence
const mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });
const userAudioChunks = {};
const recordingToMessageMap = new Map();
 
session.on("transport_event", (event) => {
  if (event.type === "input_audio_buffer.speech_started") {
    const recordingIndex = userRecordingCounter++;
    const currentUserMessageCount = session.history
      .filter(item => item.type === "message" && item.role === "user").length;
 
    recordingToMessageMap.set(recordingIndex, currentUserMessageCount);
 
    mediaRecorder.ondataavailable = (e) => {
      if (!userAudioChunks[recordingIndex]) {
        userAudioChunks[recordingIndex] = [];
      }
      userAudioChunks[recordingIndex].push(e.data);
    };
 
    mediaRecorder.start();
  } else if (event.type === "input_audio_buffer.speech_stopped") {
    mediaRecorder.stop();
  }
});

Comparison Matrix#

Quick Implementation Guide#

Prerequisites#

npm install openai @openai/agents sonner

Environment Variables#

OPENAI_API_KEY=sk-...

Sequential Recording Setup (30 mins)#

1. Create voice recorder hook
2. Create API routes
3. Build conversation component
4. Test workflow

Realtime Streaming Setup (2 hours)#

1. Install Realtime SDK
2. Create client secret endpoint
3. Build coach component
4. Implement event handlers
5. Add audio recording (optional for persistence)
6. Test real-time flow

When to Use Each Approach#

Choose Sequential Recording When:#

✅ Budget is a primary concern

• Pay only for API calls used
• Predictable costs per conversation

✅ You need full control

• Explicit start/stop recording
• Validate each step separately
• Custom processing between steps

✅ Implementing structured practice

• Question-answer format
• Assessment-focused exercises
• Clear turn boundaries

✅ Supporting older browsers

• Requires only MediaRecorder API
• Broader compatibility

✅ Debugging is critical

• Easy to inspect each pipeline stage
• Clear error boundaries

Choose Realtime Streaming When:#

✅ User experience is paramount

• Natural, free-flowing conversations
• Minimal latency (<1 second)
• Interrupt and respond (barge-in)

✅ Simulating real conversations

• Speaking fluency practice
• Interview preparation
• Casual dialogue

✅ Users have modern browsers

• WebRTC support required
• Chrome, Edge, Safari (recent versions)

✅ Volume justifies setup time

• Higher upfront complexity
• Better UX for frequent use

✅ You want built-in features

• Automatic transcription
• Voice activity detection
• Turn management included

Common Pitfalls amp; Solutions#

Sequential Recording#

Issue: Audio blob is null after stopping recording

// ❌ Wrong: Process immediately
stopRecording();
processAudio(state.audioBlob); // blob is still null!

// ✅ Correct: Use effect to wait for blob
useEffect(() => {
  if (!state.isRecording && state.audioBlob) {
    processAudio(state.audioBlob);
  }
}, [state.audioBlob, state.isRecording]);

Issue: Microphone stays active after component unmounts

// ✅ Always cleanup
useEffect(() => {
  return () => {
    if (streamRef.current) {
      streamRef.current.getTracks().forEach(track => track.stop());
    }
  };
}, []);

Realtime Streaming#

Issue: Message and audio recording out of sync

// ✅ Map recording index to expected message index
const recordingToMessageMap = new Map();
 
session.on("transport_event", (event) => {
  if (event.type === "input_audio_buffer.speech_started") {
    const recordingIndex = userRecordingCounter++;
    const expectedMessageIndex = getCurrentUserMessageCount();
    recordingToMessageMap.set(recordingIndex, expectedMessageIndex);
  }
});

Issue: Session not cleaning up properly

// ✅ Comprehensive cleanup
const stopSession = async () => {
  // Stop all recordings
  if (mediaRecorder?.state === "recording") {
    mediaRecorder.stop();
  }
 
  // Stop media streams
  mediaRecorder?.stream?.getTracks().forEach(track => track.stop());
 
  // Close session
  session?.close();
 
  // Clear refs
  sessionRef.current = null;
  mediaRecorderRef.current = null;
};

Best Practices#

Security#

// ✅ Never expose API keys in client code
// Use API routes to proxy requests
const response = await fetch("/api/openai/realtime-client-secret", {
  method: "POST"
});

// ❌ Never do this
const session = new RealtimeSession(agent, {
  apiKey: process.env.OPENAI_API_KEY // Exposed to client!
});

Performance#

// ✅ Cache TTS audio for repeated questions
const getQuestionAudio = async (questionId: string, text: string) => {
  // Check cache first
  const cached = await db.query.questionAudioCache.findFirst({
    where: eq(schema.questionAudioCache.questionId, questionId)
  });

  if (cached) return cached.audioUrl; // 90% cost reduction

  // Generate and cache
  const audio = await generateTTS(text);
  await cacheAudio(questionId, audio);
  return audio;
};

User Experience#

// ✅ Provide clear feedback at each stage
toast.info("Transcribing your response...");
toast.info("AI is thinking...");
toast.success("Ready for your next response!");

// ✅ Show recording indicator
{state.isRecording && (
  <div className="flex items-center gap-2">
    <div className="h-3 w-3 rounded-full bg-red-500 animate-pulse" />
    <span>Recording... ({state.recordingDuration}s)</span>
  </div>
)}

Error Handling#

// ✅ Graceful degradation
try {
  const transcription = await transcribe();
  const response = await generateResponse(transcription);
  await speakResponse(response);
} catch (error) {
  if (error.message.includes("microphone")) {
    toast.error("Microphone access denied. Please check permissions.");
  } else if (error.message.includes("API")) {
    toast.error("Service temporarily unavailable. Please try again.");
  } else {
    toast.error("An error occurred. Please refresh the page.");
  }
}

Advanced Topics#

Custom AI Instructions#

const INSTRUCTIONS = `You are an ESOL speaking coach.

- Adapt to CEFR level: ${userLevel}
- Focus areas: ${focusSkills.join(", ")}
- Provide feedback on: fluency, accuracy, coherence
- Keep responses under 20 seconds
- Encourage learner after each turn`;

Audio Persistence Strategy#

// Sequential: Simple - save after each turn
await saveUserRecording({
  sessionId,
  audioBlob,
  transcription,
  timestamp: new Date(),
});

// Realtime: Complex - sync with message history
// Map recording index → message index → upload when message saved

Session Analytics#

interface SessionMetrics {
  duration: number;           // Total session time
  turnCount: number;          // Number of exchanges
  userSpeakingTime: number;   // Active speaking time
  avgResponseLatency: number; // Sequential only
  completionRate: number;     // % of target turns reached
}

Conclusion#

Both approaches have their place in ESOL learning platforms:

Sequential Recording: Reliable, cost-effective, and easier to implement. Perfect for structured practice, assessments, and budget-conscious applications.

Realtime Streaming: Premium experience with natural conversation flow. Ideal for advanced learners, fluency practice, and applications where UX justifies the complexity.

Consider implementing both: use Sequential Recording for structured exercises and Realtime Streaming for conversational practice. This hybrid approach provides flexibility for different learning contexts and user needs.

Resources#

Project Files#

Sequential Implementation:

• src/components/conversation/realtime-conversation.tsx - Main component
• src/hooks/use-voice-recorder.ts - Recording logic
• src/app/api/openai/transcribe/route.ts - Whisper API
• src/app/api/openai/conversation/route.ts - GPT-4 API
• src/app/api/openai/tts/route.ts - Text-to-speech API

Realtime Implementation:

• src/components/speaking/ai-coach.tsx - Main component
• src/app/api/openai/realtime-client-secret/route.ts - Authentication

External Documentation#

• OpenAI Realtime API Guide
• OpenAI Whisper API
• MediaRecorder API (MDN)
• WebRTC API (MDN)