DEV Community: Alexander Phaiboon

Building Real-Time Voice Chat in Flutter: A WebSocket Streaming Architecture

Alexander Phaiboon — Sat, 28 Mar 2026 20:26:51 +0000

Most Flutter chat UIs handle text perfectly but fall apart when you add voice. The moment you need real-time audio streaming, you're dealing with WebSockets, buffering, and state management nightmares that make your smooth chat UI stutter and crash.

I spent three weeks building voice chat for our Flutter app and discovered that the standard chat UI patterns don't work for streaming audio. Here's the architecture that actually works, complete with code examples you can implement today.

The Problem with Standard Chat UI Patterns

Traditional Flutter chat widgets assume messages arrive complete. You build a ListView, add a new ChatMessage widget, and call setState(). Perfect for text, terrible for voice.

Voice streaming is different:

Audio chunks arrive every 100ms
You need to show real-time transcription updates
Visual feedback must update without rebuilding the entire chat
WebSocket connections can drop and reconnect
Buffer management becomes critical for smooth playback

The flutter_gen_ai_chat_ui package (864 weekly downloads) handles text streaming well but lacks voice-specific patterns. Most developers end up creating custom solutions that don't scale.

Solution: Streaming-First Architecture

The key insight is separating message state from streaming state. Instead of updating existing messages, we create a dedicated streaming layer that handles real-time updates.

Here's the core architecture:

// Core streaming interfaces
abstract class VoiceProvider {
  Stream<VoiceChunk> streamVoiceInput();
  StreamSink<VoiceChunk> get voiceOutput;
  Stream<TranscriptionUpdate> get transcriptionStream;
}

class VoiceChunk {
  final Uint8List audioData;
  final Duration timestamp;
  final String sessionId;

  const VoiceChunk({
    required this.audioData,
    required this.timestamp,
    required this.sessionId,
  });
}

class TranscriptionUpdate {
  final String text;
  final bool isComplete;
  final String sessionId;
  final Duration timestamp;

  const TranscriptionUpdate({
    required this.text,
    required this.isComplete,
    required this.sessionId,
    required this.timestamp,
  });
}

This separation lets us handle voice streaming without disrupting the main chat UI. The VoiceProvider manages WebSocket connections and audio processing while the chat UI focuses on displaying results.

Step 1: Building the Voice Streaming Controller

The streaming controller coordinates between WebSocket connections and UI updates. It handles connection failures, buffering, and state management:

class VoiceStreamController extends ChangeNotifier {
  late WebSocketChannel _channel;
  final List<VoiceChunk> _audioBuffer = [];
  StreamSubscription? _transcriptionSubscription;

  bool _isRecording = false;
  bool _isConnected = false;
  String _currentTranscription = '';

  bool get isRecording => _isRecording;
  bool get isConnected => _isConnected;
  String get currentTranscription => _currentTranscription;

  Future<void> connect() async {
    try {
      _channel = WebSocketChannel.connect(
        Uri.parse('wss://api.flai.dev/voice-stream'),
      );

      _isConnected = true;
      notifyListeners();

      // Listen for transcription updates
      _transcriptionSubscription = _channel.stream
          .map((data) => TranscriptionUpdate.fromJson(data))
          .listen(_handleTranscriptionUpdate);

    } catch (e) {
      _isConnected = false;
      notifyListeners();
      rethrow;
    }
  }

  void _handleTranscriptionUpdate(TranscriptionUpdate update) {
    _currentTranscription = update.text;
    notifyListeners();

    // If transcription is complete, add to chat history
    if (update.isComplete) {
      _addCompletedMessage(update.text);
      _currentTranscription = '';
    }
  }

  Future<void> startRecording() async {
    if (!_isConnected) await connect();

    _isRecording = true;
    notifyListeners();

    // Start audio capture and streaming
    await _startAudioCapture();
  }

  Future<void> stopRecording() async {
    _isRecording = false;
    notifyListeners();

    await _stopAudioCapture();

    // Send completion signal
    _channel.sink.add(jsonEncode({'type': 'recording_complete'}));
  }

  void _addCompletedMessage(String text) {
    // Add to your chat message list here
    // This integrates with your existing chat UI
  }
}

The controller maintains clear separation between streaming state (isRecording, currentTranscription) and final state (completed messages). This prevents UI flickering and makes the interface predictable.

Step 2: Creating the Voice Chat UI

The UI layer combines your existing chat interface with streaming overlays. The key is using StreamBuilder for real-time updates while keeping the main chat stable:

class VoiceChatWidget extends StatefulWidget {
  @override
  _VoiceChatWidgetState createState() => _VoiceChatWidgetState();
}

class _VoiceChatWidgetState extends State<VoiceChatWidget> {
  late VoiceStreamController _voiceController;
  final List<ChatMessage> _messages = [];

  @override
  void initState() {
    super.initState();
    _voiceController = VoiceStreamController();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Column(
        children: [
          // Main chat area - stable, doesn't rebuild during streaming
          Expanded(
            child: ListView.builder(
              itemCount: _messages.length,
              itemBuilder: (context, index) {
                return ChatMessageWidget(message: _messages[index]);
              },
            ),
          ),

          // Streaming overlay - only rebuilds during voice activity
          Consumer<VoiceStreamController>(
            builder: (context, controller, child) {
              if (!controller.isRecording && controller.currentTranscription.isEmpty) {
                return SizedBox.shrink();
              }

              return StreamingMessageOverlay(
                transcription: controller.currentTranscription,
                isRecording: controller.isRecording,
              );
            },
          ),

          // Voice controls
          VoiceControlsWidget(controller: _voiceController),
        ],
      ),
    );
  }
}

class StreamingMessageOverlay extends StatelessWidget {
  final String transcription;
  final bool isRecording;

  const StreamingMessageOverlay({
    required this.transcription,
    required this.isRecording,
  });

  @override
  Widget build(BuildContext context) {
    return Container(
      margin: EdgeInsets.all(16),
      padding: EdgeInsets.all(12),
      decoration: BoxDecoration(
        color: Colors.blue.withOpacity(0.1),
        borderRadius: BorderRadius.circular(8),
        border: Border.all(
          color: isRecording ? Colors.red : Colors.blue,
          width: 2,
        ),
      ),
      child: Row(
        children: [
          if (isRecording)
            Icon(Icons.mic, color: Colors.red, size: 20),
          if (!isRecording && transcription.isNotEmpty)
            Icon(Icons.edit, color: Colors.blue, size: 20),
          SizedBox(width: 8),
          Expanded(
            child: Text(
              transcription.isEmpty 
                ? (isRecording ? 'Listening...' : 'Processing...')
                : transcription,
              style: TextStyle(
                fontStyle: transcription.isEmpty ? FontStyle.italic : null,
              ),
            ),
          ),
        ],
      ),
    );
  }
}

This approach keeps your existing chat UI completely stable. The ListView never rebuilds during streaming, preventing scroll jumps and maintaining smooth performance. Only the small streaming overlay updates in real-time.

Step 3: WebSocket Connection Management

Voice streaming requires robust connection handling. WebSockets drop frequently on mobile networks, and you need seamless reconnection without losing audio data:

class WebSocketManager {
  WebSocketChannel? _channel;
  Timer? _heartbeatTimer;
  final List<Map<String, dynamic>> _messageQueue = [];

  Stream<dynamic> connect(String url) async* {
    while (true) {
      try {
        _channel = WebSocketChannel.connect(Uri.parse(url));
        _startHeartbeat();

        // Send queued messages
        for (final message in _messageQueue) {
          _channel!.sink.add(jsonEncode(message));
        }
        _messageQueue.clear();

        await for (final data in _channel!.stream) {
          yield data;
        }
      } catch (e) {
        print('WebSocket error: $e');
        _stopHeartbeat();

        // Wait before reconnecting
        await Future.delayed(Duration(seconds: 2));
      }
    }
  }

  void send(Map<String, dynamic> message) {
    if (_channel != null) {
      try {
        _channel!.sink.add(jsonEncode(message));
      } catch (e) {
        // Connection lost, queue message
        _messageQueue.add(message);
      }
    } else {
      _messageQueue.add(message);
    }
  }

  void _startHeartbeat() {
    _heartbeatTimer = Timer.periodic(Duration(seconds: 30), (timer) {
      send({'type': 'ping'});
    });
  }

  void _stopHeartbeat() {
    _heartbeatTimer?.cancel();
    _heartbeatTimer = null;
  }

  void dispose() {
    _stopHeartbeat();
    _channel?.sink.close();
  }
}

The message queue ensures no audio data is lost during reconnections. The heartbeat keeps connections alive on mobile networks that aggressively close idle sockets.

Integration with FlAI

FlAI provides pre-built components that handle these patterns for you. The voice_conversation_flow Mason brick includes the streaming controller, WebSocket management, and UI components as a complete package:

# Install FlAI CLI
dart pub global activate flai_cli

# Generate voice chat component
flai create voice_conversation_flow my_voice_chat

# Add to your app
cd my_voice_chat
flutter pub get

The generated component includes production-ready error handling, audio buffering, and integration hooks for your existing chat UI. You get the architecture I've shown here without building it from scratch.

Performance Considerations

Voice streaming puts different demands on your app than text chat:

Memory Management: Audio buffers grow quickly. Implement circular buffering and clear completed chunks:

class AudioBufferManager {
  static const int maxBufferSize = 50; // Keep last 50 chunks
  final Queue<VoiceChunk> _buffer = Queue();

  void addChunk(VoiceChunk chunk) {
    _buffer.add(chunk);

    if (_buffer.length > maxBufferSize) {
      _buffer.removeFirst();
    }
  }
}

UI Updates: Limit transcription updates to prevent excessive rebuilds:

Timer? _updateTimer;

void _handleTranscriptionUpdate(String text) {
  _updateTimer?.cancel();
  _updateTimer = Timer(Duration(milliseconds: 100), () {
    _currentTranscription = text;
    notifyListeners();
  });
}

Network Efficiency: Batch small audio chunks to reduce WebSocket overhead:

final List<VoiceChunk> _chunkBatch = [];

void _batchAndSend(VoiceChunk chunk) {
  _chunkBatch.add(chunk);

  if (_chunkBatch.length >= 5 || chunk.timestamp.inMilliseconds % 500 == 0) {
    _sendBatch();
    _chunkBatch.clear();
  }
}

Next Steps

This architecture scales to complex voice AI scenarios. You can extend it with:

Multi-participant voice chat using session IDs
Voice activity detection to optimize bandwidth
Audio quality adaptation based on connection speed
Integration with speech recognition APIs beyond transcription

The streaming-first approach works because it separates concerns cleanly. Your chat UI stays simple and fast, while the streaming layer handles all the complexity of real-time audio.

Try implementing the basic controller first, then add the UI overlay. Once you have those working, the WebSocket management becomes straightforward to integrate.

The key insight is that voice chat isn't just text chat with audio added—it's a different interaction pattern that needs different architecture. Build for streaming from the start, and your users will get the smooth, responsive voice experience they expect.

Building Voice Conversations Without Usage Limits: A Flutter Developer's Guide

Alexander Phaiboon — Sat, 28 Mar 2026 19:07:45 +0000

Most productivity tools bolt AI onto their sidebar as an afterthought. But voice-first interfaces require rethinking your entire event architecture. After watching Notion limit users to 20 lifetime AI interactions, we built a voice conversation system that scales without artificial constraints.

The Problem with Traditional AI Integration

When developers add voice features, they usually treat it like a simple API call. Press button, send audio, get response. This approach breaks down at scale because:

No conversation state management - Each voice interaction exists in isolation
Blocking UI patterns - Users wait for complete responses before continuing
No error recovery - Network issues kill entire conversations

Here's what most implementations look like:

// BAD: Blocking voice implementation
class BasicVoiceWidget extends StatefulWidget {
  @override
  _BasicVoiceWidgetState createState() => _BasicVoiceWidgetState();
}

class _BasicVoiceWidgetState extends State<BasicVoiceWidget> {
  bool isRecording = false;
  String response = '';

  Future<void> handleVoiceInput() async {
    setState(() => isRecording = true);

    // This blocks everything until complete
    final audioData = await recordAudio();
    final result = await sendToAPI(audioData);

    setState(() {
      isRecording = false;
      response = result;
    });
  }
}

This pattern forces users into a request-response cycle that feels unnatural for conversations.

Sealed Events: The Foundation of Scalable Voice

We solved this with a sealed event hierarchy that treats voice as a stream of state changes, not discrete API calls. Here's our complete VoiceEvent system:

// Type-safe voice event hierarchy
sealed class VoiceEvent {
  const VoiceEvent();
}

class VoiceStarted extends VoiceEvent {
  final String conversationId;
  final DateTime timestamp;
  final Map<String, dynamic>? context;

  const VoiceStarted({
    required this.conversationId,
    required this.timestamp,
    this.context,
  });

  @override
  String toString() => 'VoiceStarted(id: $conversationId, time: $timestamp)';
}

class VoiceRecording extends VoiceEvent {
  final String conversationId;
  final Duration duration;
  final double audioLevel;

  const VoiceRecording({
    required this.conversationId,
    required this.duration,
    required this.audioLevel,
  });

  @override
  String toString() => 'VoiceRecording(id: $conversationId, duration: ${duration.inSeconds}s)';
}

class VoicePaused extends VoiceEvent {
  final String conversationId;
  final String reason;
  final DateTime pausedAt;

  const VoicePaused({
    required this.conversationId,
    required this.reason,
    required this.pausedAt,
  });
}

class VoiceCompleted extends VoiceEvent {
  final String conversationId;
  final String audioPath;
  final Duration totalDuration;
  final Map<String, dynamic> metadata;

  const VoiceCompleted({
    required this.conversationId,
    required this.audioPath,
    required this.totalDuration,
    required this.metadata,
  });
}

class VoiceError extends VoiceEvent {
  final String conversationId;
  final String error;
  final String? recoveryAction;
  final DateTime occurredAt;

  const VoiceError({
    required this.conversationId,
    required this.error,
    this.recoveryAction,
    required this.occurredAt,
  });
}

Each event carries exactly the data needed for that state. No optional fields that might be null, no guessing what properties are available.

Stream-Based Provider Architecture

The magic happens in how these events flow through the system. Instead of blocking calls, we use a streaming provider that other widgets can subscribe to:

class VoiceProvider extends ChangeNotifier {
  final StreamController<VoiceEvent> _eventController = 
      StreamController<VoiceEvent>.broadcast();

  Stream<VoiceEvent> get eventStream => _eventController.stream;

  // Current conversation state
  Map<String, VoiceConversation> _conversations = {};
  String? _activeConversationId;

  // Public getters
  VoiceConversation? get activeConversation => 
      _activeConversationId != null 
          ? _conversations[_activeConversationId] 
          : null;

  bool get isRecording => activeConversation?.state == VoiceState.recording;

  // Start new conversation
  Future<String> startConversation({Map<String, dynamic>? context}) async {
    final conversationId = generateConversationId();
    final conversation = VoiceConversation(
      id: conversationId,
      startedAt: DateTime.now(),
      context: context ?? {},
    );

    _conversations[conversationId] = conversation;
    _activeConversationId = conversationId;

    // Emit event
    _eventController.add(VoiceStarted(
      conversationId: conversationId,
      timestamp: DateTime.now(),
      context: context,
    ));

    notifyListeners();
    return conversationId;
  }

  // Handle recording updates
  void updateRecording({
    required String conversationId,
    required Duration duration,
    required double audioLevel,
  }) {
    final conversation = _conversations[conversationId];
    if (conversation == null) return;

    // Update conversation state
    conversation.updateRecording(duration, audioLevel);

    // Emit event
    _eventController.add(VoiceRecording(
      conversationId: conversationId,
      duration: duration,
      audioLevel: audioLevel,
    ));

    notifyListeners();
  }

  // Complete conversation
  Future<void> completeConversation({
    required String conversationId,
    required String audioPath,
  }) async {
    final conversation = _conversations[conversationId];
    if (conversation == null) return;

    // Calculate metadata
    final metadata = {
      'wordCount': await estimateWordCount(audioPath),
      'fileSize': await getFileSize(audioPath),
      'quality': await assessAudioQuality(audioPath),
    };

    // Update conversation
    conversation.complete(audioPath, metadata);

    // Emit event
    _eventController.add(VoiceCompleted(
      conversationId: conversationId,
      audioPath: audioPath,
      totalDuration: conversation.duration,
      metadata: metadata,
    ));

    // Process in background
    _processAudioInBackground(conversationId, audioPath);

    notifyListeners();
  }

  // Error handling
  void handleError({
    required String conversationId,
    required String error,
    String? recoveryAction,
  }) {
    final conversation = _conversations[conversationId];
    conversation?.markError(error);

    _eventController.add(VoiceError(
      conversationId: conversationId,
      error: error,
      recoveryAction: recoveryAction,
      occurredAt: DateTime.now(),
    ));

    notifyListeners();
  }

  @override
  void dispose() {
    _eventController.close();
    super.dispose();
  }
}

This architecture gives us several advantages:

Non-blocking UI - Widgets update as events stream in
Easy debugging - Every state change is an explicit event
Testable logic - Mock the stream, verify events
Error recovery - Errors don't kill the conversation

Real Usage Patterns

The sealed events make complex UI patterns simple to implement. Here's how we handle the common "push-to-talk while showing live transcription" pattern:

class VoiceConversationWidget extends StatefulWidget {
  @override
  _VoiceConversationWidgetState createState() => _VoiceConversationWidgetState();
}

class _VoiceConversationWidgetState extends State<VoiceConversationWidget> {
  late StreamSubscription<VoiceEvent> _eventSubscription;
  String _liveTranscription = '';
  double _audioLevel = 0.0;
  String? _error;

  @override
  void initState() {
    super.initState();

    // Subscribe to voice events
    _eventSubscription = context.read<VoiceProvider>()
        .eventStream
        .listen(_handleVoiceEvent);
  }

  void _handleVoiceEvent(VoiceEvent event) {
    if (!mounted) return;

    setState(() {
      switch (event) {
        case VoiceStarted(:final conversationId):
          _liveTranscription = '';
          _error = null;
          _logEvent('Started conversation: $conversationId');

        case VoiceRecording(:final duration, :final audioLevel):
          _audioLevel = audioLevel;
          _updateLiveTranscription(duration);

        case VoicePaused(:final reason):
          _logEvent('Paused: $reason');

        case VoiceCompleted(:final audioPath, :final totalDuration):
          _logEvent('Completed: ${totalDuration.inSeconds}s, saved to $audioPath');
          _finalizeTranscription();

        case VoiceError(:final error, :final recoveryAction):
          _error = error;
          _logEvent('Error: $error');
          if (recoveryAction != null) {
            _showRecoveryOption(recoveryAction);
          }
      }
    });
  }

  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        // Live audio level indicator
        AudioLevelIndicator(level: _audioLevel),

        // Live transcription
        Container(
          padding: EdgeInsets.all(16),
          child: Text(
            _liveTranscription.isEmpty 
                ? 'Press and hold to start speaking...' 
                : _liveTranscription,
            style: TextStyle(
              fontSize: 16,
              color: _liveTranscription.isEmpty ? Colors.grey : Colors.black,
            ),
          ),
        ),

        // Error display
        if (_error != null)
          Container(
            padding: EdgeInsets.all(8),
            margin: EdgeInsets.symmetric(horizontal: 16),
            decoration: BoxDecoration(
              color: Colors.red.shade50,
              borderRadius: BorderRadius.circular(8),
            ),
            child: Text(_error!, style: TextStyle(color: Colors.red.shade700)),
          ),

        // Push-to-talk button
        VoiceFAB(),
      ],
    );
  }

  void _updateLiveTranscription(Duration duration) {
    // Simulate progressive transcription
    // In production, this would come from your speech-to-text service
    final seconds = duration.inSeconds;
    if (seconds > 0 && seconds % 2 == 0) {
      _liveTranscription += _getNextTranscriptionChunk();
    }
  }

  @override
  void dispose() {
    _eventSubscription.cancel();
    super.dispose();
  }
}

The key insight is that each event type tells the UI exactly what changed and what data is now available. No more checking multiple boolean flags or null values.

Lessons from Production

After running this system in production for three months, here are the patterns that emerged:

Event Granularity Matters: We initially had fewer event types, but debugging was harder. The current five events hit the sweet spot between detail and simplicity.

Stream Performance: Broadcasting events to multiple listeners is cheap in Flutter. We have conversations with 20+ widgets listening to the same stream without performance issues.

Error Recovery: The explicit VoiceError event with optional recovery actions let us build self-healing UIs. When network issues interrupt recording, we can offer "retry" or "save locally" options based on the error type.

Testing Wins: Sealed classes make testing voice flows trivial. Mock the event stream, verify widgets respond correctly to each event type. No more integration tests for voice features.

Beyond Voice: Event-Driven Everything

This pattern scales beyond voice. We use similar sealed event hierarchies for:

Document collaboration - DocumentEvent with ContentChanged, CursorMoved, UserJoined events
AI processing - ProcessingEvent with Started, Progress, Completed, Failed events
Real-time sync - SyncEvent with Connected, Syncing, Conflict, Resolved events

The sealed class pattern forces you to handle all possible states explicitly, making your apps more robust.

Next Steps

If you're building voice features, start with the event hierarchy. Define all possible states as sealed classes before writing any UI code. This upfront design work pays dividends when you need to debug complex interaction flows.

The complete code for this voice system is running in production at CMMD, where we use it for natural language task delegation to our AI agent workforce. Unlike tools that limit AI interactions, our voice interface scales with usage because the architecture was designed for streaming, not blocking calls.

Want to see this in action? Try starting a voice conversation with CMMD's Sidekick - the entire interaction is powered by this event-driven architecture.