DEV Community: Thor 雷神 Schaeff

Build real-time conversational agents with Gemini 3.1 Flash Live

Thor 雷神 Schaeff — Thu, 26 Mar 2026 15:25:51 +0000

Today, we’re launching Gemini 3.1 Flash Live via the Gemini Live API in Google AI Studio. Gemini 3.1 Flash Live helps enable developers to build real-time voice and vision agents that can not only process the world around them, but also respond at the speed of conversation.

This is a step change in latency, reliability and more natural-sounding dialogue, delivering the quality needed for the next generation of voice-first AI.

Experience enhanced latency, reliability and quality

For real-time interactions, every millisecond of latency strips away the natural flow of the conversation that users expect. The new model better understands tone, emphasis and intent, enabling agents with key improvements:

Higher task completion rates in noisy, real-world environments: We’ve significantly improved the model’s ability to trigger external tools and deliver information during live conversations. By better discerning relevant speech from environmental sounds like traffic or television, the model more effectively filters out background noise to remain reliable and responsive to instructions.
Better instruction-following: Adherence to complex system instructions has been boosted significantly. Your agent will stay within its operational guardrails, even when conversations take unexpected turns.
More natural and low-latency dialogue: The latest model improves on latency and is even more effective at recognizing acoustic nuances like pitch and pace compared to 2.5 Flash Native Audio, making real-time conversations feel a lot more fluid and natural.
Multi-lingual capabilities: The model supports more than 90 languages for real-time multi-modal conversations.

Build with an expanding ecosystem of integrations

The Live API is built for production environments, but real-world systems require handling of diverse inputs, from live video streams to on-demand phone calls.

For systems that require WebRTC scaling or global edge routing, we recommend exploring our partner integrations to streamline the development of real-time voice and video agents.

LiveKit — Use the Gemini Live API with LiveKit Agents.
Pipecat by Daily — Create a real-time AI chatbot using Gemini Live and Pipecat.
Fishjam by Software Mansion — Create live video and audio streaming applications with Fishjam.
Vision Agents by Stream — Build real-time voice and video AI applications with Vision Agents.
Voximplant — Connect inbound and outbound calls to Live API with Voximplant.
Firebase AI SDK — Get started with the Gemini Live API using Firebase AI Logic.

Get started with the Live API

Gemini 3.1 Flash Live is available starting today via the Gemini API and in Google AI Studio. Developers can use the Gemini Live API to integrate the model into their application.

Explore our developer documentation to learn how you can build real-time agents:

Gemini Live API documentation: Explore features like multilingual support, tool use and function calling, session management (for managing long running conversations) and ephemeral tokens.
Gemini Live API examples: Get inspiration for the kind of voice experiences you can build today with the model.
Gemini Live API Skill: For coding agents to learn and build with the Live API.

Get started with the Google GenAI SDK:

import asyncio
from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

model = "gemini-3.1-flash-live-preview"
config = {"response_modalities": ["AUDIO"]}

async def main():
    async with client.aio.live.connect(model=model, config=config) as session:
        print("Session started")
        # Send content...

if __name__ == "__main__":
    asyncio.run(main())

A Guide to Embeddings and pgvector

Thor 雷神 Schaeff — Wed, 11 Mar 2026 17:12:53 +0000

A guide to building AI-powered search using Google's Gemini Embedding 2 and Supabase.

Google recently released Gemini Embedding 2, a powerful new embedding model that supports text, images, and audio in a single unified vector space. Combined with Supabase and pgvector, you can build multimodal similarity search entirely within your Postgres database.

In this guide we'll explain what embeddings are, what makes Gemini's new model interesting, and how to store and query embeddings in PostgreSQL using pgvector.

What are embeddings?

Embeddings capture the "relatedness" of text, images, video, or other types of information. This relatedness is most commonly used for:

Search: how similar is a search term to a body of text?
Recommendations: how similar are two products?
Classifications: how do we categorize a body of text?
Clustering: how do we identify trends?

Let's explore an example of text embeddings. Say we have three phrases:

"The cat chases the mouse"
"The kitten hunts rodents"
"I like ham sandwiches"

Your job is to group phrases with similar meaning. If you are a human, this should be obvious. Phrases 1 and 2 are almost identical, while phrase 3 has a completely different meaning.

Although phrases 1 and 2 are similar, they share no common vocabulary (besides "the"). Yet their meanings are nearly identical. How can we teach a computer that these are the same?

How do embeddings work?

Embeddings compress discrete information (words & symbols) into distributed continuous-valued data (vectors). If we took our phrases from before and plotted them on a chart, it might look something like this:

        "The kitten hunts rodents"
    •

  "The cat chases the mouse"
    •



                          "I like ham sandwiches"
                              •

Phrases 1 and 2 would be plotted close to each other, since their meanings are similar. We would expect phrase 3 to live somewhere far away since it isn't related. If we had a fourth phrase, "Sally ate Swiss cheese", this might exist somewhere between phrase 3 (cheese can go on sandwiches) and phrase 1 (mice like Swiss cheese).

In this example we only have 2 dimensions: the X and Y axis. In reality, we would need many more dimensions to effectively capture the complexities of human language.

Gemini Embedding 2

Google offers an embedding API as part of the Gemini platform. You feed it text, images, or audio, and it outputs a vector of floating point numbers that represents the "meaning" of that content.

The latest model, gemini-embedding-2-preview, outputs 3072 dimensions by default. What makes it special:

Multimodal: embed text, images, and audio into the same vector space, enabling cross-modal search (e.g., search images with text)
Matryoshka Representation Learning (MRL): truncate embeddings to smaller dimensions (768, 512, 256, etc.) to trade off accuracy for storage and speed
High quality: state-of-the-art performance on the MTEB leaderboard

Why is this useful? Once we have generated embeddings on multiple pieces of content, it is trivial to calculate how similar they are using vector math operations like cosine distance. A perfect use case for this is search. Your process might look something like this:

Pre-process your knowledge base and generate embeddings for each item
Store your embeddings to be referenced later (more on this)
Build a search interface that prompts your user for input
Take the user's input, generate a one-time embedding, then perform a similarity search against your pre-processed embeddings
Return the most similar items to the user

Embeddings in practice

At a small scale, you could store your embeddings in a CSV file, load them into Python, and use a library like numPy to calculate similarity between them. Unfortunately this likely won't scale well:

What if I need to store and search over a large number of documents and embeddings (more than can fit in memory)?
What if I want to create/update/delete embeddings dynamically?
What if I'm not using Python?

Using PostgreSQL

Enter pgvector, an extension for PostgreSQL that allows you to both store and query vector embeddings within your database. Let's try it out.

First we'll enable the Vector extension. In Supabase, this can be done from the web portal through Database → Extensions. You can also do this in SQL by running:

create extension vector;

Next let's create a table to store our items and their embeddings. We'll use 768 dimensions since Gemini's MRL feature lets us truncate to smaller sizes for efficiency:

create table items (
  id bigserial primary key,
  content text,
  metadata jsonb,
  embedding vector(768)
);

pgvector introduces a new data type called vector. In the code above, we create a column named embedding with the vector data type. The size of the vector defines how many dimensions the vector holds. Gemini's gemini-embedding-2-preview model outputs 3072 dimensions by default, but thanks to MRL we can truncate to 768 dimensions without significant quality loss — saving ~75% on storage and improving query speed.

We also create a text column named content to store the original text, and a jsonb column named metadata for any additional information about each item.

Soon we're going to need to perform a similarity search over these embeddings. Let's create a function to do that:

create or replace function match_items (
  query_embedding vector(768),
  match_threshold float,
  match_count int
)
returns table (
  id bigint,
  content text,
  metadata jsonb,
  similarity float
)
language sql stable
as $$
  select
    items.id,
    items.content,
    items.metadata,
    1 - (items.embedding <=> query_embedding) as similarity
  from items
  where items.embedding <=> query_embedding < 1 - match_threshold
  order by items.embedding <=> query_embedding
  limit match_count;
$$;

pgvector introduces 3 new operators that can be used to calculate similarity:

Operator	Description
`<->`	Euclidean distance
`<#>`	negative inner product
`<=>`	cosine distance

Cosine similarity works well with normalized embeddings, so we will use that here.

Now we can call match_items(), pass in our embedding, similarity threshold, and match count, and we'll get a list of all items that match. And since this is all managed by Postgres, our application code becomes very simple.

Indexing

Once your table starts to grow with embeddings, you will likely want to add an index to speed up queries. Vector indexes are particularly important when you're ordering results because vectors are not grouped by similarity, so finding the closest by sequential scan is a resource-intensive operation.

Each distance operator requires a different type of index. We expect to order by cosine distance, so we need a vector_cosine_ops index. You can use either IVFFlat or HNSW — HNSW generally provides better recall:

create index on items using hnsw (embedding vector_cosine_ops);

For IVFFlat, a good starting number of lists is 4 * sqrt(table_rows):

create index on items using ivfflat (embedding vector_cosine_ops)
with
  (lists = 100);

You can read more about indexing on pgvector's GitHub page here.

Generating embeddings

Let's use the Google Gemini SDK to generate embeddings. First, install the dependencies:

npm install @google/genai @supabase/supabase-js

Create a helper to initialize the Gemini client and generate embeddings:

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY! });

export async function getEmbedding(input: string): Promise<number[]> {
  const response = await ai.models.embedContent({
    model: "gemini-embedding-2-preview",
    contents: [{ parts: [{ text: input }] }],
    config: {
      outputDimensionality: 768,
    },
  });

  const values = response.embeddings?.[0]?.values;
  if (!values) {
    throw new Error("No embeddings returned from API");
  }

  return normalizeVector(values);
}

function normalizeVector(vector: number[]): number[] {
  let sumOfSquares = 0;
  for (const val of vector) {
    sumOfSquares += val * val;
  }
  const magnitude = Math.sqrt(sumOfSquares);
  if (magnitude === 0) return vector;
  return vector.map((val) => val / magnitude);
}

We normalize the vectors after generation. This is a best practice when using truncated dimensions via MRL — it ensures cosine similarity calculations remain accurate.

Now let's store some documents:

import { createClient } from "@supabase/supabase-js";

const supabase = createClient(
  process.env.SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY!
);

async function storeDocuments() {
  const documents = [
    "The cat chases the mouse",
    "The kitten hunts rodents",
    "I like ham sandwiches",
  ];

  for (const content of documents) {
    // Generate embedding using Gemini
    const embedding = await getEmbedding(content);

    // Store in Supabase
    await supabase.from("items").insert({
      content,
      metadata: { source: "example" },
      embedding,
    });
  }
}

Building a search function

Now let's build a search function that takes a user's query, generates an embedding, and finds the most similar items:

async function search(query: string) {
  // Generate a one-time embedding for the query
  const embedding = await getEmbedding(query);

  // Perform similarity search via Supabase RPC
  const { data: items, error } = await supabase.rpc("match_items", {
    query_embedding: embedding,
    match_threshold: 0.5,
    match_count: 10,
  });

  if (error) throw error;

  return items;
}

// Example usage
const results = await search("small feline catching prey");
console.log(results);
// Returns "The cat chases the mouse" and "The kitten hunts rodents"
// but NOT "I like ham sandwiches"

Multimodal search

This is where Gemini Embedding 2 really shines. Unlike text-only models, you can embed images and audio into the same vector space. This means you can search for images using text, or find audio clips similar to an image.

export interface MultimodalPart {
  text?: string;
  inlineData?: {
    data: string; // base64
    mimeType: string;
  };
}

export async function getMultimodalEmbedding(
  parts: MultimodalPart[]
): Promise<number[]> {
  const response = await ai.models.embedContent({
    model: "gemini-embedding-2-preview",
    contents: [{ parts }],
    config: {
      outputDimensionality: 768,
    },
  });

  const values = response.embeddings?.[0]?.values;
  if (!values) throw new Error("No embeddings returned");

  return normalizeVector(values);
}

Now you can embed an image and search for it with text:

// Embed an image
const imageBase64 = fs.readFileSync("photo.jpg").toString("base64");
const imageEmbedding = await getMultimodalEmbedding([
  {
    inlineData: {
      data: imageBase64,
      mimeType: "image/jpeg",
    },
  },
]);

// Store it
await supabase.from("items").insert({
  content: "photo.jpg",
  metadata: { type: "image", mimeType: "image/jpeg" },
  embedding: imageEmbedding,
});

// Later, search with text — finds the image!
const results = await search("a photograph");

This cross-modal capability opens up powerful use cases like visual search engines, audio content discovery, and mixed-media recommendation systems.

Choosing dimensions

Gemini Embedding 2 supports Matryoshka Representation Learning, which means you can choose your embedding size:

Dimensions	Storage per vector	Best for
3072 (default)	~24 KB	Maximum accuracy
768	~6 KB	Good balance of accuracy and speed
256	~2 KB	Large-scale, speed-critical applications

In this project we use 768 dimensions — it gives us roughly equivalent quality to OpenAI's text-embedding-ada-002 (which is also 1536-dimensional) while using half the storage. For most applications, 768 dimensions provides an excellent accuracy-to-efficiency tradeoff.

To change the dimension, update two things:

The outputDimensionality in your API call
The vector(n) size in your SQL table definition

Wrapping up

With Gemini Embedding 2 and Supabase, you get:

Multimodal embeddings — text, images, and audio in one vector space
Flexible dimensions — choose your accuracy vs speed tradeoff with MRL
Managed infrastructure — Supabase handles your Postgres + pgvector so you don't have to
Simple queries — similarity search is just a SQL function call

The full source code for a working multimodal search application using this stack is available in this repository. Check out the Supabase AI & Vectors docs for more advanced patterns like Retrieval Augmented Generation (RAG).

How Fishjam.io Built a Multi-Speaker AI Game using Gemini Live

Thor 雷神 Schaeff — Mon, 09 Feb 2026 15:43:54 +0000

Picture a lively dinner party: glasses clinking, half-finished sentences, and three people laughing at the same time. To a human, navigating this is instinctual. To an AI, it is a nightmare. Developers have effectively mastered the predictable flow of a one-on-one chat. But handling a group conversation, where people interrupt and talk over each other, is much more difficult.

Bernard Gawor and the Fishjam team at Software Mansion set out to showcase their Selective Forwarding Unit solution by building a unique demo app that solves this problem. That’s how the Deep Sea Stories game came to life.

The premise is simple: a group of detectives enters a conference room to solve a mystery. The twist? The "Riddle Master", the entity that knows the secret solution and answers questions is actually a Gemini Voice AI Agent. This required the agent to listen, understand, and respond to a group of users in real-time.

The Anatomy of a Voice Agent

First, let’s look at how an AI Voice Agent typically processes data. It typically operates through a modular pipeline that includes the following steps:

Speech-to-Text (S2T): The system converts the user’s spoken input into text using models like Google Speech-to-Text, OpenAI Whisper or ElevenLabs’ transcription service.
Large Language Model (LLM): The transcribed text is processed by an LLM (e.g. Gemini, GPT-4, Claude) to understand the context and generate an appropriate text response.
Text-to-Speech (TTS): The text response is converted back into natural-sounding speech using services like Google Cloud TTS, ElevenLabs or Azure TTS.
Real-time Audio Streaming: The audio is delivered back to the user with minimal latency.

A second architecture gaining popularity, and notably used in the newest Gemini Live API models is Speech-to-Speech. Unlike traditional pipelines that convert speech to text and back again, this architecture feeds raw audio directly into the model and generates audio output in a single step. This unified approach not only reduces latency but also preserves non-verbal features, enabling the model to recognize and replicate subtle human emotions, tone, and pacing with high fidelity.

One-to-One vs. Group Contexts

Most standard SDKs make setting up a one-on-one conversation relatively simple. For example, using the Gemini Live API SDK:

const { GoogleGenAI } = require("@google/genai");

// 1. Setup
const ai = new GoogleGenAI({ apiKey: "YOUR_API_KEY" });

async function startAgent() {
  // 2. Connect
  const session = await ai.live.connect({
    model: "gemini-2.5-flash-native-audio-preview-12-2025",
    config: { responseModalities: ["AUDIO"] },
  });

  console.log("Agent Connected!");

  // 3. Listen for the Agent's Voice
  session.receive(async (msg) => {
    // This loop runs every time the AI sends an audio chunk
    if (msg.serverContent?.modelTurn?.parts) {
      const audioData = msg.serverContent.modelTurn.parts[0].inlineData.data;
      console.log(`Received Audio Chunk (${audioData.length} bytes)`);
      // In a real app, you would send 'audioData' to your audio output device
    }
  });

  // 4. Send Your Voice (Simulated)
  // Real apps pipe microphone data here continuously
  console.log("Sending audio...");
  await session.sendRealtimeInput([
    {
      mimeType: "audio/pcm;rate=16000",
      data: "BASE64_ENCODED_PCM_AUDIO_STRING_GOES_HERE",
    },
  ]);
}

startAgent();

However, these SDKs assume a single audio input stream. In a conference room, audio streams are distinct, asynchronous, and overlapping. They had to determine how to aggregate these inputs for the Riddle Master without losing context or introducing unacceptable latency.

They evaluated three specific architectural strategies to handle the multi-speaker environment:

Server-Side Aggregation: This method involves mixing all player audio streams into a single channel before sending it to the AI Agent. While simple to implement, mixing audio makes it incredibly difficult for the Speech-to-Text (S2T) model to transcribe accurately, especially when users talk over one another. This results in “hallucinations” or missed queries.
Agent per Client: This approach assigns a separate Voice AI agent to every single player in the room. This creates a chaotic user experience (all agents speaking at once) and prevents a shared game state. It is also cost-prohibitive, as every user stream consumes separate processing tokens.
Server-Side Filtering using VAD: In this approach, they implemented a centralized gatekeeper using Voice Activity Detection (VAD). They wait for a player to speak, lock the “input slot” and forward only that specific player’s audio to the AI agent. Once they stop speaking, the lock is released, allowing another player to ask questions. This is the solution they finally went with.

Beyond One-on-One: A “Deep Sea Stories” Game Web App

Key Technologies

Fishjam: A real-time communication platform handling peer-to-peer audio streaming via WebRTC (SFU). (Not familiar with WebRTC/SFUs? Check out their guide)
Gemini GenAI Voice Agent: Provides an easy SDK that makes creating voice agents and initializing audio conversations simple.

Architecture Overview

The game logic is handled on the backend, which manages the conferencing room and peer connections.

Player Connection: When players join the game using the frontend client, they connect audio/video via the Fishjam Web SDK. (See: Fishjam React Quick Start).
The Bridge: When the game starts, the backend creates a Fishjam Agent. This agent acts like a “ghost peer” in the audio-video room; its sole purpose is to capture audio of the players and forward it to the AI, and vice versa.
The Brain: The backend initiates a WebSocket connection with the Gemini agent and forwards the audio stream from players to Gemini and vice versa.

Implementation Details

1. Initializing Clients and game room

import { FishjamClient } from '@fishjam-cloud/js-server-sdk';
import GeminiIntegration from '@fishjam-cloud/js-server-sdk/gemini';

const fishjamClient = new FishjamClient({
  fishjamId: process.env.FISHJAM_ID!,
  managementToken: process.env.FISHJAM_TOKEN!,
});

const genAi = GeminiIntegration.createClient({
  apiKey: process.env.GOOGLE_API_KEY!,
});

const gameRoom = await fishjamClient.createRoom();

2. Creating the Fishjam Agent

When the first player joins the game room, they create the Fishjam agent to capture players' audio on the backend.

import GeminiIntegration from "@fishjam-cloud/js-server-sdk/gemini";

const { agent } = await fishjamClient.createAgent(gameRoom.id, {
  subscribeMode: "auto",
  // Use their preset to match the required audio format (16kHz)
  output: GeminiIntegration.geminiInputAudioSettings,
});
// agentTrack enables to send audio back to players
const agentTrack = agent.createTrack(
  GeminiIntegration.geminiOutputAudioSettings,
);

3. Configuring and Initializing the AI Riddle Master

When users select a story scenario, they configure the Gemini agent with the specific context (the riddle solution and the “Game Master” persona).

const session = await genAi.live.connect({
  model: GEMINI_MODEL,
  config: {
    responseModalities: [Modality.AUDIO],
    systemInstruction:
      "here's the story: ..., and its solution: ... you should answer only yes or no questions about this story",
  },
  callbacks: {
    // Gemini -> Fishjam
    onmessage: (msg) => {
      if (msg.data) {
        // send Riddle Master's audio responses back to players
        const pcmData = Buffer.from(msg.data, "base64");
        agent.sendData(agentTrack.id, pcmData);
      }

      if (msg.serverContent?.interrupted) {
        console.log("Agent was interrupted by user.");
        // Clears the buffer on the Fishjam media server
        agent.interruptTrack(agentTrack.id);
      }
    },
  },
});

4. Bridging Audio (The Glue)

The final piece of the puzzle is the bridge between the SFU and the AI. They capture audio streams from the Fishjam agent (what the players are saying) and pass them through a custom VAD (Voice Activity Detection) filter. This filter implements a “mutex” lock mechanism: it identifies the first active speaker, locks the channel to their ID, and forwards only their audio to Gemini. All other simultaneous audio is ignored until the active speaker finishes their turn.

Below is the simplified code of this logic:

// State to track who currently "holds the floor"
let activeSpeakerId: string | null = null;

// They capture audio chunks from ALL players in the room
agent.on("audioTrack", (userId, pcmChunk) => {
  vadService.process(userId, pcmChunk);
});

// VAD Processor Logic
vadService.on("activity", (userId, isSpeaking, audioData) => {
  if (activeSpeakerId === null && isSpeaking) {
    activeSpeakerId = userId; // Lock the floor
  }

  // They only forward audio if it comes from the person holding the lock
  if (userId === activeSpeakerId) {
    voiceAgentSession.sendAudio(audioData);

    // If the active speaker stops speaking (silence detected), release the lock
    if (!isSpeaking) {
      // (Optional: Add a debounce delay here to prevent cutting off pauses)
      activeSpeakerId = null;
    }
  }
});

Challenges in group AI

Building a multi-user voice interface introduces unique challenges compared to 1-on-1 chats:

Floor Control: Standard Speech-to-Text models can struggle when multiple players speak simultaneously. Determining which player the AI should respond to or if it should simply listen requires careful handling.
Latency: Real-time responsiveness is critical for immersion. The entire pipeline (Audio → Text → LLM → Audio) must execute in milliseconds.
Audio Quality: Maintaining clear audio through transcoding and streaming across different networks is essential.

Fortunately, Fishjam’s WebRTC implementation largely solves the latency and audio quality issues. The challenges of Floor Control needed carefully structured implementation on the backend, but it was not really that hard!

Try the Game Yourself!

They have implemented the functionality described above in a live demo. Gather friends and try to solve a mystery with their AI Riddle Master!

Play the Demo: Deep Sea Stories
View the Code: GitHub Repository

If someone is working on AI-based features with real-time video or audio and needs assistance, they can reach out to the team on Discord.

Realtime Multimodal AI on Ray-Ban Meta Glasses with Gemini Live & LiveKit

Thor 雷神 Schaeff — Tue, 03 Feb 2026 16:27:34 +0000

Imagine walking down the street, asking your glasses what kind of plant you're looking at, and getting a response in near real-time. With the combination of Gemini Live API, LiveKit, and Meta Wearables SDK, this isn't science fiction anymore, it's something you can build today.

// Detect dark theme var iframe = document.getElementById('tweet-2016031645003088280-823'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2016031645003088280&theme=dark" }

In this post, we’ll walk through how to set up a vision-enabled AI agent that connects to Meta Ray-Ban glasses via a secure WebRTC proxy.

The Architecture

The setup involves several layers to ensure low-latency, secure communication between the wearable device and the AI:

Meta Ray-Ban Glasses: Capture video and audio, connecting via Bluetooth to your phone.
Phone (Android/iOS): Acts as the gateway, connecting via WebRTC to LiveKit Cloud.
LiveKit Cloud: Serves as a secure, high-performance proxy for the Gemini Live API.
Gemini Live API: Processes the stream via WebSockets, enabling real-time multimodal interaction.

The Backend: Building the Gemini Live Agent

We use the LiveKit Agents framework to act as a secure WebRTC proxy for the Gemini Live API. This agent joins the LiveKit room, listens to the audio, and processes the video stream from the glasses.

Setting up the Assistant

The core of our agent is the AgentSession. We use the google.beta.realtime.RealtimeModel to interface with Gemini. Crucially, we enable video_input in the RoomOptions to allow the agent to "see."

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    ctx.log_context_fields = {"room": ctx.room.name}

    session = AgentSession(
        llm=google.beta.realtime.RealtimeModel(
            model="gemini-2.5-flash-native-audio-preview-12-2025",
            proactivity=True,
            enable_affective_dialog=True
        ),
        vad=ctx.proc.userdata["vad"],
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
        room_options=room_io.RoomOptions(
            video_input=True,
        )
    )
    await ctx.connect()
    await session.generate_reply()

By setting video_input=True, the agent automatically requests the video track from the room, which in this case is the 1FPS stream coming from the glasses.

Running the Agent

To start your agent in development mode and make it accessible globally via LiveKit Cloud, simply run:

uv run agent.py dev

Find the full Gemini Live vision agent example in the LiveKit docs.

Connection & Authentication

To connect your frontend to LiveKit, you need a short-lived access token.

CLI Token Generation

For testing and demos, you can quickly generate a token using the LiveKit CLI:

lk token create \
  --api-key <YOUR_API_KEY> \
  --api-secret <YOUR_API_SECRET> \
  --join \
  --room <ROOM_NAME> \
  --identity <PARTICIPANT_IDENTITY> \
  --valid-for 24h

In a production environment, you should always issue tokens from a secure backend to keep your API secrets safe.

The Frontend: Meta Wearables Integration

This example targets Android devices (like the Google Pixel). You'll need the Meta Wearables Toolkit and the specific sample project.

Clone the Sample: Get the Android client example.
Configure local.properties: Add your GitHub Token as required by the Meta SDK.
Update Connection Details: In StreamScreen.kt, replace the server URL and token with your LiveKit details:

// streamViewModel.connectToLiveKit
connectToLiveKit(
    url = "wss://your-project.livekit.cloud",
    token = "your-generated-token"
)

Run the App: Connect your device via USB and deploy from Android Studio.

Conclusion

By bridging Meta Wearables with Gemini Live via LiveKit, we've created a powerful, low-latency vision AI experience. This architecture is scalable and secure, providing a foundation for the next generation of wearable AI applications.

Resources

Happy hacking! 🚀

Singapore vibes together at the new Google DeepMind office

Thor 雷神 Schaeff — Tue, 23 Dec 2025 01:27:22 +0000

You weren't able to join for the vibe coworking session? Fear not, we're already planning the Gemini 3 Hackathon in Singapore for January 10. Apply here!

Recently, we hosted a hundred builders at the new Google DeepMind Singapore office for a vibe coding session with Google AI Studio and the Gemini API. Here are some of the cool things they built!

HireCoach

Chung Nguyen built HireCoach, an interactive job interview app that uses Gemini to present you with mock interview questions and then uses Gemini’s audio understanding capabilities to analyze your answers and give you actionable feedback on how to improve. Try it out in Google AI Studio.

Fridge Forager

Jo Ann Low built Fridge Forager that uses Gemini’s multimodal capabilities to understand the contents of your fridge and come up with creative recipes based on the ingredients you have available. During the demo, there wasn’t any fridge at hand, so we just used Nano Banana Pro to generate the contents of a Singaporean fridge which turned out to be a crowd pleaser. Try it out in Google AI Studio.

Yalom’s Garden

Onno Kampman built Yalom’s Garden, an online resilience and wellbeing garden helping you reflect and grow in a beautifully designed virtual garden. Try it out in Google AI Studio.

Mind Palace

Zeng Hanyi built Mind Palace, which combines Sherlock Holmes' Mind Palace and Notebook LM in a beautiful interactive voxel app. Try it out in Google AI Studio.

Gemini Flight Commander

Gabrielle Ong built Gemini Flight Commander, an immersive tactical drone simulator. It allows users to pilot through any global location or coordinate from a pilot's immersive view, while receiving real-time intelligence reports generated by Gemini. It uses the Gemini API for geocoding, summary of location, web search for links of locations, and Google Maps API grounding. Try it out in Google AI Studio.

Join the Singapore Gemini 3 Hackathon in January

You weren't able to join for the vibe coworking session? Fear not, we're already planning the Gemini 3 Hackathon in Singapore for January 10. Apply here!

Local and offline AI code assistant for VS Code with Ollama and Sourcegraph

Thor 雷神 Schaeff — Fri, 07 Jun 2024 14:51:33 +0000

I recently learned that Sourcegraph's AI coding assistant Cody can be used offline by connecting it to a local running Ollama server.

Now, unfortunately my little old MacBook Air doesn't have enough VRAM to run Mistral's 22B Codestral model, but fear not, I found that the Llama 3 8B model works quite well in powering both code completion and code chat workloads!

Let's have a look at how we can set this up with VS Code for the absolute offline / in-flight coding bliss:

Install Ollama and pull Llama 3 8B

Install Ollama
Run ollama pull llama3:8b
Once the downloade has completed, run ollama serve to start the Ollama server.

Configure Sourcegraph Cody in Vs Code

Install the Sourcegraph Cody Vs Code Extension.
Add the following to your Vs Code settings:

{
  //...
  // Cody autocomplete configuration:
  "cody.autocomplete.advanced.provider": "experimental-ollama",
  "cody.autocomplete.experimental.ollamaOptions": {
    "url": "http://127.0.0.1:11434",
    "model": "llama3:8b"
  },
  // Enable Ollama for Cody Chat:
  "cody.experimental.ollamaChat": true,
  // optional but useful to see detailed logs in the OUTPUT tab
  // (make sure to select "Cody by Sourcegraph" from the dropbdown)
  "cody.debug.verbose": true
  //...
}

Start Cody and enjoy your Local Offline AI Code Assistant

That's it, as long as Ollama is running in the background, you should now have a fully functional offline AI code assistant for Vs Code with Cody. This setup allows you to use both code completion and code chat features without relying on any external services or internet connection. In fact most of this last paragraph was written by Llama 3 8B itself.

For Cody Chat, make sure to select the llama3:8b Experimental option from the dropdown and you're good to go! Happy Cod(y)ing \o/

Getting started with Ruby on Rails and Postgres on Supabase

Thor 雷神 Schaeff — Mon, 29 Jan 2024 20:41:55 +0000

Every Supabase project comes with a full Postgres database, a free and open source database which is considered one of the world's most stable and advanced databases.

Postgres is an ideal choice for your Ruby on Rails applications as Rails ships with a built-in Postgres adapter!

In this post we'll start from scratch, creating a new Rails project, connecting it to our Supabase Postgres database, and interacting with the database using the Rails Console.

Supabase is one of the best free alternatives to Heroku Postgres. See this guide to learn how to migrate from Heroku to Supabase.

There's also a Heroku to Supabase migration tool available to migrate in just a few clicks.

If you prefer video guide, you can follow along below. And make sure to subscribe to the Supabase YouTube channel!

Create a Rails Project

Make sure your Ruby and Rails versions are up to date, then use rails new to scaffold a new Rails project. Use the -d=postgresql flag to set it up for Postgres.

Go to the Rails docs for more details.

rails new blog -d=postgresql

Set up the Postgres connection details

Go to database.new and create a new Supabase project. Save your database password securely.

When your project is up and running, navigate to the database settings to find the URI connection string.

Rails ships with a Postgres adapter included, you can simply configure it via the environment variables. You can find the database URL in your Supabase Dashboard.

export DATABASE_URL=postgres://postgres.xxxx:password@xxxx.pooler.supabase.com:5432/postgres

Create and run a database migration

Rails includes Active Record as the ORM as well as database migration tooling which generates the SQL migration files for you.

Create an example Article model and generate the migration files.

bin/rails generate scaffold Article title:string body:text
bin/rails db:migrate

Use the Model to interact with the database

You can use the included Rails console to interact with the database. For example, you can create new entries or list all entries in a Model's table.

bin/rails console

article = Article.new(title: "Hello Rails", body: "I am on Rails!")
article.save # Saves the entry to the database

Article.all

Start the app

bin/rails server

Run the development server. Go to http://127.0.0.1:3000 in a browser to see your application running.

Update the app to show articles

Currently the app shows a nice development splash screen, let's update this to show our articles from the database:

Rails.application.routes.draw do
  # Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html

  # Defines the root path route ("/")
  root "articles#index"
end

Deploy to Fly.io

In order to start working with Fly.io, you will need flyctl, our CLI app for managing apps. If you've already installed it, carry on. If not, hop over to the installation guide. Once that's installed you'll want to log in to Fly.

Provision Rails with Fly.io

To configure and launch the app, you can use fly launch and follow the wizard.

When asked "Do you want to tweak these settings before proceeding?" select y and set Postgres to none as we will be providing the Supabase database URL as a secret.

Set the connection string as secret

Use the Fly.io CLI to set the Supabase database connection URI from above as a sevret which is exposed as an environment variable to the Rails app.

fly secrets set DATABASE_URL=$DATABASE_URL

Deploy the app

Deploying your application is done with the following command:

fly deploy

This will take a few seconds as it uploads your application, builds a machine image, deploys the images, and then monitors to ensure it starts successfully. Once complete visit your app with the following command:

fly apps open

That's it! You're Rails app is up and running with Supabase Postgres and Fly.io!

Conclusion

Supabase is the ideal platform for powering your Postgres database for your Ruby on Rails applications! Every Supabase project comes with a full Postgres database and a good number of useful extensions!

Try it out now at database.new!

More Supabase

🚀 Learn more about Supabase

Auto-generated documentation in Supabase

Thor 雷神 Schaeff — Tue, 05 Jul 2022 15:10:40 +0000

The need for good API documentation

Every developer knows that documentation is an essential aspect of API adoption. With comprehensive API documentation that explains all features, developers can quickly adapt and integrate them into other applications. In addition to increasing API adoption, good documentation also improves the development process in several ways:

Simplified project management

In many projects, tasks are broken down as "Jobs to be done". Using an API to model a job task is common across several projects. Without a well-designed and documented API, you and your team members will be forced to spend more time copying data between tools and completing tasks.

Improved development velocity

Good documentation is a perfect way for app builders across your organization to self-serve their application needs. It makes information more accessible and easier to find, empowering new members to figure things out on their own. They can easily see how to work with APIs and quickly get up to speed on projects.

Better organizational consistency

Good API documentation fosters better consistency and standards within an organization. Just like designers and writers have a style guide for content, good API documentation serves as the style guide for developers using an API.

Why you should auto-generate API documentation

Well-written API documentation can make your APIs look and feel professional, but it can be a time-consuming task to maintain manually. Throughout the life-cycle of your application, the API documentation needs to be kept up to date. For example, if you create a new table, or change the schema of an existing table, you will need to update the documentation accordingly. Thanks to Supabase's auto-generated documentation, this can now be done easily. You can now automatically update documentation pages to reflect application changes and reduce the effort needed to keep up with documentation drift.

Stripe is a great example of an organization that has excellent API documentation and is highly regarded by API consumers. Did you know that they also make use of auto-generated documentation?

Getting started with auto-generated Supabase documentation

Supabase is built for developers and you can get started for free using your existing Github account. Once your Supabase account is set up, you will be able to access the Supabase dashboard. From here, go to All Project > New Project.

Give your project a name and set the database password. You can also choose the region and adjust the pricing plan based on the requirements of your project.

Once you create a new project, you will see the services offered by Supabase which includes a hosted database built on top of PostgreSQL. You can start creating tables in your database by clicking the Table editor button.

Here you can create your own tables and start defining your table schema.

Once you create a table, you can then add rows to the tables using the Table Editor.

If you create tables or update table schemas, API documentation will be auto-generated. You can then view the auto-generated API documents by going to the API section from the left menu.

The view on the right shows you how you can connect to your Supabase project from within Node.js applications using the createClient class from the supabase-js npm package.

For authentication, you can find your Supabase API keys by going to Settings > Project Settings > API.

Once your application has been created, you can use it to interact with your database tables. The table-specific documentation that is auto-generated gives you code snippets for the operations available. To view table-specific API documentation, go to API > API Docs > Tables and Views > {YOUR_TABLE}

As soon as you select the table to view, you'll see the schema definition. You can even add additional descriptions to your table fields to improve the documentation even more.

The section on the right contains the auto-generated documentation for your table. These are code snippets that can be plugged into the Supabase client application from the previous step to perform database operations. The first set of code snippets shows you how to select all values from the table for a specific field.

Scrolling down further, you can see the auto-generated documentation that lists code snippets for other CRUD operations using the Supabase client.

If you're building a real-time application, the "Subscribe to changes" section might be interesting. The code snippets show how you can subscribe to all or specific events. These events are triggered whenever changes are made to the table.

Supabase also auto-generates CURL requests for CRUD operations to run against your tables. This gives users a way to quickly interact with their tables if you're using the Linux command line.

Conclusion

The importance of creating good API documentation can't be stressed enough, and

there's no need to try to reverse engineer code written by others in order to create documentation. With Supabase's API doc auto-generation, you can skip the tedious task of creating documentation and focus more time on building your application.

It's no surprise why developers are drifting to the Supabase way!

Now, become a Supabase developer today by starting a new project on our free tier

How I built a sticker marketplace in less than a week

Thor 雷神 Schaeff — Wed, 27 May 2020 09:37:57 +0000

Timing is key

I've been playing with the idea of creating and selling my own stickers for a while now, but I've never gone through with it. A couple weeks back I partnered with Jason Lengstorf on a collection of videos about doing business on the Jamstack. In one of the episodes we built a sticker store for his learnwithjason.dev site, and Jason casually mentioned that I should make a hammer sticker, which wouldn't let me go. So when I came across sosplush.com (check out her seriously awesome stickers) and saw that she was open for commissions, everything fell into place. I messaged her on Twitter on Sunday night, she liked the idea, and by Friday morning we had made our first sale on thorsticker.store. What an exciting couple of days. 🥳

When I first contacted SoSplush, my plan was to have her develop the cute Mjölnir character, find a printer, get some stickers printed, and then figure out fulfillment. Knowing SoSplush sells here own awesome stickers, I asked how she handles printing and fulfillment. She mentioned she had switched to printing them since the current COVID19 situation has temporarily delayed many sticker producers. That's when it hit me: Rather than selling the stickers myself, I should turn this into a marketplace with Stripe Connect and outsource fulfillment to the capable hands of others. It was time to get to work. 🔨

Setting up a Connect Marketplace

With SoSplush on board as the first seller, I went ahead and created a new Stripe account. Next, I turned my account into a platform account. That was something I hadn't done in a while, and I was happy to see that Stripe Connect now includes a guided onboarding experience that helped me choose the right marketplace setup for my scenario. I was going to use Standard Connect and create the payments directly on SoPlush's Stripe account without taking any commission fees (I don't want to make money from this, I just want to see some cute Mjölnirs out there 😉)

Since I wouldn't be earning any money with this project, it was important to find a way to get started without too much upfront and ongoing cost. This is where Netlify comes in - they have a more than generous free tier for both building & hosting static sites, and serverless functions - exactly what I needed for this project.

Next, I had to connect SoSplush's Stripe account with my platform account, which happens via OAuth. Kicking off the OAuth process happens via a static link that includes your connect application ID. So after dropping my connect app ID into the Netlify environment settings I only needed a static page with a "Connect with Stripe" button (view source).

import React from "react";

const oAuthURL = `https://connect.stripe.com/oauth/authorize?response_type=code&client_id=${process.env.REACT_APP_STRIPE_CLIENT_ID}&scope=read_write`;

const Connect = () => {
  return (
    <a
      className="stripe-connect"
      href={oAuthURL}
      target="_blank"
      rel="noopener noreferrer"
    >
      <span>Connect with Stripe</span>
    </a>
  );
};

export default Connect;

After the account owner approves the connection, Stripe redirects them to a URL where you have to finalise the connection with an authorisation code. For this I've set up a Netlify function and set Stripe Connect to redirect to it (view source).

const stripe = require("stripe")(process.env.STRIPE_SECRET_KEY, {
  apiVersion: "2020-03-02",
  maxNetworkRetries: 2,
});

exports.handler = async ({ queryStringParameters }) => {
  let responseMessage = `Connection failed`;
  const { code } = queryStringParameters;

  if (code) {
    try {
      await stripe.oauth.token({
        grant_type: "authorization_code",
        code,
      });

      responseMessage = `Successfully connected`;
    } catch (error) {
      responseMessage = `${responseMessage}: ${error.message}`;
    }
  }

  return {
    statusCode: 200,
    body: responseMessage,
  };
};

Checkout & Connect

With SoSplush' Stripe account connect, we now were a real platform. To create CheckoutSession directly on the connected account, I need to set an account header with their Stripe account ID, which is returned in the connection request above. In a scalable marketplace scenario, you want to store that account ID in your database. In my case, with only a small amount of sellers, I decided that it's fine to store their account ID in an environment variable.

With only one product being sold on this site, I decided to hardcode the product information in my React app (view source). It includes two hidden input fields, one for the unique product identifier (here called sku for Stock Keeping Unit), and one for the seller ID.

<form onSubmit={handleSubmit}>
  <input type="hidden" name="sku" value="thorwebdev_standard" />
  <input type="hidden" name="seller" value="SOSPLUSH" />
  <div className="quantity-setter">
    <button
      type="button"
      className="increment-btn"
      disabled={state.quantity === 1}
      onClick={() => dispatch({ type: "decrement" })}
    >
      -
    </button>
    <input
      type="number"
      id="quantity"
      name="quantity"
      min="1"
      max="10"
      value={state.quantity}
      readOnly
    />
    <button
      type="button"
      className="increment-btn"
      disabled={state.quantity === 10}
      onClick={() => dispatch({ type: "increment" })}
    >
      +
    </button>
  </div>
  <button role="link" type="submit" disabled={state.loading}>
    {state.loading || !state.price ? `Loading...` : `Buy for ${state.price}`}
  </button>
</form>

When the form is submitted, we send a POST request with the quantity, the sku ID, and the seller ID to a Netlify function which creates a Checkout Session on the connected Stripe account (view source). In this function we retrieve the product's pricing information from a JSON file and validate that the sent quantity is within our limits (1-10) as to not exceed letter shipping weight. Note that you should never blindly trust information coming from the client as it could have been tempered with. That's why we load the product data from a JSON file, or in a more scalable application from the database, and validate the quantity.

const stripe = require("stripe")(process.env.STRIPE_SECRET_KEY, {
  apiVersion: "2020-03-02",
  maxNetworkRetries: 2,
});

/*
 * Product data can be loaded from anywhere. In this case, we’re loading it from
 * a local JSON file, but this could also come from an async call to your
 * inventory management service, a database query, or some other API call.
 *
 * The important thing is that the product info is loaded from somewhere trusted
 * so you know the pricing information is accurate.
 */
const inventory = require("./data/products.json");
const shippingCountries = require("./data/shippingCountries.json");

exports.handler = async event => {
  const { sku, quantity, seller } = JSON.parse(event.body);
  const product = inventory.find(p => p.sku === sku);

  // ensure that the quantity is within the allowed range
  const validatedQuantity = quantity > 0 && quantity < 11 ? quantity : 1;

  const session = await stripe.checkout.sessions.create(
    {
      payment_method_types: ["card"],
      billing_address_collection: "auto",
      shipping_address_collection: {
        allowed_countries: shippingCountries,
      },

      /*
       * This env var is set by Netlify and inserts the live site URL. If you want
       * to use a different URL, you can hard-code it here or check out the
       * other environment variables Netlify exposes:
       * https://docs.netlify.com/configure-builds/environment-variables/
       */
      success_url: `${process.env.URL}/success`,
      cancel_url: process.env.URL,
      line_items: [
        {
          name: product.name,
          description: product.description,
          images: [product.image],
          amount: product.amount,
          currency: product.currency,
          quantity: validatedQuantity,
        },
      ],
      // We are using the metadata to track which items were purchased.
      // We can access this meatadata in our webhook handler to then handle
      // the fulfillment process.
      // In a real application you would track this in an order object in your database.
      metadata: {
        items: JSON.stringify([
          {
            sku: product.sku,
            name: product.name,
            quantity: validatedQuantity,
          },
        ]),
      },
    },
    {
      stripeAccount: process.env[seller],
    }
  );

  return {
    statusCode: 200,
    body: JSON.stringify({
      sessionId: session.id,
      stripeAccount: process.env[seller],
    }),
  };
};

After creating a Checkout Session on the connected account, the function returns the session ID as well as the seller account ID. These are the two IDs we need to initiate the redirect to Stripe Checkout from our client (view source).

const handleSubmit = async event => {
  event.preventDefault();
  dispatch({ type: "setLoading", payload: { loading: true } });

  const form = new FormData(event.target);

  const data = {
    sku: form.get("sku"),
    seller: form.get("seller"),
    quantity: Number(form.get("quantity")),
  };
  console.log({ data });
  const { sessionId, stripeAccount } = await fetch(
    "/.netlify/functions/create-checkout",
    {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(data),
    }
  ).then(res => res.json());

  const stripe = await loadStripe(
    process.env.REACT_APP_STRIPE_PUBLISHABLE_KEY,
    { stripeAccount }
  );
  const { error } = await stripe.redirectToCheckout({
    sessionId,
  });

  if (error) {
    alert(error.message);
    dispatch({ type: "setLoading", payload: { loading: false } });
  }
};

Managing order fulfillment

Since we started off with one product and did not expect any flash sales like numbers (do feel free to prove me wrong 😉), fulfillment is being handled manually via email notifications from Stripe. In your Stripe profile settings you can enable to get an email notification any time you make a sale. So when someone puts through an order, SoSplush will receive an email notification from Stripe and then warm up the printing press.

A note on shortcuts and scalability

As you probably noticed while reading, there were a couple of instances where I took some major shortcuts that will only work when you have a small amount of products, sellers, and orders. For example, I'm storing seller account IDs in the environment variables, and product data in a static JSON file rather than a database. I'm also not tracking any inventory, which is fine as the small amount of orders we're expecting can be printed on demand, and fulfillment is handled manually via email notifications and the Stripe Dashboard.

That being said, we were able to go from idea to first sale in less than a week with minimal upfront investment. When needed, I can add on a database for product and inventory management and for scalable seller management. I can extend my application with additional third-party APIs, or replace them with my own APIs as needed, but only when needed. For me, that's the beauty of the Jamstack with Netlify and Stripe to enable quickly testing your online business ideas.

What's next

With a couple of sales in the "bank" (mainly from my American colleagues and friends), I'm looking to add multi-currency support for all my global friends, as well as adding some more payment methods.

SoSplush and I have also been talking about her own custom storefront to sell her amazing stickers and other products. Go check them out on sosplush.com!

Lastly, I'd love to hear any feedback and suggestion you have, both on functionality for thorsticker.store and things you'd love to read and learn about. Please do reach out on Twitter 🐦

// Detect dark theme var iframe = document.getElementById('tweet-1263535714777456640-112'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1263535714777456640&theme=dark" }