Expand description
This module provides functionality for working with streaming completion models. It provides traits and types for generating streaming completion requests and handling streaming completion responses.
The main traits defined in this module are:
- StreamingPrompt: Defines a high-level streaming LLM one-shot prompt interface
- StreamingChat: Defines a high-level streaming LLM chat interface with history
- StreamingCompletion: Defines a low-level streaming LLM completion interface
Structs§
- Pause
Control - Control for pausing and resuming a streaming response
- Streaming
Completion Response - The response from a streaming completion request;
message and response are populated at the end of the
innerstream.
Enums§
- RawStreaming
Choice - Enum representing a streaming chunk from the model
- Streamed
Assistant Content - Describes responses from a streamed provider response which is either text, a tool call or a final usage response.
Traits§
- Streaming
Chat - Trait for high-level streaming chat interface
- Streaming
Completion - Trait for low-level streaming completion interface
- Streaming
Prompt - Trait for high-level streaming prompt interface
Functions§
- stream_
to_ stdout - helper function to stream a completion request to stdout
Type Aliases§
- Streaming
Result Non-WebAssembly