TorchServe is a performant, flexible and easy-to-use tool for serving PyTorch eager mode and torschripted models. Multi-model management with the optimized worker to model allocation. REST and gRPC support for batched inference. Export your model for optimized inference. Torchscript out of the box, ORT, IPEX, TensorRT, FasterTransformer. Performance Guide: built-in support to optimize, benchmark and profile PyTorch and TorchServe performance. Expressive handlers: An expressive handler architecture that makes it trivial to support inferencing for your use case with many supported out of the box. Out-of-box support for system-level metrics with Prometheus exports, custom metrics and PyTorch profiler support.
Features
- REST and gRPC support for batched inference
- Deploy complex DAGs with multiple interdependent models
- Default way to serve PyTorch models
- Export your model for optimized inference
- Performance Guide
- Metrics API
Project Samples
License
Apache License V2.0Follow TorchServe
You Might Also Like
Gen AI apps are built with MongoDB Atlas
MongoDB Atlas provides built-in vector search and a flexible document model so developers can build, scale, and run gen AI apps without stitching together multiple databases. From LLM integration to semantic search, Atlas simplifies your AI architecture—and it’s free to get started.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of TorchServe!