LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

TensorRT-LLM — a leading open-source project in the AI/LLM ecosystem.

A framework for efficient model inference with omni-modality models

Olares: An Open-Source Personal Cloud to Reclaim Your Data

🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials.

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Superlinked Inference Engine is an Open-source inference server and production cluster for embeddings, reranking, and extraction.

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

Community maintained hardware plugin for vLLM on Ascend

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

Turn any computer or edge device into a command center for your computer vision projects.

🏕️ Reproducible development environment for humans and agents

AICI: Prompts as (Wasm) Programs

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

SGLang-Omni empowers high-performance serving for TTS, ASR, speech and omni models.

An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Hopsworks - Data-Intensive AI platform with a Feature Store

The simplest way to serve AI/ML models in production

Fast GPU OCR server. 270 img/s on FUNSD. TensorRT FP16, PP-OCRv5, HTTP + gRPC.

A throughput-oriented high-performance serving framework for LLMs

A scalable inference server for models optimized with OpenVINO™

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

An open-source computer vision framework to build and deploy apps in minutes

Model Deployment at Scale on Kubernetes 🦄️

Serverless LLM Serving for Everyone.

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

FastAPI Skeleton App to serve machine learning models production-ready.

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

xFasterTransformer — open-source AI/LLM project.

A REST API for Caffe using Docker and Go

Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services working together via the Lightning Apps framework.

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Work with LLMs on a local environment using containers

Auto-tuned launcher for GGUF models on llama.cpp / ik_llama.cpp — OpenAI-compatible server with multi-GPU tensor-split, MoE expert placement, measured flag tuning (AI Tune), hardware-matched HuggingFace downloads, and crash recovery. An Ollama alternative for multi-GPU rigs.

Auto-tuned launcher for GGUF models on llama.cpp / ik_llama.cpp — OpenAI-compatible server with multi-GPU tensor-split, MoE expert placement, measured flag tuning (AI Tune), hardware-matched HuggingFace downloads, and crash recovery. An Ollama alternative for multi-GPU rigs.

This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.

This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.