Batch normalization fusion for PyTorch. This is an archived repository, which is not maintained.
-
Updated
Apr 6, 2020 - Python
#
inference-optimization
Here are 138 public repositories matching this topic...
Optimize layers structure of Keras model to reduce computation time
-
Updated
Jul 18, 2020 - Python
Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agentic AI.
ai-agents context-api inference-optimization prompt-engineering context-engineering openclaw hermes-agent
-
Updated
Jul 13, 2026 - Python
[CVPR 2025] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
pruning inference-optimization multi-modality llm token-pruning vision-language-model llava multimodal-large-language-models
-
Updated
Apr 16, 2026 - Python
A set of tool which would make your life easier with Tensorrt and Onnxruntime. This Repo is designed for YoloV3
-
Updated
Dec 31, 2019 - Python
Run 70B+ LLMs on a single 4GB GPU — no quantization required.
-
Updated
Feb 28, 2026 - Python
Official Repo for SparseLLM: Global Pruning of LLMs (NeurIPS 2024)
pruning model-compression inference-optimization alternating-optimization large-language-models efficient-ai
-
Updated
Mar 27, 2025 - Python
The blog, read report and code example for AGI/LLM related knowledge.
-
Updated
Feb 1, 2025 - Python
Optimizing Monocular Depth Estimation with TensorRT: Model Conversion, Inference Acceleration, and 3D Reconstruction
optimization raft depth-estimation tensorrt inference-optimization monocular-depth-estimation 2dto3d depth-pro depth-anything-v2 vggt distill-any-depth depth-anything-ac unidepthv2 metric3dv2 unik3d moge-2 streamvggt meflow memfof neuflow
-
Updated
Mar 9, 2026 - Python
TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs
compression transformer triton quantization inference-optimization kv-cache llm vllm consumer-gpu turboquant
-
Updated
Apr 10, 2026 - Python
LightTTS is a lightweight TTS inference framework optimized for CosyVoice2 and CosyVoice3, enabling fast and scalable speech synthesis in Python and supports stream and bistream modes.
text-to-speech real-time tts speech-synthesis low-latency tensorrt inference-optimization audio-generation cosyvoice cosyvoice2 cosyvoice3
-
Updated
Apr 14, 2026 - Python
Official code of Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
multi-agent multi-agent-systems inference-optimization llms llm-inference llm-agents mixture-of-agents multi-agent-collaboration
-
Updated
May 6, 2026 - Python
Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.
machine-learning deep-learning gpu cuda transformers self-hosting quantization roofline mlops inference-optimization kv-cache llm llmops llm-serving vllm llm-inference flash-attention speculative-decoding pagedattention sglang
-
Updated
Jul 1, 2026 - Python
Official implementation of "SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching" (COLM 2025). A novel KV cache compression method that organizes cache at sentence level using semantic similarity.
natural-language-processing transformers memory-efficiency efficient-inference inference-optimization kv-cache llm semantic-caching colm2025
-
Updated
Sep 29, 2025 - Python
Python SDK for WildEdge
ai tensorflow keras inference pytorch observability mlops inference-optimization edge-ai on-device-ai onnxruntime ai-engineering llm llm-inference llm-ops llm-observability edge-ai-agents inference-statistics
-
Updated
Jul 14, 2026 - Python
Dynamic Attention Mask (DAM) generate adaptive sparse attention masks per layer and head for Transformer models, enabling long-context inference with lower compute and memory overhead without fine-tuning.
-
Updated
Jun 16, 2025 - Python
Code for paper "EdgeKE: An On-Demand Deep Learning IoT System for Cognitive Big Data on Industrial Edge Devices"
iot deep-neural-networks deep-learning inference resnet knowledge-distillation edge-computing edge-devices resnets inference-optimization early-exit jetson-nano branchynet edge-applications accuracy-requirements efficient-ai
-
Updated
Jun 18, 2026 - Python
Accelerating LLM inference with techniques like speculative decoding, quantization, and kernel fusion, focusing on implementing state-of-the-art research papers.
-
Updated
Jul 1, 2025 - Python
Adaptive speculative-decoding inference engine with Triton-optimised verification and online bandit draft selection.
machine-learning triton bandit-algorithms inference-optimization llm-inference speculative-decoding production-inference
-
Updated
Jul 9, 2026 - Python
Improve this page
Add a description, image, and links to the inference-optimization topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the inference-optimization topic, visit your repo's landing page and select "manage topics."