Ai Llm Inference

What it does

This skill provides production-ready operational patterns for optimizing Large Language Model (LLM) inference performance, cost, and reliability. It centralizes decision rules, optimization strategies, configuration templates, and operational checklists specifically for LLM inference workloads. The focus is on practical execution; the skill avoids theoretical explanations and focuses on what Codex can directly implement.

When to use it

This skill is useful in the following scenarios:

• Optimizing LLM inference latency or throughput
• Choosing quantization strategies (FP8/FP4/INT8/INT4)
• Configuring vLLM, TensorRT-LLM, or DeepSpeed inference
• Scaling LLM inference across GPUs using tensor/pipeline parallelism
• Building high-throughput LLM APIs

Key capabilities

• SLO Management: Defines and monitors Service Level Objectives (SLOs) for latency.
• Tail Latency Control: Implements scheduling, timeouts, admission control, queue caps, and backpressure to prevent latency spikes.
• Throughput Optimization: Utilizes continuous batching and KV-cache aware serving techniques like PagedAttention/vLLM.
• Cost Reduction: Employs model tiering and caching strategies.
• Long Context Handling: Optimizes prefill processes for long input sequences.
• Parallelism Strategies: Supports tensor, pipeline, and data parallelism to distribute models across GPUs.
• Reliability Patterns: Implements timeouts, circuit breakers, and idempotency for high availability.
• Speculative Decoding: Utilizes speculative decoding when latency is critical and a slightly lower quality draft model is acceptable.

Example prompts

Here are some example prompts you could give an AI agent with this skill:

• "How can I reduce the inference latency of my LLM API?"
• "What quantization strategies should I use to reduce cost per token?"
• "Configure vLLM for optimal performance."
• “Scale my LLM across multiple GPUs.”

Tips & gotchas

• Quantization Validation: Always validate any quantization strategy on your evaluation dataset to ensure acceptable quality.
• Scope Limitations: This skill focuses solely on inference optimization and does not cover prompting, tuning, RAG pipeline construction, deployment, or safety/governance—those are handled by other skills.
• Practical Focus: The skill provides actionable strategies; it doesn't delve into the underlying theory.