Local Llm Ops

What it does

This skill, Local LLM Ops (Ollama), provides a complete toolchain for running local Large Language Models (LLMs) on Apple Silicon devices. It includes setup scripts, command-line chat launchers, benchmarks, and diagnostic tools to manage the entire lifecycle of your LLM applications – from installation and model pulling to benchmarking performance. The skill streamlines the process of setting up and using LLMs locally, allowing for experimentation and development without relying on external services.

When to use it

• You want to run Large Language Models (LLMs) locally on an Apple Silicon Mac.
• You need a quick way to benchmark the performance of different LLM models.
• You're encountering issues setting up or running your local LLM environment and require diagnostic assistance.
• You want to experiment with different task modes like coding, creative writing, or analytical tasks using various LLMs.

Key capabilities

• Automated Setup: Provides a setup_chatbot.sh script for easy installation and configuration of Ollama.
• Chat Launchers: Offers multiple command-line chat launchers (chatllm, chat, chat.py) for interacting with LLMs.
• Task Modes: Supports different task modes (coding, creative, analytical) using specific models like codellama:70b and llama3.1:70b.
• Benchmarking: Includes a benchmark workflow (./scripts/run_benchmarks.sh) to evaluate LLM performance with configurable prompts and token limits.
• Diagnostics: Provides a diagnostic script (./diagnose.sh) to troubleshoot setup issues.

Example prompts

• ./chatllm - Launches the primary chat launcher for interacting with your local LLMs.
• ./chat -t coding -m codellama:70b - Starts a chat session in "coding" mode using the codellama:70b model.
• ./scripts/run_benchmarks.sh - Executes the benchmark script to measure LLM performance.

Tips & gotchas

• Prerequisites: Requires Homebrew (brew) for installing Ollama and managing services.
• Model Availability: You must pull at least one model (e.g., ollama pull mistral) before launching chat or benchmarks.
• API Endpoint: The Ollama API runs on http://localhost:11434.