Open Source · Resources

Jump to a section

Open-source projects, my Claude Code setup bundle, CLI install guides, and a local-model setup advisor — pick one.

★ Featured

Project Spotlight

🏏

● Live Python

IPL 2026 Predictor

An XGBoost-powered IPL match winner predictor trained on 18 seasons of historical data. Includes a Streamlit app with interactive team selection, impact-player logic, and live win probabilities.

Machine Learning XGBoost Streamlit Cricket Analytics

View on GitHub Open Live App Kaggle Notebook Kaggle Dataset

🖥️

Open WebUI Desktop

● Active

Electron desktop app for Open WebUI — pure black UI, venv installer, and system tray integration. Run your local AI chat interface as a native desktop app without touching the terminal every time.

Electron Desktop App Open WebUI Local AI JavaScript

View on GitHub

🧠

Local AI Memory

● Active

A local memory system for AI assistants that runs entirely on your own hardware. Store and retrieve context without cloud dependencies — full privacy, zero latency.

Local AI Memory Privacy Open Source

View on GitHub

✍️

GBlog Skill

● Active

A Claude Code skill for generating and publishing blog posts directly to Google Blogger. Automate your blogging workflow with AI. Also available as a Claude skill on Clawhub.

Claude Blogging Automation AI Skill

View on GitHub Use on Clawhub

🚀

More Projects Coming Soon

Building more tools and skills in public. Follow on GitHub to stay updated.

Follow on GitHub

Claude Code Setup Bundle

My entire Claude Code config in one download. Plugins, skills, rules, hooks, agents, commands — hand the file + prompt to Claude Code, Cursor, or Codex and it sets up everything on your machine.

plugins

custom skills

slash commands

sub-agents

Download setup.md View on GitHub

🤖

How it works

Download setup.md, open the Agent Prompt and copy it, then paste both into Claude Code / Cursor / Codex. The agent backs up your existing ~/.claude/, clones the companion repo, merges files non-destructively, and installs 11 plugins. Takes about a minute.

🎬 YouTube Shorts Series

CLI Tools for AI Agents

Install guides for every CLI worth having in your AI workflow. Part of the ongoing TechRex Shorts series.

🤖

Why this page exists

ML & Models

● Live

Download models and datasets, push repos, manage auth tokens, and run Hugging Face Hub ops from the terminal.

Official docs ↗

▲

Deployment · Install Guide

1
Install Vercel CLI globally via npm

Requires Node.js 18+. Global flag puts the `vercel` binary on your PATH so any agent shell can invoke it.
```
npm install -g vercel
```
2
(Alternative) Install via pnpm, yarn, or bun

Any Node package manager works. Pick the one you already use.
```
pnpm add -g vercel
```
3
Log in to your Vercel account

Opens a browser tab. Pick the same sign-in method (GitHub, GitLab, Bitbucket, or email) you use on vercel.com. Token is saved to `~/.local/share/com.vercel.cli/auth.json`.
```
vercel login
```
4
Link a local project

Run inside your project directory. Creates a `.vercel/` folder with the project + org IDs so every subsequent command targets this project.
```
vercel link
```
5
(Optional) Pull production env vars

Writes `.env.local` from your Vercel project — handy for local dev and for AI agents that need the same secrets.
```
vercel env pull
```

Verify installation

vercel --version

1
Install Node.js LTS (prerequisite)

Vercel CLI is a Node package. In an elevated PowerShell, install the LTS build. Skip this step if `node --version` already returns 18+.
```
winget install OpenJS.NodeJS.LTS
```
2

Open a FRESH PowerShell window

The Node installer modifies PATH. Existing terminals will not see `node`/`npm` until you reopen them.
3
Install Vercel CLI globally

Global flag is what makes `vercel` available to any shell / AI agent.
```
npm install -g vercel
```
4
Log in to your Vercel account

Opens your default browser. Pick the same sign-in method you use on vercel.com. The access token is stored at %APPDATA%\com.vercel.cli\auth.json.
```
vercel login
```
5
Link a local project

From your project directory. Select the Vercel team + project to link. Creates a `.vercel/` folder you should NOT commit.
```
vercel link
```
6
(Optional) Pull production env vars

Writes `.env.local` so local + agent commands see the same secrets as production.
```
vercel env pull
```

💡 Avoid installing Node via the Microsoft Store — it sandboxes global installs and `npm install -g` silently fails. If you see EACCES errors, follow npm's permissions guide: https://docs.npmjs.com/resolving-eacces-permissions-errors.

Verify installation

vercel --version

1
Install Node.js 18+ via nvm (recommended)

nvm avoids the `sudo npm -g` permissions dance and lets you switch versions per project.
```
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash && source ~/.bashrc && nvm install --lts
```
2
Install Vercel CLI globally
```
npm install -g vercel
```
3
Log in to your Vercel account

Prints a URL + one-time code. Copy the URL to a browser (useful on headless boxes).
```
vercel login
```
4
Link a local project
```
vercel link
```
5
(Optional) Pull production env vars
```
vercel env pull
```

Verify installation

vercel --version

🤖

Use it with an AI agent

Once installed + authenticated, just tell Claude Code / Cursor / Codex:

"Use the Vercel CLI to deploy this project to production" · "Pull my production env vars into .env.local" · "List my last 10 deployments"

The agent runs the CLI in your shell with the credentials you already saved — no extra config.

Official docs ↗

🐙

Version Control · Install Guide

1
Install via Homebrew

Official formula. Updates with `brew upgrade gh`.
```
brew install gh
```
2
Authenticate with GitHub

Interactive prompts: pick GitHub.com → HTTPS → "Login with a web browser". Copy the one-time code it prints, paste it at github.com/login/device, authorize.
```
gh auth login
```
3
Verify auth

Should print `Logged in to github.com as <your-username>` and show the token scopes.
```
gh auth status
```

Verify installation

gh --version

1
Install via winget (recommended)

The WinGet package is the only one maintained by the GitHub CLI team. Run in PowerShell — a UAC prompt will appear.
```
winget install --id GitHub.cli
```
2
(Alternative) Install via Scoop

Community-maintained. Good if you already use Scoop for dev tooling.
```
scoop install gh
```
3
(Alternative) Install via Chocolatey

Community-maintained. Requires Chocolatey installed first.
```
choco install gh
```
4

Open a FRESH PowerShell or Terminal window

The installer modifies PATH — existing shells will not see `gh` until they are restarted.
5
Authenticate with GitHub

Interactive prompts. Recommended answers: GitHub.com → HTTPS → Yes (auth git with gh credentials) → Login with a web browser. It prints a one-time code; paste it at github.com/login/device and authorize.
```
gh auth login
```
6
(Alternative) Authenticate with a Personal Access Token

If you cannot use the browser flow (CI, agent-only machine), create a classic PAT at github.com/settings/tokens, then pipe it in. Replace YOUR_PAT with the token.
```
echo YOUR_PAT | gh auth login --with-token
```
7
Verify auth

Confirms gh can talk to the API and lists the token scopes.
```
gh auth status
```

💡 WinGet uses the official signed installer. Choco/Scoop are community forks — if anything breaks, fall back to WinGet or download the .msi from https://github.com/cli/cli/releases/latest.

Verify installation

gh --version

Add the GitHub CLI apt keyring (Debian / Ubuntu)

For Fedora / RHEL / openSUSE, see the official install docs — each distro has a one-liner.

(type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) && sudo mkdir -p -m 755 /etc/apt/keyrings && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null && sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg

Add the apt source list

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null

Install gh

sudo apt update && sudo apt install gh -y

4
Authenticate with GitHub

GitHub.com → HTTPS → browser flow. Copy the one-time code to github.com/login/device.
```
gh auth login
```
5
Verify auth
```
gh auth status
```

Verify installation

gh --version

🤖

Use it with an AI agent

Once installed + authenticated, just tell Claude Code / Cursor / Codex:

"Use the GitHub CLI to open a PR from this branch" · "List my open issues across all repos" · "Trigger the deploy.yml workflow on main"

The agent runs the CLI in your shell with the credentials you already saved — no extra config.

Official docs ↗

⚡

Database · Install Guide

1
Install via Homebrew tap

Official tap maintained by Supabase. Updates with `brew upgrade supabase`.
```
brew install supabase/tap/supabase
```
2
Log in to Supabase

Opens a browser, generates a personal access token, and saves it to `~/.supabase/access-token`.
```
supabase login
```
3
Initialize a new project directory

Creates `supabase/config.toml` and the migrations folder. Run in an empty or existing app directory.
```
supabase init
```
4
Link to your cloud project

Find <project-ref> in your Supabase dashboard URL: https://supabase.com/dashboard/project/<ref>. This lets `supabase db push`, functions deploy, etc. target the right project.
```
supabase link --project-ref <your-project-ref>
```
5
(Optional) Start the local stack

Requires Docker Desktop running. Spins up Postgres + Auth + Storage + Studio on localhost.
```
supabase start
```

Verify installation

supabase --version

1
Install Scoop (if you don't have it)

Scoop is the only supported Windows installer for Supabase CLI. Run in a NON-admin PowerShell — Scoop installs per-user.
```
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser; irm get.scoop.sh | iex
```
2
Add the Supabase Scoop bucket

Pulls the bucket manifest from Supabase's official scoop repo.
```
scoop bucket add supabase https://github.com/supabase/scoop-bucket.git
```
3
Install Supabase CLI

Scoop downloads the prebuilt Windows binary and adds it to PATH.
```
scoop install supabase
```
4
Log in to Supabase

Opens your default browser. Click "Authorize" on the Supabase consent screen. The token is stored under %USERPROFILE%\.supabase\.
```
supabase login
```
5
Initialize a new project directory

Creates `supabase/config.toml`. Run in an empty directory (or your app's root).
```
supabase init
```
6
Link to your cloud project

Get <project-ref> from your Supabase dashboard URL (the 20-char slug after /project/).
```
supabase link --project-ref <your-project-ref>
```
7
(Optional) Start the local stack

Requires Docker Desktop for Windows installed and running.
```
supabase start
```

💡 Supabase EXPLICITLY does not support `npm install -g supabase`. On Windows, use Scoop. Do not follow random Stack Overflow answers telling you to npm-install it globally — it will silently break.

Verify installation

supabase --version

1
Install via Homebrew on Linux (recommended)

Linuxbrew works cross-distro and gets updates fastest.
```
brew install supabase/tap/supabase
```
2
(Alternative) Install as a dev dependency

Global npm install is NOT supported. Use it as a per-project devDependency and run via `npx supabase`.
```
npm install supabase --save-dev
```
3

(Alternative) Download a prebuilt release

Grab the .tar.gz for your arch from https://github.com/supabase/cli/releases and drop `supabase` into /usr/local/bin.
4
Log in to Supabase
```
supabase login
```
5
Initialize a new project directory
```
supabase init
```

Link to your cloud project

supabase link --project-ref <your-project-ref>

7
(Optional) Start the local stack

Requires Docker Engine running.
```
supabase start
```

Verify installation

supabase --version

🤖

Use it with an AI agent

Once installed + authenticated, just tell Claude Code / Cursor / Codex:

"Use the Supabase CLI to start a local stack" · "Create a migration for a new posts table" · "Generate TypeScript types from my remote database" · "Deploy the send-email edge function"

The agent runs the CLI in your shell with the credentials you already saved — no extra config.

Official docs ↗

☁️

Cloud · Install Guide

Install the signed .pkg (recommended)

Official AWS-signed package. Most up-to-date and supported path.

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg" && sudo installer -pkg AWSCLIV2.pkg -target /

2
(Alternative) Install via Homebrew

Community-maintained. Usually a few days behind upstream but simpler.
```
brew install awscli
```
3

Create an IAM user + access keys in AWS Console

Sign in to AWS Console → IAM → Users → Create user → attach a least-privilege policy (NEVER use root keys) → Security credentials tab → Create access key → select "CLI" as the use case. Copy the Access Key ID and Secret Access Key — the secret is shown only once.
4
Configure credentials

Prompts for 4 values: Access Key ID, Secret Access Key, default region (e.g. us-east-1), default output format (json / text / table). Stored in ~/.aws/credentials and ~/.aws/config.
```
aws configure
```
5
Test your credentials

Returns your account ID, user ARN, and user ID. If this fails, your keys are wrong or your user has no permissions.
```
aws sts get-caller-identity
```

Verify installation

aws --version

1
Install via MSI (recommended)

Runs the official AWS-signed MSI silently. Adds `aws` to the system PATH. Open an ELEVATED PowerShell (Run as Administrator).
```
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi /qn
```
2
(Alternative) Install via winget

Works in a normal PowerShell — winget handles the elevation prompt.
```
winget install --id Amazon.AWSCLI
```
3

(Alternative) Download + run the MSI manually

If you cannot run commands as admin, download the MSI from https://awscli.amazonaws.com/AWSCLIV2.msi and double-click it.
4

Open a FRESH PowerShell window

MSI modifies the SYSTEM PATH — open a brand-new PowerShell / Terminal so `aws` is on PATH. Existing ones will say "command not found".
5

Create an IAM user + access keys in AWS Console

AWS Console → IAM → Users → Create user → attach a least-privilege policy (e.g. PowerUserAccess for dev, NOT root keys) → the new user → Security credentials → Create access key → select "Command Line Interface (CLI)" → confirm. Copy both the Access Key ID and the Secret Access Key. The Secret is shown only once — save it immediately.
6
Configure credentials

Prompts for 4 values: AWS Access Key ID, AWS Secret Access Key, Default region (e.g. us-east-1), Default output (json). Stored at %USERPROFILE%\.aws\credentials and %USERPROFILE%\.aws\config.
```
aws configure
```
7
(Optional) Set up a named profile for multiple accounts

If you have work + personal AWS accounts, configure each under its own profile. Then use `aws --profile work s3 ls`, etc.
```
aws configure --profile work
```
8
Test your credentials

Returns your AWS account ID + user ARN. If this errors, your keys are wrong or the user has no permission.
```
aws sts get-caller-identity
```

💡 NEVER commit %USERPROFILE%\.aws\credentials to git. For CI or AI agents, prefer AWS IAM Identity Center SSO (`aws configure sso`) over long-lived access keys.

Verify installation

aws --version

1
Download the v2 installer (x86_64)

For ARM64 (Graviton, Raspberry Pi), use awscli-exe-linux-aarch64.zip instead.
```
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
```
2
Unzip the installer
```
unzip awscliv2.zip
```
3
Run the installer

Installs to /usr/local/aws-cli/ and symlinks `aws` into /usr/local/bin/.
```
sudo ./aws/install
```
4
(Alternative) Snap install

One-liner on distros with snapd (Ubuntu, Fedora with snapd).
```
sudo snap install aws-cli --classic
```
5

Create an IAM user + access keys

Same as the Windows step: AWS Console → IAM → create dedicated CLI user → attach least-privilege policy → generate access keys.
6
Configure credentials
```
aws configure
```
7
Test your credentials
```
aws sts get-caller-identity
```

Verify installation

aws --version

🤖

Use it with an AI agent

Once installed + authenticated, just tell Claude Code / Cursor / Codex:

"Use the AWS CLI to list my S3 buckets" · "Describe the EC2 instance i-abc123" · "Tail CloudWatch logs for /aws/lambda/my-func" · "Deploy this Lambda zip"

The agent runs the CLI in your shell with the credentials you already saved — no extra config.

Official docs ↗

🟩

Productivity · Install Guide

1
Install via Homebrew (easiest)

Pulls a signed prebuilt binary. No Rust or Node toolchain needed.
```
brew install googleworkspace-cli
```
2
(Alternative) Install via npm

Official npm wrapper. Requires Node.js 18+.
```
npm install -g @googleworkspace/cli
```
3
(Alternative) Install from source with Cargo

The CLI is written in Rust. Only needed if you want the bleeding edge.
```
cargo install --git https://github.com/googleworkspace/cli --locked
```
4
Install gcloud CLI (for automated auth)

Easiest auth path uses gcloud to provision the GCP project for you.
```
brew install --cask google-cloud-sdk
```
5
Automated setup via gcloud

Creates a Google Cloud project, enables the Workspace APIs, generates OAuth credentials, and opens a browser to log in. One command does everything.
```
gws auth setup
```
6
Verify auth

Should return a JSON list of your 5 most recent Drive files.
```
gws drive files list --params '{"pageSize": 5}'
```

Verify installation

gws --version

1
Install Node.js LTS (prerequisite for npm path)

Skip if `node --version` already returns 18+.
```
winget install OpenJS.NodeJS.LTS
```
2

Open a FRESH PowerShell window

PATH refresh. Required after Node install.
3
Install Google Workspace CLI globally via npm

On Windows, the npm path is the easiest — no Rust toolchain, no manual PATH edits.
```
npm install -g @googleworkspace/cli
```
4

(Alternative) Download a prebuilt Windows binary

From https://github.com/googleworkspace/cli/releases — grab the .exe for windows-x86_64, place it in a folder on your PATH (e.g. C:\tools\gws.exe).
5
Install gcloud CLI (for automated auth)

Google Cloud SDK includes the gcloud binary `gws auth setup` depends on. Elevated PowerShell required.
```
winget install Google.CloudSDK
```
6
Automated setup via gcloud

Creates a GCP project, enables the Workspace APIs (Drive, Gmail, Calendar, etc.), generates OAuth credentials, and opens a browser to complete the OAuth flow. Credentials saved to %USERPROFILE%\.config\gws\.
```
gws auth setup
```
7
(Manual alternative) Set up OAuth credentials yourself

If you prefer not to install gcloud: go to console.cloud.google.com/apis/credentials → Create Credentials → OAuth client ID → Desktop app. Download the JSON, save it as %USERPROFILE%\.config\gws\client_secret.json, then run `gws auth login` to complete the browser flow.
```
gws auth login
```
8
Verify auth

Lists your 5 most recent Drive files as JSON. If this errors with 401, re-run `gws auth setup` or `gws auth login`.
```
gws drive files list --params "{\"pageSize\": 5}"
```

💡 The CLI is written in Rust but is distributed as prebuilt binaries — you do NOT need to install Rust or Go on Windows. Use the npm package or the GitHub release binaries.

Verify installation

gws --version

1
Install via Homebrew on Linux (simplest)

Works across distros via Linuxbrew. Gets signed binaries.
```
brew install googleworkspace-cli
```
2
(Alternative) Install via npm

Requires Node.js 18+. Works everywhere.
```
npm install -g @googleworkspace/cli
```
3
(Alternative) Install from source

Requires Rust toolchain (`rustup`). Builds from the official repo.
```
cargo install --git https://github.com/googleworkspace/cli --locked
```
4
Install gcloud CLI (for automated auth)

Debian/Ubuntu one-liner; other distros see cloud.google.com/sdk/docs/install.
```
curl https://sdk.cloud.google.com | bash && exec -l $SHELL
```
5
Automated setup via gcloud

Provisions the Cloud project, enables APIs, runs the OAuth flow.
```
gws auth setup
```

Verify auth

gws drive files list --params '{"pageSize": 5}'

Verify installation

gws --version

🤖

Use it with an AI agent

Once installed + authenticated, just tell Claude Code / Cursor / Codex:

"Use the gws CLI to list my 10 most recent Drive files" · "Draft a Gmail reply to this thread" · "Create a Calendar event for tomorrow 3pm"

The agent runs the CLI in your shell with the credentials you already saved — no extra config.

Official docs ↗

🤗

ML & Models · Install Guide

1
Prerequisite: Python 3.8+

Check with `python3 --version`. Install via Homebrew if missing.
```
brew install python
```
2
Install huggingface_hub with CLI extras

The `[cli]` extra installs the `hf` binary (replaces the old `huggingface-cli`).
```
pip install -U "huggingface_hub[cli]"
```
3
(Alternative) Install via uv

Faster if you already use uv for Python envs.
```
uv tool install "huggingface_hub[cli]"
```
4
Log in with a user access token

Create a token at https://huggingface.co/settings/tokens (read + write scope). `hf auth login` prompts for the token and stores it in `~/.cache/huggingface/token`.
```
hf auth login
```
5
Verify auth

Returns your HF username + org memberships.
```
hf auth whoami
```
6
(Optional) Download a model

Pulls the repo into the HF cache. Agents can reference it by repo_id afterward.
```
hf download meta-llama/Llama-3.2-1B-Instruct
```

Verify installation

hf --version

1
Prerequisite: Python 3.8+

Install via winget if missing. Tick "Add python.exe to PATH" during interactive installs.
```
winget install Python.Python.3.12
```
2

Open a FRESH PowerShell window

PATH refresh. Required after Python install so `pip` resolves.
3
Install huggingface_hub with CLI extras

Installs the `hf` binary onto your Python Scripts PATH. Quotes are required in PowerShell for the bracket extra.
```
pip install -U "huggingface_hub[cli]"
```
4
Log in with a user access token

Generate a token at https://huggingface.co/settings/tokens (read + write scope). The command prompts for the token and stores it in %USERPROFILE%\.cache\huggingface\token.
```
hf auth login
```
5
Verify auth

Returns your HF username + org memberships. 401 means re-run `hf auth login`.
```
hf auth whoami
```
6
(Optional) Download a model

Cached under %USERPROFILE%\.cache\huggingface\hub — agents resolve by repo_id after this.
```
hf download meta-llama/Llama-3.2-1B-Instruct
```

💡 If `hf` is not recognised after install, the Python Scripts dir is missing from PATH. Add `%USERPROFILE%\AppData\Local\Programs\Python\Python3xx\Scripts` and reopen the shell.

Verify installation

hf --version

1
Prerequisite: Python 3.8+ and pip

Debian/Ubuntu example. For other distros, use your package manager.
```
sudo apt update && sudo apt install -y python3 python3-pip
```
2
Install huggingface_hub with CLI extras

The `[cli]` extra installs the `hf` binary onto your user bin dir (usually `~/.local/bin`).
```
pip install -U "huggingface_hub[cli]"
```
3
(Alternative) Install via pipx or uv

Isolated install — avoids polluting the system Python.
```
pipx install "huggingface_hub[cli]"
```
4
Log in with a user access token

Create a token at https://huggingface.co/settings/tokens (read + write scope). Stored in `~/.cache/huggingface/token`.
```
hf auth login
```
5
Verify auth
```
hf auth whoami
```

(Optional) Download a model

hf download meta-llama/Llama-3.2-1B-Instruct

Verify installation

hf --version

🤖

Use it with an AI agent

Once installed + authenticated, just tell Claude Code / Cursor / Codex:

"Use hf to download meta-llama/Llama-3.2-1B-Instruct" · "Upload this model folder to my HF account" · "List files in this dataset repo"

The agent runs the CLI in your shell with the credentials you already saved — no extra config.

Official docs ↗

🦙 Local Models · Setup Advisor

Run Coding & Chat Models Locally

Tell me your laptop's RAM and GPU. I'll pick a model that fits, show you the exact memory breakdown, and hand you the same llama.cpp launch command I run on my own machine — or the MLX equivalent on a Mac.

Operating system

System RAM 32 GB GPU VRAM 12 GB

Workloads

Detected tier mid-vram 32 GB RAM + 12 GB VRAM

🎯

Recommended for your rig

Qwen3.6-35B-A3B (MoE)

UD-IQ2_M · 11.52 GB llama.cpp Apache-2.0 ✓ Fits

Memory breakdown at chosen context 14.4 GB / 12 GB

Model11.52 GB

KV cache1.01 GB

Vision1.40 GB

Scratch0.50 GB

Context length 40960 tokens

What do these numbers mean?

Quantized version: A smaller copy of the model where each weight is stored in fewer bits. Q4_K_M = 4-bit, Q8_0 = 8-bit, UD-IQ2_M = Unsloth Dynamic 2-bit. Lower bits = smaller file + less VRAM, slightly less accuracy. Pick the biggest quant your hardware can fit.
Context length: How many tokens the model can "see" at once — your system prompt + your messages + the model's replies all count. 1 token ≈ ¾ of an English word. Hit the limit and the oldest tokens scroll out the back.
KV cache: Memory the model holds for every token already in its context, so it doesn't have to re-process them on every reply. It grows the more you chat — each new message adds tokens, and KV usage scales linearly with the total. That's why long conversations eat VRAM. -ctk q4_0 -ctv q4_0 compresses it to a quarter of the FP16 size.
Scratch: Working memory the GPU needs while computing one token — attention scores, intermediate activations, flash-attention buffers. It's roughly fixed and doesn't grow with conversation length.
--n-cpu-moe (MoE only): Mixture-of-Experts models like Qwen3.6-35B-A3B have many small expert layers — only 3B params activate per token. --n-cpu-moe N tells llama.cpp to keep N expert layers in CPU RAM instead of VRAM. The active experts get streamed to the GPU at run time, so you fit a bigger quant than VRAM alone would allow, with only a small speed cost. The recommended value below is auto-tuned to your VRAM — drop it to 0 for maximum speed if you have headroom, or raise it if you OOM.

View on Hugging Face ↗

⚡

Once it's running

OpenAI-compatible endpoint

Set the base URL on any OpenAI client. Drop in any non-empty key. Send the model alias as model.

Base URL

http://127.0.0.1:8080/v1

API key

sk-local

Model alias

qwen3.6-moe

⚠️ --host 0.0.0.0 exposes the server on your LAN. Drop it to 127.0.0.1 if you only want local access.

🦙

llama.cpp + Hugging Face · OpenAI-compatible

1
Install Python (for huggingface-cli) and CUDA prerequisites

Python 3.11+ from python.org. NVIDIA driver R555+ ships CUDA 13 runtime — no separate toolkit needed for the prebuilt llama-server.
```
winget install Python.Python.3.12; pip install -U huggingface_hub
```

Get the prebuilt llama.cpp server

Download Unsloth's CUDA-13 build from llama.cpp releases (or build from source). Extract to %USERPROFILE%\.unsloth\llama.cpp\build\bin\Release\ to match the reference launcher.

# Browse releases at https://github.com/ggml-org/llama.cpp/releases
# Pick: llama-bXXXX-bin-win-cuda-12.4-x64.zip (or newer CUDA build)
# Extract llama-server.exe + ggml-cuda.dll to ~/.unsloth/llama.cpp/build/bin/Release/

3
Download the GGUF from Hugging Face

Pulls just the picked quant + the vision adapter (when present). Gets cached in ~/.cache/huggingface/hub.
```
huggingface-cli download unsloth/Qwen3.6-35B-A3B-GGUF Qwen3.6-35B-A3B-UD-IQ2_M.gguf mmproj-BF16.gguf
```

Launch the llama-server

Save as a .ps1 file and double-click, or paste straight into PowerShell. One line — no continuation backticks to trip up your terminal.

& "$env:USERPROFILE\.unsloth\llama.cpp\build\bin\Release\llama-server.exe" -m (Resolve-Path "$env:USERPROFILE\.cache\huggingface\hub\models--unsloth--Qwen3.6-35B-A3B-GGUF\snapshots\*\Qwen3.6-35B-A3B-UD-IQ2_M.gguf") --mmproj (Resolve-Path "$env:USERPROFILE\.cache\huggingface\hub\models--unsloth--Qwen3.6-35B-A3B-GGUF\snapshots\*\mmproj-BF16.gguf") --alias qwen3.6-moe -ngl 99 --n-cpu-moe 5 -c 40960 --flash-attn on -ctk q4_0 -ctv q4_0 -t 16 -b 512 -ub 512 --jinja --reasoning-format deepseek --host 0.0.0.0 --port 8080

Verify it's live

curl http://127.0.0.1:8080/v1/models

Install build tools + CUDA toolkit

Ubuntu/Debian shown. For RHEL/Arch swap the package manager.

sudo apt update && sudo apt install -y build-essential cmake git python3-pip nvidia-cuda-toolkit
pip install -U huggingface_hub

Build llama.cpp with CUDA

Outputs build/bin/llama-server. Add it to PATH or call by full path below.

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Download the GGUF from Hugging Face

huggingface-cli download unsloth/Qwen3.6-35B-A3B-GGUF Qwen3.6-35B-A3B-UD-IQ2_M.gguf mmproj-BF16.gguf

Launch the llama-server

Same flags as the Windows version, just bash syntax. One line — paste straight into your terminal.

~/llama.cpp/build/bin/llama-server -m ~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/*/Qwen3.6-35B-A3B-UD-IQ2_M.gguf --mmproj ~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/*/mmproj-BF16.gguf --alias qwen3.6-moe -ngl 99 --n-cpu-moe 5 -c 40960 --flash-attn on -ctk q4_0 -ctv q4_0 -t 16 -b 512 -ub 512 --jinja --reasoning-format deepseek --host 0.0.0.0 --port 8080

Verify it's live

curl http://127.0.0.1:8080/v1/models

Install Xcode CLT + Python

xcode-select --install
brew install python@3.12

2
Install MLX runtime

mlx-lm ships an OpenAI-compatible server out of the box. mlx-vlm adds vision support — install it later if you need image input.
```
pip install -U mlx-lm huggingface_hub
```
3
Pre-download the MLX model (optional)

mlx_lm.server will lazy-download on first run. Pre-fetching just lets you see the download progress.
```
huggingface-cli download mlx-community/Qwen3.6-35B-A3B-4bit
```
4
Launch the MLX server

MLX manages KV cache + GPU offload inside unified memory automatically — no -ngl/-ctk/--mmproj flags needed.
```
mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-4bit --host 0.0.0.0 --port 8080
```

💡 Vision (mmproj) over MLX requires mlx-vlm instead of mlx-lm. Coming as a follow-up.

Verify it's live

curl http://127.0.0.1:8080/v1/models

Use these resources alongside the channel guides

Want walkthroughs for the tools listed here? Watch on the YouTube channel. Want the background on how these projects fit into the TechRex workflow? Read About.

Watch on YouTube Read about TechRex