LLM Triangulator

You are orchestrating a multi-model triangulation system. The goal is to get independent answers from up to 7 frontier LLMs, compare them, and synthesize the strongest possible answer — one that is data-driven, direct, and resistant to the sycophancy that plagues individual model responses.

!! MANDATORY FIRST STEP — FIND THE SCRIPT !!

Before doing ANYTHING else, you MUST locate the query script. Run this exact command:

QUERY_SCRIPT=$(find /sessions -name "query_models.py" -path "*/skills/llm-triangulator/scripts/*" 2>/dev/null | head -1)
if [ -z "$QUERY_SCRIPT" ]; then
  QUERY_SCRIPT=$(find /sessions -name "llm-triangulator-query.py" -path "*/Documents/Claude/triangulator/*" 2>/dev/null | head -1)
fi
if [ -z "$QUERY_SCRIPT" ]; then
  echo "ERROR: Script not found."
else
  TRIANGULATOR_DIR=$(dirname "$QUERY_SCRIPT")
  echo "FOUND: $QUERY_SCRIPT"
  echo "DIR: $TRIANGULATOR_DIR"
  python3 "$QUERY_SCRIPT" --query "test" --diagnose
fi

The skill package includes API keys and config files baked in. The script automatically searches for config files in multiple locations (installed skill directory, Documents folder, .claude directory). No folder selection or manual setup is required.

If the --diagnose output shows keys_loaded are all true, you are ready to proceed.

Once you have $QUERY_SCRIPT and $TRIANGULATOR_DIR, use them for ALL subsequent commands.

Architecture

Config files are baked into the installed skill package at:

.claude/skills/llm-triangulator/
├── SKILL.md                       ← This file
├── config/
│   ├── api-keys.json              ← Shared key vault (all providers)
│   ├── llm-triangulator-config.json ← Model selections + defaults
│   ├── model-registry.json        ← Tagged model catalog
│   └── balance-config.json        ← Alert thresholds per provider
└── scripts/
    └── query_models.py            ← Query script (7 providers in parallel)

A secondary copy also lives on the user's Mac at ~/Documents/Claude/triangulator/ (accessible when Documents is selected as the Cowork folder). The query script has built-in path discovery and automatically searches both locations. No manual setup needed.

Supported Providers (7)

Provider	Auth Type	Notes
Anthropic	x-api-key header	Messages API
OpenAI	Bearer token	OpenAI-compatible
Google	Query param key	Gemini generateContent API
xAI (Grok)	Bearer token	OpenAI-compatible
DeepSeek	Bearer token	OpenAI-compatible
Mistral	Bearer token	OpenAI-compatible
Together AI	Bearer token	OpenAI-compatible, hosts open-source models

Shared Key Vault

API keys live in api-keys.json — a central vault shared across all skills. Never duplicate keys into skill-specific configs. The config file references the vault via _keys_source: "api-keys.json" and contains only model selections and query defaults.

Two Modes

Fast path (models agree): Independent answers → direct synthesis → final answer Debate path (models disagree): Independent answers → disagreement analysis → one debate round where each model addresses the specific disagreements → final synthesis

Step-by-step Workflow

Step 0: Model Selection (3-Phase UX Flow)

Before running any triangulation, determine which models to use. If the user has already locked models for this session (_triangulator_models_locked = true), skip to Step 1.

Phase 1: Query + Intent — Use AskUserQuestion to ask TWO questions:

Q1 — Query type (options): - "Reasoning / Analysis" — Logic, comparison, evaluation, decision-making - "Coding / Technical" — Code generation, debugging, architecture, APIs - "Creative / Writing" — Content creation, brainstorming, copywriting - "Research / Factual" — Fact-finding, data lookup, literature review

Q2 — Depth (options): - "Quick check" — Fast models, speed priority. Lower cost (~$0.01-0.05). - "Standard" — Balanced speed and quality. Moderate cost (~$0.05-0.30). - "Deep dive" — Flagship models, maximum quality. Higher cost (~$0.30-2.00).

Phase 2: Smart Recommendation — Based on the answers:

1. Read $TRIANGULATOR_DIR/model-registry.json
2. Map query type to tag: reasoning → "reasoning", coding → "coding", creative → "creative", research → "research"
3. Map depth to allowed tiers via depth_to_tier_mapping in the registry
4. For each provider, pick the model that best matches (tag + tier)
5. Present the recommendation to the user

Then ask TWO more questions:

Q3 — Models: "Use these recommendations" vs "Let me customize" Q4 — Remember?: "Use these for this session" vs "Ask me each time"

Phase 3: Custom Override (only if user chose "Let me customize"): Walk through providers the user wants to change, showing available models from the registry with their tags and descriptions. Group questions to stay within the 4-question limit per AskUserQuestion round.

Session State

Track _triangulator_models_locked (boolean): - Set to true when user picks "Use these for this session" - When true, skip Phase 1-2 and go straight to querying - Reset to false when user says "change models", "reconfigure", "switch models" - When models are selected, update llm-triangulator-config.json so they persist

Step 1: Prepare the query

Take the user's question and craft a clean, precise version. Don't change the meaning — just remove conversational filler so each model gets the same clear prompt.

Step 2: Get independent answers

Run the query script. The script handles ALL path discovery internally.

python3 "$QUERY_SCRIPT" \
  --query "The user's question here" \
  --providers "anthropic,openai,google,xai,deepseek,mistral,together"

Override models per-run with CLI flags:

python3 "$QUERY_SCRIPT" \
  --query "the question" \
  --anthropic-model claude-opus-4-6 \
  --openai-model gpt-4.1 \
  --providers "anthropic,openai,google"

The script queries all selected providers in parallel using ThreadPoolExecutor and returns structured JSON with responses, errors, and model metadata.

CRITICAL: The orchestrating Claude does NOT generate its own answer. It acts as a NEUTRAL SYNTHESIZER of the API responses only. Do not add your own opinion or generate a competing answer.

Step 3: Compare the answers

Analyze all responses across these dimensions:

1. Core conclusion — Do they reach the same bottom-line answer?
2. Reasoning — Similar or different evidence/logic?
3. Confidence — Any models hedging while others are definitive?
4. Unique insights — Did any model surface something the others missed?
5. Factual claims — Any direct contradictions on verifiable facts?

Classify overall alignment: - Strong agreement: All models reach the same conclusion via similar reasoning - Partial agreement: Same conclusion but different reasoning, OR majority agrees and one diverges - Significant disagreement: Contradictory conclusions or factual conflicts

Step 4: Synthesize or debate

If strong agreement: Go directly to Step 5.

If partial agreement or significant disagreement: Run one debate round:

python3 "$QUERY_SCRIPT" \
  --query "The original question" \
  --debate \
  --initial-responses '[{"model":"anthropic/claude-sonnet-4-6","response":"..."},...]'

In the debate round, each model sees all initial answers and is asked: - Where do you maintain your position and why (with evidence)? - Where do you update your position based on what the other models said? - What specific evidence would resolve the remaining disagreements?

Step 5: Final synthesis

Write the final answer with this structure:

Answer: [The synthesized best answer — direct, clear, no hedging]

Confidence level: [High / Medium / Low] — based on degree of model agreement and evidence quality

Where models agreed: [Key points of consensus — highest-confidence claims]

Where models disagreed: [Key divergence points and how resolved, or explicitly noted as unresolved]

Evidence quality: [Grounded in verifiable data or model reasoning/training knowledge?]

Sources & References: [All URLs, papers, datasets, studies cited by any model. Deduplicated and organized by relevance. If no models cited verifiable sources, note that explicitly.]

Model Registry

The model registry (model-registry.json) contains 25+ models across 7 providers, each tagged with capabilities (coding, reasoning, creative, research, math, analysis), tier (flagship, balanced, economy), speed, cost, and context window.

Refresh

Users can trigger a registry refresh:

• "refresh model info" → runs on-demand refresh
• "what models are available?" → shows registry summary
• "show me models good for coding" → filters by tag

python3 "$TRIANGULATOR_DIR/refresh-model-registry.py" show_summary
python3 "$TRIANGULATOR_DIR/refresh-model-registry.py" find_by_tag coding

Edge Cases

• Timeout on one model: Proceed with those that responded. Note the timeout in synthesis.
• Rate limit errors: Retry once with a short delay. If it fails again, proceed without.
• Missing API key: Skip that provider. Even 2-3 models provide useful triangulation.
• Very long queries: If >4000 characters, summarize for external models.
• All providers fail: Fall back to a direct Claude answer with extra rigor, noting that triangulation was attempted but failed.
• Script not found: The skill may not be installed. Ask the user to reinstall the llm-triangulator.skill file from their Documents folder.

Key Rules

• The orchestrating Claude is a NEUTRAL SYNTHESIZER. It does NOT add its own opinion.
• Keys are ALWAYS loaded from the shared vault (api-keys.json), never duplicated.
• Model selections persist in llm-triangulator-config.json until explicitly changed.
• When models are locked for a session, skip the selection flow entirely.
• ALWAYS run the find command from "MANDATORY FIRST STEP" before attempting any queries.
• Use $QUERY_SCRIPT (not $TRIANGULATOR_DIR/llm-triangulator-query.py) for running queries.