How can we help?

LLM token tracking: troubleshooting and FAQ

Uri Nativ
Uri Nativ
  • Updated
Torii AI

This article collects the things that go sideways in real rollouts, grouped by symptom, plus a FAQ at the end. If you don't find your issue here, contact Torii support with: the tool you're configuring, the OS, the deployment path, the install script's exit code (if any), and the last 4 chars of the ingest token in use.

Exit codes from install scripts

Code Meaning What to do
0 Success. Look in the event tail - your test event should appear in < 5 seconds.
2 Invalid args: missing token. Check that your MDM is passing the token. For Jamf, that's Parameter 4 (legacy policies that used Parameter 4 = endpoint and Parameter 5 = token still work). For Workspace ONE, an environment variable named TORII_TOKEN (legacy TOKEN still works; the Windows .ps1 installers also read $env:TORII_TOKEN). For JumpCloud, use the -jumpcloud script variants with a Command Variable named TORII_TOKEN - JumpCloud variables are body placeholders, not env vars, and the variants ship with the placeholder reference already baked in. For Kandji, Mosyle, and Intune on macOS, use the pre-filled scripts from the Torii UI. The endpoint is baked into the script and never needs to be passed.
3 Could not resolve target user. The script was run as root with no logged-in console user (CI sandbox, kickstart, ssh as root with no logname). Pass --user-scope <path-to-user-home> explicitly.
4 JSON parse error reading existing config. The user's settings.json already contains invalid JSON (a manual edit went wrong). Fix the JSON by hand - the script refuses to touch a broken file rather than make it worse.
6 Permission denied writing to target file. The script can't write the target file with its current identity. Re-run as root (system-wide install) or as the correct user (per-user install).
7 --dry-run would have made changes. Not an error. Used to preview what a real run will do.

I already run an OTLP collector - will the install clobber it?

Current script versions no longer refuse to install when an existing OpenTelemetry exporter is configured (older versions exited with code 5 and offered a --force bypass; both are gone). A direct install replaces the Torii-owned telemetry keys - so if the CLI on that machine currently points at your own collector (e.g. localhost:4318), the install repoints it at Torii, after backing the previous file up to <file>.ai-otel-backup-YYYYMMDD-HHMMSS.

Two right answers:

  • You want to keep your collector in the path. Don’t run the direct install. Use the existing collector path - add Torii as a second exporter on the collector you already run, so your existing backend keeps receiving everything and Torii gets a copy.
  • The old config was stale and replacing it was intended. Nothing to do - the install already converged the file. The timestamped backup is the rollback if you change your mind.

Symptom: nothing in the event tail after install (succeeded with exit 0)

Walk through these in order:

1. The CLI is reading stale config

The CLI tools load their settings at startup. An open Claude Code / Gemini / Codex session that was running before the install will not see the new config. Close it and reopen.

2. Outbound HTTPS to Torii's ingest is blocked

From the workstation:

curl -v https://ai-events.toriihq.com/health

Expected: a TLS handshake and a small JSON response. If you get TCP timeout or "Could not resolve host", talk to networking about allow-listing ai-events.toriihq.com.

Linux installs additionally need outbound access to assets.toriihq.com at install time (for the helper binary). Once the install has run, this host is no longer needed at runtime - only ai-events.toriihq.com.

3. The wrong token is configured

Open the relevant config file on the workstation and find the token (look for the last 4 chars only - never paste full token content into Slack/Jira/etc.):

  • Claude Code: ~/.claude/settings.jsonenv.OTEL_EXPORTER_OTLP_HEADERS
  • Gemini CLI: ~/.gemini/settings.jsontelemetry.otlpEndpoint (the token is the bit after /t/)
  • Codex CLI: ~/.codex/config.toml → the Authorization = "Bearer …" in the managed block

Compare the last 4 chars against the value in Torii's Credentials card. If they don't match, your MDM has the wrong token - update the MDM parameter and re-run the policy.

4. You configured one tool but ran a different tool's test command

Each of the three tools has its own page and its own status card. Make sure you're looking at the right page for the CLI you actually configured and tested.

Symptom: events stopped arriving after token rotation

Your old token's 24-hour grace period may have ended. Confirm that every MDM policy and every collector instance has been updated to the new token, then run a test event. If you missed an instance and it's now past the grace period, contact Torii support - we can briefly re-enable the old token to give you another rollout window.

Symptom: Linux install fails with "could not fetch helper" or checksum mismatch

The Linux installer downloads ai-otel-helper from https://assets.toriihq.com/otel/latest/helper/linux-<arch>.gz. Two cases:

  • Could not fetch helper - the workstation can't reach assets.toriihq.com. For air-gapped fleets, mirror the helper internally and pass HELPER_BASE_URL=https://your-mirror.example.com/torii-otel as an extra parameter. See Rolling out LLM token tracking via MDM (overview).
  • Helper checksum mismatch - the gzipped helper at your mirror doesn't match its .sha256 sidecar. This happens when the mirror is partially updated. Re-sync both files from the canonical URL.

Symptom: PowerShell refuses to run the script

Default Windows execution policy blocks unsigned PowerShell scripts. Two options:

# Option A - bypass for a single invocation
powershell -ExecutionPolicy Bypass -File "$env:TEMP\install-claude-code-telemetry.ps1" -Token '<your-token>'

# Option B - your MDM (Intune, SCCM, etc.) runs scripts as SYSTEM by default
# and is exempt from per-user execution policy. No bypass needed when invoked
# through MDM.

Symptom: macOS Gatekeeper / signing concerns

The macOS install scripts are unsigned shell scripts run via curl | bash or via MDM as root. They:

  • Use only native macOS tooling - osascript -l JavaScript, plutil, stat, logname. No binaries are downloaded at install time.
  • Don't trigger Gatekeeper because nothing is launched as an executable binary. Gatekeeper inspects executable launches, not the interpretation of shell scripts.
  • Run cleanly under MDM as root without prompts.

For direct-user runs (one-liner), Gatekeeper does not block curl | bash; the user accepts the script by running the command. Your security team's questions about supply chain should be answered by reviewing the published script body at https://assets.toriihq.com/otel/latest/<script> directly and comparing against the manifest's SHA256.

Symptom: Gemini events show "anonymous" instead of user email

Gemini emits user.email only for OAuth-authenticated sessions. Vertex AI service-account sessions and API-key sessions appear as anonymous (Torii receives an installation ID but no user identity). This is upstream Gemini behavior and cannot be worked around from Torii's side. To get per-user attribution for Gemini, get your engineers signed in with OAuth.

Symptom: backup files are accumulating

A backup (*.ai-otel-backup-YYYYMMDD-HHMMSS) is written whenever a run actually changes the target file. Current scripts converge - a recurring MDM policy that re-runs without changing anything creates no new backups. If you see many backups, they were created by older script versions (which backed up on every run) or by repeated real changes (token rotations, script updates). The scripts never clean backups up (every change stays recoverable by design); deploy a separate MDM policy to clean up older-than-N-day backups, e.g.:

find ~/.claude ~/.gemini ~/.codex \
  -maxdepth 1 -type f -name '*.ai-otel-backup-*' -mtime +30 -delete

FAQ

Does it capture prompt contents?

No, by default. The Claude Code installer sets OTEL_LOG_USER_PROMPTS=0, the Gemini installer sets logPrompts: false, and the Codex installer sets log_user_prompt = false. You can opt in to prompt content by re-running the installer with --log-prompts true, but it's off by default.

Does it capture code from edits?

Full file contents and tool/shell output are never sent to Torii. For Claude Code, the installer enables tool-detail logging (OTEL_LOG_TOOL_DETAILS=1) so events carry which tool ran and its input arguments in truncated form - the shell command line, file paths, MCP server and tool names, and skill names. Individual values are capped at 512 characters and the whole payload at ~4 KB per event, so short text fragments passed as tool arguments can appear, but never whole files, command output, or model responses. To turn this off fleet-wide, remove the OTEL_LOG_TOOL_DETAILS key from the managed settings (or set it to "0"). Gemini and Codex report tool usage without argument detail.

What if I already use Datadog / Splunk / Elastic for observability?

Use the existing OTEL collector path. Add Torii as a fan-out destination on your collector so both your existing backend and Torii receive the OTLP signals.

Can I roll out only one of the three tools?

Yes. Each tool is independent - installer scripts, config files, and event tails are all per-tool. If your fleet uses Claude Code today and you'll add Gemini later, deploy Claude Code now and ignore the Gemini section of the dashboard.

Does the install script restart or interrupt the user's terminal?

No. The install writes files and exits. Any already-open CLI session keeps its previous (uninstrumented) config until restart - open a new terminal to start emitting telemetry.

Can I run both the MDM rollout and the collector path on the same fleet?

It's technically possible but offers no benefit. All three CLIs send to Torii over OTLP - once your collector fans out OTLP to Torii, you have complete signal coverage. Adding a per-workstation MDM install on top would just duplicate the same OTLP signals. Pick one path per tool.

Where do I find the canonical script bodies for security review?

Published bytes: https://assets.toriihq.com/otel/latest/<script-name>. Manifest with SHA256s for every published file: https://assets.toriihq.com/otel/latest/manifest.json. Fetch them with curl to review the exact bytes that will run on your workstations. If you'd like a deeper architectural walkthrough or have specific security review questions, contact Torii support.

How often do the scripts change?

Infrequently. Each release of a CLI may introduce new telemetry fields; we incorporate them into the install scripts in the same release window. The scripts' shape (where it writes, what it backs up, what it preserves) is stable across releases - only the contents grow. If a script change ever needs a re-deploy of MDM policies, Torii will surface that in the AI Telemetry page before customers see it elsewhere.

What happens if assets.toriihq.com is down?

The AI Telemetry page in Torii shows an "Assets not available" message and does not render a stale snapshot of the script. This is intentional - admins must never paste an out-of-date script body into their MDM. Customer workstations that already have the install applied are unaffected; only new rollouts pause until the host is back. Already-installed workstations continue to emit telemetry over OTLP to ai-events.toriihq.com normally.

Does this need a Torii admin role to enable?

Yes. The AI Telemetry page is visible to org admins. The ingest token is treated like an API credential - control access to it accordingly.

Still stuck?

Contact Torii support with:

  • Which tool (Claude Code / Gemini CLI / Codex CLI).
  • OS and version of the workstation.
  • Deployment path (MDM / collector / one-liner).
  • The install script's exit code, if one was emitted.
  • The last 4 chars of the ingest token (so we can identify the active credential without you sharing the full value).
  • Whether the Status card is Pending, Connected, or Stale, and any error rows visible in the event tail.

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request