How to Debug Hermes Agent: Logs, Traces, Gaps

Your Agent Did Something Weird. Where Do You Look?

Hermes Agent gives you two failure modes that are painful in different ways. The first is loud, an exception with a stack trace or a startup crash. The second is quiet, the agent produces a coherent but wrong answer, calls the wrong tool, or trails off after a tool result. The loud one shows up in errors.log. The quiet one is the harder debug, and it lives in the audit log of the specific session that went sideways.

This is a practical map of where the logs are, which file matches which symptom, and the small tools around them that turn 800 lines of JSONL into something you can actually read. It also names the gaps that are still open, so you know what Hermes will not tell you today.

Where the Logs Live

Every log Hermes writes lands under your Hermes home directory, defaulting to ~/.hermes/logs/ (and to <profile>/logs/ for non-default profiles). Four files matter:

All four are written by Python's RotatingFileHandler. When a file hits its size cap it rolls to agent.log.1, agent.log.2, and so on up to the configured backup_count. The active file is always the unsuffixed name. This matters when you are chasing a bug that happened yesterday, because "yesterday" may have already rotated. Grab the numbered files, not just the current one.

The setup path for all of this is hermes_logging.setup_logging(). The gateway calls it with mode="gateway" at startup and it attaches file handlers only, never a console handler. So hermes gateway start is silent by design. If your gateway looks dead, it is not. Tail gateway.log.

Which Log to Open First

Symptom drives file, not the other way around. A rough decision tree:

• The agent crashed on startup, or a request 500s. Open errors.log with hermes logs errors --since 30m -f. Stack traces land here first.
• The agent replied, but the reply is wrong. Skip errors.log. The tool calls that produced the bad reply are in the JSONL session transcript for that session (see the next section on trace-tree).
• A Telegram or Discord user says "it went quiet". Open gateway.log, filter to their session. hermes logs gateway --session abc123 narrows it.
• The dashboard is not updating. gui.log. Websocket disconnects and TUI-gateway sync errors surface here before they surface anywhere else.
• You do not know what is happening at all. hermes logs -f on agent.log with a second pane on errors.log. The overwhelming majority of "what is my agent doing" answers are one tail -f away.

The hermes logs CLI is your friend here. A few of the flags earn their keep on a live debugging session:

hermes logs                              # last 50 lines of agent.log
hermes logs -f                           # follow agent.log in real time
hermes logs gateway -n 100               # last 100 lines of gateway.log
hermes logs --level WARNING --since 1h   # last hour, warnings and errors
hermes logs --session abc123             # narrow to one session id
hermes logs errors --since 30m -f        # follow errors from 30 min ago
hermes logs list                         # inventory of files and sizes

Reading a Session as a Tree, Not a Wall of JSONL

Once a session ends, its full trajectory is written as line-delimited JSON in the session transcript file. This is the most useful artifact for debugging the "coherent but wrong" class of bug, because every model call, every tool call, and every tool result is captured in order with its arguments. It is also, in raw form, unreadable. An 800-line audit log for a moderately complex session is normal.

The community tool trace-tree (see Mukunda Katta's writeup on dev.to) reads that JSONL and prints a terminal tree. A session becomes a root, each tool call becomes a child, denied calls appear as children with their error attached. You open it, read the tree, close it. No login, no upload, no vendor lock-in. Point it at a session file:

trace-tree ~/.hermes/logs/sessions/2026-07-21T09-42-11.jsonl

For deeper diagnostics you can also pipe the same JSONL to jq and filter by tool name, latency, or error status. The transcripts are stable across versions in a way agent.log is not, so build your one-off queries against the transcript, not against grepping the human-readable log.

For a related "the agent is quiet, is it stuck?" investigation on messaging channels, our Telegram troubleshooting guide walks through the delivery-side symptoms first (bot token, webhook, group permissions), which often turn out to be the real cause before you ever need the trace tree.

Raising Log Levels Without Leaking Secrets

Verbose mode is one flag away. hermes chat --verbose (or -v) sets verbose_logging=True on AIAgent, which calls setup_verbose_logging() and adds a DEBUG-level console StreamHandler on top of the file handlers. You get everything the agent sees, live.

The critical detail is that every log record, at every level, is processed by RedactingFormatter from agent/redact.py before it is written. The formatter matches known credential shapes (sk-, sk-or-, sk-ant-, common OAuth patterns, and environment-variable-style secrets) and replaces the value inline. In practice this means you can raise the log level in production, or paste a redacted agent.log into a bug report, without leaking your OpenRouter key.

There is one gotcha worth naming. If you write your own logger or your own formatter and skip the pipeline, redaction does not apply. The NameError: name 'RedactingFormatter' is not defined gateway startup crash reported as issue #8090 is the same shape of mistake in reverse. Assume the formatter is load-bearing, do not bypass it, and if you write custom logging, wrap it through hermes_logging.setup_logging() rather than around it.

The Observability Gaps That Are Still Open

Hermes is honest about what it does not yet do. Two open feature requests describe the current ceiling:

• Structured spans with start and end timestamps. Today most log rows carry a single timestamp and some tool outputs include an ad-hoc duration_seconds. There is no stable schema for start_ts, end_ts, duration_ms, and parent_id across the whole trajectory. This is issue #6741, and it blocks the clean latency-per-tool dashboards you want when a session is slow.
• Live attach to an in-flight gateway session. As of the current release, once a gateway session is running you cannot observe it in real time from outside. agent.log catches up when the session completes, but there is no "watch this user's session as it happens" hook. This is issue #18127, and it is the biggest gap for anyone running Hermes as a shared service.

Two workarounds fill part of the gap in the meantime. The first is OpenTelemetry-shaped tracing to a backend like SigNoz, which lets you turn Hermes activity into spans downstream even without native span emission. The second is Langfuse (issue #1501), which some teams wire in per-turn as a manual instrumentation layer. Neither is officially first-class yet, but both are running in production somewhere today.

When You Would Rather Not Own the Log Layer at All

Everything above assumes you are running the agent yourself. If that is exactly the part you would rather delegate, Hermify runs a managed Hermes Agent for you on Telegram, with the same logging and audit trail available on the underlying container. You get the persistent memory, the session transcripts, and the ability to escalate a specific issue to us, without needing to keep hermes logs -f open on your own VPS. If you are on the fence about self-hosting versus a managed setup for reasons that include the operational overhead of observability, our take on hosted versus self-hosted Hermes covers the tradeoffs in full.

Get started with Hermify and skip the log-rotation and observability work entirely. Or keep the logs local and use this guide next time your agent goes quiet. Both are valid, and the same audit trail is available on either path.

Sources

• Chapter 9: Observability and Debugging (Claude Code vs. Hermes Agent) - Ken Huang
• Put a Microscope on Hermes: Full Visibility into Agent Execution - Alibaba Cloud
• I had 800 lines of Hermes agent audit log. trace-tree turned it into a tree I could read.
• hermes-agent/hermes_logging.py source
• Issue #18127: Observability for in-flight gateway sessions
• Issue #6741: Structured session tracing with start/end timestamps
• Hermes Monitoring and Observability with OpenTelemetry - SigNoz