Wuming 無名

Blog

Terminals in Your Pocket (5): Troubleshooting Notes

· Technology · English

Previous: The Computer Decides Who Gets In | 中文版:排错手记

The last post of the series: the problems that kept coming back over these months, and a few times the author dug a hole and fell in. This kind of thing rarely makes it into documentation, yet it is probably the most useful part for anyone building something similar.

Grouped by symptom: can’t connect, slow, no notification, won’t type.

Can’t connect

The computer shows offline. Nine times out of ten the resident process isn’t running. A closed laptop lid, an OS update and reboot, a signed-out user all stop it. Registering it as a login item on macOS and as a user-level service on Linux fixes most of it. The exception is a Linux server reached over SSH: user-level services stop by default when the user logs out. Keeping it alive means turning on session lingering for that user, which is one line in the install guide and which nearly everyone skips.

The computer is online but the phone can’t connect. Check first whether a new device is waiting for approval (the local approval gate from the previous post). One command on the computer lists what is waiting; approve it and you are in. This is the most common “online but unreachable”.

Can’t connect from a corporate network. Some enterprise networks block UDP entirely and allow only TCP 80 and 443. The system tries several paths at once: UDP direct and hole punching first, then TCP, then WebSocket on port 443 to a relay. That last route almost always gets through, at the cost of speed. If even 443 is intercepted by a decrypting proxy, the only fix is asking the administrator to allow the domain.

Can’t connect from one region. Reachability to each relay varies a lot by region. Computers reserve slots on more than one relay and phones pick the nearest by latency. Once a relay’s upstream failed, and every computer reserved on it looked “offline” for an afternoon while the computers themselves noticed nothing, because their connection to the relay was alive; the relay just couldn’t reach out. External probes of every relay were added afterwards, and computers now hold two reservations, so this is no longer a single point of failure.

Slow

Typing lags. Look at the status bar: direct or relayed. Relayed usually means the punch failed, common behind symmetric NATs such as many mobile carriers. It isn’t a fault. Switching to Wi-Fi often yields a direct connection.

Scrolling stutters. Terminal bytes go to the phone as they are, and a program that floods the screen (yes, or a progress bar redrawing hundreds of times a second) can overwhelm the phone’s renderer. The computer now coalesces bursts of output into one frame. The phone’s terminal also defaults to “fit to phone”, asking tmux to shrink the window to what the phone can show; turn it off and you get the computer’s size, complete but tiny.

Reconnecting forever because of a bad clock. Requests between devices carry a timestamp and a signature; too much clock skew looks like a replay and is refused. An old Mac that had been off for months drifted by a dozen minutes and connected and dropped in a loop. Syncing the clock fixed it instantly.

No notification

The agent stopped and the phone is silent. Check three things in order: are the hooks installed, was the agent restarted after installing them, does the phone have notification permission. Hooks are written into the agent’s configuration file, which the agent reads only at startup, so an agent already running when the hooks were installed won’t notify. This came up so often that the hook installer now prints a reminder.

The notification arrived but opens the wrong window. Hook events carry the pane id, but tmux pane ids change when a window is closed and reopened. The system now records both the window name and the pane id, matching by id first and falling back to the name.

Two notifications for one question. Agents sometimes fire a hook twice in a row, or one event arrives by two routes. The computer deduplicates by session and content, so a repeat within a few minutes is sent once.

Won’t type

Tapped allow, nothing happened. Usually the question has already passed: the agent timed out or someone at the desk answered. Under the one-time answer rules from the previous post, a late answer is refused and the phone should say so. If it doesn’t, the app is out of date.

Some keys don’t exist on the phone keyboard. No Esc, Tab, arrows or Ctrl. The system adds a row of extra keys above the keyboard, with Ctrl as a sticky key: tap it, then the letter. The remote desktop side had its own trap: the Mac’s screen sharing service expects capitals and symbols to arrive with Shift held, and rejects a bare uppercase letter from a phone keyboard. That took an afternoon to find.

tmux is too old. The system relies on control-mode features from tmux 3.0 onward. Some long-term-support Linux distributions still ship 2.x, and the symptom is that sessions list fine but windows won’t open. The installer now checks the version and asks for an upgrade.

Holes the author dug

Running kill-server inside tmux. An end-to-end test starts its own tmux server for testing and cleans up at the end. The cleanup specified a separate socket directory, but if the script itself runs inside the user’s tmux, the TMUX environment variable overrides that directory, and tmux kill-server kills the user’s own server, along with everything running in it, including the session running the script. Lesson: scripts always pass -S explicitly, or unset TMUX first.

pkill -f matching itself. Cleaning up with pkill -f "some name" when the pkill command line itself contains that name. It kills itself first. An old trap, re-stepped-in every few years.

Tests rewriting the user’s configuration. A test started a resident process under a temporary directory, and that process “helpfully” refreshed the agent’s hook configuration on startup to point at its own path. The user’s real hooks ended up pointing at a temporary directory deleted when the test finished. The fix: only a process in the default install location touches the user’s configuration; anything under a temporary home leaves it alone. A check after every test run now confirms the hooks still point where they should.

Builds overwriting a committed binary. For historical reasons a binary was committed to the repository. One go build in that directory overwrote it with a local build, and git dutifully marked it modified. This happened more than once.

Closing

That is the end of the series. Looking back, there is no startling technology in these tools. They take a few things that are each mature on their own (tmux, peer-to-peer networking, device certificates, agents’ hooks), arrange them under a handful of principles, and then make the right call in a large number of small details: answers are one-time, approval is local, relays see only ciphertext, key files are never transferred.

If a very diligent agent has you chained to your desk too, I hope these posts offer a way out.

The series

  1. Why Reach Your Computer From a Phone
  2. An Architecture of Three Parts
  3. Teaching Coding Agents to Knock
  4. The Computer Decides Who Gets In
  5. Troubleshooting Notes (this post)

Comments

  1. Loading comments…

Leave a comment

Plain text; links are fine. Comments with many links wait for review.