Skip to content

fix(ssh): report why a tunnel cannot reach the database instead of a driver timeout (#1981) - #1985

Merged
datlechin merged 6 commits into
mainfrom
fix/1981-ssh-forward-failure-attribution
Jul 29, 2026
Merged

fix(ssh): report why a tunnel cannot reach the database instead of a driver timeout (#1981)#1985
datlechin merged 6 commits into
mainfrom
fix/1981-ssh-forward-failure-attribution

Conversation

@datlechin

Copy link
Copy Markdown
Member

Fixes #1981.

The bug

MySQL 8.0 over an SSH tunnel fails at connect with:

[2013] Lost connection to server at 'handshake: reading initial communication packet', system error: 35 (SQLSTATE: HY000)

The same settings connect in TablePlus. No repro steps, no logs, no screenshots, which is itself part of the bug: TablePro had the reason and did not report it.

errno 35 carries no information

Worth stating up front, because it is the trap here. MariaDB Connector/C does not use SO_RCVTIMEO. It keeps the socket non-blocking and drives timeouts with poll():

while ((r = ma_recv(...)) == -1) {
  if (!ma_socket_wouldblock(socket_errno) || timeout == 0) return r;
  if (ma_pvio_wait_io_or_timeout(pvio, TRUE, timeout) < 1) return -1;
}

Two different failures both end at errno 35:

  • Nothing arrived. recv gives -1/EAGAIN, poll times out, the function returns -1, errno still holds 35.
  • The peer closed. recv gives -1/EAGAIN, poll reports readable because a FIN landed, recv returns 0, the loop exits. errno still holds 35 from the first call.

So errno 35 cannot tell "the tunnel never delivered a byte" from "TablePro closed the socket". Anything that reasons from it is guessing. The underlying connectivity condition is not recoverable from the report, and this PR does not pretend otherwise.

Root cause

Two defects we own, independent of whatever the reporter's server is doing.

The reason is computed and thrown away. SSHForwardChannelOpenPump.run() returns .failed(errorCode) or .timedOut. handleChannelOpenOutcome consumed it as:

case .failed, .timedOut, .cancelled:
    Darwin.close(clientFD)

It went to OSLog and nowhere else. The driver then reached the user first with an errno that means nothing.

The guard against exactly this could never fire. channelOpenDeadlineSeconds was 10s, with a comment claiming it "matches the driver's own connect timeout so TablePro reports the cause before the driver gives up blind". But its clock started in openAndRelay, which runs after accept() returned, after a Task.detached onto acceptQueue at .utility, after a 1000ms accept poll, and after a second Task.detached plus relayQueue.async. The driver's 10s starts at mysql_real_connect(). Equal budget, strictly later start, so TablePro always loses the race.

That makes this a recurrence of #1883, whose test file already describes the symptom: "a stuck open outlives the database driver's connect timeout, leaving an accepted socket that is never written to and never closed, which the driver reports as a greeting-read timeout with no stated cause."

Why a refactor and not a tighter timeout

Shaving seconds off the deadline is the same shape of fix that already failed once. It keeps the outcome dependent on winning a race, and the margin is unenforceable anyway: DatabaseType is open-ended, and a registry plugin's C-level connect timeout cannot be read from Core/SSH.

The deeper problem is that createTunnel returns a port once the SSH hop works, which says nothing about whether forwarding works. A connect() to a listening-but-unaccepted loopback socket succeeds in about 0.1ms, so the driver always connects and then waits. "Tunnel created" needed to start meaning "the destination is reachable".

I also weighed replacing the local listener with NWListener, which TN3151 prefers over BSD sockets, and rejected it. libssh2 is TN3151's own stated exception for an existing socket-based library, SSHChannelRelay is fd-based end to end, and the accept-before-verify behaviour is a kernel backlog property that is identical under either API. It would be a structural change for platform purity with no behavioural gain.

What changed

  • LibSSH2TunnelFactory.probeForwardDestination replaces verifyUnixSocketDestination. It opens the same channel a client would, closes it, and throws a typed error if it cannot. Covers TCP as well as sockets, and reuses SSHForwardChannelOpenPump, so it is bounded rather than the unbounded blocking call the socket path used to make.
  • SSHTunnelError.forwardRefused / .forwardTimedOut, alongside the existing .socketForwardingRefused.
  • SSHForwardFailureRecorder holds the reason for failures that only appear per client, once the tunnel is already running. DatabaseManager+Sessions consults it in the existing catch block and replaces the driver's error. Not a prefix: Beekeeper Studio shipped the prefix version in beekeeper-studio#1787 and reopened it as #4358 because the string still led with the generic error.
  • SSHForwardChannelAttempt.failed and ChannelOpenOutcome.failed now carry libssh2's message, which is what names the cause.
  • Deadline anchored to the accept timestamp, cut to 6s, and the accept poll cut from 1000ms to 200ms so the margin is real.
  • Relay termination logging covers all four cases. Only .transportHangup was logged, so a channel that opened just after the client gave up produced no trace at all.

The .forwardRefused message names the most likely cause directly, because it is the classic mistake and the docs already warned about it while the app did not:

The SSH server could not reach db.internal:3306. That address is resolved from the SSH server, not from your Mac, so a database that only listens on 127.0.0.1 needs Host set to localhost. Also check that sshd allows TCP forwarding (AllowTcpForwarding).

This is why "works in TablePlus" needs no library difference. sshForwardDestination sends the connection's raw host field to sshd, and MySQL 8 defaults to bind-address = 127.0.0.1. A different host value in the other app's entry is the whole story.

Also in this PR

Three confirmed defects found during the investigation, none of them the cause of #1981, each in its own commit:

  • Keep-alive. libssh2_keepalive_send returns against a non-blocking session, so it can answer LIBSSH2_ERROR_EAGAIN when the transport is busy. That was treated as fatal and tore down a healthy tunnel under load, along with every connection on it.
  • NSLocalNetworkUsageDescription. The iOS target declares it, the Mac target did not. macOS 15+ gates outbound TCP to local network addresses behind that permission for non-sandboxed apps too (TN3179), and a denied app fails fast with EHOSTUNREACH. The Mac app is not sandboxed, so entitlements are not involved.
  • ProxyCommand. Zero occurrences in the codebase. SSHConfigResolver dropped it as .unrecognized without a word, so a config that needs it connects somewhere else silently. Now logged, scoped to directives that change routing so the dozens of benign unparsed ones stay quiet. Implementing it is a feature and is not attempted here.

The trade-off

The probe opens and immediately closes a real TCP connection to the database on every tunnel creation, including health-monitor reconnects. That shows as an aborted connection in the server log and increments Aborted_connects. On max_connect_errors host blocking: MySQL resets that counter on any successful connect from the host, and the real connection follows immediately, so it should not accumulate in normal use. Cost is one extra round trip per connect. The Unix-socket path already paid this; it is now symmetric.

Tests

New pure seams, so the logic that used to be unreachable inside private methods is now testable without a live SSH server:

  • ChannelOpenOutcomeTunnelErrorTests: outcome to error mapping, TCP against socket destinations, timeout against refusal.
  • SSHForwardFailureRecorderTests: only failures recorded, consume-once, a later success does not clear an unread reason, latest failure wins.
  • SSHTunnelDeadlineMarginTests: regression guard on the 6s/10s margin and the accept poll granularity. This is the one that would have caught the original bug.
  • SSHKeepAliveResultTests, SSHUnsupportedDirectiveTests, plus a parser test confirming ProxyCommand reaches the classifier and ProxyJump still does not.
  • SSHTunnelErrorTests gains the two new descriptions.

Gap called out rather than hidden: there is still no integration test of LibSSH2Tunnel or SSHTunnelManager orchestration, because both are wired to raw libssh2 OpaquePointers with no fake layer. Building one is out of scope. No UI automation either, since triggering this needs a live SSH server with a controllable refusing target, matching the other network-error paths in the app.

Docs

docs/databases/ssh-tunneling.mdx troubleshooting previously told users to open Console.app and read the LibSSH2Tunnel log, which was a documented workaround for this defect. Replaced with entries for the two new messages, the bind-address trap named explicitly, and a note on the macOS Local Network permission.

Not verified locally

swiftlint and swiftformat are not installed on this machine, and I did not run a build. Syntax checked with swiftc -parse, line lengths checked against the 180 limit, plutil -lint on the plist, and the banned-words gate run on every staged diff.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@mintlify

mintlify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
TablePro 🟢 Ready View Preview Jul 28, 2026, 5:52 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@datlechin
datlechin merged commit e746c7d into main Jul 29, 2026
2 checks passed
@datlechin
datlechin deleted the fix/1981-ssh-forward-failure-attribution branch July 29, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

system error: 35 (SQLSTATE: HY000)

1 participant