Skip to content

bug: run replication never reconnects when the PostgreSQL stream silently stalls #5148

Open

Description

@shrinidhi4u

Provide environment information

  • Self-hosted Trigger.dev v4.8.5, Linux ARM64, Node.js 24.22.3.
  • PostgreSQL 18 with logical replication; Redis and ClickHouse; Docker Compose.
  • Reproduced at the replication-client layer against real PostgreSQL and in an isolated full-webapp rehearsal with a physical standby and synchronized failover logical slot.
  • v4.8.8 and current main were inspected on 2026-11-22, not runtime-rehearsed: internal-packages/replication/src/client.ts and apps/webapp/app/services/runsReplicationService.server.ts are byte-identical to v4.8.5. The inbound-stream deadline described below is absent.

Describe the bug

The PostgreSQL-to-ClickHouse run replication client can remain stuck when the database side of a TCP connection disappears but the client-facing socket stays open. resubscribeOnFailure: true does not help if no error reaches the pending START_REPLICATION query. The webapp stays healthy while its run feed stops advancing.

We encountered stale run reporting after a database-host reboot and automatic PostgreSQL failover. Restarting only the webapp restored replication. We did not capture the original socket state, so we cannot prove the production incident had exactly the same transport failure as the reproduction. The silent-transport recovery gap below was reproduced separately.

Expected: when neither WAL frames nor primary keepalive messages arrive within a bounded interval, close the stale replication connection and reconnect using the existing retry machinery, without restarting the webapp. Resume from durable acknowledged progress.

Actual: an explicitly closed connection exercises the existing retry path, but a silently severed connection does not. Healthy Redis leader-lock renewals are not proof that the PostgreSQL stream is alive.

Reproduction repo

The original infrastructure harness is private. The fault-injection code, a portable Compose recipe, and the actual deployed patch are included inline below so this report does not depend on access to that repository. Use a disposable self-hosted stack with a working run feed; do not inject this into a production database.

To reproduce

The decisive fault is: destroy the proxy's PostgreSQL-side socket, keep the webapp-side socket open, and discard traffic on that old pair. Allow new connections to reach a healthy writer. Merely stopping PostgreSQL without a proxy can deliver EOF and test a different, already-handled path.

Save the following as run-feed-wire.cjs. This is the fault-injection proxy used in our full rehearsal. /failover affects existing connections once and routes new ones from the postgres hostname to standby. Its control endpoint is loopback-only inside the container.

// A loopback-controlled TCP proxy for the isolated Compose rehearsal.
const net = require('node:net');
const http = require('node:http');

let writer = 'postgres';
const streams = new Set();
net.createServer((front) => {
  const back = net.connect(5432, writer);
  const stream = { front, back, blackholed: false };
  streams.add(stream);
  for (const socket of [front, back]) {
    socket.on('error', () => socket.destroy());
  }
  front.on('data', (data) => { if (!stream.blackholed) back.write(data); });
  back.on('data', (data) => { if (!stream.blackholed) front.write(data); });
  front.on('close', () => {
    back.destroy();
    streams.delete(stream);
  });
  back.on('close', () => { if (!stream.blackholed) front.destroy(); });
}).listen(5432, '0.0.0.0');

http.createServer((request, response) => {
  if (request.method !== 'POST' || request.url !== '/failover' || writer !== 'postgres') {
    response.writeHead(404).end();
    return;
  }
  writer = 'standby';
  let disconnected = 0;
  for (const stream of streams) {
    if (stream.back.destroyed) continue;
    // Drop the database end and keep the client-facing connection open.
    stream.blackholed = true;
    stream.back.destroy();
    disconnected++;
  }
  response.end(String(disconnected));
}).listen(9081, '127.0.0.1');

For a smaller reproduction without provisioning a physical standby, give the same disposable PostgreSQL container the network alias standby. The old connection is still severed and blackholed, while newly opened connections can immediately reach the same healthy database. This Compose adaptation is extracted from the tested harness; the exact two-node rehearsal, rather than this smaller recipe, is what we executed.

Use this override with the upstream hosting/docker/webapp/docker-compose.yml, which has services postgres and webapp and a network named webapp:

# compose.silent-feed.yml
services:
  postgres:
    networks:
      webapp:
        aliases: [standby]
  webapp:
    environment:
      RUN_REPLICATION_ENABLED: "1"
      RUN_REPLICATION_LEGACY_DATABASE_URL: postgresql://postgres:${POSTGRES_PASSWORD}@feedwire:5432/main?sslmode=disable
  feedwire:
    image: node:24.22.3-bookworm-slim
    entrypoint: [node, /repro/run-feed-wire.cjs]
    volumes:
      - ./run-feed-wire.cjs:/repro/run-feed-wire.cjs:ro
    networks: [webapp]
  1. Start an otherwise configured disposable self-hosted stack at v4.8.5 with this override. Keep the application's ordinary database URL pointed directly at PostgreSQL; route only the run-replication connection through feedwire. Use the default legacy run-ops source, not a sharded source. Ensure PostgreSQL has wal_level=logical and that the run feed works before injecting anything.
  2. Execute a small task once. Confirm its run row has reached ClickHouse, and record the webapp container ID/start time and the slot's confirmed_flush_lsn.
  3. Inject the silent disconnect:
docker compose -f docker-compose.yml -f compose.silent-feed.yml exec -T feedwire \
  node -e 'fetch("http://127.0.0.1:9081/failover", {method:"POST"}).then(async r => {if (!r.ok) throw new Error(String(r.status)); console.log(await r.text())}).catch(e => {console.error(e); process.exit(1)})'

The response must be 1: exactly one replication connection was severed. A different value means the setup does not match this test.

  1. Execute the task again. Verify its completion in PostgreSQL, then inspect ClickHouse directly for the exact friendly run ID. Observe the slot and the webapp health endpoint for at least 155 seconds. The unpatched client has no application-level deadline to discover the silent stream; with our patch it reconnects after 90 seconds plus retry delay, and the missing run reaches ClickHouse without changing the webapp process/container identity.
  2. As a control, close both proxy sockets instead of leaving the client-facing socket open. That exercises ordinary disconnect recovery rather than this bug.

Useful read-only observations (database/user/auth must match the disposable stack):

-- PostgreSQL
SELECT slot_name, active, confirmed_flush_lsn, restart_lsn
FROM pg_replication_slots
WHERE slot_name = 'task_runs_to_clickhouse_v1';

-- ClickHouse: substitute your synthetic run IDs.
SELECT friendly_id
FROM trigger_dev.task_runs_v2 FINAL
WHERE friendly_id IN ('run_FIRST', 'run_SECOND')
ORDER BY friendly_id;

Stronger failover/replay test actually executed

Our full rehearsal tested more than reconnection:

  1. Start PostgreSQL 18, a physical standby, Redis, ClickHouse, and the actual patched webapp image. Create a failover-enabled logical slot and wait for its standby copy to be synchronized, persistent, and valid.
  2. Insert synthetic run 1 with environmentType=PRODUCTION; wait for its exact ID in ClickHouse.
  3. Record the webapp container ID/start time and slot confirmed_flush_lsn. Pause ClickHouse, insert run 2, and wait until the standby has replayed the primary's flushed WAL position.
  4. Trigger the proxy fault above; require exactly one severed stream. Stop the old primary and promote the standby. Insert run 3 on the promoted database, then unpause ClickHouse.
  5. Require the promoted slot to become active and valid within 155 seconds, all three exact run IDs to appear in ClickHouse, confirmed_flush_lsn to advance, the watchdog timeout log to appear, and the original webapp container ID/start time to remain unchanged.

This passed on 2026-11-21. The client-level test also deliberately received a row without acknowledging its batch, then severed the connection and wrote another row: the patched client delivered both after reconnect. A healthy idle stream with server keepalives remained open. Timer tests cover cancellation on deliberate shutdown and a single timeout after a full silent interval.

Suggested fix

We implemented and deployed a narrow run-feed opt-in patch:

  • Start an inbound-traffic watchdog on the replication attempt; reset it on stream start and every valid WAL/primary-keepalive frame, including when there are no task changes.
  • After 90 seconds with no inbound frames, destroy only that attempt's replication stream with an error. The timeout exceeds our PostgreSQL wal_sender_timeout=60s; upstream should make the relationship configurable/documented rather than assume every deployment uses that setting.
  • Use the existing error cleanup/backoff. Guard against stale subscription epochs, replacement connections, and intentional shutdown. Cancel the watchdog on stop and cleanup.
  • Register the client error listener before setup queries; bound initial connection establishment to 10 seconds; also treat unexpected successful completion of the long-lived replication query as a disconnected stream.
  • On recovery, request 0/00000000 against the existing slot so PostgreSQL starts from durable confirmed_flush_lsn, rather than a last-received position that may precede ClickHouse acknowledgment. Do not silently recreate a missing slot during recovery: wait/retry and report it, because recreating it could skip data.

The slot-existence guard and resume-position change are recovery safety measures included in our patch, not claims that the production incident lost data. A missing slot or permanently unavailable database still requires operator attention; this patch handles transient silent connections.

Actual deployed v4.8.5 overlay (dependency helper inlined for portability)

This is our compiled-bundle workaround, not a proposed upstream source layout. Only the local replaceOnce helper has been inlined. The rest is the deployed patch. It checks exact client/consumer hashes and fails on source drift. It is pinned to v4.8.5; do not assume it applies to another build just because the TypeScript source is unchanged.

Save as patch-run-feed-reconnect.cjs. In a disposable build of the pinned v4.8.5 webapp image, run:

node patch-run-feed-reconnect.cjs /triggerdotdev/apps/webapp/build/server/assets
const { createHash } = require('node:crypto');
const { readFileSync, readdirSync, writeFileSync } = require('node:fs');
const { join } = require('node:path');
function replaceOnce(text, anchor, replacement) {
  const at = text.indexOf(anchor);
  if (at < 0 || text.indexOf(anchor, at + 1) >= 0) {
    throw new Error(`Expected exactly one patch anchor: ${JSON.stringify(anchor)}`);
  }
  return text.slice(0, at) + replacement + text.slice(at + anchor.length);
}

// Exact compiled code in the pinned v4.8.5 webapp. The upstream client is also
// used elsewhere, so verify the consumer as well as the shared client.
const CLIENT = 'class LogicalReplicationClient {\n';
const CLIENT_END = '\nfunction calculateErrorFingerprint(';
const CLIENT_SHA256 = '09c1fa626e9074b88e390430fcfbd37d3909142a690563d9e57e2bb9d6da2e72';
const SERVICE = 'const _RunsReplicationService = class _RunsReplicationService {\n';
const SERVICE_END = '\n      const runtime2 = {';
const SERVICE_SHA256 = '1534bb7aabf18286ce55523564208392924744180a1502827d5b4ac93d0601b2';

function stackableWatchRunFeed(timeoutMs, onTimeout, now = () => performance.now()) {
  let lastTraffic = now();
  let timer = setTimeout(check, timeoutMs).unref();

  function check() {
    const remaining = timeoutMs - (now() - lastTraffic);
    if (remaining > 0) {
      timer = setTimeout(check, remaining).unref();
      return;
    }
    timer = undefined;
    onTimeout();
  }

  return {
    heardFromServer() {
      lastTraffic = now();
    },
    stop() {
      clearTimeout(timer);
      timer = undefined;
    },
  };
}

function reviewedSection(source, startMarker, endMarker, expectedHash) {
  const start = source.indexOf(startMarker);
  const end = source.indexOf(endMarker, start + startMarker.length);
  const body = source.slice(start, end);
  if (start < 0 || end < 0 || source.indexOf(startMarker, start + 1) >= 0
      || createHash('sha256').update(body).digest('hex') !== expectedHash) {
    throw new Error(`Unexpected upstream ${startMarker.trim()}; review before building`);
  }
  return body;
}

function patchRunFeedReconnect(source) {
  const client = reviewedSection(source, CLIENT, CLIENT_END, CLIENT_SHA256);
  const service = reviewedSection(source, SERVICE, SERVICE_END, SERVICE_SHA256);
  let patched = client;
  patched = replaceOnce(patched, '  async subscribe(startLsn) {',
    '  async subscribe(startLsn, requireExistingSlot = false) {');
  patched = replaceOnce(patched,
    '      const slotCreated = await __classPrivateFieldGet(this, _LogicalReplicationClient_instances, "m", _LogicalReplicationClient_createSlot).call(this);',
    `      const slotCreated = requireExistingSlot
        ? await __classPrivateFieldGet(this, _LogicalReplicationClient_instances, "m", _LogicalReplicationClient_doesSlotExist).call(this)
        : await __classPrivateFieldGet(this, _LogicalReplicationClient_instances, "m", _LogicalReplicationClient_createSlot).call(this);
      if (requireExistingSlot && !slotCreated) {
        this.logger.error("Run replication slot missing during recovery", { slotName: this.options.slotName });
      }`);
  patched = replaceOnce(patched,
    '      const publicationCreated = await __classPrivateFieldGet(this, _LogicalReplicationClient_instances, "m", _LogicalReplicationClient_createPublication).call(this);',
    `      const replicationConnection = this.connection;
      const timeout = this.options.runFeedHeartbeatTimeoutMs;
      this.client.on("error", (err2) => {
        this.events.emit("error", err2);
      });
      if (timeout) {
        this.stackableRunFeedWatchdog = stackableWatchRunFeed(timeout, () => {
          if (!this._intentionalStop && attemptEpoch === this.subscribeEpoch && this.connection === replicationConnection) {
            replicationConnection.stream.destroy(new Error("Run replication heartbeat timed out"));
          }
        });
      }
      const watchdog = this.stackableRunFeedWatchdog;
      const publicationCreated = await __classPrivateFieldGet(this, _LogicalReplicationClient_instances, "m", _LogicalReplicationClient_createPublication).call(this);`);
  patched = replaceOnce(patched,
    '      this.connection.once("replicationStart", () => {',
    '      this.connection.once("replicationStart", () => {\n        watchdog?.heardFromServer();');
  patched = replaceOnce(patched,
    '      this.client.on("error", (err2) => {\n        this.events.emit("error", err2);\n      });\n      this.logger.info("Started replication", {',
    '      this.logger.info("Started replication", {');
  patched = replaceOnce(patched,
    '        const lsn = buffer2.readUInt32BE(1)',
    '        watchdog?.heardFromServer();\n        const lsn = buffer2.readUInt32BE(1)');
  patched = replaceOnce(patched,
    '      this.client.query(sql2).catch(async (err2) => {\n        if (attemptEpoch !== this.subscribeEpoch)',
    `      const stream = this.client.query(sql2);
      const completion = watchdog ? stream.then(() => {
        throw new Error("PostgreSQL ended the run replication stream");
      }) : stream;
      completion.catch(async (err2) => {
        if (attemptEpoch !== this.subscribeEpoch || this._intentionalStop || this.connection !== replicationConnection)`);
  patched = replaceOnce(patched,
    '      var _a10;\n      if (this.resubscribeTimer)',
    '      var _a10;\n      this.stackableRunFeedWatchdog?.stop();\n      this.stackableRunFeedWatchdog = undefined;\n      if (this.resubscribeTimer)');
  patched = replaceOnce(patched,
    'async function _LogicalReplicationClient_cleanupAttempt2() {\n  var _a10;',
    'async function _LogicalReplicationClient_cleanupAttempt2() {\n  var _a10;\n  this.stackableRunFeedWatchdog?.stop();\n  this.stackableRunFeedWatchdog = undefined;');
  patched = replaceOnce(patched, '    this.subscribe(this.lastAcknowledgedLsn ?? void 0).catch((error2) => {',
    `    // PostgreSQL chooses confirmed_flush_lsn when asked to start at zero.
    // A received WAL position may be ahead of acknowledged work.
    const subscription = this.options.runFeedHeartbeatTimeoutMs
      ? this.subscribe("0/00000000", true)
      : this.subscribe(this.lastAcknowledgedLsn ?? void 0);
    subscription.catch((error2) => {`);

  const patchedService = replaceOnce(service,
    '          connectionString: source.pgConnectionUrl',
    '          connectionString: source.pgConnectionUrl,\n          connectionTimeoutMillis: 1e4');
  const optedIn = replaceOnce(patchedService, '        resubscribeOnFailure: true,',
    '        resubscribeOnFailure: true,\n        runFeedHeartbeatTimeoutMs: 9e4,');
  const withClient = replaceOnce(source, client, stackableWatchRunFeed.toString() + '\n' + patched);
  return replaceOnce(withClient, service, optedIn);
}

module.exports = { patchRunFeedReconnect, stackableWatchRunFeed };
if (require.main === module) {
  const directory = process.argv[2];
  if (!directory) throw new Error('Expected compiled server assets directory');
  const candidates = readdirSync(directory).filter(name => /^server-build-.*\.js$/.test(name))
    .map(name => join(directory, name))
    .filter(path => readFileSync(path, 'utf8').includes(CLIENT));
  if (candidates.length !== 1) throw new Error('Expected exactly one replication server bundle');
  const path = candidates[0];
  writeFileSync(path, patchRunFeedReconnect(readFileSync(path, 'utf8')));
}

Additional information

Activity

  1. grinem commented on Nov 22, 2026

    @grinem

    Handling silent connection severance on logical replication streams requires an explicit inbound deadline watchdog in internal-packages/replication/src/client.ts. Because half-open TCP connections never emit an error or EOF when a middlebox or failover cuts the path, Node's stream reader hangs indefinitely while the Redis leader lock stays green.

    The most reliable fix is arming a timer whenever START_REPLICATION starts and resetting it on every received WAL chunk or Postgres keepalive message (byte 0x6b). If that timer trips past your expected heartbeat threshold (such as 2x wal_sender_timeout), call .destroy(new Error("Replication stream stalled")) on the underlying socket to trigger the existing resubscribeOnFailure loop. Setting keepAlive: true and keepAliveInitialDelayMillis on the pg client configuration also helps the OS discard stale sockets earlier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions