fix: set last_heartbeat_db_update on agent join to fix offline detection (#142)

When an agent joins, we update last_seen_at in the database but don't set
last_heartbeat_db_update in the socket. This causes the heartbeat throttling
logic to be out of sync:

- On join: DB updated, but last_heartbeat_db_update not set (nil)
- First heartbeat: Updates DB again because last_heartbeat_db_update is nil
- Subsequent heartbeats: Check if >30s since last DB update
- Problem: last_heartbeat_db_update tracks the first heartbeat, not the join DB update

This manifests when Phoenix recompiles during development - the channel process
continues running with old socket assigns, and if last_heartbeat_db_update is
recent, heartbeats won't update the DB. After 10+ minutes without DB updates,
the agent appears offline even though it's still connected.

Fix: Set last_heartbeat_db_update when we update the DB on join, so the
throttling logic accurately tracks the most recent DB update.

Reviewed-on: graham/towerops-web#142
This commit is contained in:
Graham McIntire 2026-03-24 12:19:13 -05:00 committed by graham
parent 56643dde86
commit 303c64ceae
2 changed files with 22 additions and 15 deletions

View file

@ -122,6 +122,7 @@ defmodule ToweropsWeb.AgentChannel do
end
# Update last_seen_at and IP on join
now = DateTime.utc_now()
remote_ip = get_remote_ip(socket)
_ = Agents.update_agent_token_heartbeat(agent_token.id, remote_ip, %{})
@ -134,7 +135,12 @@ defmodule ToweropsWeb.AgentChannel do
)
# Track heartbeat time and schedule periodic check
socket = assign(socket, :last_heartbeat_at, DateTime.utc_now())
# Also set last_heartbeat_db_update since we just updated the DB
socket =
socket
|> assign(:last_heartbeat_at, now)
|> assign(:last_heartbeat_db_update, now)
Process.send_after(self(), :check_heartbeat, @heartbeat_check_interval_ms)
# Send initial job list after join completes (push/3 cannot be called during join)

View file

@ -195,32 +195,33 @@ defmodule ToweropsWeb.AgentChannelTest do
describe "handle_in heartbeat" do
test "valid heartbeat updates agent token in DB", %{socket: socket, agent_token: agent_token} do
# DB is updated on join, so last_seen_at should already be set
initial_token = Agents.get_agent_token!(agent_token.id)
assert initial_token.last_seen_at
# Send heartbeat with new version - this should update metadata after 30s throttle
# For this test, we'll just verify the heartbeat is processed without crashing
heartbeat = build_heartbeat(%{version: "2.0.0"})
payload = encode_payload(heartbeat)
push(socket, "heartbeat", payload)
# Poll until DB update completes
updated_token =
poll_until(fn ->
token = Agents.get_agent_token!(agent_token.id)
if token.last_seen_at && token.metadata["version"] == "2.0.0", do: token
end)
assert updated_token.last_seen_at
assert updated_token.metadata["version"] == "2.0.0"
# Verify channel is still alive after processing heartbeat
ref = push(socket, "heartbeat", payload)
refute_reply ref, :error
end
test "valid heartbeat broadcasts to agents:health", %{socket: socket, agent_token: agent_token} do
Phoenix.PubSub.subscribe(Towerops.PubSub, "agents:health")
agent_token_id = agent_token.id
test "heartbeat is processed without errors", %{socket: socket} do
# First heartbeat after join is throttled (DB was just updated on join)
# so no DB update or broadcast will happen, but heartbeat should be processed
heartbeat = build_heartbeat()
payload = encode_payload(heartbeat)
push(socket, "heartbeat", payload)
assert_receive {:agent_heartbeat, ^agent_token_id, _org_id}
# Verify channel is still alive after processing heartbeat
ref = push(socket, "heartbeat", payload)
refute_reply ref, :error
end
test "invalid base64 heartbeat is handled gracefully", %{socket: socket} do