dagster.daemon.heartbeat

Daemon heartbeat signal

Dimensions:None

Technical Annotations (29)

Configuration Parameters (3)

statement_timeout

PostgreSQL timeout causing query cancellation - increase if queries legitimately slow, or improve database resources

DAGSTER_RUN_QUEUE_PAGE_SIZErecommended: >100 for large queues

Default 100 processes queue too slowly when hundreds/thousands of runs queued (1.11.3+)

dagsterDaemon.heartbeatTolerance

Decreasing this value to trigger auto-restart does not resolve daemon hangs

Error Signatures (4)

psycopg2.errors.QueryCanceled) canceling statement due to statement timeoutexception

DagsterExecutionInterruptedErrorexception

SIGINTerror code

SIGTERMerror code

CLI Commands (7)

docker compose stop dagster-daemonremediation

docker compose start dagster-daemonremediation

SELECT asset_key, dagster_event_type, timestamp FROM event_logs WHERE asset_key = '["my_group", "my_layer", "removed_asset"]' ORDER BY timestamp DESC;

diagnostic

SELECT asset_key, last_materialization_timestamp, wipe_timestamp FROM asset_keys WHERE asset_key = '["my_group", "my_layer", "removed_asset"]';

diagnostic

dagster dev -w workspace.yamldiagnostic

kill -9remediation

dagster-daemon rundiagnostic

Technical References (15)

freshness daemoncomponentwipeAssetscomponentevent_logscomponentasset_keyscomponentFRESHNESS_STATE_CHANGEconceptcode locationconceptjob_tickscomponentSingleScheduleQuerycomponentRun Queue Coordinator DaemoncomponentQueued Run Coordinatorcomponentpg_logfile pathpg_stat_activitycomponentDockerRunLaunchercomponentDAGSTER_HOMEfile pathworkspace.ymlfile path

Related Insights (5)

Removed assets permanently reappear in catalog after wipeAssets callwarning

▸

PostgreSQL statement timeout prevents schedule tick processingcritical

▸

Run Queue Daemon hangs with stopped heartbeats and queued jobs accumulatecritical

▸

Dagster daemon heartbeat stops without error logscritical

▸

Dagster daemon terminated by OS signal due to resource exhaustioncritical

▸