dagster.run.failed

Runs that failed

Dimensions:None

Technical Annotations (37)

Configuration Parameters (6)

terminationGracePeriodSecondsrecommended: 120

Grace period for pod shutdown before SIGKILL; configured but may need increase

karpenter.sh/do-not-disruptrecommended: true

Annotation to prevent Karpenter from disrupting pod; configured but seemingly ineffective

cluster-autoscaler.kubernetes.io/safe-to-evictrecommended: false

Annotation to prevent cluster autoscaler from evicting pod

DAGSTER_EXECUTE_K8S_JOB_STREAM_LOGS_RETRIESrecommended: 3

Maximum retries for log stream reads before failing

DAGSTER_EXECUTE_K8S_JOB_STREAM_LOGS_WAIT_BETWEEN_ATTEMPTSrecommended: 5

Wait time in seconds between log stream retry attempts

restart_policyrecommended: Never

Required pod spec setting for log streaming to function

Error Signatures (11)

dagster._core.errors.DagsterExecutionInterruptedErrorexception

urllib3.exceptions.ProtocolError: Response ended prematurelyexception

ValueError: invalid literal for int() with base 16: b''exception

DagsterExecutionStepExecutionErrorexception

current transaction is aborted, commands ignored until end of transaction blocklog pattern

duplicate key value violates unique constraint "concurrency_limits_concurrency_key_key"log pattern

InFailedSqlTransactionexception

UniqueViolationexception

DagsterExecutionInterruptedErrorexception

SIGINTerror code

SIGTERMerror code

CLI Commands (1)

dagster-daemon rundiagnostic

Technical References (19)

K8sRunLaunchercomponentKarpentercomponent_new_signal_handlercomponentexecute_k8s_jobcomponentkubernetes.watch.Watchcomponentdagster_k8s.ops.k8s_job_opcomponentwatch.stream()componentSqlEventLogStoragecomponentPostgresEventLogStoragecomponentConcurrencyLimitsTablecomponentsql_event_log.pyfile pathinstance_concurrency_context.pyfile pathauto-materialize policycomponentfreshness policycomponentasset materializationconceptdata versionconceptDockerRunLaunchercomponentDAGSTER_HOMEfile pathworkspace.ymlfile path

Related Insights (5)

Pod receives termination signal during asset execution on Kubernetescritical

▸

Kubernetes watch client stale state causes job execution failurescritical

▸

Concurrency limit initialization crashes PostgreSQL runs with UniqueViolationcritical

▸

Concurrent asset materializations waste resources and cause dbt run failureswarning

▸

Dagster daemon terminated by OS signal due to resource exhaustioncritical

▸