Add Databricks-native retry settings to task operators

providers/databricks/docs/operators/notebook.rst

@@ -42,3 +42,41 @@ Running a notebook in Databricks on an existing cluster
     :language: python
     :start-after: [START howto_operator_databricks_notebook_existing_cluster]
     :end-before: [END howto_operator_databricks_notebook_existing_cluster]
+
+Configuring Databricks-native task retries
+-------------------------------------------
+
+Use ``max_retries``, ``min_retry_interval_millis`` and ``retry_on_timeout`` to configure
+`Databricks-native task retries <https://docs.databricks.com/api/workspace/jobs/create#tasks-max_retries>`_.
+Databricks reruns failed task attempts within the same job run, so Airflow sees only the final result.
+Set ``max_retries`` to ``-1`` to retry indefinitely, or ``0`` to disable retries.
+
+These settings are independent of the Airflow task-level ``retries`` parameter, which retries the
+whole Airflow task:
+
+.. code-block:: python
+
+    DatabricksNotebookOperator(
+        task_id="notebook",
+        notebook_path="/path/to/notebook",
+        source="WORKSPACE",
+        existing_cluster_id="existing_cluster_id",
+        max_retries=3,
+        min_retry_interval_millis=2000,
+        retry_on_timeout=True,
+    )
+
+Airflow ``retries`` behaves differently depending on where the operator runs. For a standalone
+operator, each retry submits a new Databricks run. Inside a
+:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
+the Airflow task monitors a sub-run that was already submitted by the workflow launch task, so a
+retry only re-polls the terminal sub-run. Use ``max_retries`` to retry Databricks work inside a
+workflow task group.
+
+Inside a
+:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
+a task that exhausts a finite ``max_retries`` is reported as failed as soon as its final failed
+attempt is observed, so downstream failure handling is not delayed by long-running sibling tasks.
+Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the
+parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt
+under the same ``task_key`` until then. Sibling tasks in the run continue independently.

providers/databricks/docs/operators/task.rst

@@ -44,3 +44,30 @@ Running a SQL query in Databricks using DatabricksTaskOperator
     :language: python
     :start-after: [START howto_operator_databricks_task_sql]
     :end-before: [END howto_operator_databricks_task_sql]
+
+Configuring Databricks-native task retries
+-------------------------------------------
+
+Use ``max_retries``, ``min_retry_interval_millis`` and ``retry_on_timeout`` to configure
+`Databricks-native task retries <https://docs.databricks.com/api/workspace/jobs/create#tasks-max_retries>`_.
+Databricks reruns failed task attempts within the same job run, so Airflow sees only the final result.
+Set ``max_retries`` to ``-1`` to retry indefinitely, or ``0`` to disable retries.
+
+These settings are independent of the Airflow task-level ``retries`` parameter, which retries the
+whole Airflow task. You can set the same fields directly in ``task_config``. When both are set, the
+operator parameter takes precedence. If a field is unset, Databricks uses its default.
+
+Airflow ``retries`` behaves differently depending on where the operator runs. For a standalone
+operator, each retry submits a new Databricks run. Inside a
+:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
+the Airflow task monitors a sub-run that was already submitted by the workflow launch task, so a
+retry only re-polls the terminal sub-run. Use ``max_retries`` to retry Databricks work inside a
+workflow task group.
+
+Inside a
+:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
+a task that exhausts a finite ``max_retries`` is reported as failed as soon as its final failed
+attempt is observed, so downstream failure handling is not delayed by long-running sibling tasks.
+Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the
+parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt
+under the same ``task_key`` until then. Sibling tasks in the run continue independently.