ENG-2299: Add Databricks notebook run trigger

[kaustav98255]

Summary

This PR adds support for submitting Databricks notebook runs through the Paradime CLI using the runs/submit API.

Changes

New Files

• paradime/core/scripts/databricks_notebook.py — Core implementation for Databricks notebook operations:

  • trigger_databricks_notebooks() — Submits multiple notebook runs in parallel via ThreadPoolExecutor
  • _run_notebook() — Submits a single notebook run via POST to /api/2.1/jobs/runs/submit with notebook_task configuration
  • _wait_for_run() — Polls run status every 5 seconds via GET /api/2.1/jobs/runs/get until terminal state
  • _get_run_url() — Retrieves the run_page_url from the Databricks API

• paradime/cli/integrations/databricks_notebook.py — CLI command definition for Databricks notebook trigger

Modified Files

• paradime/cli/run.py — Registered databricks_notebook_trigger command

CLI Commands

databricks-notebook-trigger — Submit Databricks notebook runs

# Run a single notebook
paradime run databricks-notebook-trigger \
  --host https://adb-xxxx.azuredatabricks.net \
  --token <databricks-token> \
  --notebook-paths /Users/me/my_notebook \
  --cluster-id 0123-456789-abcdef

# Run multiple notebooks with parameters
paradime run databricks-notebook-trigger \
  --host https://adb-xxxx.azuredatabricks.net \
  --token <databricks-token> \
  --notebook-paths /Users/me/notebook1,/Users/me/notebook2 \
  --cluster-id 0123-456789-abcdef \
  --parameters '{"env": "production", "date": "2026-03-21"}'

# Fire and forget
paradime run databricks-notebook-trigger \
  --notebook-paths /Users/me/notebook1 \
  --cluster-id 0123-456789-abcdef \
  --no-wait

Implementation Details

Aspect	Details
Authentication	Bearer token via Authorization: Bearer {token} header
API	Databricks runs/submit API 2.1 (one-time runs, no job definition needed)
Notebook Task	Submits notebook_task with notebook_path and optional base_parameters on an existing cluster
Run Monitoring	Polls life_cycle_state and result_state every 5 seconds via GET /api/2.1/jobs/runs/get
Terminal States	TERMINATED with result: SUCCESS, FAILED, TIMEDOUT, CANCELED
Default Timeout	1440 minutes (24 hours)
Env Var Support	DATABRICKS_HOST, DATABRICKS_TOKEN via env_click_option
API Client	requests library

[linear]

ENG-2299 trigger databricks run notebook