ENG-2300: Add Databricks workflow (job) trigger and list

[kaustav98255]

Summary

This PR adds support for triggering Databricks workflow (job) runs and listing available jobs through the Paradime CLI.

Changes

New Files

• paradime/core/scripts/databricks_workflow.py — Core implementation for Databricks workflow operations:

  • trigger_databricks_workflows() — Triggers multiple job runs in parallel via ThreadPoolExecutor
  • _trigger_single_workflow() — Triggers a single job run via POST to /api/2.1/jobs/run-now
  • _wait_for_run_completion() — Polls run status every 5 seconds via GET /api/2.1/jobs/runs/get until terminal state
  • _get_run_page_url() — Extracts the run page URL from the run response
  • list_databricks_jobs() — Lists all jobs via GET /api/2.1/jobs/list with ID, name, and creation time

• paradime/cli/integrations/databricks_workflow.py — CLI command definitions for Databricks workflow operations

Modified Files

• paradime/cli/run.py — Registered databricks_workflow_trigger and databricks_workflow_list commands

CLI Commands

databricks-workflow-trigger — Trigger Databricks job runs

# Trigger a single job
paradime run databricks-workflow-trigger \
  --host https://adb-xxxx.azuredatabricks.net \
  --token <databricks-token> \
  --job-ids 12345

# Trigger multiple jobs
paradime run databricks-workflow-trigger \
  --host https://adb-xxxx.azuredatabricks.net \
  --token <databricks-token> \
  --job-ids 12345,67890 \
  --wait

# Fire and forget
paradime run databricks-workflow-trigger \
  --job-ids 12345 \
  --no-wait

databricks-workflow-list — List all Databricks jobs

paradime run databricks-workflow-list \
  --host https://adb-xxxx.azuredatabricks.net \
  --token <databricks-token>

# JSON output
paradime run databricks-workflow-list --json

Implementation Details

Aspect	Details
Authentication	Bearer token via Authorization: Bearer {token} header
API Version	Databricks Jobs API 2.1
Job Monitoring	Polls life_cycle_state and result_state every 5 seconds via GET /api/2.1/jobs/runs/get
Lifecycle States	PENDING, RUNNING, TERMINATING, TERMINATED, SKIPPED, INTERNAL_ERROR
Result States	SUCCESS, FAILED, TIMEDOUT, CANCELED, MAXIMUM_CONCURRENT_RUNS_REACHED
Default Timeout	1440 minutes (24 hours)
Env Var Support	DATABRICKS_HOST, DATABRICKS_TOKEN via env_click_option
API Client	requests library

[linear]

ENG-2300 trigger databricks run workflow