#781 Add graceful handling of no data ingestions

[yruslan]

Overview

When an ingestion job is marked as critical, and 'fail.if.no.data' is false, it is considered a warning is there is no data, but the pipeline is stopped anyway.

Release Notes

• When an ingestion job is marked as critical, and 'fail.if.no.data' is false, it is considered a warning is there is no data, but the pipeline is stopped anyway.

Related

Closes #i81

Summary by CodeRabbit

• Bug Fixes

  • Critical jobs that receive no data are now correctly reported as unsuccessful.
  • Pipelines with non-failing, no-data outcomes are now classified as warnings.
  • Warning indicators consistently include graceful no-data results.

• Improvements

  • Logging configuration now uses the current format and reports errors more clearly.
  • Added coverage for pipeline runs with both available data and no input data.

[coderabbitai]

Warning

Review limit reached

@yruslan, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 43 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: de7098a4-1be2-4fe0-aa19-5d4335590b23

📥 Commits

Reviewing files that changed from the base of the PR and between 0d86016 and 9a9bf76.

📒 Files selected for processing (2)

• pramen/core/src/main/scala/za/co/absa/pramen/core/utils/ThreadUtils.scala
• pramen/core/src/test/resources/test/config/integration_no_data_graceful.conf

Walkthrough

The change updates eager-job handling for critical NoData(false) results, classifies non-failing no-data outcomes as pipeline warnings, adds graceful no-data integration coverage, and migrates test Log4j configurations to Log4j 2 syntax.

Changes

No-data status handling

Layer / File(s)	Summary
No-data status classification pramen/core/src/main/scala/za/co/absa/pramen/core/...	Critical eager jobs now fail on NoData(false). Non-failing no-data outcomes now produce pipeline warnings.
Graceful no-data integration coverage pramen/core/src/test/..., pramen/extras/src/test/resources/log4j2.properties	Integration tests cover data and no-data runs. Test logging uses Log4j 2 properties with ERROR output to standard error.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• AbsaOSS/pramen#781 — Directly covers critical NoData(false) handling and warning classification.

Possibly related PRs

• AbsaOSS/pramen#611 — Both changes update PipelineStateImpl no-data and warning classification.
• AbsaOSS/pramen#673 — Both changes address no-data handling and pipeline status classification.
• AbsaOSS/pramen#697 — Both changes extend critical-job and no-data handling.

Poem

A rabbit checks the data stream,
No rows appear, yet warnings gleam.
Critical jobs must mark the fall,
Graceful paths still pass them all.
Tests hop through parquet bright.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: graceful handling of ingestion jobs with no data.
Description check	✅ Passed	The description includes the required Overview, Release Notes, and Related sections and accurately describes the no-data behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/781-graceful-handling-of-no-data-ingestions

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@pramen/core/src/test/resources/test/config/integration_no_data_graceful.conf`:
- Line 20: Add the missing fail.if.no.data = false setting to the
integration_no_data_graceful configuration alongside fail.if.no.late.data, so
the scenario explicitly uses RunStatus.NoData(false) for critical ingestion.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e97a064e-b0b1-4489-b114-4c9182a2223b

📥 Commits

Reviewing files that changed from the base of the PR and between 10fe0bf and 0d86016.

📒 Files selected for processing (6)

• pramen/core/src/main/scala/za/co/absa/pramen/core/runner/jobrunner/ConcurrentJobRunnerImpl.scala
• pramen/core/src/main/scala/za/co/absa/pramen/core/state/PipelineStateImpl.scala
• pramen/core/src/test/resources/log4j2.properties
• pramen/core/src/test/resources/test/config/integration_no_data_graceful.conf
• pramen/core/src/test/scala/za/co/absa/pramen/core/integration/NoDataGracefulSuite.scala
• pramen/extras/src/test/resources/log4j2.properties

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Set the no-data option used by this scenario.

Line 20 configures fail.if.no.late.data, not fail.if.no.data. Add fail.if.no.data = false so this integration scenario explicitly exercises RunStatus.NoData(false) for the critical ingestion.

Proposed fix

 pramen {
+  fail.if.no.data = false
   fail.if.no.late.data = false

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pramen/core/src/test/resources/test/config/integration_no_data_graceful.conf`
at line 20, Add the missing fail.if.no.data = false setting to the
integration_no_data_graceful configuration alongside fail.if.no.late.data, so
the scenario explicitly uses RunStatus.NoData(false) for critical ingestion.

[github-actions]

Unit Test Coverage

Overall Project	76.9%	🍏
Files changed	100%	🍏

Module	Coverage
pramen:core Jacoco Report	77.89%	🍏

Files

Module	File	Coverage
pramen:core Jacoco Report	ConcurrentJobRunnerImpl.scala	89.04%	🍏
	PipelineStateImpl.scala	84.47%	🍏
	ThreadUtils.scala	84.1%	🍏