Background
Currently, if fail.if.no.data=true and there is no data to ingest, the pipeline will continue trying to apply all the transformations. They will fail because data is not loaded. The proposal is when both:
fail.if.no.data=false
critical=true for an ingestion operation
stop the job, but not as a failure, but as a success with a warning.
Feature
Add a special behavior for fail.if.no.data=false with ingestion operations marked as critical=true.
Example [Optional]
--
Proposed Solution [Optional]
--
Background
Currently, if
fail.if.no.data=trueand there is no data to ingest, the pipeline will continue trying to apply all the transformations. They will fail because data is not loaded. The proposal is when both:fail.if.no.data=falsecritical=truefor an ingestion operationstop the job, but not as a failure, but as a success with a warning.
Feature
Add a special behavior for
fail.if.no.data=falsewith ingestion operations marked ascritical=true.Example [Optional]
--
Proposed Solution [Optional]
--