Skip to content

Add a special behavior for fail.if.no.data=false with ingestion operations marked as critical=true #781

Description

@yruslan

Background

Currently, if fail.if.no.data=true and there is no data to ingest, the pipeline will continue trying to apply all the transformations. They will fail because data is not loaded. The proposal is when both:

  • fail.if.no.data=false
  • critical=true for an ingestion operation

stop the job, but not as a failure, but as a success with a warning.

Feature

Add a special behavior for fail.if.no.data=false with ingestion operations marked as critical=true.

Example [Optional]

--

Proposed Solution [Optional]

--

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions