Support dynamic IaC for integrations

[jeniawhite]

What does this PR do?

We would like to support dynamic IaC generation for integrations, this change will allow integrations to declare IaC template manifests that will be used to create dynamic IaC for them to be deployed in a frictionless manner.

Why is it important?

Checklist

• I have added test packages to test/packages that prove my change is effective.
• I have added an entry in spec/changelog.yml.

Related issues

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 589ac42

History

• 💚 Build #2298 succeeded 7de13ab
• 💚 Build #2285 succeeded 46d2d28
• 💔 Build #2284 failed 2ce3402
• 💚 Build #2267 succeeded 2d44123

[teresaromero]

Please, lets separate this PR into two where the new type is implemented and other where the changes on input and integrations referencing the new fields.

[bhapas]

Per @teresaromero's review request to split this PR, the changes have been separated into two independent PRs off main:

1. Blueprint package type (+ related tests): Add blueprint package type for canonical IaC base blueprints #1224
2. iac_blueprints / scoped provider_permissions on integration & input (+ related tests): Add iac_blueprints and scoped provider_permissions for integrations/inputs #1225

Please close this PR in favor of those two when convenient. Credit for the original work remains with @jeniawhite (co-authored on the split commits).

[cmacknz]

Do you need a separate blueprint package type? Isn't it enough to allow input and integration packages to define blueprints?

I would expect that the permissions and other definitions in a blueprint would need to be tied to specific integration versions, so it seems preferable to bundle them into the existing package types, rather than have a separate package the likely introduces the need for version compatibility rules.

It seems much nicer to be able to test the AWS package as a complete unit for example.

[cmacknz]

CC @mrodm @teresaromero

[bhapas]

@cmacknz How do we deal if there are like multiple, like 20 odd AWS integrations that need same blueprint , If we limit the blueprint inside one integration then probably we would have to duplicate it for other similar integrations?

The purpose was to have one place of truth for all these blueprints that could be used by multiple packages. Reusing input / integration types might causes problems in Fleet UI if we do not have any Kibana/ES assets probably?

If the naming needs to be generic , like to reuse the package type for any general purpose template / prompt / text / asset storage mechanism then we can use more generic / base package naming convention.

[cmacknz]

Needing to reuse these is valid, but creating a separate package type creates a compatibility check problem as I mentioned so if reuse is the only reason for a separate package we maybe need another way to reuse configuration in packages.

It should be possible for us to create a way to reuse configuration across integrations without creating a new releasable user artifact. We should not be publishing packages whose main purpose is just config reuse in other packages.

[teresaromero]

i was thinking on this re-use and had the links idea since the last meeting, so i went ahead and checked this out.

The proposal: use elastic-package .link files to share the canonical blueprint asset across integration packages.

The .link mechanism already exists — a .link file is a plain text pointer ( ) that elastic-package resolves transparently at build time. The built package zip contains the real file content, not the pointer. elastic-package links update/check keeps checksums in sync and fails the build on drift.

I ran a POC on the elastic/integrations repo to verify this approach: elastic/integrations#20721

• Created account.cloudformation.json.link files inside the iac/ directories of aws, cloud_asset_inventory, and cloud_security_posture, all pointing to a single shared source.
• elastic-package links update computed checksums for all three.
• elastic-package build on cloud_asset_inventory produced a zip with iac/account.cloudformation.json — real content, correct filename — confirming that LinksFS resolves iac/*.link at build time, not just elasticsearch/ingest_pipeline/ and fields/.
• Cross-package links (pointing outside a package's own tree) work fine.

Where to store the shared source:

1. shared/blueprints/aws/federated-identity/account.cloudformation.json at repo root — neutral, no ownership coupling.
2. Inside one authoritative integration (e.g., aws) — makes ownership explicit but creates a dependency.

What this avoids: no new type: blueprint in the spec, no EPR changes, no new elastic-package support for a new package type. The canonical template is just a file — shared via a mechanism the toolchain already understands.

@bhapas The iac folder will contain "blueprint" + patches; would the service that patches permissions be able to grab from this folder all the files?

@mrodm will this be a challenge for backporting a package? having a linked file source on root repo? would it be better to have it on a package and link the other from there?

from this we could also update elastic-package so it can simulate the permissions patching and have this asset tested during development; as we do now with policies with fleet (thinking out loud)

[mrodm]

@mrodm will this be a challenge for backporting a package? having a linked file source on root repo? would it be better to have it on a package and link the other from there?

@teresaromero Yes, I think linking files between different packages is going to be problematic when doing backports.

Currently, we suggest that new backport branches should be created just with the required package and remove the others. If the linked file is going to be located in another package, when creating the backport branch it is likely that the linked file does not exist. In that case one option is that the owners should copy the source of the linked file from the commit where the backport branch was created.

In order to support that, scripts that copy the contents of the backport branches should also to take into account the source of the linked files to ensure that any other necessary package is kept. And this is not considered currently in this process.

If the linked file is in the root of the repository could also cause some issues. For instance, when a change in that blueprint folder in the root of the repository is introduced, it is not going to be easy to know to which packages affect and which branches should be backported. If possible, I would try to avoid this (thinking in backports).

EDIT:

And for instance, changes in the packages containing the source of the links will not be suggested as part of the backport process automation.

It is difficult to ensure that the backport process automation would take care of all these scenarios.

[mrodm]

Created issue to ensure that the source packages of the linked files are copied too elastic/integrations#20726

[mrodm]

Created issue to ensure that the source packages of the linked files are copied too elastic/integrations#20726

[bhapas]

@cmacknz Agree that we should not be publishing packages that just package a reusable config for other [ specific set of ] packages.

@teresaromero

The .link mechanism already exists — a .link file is a plain text pointer ( ) that elastic-package resolves transparently at build time. The built package zip contains the real file content, not the pointer. elastic-package links update/check keeps checksums in sync and fails the build on drift.

This sounds good. Except that I have a few questions around this.

• This approach means that every built package will have a copy of blueprint in it , right?
• Let's say , we want to develop a new development version of the blueprint [ Like .../aws/fi/account-v2-preview ] , I guess this should be possible?
• If we want multiple versions of blueprints to co-exist I guess we can just create additional folders for the versions [ aws/federated-identity/account-v1 ] [ aws/federated-identity/account-v2 ] where major bump is for non-backwards compatible changes to the template.
• For simplicity , I guess each package has to have one base template reference at all levels - policy_template , data_stream , integration , input levels.
• When there is a base template update , I guess all the downstream packages need a new version release [ Probably rightly so , rather than silently updating a template ].

@bhapas The iac folder will contain "blueprint" + patches; would the service that patches permissions be able to grab from this folder all the files?

Yes , it should be possible if we can read the specific resources from the gcloud bucket.

@mrodm will this be a challenge for backporting a package? having a linked file source on root repo? would it be better to have it on a package and link the other from there?

For backport branches , specific package versions can have specific base template instead of a link. That way we work with the specific template manually in that backport branch.

[cmacknz]

The link concept solves this, the backporting issues can be solved through the tooling updates Mario suggested hopefully. One day might just have to commit to maintaining backport branches of the entire integrations repository if this is troublesome.

I don't think the link concept prevents anything related to breaking changes or multiple versions of a blueprint at once. You can change what the links point to, have current and next versions of a blueprint etc. they would just all coexist on the main branch. Potentially you might want a version

Maybe we need to iterate on the ergonomics of this a bit but conceptually this direction makes sense to me.