feat(parameters): pluggable secret/param storage with 4 providers by fedemaleh · Pull Request #190 · nullplatform/scopes

fedemaleh · 2026-06-17T19:49:01Z

No description provided.

fedemaleh · 2026-06-23T17:26:27Z

+mkdir -p parameters/tests/providers/<provider_name>
+```
+
+`<provider_name>` is `snake_case` and is what users will set in `SECRET_PROVIDER` / `PARAMETER_PROVIDER`.


Esto en realidad tiene coincidir con el slug del provider de tipo parameters-storage.

fedemaleh · 2026-06-23T17:33:04Z

+
+---
+
+## Step 4: Write `fetch_configuration` (optional)


Eso creo que dijimos de sacarlo

fedemaleh · 2026-06-23T17:35:02Z

+1. Set the env var: `SECRET_PROVIDER=<provider_name>` and/or `PARAMETER_PROVIDER=<provider_name>`.
+2. If using `fetch_configuration`, the platform team needs to ensure the fetch mechanism (np CLI, REST endpoint, etc.) returns the JSON shape your provider expects.
+
+Done. The new provider is reachable from every workflow without any other change.


Esto no va más, vamos a sacar esta información del campo provider que viene en el payload.

fedemaleh · 2026-06-23T17:36:49Z

+
+A monolithic scope tied to one backend forces fork-and-modify for every variation. This package inverts the relationship: the **dispatch layer is the package**, the **backends are pluggable modules** dropped into `providers/`.
+
+The platform decides which provider handles each parameter — there is no per-environment / per-agent configuration of "which provider to use". The notification payload carries that information directly.


Esto no es cierto. La plataforma decide pero hay configuraciones per-environment, hay que configurarlas vía el provider de tipo parameters-storage

fedemaleh · 2026-06-23T17:39:16Z

+
+The provider's configuration travels in the same payload at `provider.attributes`. `build_context` exports it as `PROVIDER_CONFIG` (a JSON string). Each provider's `setup` reads from `PROVIDER_CONFIG` via `get_config_value --provider '.field'` to extract specific fields (region, kms_key_id, etc.).
+
+This means **there is no per-environment configuration of "which provider"** — the platform decides per-parameter. A single agent can serve parameters routed to Vault and secrets routed to Secrets Manager at the same time, without any agent-side configuration.


Lo mismo que antes, se configura vía una configuración.

Lo de que un mismo agente puede servir múltiples storage es correcto.

fedemaleh · 2026-06-23T17:40:39Z

+| **`provider.specification_id`** | **UUID identifying which provider handles this parameter** |
+| **`provider.attributes`** | **Provider-specific configuration (region, vault address, etc.)** |
+| `provider.nrn` | Provider-instance NRN (informational) |
+| `provider.dimensions` | Provider-instance dimensions (informational, different from parameter dimensions) |


Dejemos muy claro que esto no hay que usarlo

fedemaleh · 2026-06-23T17:40:52Z

+`build_context` calls:
+
+```bash
+np provider specification read --id <provider.specification_id> --output json


--format json

fedemaleh · 2026-06-23T17:42:27Z

+Two things that used to be design points but are obsolete now:
+
+- **`SECRET_PROVIDER` / `PARAMETER_PROVIDER` env vars** — not needed. The platform sends `specification_id` per parameter, so there's no global "which provider to use" setting.
+- **`fetch_configuration` scripts per provider** — not needed. Config comes in the payload as `provider.attributes`, no separate fetching step.


Borrar esto, esta es la primera implementación, no tiene sentido documentar decisiones intermedias que nunca vieron la luz.

fedemaleh · 2026-06-23T17:45:57Z

+
+| Kind        | Storage location                                  | This package |
+|-------------|---------------------------------------------------|--------------|
+| Clear-text  | nullplatform API (its own datastore)              | Not involved |


nullplatform API no es clear-text. Creo que los kind son: nullplatform-storage, third-party-storage

fedemaleh · 2026-06-23T17:46:23Z

+| Clear-text  | nullplatform API (its own datastore)              | Not involved |
+| Secret      | External provider (this package: AWS SM)          | Used         |
+
+Only **secret** parameters trigger the `store` / `retrieve` / `delete` workflows. Clear-text values never leave nullplatform and never touch AWS SM. From the operator's perspective: this provider is invisible until a parameter is marked as a secret.


Esto es mentira, se puede configurar secret manager para parámetros no secretos, es costoso pero es algo que se puede hacer.

fedemaleh · 2026-06-23T17:47:01Z

+```
+parameters/<external_id>
+```
+
+- **`parameters/`** — fixed prefix. This is the IAM anchor: it lets a single resource ARN pattern (`arn:...:secret:parameters/*`) cover everything this provider manages, and nothing else. Removing the prefix would force IAM to either allow account-wide access or maintain an enumerated list of secret names (impractical, since names are generated at runtime).
+- **`<external_id>`** — UUIDv4 generated by the `store` script via `uuidgen` (with `openssl rand -hex 16` as a portable fallback). This becomes the canonical handle nullplatform persists; subsequent `retrieve` / `delete` actions get it back via `NP_ACTION_CONTEXT.notification.external_id`.


Esto no sigue lo que hablamos, el path debería incluir los slugs / ids de entidades y el nombre del param además del uuid

También documentar en la docu global, los paths donde se guardan las cosas deben seguir el principio rector que si alguien entra al storage layer manualmente (secret manager, vault, etc.) debe poder encontrar lo que busca, siempre tiene que ser human friendly.

fedemaleh · 2026-06-23T17:50:49Z

+  "parameter_id": 42,
+  "value": "the-actual-secret",
+  "stored_at": "2026-05-05T12:34:56Z",
+  "external_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"


Agregaría un "managed_by":"nullplatform"

fedemaleh · 2026-06-23T17:52:30Z

+1. **Don't store non-secrets here.** Clear-text parameters belong in nullplatform's API. Mis-classification is the most common cost regression.
+2. **Delete promptly.** `--force-delete-without-recovery` (already used) ends billing immediately. Without it, AWS keeps charging for the 7–30 day soft-delete window.
+3. **Avoid replication unless you need DR.** Each replica is a full $0.40/month.
+4. **Cache at the consumer side.** Each `GetSecretValue` is a billable call. For high-fanout reads (e.g., autoscaling pods all pulling the same value), have a single retrieve at deploy-time and inject as env var, rather than per-pod API calls.


Esto quitar, es responsabilidad de la plataforma optimizar esto, no del user.

fedemaleh · 2026-06-23T17:52:57Z

+### Comparison with self-hosted Vault
+
+This isn't apples-to-apples:
+
+- **Vault**: no per-secret fee. Cost is the underlying infra (EC2/EKS, storage, ops time). Below ~250 secrets, self-hosted Vault tends to be cheaper on paper but more expensive in operator hours.
+- **AWS SM**: linear in secret count, zero ops. Above ~250 secrets the costs converge; the trade-off becomes operational rather than financial.
+


No me interesa esta comparación. Sacar.

fedemaleh · 2026-06-23T17:54:43Z

+
+### Idempotency
+
+- `delete` is idempotent: it suppresses errors with `|| true` and always returns `{success: true}`. Re-deleting a missing secret is a no-op.


Esto es una mala decisión, sólo tiene que hacer supress del error de not found, los otros errores (ej: falta de permiso de IAM deberían reportarse y marcarse como error)

fedemaleh · 2026-06-23T17:56:06Z

+
+### ARN shape
+
+When you `CreateSecret --name parameters/<uuid>`, AWS appends a random 6-character suffix to the ARN:


Más que parameters/ creo que el path debería ser nullplatform/

fedemaleh · 2026-06-23T17:56:52Z

+## Wildcards: which ones are OK
+
+There are two distinct uses of `*` in IAM, frequently conflated:
+
+| Pattern                                              | Meaning                              | Allowed here? |
+|------------------------------------------------------|--------------------------------------|---------------|
+| `"Resource": "*"`                                    | All resources of all types in the account | **No**        |
+| `"Resource": "arn:...:secret:parameters/*"`          | Path glob on the `parameters/` prefix     | **Yes**       |
+
+The second is not a privilege escalation — it is the only way to express "all secrets owned by this provider" given that secret names are UUIDs generated at runtime and cannot be enumerated in advance. Avoiding it would force either explicit per-secret policies (impossible for unknown UUIDs) or `Resource: "*"` (much wider).


Sólo documentemos la necesidad que es acceder a secrets cuyo inicio es nullplatform/*

fedemaleh · 2026-06-23T17:57:22Z

+
+---
+
+## Splitting agent vs consumer


Sacar esta sección, es irrelevante para esta implemetnación

fedemaleh · 2026-06-23T18:00:13Z

+
+For reference — these are commonly requested but **not** needed by the current scripts and should be denied unless a specific use case is documented:
+
+- `secretsmanager:PutSecretValue` — would let the agent overwrite values. Not used; secrets are immutable in this design.


Hay algo que creo que se está confundiendo.

El modelo de datos es así:

Hay un parameter id (ej: name=DB_PASSWORD, type=secret, id=1) que tiene múltiples values (ej: env:prod value=abcd).

Los values son inmutables, cada vez que se modifica para una misma combinación de dimensiones / scope nrn se genera una nueva versión. El modelo de datos debería guardar dentro del mismo secret todo el versionado, dado que tenemos que soportar ver y restaurar versiones viejas.

fedemaleh · 2026-06-24T12:48:19Z

+
+### `retrieve`
+
+Read the value, return `{value}` or `{value: "value not found"}` on miss.


Por que no tiras un error si value not found?

fedemaleh · 2026-06-24T12:49:26Z

+
+### `delete`
+
+Always returns `{success: true}`. Suppress errors with `|| true`.


Esto esta mal, sólo supress not found errors, el resto de los errores hay que reportarlos y fallar.

El not found error lo suprimimos para ser indepotentes y porque ya está hecho el trabajo, pero no podemos dar un resultado exitoso si el trabajo no se hizo

fedemaleh · 2026-06-24T13:02:17Z

+`build_context` resolves this UUID into a slug using the np CLI:
+
+```
+np provider specification read --id <specification_id> --output json


En np cli no existe un flag --output es --format json.

fedemaleh · 2026-06-24T13:04:29Z

+parameters/
+├── entrypoint              # Action router (action → workflow)
+├── build_context           # Resolves ACTIVE_PROVIDER from spec_id, sources setup
+├── store, retrieve,        # Dispatch one-liners


Podríamos tener un script dispatch que reciba el nombre del action y sepa hacer el dispatch. Me imagino algo así:

#!/bin/bash set -euo pipefail # Dispatch: delegate to the active provider's retrieve implementation. # PROVIDER_DIR is exported by build_context. source "$PROVIDER_DIR/$ACTION"

Y en los workflows:

- name: store type: script file: "$SERVICE_PATH/parameters/dispatch" configuration: ACTION: store

Pondría todos los scripts compartidos en utils así no quedan sueltos.

fedemaleh · 2026-06-24T13:13:45Z

+
+For a parameter with:
+
+- Entities: `organization=1255165411`, `account=95118862`, `namespace=37094320`, `application=321402625`


Falta aclarar que las dimensiones pueden no existir y que opcionalmente también puede haber una entidad scope.

Aclarar esto en el doc global y en cada provider.

fedemaleh · 2026-06-24T13:19:36Z

+| Error condition                        | `store`           | `retrieve`               | `delete`           |
+|----------------------------------------|-------------------|--------------------------|--------------------|
+| Resource exists (on store)             | New version added | N/A                      | N/A                |
+| ResourceNotFoundException              | N/A               | `{value: "value not found"}` | Idempotent success |


En retrieve hay que fallar, no mandemos el value not found te lo aclare en el general y te pedi que lleves esa decision a todos los providers.

fedemaleh · 2026-06-24T15:01:22Z

+      "region"
+    ],
+    "properties": {
+      "region": {


Quitar este. Eso viene del role que use el agente con sus credenciales.

fedemaleh · 2026-06-24T15:01:34Z

+        "title": "AWS Region",
+        "description": "AWS region where secrets will be stored (e.g. us-east-1)"
+      },
+      "name_prefix": {


Esto tampoco va.

…ng (hyphenated)

… per-instance for_each)

…ION_ACTION env var

…from CONTEXT.scope.nrn

…el secrets

…issions

…entrypoint wall clock

…xt.notification_parse + assume_role_step.resolve_arn markers

…~0.9s on warm calls)

…ials, drop NP_STS_CACHE_DIR/DISABLE env vars

…ate)

…ullplatform/ prefix, region from IRSA

… doesn't allow '=')

…, action-aware entity fetch (single-wave prefetch)

…l provider schemas

…900ms on store)

…nfiguration.json.tpl inside

…r.iam_role)

…fu strict typing

…ayload)

…cross providers

…nore *.env

fedemaleh commented Jun 23, 2026

View reviewed changes

Comment thread parameters/docs/adding_a_provider.md

fedemaleh commented Jun 23, 2026

View reviewed changes

Comment thread parameters/providers/aws-secrets-manager/docs/architecture.md

fedemaleh commented Jun 24, 2026

View reviewed changes

Comment thread parameters/docs/adding_a_provider.md

fedemaleh commented Jun 24, 2026

View reviewed changes

fedemaleh added 29 commits July 3, 2026 13:11

refactor(parameters): align provider slugs with platform-derived nami…

89bf7a9

…ng (hyphenated)

feat(parameters): tofu install modules for all four providers (spec +…

e254878

… per-instance for_each)

fix(parameters): read action from $CONTEXT.action, not from NOTIFICAT…

d9175cb

…ION_ACTION env var

fix(parameters): build NRN from CONTEXT.entities/value_entities, not …

fcef4de

…from CONTEXT.scope.nrn

fix(parameters): build_external_id reads value_entities for scope-lev…

e5a4e4c

…el secrets

perf(parameters): parallel np-read prefetch + timing logs (debug)

322f46f

docs(parameters): add secretsmanager:TagResource to required IAM perm…

eed7b69

…issions

perf(parameters): timing logs for AWS/Vault/Azure calls + dispatch + …

37fa09c

…entrypoint wall clock

perf(parameters): per-np-call timings + entrypoint.prep + build_conte…

52b136a

…xt.notification_parse + assume_role_step.resolve_arn markers

perf(parameters): cache sts:AssumeRole creds on disk with TTL (skips …

9684828

…~0.9s on warm calls)

refactor(parameters): hardcode sts cache dir to $SERVICE_PATH/credent…

9acfacf

…ials, drop NP_STS_CACHE_DIR/DISABLE env vars

fix(parameters): sts cache reads precomputed epoch (handles busybox d…

056bc0e

…ate)

docs(parameters): aws-parameter-store iam/architecture — hardcoded /n…

19b98e9

…ullplatform/ prefix, region from IRSA

fix(parameters): sanitize '=' to '_' in aws-parameter-store path (SSM…

e6af78c

… doesn't allow '=')

perf(parameters): add ps.store/retrieve/delete granular timing markers

af6ad6b

perf(parameters): use payload's specification_slug + value_dimensions…

82cc556

…, action-aware entity fetch (single-wave prefetch)

feat(parameters): wrap config under setup{} + add sensibility{} to al…

141f7b5

…l provider schemas

perf(parameters): drop managed_by tagging from AWS providers (saves ~…

1e0f9d0

…900ms on store)

refactor(parameters): rename install/ → specs/ and move <provider>-co…

d7b46af

…nfiguration.json.tpl inside

feat(parameters): optional IAM role creation in AWS specs modules (va…

df629c7

…r.iam_role)

fix(parameters): conditional IAM policy at string level to satisfy to…

c68cf49

…fu strict typing

refactor(parameters): drop spec-lookup fallback (slug now always in p…

7b4cb27

…ayload)

Do not require service path

c0fac2c

Use correct relative path to scripts

a9af6cb

feat(parameters): per-instance agent api key + notification channel a…

3293068

…cross providers

chore(parameters): document per-instance channel vars in examples; ig…

8be215a

…nore *.env

Improve terraform installation

a0b9b7c

Improve configuration specification wordings

38fdb5f

chore(parameters): remove benchmarking timers from workflow scripts

6673e0f

fedemaleh force-pushed the feature/parameters-package branch from 02d8803 to 6673e0f Compare July 3, 2026 16:11


		A monolithic scope tied to one backend forces fork-and-modify for every variation. This package inverts the relationship: the dispatch layer is the package, the backends are pluggable modules dropped into `providers/`.

		The platform decides which provider handles each parameter — there is no per-environment / per-agent configuration of "which provider to use". The notification payload carries that information directly.


		The provider's configuration travels in the same payload at `provider.attributes`. `build_context` exports it as `PROVIDER_CONFIG` (a JSON string). Each provider's `setup` reads from `PROVIDER_CONFIG` via `get_config_value --provider '.field'` to extract specific fields (region, kms_key_id, etc.).

		This means there is no per-environment configuration of "which provider" — the platform decides per-parameter. A single agent can serve parameters routed to Vault and secrets routed to Secrets Manager at the same time, without any agent-side configuration.


		### Idempotency

		- `delete` is idempotent: it suppresses errors with `\|\| true` and always returns `{success: true}`. Re-deleting a missing secret is a no-op.


		### ARN shape

		When you `CreateSecret --name parameters/<uuid>`, AWS appends a random 6-character suffix to the ARN:


		For reference — these are commonly requested but not needed by the current scripts and should be denied unless a specific use case is documented:

		- `secretsmanager:PutSecretValue` — would let the agent overwrite values. Not used; secrets are immutable in this design.


		### `retrieve`

		Read the value, return `{value}` or `{value: "value not found"}` on miss.


		### `delete`

		Always returns `{success: true}`. Suppress errors with `\|\| true`.


		For a parameter with:

		- Entities: `organization=1255165411`, `account=95118862`, `namespace=37094320`, `application=321402625`

Uh oh!

Conversation

fedemaleh commented Jun 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fedemaleh Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fedemaleh Jun 24, 2026 •

edited

Loading