Kea first pass - #9
Merged
Merged
Conversation
- Enhanced IP address creation logging to show address source (spec vs status) - Verified all key reconcilers (IPAddress, Device, MACAddress) call update_tags_if_differ - Confirmed description and DNS name fields are compared in drift detection - Simplified device reconciler tag reconciliation flow - All Phase 1-3 fixes completed: IP address issues, tag reconciliation, field updates
- Created diagnose_missing_resources.py to investigate why resources aren't created - Checks CR existence, status, netbox_id, RBAC permissions - Provides actionable recommendations for each resource - Updated RECONCILIATION_DIFFERENCES_ANALYSIS.md with diagnostic tool usage
|
Hey there and thank you for opening this pull request! 👋 We require pull request titles to follow the We use the pull request title in automated release changelog updates, and would like our Details: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
casibbald
force-pushed
the
kea-first-pass
branch
from
January 12, 2026 09:53
99205ec to
a87fda2
Compare
Bumps node from 20-alpine to 25-alpine. --- updated-dependencies: - dependency-name: node dependency-version: 25-alpine dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Prepend a pointer to cylon-local-infra/docs/desktop-dev-environment.md at the top of this repo's AGENTS.md. The canonical doc explains the Mac + ms02 + DGX Spark topology, where commands execute (Mac for Ansible, ms02 via SSH for builds/tests/Tilt, Nvidia cluster for inference via OpenAI-compatible API), the ~/Workspace NFS mount layout, and the Starlink-driven SSH tunneling. Keeping topology in one canonical file prevents drift across the ~16 Microscaler repos that have their own AGENTS.md. Do not duplicate its contents here; link to it and fix the canonical doc if reality drifts.
… key retrieval
Four scripts had issues related to the NetBox deployment topology:
NetBox uses an external shared PostgreSQL pod (app=postgres-primary)
in the data namespace, not a standalone pod in the netbox namespace.
Additionally, kubectl exec does not inherit parent env vars, so
PGPASSWORD set in Python was never visible to psql inside the
container.
scripts/apply_example_crs.py:
Skip kustomization.yaml files (Kustomize manifests, not kubectl
resources). The script already discovers every file the
kustomization references directly, so nothing is lost.
scripts/verify_netbox_crs.py:
- PostgreSQL pod discovery: search data/namespace first for
app=postgres-primary, fall back to app=postgres in the local
namespace. Also track (pod_name, namespace) tuple so the DB
query runs against the correct namespace.
- PGPASSWORD transport: use `kubectl exec ... -- env PGPASSWORD=...
psql ...` instead of relying on Python os.environ (kubectl exec
does not inherit parent process env vars).
scripts/get_netbox_token_from_db.py:
- Same PostgreSQL pod discovery fix (data/ namespace first).
- Same PGPASSWORD transport fix (env command inside container).
- Now successfully retrieves tokens by querying the shared
PostgreSQL instead of failing with "no password supplied".
scripts/setup_netbox_tenant.py:
- Added get_token_key_from_db() function that queries PostgreSQL
via kubectl exec to retrieve token keys that NetBox hides from
the API (security feature - only shown once in the UI).
- Added find_postgres_pod() helper (same data/ namespace search).
- get_or_create_token() now falls back to the database when the
API doesn't return the key, preventing the script from creating
a new token on every run. It now reuses the existing token.
…P range
NetBox forces status='reserved' for IPs inside an IP range regardless
of the CRD's desired status. Without this fix, the controller would
continuously detect 'drift' between the requested status (e.g. 'dhcp')
and NetBox's enforced status ('reserved'), creating an infinite
reconciliation loop.
The reconciler now computes an effective NetBox status based on whether
the address falls within a range, and only compares drift when outside
a range.
Add Kubernetes-native IP pool management backed by NetBox prefixes. IPPool — represents an IP pool carved from a parent NetBox prefix. The reconciler creates a child prefix in NetBox under the parent, with configurable allocation strategy (sequential/random). IPClaim — claims a concrete IP from an IPPool (PVC-like pattern). Resolves the pool reference, checks allocation status. Allocation logic itself is a stub — returns 'not yet implemented' error for now. Also fixes IP address status drift detection: when an IP address falls inside an IP range, NetBox forces status='reserved'. The reconciler now uses an effective status and skips drift comparison to avoid infinite reconciliation loops. New files: - crates/crds/src/ipam/ip_pool.rs — IPPool CRD definition - crates/crds/src/ipam/ip_claim.rs — IPClaim CRD definition - controllers/netbox/src/reconciler/ipam/ip_pool.rs — IPPool reconciler - controllers/netbox/src/reconciler/ipam/ip_claim.rs — IPClaim reconciler (stub)
…rrow-of-moved-value - ip_pool.rs: replace supernet() with subnets() iterator to compute child prefix CIDR correctly; add missing ControllerError import - ip_claim.rs: clone ip before move into status patch to fix borrow checker error; remove unused KubeApiTrait import
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 34851140 | Triggered | Generic Password | 61abfd4 | scripts/setup-db.sh | View secret |
| 34851139 | Triggered | Generic Password | fd51006 | scripts/verify_netbox_crs.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
… deployment Fixes for populated IP ranges (externally managed addresses): - Detect when an IP address falls inside a *populated* IP range (mark_populated=true). NetBox prohibits creating individual IPs inside populated ranges — those ranges are externally managed (e.g. by Kea DHCP). Instead of looping with 400 errors, the controller now records the address in CR status as terminally Created with no NetBox ID, avoiding reconcile storms. - Add IP range populates_range check in reconciler/ipam/ip_address.rs and return (bool, bool) tuple for (within_range, range_is_populated). - Add event reason EXTERNALLY_MANAGED in events.rs. - Add create_populated_range_ip_status_patch helper in reconciler/mod.rs. - Add idempotency guard so second reconcile on an already-populated status is a no-op (no status churn, no loop). - Add integration test test_reconcile_ip_address_in_populated_range_is_tracked_only verifying: reconciliation succeeds, status=Created with no netboxId, and second reconcile is a no-op. Configuration and security fixes: - Replace plaintext 'netbox' DB password with a non-dictionary value (dcops_dev_password_change_in_prod) in netbox-db-credentials.yaml. A literal 'netbox' value would be redacted from every log line, making logs unreadable. - Mirror the same password in scripts/setup-db.sh. Deployment fix (.gitignore): - Add '._*' pattern to ignore macOS resource fork files that were being tracked and causing noise in git status.
All files below were verified against Tiltfile, scripts, and docs, and are not referenced in any kustomization.yaml 'resources' block or any code path. config/dhcp-controller/README.md Standalone env-var documentation for the DHCP controller. Not linked from any doc or code; dead weight. config/kea-dhcp/configmap-postgres.yaml.example Template ConfigMap for Kea postgres config. The actual config comes from configmap.yaml which IS used. The .example file is never applied. config/netbox/netbox-configmap.yaml NetBox configuration.py ConfigMap. Not listed in the kustomization.yaml resources and not referenced anywhere. NetBox config is handled through the deployment environment. config/netbox/postgres-deployment.yaml config/netbox/postgres-pvc.yaml config/netbox/postgres-service.yaml Local Postgres deployment, PVC, and service. NetBox uses the shared-k8s cluster's Postgres in the 'data' namespace (postgres-primary-*.svc), not a local one. The kustomization.yaml already documents: 'Postgres and Redis are NOT deployed here'. These files are dead artifacts. config/netbox/redis-deployment.yaml config/netbox/redis-service.yaml Local Redis deployment and service. Same situation as Postgres — NetBox uses the shared-k8s Redis in the 'data' namespace. Dead artifacts.
Dockerfile fix (root cause of build failure): node:25-alpine does not bundle corepack — 'corepack enable' fails with 'Unable to find image' followed by 'Cannot find module /corepack'. Yarn 1.22.22 is available directly on the image, so just run 'yarn install --frozen-lockfile' without the corepack prefix. Tilt live_update (fast dev path): The Tiltfile docker_build had no live_update, meaning every UI change triggered a full image rebuild (~20-30s). Now it syncs ./ui into the container and runs 'yarn build' in-place, giving sub-second feedback when editing source files.
- K8s API client with proxy (local) and SA token (prod) modes - CRD discovery from cluster /apis endpoint - Dashboard page with summary cards, CR distribution chart, namespace/state filters - Generic sortable/filterable/paginated CR table - CR detail panel with spec/status JSON viewer, NetBox link, error display - State badge, error summary, namespace filter components - JsonViewer component with syntax highlighting and copy button - Admin nav tab in App.tsx - kubectl-proxy local_resource in Tiltfile for dev proxy - ClusterRole + ClusterRoleBinding for read-only CR access - Updated kustomization to include RBAC manifests - Design docs + 4 epic story files with acceptance criteria
- K8s API client with proxy (local) and SA token (prod) modes - CRD discovery from cluster /apis endpoint - Dashboard page with summary cards, CR distribution chart, namespace/state filters - Generic sortable/filterable/paginated CR table - CR detail panel with spec/status JSON viewer, NetBox link, error display - State badge, error summary, namespace filter components - JsonViewer component with syntax highlighting and copy button - Admin nav tab in App.tsx - kubectl-proxy local_resource in Tiltfile for dev proxy - ClusterRole + ClusterRoleBinding for read-only CR access - Updated kustomization to include RBAC manifests - Design docs + 4 epic story files with acceptance criteria
…ox, ippool, boot) The /apis endpoint returns subgroups as keys (netbox.dcops.microscaler.io, etc.) not the parent group (dcops.microscaler.io). Fixed discovery to: - Dynamically find all dcops.* subgroups by filtering /apis keys - Extract version from first subgroup entry - Pass subgroup to listCrds/getCrds for correct API path - Added subgroup field to CrdMeta interface
Also fix CRD discovery to query all dcops microgroups dynamically (netbox.dcops.microscaler.io, ippool.dcops.microscaler.io, boot.dcops.microscaler.io) instead of hardcoding dcops.microscaler.io.
…ources - Split single dcops-ui into dcops-docs (docs) and dcops-dashboard (dashboard) - Create separate Dockerfiles, nginx configs, and kustomize directories - Docs app on port 8801, Dashboard app on port 8802 - Dashboard includes RBAC ClusterRole/Binding for K8s API access - Remove old dcops-ui resource and role.yaml - kubectl-proxy local_resource for dashboard API access
| resources = rule.get("resources", []) | ||
| verbs = rule.get("verbs", []) | ||
|
|
||
| if "dcops.microscaler.io" in api_groups: |
Migrate from Tilt/docker_build deploying workloads to Flux-driven deployment pattern, matching hauliage's architecture: - Replace custom_build + k8s_yaml(kustomize) for controllers with local_resource that builds images and pushes dev-<timestamp> tags for Flux ImagePolicyWatch discovery - Remove k8s_yaml/k8s_resource for netbox-controller, dhcp-controller, pxe-server, kea-dhcp (Flux owns all workload deployment) - Keep NetBox deployed via Tilt (dev infra, like hauliage's supabase) - Keep kubectl proxy + Vite dev servers (local dev only) - Keep CRD generation, token management, tenant setup - Match hauliage pattern: local_resource -> docker build + push to shared LAN registry, Flux handles runtime reconciliation
…t Tilt build storms - Replace generate_crds.py with generate_crds_local.py that only generates CRD YAML locally without applying to the cluster - CRDs in the cluster are managed by Flux (production) or applied manually - This prevents generate-crds from blocking all build resources on kubectl operations, which was causing recursive build storms in Tilt - Build resources still depend on generate-crds for the local YAML artifact
…ng staging directory The root cause: image-pxe-server ran docker buildx build . which included the entire repo as context (including target/). Docker ops on target/ files triggered Tilt file watchers, which triggered another docker build, creating an infinite loop. Fix: Copy the binary to an unwatched .tilt-staging/pxe-server directory before running docker build, so Tilt doesn't see docker's filesystem access as changes.
Removed duplicate PXE_IMAGE_NAME argument — was 11 args for 10 %s placeholders.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.