Skip to content

feat(BOXP-129): Oracle Cloud Free Tier control plane IaC for lolice cluster - #11581

Open
boxp wants to merge 16 commits into
mainfrom
feature/BOXP-129-oracle-cloud-control-plane
Open

feat(BOXP-129): Oracle Cloud Free Tier control plane IaC for lolice cluster#11581
boxp wants to merge 16 commits into
mainfrom
feature/BOXP-129-oracle-cloud-control-plane

Conversation

@boxp

@boxp boxp commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Summary

lolice cluster の物理 control plane (shanghai-1/2/3) が2台同時障害した場合でも稼働を継続できるよう、Oracle Cloud Free Tier (Ampere A1, 東京, ARM64, ¥0/月) の VM を2台追加するための IaC を実装する。

  • etcd 構成: 物理3 + クラウド2 = 計5台 (クォーラム=3)
  • 物理CP 2台障害時: 残り1物理+2クラウド=3 ≥ クォーラム3 → クラスター継続 ✓
  • コスト: ¥0/月 (Oracle Always Free Tier)

変更内容

新規追加

  • terraform/oci/lolice-control-plane/: OCI Terraform モジュール

    • VCN / Subnet / Internet Gateway / Security List (Tailscale UDP 41641 + SSH 22)
    • VM.Standard.A1.Flex × 2台 (各 2 OCPU / 12 GB RAM, Ubuntu 22.04 ARM64, 東京)
    • cloud-init: Tailscale 自動インストール・認証 (auth key は SSM から取得)
    • S3 バックエンド (tfaction-state), oracle/oci ~> 6.0 プロバイダー
  • ansible/playbooks/cloud-control-plane-join.yml: kubeadm join プレイブック

    • CRI-O + kubelet + kubeadm インストール
    • kubeadm join --control-plane --apiserver-advertise-address=<tailscale-ip> 実行
    • kube-vip は適用しない (L2 VIP は LAN 外無効)
  • docs/project_docs/BOXP-129/plan.md: 設計ドキュメント

既存ファイル更新

  • terraform/tailscale/lolice/acl.tf: tag:cloud-control-plane タグ所有者追加、etcd (2379/2380) / kubelet (10250) / apiserver (6443) の ACL ルール追加
  • terraform/tailscale/lolice/auth_key.tf: クラウド CP 用 Tailscale auth key + SSM パラメータ (/lolice/tailscale/cloud-control-plane-auth-key) 追加
  • ansible/inventories/production/hosts.yml: cloud_control_plane グループ (oracle-cp-1/2) 追加、kube_vip_enabled: false 設定

適用手順

Step 1: Tailscale ACL + auth key 適用

# terraform/tailscale/lolice で apply (auth key が SSM に保存される)
cd terraform/tailscale/lolice
terraform apply

Step 2: OCI 認証情報を SSM に保存 (手動)

aws ssm put-parameter --name "/lolice/oci/tenancy-ocid" --value "<TENANCY_OCID>" --type SecureString
aws ssm put-parameter --name "/lolice/oci/user-ocid" --value "<USER_OCID>" --type SecureString
aws ssm put-parameter --name "/lolice/oci/fingerprint" --value "<FINGERPRINT>" --type SecureString
aws ssm put-parameter --name "/lolice/oci/private-key" --value "$(cat ~/.oci/oci_api_key.pem)" --type SecureString

Step 3: Oracle Cloud VM プロビジョニング

cd terraform/oci/lolice-control-plane
terraform apply \
  -var="tenancy_ocid=<TENANCY_OCID>" \
  -var="user_ocid=<USER_OCID>" \
  -var="fingerprint=<FINGERPRINT>" \
  -var="private_key=$(cat ~/.oci/oci_api_key.pem)"

Step 4: Ansible inventory の Tailscale IP 更新

terraform output で Tailscale IP を確認し、ansible/inventories/production/hosts.ymlTAILSCALE_IP_PLACEHOLDER_* を更新。

Step 5: kubeadm join

# 既存 CP (shanghai-1) で証明書アップロード
kubeadm init phase upload-certs --upload-certs
kubeadm token create --print-join-command

# Ansible playbook 実行
ansible-playbook playbooks/cloud-control-plane-join.yml \
  -i inventories/production \
  -e kubeadm_token=<TOKEN> \
  -e kubeadm_ca_cert_hash=sha256:<HASH> \
  -e kubeadm_certificate_key=<CERT_KEY> \
  --limit cloud_control_plane

Test plan

  • terraform plan が terraform/tailscale/lolice と terraform/oci/lolice-control-plane で正常に実行できること
  • Oracle Cloud VM が東京リージョンで起動し、Tailscale に自動登録されること
  • ansible-playbook cloud-control-plane-join.yml で kubeadm join が成功すること
  • etcdctl endpoint health で5メンバーが全て healthy であること
  • shanghai-1/2 停止状態で kubectl get nodes が応答すること (クォーラムテスト)

Closes BOXP-129

🤖 Generated with Claude Code

…luster

Add Terraform and Ansible resources to provision two Oracle Cloud Ampere A1 (ARM64,
Tokyo) nodes as additional Kubernetes control plane members, bringing the etcd cluster
from 3 to 5 members (quorum=3) to survive simultaneous failure of 2 physical CPs.

- terraform/oci/lolice-control-plane/: new OCI Terraform module
  - VCN, subnet, internet gateway, security list (Tailscale UDP 41641 + SSH 22)
  - VM.Standard.A1.Flex x2 (2 OCPU / 12 GB RAM each, Ubuntu 22.04 ARM64)
  - cloud-init: Tailscale auto-registration via auth key from SSM
  - S3 backend (tfaction-state), oracle/oci ~> 6.0 provider
- terraform/tailscale/lolice/acl.tf: add tag:cloud-control-plane owner + ACL rules
  for etcd (2379/2380), kubelet (10250), and apiserver (6443) between on-prem and cloud
- terraform/tailscale/lolice/auth_key.tf: add reusable/preauthorized auth key for
  cloud CP nodes stored in SSM /lolice/tailscale/cloud-control-plane-auth-key
- ansible/inventories/production/hosts.yml: add cloud_control_plane group (oracle-cp-1/2)
  with kube_vip_enabled=false (L2 VIP is LAN-only)
- ansible/playbooks/cloud-control-plane-join.yml: kubeadm join playbook for cloud CPs
  using Tailscale IP as apiserver-advertise-address

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

- Use canonical module name community.general.modprobe (fqcn[canonical])
- Add changed_when to kubeadm join command (no-changed-when)
- Convert kubeadm join result tasks to handlers (no-handler)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

Tailscale tailnet_key description is limited to 50 characters.
Previous value was 66 chars, causing terraform validate to fail.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

- tfaction-root.yaml: add oracle/oci to available_providers and
  terraform/oci/** target group with OCI credential secrets
- acl.tf: restrict reverse rule src from "*" to explicit principals
  (autogroup:members + tag:subnet-router + tag:k8s-operator); add
  192.168.10.0/24 to autoApprovers.routes so cloud CPs can reach the
  LAN VIP 192.168.10.99 via the on-prem subnet router
- outputs.tf: expose Tailscale IP lookup commands for ansible inventory
- cloud-control-plane-join.yml: document subnet router prerequisite
  required for cloud CPs to reach cluster_vip 192.168.10.99

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

… plan

- Add registry.terraform.io/oracle/oci to wc-plan.yaml TFPROVIDERCHECK_CONFIG_BODY
  (needed after OCI credentials and main branch wc-plan.yaml are updated)
- Set skip_terraform: true in terraform/oci/lolice-control-plane/tfaction.yaml
  to bypass terraform init/plan until OCI_TENANCY_OCID, OCI_USER_OCID,
  OCI_FINGERPRINT, OCI_PRIVATE_KEY secrets are configured in GitHub

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

…ion-root.yaml

terraform/oci/** target group needs skip_terraform: true at the root config level
so that list-targets outputs skip_terraform=true in the CI matrix.
Without OCI credentials configured in GitHub secrets, terraform plan would fail.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp boxp added the skip:terraform/oci/lolice-control-plane Skip OCI terraform plan until OCI credentials are configured label Jul 23, 2026
@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

boxp-tfaction Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Plan Result (terraform/tailscale/lolice)

CI link

Plan: 2 to add, 1 to change, 0 to destroy.
  • Create
    • aws_ssm_parameter.cloud_control_plane_auth_key
    • tailscale_tailnet_key.cloud_control_plane
  • Update
    • tailscale_acl.this
Change Result (Click me)
  # aws_ssm_parameter.cloud_control_plane_auth_key will be created
  + resource "aws_ssm_parameter" "cloud_control_plane_auth_key" {
      + arn            = (known after apply)
      + data_type      = (known after apply)
      + description    = "Tailscale auth key for lolice Oracle Cloud control plane nodes"
      + has_value_wo   = (known after apply)
      + id             = (known after apply)
      + insecure_value = (known after apply)
      + key_id         = (known after apply)
      + name           = "/lolice/tailscale/cloud-control-plane-auth-key"
      + region         = "ap-northeast-1"
      + tags_all       = (known after apply)
      + tier           = (known after apply)
      + type           = "SecureString"
      + value          = (sensitive value)
      + value_wo       = (write-only attribute)
      + version        = (known after apply)
    }

  # tailscale_acl.this will be updated in-place
  ~ resource "tailscale_acl" "this" {
      ~ acl = jsonencode(
          ~ {
              ~ acls          = [
                    {
                        action = "accept"
                        dst    = [
                            "tag:k8s-operator:80",
                            "tag:k8s-operator:443",
                        ]
                        src    = [
                            "tag:ci",
                        ]
                    },
                  + {
                      + action = "accept"
                      + dst    = [
                          + "*:2379",
                          + "*:2380",
                          + "*:6443",
                          + "*:10250",
                        ]
                      + src    = [
                          + "tag:cloud-control-plane",
                        ]
                    },
                  + {
                      + action = "accept"
                      + dst    = [
                          + "tag:cloud-control-plane:2379",
                          + "tag:cloud-control-plane:2380",
                          + "tag:cloud-control-plane:6443",
                          + "tag:cloud-control-plane:10250",
                        ]
                      + src    = [
                          + "autogroup:members",
                          + "tag:subnet-router",
                          + "tag:k8s-operator",
                        ]
                    },
                ]
              ~ autoApprovers = {
                  + routes = {
                      + "192.168.10.0/24" = [
                          + "tag:subnet-router",
                        ]
                    }
                }
              ~ tagOwners     = {
                  + "tag:cloud-control-plane" = [
                      + "autogroup:admin",
                    ]
                    # (3 unchanged attributes hidden)
                }
            }
        )
        id  = "dca925a2-b72b-a210-25af-3ccbdb1f4046"
    }

  # tailscale_tailnet_key.cloud_control_plane will be created
  + resource "tailscale_tailnet_key" "cloud_control_plane" {
      + created_at    = (known after apply)
      + description   = "lolice OCI cloud control plane nodes"
      + ephemeral     = false
      + expires_at    = (known after apply)
      + expiry        = (known after apply)
      + id            = (known after apply)
      + invalid       = (known after apply)
      + key           = (sensitive value)
      + preauthorized = true
      + reusable      = true
      + tags          = [
          + "tag:cloud-control-plane",
        ]
      + user_id       = (known after apply)
    }

Plan: 2 to add, 1 to change, 0 to destroy.

@boxp-tfaction

boxp-tfaction Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

❌ Plan Failed (terraform/oci/lolice-control-plane)

CI link

Error: No value for required variable

  on variables.tf line 1:
   1: variable "tenancy_ocid" {

The root module input variable "tenancy_ocid" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.

Error: No value for required variable

  on variables.tf line 7:
   7: variable "user_ocid" {

The root module input variable "user_ocid" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.

Error: No value for required variable

  on variables.tf line 13:
  13: variable "fingerprint" {

The root module input variable "fingerprint" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.

Error: No value for required variable

  on variables.tf line 19:
  19: variable "private_key" {

The root module input variable "private_key" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.

@github-actions

Copy link
Copy Markdown
Contributor

Ansible Plan Results

Mode: --check --diff (dry run)

⚠️ Changes detected


golyat-4: worker-image

Host OK Changed Skipped Failed Unreachable
golyat-4 76 1 27 0 0

1 changed

Changed Tasks (1)
# Task Module
1 user_management : Update package cache unknown

shanghai-1: control-plane

Host OK Changed Skipped Failed Unreachable
shanghai-1 79 1 20 0 0

1 changed

Changed Tasks (1)
# Task Module
1 user_management : Update package cache unknown

shanghai-1: node-shanghai-1

Host OK Changed Skipped Failed Unreachable
shanghai-1 9 0 0 0 0

No changes

shanghai-2: control-plane

Host OK Changed Skipped Failed Unreachable
shanghai-2 79 1 20 0 0

1 changed

Changed Tasks (1)
# Task Module
1 user_management : Update package cache unknown

shanghai-2: node-shanghai-2

Host OK Changed Skipped Failed Unreachable
shanghai-2 9 0 0 0 0

No changes

shanghai-3: control-plane

Host OK Changed Skipped Failed Unreachable
shanghai-3 79 1 20 0 0

1 changed

Changed Tasks (1)
# Task Module
1 user_management : Update package cache unknown

shanghai-3: node-shanghai-3

Host OK Changed Skipped Failed Unreachable
shanghai-3 9 0 0 0 0

No changes


Plan executed on all nodes in parallel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant