1. Provider Configuration

To interact with Vault and generate secure tokens, the environment must specify the required providers.

terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
    }
    random = {
      source = "hashicorp/random"
    }
  }
}

2. Generating the Secret Data

In this implementation, a token is required to bootstrap a K3s cluster. The random provider is utilized to generate a secure, 32-character string.

resource "random_password" "k3s_cluster_token" {
  length  = 32
  special = false
}

3. Storing Sensitive Data in Vault (Set)

The generated token is injected into Vault using the vault_kv_secret_v2 resource. This operation assumes the target Vault mount path is configured for KV Version 2.

• Mount Point: home-lab-secrets

• Secret Path: k3s-cluster-token

• Payload Format: JSON encoded key-value pair.

  resource "vault_kv_secret_v2" "k3s_cluster_token" {
    mount     = "home-lab-secrets"
    name      = "k3s-cluster-token"
    data_json = jsonencode({
      token = random_password.k3s_cluster_token.result
    })
  }

4. Retrieving Sensitive Data from Vault (Get)

To utilize the stored secret elsewhere in the infrastructure code, a Terraform data source fetches the values from Vault. The depends_on meta-argument ensures the secret is fully provisioned before retrieval is attempted.

data "vault_kv_secret_v2" "k3s_cluster_token" {
  depends_on = [vault_kv_secret_v2.k3s_cluster_token]
  mount      = "home-lab-secrets"
  name       = "k3s-cluster-token"
}

5. Usage Example: Bootstrapping a Node

Once retrieved, the secret data can be accessed via data.vault_kv_secret_v2.<name>.data.<key>. In the provided example, the token is passed as an environment variable (K3S_TOKEN) over an SSH connection using a remote-exec provisioner to join a node to the existing K3s cluster.

provisioner "remote-exec" {
    inline = [
      "cloud-init status --wait",
      "curl -sfL https://get.k3s.io | K3S_URL='https://${local.master_node.network.ip}:6443' K3S_TOKEN='${data.vault_kv_secret_v2.k3s_cluster_token.data.token}' sh -"
    ]
  }

Lesson learnt from previous work, I will conclude them into steps and hope it can help others.Prerequisites

1. Ensure all Proxmox nodes are same version.
   It can done by run commands in each node below:

   ## Update package list.
   sudo apt update
   
   ## Ensure all proxmox node in same version.
   sudo apt dist-upgrade -y 
   
   ## Update package.
   sudo apt upgrade
   
   ## Clean up
   sudo apt clean

2. Ensure all node connect to network;
3. Ensure all node can resolve each other’s DNS record;

In there, it will use 2 nodes (nodeA, nodeB) in command as sample/

Steps:

1. Create cluster.
   In nodeA, use command below to create cluster.

   pvecm create demo-cluster

2. Grand cluster firewall port.
   In nodeA, open /etc/pve/firewall/cluster.fw , add rules inside section [RULES], sample as below:

   [RULES]
   
   IN SSH(ACCEPT) -i vmbr0 -log nolog # SSH between nodes
   IN NTP(ACCEPT) -i vmbr0 -log nolog # NTP sync time
   IN DNS(ACCEPT) -i vmbr0 -log nolog # DNS resolve
   IN ACCEPT -i vmbr0 -p tcp -dport 5403 -sport 5403 -log nolog # Proxmox VE Cluster Daemon
   IN ACCEPT -i vmbr0 -p tcp -dport 60050 -sport 60000 -log nolog # Proxmox VE backup and migration traffic
   IN ACCEPT -i vmbr0 -p udp -dport 5405 -sport 5404 -log nolog # Proxmox corosync
   IN ACCEPT -i vmbr0 -p tcp -dport 3128 -sport 3128 -log nolog # Proxmox VE HA (High Availability) manager
   IN ACCEPT -i vmbr0 -p tcp -dport 5999 -sport 5900 -log nolog # SPICE console connections.
   IN ACCEPT -i vmbr0 -p tcp -dport 8009 -sport 8007 -log nolog # Proxmox VE cluster traffic
   IN ACCEPT -i vmbr0 -p tcp -dport 8006 -sport 8006 -log nolog # Proxmox VM

3. Join node.
   In nodeB, use command below to join cluster.

   pvecm add <node1 IP address>

4. Verify result
   In nodeB, run command below, if it can found nodeB IP address, means node join successfully.

   pvecm status

   

It cause by dangling commit and dangling blob in repository.

Dangling commit:  Commit that isn’t directly linked to by any child commit, branch, tag or other reference

Dangling blob: A change that made it to the staging area/index, but never got committed.

These dragging commit and blob can found in command below:

git fsck --name-objects

Output:

Run command below to clean danging commit / blob and clean up repository

cd $REPO_PATH 
## Remove all danging commit and blob and GC collect. 
git reflog expire --expire=now --all 
git gc --prune=now 

## Clean up local old / conflict branch 
git remote prune origin

In this example, it will use Gradle as example to show how to get package version and call it in in other steps.

Steps

1. Update GitHub Action.
   Add steps below in GiutHub action.

   - name: Read package version
     id: get-version
     run: |
       version=$(grep -oP "version\s*=\s*'[^']+'" build.gradle | grep -oP "[^']*" | tr -d '\n' | tr -d ' ' | tr -d 'version=')
       echo "::set-output name=version::${version}"
   
   - name: Print package version
     run: |
       echo "Testing current version."
       echo "Package version is ${{ steps.get-version.outputs.version }}"

2. Check result
   Execute the pipeline to check result, expect version in build.gradle mentioned will shown.

Prerequisites

1. Ensure required windows feature are installed properly.
   In Windows feature, ensure Hyper-V, Virtual Machine Platform and Windows Subsystem for Linux are checked, if not, tick missing items and click OK to install it. Restart computer after installed.
   
2. Ensure required packages are installed.
   Check docker engine install or not; install it and restart computer if missing;

Steps

1. Install act
   Execute command in Terminal below to install act. Or you can follow act user guide to install in another way.

   choco install act-cli

2. Execute and verify
   Execute command in Terminal below, expected it can executed successfully if step do not required specific credentials.

   act --pull=false

   

Reference

1. act user guide, nektosact, https://nektosact.com/

It caused by GitLab not support cert verify with legacy certificate attribute. To resolve that problem it need to genearate certificate in server again. And steps will show as below.

1. Generate new certificate and install in GitLab server
   Execute command below in GitLab server:

   export HOSTNAME=home-gitlb-sr01
   export CERT_VALID_DAYS=36500
   export CERT_PATH=/etc/gitlab/ssl
   
   ## Install OpenSSL
   sudo apt install -y openssl
   
   ## Generate RSA key with 2048 length.
   sudo openssl genrsa -out ${HOSTNAME}.key 2048
   
   ## Generate x509 certificate which contain SANs with key generated. 
   sudo openssl req -new -x509 -addext "subjectAltName = DNS:localhost,DNS:${HOSTNAME}" -days ${CERT_VALID_DAYS} -key ${HOSTNAME}.key -out ${HOSTNAME}.crt
   
   ## Backup original certificate and replace with generated one.
   sudo tar -czvf ${CERT_PATH} ${CERT_PATH}.tar.gz
   sudo cp ${HOSTNAME}.crt ${CERT_PATH}/
   
   ## Restart GitLab server to apply new certificate.
   sudo gitlab-ctl reconfigure
   sudo gitlab-ctyl restart

2. Import certificate in Gitlab Runner and apply settings.

   export HOSTNAME=home-gitlb-sr01
   
   ## Get Cert from GitLab server and copy to runner cert directory.
   openssl s_client -showcerts -connect ${HOSTNAME}:443 -servername ${HOSTNAME} < /dev/null 2>/dev/null | openssl x509 -outform PEM > ${HOSTNAME}.crt
   sudo cp ${HOSTNAME}.crt /etc/gitlab-runner/certs/
   
   sudo gitlab-runner register --tls-ca-file=${HOSTNAME}.crt
   sudo gitlab-runner restart
   sudo gitlab-runner run

    

Problem

For some reasons, some of the services cannot use common method to communicate and it cannot be change due to their own constrains. However, application still need to get resource and execute command from it. It is require to convert the response and executed result to application compatible format.

It will be time consuming if each service implement it’s own logic to communicate such “out-standing” service. Also, it will be more complicate on change when such system and it will impact to all involved services.

Solution

Implement the middle layer between services which role as a middleman to:

1. Collect consumer request;
2. Transform request format / communication channel and send to target system;
3. Receive target system response;
4. Transform response format / communication channel and send back to consumer;

Anti-corruption layer is the middleman which do tasks above. When the target system change, it can eliminate the change scale by only need to update anti-corruption layer, consumer do not need to update in their side.

Use case

1. Communicate with legacy or specific protocol (e.g. SOAP, HL7, etc)
2. Communicate with legacy channel (e.g. COM / RPC call, etc)

Implementation

Consideration

Compare with façade and adapter

Façade is set of interface and adapter is a converter between different system.

Anti-corruption layer combine the characteristic of façade and adapter, it provides the interfaces / event handlers to collect request and send out response between target system; In addition, anti-corruption layer contains a certain logics to translate the content to fulfill each others needs;

Compare with Ambassador

Ambassador is focus on non-fictional requirement, and anti-corruption layer is a sort of converter, so certain business logic might contain in there.

Reference

1. Anti-corruption Layer, Microsoft Azure
   https://learn.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer
2. The Anti-Corruption Layer Pattern, Ahmed Shirin, Dev.to
   https://dev.to/asarnaout/the-anti-corruption-layer-pattern-pcd

XCP-ng is free and open source edition of XenServer, which focked from Citrix since 2018. As inherited from XenServer, it is type 1 hypervisor so it can utilize hardware resources.

XCP-ng combine with 4 major components: Computed, Network, Storage and API. The UI need to install separate Xen-Orchesta (XOA) VM, which role as a controller which similar VMWare vCenter. In XOA, it also integrate XOSAN for storage management. Also, to secure access, XCP-ng Center enables administrators to manage hypervisers via desktop applicaition.

Unfortunately, It is not fully support IaC (Infrastructure as Code). There has unofficial provider in Terraform registry. As it is not XCP-ng clould still in process, hopefully it will have official IaC solution once go live.

Reference

1. XCP-ng.org
   https://xcp-ng.org/
2. XCP, GitHub
   https://github.com/xcp-ng/xcp
3. XOA, Xen Orchestra
   https://xen-orchestra.com/#!/xo-home
4. Xen Orchestra Provider, Terraform Registry
   https://registry.terraform.io/providers/terra-farm/xenorchestra/latest/docs
5. XCP-ng Center, GitHub
   https://github.com/xcp-ng/xenadmin

This problem caused by escape character found in String. As LogstashEncoder will serialize object to JSON, it will replace escape ASCII character to encoded value (e.g. new line => \n).

To resolve it, it can use CharacterEscapesJsonFactoryDecorator in log settings file (e.g. log4j.xml) to override specific ASCII code and replaced by defined value.

This example will replace string to empty string.

 <appender name="console-json" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" >
            <jsonFactoryDecorator class="net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator">
                <escape>
                    <!-- ASCII 10 => new line  -->
                    <targetCharacterCode>10</targetCharacterCode>
                    <escapeSequence></escapeSequence>
                </escape>
            </jsonFactoryDecorator>
        </encoder>
</appender>

Test result:

Steps

1. Enable kernal modules for containerd.
   In shell, execute command below.

   sudo cat << EOF | sudo tee /etc/modules-load.d/containerd.conf 
   overlay 
   br_netfilter 
   EOF
   sudo modprobe overlay
   sudo modprobe br_netfilter

2. Initialize system settings for K8S networking.
   In shell, execute command below.

   sudo cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
   net.bridge.bridge-nf-call-iptables = 1
   net.ipv4.ip_forward = 1
   net.bridge.bridge-nf-call-ip6tables = 1
   EOF
   sudo sysctl --system

3. Prepare applications to use in following steps.
   In shell, execute command below.

   sudo apt update
   sudo apt install -y apt-transport-https curl

4. Disable swap in OS.
   In shell, execute command below.

   sudo swapoff -a
   sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

5. Install containerd
   In shell, execute command below.

   sudo apt install -y ca-certificates curl gnupg lsb-release
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
   echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
   sudo apt update
   sudo apt install -y containerd.io

6. Configure containerd
   In shell, execute command below.

   sudo mkdir -p /etc/containerd 
   sudo containerd config default | sudo tee /etc/containerd/config.toml 
   sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml && grep 'SystemdCgroup' -B 11 /etc/containerd/config.toml
   sudo systemctl enable --now containerd
   sudo systemctl restart containerd

7. Install K8S
   In shell, execute command below.

   curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
   cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
   deb https://apt.kubernetes.io/ kubernetes-xenial main
   EOF
   sudo apt update
   sudo apt install -y kubelet=$KUBERNETES_VERSION kubectl=$KUBERNETES_VERSION kubeadm=$KUBERNETES_VERSION
   sudo apt-mark hold kubelet kubeadm kubectl
   sudo systemctl enable --now kubelet

8. [For Master node only] Initialize K8S cluster.
   In shell, execute command below.

   sudo kubeadm init --pod-network-cidr $POD_NETWORK_CIDR --kubernetes-version $KUBERNETES_VERSION

9. [For Master node only] Setup master node local settings.
   In shell, execute command below.

   mkdir -p $HOME/.kube
   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   sudo chown $(id -u):$(id -g) $HOME/.kube/config

10. [For Master node only] Setup Calico.
    In shell, execute command below.

    kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

11. [For Master node only] Generate command for worker nodes joining cluster.
    In shell, execute command below.

    kubeadm token create --print-join-command

    It should be generate command like this, copy it to clipboard.
    

12. [For Worker node only] Join K8S cluster.
    In shell, execute command which copied from previous steps.
    
13. Verify result.
    In master node shell, execute command below

    kubectl get po -A -o wide

    Expected it will show all namespace pods and its deployed pod.
    

Related shell script has been push in GitHub repository.