In a previous post, I mentioned that I struggled to get OpenEBS working in Talos and instead went with democratic-csi. In recent weeks, I decided to revisit this and figure out how to get OpenEBS replicated storage working in order to evaluate replicated storage in my cluster. I now have multiple disks that I can dedicate to my Kubernetes cluster and wanted to avoid the issue with the single point of failure using a TrueNAS VM for democratic-csi.

If you are following along, I will assume you are familiar with deploying Talos Linux itself and have talosctl installed with an existing cluster running. If you need more details on how to do that, check out https://blog.dalydays.com/post/kubernetes-homelab-series-part-1-talos-linux-proxmox/.

Dedicated Storage Node

It’s not absolutely necessary to use a dedicated storage node. I’m doing this because I want to pass a disk directly to a VM for storage on each physical host and want to keep storage somewhat isolated from other worker nodes, and I can spare the few extra resources to dedicate to this purpose. If you want to use existing worker nodes, just follow this process for your existing nodes instead of creating new ones.

Create New Talos Node(s)

• Create a VM in Proxmox with 4GB RAM and 4 vCPU cores (2GB RAM is not enough due to the fact that you will be enabling hugepages which takes up 2GB and you would see oom-kills otherwise. You also need a dedicated CPU core just for the io-engine, along with all the other stuff that runs. I tried with 2vCPU and it wouldn’t schedule the io-engine pod due to insufficient resources). I named my first one talos-storage-1
  • Attach a Talos ISO to the CD ROM and boot from it
  • Get the IP address from the node
• Install Talos using the worker.yaml template used for other worker nodes (you may want to get a current or updated version of Talos from the image factory):
  • talosctl apply-config --insecure -n 10.0.50.135 --file _out/worker.yaml
• Apply a patch to set a static IP and node label, e.g.

# ./patches/storage1.patch
machine:
  network:
    hostname: talos-storage-1
    interfaces:
      - deviceSelector:
          busPath: "0*"
        addresses:
          - 10.0.50.31/24
        routes:
          - network: 0.0.0.0/0
            gateway: 10.0.50.1
    nameservers:
      - 192.168.1.22

• talosctl patch mc -n 10.0.50.135 --patch @patches/storage1.patch
  • I’m having trouble here with the node name changing, and I have to manually delete the random name from the cluster:
  • e.g. kubectl delete node talos-lry-si8
  • Also you might need to delete the label openebs.io/nodename= if you already have openebs running and are adding/changing nodes
    • kubectl edit node talos-storage-1 and change the value to the current node name
• Apply a patch to set some machine config stuff for OpenEBS which includes hugepages, a nodeLabel for where mayastor engine should run, and the /var/local bind mount:

# ./patches/openebs.patch
machine:
  sysctls:
    vm.nr_hugepages: "1024"
  nodeLabels:
    openebs.io/engine: mayastor
  extraMounts:
    - destination: /var/local
      type: bind
      source: /var/local
      options:
        - rbind
        - rshared
        - rw

• talosctl patch mc -n 10.0.50.31 --patch @patches/openebs.patch
• If you have an additional disk to use with OpenEBS, you’ll need to pass it directly to the Talos node VM. I’m using Proxmox
  • SSH into the Proxmox host and find the disk ID to be passed. I just run ls -lh /dev/disk/by-id/ and get the root disk (not containing any "_1" or "_part*", for example /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S59CNM0W635077P`)
  • Pass the disk directly to the Talos VM, where 511 is the VM ID and assuming you only have 1 disk already on scsi0: qm set 511 -scsi1 /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S59CNM0W635077P
  • Checking the hardware tab on VM 511 in Proxmox, you should see this new disk. Double click it and make sure to check “Advanced”, “Discard”, and “SSD emulation”
    • If you see orange on these settings, you will need to shut down the VM, then power it back on for the changes to apply. Rebooting won’t do it.
  • Now that the disk has been added, look for it with talosctl: talosctl get disks -n 10.0.50.31
    • In my case I see a disk named sdb which is 2.0TB with model “QEMU HARDDISK”
  • Mount the disk to be passed to containers with appropriate privileges. This is required for openebs-io-engine to access the extra disk.

  # ./patches/mount-sdb.patch
  machine:
    disks:
      - device: /dev/sdb
  

  • Apply: talosctl patch mc -n 10.0.50.31 --patch @patches/mount-sdb.patch - At this point the Talos node will reboot and should come back up healthy in a minute.
  • View the console or check the dashboard with talosctl dashbord -n 10.0.50.31
  • If you see an error about being unable to mount the disk or the partition being the wrong type, etc. you will need to wipe the disk and create a fresh GPT partition. As of Talos 1.9.0 this can be done with talosctl wipe disk sdb -n 10.0.50.31, otherwise you would need to do this outside of Talos.
    • talosctl wipe disk sdb -n 10.0.50.31 - where sdb is the device, you confirmed this right? Confirm using talosctl get disks -n 10.0.50.31
    • Otherwise from Proxmox you could do this: Shut down the VM and do this in proxmox with wipefs /dev/yourdev and then use fdisk /dev/yourdev > g > w (g writes a new GPT table, w writes to disk). Now power Talos back on and it should do its thing.
    • Yet another option would be to boot into a different Linux ISO on the VM and use a tool like Gparted. Whatever you like best.

Let’s Verify Our Disk Mount

When Talos successfully mounts the extra disk, we should see it listed with lsblk without any partitions. We want to pass the raw disk to OpenEBS. In order to check, run a debug pod on your storage node and check the bind mounts.

• kubectl debug node/talos-storage-1 -it --image=alpine -- /bin/sh
• apk add lsblk
• lsblk
• Check for the mount, showing the full capacity of your disk.

Now repeat this whole process for any other Talos nodes you need. I have 3, so I’m doing talos-storage-1, talos-storage-2 and talos-storage-3.

Worker Nodes Also Need /var/local Mounted

Certain OpenEBS components run on any node, and this requires all worker nodes to have /var/local mounted.

# ./patches/mount-var-local.patch
machine:
  kubelet:
    extraMounts:
      - destination: /var/local
        type: bind
        source: /var/local
        options:
          - rbind
          - rshared
          - rw

In my case, I applied this to my 3 worker nodes. I don’t think a reboot is required, but you could if you wanted to:

• talosctl patch mc -n 10.0.50.21 --patch @patches/mount-var-local.patch
• talosctl patch mc -n 10.0.50.22 --patch @patches/mount-var-local.patch
• talosctl patch mc -n 10.0.50.23 --patch @patches/mount-var-local.patch

In order to check the other bind mount for /var/local, we have to wait until after deploying OpenEBS because the mount isn’t utilized until a pod is deployed with a HostPath volume at or below this path. Specifically, the openebs-io-engine-* Daemonset maps to this path.

Installing OpenEBS

This was a pain to figure out. Documentation from OpenEBS is lacking, and so is documentation from Talos on the same topic. Here’s what I found to work. You need a privileged namespace, bind mounts on all worker nodes, then DiskPools before you can start testing PVCs.

Privileged Namespace

OpenEBS requires privileges, and the easiest way to handle that is by making the namespace privileged (rather than messing with machine configs).

• Add a new privileged namespace. The Helm chart wants you to use openebs so do this:

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: openebs
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/audit: privileged

• kubectl apply -f namespace.yaml

Helm Installation

• helm repo add openebs https://openebs.github.io/openebs
• helm repo update
• Grab the values from the Helm chart (helm show values openebs/openebs > values.yaml), or use this. I have already modified the config to disable initContainers which is a known issue with Talos, and disabled local provisioners that I’m not interested in.

# values.yaml
openebs-crds:
  csi:
    volumeSnapshots:
      enabled: true
      keep: true

# Refer to https://github.com/openebs/dynamic-localpv-provisioner/blob/v4.1.2/deploy/helm/charts/values.yaml for complete set of values.
localpv-provisioner:
  rbac:
    create: true

# Refer to https://github.com/openebs/zfs-localpv/blob/v2.6.2/deploy/helm/charts/values.yaml for complete set of values.
zfs-localpv:
  crds:
    zfsLocalPv:
      enabled: true
    csi:
      volumeSnapshots:
        enabled: false

# Refer to https://github.com/openebs/lvm-localpv/blob/lvm-localpv-1.6.2/deploy/helm/charts/values.yaml for complete set of values.
lvm-localpv:
  crds:
    lvmLocalPv:
      enabled: true
    csi:
      volumeSnapshots:
        enabled: false

# Refer to https://github.com/openebs/mayastor-extensions/blob/v2.7.2/chart/values.yaml for complete set of values.
mayastor:
  csi:
    node:
      initContainers:
        enabled: false
  etcd:
    # -- Kubernetes Cluster Domain
    clusterDomain: cluster.local
  localpv-provisioner:
    enabled: false
  crds:
    enabled: false

# -- Configuration options for pre-upgrade helm hook job.
preUpgradeHook:
  image:
    # -- The container image registry URL for the hook job
    registry: docker.io
    # -- The container repository for the hook job
    repo: bitnami/kubectl
    # -- The container image tag for the hook job
    tag: "1.25.15"
    # -- The imagePullPolicy for the container
    pullPolicy: IfNotPresent

engines:
  local:
    lvm:
      enabled: false
    zfs:
      enabled: false
  replicated:
    mayastor:
      enabled: true

• helm install openebs -n openebs openebs/openebs -f values.yaml
• Verify: kubectl get po -n openebs and it should look something like this:

NAME                                          READY   STATUS    RESTARTS      AGE
openebs-agent-core-74d4ddc7c5-hjnxl           2/2     Running   0             9m23s
openebs-agent-ha-node-f9bsb                   1/1     Running   0             9m23s
openebs-agent-ha-node-gjdbt                   1/1     Running   0             9m23s
openebs-agent-ha-node-mwjq9                   1/1     Running   0             9m23s
openebs-agent-ha-node-rfjrw                   1/1     Running   0             93s
openebs-api-rest-757d87d4bd-zd2ms             1/1     Running   0             9m23s
openebs-csi-controller-58c7dfcd5b-6jtcq       6/6     Running   0             9m23s
openebs-csi-node-fmmwg                        2/2     Running   0             9m23s
openebs-csi-node-j95f5                        2/2     Running   2 (60s ago)   93s
openebs-csi-node-jxkvq                        2/2     Running   0             9m23s
openebs-csi-node-xtsnt                        2/2     Running   0             9m23s
openebs-etcd-0                                1/1     Running   0             9m23s
openebs-etcd-1                                1/1     Running   0             9m23s
openebs-etcd-2                                1/1     Running   0             9m23s
openebs-io-engine-lb8zr                       2/2     Running   0             9m23s
openebs-localpv-provisioner-657c44878-wjmwr   1/1     Running   0             9m23s
openebs-loki-0                                1/1     Running   0             9m23s
openebs-nats-0                                3/3     Running   0             9m23s
openebs-nats-1                                3/3     Running   0             9m23s
openebs-nats-2                                3/3     Running   0             9m23s
openebs-obs-callhome-8665bb8f6f-4ntrd         2/2     Running   0             9m23s
openebs-operator-diskpool-6d44884f8f-52rrx    1/1     Running   0             9m23s
openebs-promtail-2g6kz                        1/1     Running   0             9m23s
openebs-promtail-d72cx                        1/1     Running   0             93s
openebs-promtail-hsxlc                        1/1     Running   0             9m23s
openebs-promtail-npw7f                        1/1     Running   0             9m23s

• If pods are stuck initializing after a few minutes, start checking logs with openebs-etcd-0 since a lot of things depend on that being up before it will initialize.
  • Related to /var/local bind mounts not being added to worker nodes, openebs-etcd-* will have Warning events about “FailedMount”, stating that a PVC doesn’t exist. If you look closely, the path starts with /var/local/... which means you’re missing the /var/local bind mount on one or more Talos worker/storage nodes.
  • The fix in this situation would be to fix the bind mount on the worker nodes, and you could also try a reboot (you can identify the node based on the pod having issues). There’s no need to change the OpenEBS deployment.
  • After fixing it, you might see stale pods with errors. You can terminate pods in an error state, and if they are needed the Deployment or Daemonset responsible for them will recreate them as needed.

Add DiskPool(s)

I was excited at this point to test with a PVC, but then was confused about why it wouldn’t provision anything. The Talos docs feel a bit sparse and seem to imply that now is the time to test with a PVC. But if you pay close attention they only mention testing the local provisioner, adding to my confusion. It turns out you need to add DiskPools, which makes sense in hindsight. If you’ve ever used Longhorn there is a similar config needed after the initial install, so that you know what disk capacity you have to work with.

• Earlier, we mounted that 2TB disk in the talos-storage-1 node. Now we’ll use that for our first DiskPool
• Get the disk ID by exec-ing into the openebs-io-engine pod
  • Identify one of the io-engine pods: kubectl get po -l app=io-engine -n openebs
  • Exec into the pod: kubectl exec -it openebs-io-engine-jpnrh -c io-engine -n openebs -- bin/sh
  • ls -lh /dev/disk/by-id/ - grab the one pointing to /dev/sdb in our case, which for me is scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
• FYI I went with uring instead of aio since it’s the new kid on the block

# diskpool-1.yaml
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: pool-1
  namespace: openebs
spec:
  node: talos-storage-1
  disks: ["uring:///dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1"]

• kubectl apply -f diskpool-1.yaml
• Verify: kubectl get dsp -n openebs - quickly it should change to Created state and Online POOL_STATUS

Repeat this process for any/all storage nodes you have. Since I’m virtualizing Talos, the disk path is exactly the same on all 3 nodes so I can reuse the config, just updating the pool name and the node name.

Troubleshooting

Sorry I can’t be a ton of help here since I have only done really limited troubleshooting. If you run into issues with diskpools stuck in Creating status, you will need to describe the dsp or check logs.

What I can say is that I’m running a homelab, which means I have 3 different disks, from 3 different manufacturers. Two of them worked in this configuration, but one did not. The disk is fine, it’s fairly new, I’ve tried wiping it multiple times, multiple ways, but it just would not work with the diskpool. I tried doing it as a bind mount and using /mnt/local/nvme2tb (this one requires adding the volume to the io-engine Daemonset), mounting /dev/sdb, mounting /dev/sdb1, everything I could think of, but it would not create. I rebuilt the Talos node but got the same results. I switched to a different disk and changed nothing else, and it works totally fine. For prosperity, these are the disks I have and which ones worked in this configuration.

• Samsung 970 EVO Plus 2TB - no problems
• Samsung 990 EVO 2TB - no problems
• WD BLACK SN770 2TB - no problems
• Crucial P3 Plus 2TB (CT2000P3PSSD8) - COULD NOT GET THIS WORKING :(

If you are reading this and you know or think you might know why this didn’t work, please reach out! I’m interested in understanding why this wouldn’t work and how I could troubleshoot better.

Testing A Replicated PVC

If you are here, you have at least one working diskpool and are ready to test that PVC provisioning works and can be attached to a running pod. Let’s test that.

• Verify diskpools: kubectl get dsp -n openebs - for my 3 storage nodes with 2TB volumes, I’m seeing this

NAME     NODE              STATE     POOL_STATUS   CAPACITY        USED   AVAILABLE
pool-1   talos-storage-1   Created   Online        1998443249664   0      1998443249664
pool-2   talos-storage-2   Created   Online        1998443249664   0      1998443249664
pool-3   talos-storage-3   Created   Online        1998443249664   0      1998443249664

• Check what storage classes are available: kubectl get sc

NAME                     PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
mayastor-etcd-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
mayastor-loki-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
openebs-hostpath         openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
openebs-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   6h15m

• For now, we’re interested in testing that openebs-single-replica SC that uses Mayastor, so write this file. Note that this uses the default namespace:

# test-pvc-openebs-single-replica.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openebs-testpvc
spec:
  storageClassName: openebs-single-replica
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

• Apply it: kubectl apply -f test-pvc-openebs-single-replica.yaml
• Check PV and PVC:
  • kubectl get pv
  • kubectl get pvc - you should see a PVC named openebs-testpvc with status Bound and storage class openebs-single-replica
• Deploy a test pod to attach the PVC to - this pod is also in the default namespace:

# pod-using-testpvc.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: testlogger
spec:
    containers:
    - name: testlogger
      image: alpine:3.20
      command: ["/bin/ash"]
      args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
      volumeMounts:
      - name: testvol
        mountPath: /mnt
    volumes:
    - name: testvol
      persistentVolumeClaim:
        claimName: openebs-testpvc

• kubectl apply -f pod-using-testpvc.yaml
• Verify it’s running: kubectl get po testlogger
• Exec into the test pod: kubectl exec -it testlogger -- /bin/sh
• Look at your mounts with df -h /mnt. Since OpenEBS Mayastor uses NVMe-oF you should see that mount path /mnt attached to what looks like a NVMe block device.

Filesystem                Size      Used Available Use% Mounted on
/dev/nvme0n1              9.7G     28.0K      9.2G   0% /mnt

• Hmm. The size doesn’t quite add up. 9.7G is close to 10Gi but then 28.0K used does not equal 9.2G available. Hmm…
• Cleanup:
  • kubectl delete -f pod-using-testpvc.yaml
  • kubectl delete -f test-pvc-openebs-single-replica.yaml

What Is A “Single” Replica Anyway?

A replica is an exact reproduction of something…

A replica by definition cannot exist without copying something that already exists, which inherently means there must be at least 2. In the context of OpenEBS, a single replica just means you only have ONE copy of the data. It gets randomly assigned to one of the diskpools available. If that disk fails, that data is gone. But we are using OpenEBS for the purpose of replication, so how do we get more replicas??? Follow me.

• Create a new storage class with 2 replicas (feel free to do 3 or any value at this point):

# openebs-2-replicas-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-2-replicas
parameters:
  protocol: nvmf
  repl: "2"
provisioner: io.openebs.csi-mayastor

• kubectl apply -f openebs-2-replicas-sc.yaml
• Verify: kubectl get sc
• Test - deploy another test-pvc using the new SC:

# test-pvc-openebs-2-replicas.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openebs-testpvc
spec:
  storageClassName: openebs-2-replicas
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

• kubectl apply -f openebs-2-replicas.yaml
• Test with a pod, using the same pod from earlier - kubectl apply -f pod-using-testpvc.yaml
• Exec into the test pod: kubectl exec -it testlogger -- /bin/sh
• Check your mounted disk: df -h /mnt

Conclusion

I’m going to stop here. I didn’t go into any details about how to check which diskpools hold the replica(s) for the PV, but I assume there is a way to do that. I also did not look at how to recover in case there is a diskpool or storage node failure. Assuming you had 3 replicas, and one went down, there would be no data loss.

I didn’t cover performance, monitoring, recovery, or anything else that you probably care about long term. That could be a future post, but my next stop is actually evaluating Longhorn with the v2 engine. As of today, they have released 1.8.0-rc5, which enables support for their v2 engine with Talos (this just means they support NVMe-oF). If Longhorn can now work with NVMe-oF and Talos, to me that is a much more mature and feature rich product, with more community support than OpenEBS. I believe it also supports snapshots and other features that OpenEBS does not currently.

My next post will be all about blowing this setup away and doing it all over with Longhorn. Hopefully by then the stable 1.8.0 will have been released.

There are several questions to answer when deciding how to handle storage for Kubernetes.

• How many disks do you have?
• Do you want/need replicated storage?
• What are your storage capacity requirements?
• What are your performance requirements?
• Do you need dynamically provisioned storage or will you be doing it manually?
• NFS or iSCSI or NVMe-oF?
• How will you back up your data?
• Do you need snapshots?
• Does your storage need to be highly available?
• Does it need to be accessible from any node in the cluster, or are you good with node local storage?

Ultimately it’s not actually hard, it’s just complex if you want to achieve anything like you get with EBS volumes in AWS, but in your homelab. Here’s how I always try to approach complex problems: start as simple as possible, make it work, then add complexity only as needed.

My Requirements

• To have dynamically provisioned persistent volumes
• To have that persistent volume be accessible from any node in my cluster
• Keep it as simple as reasonably possible

How Am I Doing It?

Currently I have a single 2TB NVMe disk and I want to use that for dynamically provisioned storage. I’m not worried about replication right now since I have backups in place and this is just for my homelab. If I wanted to do replicated storage, I might consider a Ceph cluster, but that realistically requires a decent amount of hardware and fast networking interconnectivity (greater than 1GB, ideally 10GB minimum for replication to keep up).

In order to manage the disk I’m using TrueNAS Scale, which is basically ZFS on Linux with a nice web GUI to manage things. This actually provides the option of doing a zpool with maybe 2 disks mirrored, or even a RAIDZ6 as your storage target to easily solve for replication across disks.

In Proxmox, I’m passing the disk itself directly to TrueNAS VM. You should pass an entire disk controller if you need to pool multiple drives together in ZFS. If you’re dealing with a single disk, it’s OK to do it this way.

Spin up the VM, format the disk, ZFS, etc. and you’re ready to go. In my case, this is on the Kubernetes VLAN and assigned a static IP. Originally I had a second NIC attached to the primary VLAN but this caused some weird web UI performance issues so I removed that one.

TrueNAS Setup In Proxmox

This isn’t intended to be a TrueNAS tutorial, so I’ll just list the steps at a high level. Basically, get a TrueNAS server running and then proceed.

• Pass a disk (or HBA controller) to a VM in Proxmox
• Install TrueNAS Scale
• Create a zpool
• Generate an API Key - in the top right corner go to Admin > API Keys
• Make sure the network is accessible from your Kubernetes cluster

Install democratic-csi With TrueNAS

https://github.com/democratic-csi/democratic-csi

This is a straightforward CSI provider that focuses on dynamically provisioned storage from TrueNAS or generic ZFS on Linux backends. Protocols include NFS, iSCSI, and NVMe-oF. I’ll show you how to use the API variation and do NFS and iSCSI shares, plus talk about almost getting NVMe-oF working.

You will install a separate Helm chart for each provisioner, and you can actually run multiple at the same time, which is what I will be doing with both NFS and iSCSI. This is helpful since NFS even supports RWX volumes (if you actually have a use case for that), while iSCSI is a good default for RWO volumes.

VolumeSnapshot Support

This is optional, but if you want to utilize volume snapshots (which became GA as of Kubernetes 1.20), you will need to install the CRDs which aren’t included with vanilla Talos, along with installing the “snapshotter.” This implements Volume Snapshots - https://kubernetes.io/docs/concepts/storage/volume-snapshots/

• Clone repo: git clone https://github.com/kubernetes-csi/external-snapshotter.git
• cd external-snapshotter
• Apply CRDs: kubectl kustomize client/config/crd | kubectl create -f -
• Install snapshotter into kube-system: kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -
• Verify: kubectl get deploy snapshot-controller -n kube-system

Dynamic iSCSI Provisioner With freenas-api-iscsi

My single 2TB disk is in a pool named nvme2tb. I created a dataset in TrueNAS named iscsi. Those may vary in your case, so pay attention to the configuration and update those values according to your environment.

Don’t forget, your Talos installation needs to include the iscsi extension or the nodes won’t be able to connect to TrueNAS.

• Create a dataset named iscsi
• Make sure Block (iSCSI) Shares Targets is running, and click Configure
• Save the defaults for Target Global Configuration
• Add a portal on 0.0.0.0:3260 named k8s-democratic-csi
• Add an Initiator Group, Allow all initiators, and name it something like k8s-talos
• Create a Target named donotdelete and alias donotdelete, then add iSCSI group selecting the Portal and Initiator Group you just created. This prevents TrueNAS from deleting the Initiator Group if you’re testing and you delete the one and only PV.
• Make note of the portal ID and the Initiator Group ID and update these values in the file freenas-api-iscsi.yaml if needed
  • During testing, the manually created Initiator Group was getting deleted whenever deleting the last PV. This appears to be a bug in TrueNAS somewhere according to https://github.com/democratic-csi/democratic-csi/issues/412. Essentially TrueNAS deletes the Initiator Group automatically if an associated Target is deleted and no others exist. If you followed the instructions and created a manual Target this won’t be an issue :)
• Create the democratic-csi namespace: kubectl create ns democratic-csi
• Make that namespace privileged: kubectl label --overwrite namespace democratic-csi pod-security.kubernetes.io/enforce=privileged
• Create freenas-api-iscsi.yaml and update apiKey, host, targetPortal, datasetParentName, and detachedSnapshotsDatasetParentName. Other common considerations in storage class YAML config are storageClasses.defaultClass (true/false, there can only be one), and storageClasses.reclaimPolicy (Delete/Retain) and if you set this to Retain, there is less chance that data will be deleted if you delete a pod (for example), but you are also responsible for deleting this manually in TrueNAS if you ever need to.

  driver:
    config:
      driver: freenas-api-iscsi
      httpConnection:
        protocol: https
        apiKey: [your-truenas-api-key-here]
        host: 10.0.50.99
        port: 443
        allowInsecure: true
      zfs:
        datasetParentName: nvme2tb/iscsi/volumes
        detachedSnapshotsDatasetParentName: nvme2tb/iscsi/snapshots
        zvolCompression:
        zvolDedup:
        zvolEnableReservation: false
        zvolBlockSize:
      iscsi:
        targetPortal: "10.0.50.99:3260"
        targetPortals: []
        interface:
        namePrefix: csi-
        nameSuffix: "-talos"
        targetGroups:
          - targetGroupPortalGroup: 1
            targetGroupInitiatorGroup: 5
            targetGroupAuthType: None
            targetGroupAuthGroup:
        extentInsecureTpc: true
        extentXenCompat: false
        extentDisablePhysicalBlocksize: true
        extentBlocksize: 512
        extentRpm: "SSD"
        extentAvailThreshold: 0
  
  csiDriver:
    # should be globally unique for a given cluster
    name: "org.democratic-csi.freenas-api-iscsi"
  
  storageClasses:
    - name: truenas-iscsi
      defaultClass: true
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: ext4
        detachedVolumesFromSnapshots: "false"
      mountOptions: []
      secrets:
        provisioner-secret:
        controller-publish-secret:
        node-stage-secret:
        node-publish-secret:
        controller-expand-secret:
  
  volumeSnapshotClasses:
    - name: truenas-iscsi
      parameters:
        detachedSnapshots: "true"
  
  node:
    hostPID: true
    driver:
      extraEnv:
        - name: ISCSIADM_HOST_STRATEGY
          value: nsenter
        - name: ISCSIADM_HOST_PATH
          value: /usr/local/sbin/iscsiadm
      iscsiDirHostPath: /usr/local/etc/iscsi
      iscsiDirHostPathType: ""
  

• Deploy: helm upgrade --install --namespace democratic-csi --values freenas-api-iscsi.yaml truenas-iscsi democratic-csi/democratic-csi
• Verify:
  • You’re looking to see that everything is fully running. It may take a minute to spin up.
  • kubectl get all -n democratic-csi
  • kubectl get storageclasses or kubectl get sc

Test - Deploy A PVC

• Test with a simple PVC, targeting our new truenas-iscsi storage class, test-pvc-truenas-iscsi.yaml:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: testpvc-iscsi
  spec:
    storageClassName: truenas-iscsi
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
  

  • kubectl apply -f test-pvc-truenas-iscsi.yaml
  • Be patient, this can take a minute to provision the new Zvol on TrueNAS (looks something like nvme2tb/iscsi/volumes/pvc-25e70f84-91c7-4e49-a9f1-e324681a3b7d) and get everything mapped in Kubernetes.
  • Check the Persistent Volume itself: kubectl get pv
    • Looking for a new entry here
  • Check the Persistent Volume Claim: kubectl get pvc
    • Looking for status Bound to the newly created PV
  • If you need to investigate, next look at kubectl describe pvc and kubectl describe pv, or go look in the TrueNAS UI to see if a new disk has been created

Test - Deploy A Pod

At this point there should be a PV and PVC, but they are not actually connected to a pod yet. The moment a pod claims a PVC, that’s when the actual node that the pod is running on will mount the iSCSI target, and this is where the iscsi-utils extension comes into play in Talos Linux. Let’s test to make sure we can actually connect to the PVC from a pod.

This test pod uses a small Alpine image and writes to a log file every second. The two lines commented out at the bottom are there in case you want to target a specific node. I would recommend if you’re not sure all your Talos Linux nodes are configured properly for iSCSI to target each of them and verify from every pod. You can delete the pod, but preserve the PVC and if you reconnect to the PVC from another pod, even if it’s running on another node, it should still contain the same data.

• Create pod-using-testpvc-iscsi.yaml:

  apiVersion: v1
  kind: Pod
  metadata:
    name: testlogger-iscsi
  spec:
      containers:
      - name: testlogger-iscsi
        image: alpine:3.20
        command: ["/bin/ash"]
        args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
        volumeMounts:
        - name: testvol
          mountPath: /mnt
      volumes:
      - name: testvol
        persistentVolumeClaim:
          claimName: testpvc-iscsi
  #    nodeSelector:
  #      kubernetes.io/hostname: taloswk1
  

• Deploy: kubectl apply -f pod-using-testpvc-iscsi.yaml
• Verify kubectl get po
  • Check which node it’s on with kubectl get po -o wide or kubectl describe po testlogger-iscsi | grep Node:
• Validate data is being written to the PVC:
  • Exec into the pod: kubectl exec -it testlogger-iscsi -- /bin/sh
  • Look at the file: cat /mnt/test.log
  • Show line count: wc -l /mnt/test.log

Test - Cleanup

• Delete pod: kubectl delete -f pod-using-testpvc-iscsi.yaml
• Delete PVC: kubectl delete -f test-pvc-truenas-iscsi.yaml

Dynamic NFS Provisioner With freenas-api-nfs

This one’s a little simpler than iSCSI since support is built into Talos automatically, and there’s less setup on the TrueNAS side.

• Create a dataset named nfs
• Create the democratic-csi namespace: kubectl create ns democratic-csi
• Make that namespace privileged: kubectl label --overwrite namespace democratic-csi pod-security.kubernetes.io/enforce=privileged
• Create freenas-api-nfs.yaml and update apiKey, host, shareHost, datasetParentName, and detachedSnapshotsDatasetParentName

  driver:
    config:
      driver: freenas-api-nfs
      httpConnection:
        protocol: https
        apiKey: [api-key-goes-here]
        host: 10.0.50.99
        port: 443
        allowInsecure: true
      zfs:
        datasetParentName: nvme2tb/nfs
        detachedSnapshotsDatasetParentName: nvme2tb/nfs/snaps
        datasetEnableQuotas: true
        datasetEnableReservation: false
        datasetPermissionsMode: "0777"
        datasetPermissionsUser: 0
        datasetPermissionsGroup: 0
      nfs:
        shareCommentTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
        shareHost: 10.0.50.99
        shareAlldirs: false
        shareAllowedHosts: []
        shareAllowedNetworks: []
        shareMaprootUser: root
        shareMaprootGroup: root
        shareMapallUser: ""
        shareMapallGroup: ""
  
  csiDriver:
    # should be globally unique for a given cluster
    name: "org.democratic-csi.freenas-api-nfs"
  
  storageClasses:
    - name: truenas-nfs
      defaultClass: false
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: nfs
      mountOptions:
        - noatime
        - nfsvers=4
  
  volumeSnapshotClasses:
    - name: truenas-nfs
  

• Since we are using snapshots, you also need to install a snapshot controller such as https://github.com/democratic-csi/charts/tree/master/stable/snapshot-controller
  • You can skip this step if you disable snapshots in your YAML file
  • helm upgrade --install --namespace kube-system --create-namespace snapshot-controller democratic-csi/snapshot-controller
  • kubectl -n kube-system logs -f -l app=snapshot-controller
• Deploy: helm upgrade --install --namespace democratic-csi --values freenas-api-nfs.yaml truenas-nfs democratic-csi/democratic-csi
• Verify:
  • You’re looking to see that everything is fully running. It may take a minute to spin up.
  • kubectl get all -n democratic-csi
  • kubectl get storageclasses or kubectl get sc

Test - Deploy A PVC

• Test with a simple PVC, targeting our new truenas-nfs storage class, test-pvc-truenas-nfs.yaml:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: testpvc-nfs
  spec:
    storageClassName: truenas-nfs
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
  

  • kubectl apply -f test-pvc-truenas-nfs.yaml
  • Be patient, this can take a minute to provision the new block device on TrueNAS and get everything mapped in Kubernetes.
  • Check the Persistent Volume itself: kubectl get pv
    • Looking for a new entry here
  • Check the Persistent Volume Claim: kubectl get pvc
    • Looking for status Bound to the newly created PV
  • If you need to investigate, next look at kubectl describe pvc and kubectl describe pv, or go look in the TrueNAS UI to see if a new disk has been created

Test - Deploy A Pod

At this point there should be a PV and PVC, but they are not actually connected to a pod yet. The moment a pod claims a PVC, that’s when the actual node that the pod is running on will mount the NFS share.

This test pod uses a small Alpine image and writes to a log file every second. The two lines commented out at the bottom are there in case you want to target a specific node. I would recommend if you’re not sure all your Talos Linux nodes are configured properly for iSCSI to target each of them and verify from every pod. You can delete the pod, but preserve the PVC and if you reconnect to the PVC from another pod, even if it’s running on another node, it should still contain the same data.

• Create pod-using-testpvc-nfs.yaml:

  apiVersion: v1
  kind: Pod
  metadata:
    name: testlogger-nfs
  spec:
      containers:
      - name: testlogger-nfs
        image: alpine:3.20
        command: ["/bin/ash"]
        args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
        volumeMounts:
        - name: testvol
          mountPath: /mnt
      volumes:
      - name: testvol
        persistentVolumeClaim:
          claimName: testpvc-nfs
  #    nodeSelector:
  #      kubernetes.io/hostname: taloswk1
  

• Deploy: kubectl apply -f pod-using-testpvc-nfs.yaml
• Verify kubectl get po
  • Check which node it’s on with kubectl get po -o wide or kubectl describe po testlogger-nfs | grep Node:
• Validate data is being written to the PVC:
  • Exec into the pod: kubectl exec -it testlogger-nfs -- /bin/sh
  • Look at the file: cat /mnt/test.log
  • Show line count: wc -l /mnt/test.log

Test - Cleanup

• kubectl delete -f pod-using-testpvc-nfs.yaml
• kubectl delete -f test-pvc-truenas-nfs.yaml

Dynamic NVMe-oF Storage For Kubernetes (I was unable to make this work)

I spent some time trying to get this to work. TrueNAS doesn’t currently support NVMe-oF through the interface, but since it’s just a Linux box you can simply (almost simply) install extra packages needed and configure them as root. After doing that, I tested manually by connecting from another Linux machine to validate that I could indeed mount NVMe over TCP using TrueNAS.

From there, I figured out the configuration needed for democratic-csi zfs-generic-nvmeof driver and started testing. I got as far as getting it to provision a new dataset on TrueNAS, create the mount, and create the PV and PVC in the cluster, showing as Bound. However, when I would actually attempt to connect to it from a pod, it would fail. It may have something to do with how democratic-csi does the mount from the node, or otherwise I might have something wrong in my configuration that I can’t figure out.

If I could get this working, I might not even bother running a TrueNAS instance and just run some lightweight Linux server to interface between democratic-csi and the disk(s).

Here’s some extra details on exactly what I tried:

• https://github.com/siderolabs/talos/issues/9255
• https://github.com/democratic-csi/democratic-csi/issues/418

Please help me if you know how to make this work, as I’d much rather be using this than iSCSI :)

Here’s my almost working config for reference:

csiDriver:
  name: "org.democratic-csi.nvmeof"

storageClasses:
  - name: truenas-nvmeof
    defaultClass: false
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    allowVolumeExpansion: true
    parameters:
      fsType: ext4

    mountOptions: []
    secrets:
      provisioner-secret:
      controller-publish-secret:
      node-stage-secret:
      node-publish-secret:
      controller-expand-secret:

volumeSnapshotClasses: []

driver:
  config:
    driver: zfs-generic-nvmeof
    sshConnection:
      host: 10.0.50.99
      port: 22
      username: root
      privateKey: |
        -----BEGIN RSA PRIVATE KEY-----
        REDACTED!
        -----END RSA PRIVATE KEY-----        
    zfs:
      datasetParentName: nvme2tb/k8s/nvmeof
      detachedSnapshotsDatasetParentName: nvme2tb/k8s/nvmeof-snapshots

      zvolCompression:
      zvolDedup:
      zvolEnableReservation: false
      zvolBlocksize: 16K

    nvmeof:
      transports:
        - tcp://0.0.0.0:4420

      namePrefix:
      nameSuffix:

      shareStrategy: "nvmetCli"

      shareStrategyNvmetCli:
        nvmetcliPath: nvmetcli
        configIsImportedFilePath: /var/run/nvmet-config-loaded
        configPath: /etc/nvmet/config.json
        basename: "nqn.2003-01.org.linux-nvme"
        ports:
          - "1"
        subsystem:
          attributes:
            allow_any_host: 1

What About Other Options?

What About Local Storage?

This fails to meet one of my main requirements of being able to mount a persistent volume to any node. The whole point of Kubernetes, at least for me, is the ability to take a node offline with no actual downtime of any services. If you have a pod connected to a PVC on a certain node, but I need to move that pod to another node, I need to reconnect to the existing persistent volume, but this method doesn’t allow for that.

What About proxmox-csi-plugin?

I also looked at the proxmox-csi-plugin, but that has the same problem as with node local storage so doesn’t fit my requirements.

What About Rancher Longhorn?

Longhorn is nice and easy. It uses either NFS or iSCSI. The Talos team recommends against both NFS and iSCSI (although either can be used). I would say to use Longhorn if you need replicated storage in a homelab (like 3 disks) but don’t want to figure out Ceph, etc. It’s straightforward and well supported. As for downsides, I don’t have firsthand experience. I hear it’s great for usability, but some users complain that it’s not reliable.

It’s also a little more complicated to set up specifically with Talos, but Longhorn has specific instructions for installation with Talos so that shouldn’t be a man blocker.

At some point I might try this out, but for now I’m sticking with the simple approach of using TrueNAS Scale with any type of zpool you want, and dynamically provisioning NFS or iSCSI using democratic-csi.

What About Mayastor?

I tried getting this to work because the Talos team recommends it if you don’t want to do a full blown Ceph cluster, etc. I was also super interested in the fact that it uses NVMe-oF which is a newer protocol (basically a modern replacement for iSCSI). I wasn’t able to get it working, but I discovered a little bit about it and decided to keep it simpler for the following reasons:

• I only have a single disk currently, so I don’t need any replicated storage
• It seems to be more resource intensive than democratic-csi and has more components.
• The documentation is kind of hectic. It used to be Mayastor, but now it’s OpenEBS Replicated PV Mayastor or something. When deploying, it’s hard to tell if I’m deploying other OpenEBS stuff I don’t need or what exactly PV type I need. I think you need one type of PV to store data for one of the components even if you are ultimately trying to run the replicated PV type (mayastor) for your primary cluster storage. I don’t know, it was confusing.

My failure to get this working is 100% a skill issue, but going back to my requirements I really don’t need this for my homelab at this point. I may revisit this in the future.

What About Ceph?

I would need enough disks and resources to run Ceph. It’s something I really want to test out and potentially use, although probably way overkill for my homelab. I’m currently running 2 Minisforum MS-01 servers and planning on getting a third to do a true Proxmox cluster and replace my old 2U power hungry server. At that point, I might actually give Ceph a shot (trying both the Proxmox Ceph installation and also Rook/Ceph on top of Kubernetes). This would solve for truly HA storage, plus meet all other requirements I have, assuming I don’t add a requirement for low resource utilization just to run storage.