Inheriting legacy infrastructure is an adventure. We had been using the legacy runners for many years now, before I started with the company. They had some issues but overall they served their purpose. They were based on docker+machine which allowed for dynamically scaling EC2 instances in AWS. Those runners checked all the boxes originally - they ran jobs, they autoscaled, and they were easy to configure when needed. It was a great starting point, maybe even the best available option at the time. But over the years the pain points continued to grow, and I was already passively considering alternatives. Some of the pain points include:

1. One job per EC2 instance - I was disappointed when I found this out, to say the least
2. Slow scale up - it would take about 5 minutes for a new EC2 instance to bootstrap before it could accept a job
3. docker+machine weirdness - it would orphan EC2 instances forever, requiring periodic manual cleanup
4. No shared image cache - job startup time was abysmal due to a combination of insane image sizes and essentially no reuse of cached images (due to new EC2 instances for every job)
5. Base AMI images needed to be updated manually when the security team asked (bad, we should obviously be automating and doing this proactively, but nobody had time or desire to prioritize this)
6. docker+machine sometimes got in a weird state where it could not scale up new EC2 instances and instead would get stuck in a retry loop, preventing any new jobs in that queue from running until we resolved it manually
7. docker+machine has been deprecated for several years - we tried updating to a later version but things broke. It wasn’t worth the burden of tracking down why, rather than looking for a better path forward.
8. Observability. We just don’t have a good way to measure performance or monitor for issues (though admittedly that is largely an “us” issue)
9. Long lived EC2 instances. In our configuration, docker+machine idles down to 1 node per queue. In practice, this means that the first ever EC2 instance that was deployed for that queue remains online as long as possible, storing up image cache and other ephemeral data that is unneeded. Even with 200GB disks, we’ve had many occasions where it was necessary to SSH into the EC2 instances and run prune commands. We tried automating this but something about docker system prune -af non-interactively is unreliable over an SSH session.
10. Troubleshooting is tricky. We can SSH into the host running docker+machine, get a list of EC2 instances, then docker-machine ssh instance-name to access the instance and troubleshoot.
11. Cost (and waste). One EC2 instance per job adds up, especially when the vast majority of those jobs use under 10% of the allocated resources.

This project kind of developed organically out of multiple needs that seemed to come in all around the same time. I was already bored with the idea of the existing maintenance plan, and we were not proactive about it partly due to the toil involved. Developers started complaining about slow startup time - this was because of the lack of shared image caching, combined with developers pulling multi gigabyte images for test jobs. Sure, it’s easy to say they should optimize their images (and it couldn’t be more true). Security started pushing on us to “patch” the runners, which meant updating the base AMI images manually, but if we needed to keep this up to date monthly, we were going to need to automate somewhere. Now there was a compelling reason for me to get buy-in from leadership to assign this project, and I already had a plan in mind for how we could replace docker+machine and simplify the maintenance burden - Kubernetes. I had hopes that it would solve every pain point we had with docker+machine:

1. Workers could run multiple jobs, making better use of available resources. After all, that was supposed to be one of the main benefits of using Docker.
2. Fast scale up. Larger EC2 instances per worker meant that adding one more EC2 instance allowed X more jobs to run as soon as the new instance was ready, plus we could use lightweight base images rather than our bulky AMI.
3. Cluster Autoscaler should be capable of handling scaling in production environments, and it is actively maintained unlike docker+machine.
4. Better shared image cache. Since many jobs would now run per EC2 instance, all of those jobs would share the image cache that was pulled to that instance. There are better ways, but this was still a good start, and significantly better than the existing option.
5. Using AWS provided AMIs meant we no longer have to maintain or build custom AMIs with security and logging agents, etc.
6. Ideally, eliminate docker+machine weirdness by simply eliminating docker+machine itself.
7. EKS is supported by AWS, along with their AMIs. Kubernetes is actively maintained. Cluster Autoscaler is actively maintained. GitLab runners support Kubernetes.
8. We can get insight into overall health and performance using EKS container insights. It’s a major leap forward from what we had before, although not the most impressive tool in the world either.
9. No more long-lived EC2 instances. Cluster autoscaler happily scales down any which one, and given even a low amount of activity throughout the days and weeks it does a nice job balancing instances and cycling them out, which I love to see. I don’t need stale workers starting to exhibit weird issues that I probably don’t care about the root cause.
10. Maybe this one is personal preference, but I find it much easier to troubleshoot within Kubernetes. I use k9s which makes it amazingly easy (dare I say fun) to see the state of all pods running in the namespace, exec into them if desired, and easily get a picture of the pods, nodes, resource utilization, logs, etc. in realtime.
11. Thanks to the magic of Cluster Autoscaler, we can idle down the worker nodes very, very low while there are no jobs demanding resources. On average, I estimated this to save at least 50%, assuming the exact same workload we ran last year, with potential to save even more if we continue monitoring and tuning. And this still includes all the other benefits including less maintenance and faster performance.

Our team also explored a simpler option where we could build and maintain a few EC2 instances running Docker and use the docker executor. This would be a lot more in line with what the original engineer thought was going to happen when he decided on docker+machine years ago, running more than 1 job per EC2 instance. This would solve some of our pain points, but still leave us with some maintenance burden, having to care about long-lived EC2 instances, and still having to patch or automate a rolling build. It would provide better scaling and resource utilization, but still require us to manually right-size forever. While the team built and benchmarked a POC on this idea, I was testing on EKS. We figured we knew which option was going to win, but of course you don’t know until you try.

It turns out that yes, EKS is better in almost every way. It really did solve the pain points, got us into a much better security posture, and reduced our maintenance burden by well over 50%. Job startup time increased on average, and job performance even increased significantly. I may elaborate on that piece later on.

It was also the perfect way to introduce a production Kubernetes environment to the organization (yes this is the first thing we have ever done as a company with Kubernetes) because of the relative level of complexity.

Design

This was the goal: to build a scalable, maintainable platform on which to run stateless jobs for GitLab as efficiently as possible. I was solving for all of the pain points listed above, but primarily thinking about job startup performance, security, and long term maintenance. Cost savings is always good, but for this project it was just the icing on the cake.

Build Tooling

I’m not new to Kubernetes, but I was new to EKS. I wanted to use a GitLab pipeline to build the EKS infrastructure. After some quick research, I found eksctl to be the easiest way to build a cluster. It simplifies many aspects that would otherwise require a fair bit of CloudFormation, but is flexible enough to get the job done, with only minor effort required outside of that.

Other options included CloudFormation or CDK, but why bother when there is a bespoke tool directly from Amazon.

I built a pipeline in GitLab to deploy EKS using eksctl and AWS CLI. It relies on a YAML file that is specific to eksctl to define some specifics that I needed. I needed to use the AWS CLI to change the EKS upgrade policy to STANDARD. The pipeline is idempotent, so that I can safely run it as many times as I want without making unnecessary changes (think “if cluster exists, then eksctl upgrade cluster else eksctl create cluster”). I built some logic around nodegroups, so that when I change or add nodegroups they are deployed properly, but existing nodegroups are not changed or removed. I also opted to use access entries for authentication, so by including the name of the role in the config file, the pipeline automatically ensures the proper access entry is added for that role. That allows us to authenticate with the control plane through our AWS SSO configuration.

Networking

Originally, I was leaning toward putting everything in its own VPC. However, there was additional work to get the GitLab VPC talking to the EKS VPC, but most importantly I quickly realized that some jobs were connecting to external systems over the public internet with IP allow listing. I needed to reuse the existing EIP if at all possible.

I chose to place the EKS workers in the same VPC subnet as the existing runners due to IP allow listing on external systems from the NAT gateway EIP. It was easier if all outbound traffic came from the same IP. Don’t get me started on why we send all of this traffic over the public internet to begin with, that’s a battle for another day.

Compute

The other primary decision to make was compute. EKS delivers the control plane, but you still need somewhere to run your stuff. Our two options are EC2 or Fargate.

Based on this project’s goal, EC2 was the obvious choice because spinning up a single EC2 instance instantly provides capacity for X number of jobs to run in parallel. Maybe even more importantly is that jobs within the EC2 instance can share the image cache, greatly increasing the odds that a given job will start significantly faster if its target image is already cached.

Within EC2, I had 3 options: bring your own EC2, managed nodegroups, or EKS Auto Mode. At the time, Auto mode was so new that I wasn’t ready to commit to that for production workloads. Perhaps that would have turned out great, but it wasn’t ready in my opinion. Bring your own has its own obvious downsides, and we have another option so it was the obvious choice to go with managed nodegroups. It’s yet another thing that simplifies some of the deployment and maintenance burden, and we generally trust AWS solutions to work well which has been the case with managed nodegroups for us.

Access Control

I’m talking about specifically how to authenticate to the cluster API. It was a no-brainer to map IAM roles that our team already uses with SSO to Access Entries on the cluster. This allows us to go into the terminal, run assume (a granted.dev tool) and authenticate to AWS SSO.

I evaluated IRSA, but there was more effort to integrate with our OIDC provider and we didn’t have a need for fine-grained access controls up front.

RBAC is an area where we have lots of room for improvement. For now, it’s like giving the whole team root access to the whole cluster. Granted it’s only our team who has any access, but we still don’t follow the principle of least privilege here at this time, partly due to the process requiring manual updates via Helm, etc. No other teams require any level of access to the cluster, so we aren’t concerned with taking a tiered approach in this particular situation.

Config Management

Let’s talk about updating config.toml. I originally included the GitLab runner config within the same project repo that builds and deploys the EKS cluster. I now realize my mistake, because changing a runner config (maybe we need to increase the log limit) means running the EKS deployment pipeline. While it is idempotent, it’s still awkward at best. Also, the process is to update the repo, run through the pipeline, and on top of that we still have a manual process to authenticate and manually run helm commands to upgrade deployments.

I see a couple of opportunities for improvement:

• I should have put gitlab runner jobs in their own namespace. I did create a namespace for everything gitlab runner related, but this also includes the gitlab-runner deployment itself which is a minor annoyance, and doesn’t help when aggregating things like performance metrics.
• Config for gitlab-runner deployments (used with the Helm charts) are mixed into the same repo as the EKS infrastructure build. This should go in its own repo.
• GitOps. What was I thinking? I should have used FluxCD up front. Even fixing some issues with config management and splitting out namespaces properly leaves us with a significant amount of manual effort to make simple config changes.
  • The process is more tedious than I would expect. With the old system it was: SSH into the test GitLab host > backup and live edit config.toml > test > repeat for production. With the new system, the process is longer: checkout a feature branch > update the config > manually test the config by running helm upgrade commands (assuming I’m already set up with kubectl, kubeconfig, authentication, helm, helm repos, etc.) > test > merge to main > repeat for production.
  • Why bother updating a config in a repo when you’re going to change it manually in production? We all hope nobody ever makes a mistake and they get out of sync, but I’ve never seen this work well, anywhere. I assume there will ALWAYS be drift because we built a system that allows that there CAN be drift. What if there’s some production outage, and the on-call needs to fix it quickly overnight? The chances it will go back into the repo later are low, despite everyone’s best intentions.
  • All this complexity, all a waste of time and effort. What if there was a system that could proactively sync what’s in the config repo with what’s currently running in the cluster? Welcome to GitOps.
    • This is on the roadmap before we do anything new in the cluster. The new process will be: update the config in the repo > open a MR > test (in the cluster) > merge to main > validate in production
    • Anyone can do it
    • Nobody needs to install special tools or question IAM permissions, including junior engineers or even interns
    • We can enforce change approvals
    • Visibility - we have an audit trail. If it’s not in the repo, it’s not in the cluster
  • I haven’t even talked about what else is deployed in the cluster beyond gitlab runners. It’s not much, but it’s still something. FluxCD can and should also manage Cluster Autoscaler and Metrics Server.
  • FluxCD flips the traditional deployment model on its head, which has major security implications. If we grant FluxCD enough privileges to do its job, then we no longer need every engineer on our team to have full root access to the cluster via kubectl commands (any changes need to go through the repo, through FluxCD). Let’s go into read only mode for debugging/troubleshooting but leave updates/writes to GitOps.
    • Sure, we still need a way in, maybe a break-glass option. One simple approach would be to authenticate to the AWS account where EKS lives, manually add a new access entry, and boom, you’re in. Document this. Or even automate this, put a manual job in the repo pipeline called “break glass” with clear, simple instructions. It should be easy for anyone to find and execute, but also auditable.

All Together

I’ve talked through most of the design, much of which came from the initial POC but some of which comes from having been running this in production for over a year and what I’ve learned as new needs arise.

This also doesn’t give the full picture of everything involved in making it work. Beyond deploying EKS, managed nodegroups, cluster autoscaler, metrics server, and gaining access to the cluster, we still have to deploy gitlab-runners and wire them up to gitlab to accept jobs. We also have to consider nuances when it comes to how cluster autoscaler works and some quirks that it has.

Deploying this was a very low risk deployment due to the ability to add the new EKS infrastructure in parallel with existing runners, allowing us to phase out the old stuff tag by tag. EKS runners have new, unique tags, and eventually we would add legacy tags, and finally phase out the old runners once we validated that jobs worked reliably.

In the end it has been a success, despite having some snags along the way. We learned new things, and always came up with a new, better way to move forward. One of the major benefits to this architecture is the flexibility. It feels unlimited, there doesn’t seem to be any hurdle too big to get past, and most hurdles have been fairly minor.

Other factors I considered throughout this process:

• Skillset. Our team is not experienced in Kubernetes, though several on the team are working towards the CKA certification. But with a team of Linux Admins, I know everyone is capable of following the steps. Once you get past the initial hurdle of getting authenticated and setting up local tools, the rest is pretty straightforward even for inexperienced engineers.
  • The way I see it, this is a good way to level up our team in general. In many ways already, it demonstrates best practices and the way things could/should be. It gives everyone production experience with Kubernetes and this only makes us better.
• Unknown behavior with migrating all of the different pipeline jobs to the new system. In theory it should work the same because it has the same default CPU/RAM resources available. In practice, resource limits are handled very differently. We are essentially introducing new limits or enforcing stricter limits than what we had previously. And this is essential so that we can tune scaling, establish safe defaults, and manage cost over time.
• New maintenance schedule and process. EKS upgrades. Add-on upgrades. Component upgrades. Nodegroup AMI upgrades. The great news here is that it’s possible to do all of these while workloads continue running, so we just unlocked significant maintenance improvements and security by extension by being able to keep things updated more frequently with less friction. And we can patch zero-day vulnerabilities any time with very little risk.
• Migration from legacy runners. The plan was to introduce the new runners and ask developers to start testing. We had mixed results. The backup plan was to start switching tags to the new runners, and then removing those tags from the old runners, phasing them out. Once we phase the old runners out and haven’t had any outages or break/fix tickets come in, we can finally decommission the old infrastructure and fully realize the security and cost savings.

Gotchas

> eksctl

This is a handy utility, but is not idempotent by design. If you deploy a new cluster with this tool, and later on decide to change something about that cluster (but don’t want to rebuild), you can’t necessarily update your config and redeploy. Even the command is different: new builds are eksctl create cluster where certain changes can be done with eksctl upgrade cluster. Some changes to managed nodegroups can be done with eksctl update nodegroup but others require completely rebuilding the nodegroup. But in the end, the total cluster config and deployment pipeline is significantly simpler to build and understand than it would have been using CloudFormation.

> EKS Built-in CNI

The AWS CNI that comes out of the box is that assigns a unique IP from your VPC subnet to every pod. I was not expecting this, and during initial POC testing I quickly found out when I started scaling up parallel jobs. Luckily it’s possible to add a new CIDR block to the existing VPC, so I went ahead and added a new block and split that into 2 subnets in different AZs, then wired those up to the existing route table and out through the existing NAT gateway. Crisis averted. We could have switched CNIs as well, but I was ready to move forward, and since this was supported it was a little bit less effort in the short term. If we start going beyond 1000 parallel jobs then I’ll be revisiting this one and moving to something else. For now, we’re well below that, so it’s noted and we can move on.

> Manual Changes

Manual changes still need to be made in the current state. After building the cluster (fully automated), it’s in a vanilla state and requires you to manually deploy your resources. For us this includes Metrics Server, Cluster Autoscaler and gitlab-runners. This is an area where GitOps would help tremendously.

> Spot Instances

This isn’t too difficult. In the eksctl config, enable spot: true and be sure to specify at least 2 or 3 different instance types within the same family with the same minimum CPU/RAM specs. Just in case one type is not available during spot request, it can pick from something else so that jobs are not stuck pending. I also ran into this. You’re already saving so much money, this is not the time to be a stickler.

I think that’s it for spot instances.

> OOMKilled Jobs

In our case going from EC2 instances with 8GB RAM and 4GB swap to pods with a strict 8GB limit caused a couple of OOMKills here and there. It turns out this was one of the unknowns we anticipated going into the migration.

Why does this happen if the RAM limit matches? On the EC2 instance, the job has full reign of the VM it runs on, so not only is the Linux kernel more conservative about preventing OOMkills, it also has a 4GB swap buffer that it can exhaust before giving up.

This is actually a major factors I found to be related to the significant performance increase for some workloads on EKS, despite no changes to the pipeline and similar resource limites from a CPU/RAM perspective. Why would a job run 25% faster? It could be that we were silently exceeding the resource limits on legacy runners but it wasn’t bad enough to break jobs, just bad enough to make them slow.

Does this mean EKS is worse or less reliable in a way? No. This makes it obvious that jobs were already overprovisioned, but nobody was aware of it. Nobody was even aware of the performance degradation due to heavy over-utilization of swap. I see it as a feature, it highlights the need for better visibility, and it highlights the importance of understanding what we truly need from a resource perspective to run jobs. No more throwing hardware at the problem.

We should understand what jobs cost from a resource and financial perspective, because that will allow us properly tune autoscaling, make things run much more efficiently, likely save significantly on cost, and make informed decisions on resource quota increases if needed.

We’re almost done with OOMkilled, but not quite. Guess what, it’s super easy to override requests and limits in GitLab pipelines to ask for more CPU or RAM. I have this documented in a simple to follow guide with screenshots and specific examples of different messages that developers might see which could be related. I set upper limits on CPU and RAM requests. I explain why you should probably not set CPU limits, especially with Java workloads. And I just handed them self-service autonomy that frees up our team from troubleshooting OOMkill issues and rather provides developers what they need to keep working 24/7.

There’s more to talk about on this topic, including setting lower default limits once developers are more comfortable overriding these values. Looking at utilization, 8GB RAM is a pretty high limit which means we are very loosely packing jobs and wasting a lot of resources still. Keep in mind this still fits within the 50% savings estimate, but we could be doing significantly better by tightening up requests for most jobs that only need 1-2GB, and packing more of those jobs into a single worker. Another approach to this problem is encouraging developers who will listen to override their requests for lightweight jobs to lower values, proactively. Even if we leave the default at 8GB, if developers are proactive about this it can still offer them tangible benefits. They are more likely to get their jobs scheduled sooner, because a 2GB slot is easier to schedule than an 8GB slot, etc.

> Cluster Autoscaler

To me, Cluster Autoscaler is the source of the most interesting gotchas I ran into. These took the most time and caused most of the hurdles that we have had to overcome. Here’s what I ran into, in order.

» Scaling Up

This is the easy part. Deploy Cluster Autoscaler (CA) and configure its min/max thresholds. However, there’s a little bit of glue you need to tie this together with the Managed Nodegroups and the AutoScalingGroups (ASG) they manage. It’s not 100% ready out of the box. Here’s a recap of what needs to happen, how it works:

• CA needs permission to read and make changes to the ASG. This is done via an IAM role attached to the EC2 instance powering your worker nodes (every worker node has the same profile). If you specify autoScaler: true in the eksctl config, this is all handled automatically.
• Cluster Autoscaler (CA) interacts directly with the ASG, increasing desired capacity to scale up. It identifies the correct ASG by matching with tags on the ASG. These are already set in the eksctl config.
• It’s pretty easy to check if this works. If you submit more jobs from GitLab, wait a minute and then check to see if the ASG requested more instances. If that doesn’t work, move to troubleshooting (checking CA logs, IAM permissions, etc.).

» Scaling Down

This is where it starts to get really interesting. During initial testing, scale-up and scale-down worked great. I believe the default was 10 minutes before scaling down, but I didn’t experience any problems. After running more workloads over time, I got a ticket that jobs were pending for a long time. I quickly realized that autoscaling wasn’t working. The online worker nodes were taking jobs just fine, but CA wasn’t doing what I expected.

What went wrong here?

The initial design was intentionally simple (KISS). I deployed a single managed nodegroup and deployed everything on it, including all gitlab-runner deployments, and CA itself. Now it’s obvious in hindsight. When you run important things on the very nodes you are potentially scaling down and terminating, bad things can happen. CA terminated the node where the CA deployment itself lived, and it got stuck in a bad state.

• “If we were your kids, we’d punish ourselves.” - Little Rascals, 1994

CA was punishing itself and I gave it no other choice.

The solution? What if I deployed another, smaller managed nodegroup (2 nodes for HA), labeled it management, and isolated it from CA? So that’s exactly what I did. Add that to the EKS deployment, add labels and taints, and update taints/tolerations on the CA deployment. I protected it from itself, and this problem is solved.

> Management Nodegroup

Without going into too much detail, just keep in mind that this means we have to manage taints and tolerations so that management workloads go to management nodes, and job workloads go to job nodes. This involves node selectors and tolerations on the deployment side of things.

» Scaling Down To 0

This actually began as a request for a brand new type of runner we had not implemented before. There was a new need to build Dockerfiles on arm64, so I began evaluating options. Just because you can doesn’t mean you should, and while my first option was running a QEMU based build which could spit out multi-platform images from a single amd64 node, it wasn’t efficient and wasn’t worth the effort. The developers already had a completely separate Dockerfile with different build steps, so why not just give them a runner to build on arm64 natively?

I decided to create another managed nodegroup, but given the type of workload, the infrequency, and risk if jobs get interrupted, I decided it was worthwhile to base this on spot instances, plus while we’re at it let’s scale this one down to 0 since most of the time it’s unused and would be sitting idle, wasting money.

> Scaling Back Up From 0

The easy part is scaling down, that was already functioning. But Once you remove all instances from the ASG, now there’s a problem. How does CA know which ASG to go modify? Previously, it would use the running Kubernetes node to discover ASG info. When you get to the point where this ASG has no worker nodes running in the cluster, CA has less visibility, requiring you to add a specific tag to the ASG that it uses to identify it.

Specifically, you need to manually add a tag on the ASG k8s.io/cluster-autoscaler/node-template/label/nodegroup: [your-nodegroup-name] where you substitute the name of your nodegroup there, plus you need to have a label on your managed nodegroup that matches, which can be predefined in the eksctl config when you create the nodegroup.

I may have missed how to do this. I scoured the documentation and spent some trial/error time on this, but was unable to find a solution to fully automate this tagging in the build pipeline. Therefore I have instructions to specify that whenever using a managed nodegroup that scales to 0, you need to manually add the appropriate tag on the ASG. It’s one extra manual step. It would still be nice if it could be automated, and perhaps Copilot is right when it suggests it could be done by adding the right label in the right format within eksctl config.

I should revisit this some time. Even 1 simple manual step can become a larger issue over time.

» Scaling Down, Again

Wait, there’s more?! Yes, there’s more. This is the one I find the most interesting. I’m not sure if favorite is the right word but…

So we’re finally in a pretty good state, and people know what to do if they see OOMKilled (they have documentation and practice adjusting limits). I receive another troubleshooting ticket. This time a job has failed because the pod was terminated unexpectedly. Let’s dive into what happened.

Starting with the failed job log, it clearly states that the pod was terminated before the timeout, which was unexpected. I looked at the cluster autoscaler logs to find that it scaled down the very node this job was running on.

Back to the documentation: https://docs.aws.amazon.com/eks/latest/best-practices/cas.html

• I find a section named “Prevent Scale Down Eviction” which clearly states that expensive to evict pods should have the annotation cluster-autoscaler.kubernetes.io/safe-to-evict=false
• I apply the annotation and test in a test environment (run several long-running jobs in parallel to trigger scale-up, then wait)
• It doesn’t help, and Cluster Autoscaler is still resulting in premature job termination during scale-down

What’s wrong with Cluster Autoscaler on EKS, and why doesn’t it behave as advertised in the AWS documentation? I can’t answer that question, to be honest. The documentation is in fact wrong. It actually doesn’t work. Is there a bug with the Cluster Autoscaler?

I found an issue on Cluster Autoscaler’s GitHub page that matches my symptoms: https://github.com/kubernetes/autoscaler/issues/8196

If I understand the analysis correctly, the root cause is related to AZRebalance and how AWS terminates nodes when decreasing the desired capacity on the ASG. It’s not a bug that can be fixed in Cluster Autoscaler, and AWS is not going to address this issue on their end.

OK… so what can we do?

I figured since there will be no official fix, we should try to work around the issue. PodDisruptionBudget came to mind. What if you can apply a PDB with maxUnavailable: 0? I tried it, and this time it worked. Great.

I posted my workaround in the GitHub issue so that hopefully it helps some people:https://github.com/kubernetes/autoscaler/issues/8196#issuecomment-3353620324

One last thing to note about this workaround: The documentation explains that using .spec.maxUnavailable is unsupported when using a PDB selector on a resource that is managed by something else. We are in a gray area, because while the gitlab-runner deployment is responsible for deploying job pods, the pods themselves aren’t controlled by something else like a ReplicaSet or Deployment. So having the label/selector on the pod, combined with .spec.maxUnavailable does work as expected. However, there are UnmanagedPods warnings in the PDB event history: Pods selected by this PodDisruptionBudget (selector: &LabelSelector{MatchLabels:map[string]string{eviction-protection-pdb: true,},MatchExpressions:[]LabelSelectorRequirement{},}) were found to be unmanaged. As a result, the status of the PDB cannot be calculated correctly, which may result in undefined behavior. To account for these pods please set ".spec.minAvailable" field of the PDB to an integer value.

I’m OK with this warning because this works exactly as I expect it to. I have this documented thoroughly for future reference, and I consider this one mitigated.

» AZRebalance

Here we go again. Another ticket comes in, another job terminated prematurely. I thought we fixed autoscaling (and we kind of did), but now the AZRebalance feature is biting us.

This is not exactly related to Cluster Autoscaler, but kind of similar in a way. Instead of scaling up or down, it rebalances which is a fancy way of saying add a new node in a different AZ, then terminate a node in the hot-spot AZ.

Verifying the failed job log, it failed prematurely. Digging deeper, we look at ASG events and also CloudTrail, and find a tight correlation: a node was terminated seconds after the job failed unexpectedly. Full disclosure, kiro-cli is an amazing tool for digging through event history and logs in AWS, and correlating data with event times. It basically did all the work in this case, although it had completely the wrong event initially, I called out the timing mismatch, and it found another AZRebalance event that matched up perfectly.

What happened in this case?

• As nodes are scaled up and down, it’s possible to find yourself in a situation where there’s an unbalanced number of nodes in a single AZ.
• Independently, AZRebalance is checking for this case. Its job is to rebalance when things are not balanced.
• In order to balance things, AZRebalance will spin up a new node in another AZ with fewer nodes, and then it will terminate the extras in the hot-spot AZ.
• AZRebalance doesn’t discriminate, it doesn’t even check what’s running. It cares not about your workloads (and rightfully so, we need to build around this understanding). In this case, the node it selected as tribute was one that had production jobs running on it.

What can we do about it?

• We can add yet another tool, AWS Node Termination Handler. That also involves setting up an SQS Queue and EventBridge. Not the end of the world, but one more thing to maintain.
  • More importantly, we are under pressure to deliver on other things right now, so building this out requires more engineering effort.
• OR the quick and dirty approach is to disable AZRebalance. It’s not ideal, but it got us in a more reliable state immediately. The caveat is that we document this, and make sure it’s included in our backlog to resolve later on. This still leaves us in a potentially bad situation where nodes can hot-spot in a single AZ, exposing us to more risk if that AZ has an outage. But we are OK with the tradeoffs for the short term.

My tentative plan is to incorporate AWS Node Termination Handler. It appears to be relatively straightforward to implement and deploy. Obviously this would be built right into the pipeline that builds the EKS cluster, automating all of the setup. I’m not sure if I would jump to CloudFormation or the CDK, but whichever seems simpler to configure and maintain is what I would lean toward, starting with CloudFormation just because we are using it for another small piece of the pipeline already.

Disabling the rebalancer is not maintainable long term, as any new managed nodegroups would have it enabled by default and require someone to remember (never ideal). Even upgrading a nodegroup as we did when moving from BottleRocket to AmazonLinux2023 would have the same impact.

Conclusion

What began as an oversimplified idea in my head evolved into a production ready, scalable project that resulted in noticeably better performance, significantly reduced maintenance burden, better observability, and significant cost savings. Despite thinking this would be a relatively straightforward intro to Kubernetes for our team, we still ran into lots of mostly small gotchas. Luckily, similar to Linux, Kubernetes is extremely flexible and powerful, making it possible to find a way forward in every circumstance. This is where Kubernetes shines, helping us manage a sufficiently complex architecture and standardizing deployment, troubleshooting and observability. Developers are happy, the business is happy, our team is happy, and we learned a lot along the way.

In a previous post, I mentioned that I struggled to get OpenEBS working in Talos and instead went with democratic-csi. In recent weeks, I decided to revisit this and figure out how to get OpenEBS replicated storage working in order to evaluate replicated storage in my cluster. I now have multiple disks that I can dedicate to my Kubernetes cluster and wanted to avoid the issue with the single point of failure using a TrueNAS VM for democratic-csi.

If you are following along, I will assume you are familiar with deploying Talos Linux itself and have talosctl installed with an existing cluster running. If you need more details on how to do that, check out https://blog.dalydays.com/post/kubernetes-homelab-series-part-1-talos-linux-proxmox/.

Dedicated Storage Node

It’s not absolutely necessary to use a dedicated storage node. I’m doing this because I want to pass a disk directly to a VM for storage on each physical host and want to keep storage somewhat isolated from other worker nodes, and I can spare the few extra resources to dedicate to this purpose. If you want to use existing worker nodes, just follow this process for your existing nodes instead of creating new ones.

Create New Talos Node(s)

• Create a VM in Proxmox with 4GB RAM and 4 vCPU cores (2GB RAM is not enough due to the fact that you will be enabling hugepages which takes up 2GB and you would see oom-kills otherwise. You also need a dedicated CPU core just for the io-engine, along with all the other stuff that runs. I tried with 2vCPU and it wouldn’t schedule the io-engine pod due to insufficient resources). I named my first one talos-storage-1
  • Attach a Talos ISO to the CD ROM and boot from it
  • Get the IP address from the node
• Install Talos using the worker.yaml template used for other worker nodes (you may want to get a current or updated version of Talos from the image factory):
  • talosctl apply-config --insecure -n 10.0.50.135 --file _out/worker.yaml
• Apply a patch to set a static IP and node label, e.g.

# ./patches/storage1.patch
machine:
  network:
    hostname: talos-storage-1
    interfaces:
      - deviceSelector:
          busPath: "0*"
        addresses:
          - 10.0.50.31/24
        routes:
          - network: 0.0.0.0/0
            gateway: 10.0.50.1
    nameservers:
      - 192.168.1.22

• talosctl patch mc -n 10.0.50.135 --patch @patches/storage1.patch
  • I’m having trouble here with the node name changing, and I have to manually delete the random name from the cluster:
  • e.g. kubectl delete node talos-lry-si8
  • Also you might need to delete the label openebs.io/nodename= if you already have openebs running and are adding/changing nodes
    • kubectl edit node talos-storage-1 and change the value to the current node name
• Apply a patch to set some machine config stuff for OpenEBS which includes hugepages, a nodeLabel for where mayastor engine should run, and the /var/local bind mount:

# ./patches/openebs.patch
machine:
  sysctls:
    vm.nr_hugepages: "1024"
  nodeLabels:
    openebs.io/engine: mayastor
  extraMounts:
    - destination: /var/local
      type: bind
      source: /var/local
      options:
        - rbind
        - rshared
        - rw

• talosctl patch mc -n 10.0.50.31 --patch @patches/openebs.patch
• If you have an additional disk to use with OpenEBS, you’ll need to pass it directly to the Talos node VM. I’m using Proxmox
  • SSH into the Proxmox host and find the disk ID to be passed. I just run ls -lh /dev/disk/by-id/ and get the root disk (not containing any "_1" or "_part*", for example /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S59CNM0W635077P`)
  • Pass the disk directly to the Talos VM, where 511 is the VM ID and assuming you only have 1 disk already on scsi0: qm set 511 -scsi1 /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S59CNM0W635077P
  • Checking the hardware tab on VM 511 in Proxmox, you should see this new disk. Double click it and make sure to check “Advanced”, “Discard”, and “SSD emulation”
    • If you see orange on these settings, you will need to shut down the VM, then power it back on for the changes to apply. Rebooting won’t do it.
  • Now that the disk has been added, look for it with talosctl: talosctl get disks -n 10.0.50.31
    • In my case I see a disk named sdb which is 2.0TB with model “QEMU HARDDISK”
  • Mount the disk to be passed to containers with appropriate privileges. This is required for openebs-io-engine to access the extra disk.

  # ./patches/mount-sdb.patch
  machine:
    disks:
      - device: /dev/sdb
  

  • Apply: talosctl patch mc -n 10.0.50.31 --patch @patches/mount-sdb.patch - At this point the Talos node will reboot and should come back up healthy in a minute.
  • View the console or check the dashboard with talosctl dashbord -n 10.0.50.31
  • If you see an error about being unable to mount the disk or the partition being the wrong type, etc. you will need to wipe the disk and create a fresh GPT partition. As of Talos 1.9.0 this can be done with talosctl wipe disk sdb -n 10.0.50.31, otherwise you would need to do this outside of Talos.
    • talosctl wipe disk sdb -n 10.0.50.31 - where sdb is the device, you confirmed this right? Confirm using talosctl get disks -n 10.0.50.31
    • Otherwise from Proxmox you could do this: Shut down the VM and do this in proxmox with wipefs /dev/yourdev and then use fdisk /dev/yourdev > g > w (g writes a new GPT table, w writes to disk). Now power Talos back on and it should do its thing.
    • Yet another option would be to boot into a different Linux ISO on the VM and use a tool like Gparted. Whatever you like best.

Let’s Verify Our Disk Mount

When Talos successfully mounts the extra disk, we should see it listed with lsblk without any partitions. We want to pass the raw disk to OpenEBS. In order to check, run a debug pod on your storage node and check the bind mounts.

• kubectl debug node/talos-storage-1 -it --image=alpine -- /bin/sh
• apk add lsblk
• lsblk
• Check for the mount, showing the full capacity of your disk.

Now repeat this whole process for any other Talos nodes you need. I have 3, so I’m doing talos-storage-1, talos-storage-2 and talos-storage-3.

Worker Nodes Also Need /var/local Mounted

Certain OpenEBS components run on any node, and this requires all worker nodes to have /var/local mounted.

# ./patches/mount-var-local.patch
machine:
  kubelet:
    extraMounts:
      - destination: /var/local
        type: bind
        source: /var/local
        options:
          - rbind
          - rshared
          - rw

In my case, I applied this to my 3 worker nodes. I don’t think a reboot is required, but you could if you wanted to:

• talosctl patch mc -n 10.0.50.21 --patch @patches/mount-var-local.patch
• talosctl patch mc -n 10.0.50.22 --patch @patches/mount-var-local.patch
• talosctl patch mc -n 10.0.50.23 --patch @patches/mount-var-local.patch

In order to check the other bind mount for /var/local, we have to wait until after deploying OpenEBS because the mount isn’t utilized until a pod is deployed with a HostPath volume at or below this path. Specifically, the openebs-io-engine-* Daemonset maps to this path.

Installing OpenEBS

This was a pain to figure out. Documentation from OpenEBS is lacking, and so is documentation from Talos on the same topic. Here’s what I found to work. You need a privileged namespace, bind mounts on all worker nodes, then DiskPools before you can start testing PVCs.

Privileged Namespace

OpenEBS requires privileges, and the easiest way to handle that is by making the namespace privileged (rather than messing with machine configs).

• Add a new privileged namespace. The Helm chart wants you to use openebs so do this:

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: openebs
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/audit: privileged

• kubectl apply -f namespace.yaml

Helm Installation

• helm repo add openebs https://openebs.github.io/openebs
• helm repo update
• Grab the values from the Helm chart (helm show values openebs/openebs > values.yaml), or use this. I have already modified the config to disable initContainers which is a known issue with Talos, and disabled local provisioners that I’m not interested in.

# values.yaml
openebs-crds:
  csi:
    volumeSnapshots:
      enabled: true
      keep: true

# Refer to https://github.com/openebs/dynamic-localpv-provisioner/blob/v4.1.2/deploy/helm/charts/values.yaml for complete set of values.
localpv-provisioner:
  rbac:
    create: true

# Refer to https://github.com/openebs/zfs-localpv/blob/v2.6.2/deploy/helm/charts/values.yaml for complete set of values.
zfs-localpv:
  crds:
    zfsLocalPv:
      enabled: true
    csi:
      volumeSnapshots:
        enabled: false

# Refer to https://github.com/openebs/lvm-localpv/blob/lvm-localpv-1.6.2/deploy/helm/charts/values.yaml for complete set of values.
lvm-localpv:
  crds:
    lvmLocalPv:
      enabled: true
    csi:
      volumeSnapshots:
        enabled: false

# Refer to https://github.com/openebs/mayastor-extensions/blob/v2.7.2/chart/values.yaml for complete set of values.
mayastor:
  csi:
    node:
      initContainers:
        enabled: false
  etcd:
    # -- Kubernetes Cluster Domain
    clusterDomain: cluster.local
  localpv-provisioner:
    enabled: false
  crds:
    enabled: false

# -- Configuration options for pre-upgrade helm hook job.
preUpgradeHook:
  image:
    # -- The container image registry URL for the hook job
    registry: docker.io
    # -- The container repository for the hook job
    repo: bitnami/kubectl
    # -- The container image tag for the hook job
    tag: "1.25.15"
    # -- The imagePullPolicy for the container
    pullPolicy: IfNotPresent

engines:
  local:
    lvm:
      enabled: false
    zfs:
      enabled: false
  replicated:
    mayastor:
      enabled: true

• helm install openebs -n openebs openebs/openebs -f values.yaml
• Verify: kubectl get po -n openebs and it should look something like this:

NAME                                          READY   STATUS    RESTARTS      AGE
openebs-agent-core-74d4ddc7c5-hjnxl           2/2     Running   0             9m23s
openebs-agent-ha-node-f9bsb                   1/1     Running   0             9m23s
openebs-agent-ha-node-gjdbt                   1/1     Running   0             9m23s
openebs-agent-ha-node-mwjq9                   1/1     Running   0             9m23s
openebs-agent-ha-node-rfjrw                   1/1     Running   0             93s
openebs-api-rest-757d87d4bd-zd2ms             1/1     Running   0             9m23s
openebs-csi-controller-58c7dfcd5b-6jtcq       6/6     Running   0             9m23s
openebs-csi-node-fmmwg                        2/2     Running   0             9m23s
openebs-csi-node-j95f5                        2/2     Running   2 (60s ago)   93s
openebs-csi-node-jxkvq                        2/2     Running   0             9m23s
openebs-csi-node-xtsnt                        2/2     Running   0             9m23s
openebs-etcd-0                                1/1     Running   0             9m23s
openebs-etcd-1                                1/1     Running   0             9m23s
openebs-etcd-2                                1/1     Running   0             9m23s
openebs-io-engine-lb8zr                       2/2     Running   0             9m23s
openebs-localpv-provisioner-657c44878-wjmwr   1/1     Running   0             9m23s
openebs-loki-0                                1/1     Running   0             9m23s
openebs-nats-0                                3/3     Running   0             9m23s
openebs-nats-1                                3/3     Running   0             9m23s
openebs-nats-2                                3/3     Running   0             9m23s
openebs-obs-callhome-8665bb8f6f-4ntrd         2/2     Running   0             9m23s
openebs-operator-diskpool-6d44884f8f-52rrx    1/1     Running   0             9m23s
openebs-promtail-2g6kz                        1/1     Running   0             9m23s
openebs-promtail-d72cx                        1/1     Running   0             93s
openebs-promtail-hsxlc                        1/1     Running   0             9m23s
openebs-promtail-npw7f                        1/1     Running   0             9m23s

• If pods are stuck initializing after a few minutes, start checking logs with openebs-etcd-0 since a lot of things depend on that being up before it will initialize.
  • Related to /var/local bind mounts not being added to worker nodes, openebs-etcd-* will have Warning events about “FailedMount”, stating that a PVC doesn’t exist. If you look closely, the path starts with /var/local/... which means you’re missing the /var/local bind mount on one or more Talos worker/storage nodes.
  • The fix in this situation would be to fix the bind mount on the worker nodes, and you could also try a reboot (you can identify the node based on the pod having issues). There’s no need to change the OpenEBS deployment.
  • After fixing it, you might see stale pods with errors. You can terminate pods in an error state, and if they are needed the Deployment or Daemonset responsible for them will recreate them as needed.

Add DiskPool(s)

I was excited at this point to test with a PVC, but then was confused about why it wouldn’t provision anything. The Talos docs feel a bit sparse and seem to imply that now is the time to test with a PVC. But if you pay close attention they only mention testing the local provisioner, adding to my confusion. It turns out you need to add DiskPools, which makes sense in hindsight. If you’ve ever used Longhorn there is a similar config needed after the initial install, so that you know what disk capacity you have to work with.

• Earlier, we mounted that 2TB disk in the talos-storage-1 node. Now we’ll use that for our first DiskPool
• Get the disk ID by exec-ing into the openebs-io-engine pod
  • Identify one of the io-engine pods: kubectl get po -l app=io-engine -n openebs
  • Exec into the pod: kubectl exec -it openebs-io-engine-jpnrh -c io-engine -n openebs -- bin/sh
  • ls -lh /dev/disk/by-id/ - grab the one pointing to /dev/sdb in our case, which for me is scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
• FYI I went with uring instead of aio since it’s the new kid on the block

# diskpool-1.yaml
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: pool-1
  namespace: openebs
spec:
  node: talos-storage-1
  disks: ["uring:///dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1"]

• kubectl apply -f diskpool-1.yaml
• Verify: kubectl get dsp -n openebs - quickly it should change to Created state and Online POOL_STATUS

Repeat this process for any/all storage nodes you have. Since I’m virtualizing Talos, the disk path is exactly the same on all 3 nodes so I can reuse the config, just updating the pool name and the node name.

Troubleshooting

Sorry I can’t be a ton of help here since I have only done really limited troubleshooting. If you run into issues with diskpools stuck in Creating status, you will need to describe the dsp or check logs.

What I can say is that I’m running a homelab, which means I have 3 different disks, from 3 different manufacturers. Two of them worked in this configuration, but one did not. The disk is fine, it’s fairly new, I’ve tried wiping it multiple times, multiple ways, but it just would not work with the diskpool. I tried doing it as a bind mount and using /mnt/local/nvme2tb (this one requires adding the volume to the io-engine Daemonset), mounting /dev/sdb, mounting /dev/sdb1, everything I could think of, but it would not create. I rebuilt the Talos node but got the same results. I switched to a different disk and changed nothing else, and it works totally fine. For prosperity, these are the disks I have and which ones worked in this configuration.

• Samsung 970 EVO Plus 2TB - no problems
• Samsung 990 EVO 2TB - no problems
• WD BLACK SN770 2TB - no problems
• Crucial P3 Plus 2TB (CT2000P3PSSD8) - COULD NOT GET THIS WORKING :(

If you are reading this and you know or think you might know why this didn’t work, please reach out! I’m interested in understanding why this wouldn’t work and how I could troubleshoot better.

Testing A Replicated PVC

If you are here, you have at least one working diskpool and are ready to test that PVC provisioning works and can be attached to a running pod. Let’s test that.

• Verify diskpools: kubectl get dsp -n openebs - for my 3 storage nodes with 2TB volumes, I’m seeing this

NAME     NODE              STATE     POOL_STATUS   CAPACITY        USED   AVAILABLE
pool-1   talos-storage-1   Created   Online        1998443249664   0      1998443249664
pool-2   talos-storage-2   Created   Online        1998443249664   0      1998443249664
pool-3   talos-storage-3   Created   Online        1998443249664   0      1998443249664

• Check what storage classes are available: kubectl get sc

NAME                     PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
mayastor-etcd-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
mayastor-loki-localpv    openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
openebs-hostpath         openebs.io/local          Delete          WaitForFirstConsumer   false                  6h15m
openebs-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   6h15m

• For now, we’re interested in testing that openebs-single-replica SC that uses Mayastor, so write this file. Note that this uses the default namespace:

# test-pvc-openebs-single-replica.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openebs-testpvc
spec:
  storageClassName: openebs-single-replica
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

• Apply it: kubectl apply -f test-pvc-openebs-single-replica.yaml
• Check PV and PVC:
  • kubectl get pv
  • kubectl get pvc - you should see a PVC named openebs-testpvc with status Bound and storage class openebs-single-replica
• Deploy a test pod to attach the PVC to - this pod is also in the default namespace:

# pod-using-testpvc.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: testlogger
spec:
    containers:
    - name: testlogger
      image: alpine:3.20
      command: ["/bin/ash"]
      args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
      volumeMounts:
      - name: testvol
        mountPath: /mnt
    volumes:
    - name: testvol
      persistentVolumeClaim:
        claimName: openebs-testpvc

• kubectl apply -f pod-using-testpvc.yaml
• Verify it’s running: kubectl get po testlogger
• Exec into the test pod: kubectl exec -it testlogger -- /bin/sh
• Look at your mounts with df -h /mnt. Since OpenEBS Mayastor uses NVMe-oF you should see that mount path /mnt attached to what looks like a NVMe block device.

Filesystem                Size      Used Available Use% Mounted on
/dev/nvme0n1              9.7G     28.0K      9.2G   0% /mnt

• Hmm. The size doesn’t quite add up. 9.7G is close to 10Gi but then 28.0K used does not equal 9.2G available. Hmm…
• Cleanup:
  • kubectl delete -f pod-using-testpvc.yaml
  • kubectl delete -f test-pvc-openebs-single-replica.yaml

What Is A “Single” Replica Anyway?

A replica is an exact reproduction of something…

A replica by definition cannot exist without copying something that already exists, which inherently means there must be at least 2. In the context of OpenEBS, a single replica just means you only have ONE copy of the data. It gets randomly assigned to one of the diskpools available. If that disk fails, that data is gone. But we are using OpenEBS for the purpose of replication, so how do we get more replicas??? Follow me.

• Create a new storage class with 2 replicas (feel free to do 3 or any value at this point):

# openebs-2-replicas-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-2-replicas
parameters:
  protocol: nvmf
  repl: "2"
provisioner: io.openebs.csi-mayastor

• kubectl apply -f openebs-2-replicas-sc.yaml
• Verify: kubectl get sc
• Test - deploy another test-pvc using the new SC:

# test-pvc-openebs-2-replicas.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: openebs-testpvc
spec:
  storageClassName: openebs-2-replicas
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

• kubectl apply -f openebs-2-replicas.yaml
• Test with a pod, using the same pod from earlier - kubectl apply -f pod-using-testpvc.yaml
• Exec into the test pod: kubectl exec -it testlogger -- /bin/sh
• Check your mounted disk: df -h /mnt

Conclusion

I’m going to stop here. I didn’t go into any details about how to check which diskpools hold the replica(s) for the PV, but I assume there is a way to do that. I also did not look at how to recover in case there is a diskpool or storage node failure. Assuming you had 3 replicas, and one went down, there would be no data loss.

I didn’t cover performance, monitoring, recovery, or anything else that you probably care about long term. That could be a future post, but my next stop is actually evaluating Longhorn with the v2 engine. As of today, they have released 1.8.0-rc5, which enables support for their v2 engine with Talos (this just means they support NVMe-oF). If Longhorn can now work with NVMe-oF and Talos, to me that is a much more mature and feature rich product, with more community support than OpenEBS. I believe it also supports snapshots and other features that OpenEBS does not currently.

My next post will be all about blowing this setup away and doing it all over with Longhorn. Hopefully by then the stable 1.8.0 will have been released.

https://www.talos.dev/v1.9/kubernetes-guides/upgrading-kubernetes/

Upgrading Kubernetes is non-disruptive to the cluster workloads.

You can do this live, assuming you don’t have single-replica workloads that are node-specific.

Today I will be upgrading to Kubernetes version v1.31.5. I’m currently on v1.30.0 but I want to make sure I’m running the same version that is being tested on the CKA exam that I’m studying for which is currently 1.31.

Talos recommends using the talosctl upgrade-k8s command which automatically upgrades the entire cluster and has built in safety checks. They explain how to do it manually, but I chose Talos Linux partly based on the ease of ongoing maintenance and upgrades so I will be using the easy button here!

• Check current version: kubectl get node
• Upgrade: talosctl -n 10.0.50.11 upgrade-k8s --to 1.31.5
  • You only need to specify a single control plane node, but this will upgrade the whole cluster
  • You need to choose a real Kubernetes version - https://kubernetes.io/releases/
  • This will take a while, so try to be patient.
• Verify version: kubectl get node

I like to update my local talosconfig repo which was used to deploy the original Talos cluster and also includes secrets used to recover in case of any problems. This is a good time to update the Kubernetes version in controlplane.yaml and worker.yaml for any new nodes you deploy.

Upgrade Talos OS

https://www.talos.dev/v1.9/talos-guides/upgrading-talos/

The Talos team recommends using the same version of talosctl that your nodes are running. You will then upgrade talosctl after the node upgrades are complete.

Be sure to upgrade one node at a time and check that it’s healthy before moving on. You can blast through them using a for loop, or do them by hand. Just don’t do them all at the same time :)

• Check versions:
  • Client: talosctl version --client
  • Server: kubectl get node -o wide (OS-IMAGE column)
• Get a new Talos OS image from the factory: https://factory.talos.dev
  • Make sure to add any existing extensions you’re using such is iscsi-tools
  • Copy the image string under the “Upgrading Talos Image” header. In my case this looks like factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2
• Upgrade one node: talosctl upgrade -n 10.0.50.11 --image factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2 --preserve
  • -n: Specify the node to upgrade
  • --image: Specify the factory image to use
  • --preserve: Don’t wipe extraMounts if applicable. I default to using this unless I have a specific reason to wipe additional mounts.
• Repeat the upgrade command for each node, one at a time, until all nodes have been upgraded.

In my homelab, I am comfortable blasting through upgrades with a for loop, so my upgrade command looks like this:

for node in 11 12 13 21 22 23 31 32 33; do talosctl upgrade -n 10.0.50.$node --image factory.talos.dev/installer/88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b:v1.9.2 --preserve; done

Once Nodes have been upgraded, upgrade the talosctl client so the version matches the Talos node version.

• rm /usr/local/bin/talosctl
• curl -sL https://talos.dev/install | sh (this gets the latest version)
  • You can also download a specific release from https://github.com/siderolabs/talos/releases, e.g. curl -LJO https://github.com/siderolabs/talos/releases/download/v1.8.3/talosctl-linux-amd64
• Verify: talosctl version --client

I like to update my local talosconfig repo which was used to deploy the original Talos cluster and also includes secrets used to recover in case of any problems. This is a good time to update the factory image in controlplane.yaml and worker.yaml for any new nodes you deploy.

That’s it!

External services are anything you want to route traffic to that does not live in your Kubernetes cluster. For example, you might be running MinIO in a VM and accessing it via IP address, but you want to basically reverse proxy to it. You can use Kubernetes Ingress to act as a reverse proxy to pretty much anything, even if it doesn’t live within your cluster.

It’s pretty straightforward to do this, and you just need to create a few resources: Service, EndpointSlice and Ingress. For the sake of organization, I like to use a separate namespace to separate any external service resources. If you’re doing this, just create a new namespace as your first step.

MinIO Example

I have MinIO running outside my Kubernetes cluster. For now, I just want basically a reverse proxy in front of it, but I want to use Kubernetes Ingress since I already have that running and configured to get TLS certificates from Let’s Encrypt.

• Optionally, create a new Namespace. I’m using “externalservices”

  apiVersion: v1
  kind: Namespace
  metadata:
    name: externalservices
  

• Ceate an EndpointSlice. This object is the newer replacement for Endpoints which are now managed by EndpointSlice, so it’s no longer recommended to manually create Endpoints directly. EndpointSlice requires a name and a label that matches the service name you will be using. The label’s key is kubernetes.io/service-name. namespace is optional but recommended. Then you just specify address type (IPv4 or IPv6), ports, and the actual endpoint with the IP address. Here’s an example of my EndpointSlice manifest where MinIO listening at 192.168.1.35 on port 80. I’m using a namespace called “externalservices” and the service name is minio-service.

  apiVersion: discovery.k8s.io/v1
  kind: EndpointSlice
  metadata:
    name: minio-service
    namespace: externalservices
    labels:
      kubernetes.io/service-name: minio-service
  addressType: IPv4
  ports:
    - port: 80
  endpoints:
    - addresses:
      - "192.168.1.35"
      conditions:
        ready: true
  

• Next, create a ClusterIP Service without a selector. Having no selectors allows the service to be attached directly to an Endpoint that you create instead of only being able to attach to a pod. The way it knows how to attach to the EndpointSlice is based on the label you used in the EndpointSlice manifest which has to exactly match the name you use in this service (in this case minio-service). This service just maps port 80 to targetPort 80, and we will handle HTTP to HTTPS redirection, plus TLS termination in the Ingress resource.

  apiVersion: v1
  kind: Service
  metadata:
    name: minio-service
    namespace: externalservices
  spec:
    type: ClusterIP
    ports:
      - port: 80
        protocol: TCP
        targetPort: 80
  

• Finally, create an Ingress. Here’s a basic example using a subdomain, using a Let’s Encrypt ClusterIssuer, and using traefik as the ingress class.

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: minio-service-ingress
    namespace: externalservices
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-staging"
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
  spec:
    ingressClassName: traefik
    tls:
    - hosts:
      - minio.example.com
      secretName: tls-example-com
    rules:
    - host: minio.example.com
      http:
        paths:
        - backend:
            service:
              name: minio-service
              port:
                number: 80
          path: /
          pathType: Prefix
  

TLDR; Put it all together into a single manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: externalservices
---
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: minio-service
  namespace: externalservices
  labels:
    kubernetes.io/service-name: minio-service
addressType: IPv4
ports:
  - port: 80
endpoints:
  - addresses:
    - "192.168.1.35"
    conditions:
      ready: true
---
apiVersion: v1
kind: Service
metadata:
  name: minio-service
  namespace: externalservices
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-service-ingress
  namespace: externalservices
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - minio.example.com
    secretName: tls-example-com
  rules:
  - host: minio.example.com
    http:
      paths:
      - backend:
          service:
            name: minio-service
            port:
              number: 80
        path: /
        pathType: Prefix

• Apply kubectl apply -f minio-external-service.yaml
• Shortly after deploying the resources, update DNS or your local hosts file and point to minio.example.com. It may take a bit to generate a signed certificate from the issuer, but you should still be able to route to your endpoint through the Ingress and verify that it’s working.

That’s it. That’s the end. You are now done.

It’s easy to gloss over backups especially when you are setting up a new cluster, finally getting stuff running on it, and feeling excited that you can now deploy some service and basically automate certificates, ingress, and everything else that used to be a pain to do manually. But as soon as something fails, you will wish you spent a little more time not only thinking about backups, but also documenting step by step how to recover from a disaster.

In Kubernetes, we have two main things to back up: persistent data, and the state of the cluster resources.

With Velero, backups in Kubernetes are EASY. Just do it now, before you put stuff on the cluster, and write down the steps to recover. Why not?

etcd Backups In Talos Linux

Before we dive into Velero, I want to mention that theoretically, you could just use Velero for backups and never worry about etcd. However, it’s a best practice to backup etcd and it’s easy to do, there’s no reason not to have another recovery point.

What is etcd (I think it’s pronounced “et-see-dee”)? This is the database that the control plane nodes use to keep track of any and all resources in the cluster. It’s the cluster database that keeps track of every resource, status, etc. in the cluster. This includes every node, pod, deployment, secret, etc. If you had a catastrophe and lost all your cluster resource state, or had something get corrupted, you should be able to restore the etcd database and get back to the exact state it was in when the backup was taken.

Side note: This tracks real time data so whatever storage the etcd cluster is running on (the actual disks the control plane node stores this data on) needs to be pretty fast. If it’s slow, you can run into issues where kubectl commands are very slow, or stuff stops working as expected in the cluster. Maybe this note belongs in the Talos Linux setup section, but here we are.

Talos Linux, unlike a traditional kubeadm install, does not run etcd as a static pod. Instead, it’s hidden behind the Talos API, meaning you have to use talosctl to manage it: https://www.talos.dev/v1.8/advanced/etcd-maintenance/

How To Back Up etcd In Talos

https://www.talos.dev/v1.8/advanced/disaster-recovery/#backup

• talosctl -n <IP> etcd snapshot /backup_path/etcd.snapshot.$(/bin/date +%Y%m%d)
  • You need to pass any one of the control plane node IPs to the -n flag

You could automate this somewhere so you have regular backups. Highly recommended! If you do a bash script, you’ll need to pass the --talosconfig /path/to/talosconfig option.

If you’re currently in a Disaster Recovery scenario and the snapshot command is failing, copy the etcd database directly: https://www.talos.dev/v1.8/advanced/disaster-recovery/#disaster-database-snapshot

• talosctl -n <IP> cp /var/lib/etcd/member/snap/db .

How To Recover etcd In Talos

https://www.talos.dev/v1.8/advanced/disaster-recovery/#recovery

I haven’t done this process yet (and hopefully I don’t need it). Follow the official guide.

What’s Velero?

https://velero.io/

Velero is an open source DR and backup/migration/recovery tool for Kubernetes. It backs up all cluster resources (or whatever you specify) and can be used for backup, restore, disaster recovery and migrating to new clusters. It’s also capable of backing up PVs. Backups are stored on external endpoints (mostly S3 capable services including MinIO). Backup/restore is managed through CRDs. There are two components: cluster resources and a client side utility velero.

Don’t trust their blog to be up to date because the current release according to the blog is 1.11, released April 26, 2023, although I’m currently running 1.15 which was released November 5, 2024.

Get an accurate changelog right from the source: https://github.com/vmware-tanzu/velero/releases

Velero Installation/Setup

I’ll walk through installing Velero with some options you might want to consider. Then I’ll dive into running a backup and restore manually, scheduling backups, and backing up persistent volumes with Kopia.

Prerequisites

• Access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled. For more information on supported Kubernetes versions, see the Velero compatibility matrix.
• kubectl installed locally
• Object storage. I’m running MinIO in my lab. You could use MinIO, Ceph, AWS S3, Backblaze B2 (S3 API), or any other object storage you prefer.

Installing Velero - Client Side

https://velero.io/docs/v1.15/basic-install/

• Install the velero client utility

  VERSION=v1.15.0
  curl -LJO https://github.com/vmware-tanzu/velero/releases/download/$VERSION/velero-$VERSION-linux-amd64.tar.gz
  tar zxvf velero-$VERSION-linux-amd64.tar.gz
  install -m 755 velero-$VERSION-linux-amd64/velero /usr/local/bin/velero
  rm -rf velero-$VERSION-linux-amd64
  rm -rf velero-$VERSION-linux-amd64.tar.gz
  velero version
  

Installing Velero - Server Side

There are two installation methods, using the Helm chart or using the velero utility. Let’s use velero CLI since the Helm method seems complicated based on what I can find in the docs.

MinIO Setup

We need to start with an access key for the S3 backup target. I’m using MinIO: https://velero.io/docs/v1.15/contributions/minio/

• Create a bucket velero-talos
• Create a group velero-svc
• Create a service account user velero and add to velero-svc group
• Create a policy velero-rw. Raw Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::velero-talos",
                "arn:aws:s3:::velero-talos/*"
            ]
        }
    ]
}

• Assign that policy to the velero-svc group
• Create an access key on the velero user
• Create a file named minio-access-key.txt, replacing the values from your access key

[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

• Decide what you think is best for storing this file or the credentials somewhere securely. I would suggest using SOPS, but I will leave the implementation up to you.

Install Using velero Utility

• Arguments reference:
  • --provider - Use the AWS provider which is also used for any generic S3 object storage, including MinIO
  • --secret-file - Specifies the secrets to use for authentication against the MinIO storage
  • --plugins - Required for Velero to connect to an S3 backend
  • --bucket - specifies the name of the bucket on the S3 backend
  • --use-volume-snapshots=true - Enables volume snapshots
  • --backup-location-config - Configures the connection URL and options for the S3 backend
  • --features=EnableCSI - Enables Velero to use a built in CSI snapshot driver, such as democratic-csi and take snapshots using a volumesnapshotclass. To be clear, Velero would create a volumesnapshot and it would be stored wherever democratic-csi stores snapshots (in our case that’s in another dataset on the TrueNAS instance). This is the easy button if you already have CSI snapshots enabled, but the alternative is using Kopia and storing snapshots on the MinIO backend.
• Install:
  • velero install --provider aws --secret-file minio-access-key.txt --plugins velero/velero-plugin-for-aws:v1.11.0 --bucket velero --use-volume-snapshots=true --backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=http://192.168.1.35:9000 --features=EnableCSI
    • You may need to update the plugin version and s3URL.
    • Optionally, set use-volume-snapshots=false if you don’t want to back up PVs.
• Verify:
  • kubectl get all -n velero
  • kubectl -n velero get pod -l deploy=velero
  • Check logs, in particular to verify that Velero can reach the backup location
    • kubectl -n velero logs -l deploy=velero

    level=info msg="BackupStorageLocations is valid, marking as available"
    

• Post install:
  • If you are using democratic-csi for snapshots, you also need to add a label on the VolumeSnapshotClass to let Velero know which one to use by default. This must only be set on 1 volumeSnapshotClass
    • kubectl edit volumesnapshotclass truenas-iscsi

      velero.io/csi-volumesnapshot-class: "true"
    

  • If you’re ADDING this funtionality after you’ve already installed Velero, you just need to modify the server deployment and also enable on the client: velero.io/csi-volumesnapshot-class: “true”
    • Server:
      • kubectl edit -n velero deploy/velero

      --features=EnableCSI
      

    • Client:
      • velero client config set features=EnableCSI
      • velero client config get features

Testing

I’ll go through a specific example that should work to follow along if you’ve followed the series up to now. Otherwise, there are some good examples in the docs that you can reference: https://velero.io/docs/v1.15/examples/

Manual Backup/Restore

Let’s back up the traefik namespace:

• velero backup create traefik-backup --include-namespaces traefik
• Verify: velero backup describe traefik-backup
  • Looking for Phase: Complete
• Test a DR scenario:
  • Blow away the traefik namespace: kubectl delete ns traefik
  • Restore traefik-backup with Velero: velero restore create --from-backup traefik-backup
  • Verify: kubectl get ns or velero restore describe traefik-backup-20241130181633 (get the restore name from the previous restore command)

Create a full backup: velero backup create full-20241130 - If you don’t specify namespaces, etc. then everything gets backed up.

Create A Backup Schedule

I like to do daily full backups at 7am. TTL is the expiration time for each backup created, 720h is 30 days. So I get daily backups that fall off after 30 days.

• Create schedule: velero schedule create daily-full --schedule "0 7 * * *" --ttl 720h
• Verify: velero get schedules

Alternatively, you can use a manifest to define your backup schedule. You may not want to include certain namespaces in your daily backups since most likely you won’t need/want to restore namespaces like kube-system, kube-public, and kube-node-lease. Let’s look at defining a backup schedule using a manifest:

• Create the Schedule manifest, schedule.yaml:

  apiVersion: velero.io/v1
  kind: Schedule
  metadata:
    name: daily-skipping-system-namespaces
  spec:
    schedule: "0 12 * * *"
    skipImmediately: false
    template:
      excludedNamespaces:
        - kube-public
        - kube-system
        - kube-node-lease
  

• Apply the schedule: kubectl apply -f schedule.yaml
• Verify: velero get schedules

Restore From Scheduled Backup

• velero restore create --from-schedule SCHEDULE_NAME

Restores In General

See previous examples for specific restore commands. The basic syntax will be velero restore create [RESTORE_NAME] [--from-backup BACKUP_NAME | --from-schedule SCHEDULE_NAME] [flags]

You can restore from a manually created backup or a scheduled backup. You can also restore specific items from a backup, so let’s say you just want to restore your secret key to your sealed secrets from a daily-full scheduled backup, you can use flags to specify exactly what from that backup you want to restore. For example:

• velero restore create restore-sealed-secret-key --from-schedule daily-full --selector sealedsecrets.bitnami.com/sealed-secrets-key=active

Back Up PVCs Using democratic-csi Snapshots

If you enabled snapshots in democratic-csi and also enabled snapshos in Velero as described above, then anything you back up with Velero that includes a PVC will be snapshotted. I tested this by deploying a pod/deployment in the default namespace, attached to a test PVC, then ran a Velero backup against it.

• velero backup create full-backup

Kopia

Velero uses Kopia (as a plugin) to copy existing snapshots to your S3 backend. This means that if you’re not already creating “regular” snapshots already outside of Velero, you need to have EnableCSI enabled and working already so that a VolumeSnapshot is created, which can then be copied out to S3. In a homelab environment, this may be redundant if your MinIO storage is backed by the same storage you’re using for snapshots, but for the sake of completeness I will run through how we can make this work and test it.

There are several questions to answer when deciding how to handle storage for Kubernetes.

• How many disks do you have?
• Do you want/need replicated storage?
• What are your storage capacity requirements?
• What are your performance requirements?
• Do you need dynamically provisioned storage or will you be doing it manually?
• NFS or iSCSI or NVMe-oF?
• How will you back up your data?
• Do you need snapshots?
• Does your storage need to be highly available?
• Does it need to be accessible from any node in the cluster, or are you good with node local storage?

Ultimately it’s not actually hard, it’s just complex if you want to achieve anything like you get with EBS volumes in AWS, but in your homelab. Here’s how I always try to approach complex problems: start as simple as possible, make it work, then add complexity only as needed.

My Requirements

• To have dynamically provisioned persistent volumes
• To have that persistent volume be accessible from any node in my cluster
• Keep it as simple as reasonably possible

How Am I Doing It?

Currently I have a single 2TB NVMe disk and I want to use that for dynamically provisioned storage. I’m not worried about replication right now since I have backups in place and this is just for my homelab. If I wanted to do replicated storage, I might consider a Ceph cluster, but that realistically requires a decent amount of hardware and fast networking interconnectivity (greater than 1GB, ideally 10GB minimum for replication to keep up).

In order to manage the disk I’m using TrueNAS Scale, which is basically ZFS on Linux with a nice web GUI to manage things. This actually provides the option of doing a zpool with maybe 2 disks mirrored, or even a RAIDZ6 as your storage target to easily solve for replication across disks.

In Proxmox, I’m passing the disk itself directly to TrueNAS VM. You should pass an entire disk controller if you need to pool multiple drives together in ZFS. If you’re dealing with a single disk, it’s OK to do it this way.

Spin up the VM, format the disk, ZFS, etc. and you’re ready to go. In my case, this is on the Kubernetes VLAN and assigned a static IP. Originally I had a second NIC attached to the primary VLAN but this caused some weird web UI performance issues so I removed that one.

TrueNAS Setup In Proxmox

This isn’t intended to be a TrueNAS tutorial, so I’ll just list the steps at a high level. Basically, get a TrueNAS server running and then proceed.

• Pass a disk (or HBA controller) to a VM in Proxmox
• Install TrueNAS Scale
• Create a zpool
• Generate an API Key - in the top right corner go to Admin > API Keys
• Make sure the network is accessible from your Kubernetes cluster

Install democratic-csi With TrueNAS

https://github.com/democratic-csi/democratic-csi

This is a straightforward CSI provider that focuses on dynamically provisioned storage from TrueNAS or generic ZFS on Linux backends. Protocols include NFS, iSCSI, and NVMe-oF. I’ll show you how to use the API variation and do NFS and iSCSI shares, plus talk about almost getting NVMe-oF working.

You will install a separate Helm chart for each provisioner, and you can actually run multiple at the same time, which is what I will be doing with both NFS and iSCSI. This is helpful since NFS even supports RWX volumes (if you actually have a use case for that), while iSCSI is a good default for RWO volumes.

VolumeSnapshot Support

This is optional, but if you want to utilize volume snapshots (which became GA as of Kubernetes 1.20), you will need to install the CRDs which aren’t included with vanilla Talos, along with installing the “snapshotter.” This implements Volume Snapshots - https://kubernetes.io/docs/concepts/storage/volume-snapshots/

• Clone repo: git clone https://github.com/kubernetes-csi/external-snapshotter.git
• cd external-snapshotter
• Apply CRDs: kubectl kustomize client/config/crd | kubectl create -f -
• Install snapshotter into kube-system: kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -
• Verify: kubectl get deploy snapshot-controller -n kube-system

Dynamic iSCSI Provisioner With freenas-api-iscsi

My single 2TB disk is in a pool named nvme2tb. I created a dataset in TrueNAS named iscsi. Those may vary in your case, so pay attention to the configuration and update those values according to your environment.

Don’t forget, your Talos installation needs to include the iscsi extension or the nodes won’t be able to connect to TrueNAS.

• Create a dataset named iscsi
• Make sure Block (iSCSI) Shares Targets is running, and click Configure
• Save the defaults for Target Global Configuration
• Add a portal on 0.0.0.0:3260 named k8s-democratic-csi
• Add an Initiator Group, Allow all initiators, and name it something like k8s-talos
• Create a Target named donotdelete and alias donotdelete, then add iSCSI group selecting the Portal and Initiator Group you just created. This prevents TrueNAS from deleting the Initiator Group if you’re testing and you delete the one and only PV.
• Make note of the portal ID and the Initiator Group ID and update these values in the file freenas-api-iscsi.yaml if needed
  • During testing, the manually created Initiator Group was getting deleted whenever deleting the last PV. This appears to be a bug in TrueNAS somewhere according to https://github.com/democratic-csi/democratic-csi/issues/412. Essentially TrueNAS deletes the Initiator Group automatically if an associated Target is deleted and no others exist. If you followed the instructions and created a manual Target this won’t be an issue :)
• Create the democratic-csi namespace: kubectl create ns democratic-csi
• Make that namespace privileged: kubectl label --overwrite namespace democratic-csi pod-security.kubernetes.io/enforce=privileged
• Create freenas-api-iscsi.yaml and update apiKey, host, targetPortal, datasetParentName, and detachedSnapshotsDatasetParentName. Other common considerations in storage class YAML config are storageClasses.defaultClass (true/false, there can only be one), and storageClasses.reclaimPolicy (Delete/Retain) and if you set this to Retain, there is less chance that data will be deleted if you delete a pod (for example), but you are also responsible for deleting this manually in TrueNAS if you ever need to.

  driver:
    config:
      driver: freenas-api-iscsi
      httpConnection:
        protocol: https
        apiKey: [your-truenas-api-key-here]
        host: 10.0.50.99
        port: 443
        allowInsecure: true
      zfs:
        datasetParentName: nvme2tb/iscsi/volumes
        detachedSnapshotsDatasetParentName: nvme2tb/iscsi/snapshots
        zvolCompression:
        zvolDedup:
        zvolEnableReservation: false
        zvolBlockSize:
      iscsi:
        targetPortal: "10.0.50.99:3260"
        targetPortals: []
        interface:
        namePrefix: csi-
        nameSuffix: "-talos"
        targetGroups:
          - targetGroupPortalGroup: 1
            targetGroupInitiatorGroup: 5
            targetGroupAuthType: None
            targetGroupAuthGroup:
        extentInsecureTpc: true
        extentXenCompat: false
        extentDisablePhysicalBlocksize: true
        extentBlocksize: 512
        extentRpm: "SSD"
        extentAvailThreshold: 0
  
  csiDriver:
    # should be globally unique for a given cluster
    name: "org.democratic-csi.freenas-api-iscsi"
  
  storageClasses:
    - name: truenas-iscsi
      defaultClass: true
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: ext4
        detachedVolumesFromSnapshots: "false"
      mountOptions: []
      secrets:
        provisioner-secret:
        controller-publish-secret:
        node-stage-secret:
        node-publish-secret:
        controller-expand-secret:
  
  volumeSnapshotClasses:
    - name: truenas-iscsi
      parameters:
        detachedSnapshots: "true"
  
  node:
    hostPID: true
    driver:
      extraEnv:
        - name: ISCSIADM_HOST_STRATEGY
          value: nsenter
        - name: ISCSIADM_HOST_PATH
          value: /usr/local/sbin/iscsiadm
      iscsiDirHostPath: /usr/local/etc/iscsi
      iscsiDirHostPathType: ""
  

• Deploy: helm upgrade --install --namespace democratic-csi --values freenas-api-iscsi.yaml truenas-iscsi democratic-csi/democratic-csi
• Verify:
  • You’re looking to see that everything is fully running. It may take a minute to spin up.
  • kubectl get all -n democratic-csi
  • kubectl get storageclasses or kubectl get sc

Test - Deploy A PVC

• Test with a simple PVC, targeting our new truenas-iscsi storage class, test-pvc-truenas-iscsi.yaml:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: testpvc-iscsi
  spec:
    storageClassName: truenas-iscsi
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
  

  • kubectl apply -f test-pvc-truenas-iscsi.yaml
  • Be patient, this can take a minute to provision the new Zvol on TrueNAS (looks something like nvme2tb/iscsi/volumes/pvc-25e70f84-91c7-4e49-a9f1-e324681a3b7d) and get everything mapped in Kubernetes.
  • Check the Persistent Volume itself: kubectl get pv
    • Looking for a new entry here
  • Check the Persistent Volume Claim: kubectl get pvc
    • Looking for status Bound to the newly created PV
  • If you need to investigate, next look at kubectl describe pvc and kubectl describe pv, or go look in the TrueNAS UI to see if a new disk has been created

Test - Deploy A Pod

At this point there should be a PV and PVC, but they are not actually connected to a pod yet. The moment a pod claims a PVC, that’s when the actual node that the pod is running on will mount the iSCSI target, and this is where the iscsi-utils extension comes into play in Talos Linux. Let’s test to make sure we can actually connect to the PVC from a pod.

This test pod uses a small Alpine image and writes to a log file every second. The two lines commented out at the bottom are there in case you want to target a specific node. I would recommend if you’re not sure all your Talos Linux nodes are configured properly for iSCSI to target each of them and verify from every pod. You can delete the pod, but preserve the PVC and if you reconnect to the PVC from another pod, even if it’s running on another node, it should still contain the same data.

• Create pod-using-testpvc-iscsi.yaml:

  apiVersion: v1
  kind: Pod
  metadata:
    name: testlogger-iscsi
  spec:
      containers:
      - name: testlogger-iscsi
        image: alpine:3.20
        command: ["/bin/ash"]
        args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
        volumeMounts:
        - name: testvol
          mountPath: /mnt
      volumes:
      - name: testvol
        persistentVolumeClaim:
          claimName: testpvc-iscsi
  #    nodeSelector:
  #      kubernetes.io/hostname: taloswk1
  

• Deploy: kubectl apply -f pod-using-testpvc-iscsi.yaml
• Verify kubectl get po
  • Check which node it’s on with kubectl get po -o wide or kubectl describe po testlogger-iscsi | grep Node:
• Validate data is being written to the PVC:
  • Exec into the pod: kubectl exec -it testlogger-iscsi -- /bin/sh
  • Look at the file: cat /mnt/test.log
  • Show line count: wc -l /mnt/test.log

Test - Cleanup

• Delete pod: kubectl delete -f pod-using-testpvc-iscsi.yaml
• Delete PVC: kubectl delete -f test-pvc-truenas-iscsi.yaml

Dynamic NFS Provisioner With freenas-api-nfs

This one’s a little simpler than iSCSI since support is built into Talos automatically, and there’s less setup on the TrueNAS side.

• Create a dataset named nfs
• Create the democratic-csi namespace: kubectl create ns democratic-csi
• Make that namespace privileged: kubectl label --overwrite namespace democratic-csi pod-security.kubernetes.io/enforce=privileged
• Create freenas-api-nfs.yaml and update apiKey, host, shareHost, datasetParentName, and detachedSnapshotsDatasetParentName

  driver:
    config:
      driver: freenas-api-nfs
      httpConnection:
        protocol: https
        apiKey: [api-key-goes-here]
        host: 10.0.50.99
        port: 443
        allowInsecure: true
      zfs:
        datasetParentName: nvme2tb/nfs
        detachedSnapshotsDatasetParentName: nvme2tb/nfs/snaps
        datasetEnableQuotas: true
        datasetEnableReservation: false
        datasetPermissionsMode: "0777"
        datasetPermissionsUser: 0
        datasetPermissionsGroup: 0
      nfs:
        shareCommentTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
        shareHost: 10.0.50.99
        shareAlldirs: false
        shareAllowedHosts: []
        shareAllowedNetworks: []
        shareMaprootUser: root
        shareMaprootGroup: root
        shareMapallUser: ""
        shareMapallGroup: ""
  
  csiDriver:
    # should be globally unique for a given cluster
    name: "org.democratic-csi.freenas-api-nfs"
  
  storageClasses:
    - name: truenas-nfs
      defaultClass: false
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: nfs
      mountOptions:
        - noatime
        - nfsvers=4
  
  volumeSnapshotClasses:
    - name: truenas-nfs
  

• Since we are using snapshots, you also need to install a snapshot controller such as https://github.com/democratic-csi/charts/tree/master/stable/snapshot-controller
  • You can skip this step if you disable snapshots in your YAML file
  • helm upgrade --install --namespace kube-system --create-namespace snapshot-controller democratic-csi/snapshot-controller
  • kubectl -n kube-system logs -f -l app=snapshot-controller
• Deploy: helm upgrade --install --namespace democratic-csi --values freenas-api-nfs.yaml truenas-nfs democratic-csi/democratic-csi
• Verify:
  • You’re looking to see that everything is fully running. It may take a minute to spin up.
  • kubectl get all -n democratic-csi
  • kubectl get storageclasses or kubectl get sc

Test - Deploy A PVC

• Test with a simple PVC, targeting our new truenas-nfs storage class, test-pvc-truenas-nfs.yaml:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: testpvc-nfs
  spec:
    storageClassName: truenas-nfs
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
  

  • kubectl apply -f test-pvc-truenas-nfs.yaml
  • Be patient, this can take a minute to provision the new block device on TrueNAS and get everything mapped in Kubernetes.
  • Check the Persistent Volume itself: kubectl get pv
    • Looking for a new entry here
  • Check the Persistent Volume Claim: kubectl get pvc
    • Looking for status Bound to the newly created PV
  • If you need to investigate, next look at kubectl describe pvc and kubectl describe pv, or go look in the TrueNAS UI to see if a new disk has been created

Test - Deploy A Pod

At this point there should be a PV and PVC, but they are not actually connected to a pod yet. The moment a pod claims a PVC, that’s when the actual node that the pod is running on will mount the NFS share.

This test pod uses a small Alpine image and writes to a log file every second. The two lines commented out at the bottom are there in case you want to target a specific node. I would recommend if you’re not sure all your Talos Linux nodes are configured properly for iSCSI to target each of them and verify from every pod. You can delete the pod, but preserve the PVC and if you reconnect to the PVC from another pod, even if it’s running on another node, it should still contain the same data.

• Create pod-using-testpvc-nfs.yaml:

  apiVersion: v1
  kind: Pod
  metadata:
    name: testlogger-nfs
  spec:
      containers:
      - name: testlogger-nfs
        image: alpine:3.20
        command: ["/bin/ash"]
        args: ["-c", "while true; do echo \"$(date) - test log\" >> /mnt/test.log && sleep 1; done"]
        volumeMounts:
        - name: testvol
          mountPath: /mnt
      volumes:
      - name: testvol
        persistentVolumeClaim:
          claimName: testpvc-nfs
  #    nodeSelector:
  #      kubernetes.io/hostname: taloswk1
  

• Deploy: kubectl apply -f pod-using-testpvc-nfs.yaml
• Verify kubectl get po
  • Check which node it’s on with kubectl get po -o wide or kubectl describe po testlogger-nfs | grep Node:
• Validate data is being written to the PVC:
  • Exec into the pod: kubectl exec -it testlogger-nfs -- /bin/sh
  • Look at the file: cat /mnt/test.log
  • Show line count: wc -l /mnt/test.log

Test - Cleanup

• kubectl delete -f pod-using-testpvc-nfs.yaml
• kubectl delete -f test-pvc-truenas-nfs.yaml

Dynamic NVMe-oF Storage For Kubernetes (I was unable to make this work)

I spent some time trying to get this to work. TrueNAS doesn’t currently support NVMe-oF through the interface, but since it’s just a Linux box you can simply (almost simply) install extra packages needed and configure them as root. After doing that, I tested manually by connecting from another Linux machine to validate that I could indeed mount NVMe over TCP using TrueNAS.

From there, I figured out the configuration needed for democratic-csi zfs-generic-nvmeof driver and started testing. I got as far as getting it to provision a new dataset on TrueNAS, create the mount, and create the PV and PVC in the cluster, showing as Bound. However, when I would actually attempt to connect to it from a pod, it would fail. It may have something to do with how democratic-csi does the mount from the node, or otherwise I might have something wrong in my configuration that I can’t figure out.

If I could get this working, I might not even bother running a TrueNAS instance and just run some lightweight Linux server to interface between democratic-csi and the disk(s).

Here’s some extra details on exactly what I tried:

• https://github.com/siderolabs/talos/issues/9255
• https://github.com/democratic-csi/democratic-csi/issues/418

Please help me if you know how to make this work, as I’d much rather be using this than iSCSI :)

Here’s my almost working config for reference:

csiDriver:
  name: "org.democratic-csi.nvmeof"

storageClasses:
  - name: truenas-nvmeof
    defaultClass: false
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    allowVolumeExpansion: true
    parameters:
      fsType: ext4

    mountOptions: []
    secrets:
      provisioner-secret:
      controller-publish-secret:
      node-stage-secret:
      node-publish-secret:
      controller-expand-secret:

volumeSnapshotClasses: []

driver:
  config:
    driver: zfs-generic-nvmeof
    sshConnection:
      host: 10.0.50.99
      port: 22
      username: root
      privateKey: |
        -----BEGIN RSA PRIVATE KEY-----
        REDACTED!
        -----END RSA PRIVATE KEY-----        
    zfs:
      datasetParentName: nvme2tb/k8s/nvmeof
      detachedSnapshotsDatasetParentName: nvme2tb/k8s/nvmeof-snapshots

      zvolCompression:
      zvolDedup:
      zvolEnableReservation: false
      zvolBlocksize: 16K

    nvmeof:
      transports:
        - tcp://0.0.0.0:4420

      namePrefix:
      nameSuffix:

      shareStrategy: "nvmetCli"

      shareStrategyNvmetCli:
        nvmetcliPath: nvmetcli
        configIsImportedFilePath: /var/run/nvmet-config-loaded
        configPath: /etc/nvmet/config.json
        basename: "nqn.2003-01.org.linux-nvme"
        ports:
          - "1"
        subsystem:
          attributes:
            allow_any_host: 1

What About Other Options?

What About Local Storage?

This fails to meet one of my main requirements of being able to mount a persistent volume to any node. The whole point of Kubernetes, at least for me, is the ability to take a node offline with no actual downtime of any services. If you have a pod connected to a PVC on a certain node, but I need to move that pod to another node, I need to reconnect to the existing persistent volume, but this method doesn’t allow for that.

What About proxmox-csi-plugin?

I also looked at the proxmox-csi-plugin, but that has the same problem as with node local storage so doesn’t fit my requirements.

What About Rancher Longhorn?

Longhorn is nice and easy. It uses either NFS or iSCSI. The Talos team recommends against both NFS and iSCSI (although either can be used). I would say to use Longhorn if you need replicated storage in a homelab (like 3 disks) but don’t want to figure out Ceph, etc. It’s straightforward and well supported. As for downsides, I don’t have firsthand experience. I hear it’s great for usability, but some users complain that it’s not reliable.

It’s also a little more complicated to set up specifically with Talos, but Longhorn has specific instructions for installation with Talos so that shouldn’t be a man blocker.

At some point I might try this out, but for now I’m sticking with the simple approach of using TrueNAS Scale with any type of zpool you want, and dynamically provisioning NFS or iSCSI using democratic-csi.

What About Mayastor?

I tried getting this to work because the Talos team recommends it if you don’t want to do a full blown Ceph cluster, etc. I was also super interested in the fact that it uses NVMe-oF which is a newer protocol (basically a modern replacement for iSCSI). I wasn’t able to get it working, but I discovered a little bit about it and decided to keep it simpler for the following reasons:

• I only have a single disk currently, so I don’t need any replicated storage
• It seems to be more resource intensive than democratic-csi and has more components.
• The documentation is kind of hectic. It used to be Mayastor, but now it’s OpenEBS Replicated PV Mayastor or something. When deploying, it’s hard to tell if I’m deploying other OpenEBS stuff I don’t need or what exactly PV type I need. I think you need one type of PV to store data for one of the components even if you are ultimately trying to run the replicated PV type (mayastor) for your primary cluster storage. I don’t know, it was confusing.

My failure to get this working is 100% a skill issue, but going back to my requirements I really don’t need this for my homelab at this point. I may revisit this in the future.

What About Ceph?

I would need enough disks and resources to run Ceph. It’s something I really want to test out and potentially use, although probably way overkill for my homelab. I’m currently running 2 Minisforum MS-01 servers and planning on getting a third to do a true Proxmox cluster and replace my old 2U power hungry server. At that point, I might actually give Ceph a shot (trying both the Proxmox Ceph installation and also Rook/Ceph on top of Kubernetes). This would solve for truly HA storage, plus meet all other requirements I have, assuming I don’t add a requirement for low resource utilization just to run storage.

Ingress is a way to expose your applications running in Kubernetes to something outside the cluster. It’s like a reverse proxy for Kubernetes. It defines how traffic is routed from the point that it hits the cluster, specifically the ingress controller, to the running pod. You might already know about the different Kubernetes service types such as ClusterIP, NodePort and LoadBalancer, but this takes it to another level that can handle TLS termination and advanced routing rules, among other nice features, depending on which Ingress Controller you pick. That determines exactly which features are available.

An ingress controller is the software that contains the actual logic to implement Ingress rules and functionality. These can vary depending on the ingress controller you pick.

The awesome thing about this component is that it deploys a LoadBalancer type service, and if you used MetalLB or kube-vip, that means you get a virtual IP dynamically assigned to one of your ingress controller replicas. That means if the pod goes down that’s running the ingress controller (assuming you have more than 1 replica or use a DaemonSet) then the IP automatically moves to another replica. In other words, between MetalLB and Traefik (or insert your tool of choice for either), you get highly available Ingress to your Kubernetes cluster with no manual effort. Pretty nifty!

What About NGINX?

I feel like I should mention that one of the most widely known ingress controllers is nginx. Before you go down that rabbit hole, there are 2 different ingress controllers based on nginx. One used to be called nginx-ingress which was/is the official open source nginx ingress controller (now it’s kubernetes-ingress: https://github.com/nginxinc/kubernetes-ingress), and maintained by the engineers at NGINX. Hard to go wrong there. The other popular option is ingress-nginx which is the community maintained, NGINX supported project. It’s been a while since I tried either one, but I hear good things about ingress-nginx because there’s a large community behind it and it should be easier to find answers if you have questions. If you need something more production ready (and have your heart set on NGINX), then maybe consider kubernetes-ingress.

Why Traefik?

A valid question you might ask, and I’ve asked myself, is why would I consider using Traefik instead of many people’s go-to NGINX? These are my reasons:

• I settled on Traefik years ago when it was still v1 for things I was running in Docker. It provided the ability to EASILY reverse proxy to my applications by simply setting a few labels in my docker-compose.yml files, and INCLUDING the ability to automatically get certificates from Let’s Encrypt using DNS-01 challenge (offering legitimate certificates even for internal only applications).
• It’s open-source, it’s fast, and it just works (I’ve never had issues or run into bugs personally using Traefik)
• When I first went to Traefik, it had more features than NGINX. Notably it can dynamically add routes/rules without restarting or reloading. Maybe the NGINX controller functions the same way today, but in the Docker compose world that was not the case.
• Native support for HTTP/3 - it seems to be more on the cutting edge, supporting new technologies faster than NGINX. I know - newer isn’t always better, just look at Debian. But for my homelab, newer is almost always better :)
• Middleware - more features readily available such as IP whitelisting, WAF options, etc. that you can easily plug into routes for additional functionality

In Kubernetes it matters a lot less since the Ingress resource is another layer of abstraction, and either way your stuff will get routed. But at some point you may have specific requirements, certain middlewares functionality, performance requirements, etc. and that may drive the decision to go with whatever tool best meets those requirements.

Why Not NGINX Or Kong Or Something Else?

I have no argument against using NGINX or anything else. I just like the Traefik project and it does what I need in my homelab, so I will continue using it for now :)

Installation/Setup

You can install the helm chart with no extra options and it will “just work.” There are also some good options you might want to at least know about. These can also be changed later on, so it’s not a big deal if you change your mind later.

https://doc.traefik.io/traefik/getting-started/install-traefik/#use-the-helm-chart

Vanilla Installation

• For any helm installation, always start by adding the helm repo:
  • helm repo add traefik https://traefik.github.io/charts
  • helm repo update
• Vanilla Installation:
  • helm install traefik traefik/traefik

You can start deploying Ingress resources. But keep reading to understand some of the ingress controller options and why you might consider changing them.

Deployment Vs. DaemonSet

By default the Helm chart creates a Deployment with 1 replica, based on the values.yml file. You could scale this up to 3 replicas if you’re looking for something more highly available, or you could consinder a DaemonSet where there will always be one Traefik instance on each worker node to handle routing. Keep in mind with a DaemonSet that if you ever scale up the number of worker nodes, that will also scale up the number of total Ingress Controller pods.

If you wanted to do something a little more elaborate like only run 3 replicas, but make sure they never run on the same node as each other (3 different physical nodes), you would need to get into some node affinity rules that I won’t explain here because I haven’t done that (yet).

When customizing Helm charts, I tend to prefer downloading the values.yaml file, customizing it, then deploying using the custom values file. That way I can store that in version control and not have to think about messy JSON stuff in my shell commands.

• Download values file: helm show values traefik/traefik > values.yaml

• Customize. Let’s change to a DaemonSet by changing deployment.kind from Deployment to DaemonSet. Capitalization matters here! DaemonSet with a capital S:

  ...
  deployment:
    # -- Enable deployment
    enabled: true
    # -- Deployment or DaemonSet
    kind: DaemonSet
  ...
  

• Let’s say you have 10 worker nodes and you want Traefik to be replicated, but you don’t need 10 instances running. In that case you could do a Deployment and set the number of replicas to 3.

  ...
  deployment:
    # -- Enable deployment
    enabled: true
    # -- Deployment or DaemonSet
    kind: Deployment
    replicas: 3
  ...
  

• Deploy: helm install traefik traefik/traefik -f values.yaml

Namespace

Another consideration when deploying Traefik is whether you want to use a custom namespace. Namespaces are a best practice instead of dumping everything into the default namespace for security and organization. It’s easy enough to split out Traefik, so why not?

• Create a namespace: kubectl create ns traefik
• Deploy: helm install traefik traefik/traefik -f values.yaml -n traefik

OR do it all in a single step by adding --create-namespace

• helm install traefik traefik/traefik -f values.yaml -n traefik --create-namespace

Verify Your Deployment (or DaemonSet)

• Vanilla install: kubectl get deploy

  NAME      READY   UP-TO-DATE    AVAILABLE   AGE
  traefik   1/1     1             1           3m
  

• Namespaced Deployment: kubectl get deploy -n traefik

  NAME      READY   UP-TO-DATE    AVAILABLE   AGE
  traefik   3/3     3             3           14s
  

• Namespaced DaemonSet: kubectl get daemonset -n traefik

  NAME      DESIRED   CURRENT   READY   UP-TO-DATE    AVAILABLE   NODE SELECTOR   AGE
  traefik   3         3         3       3             3           <none>          12s
  

Traefik Dashboard

Traefik has a cool dashboard showing some details about routes, middlewares, services, and other stuff. It’s all read-only, but also nice to look at and sometimes helpful in troubleshooting why a route isn’t working as expected or what exactly it’s routing to. But you have to enable access to it, and consider what type of security you want to have in place for Dashboard access.

They don’t recommend enabling the dashboard in production, but if you do, at least secure it. The easiest way to expose but also keep somewhat secure is using basic authentication. Sounds good enough for me, let’s try it out!

https://github.com/traefik/traefik-helm-chart/blob/master/EXAMPLES.md#publish-and-protect-traefik-dashboard-with-basic-auth

• You can throw this in its own yaml file and apply this specific config to the existing installation. Let’s call this dashboard-basicauth.yaml:
  • You’ll need a DNS record pointing the host in the matchRule to the IP address on Traefik’s LoadBalancer service.
  • Using websecure without specifying TLS options will result in a self-signed certificate and you’ll get a warning in the browser.
  • Changing these things is beyond the scope of this tutorial - this is where I draw the line :)

# Create an IngressRoute for the dashboard
ingressRoute:
  dashboard:
    enabled: true
    # Custom match rule with host domain
    matchRule: Host(`traefik-dashboard.example.com`)
    entryPoints: ["web","websecure"]
    # Add custom middlewares : authentication and redirection
    middlewares:
      - name: traefik-dashboard-auth

# Create the custom middlewares used by the IngressRoute dashboard (can also be created in another way).
# /!\ Yes, you need to replace "changeme" password with a better one. /!\
extraObjects:
  - apiVersion: v1
    kind: Secret
    metadata:
      name: traefik-dashboard-auth-secret
    type: kubernetes.io/basic-auth
    stringData:
      username: admin
      password: changeme

  - apiVersion: traefik.io/v1alpha1
    kind: Middleware
    metadata:
      name: traefik-dashboard-auth
    spec:
      basicAuth:
        secret: traefik-dashboard-auth-secret

• Apply the config: helm upgrade --reuse-values -n traefik -f dashboard-basicauth.yaml traefik traefik/traefik

Note that the ingressRoute you set up in this config will NOT appear in your cluster as an Ingress type resource. So if you run kubectl get ingress -A you won’t find it and can’t verify it that way. It’s an internal ingress route that Traefik uses.

HTTP Redirect To HTTPS

This is another common thing that most people probably do. You can do this on a per-service basis, or configure Traefik to automatically redirect HTTP to HTTPS for everything. In that case, if you happen to have something that you don’t want redirected, you would need to create another entrypoint and use a different port besides 80.

https://doc.traefik.io/traefik/routing/entrypoints/#redirection

I’m confused about the mismatch between the values.yaml file and the documentation. In values.yaml there’s a field called redirectTo that expects an object. I worked out that it requires a port and possibly can take another field named permanent (true/false). However, when I upgraded the helm release with these values it didn’t seem to have any effect. The documentation, on the other hand, talks about a structure that looks like entryPoints.web.http.redirections.entryPoint.[to|scheme|permanent|priority]. Since that’s how it’s configured everywhere else, why not use the same structure in values.yaml???

We can (and should?) ignore redirectTo and just go right to additionalArguments instead, using the familiar syntax in the docs:

additionalArguments:
  - "--entrypoints.web.http.redirections.entryPoint.to=:443"
  - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
  - "--entrypoints.web.http.redirections.entryPoint.permanent=true"

Note the fact that I used :443 to redirect to, instead of the entryPoint name since using websecure here will actually cause it to redirect to port 8443. I’m not sure if that’s a bug or not in the Helm chart, but this still works and makes sense.

And in case you’re not familiar, “permanent” just means the difference between a normal redirect (HTTP 302) and permanent redirect (HTTP 301) which determines which HTTP response is used.

Logging, externalTrafficPolicy and IPAllowList Middleware

Note: This is an update as of 3/10/2025

If you want access logs enabled which are not by default, add this section to your values.yaml file:

logs:
  access:
    enabled: true

This makes it so that you can view access logs from the pod or deployment using something like kubectl -n traefik logs deployments/traefik

After enabling this, I realized all requests had the same source IP of 10.244.1.0. I was trying to use the IPAllowList middleware for a /admin endpoint on a service but it wasn’t working due to Traefik being unable to see the true source IP. After a bit of research, I found that I could change the externalTrafficPolicy from Cluster to Local. Read https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip for more info.

There are pros and cons to this approach. If you need to preserve the source IP, I found this to be the best option. The down side is that traffic can only come in on a node where one of the Traefik pods lives. So earlier when you decided whether to run 2 replicas, a daemonset, etc. that comes into play here. But if you run 2-3 replicas or use a daemonset, this is not a problem.

Change externalTrafficPolicy to Local:

service:
  spec:
    externalTrafficPolicy: Local

I read that it might be possible to use X-Forwarded-For, I didn’t dive very far into X-Forwarded-For settings or depth on the middleware. It might be possible to leave this policy set to Cluster and use the header info instead. However, you will need to consider security as you might not necessarily trust the X-Forwarded-For header from internet traffic as this can easily be spoofed.

IPAllowList Middleware

So anyway, we added logging and changed the externalTrafficPolicy to Local so that Traefik sees the true source IP from the request and we can see the request logs. Now I want to show a quick example of how to restrict access to a certain path on a service, while allowing all traffic to reach all other pages. While I’m sure there are many ways to tackle this problem, this is a straightforward way to block access to /admin for a service.

I have an ecommerce shop that I want to make available on the internet. It has an admin login page at /admin so I want to just block any public traffic from reaching that page, no exceptions. See below for info about IngressRoute which is what I’m using in this example:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: evershop.example.com
  namespace: evershop
spec:
  secretName: evershop.example.com-secret
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  dnsNames:
    - evershop.example.com
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: evershop-service-ingressroute
  namespace: evershop
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  tls:
    secretName: evershop.example.com-secret
  routes:
  - match: Host(`evershop.example.com`) && PathPrefix(`/admin`)
    kind: Rule
    services:
      - name: app
        port: 3000
        scheme: http
    middlewares:
      - name: admin-ipallowlist
  - match: Host(`evershop.example.com`)
    kind: Rule
    services:
      - name: app
        port: 3000
        scheme: http
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: admin-ipallowlist
  namespace: evershop
spec:
  ipAllowList:
    sourceRange:
      - 192.168.0.0/23
      - 10.0.50.0/24

To break down what is happening there, we have a Certificate resource which is needed to get a signed certificate through Cert-Manager. At the bottom we have a Middleware which is a Traefik specific CRD. This one uses ipAllowList and I specified 2 local subnets that I want to allow. Finally, we have an IngressRoute. Within that, there are two routes that can be matched, and the one that includes the PathPrefix of /admin in the match rule has an extra middlewares: section that references admin-ipallowlist. That tells Traefik that any inbound traffic to this service that has /admin in the path should also be checked against the IPAllowList before routing traffic to the backend app service. If the source IP doesn’t match the sourceRange, it will return a 403 Forbidden page.

You can customize the 403 Forbidden page, but I didn’t want to do that at this time. Just know that it can be routed to another page (or another service) by using another Middleware of type errors: https://doc.traefik.io/traefik/middlewares/http/errorpages/

Other Options

Some of the options you might notice depend on persistent storage. Just be sure you have storage configured in your cluster before using any of those options (which I’ll talk about in my next post).

Specifically I noticed the persistence section in the values.yaml file, but that only pertains to storing certificates acquired directly through Traefik. I would just recommend ignoring that option and sticking with cert-manager.

Traefik provides some examples of common use cases which might also be helpful: https://github.com/traefik/traefik-helm-chart/blob/master/EXAMPLES.md

Beyond all of that, if you need to configure custom entryPoints (say you need ingress on a weird port like 4567), read the Traefik docs, make the change in the Helm values and run the upgrade. I might come back later and add a section explaining this in more detail, but for now I just want to mention it’s a possibility. I’ve used it for the GitLab container registry among other things.

Changed Your Mind After Installing Or Just Need To Upgrade?

This is basic Helm stuff, but basically just update your values and use the same command as before except replacing install with upgrade.

• Update values.yaml
• Upgrade Helm chart: helm upgrade -n traefik -f values.yaml traefik traefik/traefik

It almost feels like you just type “traefik traefik traefik” over again a few times, but slightly more nuanced. See the Helm docs for more details.

https://helm.sh/docs/helm/helm_upgrade/

Traefik Version Upgrade

Always read release notes before upgrading! Sometimes there is more to it than just running the Helm upgrade command. See https://github.com/traefik/traefik-helm-chart?tab=readme-ov-file#upgrading for more details around CRDs and other considerations.

TLDR; Deploy Using Options I’m Using

• Deployment: Deployment with 2 replicas
  • I don’t want to run too many replicas with a DaemonSet if I scale up the number of workers.
  • This still helps me if a pod crashes, there is still another replica running while the other pod gets recycled.
• Namespace: traefik
  • I’m trying to follow best practices and use namespaces for different things
• Dashboard: Yes
  • I like having access to this in my homelab, but I will protect it with basic auth just as a best practice to secure it in some way.
• HTTP Permanent Redirect: True
  • I don’t have any special use cases for HTTP now, but if needed I would just create a new entrypoint on a different port if something needed to be non-TLS.

Write values.yaml

• values.yaml ONLY needs to contain what you’re changing, and defaults will be used automatically. So KISS and don’t include default values.

  # Deployment with 2 replicas
  deployment:
    replicas: 2
  
  # Dashboard with basic auth
  ingressRoute:
    dashboard:
      enabled: true
      matchRule: Host(`traefik-dashboard.example.com`)
      entryPoints: ["web","websecure"]
      middlewares:
        - name: traefik-dashboard-auth
  extraObjects:
    - apiVersion: v1
      kind: Secret
      metadata:
        name: traefik-dashboard-auth-secret
      type: kubernetes.io/basic-auth
      stringData:
        # Change the password PLEASE!!!
        username: admin
        password: changeme
    - apiVersion: traefik.io/v1alpha1
      kind: Middleware
      metadata:
        name: traefik-dashboard-auth
      spec:
        basicAuth:
          secret: traefik-dashboard-auth-secret
  
  # HTTP permanent redirect
  additionalArguments:
    - "--entrypoints.web.http.redirections.entryPoint.to=:443"
    - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
    - "--entrypoints.web.http.redirections.entryPoint.permanent=true"
  

• Deploy using one-size-fits-all command:
  • helm upgrade --install -n traefik --create-namespace -f values.yaml traefik traefik/traefik

Testing Ingress

Finally, the easy part. Deploy something, then deploy an Ingress resource, and test. Let’s try it.

Deployment

Create a test deployment with 2 replicas. Traefik provides a utility called “whoami” which is helpful. This deployment also includes a Service (defaults to type ClusterIP) which is needed before deploying an Ingress resource.

• Create whoami-deployment.yaml:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: whoami-deployment
  spec:
    selector:
      matchLabels:
        app: whoami
    replicas: 2
    template:
      metadata:
        labels:
          app: whoami
      spec:
        containers:
        - image: traefik/whoami:latest
          name: whoami
          ports:
          - containerPort: 80
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: whoami-svc
  spec:
    ports:
    - port: 80
      targetPort: 80
      protocol: TCP
    selector:
      app: whoami
  

• Create the deployment: kubectl apply -f whoami-deployment.yaml
• Verify: k get deploy

Ingress

This is a super basic Ingress resource which can be useful for testing. It includes two annotations:

• One for cert-manager to issue a certificate from the letsencrypt-staging ClusterIssuer for the hostname specified under Ingress.spec.tls.hosts. If you had used an Issuer instead of ClusterIssuer, you would need to change that line to cert-manager.io/issuer: "letsencrypt-staging"

• One to specify which entryPoints Traefik should route traffic on. This can be a single entrypoint, or a comma separated list like web,websecure - see https://doc.traefik.io/traefik/routing/providers/kubernetes-ingress/#on-ingress

• Create whoami-ingress.yaml and update host in both places, and cluster issuer if yours has a different name:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: whoami-ingress
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-staging"
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
  spec:
    ingressClassName: traefik
    tls:
    - hosts:
      - whoami.example.com
      secretName: whoami-example-tls
    rules:
    - host: whoami.example.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: whoami-svc
              port:
                number: 80
  

• Verify: kubectl get ingress

• Test

  • Update DNS or your local hosts file to point your host (whoami.example.com) to the IP address of Traefik’s LoadBalancer service (kubectl get svc -n traefik)
  • Navigate to whoami.example.com in the browser. If you test too soon, you will get the self signed cert from Traefik. If you wait long enough and are still using the staging certificate issuer, you will still have to click through the warning message but it should say it’s issued by Let’s Encrypt Staging. If you use a production issuer, this should be secure with no warnings.

IngressRoute

This is a Traefik specific way to create an Ingress. It makes things easier if you want to use any Traefik features such as Middleware. I’ve been using this method for everything just to be consistent, plus for some reason my certificates get generated quickly with cert-manager using this method over Ingress with annotations. Let’s look at the last Ingress example but as an IngressResource. You will need to add a separate Certificate resource.

# whoami-ingressroute.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: whoami.example.com
  namespace: whoami
spec:
  secretName: whoami.example.com-secret
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  dnsNames:
    - whoami.example.com
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-service-ingressroute
  namespace: whoami
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  tls:
    secretName: whoami.example.com-secret
  routes:
  - match: Host(`whoami.example.com`)
    kind: Rule
    services:
      - name: whoami-svc
        port: 80
        scheme: http

• Apply using kubectl apply -f whoami-ingressroute.yaml
• View IngressRoute kubectl get ingressroute -n whoami
  • This will not appear as an Ingress anymore, since it’s an IngressRoute CRD now. So kubectl get ing -n whoami will not list it.

What About Gateway API?

This Kong article has a lot of good information on the topic: https://konghq.com/blog/engineering/gateway-api-vs-ingress

I have read that Gateway API is the successor to ingress controllers. The official FAQ says that Gateway API will not replace Ingress, so we are safe to continue using them (https://gateway-api.sigs.k8s.io/faq/#will-gateway-api-replace-the-ingress-api). There is no need to use Gateway API if all you need is Ingress.

Gateway API solves some inherent limitations with Ingress, primarily the fact that Ingress is Layer 7 only.

Gateway API Tutorial?

Nah.

We would need a separate tutorial on Gateway API, so I don’t even try to begin here. The examples from above mention that you even have to enable it specifically in the Traefik installation before using it: https://github.com/traefik/traefik-helm-chart/blob/master/EXAMPLES.md#use-kubernetes-gateway-api

Also - How to migrate from Ingress to Gateway API: https://gateway-api.sigs.k8s.io/guides/migrating-from-ingress/

If you’re like me and you have a habit of SSHing into an environment to troubleshoot an issue, you may be at a loss with Talos Linux since one does not simply SSH into a Talos node. Lord Of The Rings reference right there :)

What you can do is run a “debug” container with elevated privileges, in the kube-system namespace, which essentially gives you roughly the same thing. This is how.

Real quick, Talos themselves wrote an article on this including recommendations for alternative methods. I’m forging ahead the “old” way, but don’t discredit the new approach either as this type of thing is likely going to be the future and we need to adapt! Talos offers a lot of excellent troubleshooting tools via their talosctl API and if all you need is something like netstat, they’ve got you covered.

https://www.siderolabs.com/blog/how-to-ssh-into-talos-linux/

On the other hand, I just needed to test manually mounting NVMe over TCP when democratic-csi wasn’t working as expected, and this was the only way :)

In case anyone is wondering, why not just run kubectl debug and target the node, that method doesn’t have the ability to mount /dev which was needed in Talos to debug CSI driver stuff. So the alternative is to run a pod with /dev mounted, and then attach to it.

Show Me The Money!

https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/#static-profile

Anyway, here’s how to deploy the debug container.

• Write a new file debugpod.yaml and replace [nodename] with the name of the specific node you want this deployed to in your cluster. Optionally change the image if you don’t want to use Alpine. This is my go-to default because of how small and fast it is, and installing packages with APK is about as fast as it gets.

  apiVersion: v1
  kind: Pod
  metadata:
    name: debugpod
    namespace: kube-system
  spec:
    hostPID: true
    containers:
    - name: debugcontainer
      image: alpine:3.20
      stdin: true
      tty: true
      securityContext:
        privileged: true
      volumeMounts:
      - name: dev-mount
        mountPath: /dev
    volumes:
    - name: dev-mount
      hostPath:
        path: /dev
    nodeSelector:
      kubernetes.io/hostname: [nodename]
  

• Deploy the pod: kubectl apply -f debug-pod.yaml
• Drop into a shell inside the container: kubectl exec -it debugpod -n kube-system -- /bin/sh
• Alternatively, run an ephemeral container in the pod with sysadmin privileges (root): kubectl debug -it debugpod --image=alpine:3.21 --target=debugcontainer -n kube-system --profile=sysadmin
• Do something useful. In my case, I wanted to check if a pod running in this cluster can reach out to Cloudflare and Google DNS servers so:
  • Install telnet: apk add --update busybox-extras
  • telnet 1.1.1.1 53 - Ctrl+C, then e to exit
  • telnet 8.8.8.8 53 - Ctrl+C, then e to exit
• Whatever else you want to test :) For example, check this democratic-csi issue for what I did from here to test nvmeof mounts: https://github.com/democratic-csi/democratic-csi/issues/418

https://cert-manager.io/
cert-manager is a certificate controller for Kubernetes which is capable of handling all your certificate needs. This of course includes acquiring and automatically renewing certificates from Let’s Encrypt, but it can also be used as a local CA (certificate authority) for private certificates between services, etc.

Installation/Setup

https://cert-manager.io/docs/
I run piHole internally on my network and also use DNS challenge for internal only hostnames. This means when I request a new certificate and cert-manager attempts to look up the DNS challenge record, it won’t be able to query it through my piHole.

The solution is to configure cert-manager to specifically use public DNS servers to do the lookup, and that is done by setting dns01-recursive-nameservers and dns01-recursive-nameservers-only.

For that reason, I don’t use the “Default static install” method by just installing the manifest using kubectl. Instead, I use the Helm chart so that I can apply those overrides during the installation.

If you don’t need to override the nameservers used for DNS challenge, I would recommend using their Default static install method: https://cert-manager.io/docs/installation/#default-static-install

If you’re still with me and want to apply using the Helm chart, follow along!

• Make sure you have Helm version 3 or later: https://helm.sh/docs/intro/install/
• Add the helm repo:

  helm repo add jetstack https://charts.jetstack.io --force-update
  

• Install the Helm chart. Note the extraArgs for DNS nameserver settings:

  helm install \
    cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --version v1.16.1 \
    --set crds.enabled=true
    --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'
  

  • If you’re doing anything fancy on your network like me, double check that your Kubernetes nodes running cert-manager are able to reach the dns01-recursive-nameservers on port 53
• Verify deployment: kubectl get all -n cert-manager
• (optional) Verify dns01-recursive-nameserver settings were applied: kubectl -n cert-manager describe deploy cert-manager | grep dns01

Verify Installation

At this point cert-manager should be running and ready to issue self-signed certificates. We can test that.

• Test with this file named cert-manager-verify.yaml:
  • Apply: kubectl apply -f cert-manager-verify.yaml
  • Verify: kubectl describe cert -n cert-manager-test
  • Cleanup: kubectl delete -f cert-manager-verify.yaml

  apiVersion: v1
  kind: Namespace
  metadata:
    name: cert-manager-test
  ---
  apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    name: test-selfsigned
    namespace: cert-manager-test
  spec:
    selfSigned: {}
  ---
  apiVersion: cert-manager.io/v1
  kind: Certificate
  metadata:
    name: selfsigned-cert
    namespace: cert-manager-test
  spec:
    dnsNames:
      - example.com
    secretName: selfsigned-cert-tls
    issuerRef:
      name: test-selfsigned
  

Getting Certificates From Let’s Encrypt (using Cloudflare DNS)

Now we are ready to hook up Let’s Encrypt so we can get certificates that are signed by a trusted authority. It’s always a good idea to start with the staging issuer when testing Let’s Encrypt so you don’t run into rate limits with their production infrastructure.

I’m using Cloudflare for my DNS nameservers so these steps will be based on that setup.

Getting A Cloudflare API Token

• Create a Kubernetes secret resource with your Cloudflare API token. This allows cert-manager to add custom _acme-challenge records for domain validation. This will be the same API token you use for both Staging and Production certificates, so you only need to create one of these.
  • In Cloudflare, you can generate a new API token by navigating to your user account > My Profile > API Tokens > Create Token. Make sure token permissions include All zones - Zone Settings:Read, Zone:Read, DNS:Edit or you can limit to a specific zone if you have multiple and want to use different API Tokens for different zones.

  apiVersion: v1
  kind: Secret
  metadata:
    name: letsencrypt-cloudflare-api-token-secret
    namespace: cert-manager
  type: Opaque
  stringData:
    api-token: <api-token-goes-here>
  

• Apply the secret manifest: kubectl apply -f letsencrypt-cloudflare-api-token-secret.yaml

Deploying A cert-manager Issuer - Staging

cert-manager has two types of issuers: Issuer and ClusterIssuer. Issuer is used for more fine-grained control, it can be placed within a specific namespace, etc. ClusterIssuer, as the name implies, can be accessed across the whole Kubernetes cluster. Since I’m not getting too complicated, and also want to be able to easily utilize cert-manager from multiple namespaces, I’m using ClusterIssuer. If you wanted to use Issuer, just pick a namespace and the steps are almost identical to this.

It’s strongly recommended to test with Let’s Encrypt’s staging server to avoid any rate limiting. You are far less likely to run into rate limits during testing while using the staging issuer, and once you get this working it’s simply a matter of updating the ACME server URL to the production issuer and changing the name on the Issuer/ClusterIssuer (or you can run boht at the same time).

https://letsencrypt.org/docs/staging-environment/

You can use any valid email address. When you request a new cert, the email address is registered with that particular request, and that’s where they will send renewal/expiry notices.

• Create a manifest file for the staging issuer letsencrypt-staging-clusterissuer.yaml. Update your email address:

  apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt-staging
  spec:
    acme:
      # The ACME server URL
      server: https://acme-staging-v02.api.letsencrypt.org/directory
      # Email address used for ACME registration
      email: mail@example.com
      # Name of a secret used to store the ACME account private key
      privateKeySecretRef:
        name: letsencrypt-staging-issuer-secret
      # Enable the DNS-01 challenge provider
      solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: letsencrypt-cloudflare-api-token-secret
              key: api-token
  

• Apply the staging issuer: kubectl apply -f letsencrypt-staging-clusterissuer.yaml
• Verify your ClusterIssuer exists (or Issuer if that’s what you’re using): kubectl get clusterissuer

Request A Real (Staging) Certificate From Let’s Encrypt

Now we need to deploy a Certificate resource (again) to test our Let’s Encrypt staging ClusterIssuer. This is similar to the verify manifest like before, but only needs to include the Certificate resource. However, we also need to update a couple things, notably adding the field Certificate.spec.issuerRef.kind and setting its value to ClusterIssuer (see https://cert-manager.io/docs/usage/certificate/)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-staging-certificate
  namespace: default
spec:
  dnsNames:
    - justkidding.example.com
  secretName: letsencrypt-staging-cert-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging

• Apply the Certificate manifest: kubectl apply -f letsencrypt-staging-certificate.yaml
• Check the Certificate is ready: kubectl get certificate - this will show Ready=False until it’s fully issued
• Be patient. This can sometimes be quick, but sometimes can take a while. My most recent attempt took 38 minutes (see Troubleshooting section below to dive into what’s happening during the process).

Once you get a certificate issued using the staging issuer, you are ready to move to production!

Deploying A cert-manager Issuer - Production

Follow the same steps as in Staging, but using the production ACME URL and probably not using the name “staging”.

• Create a manifest file for the production issuer letsencrypt-production-clusterissuer.yaml. Update your email address:

  apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt-production
  spec:
    acme:
      # The ACME server URL
      server: https://acme-v02.api.letsencrypt.org/directory
      # Email address used for ACME registration
      email: mail@example.com
      # Name of a secret used to store the ACME account private key
      privateKeySecretRef:
        name: letsencrypt-production-issuer-secret
      # Enable the DNS-01 challenge provider
      solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: letsencrypt-cloudflare-api-token-secret
              key: api-token
  

• Apply the staging issuer: kubectl apply -f letsencrypt-production-clusterissuer.yaml
• Verify your ClusterIssuer exists (or Issuer if that’s what you’re using): kubectl get clusterissuer

Request A Real (Production) Certificate From Let’s Encrypt

This is exactly the same as for staging, except we make the request from the production ClusterIssuer (the last line in the yaml file says which ClusterIssuer to use).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-production-certificate
  namespace: default
spec:
  dnsNames:
    - justkidding.example.com
  secretName: letsencrypt-production-cert-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-production

• Apply the Certificate manifest: kubectl apply -f letsencrypt-production-certificate.yaml
• Check the Certificate is ready: kubectl get certificate - this will show Ready=False until it’s fully issued
• Be patient. This can sometimes be quick, but sometimes can take a while depending on a lot of factors. See Troubleshooting below for some tips.

Troubleshooting Certificate Requests

• Definitive guide: https://cert-manager.io/docs/troubleshooting/
  • Also for Let’s Encrypt: https://cert-manager.io/docs/troubleshooting/acme/
• kubectl get certificaterequests
• kubectl describe certificaterequests
• kubectl get order
• kubectl describe order
• kubectl get challenge
• kubectl describe challenge
  • This level will tell you if it’s stuck waiting for DNS challenge (basically it’s waiting for the _acme-challenge record to be added and validated from the cert-manager service. This is where sometimes you can get timeouts if your network doesn’t allow cert-manager to query public DNS servers to check).
  • Troubleshooting steps I’ve had here were basically verifying that the DNS record exists in Cloudflare while it’s waiting. If it is, wait a while to see if it eventually solves the challenge. If not, you might need to dive deeper into dns01-recursive-nameserver settings (see above) or otherwise make sure your internal DNS resolver is able to query the new record.
  • In the output, look for Challenge.Spec.Key which is the value you should see in the _acme-challenge TXT record
• You can also check the logs on the cert-manager container
  • Find the pod name: kubectl get po -n cert-manager
  • kubectl logs cert-manager-556766675f-pt123 -n cert-manager
• See cert-manager issue for more discussion: https://github.com/cert-manager/cert-manager/issues/5917

What About Ingress?

I’ll revisit this topic after we get to Ingress Controllers (using Traefik) and how to get certificates from Let’s Encrypt. It’s significantly easier than this, and since we already have this production ClusterIssuer in place, you basically just add a line in your Ingress resource to use this ClusterIssuer to attach a certificate and cert-manager does the rest!

There are a few different types of “Services” in Kubernetes. They provide solutions for different types of network communication between internal cluster resources and externally. Depending on what you need to connect to what, you will use a suitable Service type to provide that connection. I’m specifically not mentioning those other types because this isn’t really a tutorial on Kubernetes or what Service type you might need. But you will most likely need/want a LoadBalancer service if you need external traffic to reach within your cluster.

One of the first things you might run into after building a new Kubernetes cluster is that as soon as you go to build something that has a LoadBalancer type service, it’s stuck in a Pending status. That’s because you don’t have a “controller” (is that the right term?) to handle this type of service. It’s not included out of the box.

Why not include LoadBalancer out of the box? Because it requires some environment specific configuration, and most importantly in our case that’s a pool of IP addresses that are dedicated for this purpose. So how do we tell Kubernetes what IPs it can use for LoadBalancer services? MetalLB - https://metallb.io/ There’s also the difference between how to advertise provisioned IPs to the rest of the network so that traffic can be routed properly. MetalLB supports two options: layer 2 mode and BGP. I don’t have a need for BGP in my homelab so I’m just using layer 2 mode. Layer 2 mode uses ARP on IPv4 networks and NDP on IPv6 networks - https://metallb.io/concepts/

There are other options such as KubeVIP (https://kube-vip.io/), which is also a great project. I’m going to use MetalLB because that’s what I have already figured out. You could even use KubeVIP as a control plane load balancer, in addition to MetalLB as a service load balancer. One is dedicated to load balancing the control plane, which means you can set one IP address for all 3 control plane nodes as an interface between kubectl and your cluster’s API. But what we’re focused on here is Service load balancing, so the same thing but for all the stuff you’re actually running on your cluster like websites, etc. And eventually this will tie into Ingress and Gateway API, but let’s not get too far into the weeds yet.

See For Yourself

If you want to see the behavior wen you create a LoadBalancer service without something that implements this type of service such as MetalLB, follow along. The symptom is that instead of getting an IP assigned, you will see the EXTERNAL-IP “stuck” as <pending>.

• Save this as test-lb-svc.yaml and then run kubectl apply -f test-lb-svc.yaml. This creates a deployment running 2 replicas of whoami (which is a simple web server that shows you some details about the connection provided by Traefik), then creates a LoadBalancer service in front of it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-deployment
  labels:
    app: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: traefik/whoami:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami-loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: whoami
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80