* This version is deprecated since it is no longer supported by its creator. We continue to support it, but support will be removed in the future.
Host packages are bundled and installed by kURL without the need for external package repositories except for in the case of Red Hat Enterprise Linux 9 and Rocky Linux 9.
For these OSes, the following packages are required per add-on:
| Add-on | Packages |
|---|---|
| * kURL Core | curl openssl tar |
| Collectd | bash glibc libcurl libcurl-minimal libgcrypt libgpg-error libmnl openssl-libs rrdtool systemd systemd-libs yajl |
| Containerd | bash libseccomp libzstd systemd |
| Kubernetes | conntrack-tools ethtool glibc iproute iptables-nft socat util-linux |
| Longhorn | iscsi-initiator-utils nfs-utils |
| OpenEBS \versions 1.x and 2.x* | iscsi-initiator-utils |
| Rook | lvm2 |
| Velero | nfs-utils |
The following table lists information about the core directory requirements.
| Name | Location | Requirements | Description |
|---|---|---|---|
| etcd | /var/lib/etcd/ | 2 GB | Kubernetes etcd cluster data directory. See the etcd documentation and Cloud Disk Performance for more information and recommendations. |
| kubelet | /var/lib/kubelet/ | 30 GB * | Used for local disk volumes, emptyDir, log storage, and more. See the Kubernetes Resource Management for Pods and Containers documentation for more information. |
| containerd | /var/lib/containerd/ | 30 GB * | Snapshots, content, metadata for containers and image, as well as any plugin data will be kept in this location. See the containerd documentation for more information. |
| kube-apiserver | /var/log/apiserver/ | 1 GB | Kubernetes audit logs. See Kubernetes Auditing documentation for mode information. |
| kURL | /var/lib/kurl/ | 10 GB ** | kURL data directory used to store utilities, system packages, and container images. This directory can be overridden with the flag kurl-install-directory (see kURL Advanced Install Options) |
| Root Disk | / | 100 GB | Based on the aggregate requirements above and the fact that Kubernetes will start to reclaim space at 85% full disk, the minimum recommended root partition is 100 GB. See details above for each component. |
* This requirement depends on the size of the container images and the amount of ephemeral data used by your application containers.
** This requirement can vary depending on your choice of kURL add-ons and can grow over time.
In addition to the storage requirements, the Kubernetes garbage collection process attempts to ensure that the Node and Image filesystems do not reach their minimum available disk space thresholds of 10% and 15% respectively. For this reason, kURL recommends an additional 20% overhead on top of these disk space requirements for the volume or volumes containing the directories /var/lib/kubelet/ and /var/lib/containerd/. For more information see the Kubernetes Reclaiming node level resources documentation.
The following table lists the add-on directory locations and disk space requirements, if applicable. For any additional requirements, see the specific topic for the add-on.
| Name | Location | Requirements | Description |
|---|---|---|---|
| Docker | /var/lib/docker/ | 30 GB * | Images, containers and volumes, and more will be kept in this location. See the Docker Storage documentation for more information. When using the Docker runtime, /var/lib/containerd/ is not required. |
| Docker | /var/lib/dockershim/ | N/A | Kubernetes dockershim data directory |
| Weave | /var/lib/cni/ | N/A | Container networking data directory |
| Weave | /var/lib/weave/ | N/A | Weave data directory |
| Rook | /var/lib/rook/ | 10 GB | Ceph monitor metadata directory. See the ceph-mon Minimum Hardware Recommendations for more information. |
| Registry | PVC ** | N/A | Stores container images only in airgapped clusters. Data will be stored in Persistent Volumes. |
| Velero | PVC ** | N/A | Stores snapshot data. Data will be stored in Persistent Volumes. |
* This requirement depends on the size of the container images and the amount of ephemeral data used by your application containers.
** Data will be stored in Persistent Volumes. Requirements depend on the provisioner of choice. See Persistent Volume Requirements for more information.
Depending on the amount of persistent data stored by your application, you will need to allocate enough disk space at the following location dependent on your PVC provisioner or provisioners.
| Name | Location | Description |
|---|---|---|
| OpenEBS | /var/openebs/local/ | OpenEBS Local PV Hostpath volumes will be created under this directory. See the OpenEBS Add-on documentation for more information. |
| Rook (Block Storage) | Rook add-on version 1.4.3 and later requires an unformatted storage device on each node in the cluster for Ceph volumes. See the Rook Block Storage documentation for more information. | |
| Rook (version 1.0.x) | /opt/replicated/rook/ | Rook Filesystem volumes will be created under this directory. See the Rook Filesystem Storage documentation for more information. |
| Longhorn | /var/lib/longhorn/ | Longhorn volumes will be created under this directory. See the Longhorn Add-on documentation for more information. |
The fully-qualified domain name (FQDN) of any host used with kURL must be a valid DNS subdomain name, and its name must be resolvable by DNS. For more information, see DNS Subdomain Names in the Kubernetes documentation.
After a host is added to a Kubernetes cluster, Kubernetes assumes that the hostname and IP address of the host will not change. If you need to change the hostname or IP address of a node, you must first remove the node from the cluster.
To change the hostname or IP address of a node in clusters that do not have three or more nodes, use snapshots to move the application to a new cluster before you attempt to remove the node. For more information about using snapshots, see Velero Add-on.
For more information about the requirements for naming nodes, see Node naming uniqueness in the Kubernetes documentation.
The following domains need to be accessible from servers performing online kURL installs. IP addresses for these services can be found in replicatedhq/ips.
| Host | Description |
|---|---|
| amazonaws.com | tar.gz packages are downloaded from Amazon S3 during embedded cluster installations. The IP ranges to allowlist for accessing these can be scraped dynamically from the AWS IP Address Ranges documentation. |
| k8s.gcr.io, registry.k8s.io | Images for the Kubernetes control plane are downloaded from the Google Container Registry repository used to publish official container images for Kubernetes. Starting March 20, 2023, these requests are proxied to the new address registry.k8s.io. Both of these URLs must be allowed network traffic using firewall rules. For more information on the Kubernetes control plane components, see the Kubernetes documentation. |
| k8s.kurl.sh, s3.kurl.sh | Kubernetes cluster installation scripts and artifacts are served from kurl.sh. Bash scripts and binary executables are served from kurl.sh. This domain is owned by Replicated, Inc which is headquartered in Los Angeles, CA. |
No outbound internet access is required for airgapped installations.
The kURL install script will prompt to disable firewalld. Note that firewall rules can affect communications between containers on the same machine, so it is recommended to disable these rules entirely for Kubernetes. Firewall rules can be added after or preserved during an install, but because installation parameters like pod and service CIDRs can vary based on local networking conditions, there is no general guidance available on default requirements. See Advanced Options for installer flags that can preserve these rules.
The following ports must be open between nodes for multi-node clusters:
| Protocol | Direction | Port Range | Purpose | Used By |
|---|---|---|---|---|
| TCP | Inbound | 6443 | Kubernetes API server | All |
| TCP | Inbound | 2379-2380 | etcd server client API | Primary |
| TCP | Inbound | 10250 | kubelet API | Primary |
| UDP | Inbound | 8472 | Flannel VXLAN | All |
| TCP | Inbound | 6783 | Weave Net control | All |
| UDP | Inbound | 6783-6784 | Weave Net data | All |
| TCP | Inbound | 9090 | Rook CSI RBD Plugin Metrics | All |
| Protocol | Direction | Port Range | Purpose | Used By |
|---|---|---|---|---|
| TCP | Inbound | 10250 | kubelet API | Primary |
| UDP | Inbound | 8472 | Flannel VXLAN | All |
| TCP | Inbound | 6783 | Weave Net control | All |
| UDP | Inbound | 6783-6784 | Weave Net data | All |
| TCP | Inbound | 9090 | Rook CSI RBD Plugin Metrics | All |
These ports are required for Kubernetes, Flannel, and Weave Net.
In addition to the ports listed above that must be open between nodes, the following ports should be available on the host for components to start TCP servers accepting local connections.
| Port | Purpose |
|---|---|
| 2381 | etcd health and metrics server |
| 6781 | weave network policy controller metrics server |
| 6782 | weave metrics server |
| 10248 | kubelet health server |
| 10249 | kube-proxy metrics server |
| 9100 | prometheus node-exporter metrics server |
| 10257 | kube-controller-manager health server |
| 10259 | kube-scheduler health server |
When using the Flannel CNI, to allow for outgoing TCP connections from pods, you must configure stateless packet filtering firewalls to allow all packets with TCP flags "ack" with destination port range 1024-65535. For more information see the Flannel Firewalls add-on documentation.
| Name | Source IP | Destination IP | Source port | Destination port | Protocol | TCP flags | Action |
| ---- | --------- | -------------- | ----------- | ---------------- | -------- | --------- | ------ |
| Allow outgoing TCP | 0.0.0.0/0 | 0.0.0.0/0 | 0-65535 | 1024-65535 | tcp | ack | accept |
In addition to the networking requirements described in the previous section, operating a cluster with high availability adds additional constraints.
To operate the Kubernetes control plane in HA mode, it is recommended to have a minimum of 3 primary nodes.
In the event that one of these nodes becomes unavailable, the remaining two will still be able to function with an etcd quorum.
As the cluster scales, dedicating these primary nodes to control-plane only workloads using the noSchedule taint should be considered.
This will affect the number of nodes that need to be provisioned.
The number of required secondary nodes is primarily a function of the desired application availability and throughput. By default, primary nodes in kURL also run application workloads. At least 2 nodes should be used for data durability for applications that use persistent storage (i.e. databases) deployed in-cluster.
Highly available cluster setups that do not leverage EKCO's internal load balancing capability require a load balancer to route requests to healthy nodes. The following requirements need to be met for load balancers used on the control plane (primary nodes):
The load balancer must support hairpinning, i.e. nodes referring to each other through the load balancer IP.
The IP or DNS name and port of the load balancer should be provided as an argument to kURL during the HA setup. See Highly Available K8s for more install information.
For more information on configuring load balancers in the public cloud for kURL installs see Public Cloud Load Balancing.
Load balancer requirements for application workloads vary depending on workload.
The following example cloud VM instance/disk combinations are known to provide sufficient performance for etcd and will pass the write latency preflight.