Skip to content

ClickHouse on Kubernetes

Last updated

View as Markdown

Monitor ClickHouse that runs under the Altinity Kubernetes Operator, as a ClickHouseInstallation.

The chart, the namespace, secrets, and the install command work as described on the Kubernetes page.

The requirements in Before you start on the Kubernetes page apply here, along with the following:

  • Check the version on any replica. The collector supports 25.8 and later.

  • Ensure that you have DBGorilla collector and chart 0.17.0 or later

  • Ensure that the collector runs inside the same Kubernetes cluster as the installation.

  • Ensure that the collector pod can reach every replica on port 8123, or 8443 with TLS.

  • Create the dbg_collector user from Prepare the cluster in ClickHouse. If the installation defines its users in the ClickHouseInstallation and not in SQL, add the user there.

  • Add a collector in the DBGorilla web admin interface and record the values provided to include in your collector config file.

The provider reads two kinds of object in the installation’s namespace: the StatefulSets the operator creates, one per replica, and their pods. It gets three verbs, get, list and watch. No writes, no exec, and no access to Secrets. Nothing the collector is granted can change a database or a Kubernetes object.

The chart renders this when rbac.create is on. The only decision is where to bind it:

rbac:
create: true
scope: namespaced
namespaces:
- analytics # the namespace that HOLDS the installation

namespaces is where the ClickHouseInstallation lives, which is often not where the collector runs. A binding in the wrong namespace installs cleanly and grants nothing.

rbac.nodeMetrics is on by default and is separate from scope. The collector reads each ClickHouse pod’s CPU, memory, and volume usage from the kubelet on the node that runs it. The objects that authorize that read, nodes, nodes/stats, and nodes/pods, are cluster scoped, so this binding is always a ClusterRoleBinding, whatever scope says.

ModeWhat it doesUse it when
auto (default)Collects everything it can. If the node grant is missing, it logs that pod metrics are off.You want an install that never fails on the node grant
enabledRequires the node grant. The component does not start without it.You want to know the moment the grant is wrong

This provider finds its replicas through Kubernetes, so mode = "disabled" is refused. To collect with no Kubernetes access at all, use type = "self_hosted" as on ClickHouse.

collector.toml, one [[component]] per installation:

[dbgorilla]
agent_id = "<agent id from the app>"
tenant_id = "<tenant id from the app>"
secret = { env = "DBG_SERVER_SECRET" }
[[component]]
name = "events"
engine = "clickhouse"
[component.provider]
type = "altinity"
namespace = "analytics"
installation = "events"
[component.auth]
method = "password"
user = "dbg_collector"
password = { env = "CLICKHOUSE_COLLECTOR_PASSWORD" }
[component.connect]
host = "clickhouse-events.analytics.svc"

namespace is where the ClickHouseInstallation lives and installation is its name. host is the installation’s own Service, which the operator names clickhouse-<installation>.

values.yaml:

rbac:
create: true
scope: namespaced
namespaces:
- analytics
secrets:
serverSecret: "<secret from the app>"
databasePasswords:
CLICKHOUSE_COLLECTOR_PASSWORD: "<the dbg_collector password>"

Then:

Terminal window
helm install dbg-collector oci://dbgorillapublic.azurecr.io/charts/dbg-collector \
--namespace analytics \
--values ./values.yaml \
--set-file config.inline=./collector.toml

If a NetworkPolicy guards the ClickHouse pods, give the collector the label it admits under podLabels in values.yaml.

By default the collector monitors every cluster the installation defines. You may specify a single cluster directly.

[component.provider]
type = "altinity"
namespace = "analytics"
installation = "events"
cluster = "main"

If the HTTP interface uses TLS, set the HTTPS port and ssl_mode, and mount the CA from its Secret. The default port is 8123 and the default ssl_mode is disable.

[component.connect]
host = "clickhouse-events.analytics.svc"
port = 8443
ssl_mode = "verify_full"
ca_cert = "/etc/dbg-collector-certs/ca.crt"
extraVolumes:
- name: clickhouse-ca
secret:
secretName: clickhouse-ca # the Secret that holds your CA
items:
- key: ca.crt
path: ca.crt
extraVolumeMounts:
- name: clickhouse-ca
mountPath: /etc/dbg-collector-certs
readOnly: true

With verify_full, each replica’s certificate needs the name the collector connects to, <service>.<namespace>.svc. If the certificates carry other names, use require, which encrypts without checking the certificate.

kubernetes.mode is auto by default. See Choosing kubernetes.mode.

[component.provider.kubernetes]
mode = "enabled"

RPC Commands allow DBGorilla agents to run commands on your database. By default commands are disabled.

[commands]
enabled = true
allowed = ["explain", "execute_query"]

Commands in ClickHouse lists what each command does and the grant it needs.

databases and exclude_databases work as described under Choose databases in ClickHouse.

Add a [[component]] block per installation, each with its own namespace and installation, and list every namespace under rbac.namespaces:

[[component]]
name = "events"
engine = "clickhouse"
# ...
[[component]]
name = "billing"
engine = "clickhouse"
# ...

Read the collector log:

Terminal window
kubectl -n analytics logs -f deploy/dbg-collector

A healthy start has opamp handshake complete and discovered component. The first discovered component line carries "engine":"clickhouse" and a "nodes" count, which is the number of replicas the installation declares.

Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked Monitored, as ClickHouse with its version.

The log names anything that is wrong:

Log lineFix
no StatefulSet in namespaceCheck namespace and installation in collector.toml.
the Kubernetes API could not be readSet rbac.create: true and list the installation’s namespace under rbac.namespaces.
kubernetes.mode = "disabled" is not supportedUse auto or enabled, or switch to type = "self_hosted".
pod CPU, memory and volume metrics will not be collectedSet rbac.nodeMetrics: true. On a kubelet older than Kubernetes 1.33 the read also needs nodes/proxy, which the chart leaves for you to grant.
replica is not running; it stays a member and is retriedA replica’s pod is down. The line names the pod and the reason, such as StatefulSet chi-events-main-1-0 is scaled to 0.
Authentication failedThe dbg_collector user is missing on a replica, or the password is wrong.
unsupported combination (engine=clickhouse, provider=altinity, auth=password)The collector is older than 0.17.0. Use 0.17.0 or later.

The helm upgrade command is under Upgrading and removing in Manage collectors.

helm uninstall is under Upgrading and removing in Manage collectors. It removes the Role and the binding with the rest of the release. Then drop the dbg_collector user.