ClickHouse on Kubernetes
Monitor ClickHouse that runs under the Altinity Kubernetes Operator, as a ClickHouseInstallation.
The chart, the namespace, secrets, and the install command work as described on the Kubernetes page.
Before you start
Section titled “Before you start”The requirements in Before you start on the Kubernetes page apply here, along with the following:
-
Check the version on any replica. The collector supports 25.8 and later.
-
Ensure that you have DBGorilla collector and chart 0.17.0 or later
-
Ensure that the collector runs inside the same Kubernetes cluster as the installation.
-
Ensure that the collector pod can reach every replica on port 8123, or 8443 with TLS.
-
Create the
dbg_collectoruser from Prepare the cluster in ClickHouse. If the installation defines its users in theClickHouseInstallationand not in SQL, add the user there. -
Add a collector in the DBGorilla web admin interface and record the values provided to include in your collector config file.
Kubernetes read access
Section titled “Kubernetes read access”The provider reads two kinds of object in the installation’s namespace: the StatefulSets the
operator creates, one per replica, and their pods. It gets three verbs, get, list and watch.
No writes, no exec, and no access to Secrets. Nothing the collector is granted can change a
database or a Kubernetes object.
The chart renders this when rbac.create is on. The only decision is where to bind it:
rbac: create: true scope: namespaced namespaces: - analytics # the namespace that HOLDS the installationnamespaces is where the ClickHouseInstallation lives, which is often not where the collector
runs. A binding in the wrong namespace installs cleanly and grants nothing.
Pod CPU, memory and disk
Section titled “Pod CPU, memory and disk”rbac.nodeMetrics is on by default and is separate from scope. The collector reads each
ClickHouse pod’s CPU, memory, and volume usage from the kubelet on the node that runs it. The
objects that authorize that read, nodes, nodes/stats, and nodes/pods, are cluster scoped, so
this binding is always a ClusterRoleBinding, whatever scope says.
Choosing kubernetes.mode
Section titled “Choosing kubernetes.mode”| Mode | What it does | Use it when |
|---|---|---|
auto (default) | Collects everything it can. If the node grant is missing, it logs that pod metrics are off. | You want an install that never fails on the node grant |
enabled | Requires the node grant. The component does not start without it. | You want to know the moment the grant is wrong |
This provider finds its replicas through Kubernetes, so mode = "disabled" is refused. To collect
with no Kubernetes access at all, use type = "self_hosted" as on
ClickHouse.
Install
Section titled “Install”collector.toml, one [[component]] per installation:
[dbgorilla]agent_id = "<agent id from the app>"tenant_id = "<tenant id from the app>"secret = { env = "DBG_SERVER_SECRET" }
[[component]]name = "events"engine = "clickhouse"
[component.provider]type = "altinity"namespace = "analytics"installation = "events"
[component.auth]method = "password"user = "dbg_collector"password = { env = "CLICKHOUSE_COLLECTOR_PASSWORD" }
[component.connect]host = "clickhouse-events.analytics.svc"namespace is where the ClickHouseInstallation lives and installation is its name. host is
the installation’s own Service, which the operator names clickhouse-<installation>.
values.yaml:
rbac: create: true scope: namespaced namespaces: - analytics
secrets: serverSecret: "<secret from the app>" databasePasswords: CLICKHOUSE_COLLECTOR_PASSWORD: "<the dbg_collector password>"Then:
helm install dbg-collector oci://dbgorillapublic.azurecr.io/charts/dbg-collector \ --namespace analytics \ --values ./values.yaml \ --set-file config.inline=./collector.tomlIf a NetworkPolicy guards the ClickHouse pods, give the collector the label it admits under
podLabels in values.yaml.
Additional configuration
Section titled “Additional configuration”Cluster Selection
Section titled “Cluster Selection”By default the collector monitors every cluster the installation defines. You may specify a single cluster directly.
[component.provider]type = "altinity"namespace = "analytics"installation = "events"cluster = "main"If the HTTP interface uses TLS, set the HTTPS port and ssl_mode, and mount the CA from its
Secret. The default port is 8123 and the default ssl_mode is disable.
[component.connect]host = "clickhouse-events.analytics.svc"port = 8443ssl_mode = "verify_full"ca_cert = "/etc/dbg-collector-certs/ca.crt"extraVolumes: - name: clickhouse-ca secret: secretName: clickhouse-ca # the Secret that holds your CA items: - key: ca.crt path: ca.crt
extraVolumeMounts: - name: clickhouse-ca mountPath: /etc/dbg-collector-certs readOnly: trueWith verify_full, each replica’s certificate needs the name the collector connects to,
<service>.<namespace>.svc. If the certificates carry other names, use require, which encrypts
without checking the certificate.
Kubernetes mode
Section titled “Kubernetes mode”kubernetes.mode is auto by default. See Choosing kubernetes.mode.
[component.provider.kubernetes]mode = "enabled"Commands
Section titled “Commands”RPC Commands allow DBGorilla agents to run commands on your database. By default commands are disabled.
[commands]enabled = trueallowed = ["explain", "execute_query"]Commands in ClickHouse lists what each command does and the grant it needs.
Choose databases
Section titled “Choose databases”databases and exclude_databases work as described under
Choose databases in ClickHouse.
Several installations, one collector
Section titled “Several installations, one collector”Add a [[component]] block per installation, each with its own namespace and installation, and
list every namespace under rbac.namespaces:
[[component]]name = "events"engine = "clickhouse"# ...
[[component]]name = "billing"engine = "clickhouse"# ...Check it is working
Section titled “Check it is working”Read the collector log:
kubectl -n analytics logs -f deploy/dbg-collectorA healthy start has opamp handshake complete and discovered component. The first
discovered component line carries "engine":"clickhouse" and a "nodes" count, which is the
number of replicas the installation declares.
Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked Monitored, as ClickHouse with its version.
The log names anything that is wrong:
| Log line | Fix |
|---|---|
no StatefulSet in namespace | Check namespace and installation in collector.toml. |
the Kubernetes API could not be read | Set rbac.create: true and list the installation’s namespace under rbac.namespaces. |
kubernetes.mode = "disabled" is not supported | Use auto or enabled, or switch to type = "self_hosted". |
pod CPU, memory and volume metrics will not be collected | Set rbac.nodeMetrics: true. On a kubelet older than Kubernetes 1.33 the read also needs nodes/proxy, which the chart leaves for you to grant. |
replica is not running; it stays a member and is retried | A replica’s pod is down. The line names the pod and the reason, such as StatefulSet chi-events-main-1-0 is scaled to 0. |
Authentication failed | The dbg_collector user is missing on a replica, or the password is wrong. |
unsupported combination (engine=clickhouse, provider=altinity, auth=password) | The collector is older than 0.17.0. Use 0.17.0 or later. |
Upgrade
Section titled “Upgrade”The helm upgrade command is under
Upgrading and removing in Manage collectors.
Remove
Section titled “Remove”helm uninstall is under
Upgrading and removing in Manage collectors. It
removes the Role and the binding with the rest of the release. Then drop the dbg_collector user.
- Topology shows what the collector found.
- Organization Management lets you bring in your team.