# ClickHouse on Kubernetes

Monitor ClickHouse that runs under the Altinity Kubernetes Operator, as a `ClickHouseInstallation`.

The chart, the namespace, secrets, and the install command work as described on the
[Kubernetes](/docs/connect/kubernetes/) page.

## Before you start

The requirements in [Before you start](/docs/connect/kubernetes/#before-you-start) on the Kubernetes
page apply here, along with the following:

- Check the version on any replica.  The collector supports 25.8 and later.

- Ensure that you have DBGorilla collector and chart 0.17.0 or later

- Ensure that the collector runs inside the same Kubernetes cluster as the installation.

- Ensure that the collector pod can reach every replica on port 8123, or 8443 with TLS.

- Create the `dbg_collector` user from
  [Prepare the cluster](/docs/connect/clickhouse/#prepare-the-cluster) in ClickHouse.  If the
  installation defines its users in the `ClickHouseInstallation` and not in SQL, add the user there.

- Add a collector in the DBGorilla web admin interface and record the values provided to include in your collector config file.

### Kubernetes read access

The provider reads two kinds of object in the installation's namespace: the StatefulSets the
operator creates, one per replica, and their pods.  It gets three verbs, `get`, `list` and `watch`.
No writes, no `exec`, and no access to Secrets.  Nothing the collector is granted can change a
database or a Kubernetes object.

The chart renders this when `rbac.create` is on.  The only decision is where to bind it:

```yaml
rbac:
  create: true
  scope: namespaced
  namespaces:
    - analytics        # the namespace that HOLDS the installation
```

`namespaces` is where the `ClickHouseInstallation` lives, which is often not where the collector
runs.  A binding in the wrong namespace installs cleanly and grants nothing.

#### Pod CPU, memory and disk

`rbac.nodeMetrics` is on by default and is separate from `scope`.  The collector reads each
ClickHouse pod's CPU, memory, and volume usage from the kubelet on the node that runs it.  The
objects that authorize that read, `nodes`, `nodes/stats`, and `nodes/pods`, are cluster scoped, so
this binding is always a ClusterRoleBinding, whatever `scope` says.

#### Choosing `kubernetes.mode`

| Mode | What it does | Use it when |
|---|---|---|
| `auto` (default) | Collects everything it can.  If the node grant is missing, it logs that pod metrics are off. | You want an install that never fails on the node grant |
| `enabled` | Requires the node grant.  The component does not start without it. | You want to know the moment the grant is wrong |

This provider finds its replicas through Kubernetes, so `mode = "disabled"` is refused.  To collect
with no Kubernetes access at all, use `type = "self_hosted"` as on
[ClickHouse](/docs/connect/clickhouse/).

## Install

`collector.toml`, one `[[component]]` per installation:

```toml
[dbgorilla]
agent_id  = "<agent id from the app>"
tenant_id = "<tenant id from the app>"
secret    = { env = "DBG_SERVER_SECRET" }

[[component]]
name   = "events"
engine = "clickhouse"

[component.provider]
type         = "altinity"
namespace    = "analytics"
installation = "events"

[component.auth]
method   = "password"
user     = "dbg_collector"
password = { env = "CLICKHOUSE_COLLECTOR_PASSWORD" }

[component.connect]
host = "clickhouse-events.analytics.svc"
```

`namespace` is where the `ClickHouseInstallation` lives and `installation` is its name.  `host` is
the installation's own Service, which the operator names `clickhouse-<installation>`.

`values.yaml`:

```yaml
rbac:
  create: true
  scope: namespaced
  namespaces:
    - analytics

secrets:
  serverSecret: "<secret from the app>"
  databasePasswords:
    CLICKHOUSE_COLLECTOR_PASSWORD: "<the dbg_collector password>"
```

Then:

```bash
helm install dbg-collector oci://dbgorillapublic.azurecr.io/charts/dbg-collector \
  --namespace analytics \
  --values ./values.yaml \
  --set-file config.inline=./collector.toml
```

If a NetworkPolicy guards the ClickHouse pods, give the collector the label it admits under
`podLabels` in `values.yaml`.

## Additional configuration

### Cluster Selection

By default the collector monitors every cluster the installation defines.  You may specify a single cluster directly.

```toml
[component.provider]
type         = "altinity"
namespace    = "analytics"
installation = "events"
cluster      = "main"
```

### TLS

If the HTTP interface uses TLS, set the HTTPS port and `ssl_mode`, and mount the CA from its
Secret.  The default port is 8123 and the default `ssl_mode` is `disable`.

```toml
[component.connect]
host     = "clickhouse-events.analytics.svc"
port     = 8443
ssl_mode = "verify_full"
ca_cert  = "/etc/dbg-collector-certs/ca.crt"
```

```yaml
extraVolumes:
  - name: clickhouse-ca
    secret:
      secretName: clickhouse-ca    # the Secret that holds your CA
      items:
        - key: ca.crt
          path: ca.crt

extraVolumeMounts:
  - name: clickhouse-ca
    mountPath: /etc/dbg-collector-certs
    readOnly: true
```

With `verify_full`, each replica's certificate needs the name the collector connects to,
`<service>.<namespace>.svc`.  If the certificates carry other names, use `require`, which encrypts
without checking the certificate.

### Kubernetes mode

`kubernetes.mode` is `auto` by default.  See [Choosing `kubernetes.mode`](#choosing-kubernetesmode).

```toml
[component.provider.kubernetes]
mode = "enabled"
```

### Commands

RPC Commands allow DBGorilla agents to run commands on your database.  By default commands are disabled.

```toml
[commands]
enabled = true
allowed = ["explain", "execute_query"]
```

[Commands](/docs/connect/clickhouse/#commands) in ClickHouse lists what each command does and the
grant it needs.

### Choose databases

`databases` and `exclude_databases` work as described under
[Choose databases](/docs/connect/clickhouse/#choose-databases) in ClickHouse.

### Several installations, one collector

Add a `[[component]]` block per installation, each with its own `namespace` and `installation`, and
list every namespace under `rbac.namespaces`:

```toml
[[component]]
name   = "events"
engine = "clickhouse"
# ...

[[component]]
name   = "billing"
engine = "clickhouse"
# ...
```

## Check it is working

Read the collector log:

```bash
kubectl -n analytics logs -f deploy/dbg-collector
```

A healthy start has `opamp handshake complete` and `discovered component`.  The first
`discovered component` line carries `"engine":"clickhouse"` and a `"nodes"` count, which is the
number of replicas the installation declares.

Then open Home in DBGorilla.  The cluster appears under the name you gave the component, marked
**Monitored**, as ClickHouse with its version.

The log names anything that is wrong:

| Log line | Fix |
|---|---|
| `no StatefulSet in namespace` | Check `namespace` and `installation` in `collector.toml`. |
| `the Kubernetes API could not be read` | Set `rbac.create: true` and list the installation's namespace under `rbac.namespaces`. |
| `kubernetes.mode = "disabled" is not supported` | Use `auto` or `enabled`, or switch to `type = "self_hosted"`. |
| `pod CPU, memory and volume metrics will not be collected` | Set `rbac.nodeMetrics: true`.  On a kubelet older than Kubernetes 1.33 the read also needs `nodes/proxy`, which the chart leaves for you to grant. |
| `replica is not running; it stays a member and is retried` | A replica's pod is down.  The line names the pod and the reason, such as `StatefulSet chi-events-main-1-0 is scaled to 0`. |
| `Authentication failed` | The `dbg_collector` user is missing on a replica, or the password is wrong. |
| `unsupported combination (engine=clickhouse, provider=altinity, auth=password)` | The collector is older than 0.17.0.  Use 0.17.0 or later. |

## Upgrade

The `helm upgrade` command is under
[Upgrading and removing](/docs/administer/manage-collectors/#kubernetes) in Manage collectors.

## Remove

`helm uninstall` is under
[Upgrading and removing](/docs/administer/manage-collectors/#kubernetes) in Manage collectors.  It
removes the Role and the binding with the rest of the release.  Then drop the `dbg_collector` user.

## Next

- [Topology](/docs/using-dbgorilla/topology/) shows what the collector found.
- [Organization Management](/docs/administer/members-and-roles/) lets you bring in your team.