# Instaclustr ClickHouse

Monitor a ClickHouse cluster that NetApp Instaclustr runs for you.

## Before you start

- Check the ClickHouse version in the Instaclustr console.  The collector supports 25.8 and later.

- Ensure that you have DBGorilla collector 0.17.0 or later

- Ensure that the collector's egress IP address is in the cluster's firewall rules for the HTTPS
  port, 8443.

- Create a read-only provisioning API key in the Instaclustr console.

- For host metrics, also create a Prometheus API key.  It is a separate key type.

- Ensure the collector can reach `otlp.dbgorilla.com`, `auth.dbgorilla.com`, `api.instaclustr.com`
  and `*.prometheus.monitoring.instaclustr.com` on port 443.

- Install Docker on the machine that will run the collector.

- Add a collector in the DBGorilla web admin interface and record the values provided to include in your collector config file.

- Record the cluster ID and one node's host name.  Both are on the cluster's page in the
  Instaclustr console, the host name under **Connection Info**.

## Prepare the cluster

Connect as the default `icclickhouse` user from **Connection Info** in the Instaclustr console.
That user is an administrator, so give the collector its own user:

```sql
CREATE USER dbg_collector ON CLUSTER '<cluster id>' IDENTIFIED WITH sha256_password BY '<strong password>';
GRANT ON CLUSTER '<cluster id>' SELECT ON system.* TO dbg_collector;
```

Then run this statement on every node.  It does not take `ON CLUSTER`:

```sql
GRANT CURRENT GRANTS(SHOW ON *.*) TO dbg_collector;
```

Instaclustr keeps a database of its own that `icclickhouse` cannot grant on, so a plain
`GRANT SHOW ON *.*` is refused.  `CURRENT GRANTS` passes on everything `icclickhouse` holds.

[Prepare the cluster](/docs/connect/clickhouse/#prepare-the-cluster) in ClickHouse lists what each
grant adds.

## Install

Write `collector.toml`.  `host` is the host name of any one node, as **Connection Info** shows it.
Use the name and not the IP address: the nodes' certificate covers their names, and the collector
works out the other nodes' names from this one.

```toml
[dbgorilla]
agent_id  = "<agent id from the app>"
tenant_id = "<tenant id from the app>"
secret    = "${DBG_SERVER_SECRET}"

[[component]]
name   = "events-instaclustr"
engine = "clickhouse"

[component.provider]
type               = "instaclustr"
cluster_id         = "<cluster id>"
api_username       = "you@example.com"
api_key            = { env = "IC_READONLY_API_KEY" }
prometheus_api_key = { env = "IC_PROMETHEUS_API_KEY" }

[component.auth]
method   = "password"
user     = "dbg_collector"
password = { env = "INSTACLUSTR_CLICKHOUSE_PASSWORD" }

[component.connect]
host     = "ip-203-0-113-20.<cluster hash>.cnodes.io"
port     = 8443
ssl_mode = "verify_full"
```

`api_username` is the Instaclustr console user that the API keys belong to.  Leave out
`prometheus_api_key` if you did not create one.  The nodes' certificate is publicly trusted, so
`verify_full` needs no `ca_cert`.

Run the collector from the directory that contains both files:

```sh
docker run -d --name dbg-collector --restart unless-stopped \
  -v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \
  --env-file secrets.env \
  dbgorillapublic.azurecr.io/dbg-collector:0.17.0 \
  --config-file /etc/dbg-collector/collector.toml
```

## Additional configuration

### Private addresses

If the collector runs in a network peered with the cluster's VPC, have it connect to the nodes'
private addresses:

```toml
[component.provider]
use_private_addresses = true
```

Set `host` to a node's private host name too.

### Platform metrics

With `prometheus_api_key` set, each node also reports CPU, memory, disk and network, along with
Instaclustr's own read and write probe, backup integrity and ClickHouse Keeper status.

### Commands

RPC Commands allow DBGorilla agents to run commands on your database.  By default commands are disabled.

```toml
[commands]
enabled = true
allowed = ["explain", "execute_query"]
```

[Commands](/docs/connect/clickhouse/#commands) in ClickHouse lists what each command does and the
grant it needs.

## Check it is working

Read the collector log:

```sh
docker logs dbg-collector 2>&1 | grep -E 'opamp handshake complete|discovered component|snapshot serialized|WARN|ERROR'
```

A healthy start has these three messages:

```text
opamp handshake complete
discovered component
topology snapshot serialized for OTLP post
```

The first `discovered component` line carries `"engine":"clickhouse"` and a `"nodes"` count.  The
count is the number of ClickHouse server nodes in the cluster.  Dedicated ClickHouse Keeper nodes
take no queries, so they are not counted.

Then open Home in DBGorilla.  The cluster appears under the name you gave the component, marked
**Monitored**, as ClickHouse with its version.  Its databases and tables arrive first, then server
metrics.  The workload arrives after the cluster has taken queries.  The queries Instaclustr's own
monitoring runs are left out of it.

If nothing arrives, the log names the cause:

| Log line | Fix |
|---|---|
| `unsupported combination (engine=clickhouse, provider=instaclustr, auth=password)` | The collector is older than 0.17.0.  Use 0.17.0 or later. |
| `instaclustr cluster-detail` | Check `cluster_id`, `api_username` and `IC_READONLY_API_KEY`. |
| `has no addressable nodes` | The nodes have no public addresses.  Set `use_private_addresses = true`. |
| `is PrivateLink-enabled` | The collector cannot monitor a PrivateLink data centre. |
| `Authentication failed` | Check the user name and `INSTACLUSTR_CLICKHOUSE_PASSWORD`. |
| `Not enough privileges` | Run the grants in [Prepare the cluster](#prepare-the-cluster), the `SHOW` grant on every node. |
| A certificate error naming an IP address | Set `host` to the node's host name from **Connection Info**. |

## Upgrade

Change the tag in the `docker run` command to the new release, remove the container with
`docker rm -f dbg-collector`, and run the command again.  The collector reconnects under the same
agent ID.

## Remove

Remove the container with `docker rm -f dbg-collector`.  To delete the cluster from Home, follow
[Removing the database from DBGorilla](/docs/administer/manage-collectors/#removing-the-database-from-dbgorilla)
in Manage collectors.  Then drop the user:

```sql
DROP USER dbg_collector ON CLUSTER '<cluster id>';
```

## Next

- [Topology](/docs/using-dbgorilla/topology/) shows what the collector found.
- [Organization Management](/docs/administer/members-and-roles/) lets you bring in your team.