Instaclustr ClickHouse
Monitor a ClickHouse cluster that NetApp Instaclustr runs for you.
Before you start
Section titled “Before you start”-
Check the ClickHouse version in the Instaclustr console. The collector supports 25.8 and later.
-
Ensure that you have DBGorilla collector 0.17.0 or later
-
Ensure that the collector’s egress IP address is in the cluster’s firewall rules for the HTTPS port, 8443.
-
Create a read-only provisioning API key in the Instaclustr console.
-
For host metrics, also create a Prometheus API key. It is a separate key type.
-
Ensure the collector can reach
otlp.dbgorilla.com,auth.dbgorilla.com,api.instaclustr.comand*.prometheus.monitoring.instaclustr.comon port 443. -
Install Docker on the machine that will run the collector.
-
Add a collector in the DBGorilla web admin interface and record the values provided to include in your collector config file.
-
Record the cluster ID and one node’s host name. Both are on the cluster’s page in the Instaclustr console, the host name under Connection Info.
Prepare the cluster
Section titled “Prepare the cluster”Connect as the default icclickhouse user from Connection Info in the Instaclustr console.
That user is an administrator, so give the collector its own user:
CREATE USER dbg_collector ON CLUSTER '<cluster id>' IDENTIFIED WITH sha256_password BY '<strong password>';GRANT ON CLUSTER '<cluster id>' SELECT ON system.* TO dbg_collector;Then run this statement on every node. It does not take ON CLUSTER:
GRANT CURRENT GRANTS(SHOW ON *.*) TO dbg_collector;Instaclustr keeps a database of its own that icclickhouse cannot grant on, so a plain
GRANT SHOW ON *.* is refused. CURRENT GRANTS passes on everything icclickhouse holds.
Prepare the cluster in ClickHouse lists what each grant adds.
Install
Section titled “Install”Write collector.toml. host is the host name of any one node, as Connection Info shows it.
Use the name and not the IP address: the nodes’ certificate covers their names, and the collector
works out the other nodes’ names from this one.
[dbgorilla]agent_id = "<agent id from the app>"tenant_id = "<tenant id from the app>"secret = "${DBG_SERVER_SECRET}"
[[component]]name = "events-instaclustr"engine = "clickhouse"
[component.provider]type = "instaclustr"cluster_id = "<cluster id>"api_username = "you@example.com"api_key = { env = "IC_READONLY_API_KEY" }prometheus_api_key = { env = "IC_PROMETHEUS_API_KEY" }
[component.auth]method = "password"user = "dbg_collector"password = { env = "INSTACLUSTR_CLICKHOUSE_PASSWORD" }
[component.connect]host = "ip-203-0-113-20.<cluster hash>.cnodes.io"port = 8443ssl_mode = "verify_full"api_username is the Instaclustr console user that the API keys belong to. Leave out
prometheus_api_key if you did not create one. The nodes’ certificate is publicly trusted, so
verify_full needs no ca_cert.
Run the collector from the directory that contains both files:
docker run -d --name dbg-collector --restart unless-stopped \ -v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \ --env-file secrets.env \ dbgorillapublic.azurecr.io/dbg-collector:0.17.0 \ --config-file /etc/dbg-collector/collector.tomlAdditional configuration
Section titled “Additional configuration”Private addresses
Section titled “Private addresses”If the collector runs in a network peered with the cluster’s VPC, have it connect to the nodes’ private addresses:
[component.provider]use_private_addresses = trueSet host to a node’s private host name too.
Platform metrics
Section titled “Platform metrics”With prometheus_api_key set, each node also reports CPU, memory, disk and network, along with
Instaclustr’s own read and write probe, backup integrity and ClickHouse Keeper status.
Commands
Section titled “Commands”RPC Commands allow DBGorilla agents to run commands on your database. By default commands are disabled.
[commands]enabled = trueallowed = ["explain", "execute_query"]Commands in ClickHouse lists what each command does and the grant it needs.
Check it is working
Section titled “Check it is working”Read the collector log:
docker logs dbg-collector 2>&1 | grep -E 'opamp handshake complete|discovered component|snapshot serialized|WARN|ERROR'A healthy start has these three messages:
opamp handshake completediscovered componenttopology snapshot serialized for OTLP postThe first discovered component line carries "engine":"clickhouse" and a "nodes" count. The
count is the number of ClickHouse server nodes in the cluster. Dedicated ClickHouse Keeper nodes
take no queries, so they are not counted.
Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked Monitored, as ClickHouse with its version. Its databases and tables arrive first, then server metrics. The workload arrives after the cluster has taken queries. The queries Instaclustr’s own monitoring runs are left out of it.
If nothing arrives, the log names the cause:
| Log line | Fix |
|---|---|
unsupported combination (engine=clickhouse, provider=instaclustr, auth=password) | The collector is older than 0.17.0. Use 0.17.0 or later. |
instaclustr cluster-detail | Check cluster_id, api_username and IC_READONLY_API_KEY. |
has no addressable nodes | The nodes have no public addresses. Set use_private_addresses = true. |
is PrivateLink-enabled | The collector cannot monitor a PrivateLink data centre. |
Authentication failed | Check the user name and INSTACLUSTR_CLICKHOUSE_PASSWORD. |
Not enough privileges | Run the grants in Prepare the cluster, the SHOW grant on every node. |
| A certificate error naming an IP address | Set host to the node’s host name from Connection Info. |
Upgrade
Section titled “Upgrade”Change the tag in the docker run command to the new release, remove the container with
docker rm -f dbg-collector, and run the command again. The collector reconnects under the same
agent ID.
Remove
Section titled “Remove”Remove the container with docker rm -f dbg-collector. To delete the cluster from Home, follow
Removing the database from DBGorilla
in Manage collectors. Then drop the user:
DROP USER dbg_collector ON CLUSTER '<cluster id>';- Topology shows what the collector found.
- Organization Management lets you bring in your team.