Skip to content

Instaclustr ClickHouse

Last updated

View as Markdown

Monitor a ClickHouse cluster that NetApp Instaclustr runs for you.

  • Check the ClickHouse version in the Instaclustr console. The collector supports 25.8 and later.

  • Ensure that you have DBGorilla collector 0.17.0 or later

  • Ensure that the collector’s egress IP address is in the cluster’s firewall rules for the HTTPS port, 8443.

  • Create a read-only provisioning API key in the Instaclustr console.

  • For host metrics, also create a Prometheus API key. It is a separate key type.

  • Ensure the collector can reach otlp.dbgorilla.com, auth.dbgorilla.com, api.instaclustr.com and *.prometheus.monitoring.instaclustr.com on port 443.

  • Install Docker on the machine that will run the collector.

  • Add a collector in the DBGorilla web admin interface and record the values provided to include in your collector config file.

  • Record the cluster ID and one node’s host name. Both are on the cluster’s page in the Instaclustr console, the host name under Connection Info.

Connect as the default icclickhouse user from Connection Info in the Instaclustr console. That user is an administrator, so give the collector its own user:

CREATE USER dbg_collector ON CLUSTER '<cluster id>' IDENTIFIED WITH sha256_password BY '<strong password>';
GRANT ON CLUSTER '<cluster id>' SELECT ON system.* TO dbg_collector;

Then run this statement on every node. It does not take ON CLUSTER:

GRANT CURRENT GRANTS(SHOW ON *.*) TO dbg_collector;

Instaclustr keeps a database of its own that icclickhouse cannot grant on, so a plain GRANT SHOW ON *.* is refused. CURRENT GRANTS passes on everything icclickhouse holds.

Prepare the cluster in ClickHouse lists what each grant adds.

Write collector.toml. host is the host name of any one node, as Connection Info shows it. Use the name and not the IP address: the nodes’ certificate covers their names, and the collector works out the other nodes’ names from this one.

[dbgorilla]
agent_id = "<agent id from the app>"
tenant_id = "<tenant id from the app>"
secret = "${DBG_SERVER_SECRET}"
[[component]]
name = "events-instaclustr"
engine = "clickhouse"
[component.provider]
type = "instaclustr"
cluster_id = "<cluster id>"
api_username = "you@example.com"
api_key = { env = "IC_READONLY_API_KEY" }
prometheus_api_key = { env = "IC_PROMETHEUS_API_KEY" }
[component.auth]
method = "password"
user = "dbg_collector"
password = { env = "INSTACLUSTR_CLICKHOUSE_PASSWORD" }
[component.connect]
host = "ip-203-0-113-20.<cluster hash>.cnodes.io"
port = 8443
ssl_mode = "verify_full"

api_username is the Instaclustr console user that the API keys belong to. Leave out prometheus_api_key if you did not create one. The nodes’ certificate is publicly trusted, so verify_full needs no ca_cert.

Run the collector from the directory that contains both files:

Terminal window
docker run -d --name dbg-collector --restart unless-stopped \
-v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \
--env-file secrets.env \
dbgorillapublic.azurecr.io/dbg-collector:0.17.0 \
--config-file /etc/dbg-collector/collector.toml

If the collector runs in a network peered with the cluster’s VPC, have it connect to the nodes’ private addresses:

[component.provider]
use_private_addresses = true

Set host to a node’s private host name too.

With prometheus_api_key set, each node also reports CPU, memory, disk and network, along with Instaclustr’s own read and write probe, backup integrity and ClickHouse Keeper status.

RPC Commands allow DBGorilla agents to run commands on your database. By default commands are disabled.

[commands]
enabled = true
allowed = ["explain", "execute_query"]

Commands in ClickHouse lists what each command does and the grant it needs.

Read the collector log:

Terminal window
docker logs dbg-collector 2>&1 | grep -E 'opamp handshake complete|discovered component|snapshot serialized|WARN|ERROR'

A healthy start has these three messages:

opamp handshake complete
discovered component
topology snapshot serialized for OTLP post

The first discovered component line carries "engine":"clickhouse" and a "nodes" count. The count is the number of ClickHouse server nodes in the cluster. Dedicated ClickHouse Keeper nodes take no queries, so they are not counted.

Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked Monitored, as ClickHouse with its version. Its databases and tables arrive first, then server metrics. The workload arrives after the cluster has taken queries. The queries Instaclustr’s own monitoring runs are left out of it.

If nothing arrives, the log names the cause:

Log lineFix
unsupported combination (engine=clickhouse, provider=instaclustr, auth=password)The collector is older than 0.17.0. Use 0.17.0 or later.
instaclustr cluster-detailCheck cluster_id, api_username and IC_READONLY_API_KEY.
has no addressable nodesThe nodes have no public addresses. Set use_private_addresses = true.
is PrivateLink-enabledThe collector cannot monitor a PrivateLink data centre.
Authentication failedCheck the user name and INSTACLUSTR_CLICKHOUSE_PASSWORD.
Not enough privilegesRun the grants in Prepare the cluster, the SHOW grant on every node.
A certificate error naming an IP addressSet host to the node’s host name from Connection Info.

Change the tag in the docker run command to the new release, remove the container with docker rm -f dbg-collector, and run the command again. The collector reconnects under the same agent ID.

Remove the container with docker rm -f dbg-collector. To delete the cluster from Home, follow Removing the database from DBGorilla in Manage collectors. Then drop the user:

DROP USER dbg_collector ON CLUSTER '<cluster id>';