Skip to content

Instaclustr Cassandra

Last updated

View as Markdown

Monitor an Apache Cassandra cluster that NetApp Instaclustr runs for you. The collector finds the cluster’s nodes through the Instaclustr API, connects to each node over CQL with a read-only role, and, if you give it the Prometheus API key, adds each node’s host and JVM metrics from the Instaclustr platform.

  • Check the Cassandra version in the Instaclustr console. The collector supports 4.0, 4.1 and 5.0.

  • The collector must reach the cluster nodes directly.

  • Add the collector’s egress IP address to the cluster’s firewall, under Cassandra Allowed Addresses. The collector connects to port 9042 on every node, at each node’s public address. If the collector reaches the cluster over VPC peering instead, it uses the private addresses; see Private addresses.

  • Create a read-only provisioning API key in the Instaclustr console. The collector uses it to list the cluster’s nodes, and for nothing else.

  • For host metrics, also create a Prometheus API key. It is a separate key type; a provisioning key does not work in its place.

  • Let the collector reach otlp.dbgorilla.com and auth.dbgorilla.com on port 443, and also api.instaclustr.com and *.prometheus.monitoring.instaclustr.com on port 443.

  • Install Docker on the machine that will run the collector.

  • Add a collector in the app, and keep the agent ID, tenant ID and secret it shows. The secret is shown only once.

  • Keep the cluster ID. It is on the cluster’s page in the Instaclustr console.

Connect with cqlsh as the default iccassandra user from Connection Info in the Instaclustr console. That user is a superuser, so give the collector its own role:

CREATE ROLE dbg_collector WITH LOGIN = true AND PASSWORD = '<strong password>';
GRANT SELECT ON KEYSPACE system_views TO dbg_collector;
GRANT SELECT ON TABLE system.prepared_statements TO dbg_collector;
GRANT DESCRIBE ON ALL ROLES TO dbg_collector;

Prepare the cluster in Apache Cassandra lists what each one adds, and what to grant instead if the keyspace grant fails.

Write collector.toml. host can be the public address of any one node; the collector finds the others through the Instaclustr API.

[dbgorilla]
agent_id = "<agent id from the app>"
tenant_id = "<tenant id from the app>"
secret = "${DBG_SERVER_SECRET}"
[[component]]
name = "orders-instaclustr"
engine = "cassandra"
[component.provider]
type = "instaclustr"
cluster_id = "<cluster id>"
api_username = "you@example.com"
api_key = { env = "IC_READONLY_API_KEY" }
prometheus_api_key = { env = "IC_PROMETHEUS_API_KEY" }
[component.auth]
method = "password"
user = "dbg_collector"
password = { env = "INSTACLUSTR_CASSANDRA_PASSWORD" }
[component.connect]
host = "203.0.113.20"
port = 9042
ssl_mode = "disable"

api_username is the Instaclustr console user that the API keys belong to. Leave out prometheus_api_key if you did not create one.

Put the secrets in secrets.env, next to collector.toml:

Terminal window
DBG_SERVER_SECRET=<secret from the app>
IC_READONLY_API_KEY=<provisioning API key>
IC_PROMETHEUS_API_KEY=<Prometheus API key>
INSTACLUSTR_CASSANDRA_PASSWORD=<dbg_collector password>

Start the collector:

Terminal window
docker run -d --name dbg-collector --restart unless-stopped \
-v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \
--env-file secrets.env \
dbgorillapublic.azurecr.io/dbg-collector:0.16.0 \
--config-file /etc/dbg-collector/collector.toml

If the cluster has client-to-node encryption turned on, download the cluster’s CA certificate from Connection Info in the Instaclustr console, set ssl_mode, and mount the certificate:

[component.connect]
host = "203.0.113.20"
port = 9042
ssl_mode = "verify_full"
ca_cert = "/etc/dbgorilla/instaclustr-cluster-ca.pem"

Then start the collector with the certificate mounted:

Terminal window
docker run -d --name dbg-collector --restart unless-stopped \
-v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \
-v "$PWD/instaclustr-cluster-ca.pem:/etc/dbgorilla/instaclustr-cluster-ca.pem:ro" \
--env-file secrets.env \
dbgorillapublic.azurecr.io/dbg-collector:0.16.0 \
--config-file /etc/dbg-collector/collector.toml

The ssl_mode values are the same as for self-managed Cassandra; see Client encryption in Apache Cassandra.

If the collector runs in a network peered with the cluster’s VPC, have it connect to the nodes’ private addresses:

[component.provider]
use_private_addresses = true

Set host to a node’s private address too.

With prometheus_api_key set, each node also reports CPU, memory, disk and network, JVM heap and garbage collection, compactions and repairs, and the partition count and partition sizes of each table. These come from the Instaclustr platform, so node_exporter is not needed. Without the key, the cluster’s schema, workload and node metrics still arrive.

If you would like users in the DBGorilla UI to be able to run queries against the live cluster, that capability MUST be enabled explicitly in the collector config. DBGorilla AI agents will never have the ability to execute queries regardless of configuration.

[commands]
enabled = true
allowed = ["execute_query"]

Then grant SELECT on each keyspace it may read:

GRANT SELECT ON KEYSPACE shop TO dbg_collector;

Read the collector log:

Terminal window
docker logs dbg-collector 2>&1 | grep -E 'opamp handshake complete|discovered component|snapshot serialized|WARN|ERROR'

A healthy start has these three messages:

opamp handshake complete
discovered component
topology snapshot serialized for OTLP post

The first discovered component line carries "engine":"cassandra" and a "nodes" count. The count is the number of Cassandra nodes in the cluster.

Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked Monitored, with its engine and version, such as Cassandra 5.0.9. Its keyspaces and tables arrive within a minute of startup, and node metrics arrive within two. The workload arrives within a minute of the cluster taking traffic. Instaclustr’s own instaclustr keyspace is left out.

If nothing arrives, the log names the cause:

Log lineFix
unsupported combination (engine=cassandra, provider=instaclustr, auth=password)The collector is older than 0.16.0. Use 0.16.0 or later.
instaclustr cluster-detailCheck cluster_id, api_username and IC_READONLY_API_KEY.
has no addressable nodesThe nodes have no public addresses. Set use_private_addresses = true.
is PrivateLink-enabledThe collector cannot monitor a PrivateLink data centre.
bad credentials (0x0100)Check the role name and INSTACLUSTR_CASSANDRA_PASSWORD.
no SELECT permission on <table system_views.thread_pools>Run the grants in Prepare the cluster.
read the response header: Connection reset by peerClient-to-node encryption is on. Set ssl_mode.
seed identify timed outThe collector’s address is not in Cassandra Allowed Addresses.

Change the tag in the docker run command to the new release, then replace the container:

Terminal window
docker pull dbgorillapublic.azurecr.io/dbg-collector:<new version>
docker rm -f dbg-collector
# run the docker run command again, with the new tag

The agent ID and secret stay the same, so it reconnects as the same collector.

Terminal window
docker rm -f dbg-collector

The cluster stays on Home, marked Not monitored, with its history, until you delete it. To delete it, follow Removing the database from DBGorilla in Manage collectors.

Then drop the role on any node:

DROP ROLE dbg_collector;