Skip to content

Apache Cassandra

Last updated

View as Markdown

Monitor Apache Cassandra 4.0, 4.1 or 5.0 that you run yourself. You create one read-only role, write collector.toml by hand, and run the collector in a Docker container that can reach every node. The collector installs nothing on the nodes.

NetApp Instaclustr managed Cassandra is not covered here.

  • Check the version on any node. The collector supports 4.0, 4.1 and 5.0.

    SELECT release_version FROM system.local;

    Cassandra 3.11 is refused during discovery, and nothing is sent:

    Cassandra 3.11.19 is not supported: the collector needs 4.0 or later (3.x has no virtual tables and reached end of life with 5.0)
  • Turn on password authentication and authorization in cassandra.yaml, if they are not already enabled. The grants below require the authorizer. Changing either setting requires a restart of each node.

    authenticator: PasswordAuthenticator
    authorizer: CassandraAuthorizer
  • Let the collector reach port 9042 on every node at the address that each node advertises to clients. You give the collector one node. It finds the others and connects to each of them directly.

  • Let the collector reach otlp.dbgorilla.com and auth.dbgorilla.com on port 443. See What the collector is in Choose your setup.

  • Install Docker on the machine that will run the collector.

  • Add a collector in the app, and keep the agent ID, tenant ID and secret it shows. The secret is shown only once.

Create the role as a superuser on any node. Roles and grants apply to the whole cluster.

CREATE ROLE dbg_collector WITH LOGIN = true AND PASSWORD = '<strong password>';
GRANT SELECT ON KEYSPACE system_views TO dbg_collector;
GRANT SELECT ON TABLE system.prepared_statements TO dbg_collector;
GRANT DESCRIBE ON ALL ROLES TO dbg_collector;

None of these grants read your tables. system_views contains node statistics, and system.prepared_statements contains the text of the statements that your applications prepare.

GrantWhat it addsWithout it
SELECT ON KEYSPACE system_viewsNode metrics, the workload, in-flight requests and node settingsOnly the keyspaces, tables and node uptime arrive. The collector log names the missing grant: User dbg_collector has no SELECT permission on <table system_views.thread_pools>.
SELECT ON TABLE system.prepared_statementsThe statement text behind each workload entryThe workload lists each table operation, such as READ shop.customers, without its statements.
DESCRIBE ON ALL ROLESEvery role in the topologyThe topology lists only dbg_collector.

With the login alone, the collector still reads the schema: keyspaces, tables, columns, secondary indexes and user-defined types.

Some older patch releases refuse the keyspace grant, including 5.0.4:

Resource <keyspace system_views> doesn't exist

Grant each table instead:

GRANT SELECT ON TABLE system_views.thread_pools TO dbg_collector;
GRANT SELECT ON TABLE system_views.caches TO dbg_collector;
GRANT SELECT ON TABLE system_views.internode_inbound TO dbg_collector;
GRANT SELECT ON TABLE system_views.internode_outbound TO dbg_collector;
GRANT SELECT ON TABLE system_views.clients TO dbg_collector;
GRANT SELECT ON TABLE system_views.sstable_tasks TO dbg_collector;
GRANT SELECT ON TABLE system_views.pending_hints TO dbg_collector;
GRANT SELECT ON TABLE system_views.cql_metrics TO dbg_collector;
GRANT SELECT ON TABLE system_views.queries TO dbg_collector;
GRANT SELECT ON TABLE system_views.coordinator_read_latency TO dbg_collector;
GRANT SELECT ON TABLE system_views.coordinator_write_latency TO dbg_collector;
GRANT SELECT ON TABLE system_views.coordinator_scan_latency TO dbg_collector;
GRANT SELECT ON TABLE system_views.local_read_latency TO dbg_collector;
GRANT SELECT ON TABLE system_views.local_write_latency TO dbg_collector;
GRANT SELECT ON TABLE system_views.local_scan_latency TO dbg_collector;
GRANT SELECT ON TABLE system_views.disk_usage TO dbg_collector;
GRANT SELECT ON TABLE system_views.max_partition_size TO dbg_collector;
GRANT SELECT ON TABLE system_views.tombstones_per_read TO dbg_collector;
GRANT SELECT ON TABLE system_views.settings TO dbg_collector;
GRANT SELECT ON TABLE system_views.gossip_info TO dbg_collector;

Write collector.toml. host can be any one node.

[dbgorilla]
agent_id = "<agent id from the app>"
tenant_id = "<tenant id from the app>"
secret = "${DBG_SERVER_SECRET}"
[[component]]
name = "orders-ring"
engine = "cassandra"
[component.provider]
type = "self_hosted"
[component.auth]
method = "password"
user = "dbg_collector"
password = { env = "CASSANDRA_COLLECTOR_PASSWORD" }
[component.connect]
host = "cass-1.internal.example.com"
port = 9042
ssl_mode = "disable"

Keep the secrets in a separate file with mode 0600, and keep that file out of version control:

secrets.env
DBG_SERVER_SECRET=your-client-secret
CASSANDRA_COLLECTOR_PASSWORD=your-role-password

Run the collector from the directory that contains both files:

Terminal window
docker run -d --name dbg-collector --restart unless-stopped \
-v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \
--env-file secrets.env \
dbgorillapublic.azurecr.io/dbg-collector:0.16.0 \
--config-file /etc/dbg-collector/collector.toml

Cassandra requires collector 0.16.0 or later. Pin the release tag as shown here so that a restart cannot change the running version.

If the cluster has client_encryption_options enabled, set ssl_mode and mount the CA that signed the node certificates:

[component.connect]
host = "cass-1.internal.example.com"
port = 9042
ssl_mode = "verify_full"
ca_cert = "/etc/dbgorilla/cassandra-ca.pem"
Terminal window
docker run -d --name dbg-collector --restart unless-stopped \
-v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \
-v "$PWD/cassandra-ca.pem:/etc/dbgorilla/cassandra-ca.pem:ro" \
--env-file secrets.env \
dbgorillapublic.azurecr.io/dbg-collector:0.16.0 \
--config-file /etc/dbg-collector/collector.toml
ssl_modeWhat it checks
disableNothing. The default. Use it only when client encryption is off.
requireEncrypts, but does not check the certificate.
verify_caThe certificate chains to ca_cert. The name in it is not checked. ca_cert is required.
verify_fullThe chain, and that the certificate names the address the collector connects to. Without ca_cert, the image’s own trust store is used.

The collector connects to each node at the address that the node advertises, usually an IP address. With verify_full, every node’s certificate needs that address as a subject alternative name. If the certificates contain only DNS names, use verify_ca. Otherwise, the seed connects and every node then fails with:

x509: cannot validate certificate for 10.250.50.10 because it doesn't contain any IP SANs

If the cluster sets require_client_auth: true, add a client certificate and its key, and mount them in the same way:

client_cert = "/etc/dbgorilla/collector.pem"
client_key = "/etc/dbgorilla/collector.key"

Cassandra takes one command, execute_query: a single CQL SELECT with at most 1,000 rows and no ALLOW FILTERING. It reads only the keyspaces granted to the role. To allow it, add this to collector.toml:

[commands]
enabled = true
allowed = ["execute_query"]

Then grant SELECT on each keyspace it may read:

GRANT SELECT ON KEYSPACE shop TO dbg_collector;

explain and collect_statistics do not apply to Cassandra. If they are listed, the collector leaves them out.

For CPU, memory and disk metrics, run Prometheus node_exporter on each node on its default port 9100. The collector finds it there. See node_exporter in Self-managed server.

Read the collector log:

Terminal window
docker logs dbg-collector 2>&1 | grep -E 'opamp handshake complete|discovered component|snapshot serialized|WARN|ERROR'

A healthy start has these three messages:

opamp handshake complete
discovered component
topology snapshot serialized for OTLP post

The first discovered component line carries "engine":"cassandra" and a "nodes" count. The count is the number of nodes in the cluster.

Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked Monitored, with its engine and version, such as Cassandra 5.0.9. Its keyspaces and tables arrive within a minute of startup, and node metrics arrive within two. The workload arrives within a minute of the cluster taking traffic. Each workload entry is a table operation, such as READ shop.customers, with its call count, mean time and the statements behind it.

In-flight requests and the workload’s times require Cassandra 4.1.5 or later. On 4.0, each workload entry has its call count and statements, and node metrics arrive as usual.

If nothing arrives, the log names the cause:

Log lineFix
Cassandra 3.11.19 is not supportedUpgrade to 4.0 or later.
unsupported combination (engine=cassandra, provider=self_hosted, auth=password)The collector is older than 0.16.0. Use 0.16.0 or later.
bad credentials (0x0100): Provided username dbg_collector and/or password are incorrectCheck the role name and CASSANDRA_COLLECTOR_PASSWORD.
no SELECT permission on <table system_views.thread_pools>Run the grants in Prepare the cluster.
read the response header: Connection reset by peerClient encryption is on. Set ssl_mode.
invalid peer certificate: UnknownIssuerSet ca_cert to the CA that signed the node certificates.
x509: cannot validate certificate for (an IP address)Use verify_ca, or reissue the node certificates with their IP addresses.
received fatal alert: CertificateRequiredThe cluster requires client certificates. Set client_cert and client_key.

Change the tag in the docker run command to the new release, then replace the container:

Terminal window
docker pull dbgorillapublic.azurecr.io/dbg-collector:<new version>
docker rm -f dbg-collector
# run the docker run command again, with the new tag

The agent ID and secret stay the same, so it reconnects as the same collector.

Terminal window
docker rm -f dbg-collector

The cluster stays on Home, marked Not monitored, with its history, until you delete it. To delete it, follow Removing the database from DBGorilla in Manage collectors.

Then drop the role on any node:

DROP ROLE dbg_collector;