Apache Cassandra
Monitor Apache Cassandra 4.0, 4.1 or 5.0 that you run yourself. You create one
read-only role, write collector.toml by hand, and run the collector in a Docker container that
can reach every node. The collector installs nothing on the nodes.
NetApp Instaclustr managed Cassandra is not covered here.
Before you start
Section titled “Before you start”-
Check the version on any node. The collector supports 4.0, 4.1 and 5.0.
SELECT release_version FROM system.local;Cassandra 3.11 is refused during discovery, and nothing is sent:
Cassandra 3.11.19 is not supported: the collector needs 4.0 or later (3.x has no virtual tables and reached end of life with 5.0) -
Turn on password authentication and authorization in
cassandra.yaml, if they are not already enabled. The grants below require the authorizer. Changing either setting requires a restart of each node.authenticator: PasswordAuthenticatorauthorizer: CassandraAuthorizer -
Let the collector reach port 9042 on every node at the address that each node advertises to clients. You give the collector one node. It finds the others and connects to each of them directly.
-
Let the collector reach
otlp.dbgorilla.comandauth.dbgorilla.comon port 443. See What the collector is in Choose your setup. -
Install Docker on the machine that will run the collector.
-
Add a collector in the app, and keep the agent ID, tenant ID and secret it shows. The secret is shown only once.
Prepare the cluster
Section titled “Prepare the cluster”Create the role as a superuser on any node. Roles and grants apply to the whole cluster.
CREATE ROLE dbg_collector WITH LOGIN = true AND PASSWORD = '<strong password>';GRANT SELECT ON KEYSPACE system_views TO dbg_collector;GRANT SELECT ON TABLE system.prepared_statements TO dbg_collector;GRANT DESCRIBE ON ALL ROLES TO dbg_collector;None of these grants read your tables. system_views contains node statistics, and
system.prepared_statements contains the text of the statements that your applications prepare.
| Grant | What it adds | Without it |
|---|---|---|
SELECT ON KEYSPACE system_views | Node metrics, the workload, in-flight requests and node settings | Only the keyspaces, tables and node uptime arrive. The collector log names the missing grant: User dbg_collector has no SELECT permission on <table system_views.thread_pools>. |
SELECT ON TABLE system.prepared_statements | The statement text behind each workload entry | The workload lists each table operation, such as READ shop.customers, without its statements. |
DESCRIBE ON ALL ROLES | Every role in the topology | The topology lists only dbg_collector. |
With the login alone, the collector still reads the schema: keyspaces, tables, columns, secondary indexes and user-defined types.
If the keyspace grant fails
Section titled “If the keyspace grant fails”Some older patch releases refuse the keyspace grant, including 5.0.4:
Resource <keyspace system_views> doesn't existGrant each table instead:
GRANT SELECT ON TABLE system_views.thread_pools TO dbg_collector;GRANT SELECT ON TABLE system_views.caches TO dbg_collector;GRANT SELECT ON TABLE system_views.internode_inbound TO dbg_collector;GRANT SELECT ON TABLE system_views.internode_outbound TO dbg_collector;GRANT SELECT ON TABLE system_views.clients TO dbg_collector;GRANT SELECT ON TABLE system_views.sstable_tasks TO dbg_collector;GRANT SELECT ON TABLE system_views.pending_hints TO dbg_collector;GRANT SELECT ON TABLE system_views.cql_metrics TO dbg_collector;GRANT SELECT ON TABLE system_views.queries TO dbg_collector;GRANT SELECT ON TABLE system_views.coordinator_read_latency TO dbg_collector;GRANT SELECT ON TABLE system_views.coordinator_write_latency TO dbg_collector;GRANT SELECT ON TABLE system_views.coordinator_scan_latency TO dbg_collector;GRANT SELECT ON TABLE system_views.local_read_latency TO dbg_collector;GRANT SELECT ON TABLE system_views.local_write_latency TO dbg_collector;GRANT SELECT ON TABLE system_views.local_scan_latency TO dbg_collector;GRANT SELECT ON TABLE system_views.disk_usage TO dbg_collector;GRANT SELECT ON TABLE system_views.max_partition_size TO dbg_collector;GRANT SELECT ON TABLE system_views.tombstones_per_read TO dbg_collector;GRANT SELECT ON TABLE system_views.settings TO dbg_collector;GRANT SELECT ON TABLE system_views.gossip_info TO dbg_collector;Install
Section titled “Install”Write collector.toml. host can be any one node.
[dbgorilla]agent_id = "<agent id from the app>"tenant_id = "<tenant id from the app>"secret = "${DBG_SERVER_SECRET}"
[[component]]name = "orders-ring"engine = "cassandra"
[component.provider]type = "self_hosted"
[component.auth]method = "password"user = "dbg_collector"password = { env = "CASSANDRA_COLLECTOR_PASSWORD" }
[component.connect]host = "cass-1.internal.example.com"port = 9042ssl_mode = "disable"Keep the secrets in a separate file with mode 0600, and keep that file out of version control:
DBG_SERVER_SECRET=your-client-secretCASSANDRA_COLLECTOR_PASSWORD=your-role-passwordRun the collector from the directory that contains both files:
docker run -d --name dbg-collector --restart unless-stopped \ -v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \ --env-file secrets.env \ dbgorillapublic.azurecr.io/dbg-collector:0.16.0 \ --config-file /etc/dbg-collector/collector.tomlCassandra requires collector 0.16.0 or later. Pin the release tag as shown here so that a restart cannot change the running version.
Client encryption
Section titled “Client encryption”If the cluster has client_encryption_options enabled, set ssl_mode and mount the CA that signed
the node certificates:
[component.connect]host = "cass-1.internal.example.com"port = 9042ssl_mode = "verify_full"ca_cert = "/etc/dbgorilla/cassandra-ca.pem"docker run -d --name dbg-collector --restart unless-stopped \ -v "$PWD/collector.toml:/etc/dbg-collector/collector.toml:ro" \ -v "$PWD/cassandra-ca.pem:/etc/dbgorilla/cassandra-ca.pem:ro" \ --env-file secrets.env \ dbgorillapublic.azurecr.io/dbg-collector:0.16.0 \ --config-file /etc/dbg-collector/collector.tomlssl_mode | What it checks |
|---|---|
disable | Nothing. The default. Use it only when client encryption is off. |
require | Encrypts, but does not check the certificate. |
verify_ca | The certificate chains to ca_cert. The name in it is not checked. ca_cert is required. |
verify_full | The chain, and that the certificate names the address the collector connects to. Without ca_cert, the image’s own trust store is used. |
The collector connects to each node at the address that the node advertises, usually an IP
address. With verify_full, every node’s certificate needs that address as a subject alternative
name. If the certificates contain only DNS names, use verify_ca. Otherwise, the seed connects
and every node then fails with:
x509: cannot validate certificate for 10.250.50.10 because it doesn't contain any IP SANsIf the cluster sets require_client_auth: true, add a client certificate and its key, and mount
them in the same way:
client_cert = "/etc/dbgorilla/collector.pem"client_key = "/etc/dbgorilla/collector.key"Commands
Section titled “Commands”Cassandra takes one command, execute_query: a single CQL SELECT with at most 1,000 rows and no
ALLOW FILTERING. It reads only the keyspaces granted to the role. To allow it, add this to
collector.toml:
[commands]enabled = trueallowed = ["execute_query"]Then grant SELECT on each keyspace it may read:
GRANT SELECT ON KEYSPACE shop TO dbg_collector;explain and collect_statistics do not apply to Cassandra. If they are listed, the collector
leaves them out.
Host metrics
Section titled “Host metrics”For CPU, memory and disk metrics, run Prometheus node_exporter on each node on its default port
9100. The collector finds it there. See node_exporter
in Self-managed server.
Check it is working
Section titled “Check it is working”Read the collector log:
docker logs dbg-collector 2>&1 | grep -E 'opamp handshake complete|discovered component|snapshot serialized|WARN|ERROR'A healthy start has these three messages:
opamp handshake completediscovered componenttopology snapshot serialized for OTLP postThe first discovered component line carries "engine":"cassandra" and a "nodes" count. The
count is the number of nodes in the cluster.
Then open Home in DBGorilla. The cluster appears under the name you gave the component, marked
Monitored, with its engine and version, such as Cassandra 5.0.9. Its keyspaces and tables
arrive within a minute of startup, and node metrics arrive within two. The workload arrives within
a minute of the cluster taking traffic. Each workload entry is a table operation, such as
READ shop.customers, with its call count, mean time and the statements behind it.
In-flight requests and the workload’s times require Cassandra 4.1.5 or later. On 4.0, each workload entry has its call count and statements, and node metrics arrive as usual.
If nothing arrives, the log names the cause:
| Log line | Fix |
|---|---|
Cassandra 3.11.19 is not supported | Upgrade to 4.0 or later. |
unsupported combination (engine=cassandra, provider=self_hosted, auth=password) | The collector is older than 0.16.0. Use 0.16.0 or later. |
bad credentials (0x0100): Provided username dbg_collector and/or password are incorrect | Check the role name and CASSANDRA_COLLECTOR_PASSWORD. |
no SELECT permission on <table system_views.thread_pools> | Run the grants in Prepare the cluster. |
read the response header: Connection reset by peer | Client encryption is on. Set ssl_mode. |
invalid peer certificate: UnknownIssuer | Set ca_cert to the CA that signed the node certificates. |
x509: cannot validate certificate for (an IP address) | Use verify_ca, or reissue the node certificates with their IP addresses. |
received fatal alert: CertificateRequired | The cluster requires client certificates. Set client_cert and client_key. |
Upgrade
Section titled “Upgrade”Change the tag in the docker run command to the new release, then replace the container:
docker pull dbgorillapublic.azurecr.io/dbg-collector:<new version>docker rm -f dbg-collector# run the docker run command again, with the new tagThe agent ID and secret stay the same, so it reconnects as the same collector.
Remove
Section titled “Remove”docker rm -f dbg-collectorThe cluster stays on Home, marked Not monitored, with its history, until you delete it. To delete it, follow Removing the database from DBGorilla in Manage collectors.
Then drop the role on any node:
DROP ROLE dbg_collector;- Topology shows what the collector found.
- Organization Management lets you bring in your team.