Skip to main content
This topic explains how to use LaunchDarkly observability features to monitor a ClickHouse database, and optionally a PostgreSQL database. You run an OpenTelemetry (OTel) Collector that queries your database’s system tables once a minute and sends the results to LaunchDarkly as observability metrics. You can then graph the metrics, alert on them, and use them when you investigate slow queries alongside your traces, logs, and errors.Database monitoring does not require an agent inside your database. The Collector connects as a read-only user and runs a fixed set of queries against system tables. Your application data never leaves your network.

​
Prerequisites

To use database monitoring, you need:
  • A LaunchDarkly project key for your target environment.
  • A ClickHouse server with the system.query_log and system.part_log tables enabled. Older ClickHouse versions lack some of the server metrics listed below. The Collector omits any metric your version does not report.
  • (Optional) A PostgreSQL server, if you also want lock monitoring.
  • A host that can run the OpenTelemetry Collector Contrib container image and reach both your database and the LaunchDarkly OpenTelemetry endpoint.
To learn how to find and copy your LaunchDarkly project key, read Project keys.

​
How database monitoring works

The OpenTelemetry Collector Contrib distribution includes the SQL query receiver. The receiver connects to a database, runs the SQL you give it on a schedule, and turns each result row into OpenTelemetry metric data points. LaunchDarkly provides a Collector configuration that uses this receiver to collect metrics from ClickHouse system tables.Every 60 seconds, the Collector:
  1. Reads system.query_log for the queries that finished in the previous minute, and groups them by the table they selected from or inserted into.
  2. Reads system.part_log for the merges that completed in the previous minute, grouped by table.
  3. Reads system.parts for the active part count and primary key memory of your largest tables.
  4. Reads system.metrics, system.asynchronous_metrics, and system.events for server-wide memory, cache, and throughput counters.
  5. Stamps every metric with your LaunchDarkly project key and sends it to the LaunchDarkly OpenTelemetry endpoint.
The query log window ends 15 seconds before the scrape runs. ClickHouse flushes system.query_log a few seconds after each query finishes. The delay keeps the Collector from missing queries that are still waiting to be flushed.

​
Metrics collected

Database monitoring sends the following metrics to LaunchDarkly. All ClickHouse metrics are prefixed with clickhouse.. Table-level metrics carry a database attribute and one of select_table, insert_table, merge_table, or part_table.

​
Select query metrics

These metrics describe the SELECT queries that finished in each 60-second window, grouped by the table they read. A query that reads several tables is counted on each table’s series. The attribute is select_table, and the value includes the database prefix, for example default.events.
MetricUnitDescription
clickhouse.select.queriescountSelects that finished in the window.
clickhouse.select.selected_rows, clickhouse.select.selected_bytescount, bytesRows and bytes read to answer the queries.
clickhouse.select.selected_marks, clickhouse.select.selected_partscountIndex marks and data parts the queries had to scan. A high mark count relative to rows returned suggests the primary key is not filtering the query.
clickhouse.select.primary_key_timemicrosecondsTime spent filtering marks with the primary key.
clickhouse.select.loaded_marks_bytes, clickhouse.select.mark_cache_hits, clickhouse.select.mark_cache_missesbytes, countMark cache activity. A rising miss count means queries are loading index marks from disk.
clickhouse.select.disk_read_time, clickhouse.select.read_compressed_bytesmicroseconds, bytesLocal disk reads.
clickhouse.select.s3_read_time, clickhouse.select.s3_read_bytes, clickhouse.select.s3_read_requestsmicroseconds, bytes, countObject storage reads, for tables stored on S3-compatible disks.
clickhouse.select.filesystem_cache_bytes, clickhouse.select.filesystem_cache_miss_bytesbytesBytes served from the filesystem cache and bytes that missed it.
clickhouse.select.real_time, clickhouse.select.cpu_timemicrosecondsWall clock and CPU time across all selects in the window.

​
Insert query metrics

These metrics describe the INSERT queries that finished in each 60-second window. The attribute is insert_table, which is the INSERT INTO target without a database prefix.
MetricUnitDescription
clickhouse.insert.queriescountInserts that finished in the window.
clickhouse.insert.inserted_rows, clickhouse.insert.inserted_bytescount, bytesRows and bytes written.
clickhouse.insert.disk_write_timemicrosecondsLocal disk write time.
clickhouse.insert.s3_write_time, clickhouse.insert.s3_write_bytes, clickhouse.insert.s3_write_requestsmicroseconds, bytes, countObject storage writes.
clickhouse.insert.delayed_inserts, clickhouse.insert.rejected_insertscountInserts that ClickHouse throttled or rejected because a table had too many parts. Any non-zero value means merges are falling behind.
clickhouse.insert.real_time, clickhouse.insert.cpu_timemicrosecondsWall clock and CPU time across all inserts in the window.

​
Merge and part metrics

These metrics describe background merges and the parts that make up each table. The attributes are merge_table and part_table.
MetricUnitDescription
clickhouse.merges.completedcountMerges that finished in the window.
clickhouse.merges.durationmillisecondsTotal merge time in the window.
clickhouse.merges.rows, clickhouse.merges.rows_readcountRows written by and read into merges.
clickhouse.merges.peak_memory_bytesbytesLargest peak memory of any merge in the window.
clickhouse.parts.activecountActive parts per table, for the 25 tables with the most primary key memory.
clickhouse.parts.primary_key_bytes_allocatedbytesMemory allocated to each table’s primary key index.

​
Server metrics

These metrics describe the whole server and have no table attribute. The metric name ends with the ClickHouse metric name. You can look up each one in the ClickHouse system tables documentation.
Metric prefixSourceMetrics
clickhouse.server.metrics.system.metricsMemoryTracking, MergesMutationsMemoryTracking, MarkCacheBytes, UncompressedCacheBytes, QueryCacheBytes, QueryConditionCacheBytes, FilesystemCacheSize, PageCacheBytes, PartsActive, Query
clickhouse.server.async_metrics.system.asynchronous_metricsMemoryResident, OSMemoryAvailable, jemalloc.allocated, jemalloc.resident, TotalPrimaryKeyBytesInMemory, TotalPrimaryKeyBytesInMemoryAllocated, QueriesMemoryUsage, FilesystemCacheBytes
clickhouse.server.events.system.eventsSelectedRows, SelectedBytes, MarkCacheHits, MarkCacheMisses, DiskReadElapsedMicroseconds. These are cumulative counters since the server started.

​
PostgreSQL lock metrics

If you enable the optional PostgreSQL receiver, the Collector also sends postgresql.locks, a gauge of lock requests in pg_locks that are still waiting to be granted. Its attributes are lock_mode, lock_type, schema, db, table, granted, and fastpath. When nothing is waiting, the Collector sends a single point with the value 0. This lets you tell a quiet database from a Collector that has stopped reporting.

​
Create a read-only database user

The Collector needs a database user that can read system tables and nothing else. We recommend a dedicated user so you can audit and revoke its access independently.

​
ClickHouse

To create a monitoring user, run the following statements as an administrator:
Grant access to each system table separately rather than running GRANT SELECT ON system.*. On ClickHouse Cloud, the wildcard grant fails because the default administrator does not have grant rights on every system table.The Collector also sets the readonly setting to 1 on its own session, so it cannot modify data even if the user is later given broader grants.

​
PostgreSQL

If you also want lock monitoring, create a PostgreSQL user and add it to the built-in pg_monitor role:

​
Configure the Collector

Create a file named database-monitoring.yaml with the following configuration. It defines one SQL query receiver for ClickHouse, a resource processor that stamps your project key on every metric, and an exporter that sends the metrics to LaunchDarkly:
The database attribute and the current_database filter scope the table-level queries to one ClickHouse database. If your workload spans several databases, run one Collector per database, or remove the current_database and database filters to see every database at once.

​
Configuration options

The configuration reads its connection details from environment variables. You can keep the same file in every environment. The following table describes each variable:
VariableDescriptionRequired
LAUNCHDARKLY_PROJECT_KEYYour LaunchDarkly project key. LaunchDarkly uses it to route metrics to your project.Yes
CLICKHOUSE_HOSTHostname of your ClickHouse server. If you run separate read and write replicas, read Monitor separate read and write clusters.Yes
CLICKHOUSE_PORTNative protocol port. Defaults to 9440, the TLS port. Use 9000 with CLICKHOUSE_SECURE=false for a server without TLS.No
CLICKHOUSE_SECUREWhether to connect with TLS. Defaults to true.No
CLICKHOUSE_USERNAME, CLICKHOUSE_PASSWORDCredentials for the monitoring user you created. The username defaults to otel_monitoring.Yes
CLICKHOUSE_DATABASEThe ClickHouse database whose tables you want to monitor. Defaults to default.No
ENVIRONMENTValue for the deployment.environment.name attribute. Use it to filter metrics by environment. Defaults to production.No
SERVICE_NAMEValue for the service.name attribute. Defaults to clickhouse. Use the same name on your application’s database spans to correlate them with these metrics.No

​
Add PostgreSQL lock monitoring

To also monitor waiting lock requests in a PostgreSQL database, add a second receiver and pipeline to the same file:
Set POSTGRES_HOST, POSTGRES_DATABASE, POSTGRES_USERNAME, and POSTGRES_PASSWORD when you run the Collector. POSTGRES_SSLMODE defaults to require. Set it to disable only for a server without TLS.

​
Monitor separate read and write clusters

Many ClickHouse deployments send queries to a read replica and send inserts to a writer. In that case, the query log on each host only records the queries that host ran. To monitor both, duplicate the sqlquery/clickhouse receiver as sqlquery/clickhouse-read and sqlquery/clickhouse-write, point each at the appropriate host, and list both receivers in the metrics pipeline. Keep the select, parts, and server memory queries on the read receiver, and the insert and merge queries on the write receiver. That way you do not count the same work twice.

​
Run the Collector

Use the OpenTelemetry Collector Contrib image, which includes the SQL query receiver. We recommend pinning the image to the version you tested with rather than using latest.

​
Docker

To run the Collector with Docker:
Replace YOUR_PROJECT_KEY with your LaunchDarkly project key and YOUR_PASSWORD with the monitoring user’s password.

​
Docker Compose

To run the Collector alongside your database with Docker Compose:

​
Kubernetes

Run the Collector as a single-replica Deployment so each scrape runs once. Store the configuration in a ConfigMap and the database password in a Secret. The OpenTelemetry Operator for Kubernetes can manage the Deployment for you. Set mode: deployment and replicas: 1 on the OpenTelemetryCollector resource.

​
Validate the configuration

Before you deploy, check that the Collector accepts the configuration and that every environment variable resolves:

​
Correlate database metrics with traces

Database monitoring tells you what each table is doing. Your application’s traces tell you which requests are waiting on it. To connect the two, instrument database calls in your application with the OpenTelemetry database semantic conventions, and use the same service.name on the Collector and in your application’s database client spans.The most useful span attributes are:
  • db.system.name: Set to clickhouse or postgresql.
  • db.namespace: The database name, matching the database attribute on your metrics.
  • db.collection.name: The table name, matching select_table or insert_table on your metrics.
  • db.query.summary: A low-cardinality summary of the statement, such as SELECT events.
Most OpenTelemetry database instrumentations set these attributes for you. With them in place, you can open a slow span in Traces, read the table it touched, and compare it against that table’s select metrics for the same minute.

​
View database metrics in LaunchDarkly

After you start the Collector, metrics begin arriving in LaunchDarkly within two minutes.To review your database metrics:
  1. Open the Telemetry section and navigate to the Observability metrics list.
  2. Search for clickhouse. to list the database metrics.
  3. Click a metric name to open its details, then group by select_table, insert_table, or merge_table to compare tables.
To learn more about graphing metrics, read Observability metrics. To add database graphs to a custom dashboard, read Creating custom dashboards.

​
Alert on database metrics

You can create alerts on any database metric. Useful starting points include:
  • clickhouse.insert.rejected_inserts or clickhouse.insert.delayed_inserts above 0, which means a table has too many parts and merges are falling behind.
  • clickhouse.parts.active rising steadily for one table, which usually precedes rejected inserts.
  • clickhouse.server.metrics.MemoryTracking approaching your server’s memory limit.
  • clickhouse.select.mark_cache_misses rising for a table while clickhouse.select.selected_marks stays flat, which means the mark cache is too small for your working set.
  • postgresql.locks above 0 for more than a few minutes, which means a transaction is blocking others.
To learn how to create alerts, read Alerts.

​
Investigate with Vega

Vega, the LaunchDarkly AI-powered agent, can read your database metrics during an investigation. When an alert fires on a slow endpoint or an error spike, Vega can check whether the affected service’s database tables saw a merge backlog, a mark cache miss spike, rejected inserts, or lock contention in the same window, and include that in its summary. If your traces carry the database semantic convention attributes described above, Vega can also name the table and query shape behind a slow span.To learn more, read Vega for auto-remediation.

​
Troubleshooting

If database metrics are not appearing in LaunchDarkly:
  1. Check the Collector logs: Run docker logs database-monitoring or the equivalent for your platform. A connection error names the host and port the receiver tried to reach. An authentication error means the monitoring user’s credentials or grants are wrong.
  2. Check the project key: Verify that LAUNCHDARKLY_PROJECT_KEY matches the project key in your LaunchDarkly project settings, not an environment key or SDK key.
  3. Verify that ClickHouse writes the query log: Run SELECT count() FROM system.query_log WHERE event_time > now() - INTERVAL 5 MINUTE as the monitoring user. If the table does not exist or is empty while queries are running, enable query_log and part_log in your ClickHouse server configuration.
  4. Verify network connectivity: Confirm the Collector host can reach otel.observability.app.launchdarkly.com on port 443:
  1. Check for empty windows: Table-level metrics only have data points for tables that ran queries in the previous minute. A table that received no selects or inserts has no points for that minute, which is expected. Server metrics are reported every minute regardless.
  2. Check TLS settings: If the Collector reports a handshake error, confirm that CLICKHOUSE_SECURE and CLICKHOUSE_PORT match your server. The TLS native port is usually 9440 and the plain port is 9000.

​
Related content