Merge pull request #89 from james-laing/streamline_arrays_uri

Split buckets and filesystems URI into endpoints

Author: sdodsley

README.md

@@ -128,17 +128,17 @@ Authentication is used by the exporter as the mechanism to cross authenticate to
 The exporter understands the following requests:
 
 
-| URL                                                | GET parameters | description               |
-| ---------------------------------------------------| -------------- | --------------------------|
-| http://\<exporter-host\>:\<port\>/metrics          | endpoint       | Full array metrics        |
-| http://\<exporter-host\>:\<port\>/metrics/array    | endpoint       | Array metrics             |
-| http://\<exporter-host\>:\<port\>/metrics/clients  | endpoint       | Clients metrics           |
-| http://\<exporter-host\>:\<port\>/metrics/usage    | endpoint       | Quotas usage metrics      |
-| http://\<exporter-host\>:\<port\>/metrics/policies | endpoint       | NFS policies info metrics |
+| URL                                                   | GET parameters | description               |
+| ----------------------------------------------------- | -------------- | ------------------------- |
+| http://\<exporter-host\>:\<port\>/metrics             | endpoint       | Full array metrics        |
+| http://\<exporter-host\>:\<port\>/metrics/array       | endpoint       | Array metrics             |
+| http://\<exporter-host\>:\<port\>/metrics/clients     | endpoint       | Clients metrics           |
+| http://\<exporter-host\>:\<port\>/metrics/filesystems | endpoint       | File System metrics       |
+| http://\<exporter-host\>:\<port\>/metrics/objectstore | endpoint       | Object Store metrics      |
+| http://\<exporter-host\>:\<port\>/metrics/policies    | endpoint       | NFS policies info metrics |
+| http://\<exporter-host\>:\<port\>/metrics/usage       | endpoint       | Quotas usage metrics      |
 
 
-Depending on the target array, scraping for the whole set of metrics could result into timeout issues, in which case it is suggested either to increase the scraping timeout or to scrape each single endpoint instead.
-
 ### Usage examples
 
 In a typical production scenario, it is recommended to use a visual frontend for your metrics, such as [Grafana](https://github.com/grafana/grafana). Grafana allows you to use your Prometheus instance as a datasource, and create Graphs and other visualizations from PromQL queries. Grafana, Prometheus, are all easy to run as docker containers.
@@ -158,10 +158,42 @@ Please have a look at the documentation of each image/application for adequate c
 
 A simple but complete example to deploy a full monitoring stack on kubernetes can be found in the [examples](examples/config/k8s) directory  
 
+### Upgrade Information
+
+v1.1.0 - New URIs `filesystem` and `objectstore` were added to split off these metric instruments to ensure metrics in the `array` URI remain quick to scrape in large environments to comply with the limitations recommendation set below in [Bugs and Limitations](#bugs-and-limitations).
+
+If you require the following metrics for your dataset for your observability toolset, please add the new endpoint(s) as required.
+
+
+The following metrics have been moved from `array` to `filesystem`
+
+* `purefb_file_systems_performance_average_bytes`
+* `purefb_file_systems_performance_bandwidth_bytes`
+* `purefb_file_systems_performance_latency_usec`
+* `purefb_file_systems_performance_throughput_iops`
+* `purefb_file_systems_space_bytes`
+* `purefb_file_systems_space_data_reduction_ratio`
+
+The following metrics have been moved from `array` to `objectstore`
+
+* `purefb_buckets_object_count`
+* `purefb_buckets_performance_average_bytes`
+* `purefb_buckets_performance_bandwidth_bytes`
+* `purefb_buckets_performance_latency_usec`
+* `purefb_buckets_performance_throughput_iops`
+* `purefb_buckets_quota_space_bytes`
+* `purefb_buckets_s3_specific_performance_latency_usec`
+* `purefb_buckets_s3_specific_performance_throughput_iops`
+* `purefb_buckets_space_bytes`
+* `purefb_buckets_space_data_reduction_ratio`
+* `purefb_object_store_accounts_data_reduction_ratio`
+* `purefb_object_store_accounts_object_count`
+* `purefb_object_store_accounts_space_bytes`
+
 
 ### Bugs and Limitations
 
-* Pure FlashBlade REST APIs are not designed for efficiently reporting on full clients and objects quota KPIs, therefrore it is suggested to scrape the "array" metrics preferably and use the "clients" and "usage" metrics individually and with a lower frequency than the other.. In any case, as a general rule, it is advisable to do not lower the scraping interval down to less than 30 sec. In case you experience timeout issues, you may want to increase the Prometheus scraping timeout and interval approriately.
+* Pure FlashBlade REST APIs are not designed for efficiently reporting on full clients and objects quota metrics. Therefore, it is suggested to scrape the `array` metrics most frequently and use the `clients`, `filesystem`, `objectstore`, `policies`, and `usage` endpoints individually and with a lower frequency. As a general rule, it is not advisable to lower the scraping interval to less than 30 seconds for for any endpoint. In case you experience timeout issues, you may want to increase the Prometheus scraping timeout and interval appropriately. It is very important to have the collection interval (frequency) safely higher than the scrape duration.
 
 ### Metrics Collected
 

cmd/fb-om-exporter/main.go

@@ -118,6 +118,14 @@ func main() {
 			w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
 			metricsHandler(w, r)
 		})
+		http.HandleFunc("/metrics/filesystems", func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
+			metricsHandler(w, r)
+		})
+		http.HandleFunc("/metrics/objectstore", func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
+			metricsHandler(w, r)
+		})
 		http.HandleFunc("/metrics/usage", func(w http.ResponseWriter, r *http.Request) {
 			w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
 			metricsHandler(w, r)
@@ -139,6 +147,12 @@ func main() {
 		http.HandleFunc("/metrics/clients", func(w http.ResponseWriter, r *http.Request) {
 			metricsHandler(w, r)
 		})
+		http.HandleFunc("/metrics/filesystems", func(w http.ResponseWriter, r *http.Request) {
+			metricsHandler(w, r)
+		})
+		http.HandleFunc("/metrics/objectstore", func(w http.ResponseWriter, r *http.Request) {
+			metricsHandler(w, r)
+		})
 		http.HandleFunc("/metrics/usage", func(w http.ResponseWriter, r *http.Request) {
 			metricsHandler(w, r)
 		})
@@ -164,8 +178,10 @@ func metricsHandler(w http.ResponseWriter, r *http.Request) {
 	} else {
 		metrics = path[2]
 		switch metrics {
-		case "clients":
 		case "array":
+		case "buckets":
+		case "clients":
+		case "filesystems":
 		case "usage":
 		case "policies":
 		default:
@@ -237,10 +253,22 @@ func index(w http.ResponseWriter, r *http.Request) {
             <td>Provides only array related metrics.</td>
         </tr>
         <tr>
+            <td>Bucket metrics</td>
+            <td><a href="/metrics/objectstore?endpoint=host">/metrics/objectstore</a></td>
+            <td>endpoint</td>
+            <td>Provides only object store related metrics.</td>
+        </tr>
+		<tr>
             <td>Client metrics</td>
             <td><a href="/metrics/clients?endpoint=host">/metrics/clients</a></td>
             <td>endpoint</td>
-            <td>Provides only client related metrics. This is the most time expensive query</td>
+            <td>Provides only client related metrics. This is the most time expensive query.</td>
+        </tr>
+	   <tr>
+            <td>File System metrics</td>
+            <td><a href="/metrics/filesystems?endpoint=host">/metrics/filesystems</a></td>
+            <td>endpoint</td>
+            <td>Provides only file system related metrics.</td>
         </tr>
         <tr>
             <td>Quota metrics</td>

internal/openmetrics-exporter/collector.go

@@ -23,38 +23,46 @@ func Collector(ctx context.Context, metrics string, registry *prometheus.Registr
 		httpPerfCollector := NewHttpPerfCollector(fbclient)
 		nfsPerfCollector := NewNfsPerfCollector(fbclient)
 		perfReplCollector := NewPerfReplicationCollector(fbclient)
-		bucketsPerfCollector := NewBucketsPerfCollector(fbclient, buckets)
-		buckestS3PerfCollector := NewBucketsS3PerfCollector(fbclient, buckets)
-		filesystemsPerfCollector := NewFileSystemsPerfCollector(fbclient, filesystems)
-		filesystemsSpaceCollector := NewFileSystemsSpaceCollector(filesystems)
 		arraySpaceCollector := NewArraySpaceCollector(fbclient)
-		bucketsSpaceCollector := NewBucketsSpaceCollector(buckets)
 		alertsCollector := NewAlertsCollector(fbclient)
 		hardwareCollector := NewHardwareCollector(fbclient)
 		hwPerfConnectorsCollector := NewHwConnectorsPerfCollector(fbclient)
-		objstoreacctsCollector := NewObjectStoreAccountsCollector(fbclient)
 		registry.MustRegister(
 			perfCollector,
 			s3perfCollector,
 			httpPerfCollector,
 			nfsPerfCollector,
 			perfReplCollector,
-			bucketsPerfCollector,
-			buckestS3PerfCollector,
-			filesystemsPerfCollector,
 			arraySpaceCollector,
-			bucketsSpaceCollector,
-			filesystemsSpaceCollector,
 			alertsCollector,
 			hardwareCollector,
 			hwPerfConnectorsCollector,
-			objstoreacctsCollector,
+		)
+	}
+	if metrics == "all" || metrics == "filesystems" {
+		filesystemsPerfCollector := NewFileSystemsPerfCollector(fbclient, filesystems)
+		filesystemsSpaceCollector := NewFileSystemsSpaceCollector(filesystems)
+		registry.MustRegister(
+			filesystemsSpaceCollector,
+			filesystemsPerfCollector,
 		)
 	}
 	if metrics == "all" || metrics == "clients" {
 		clientsPerfCollector := NewClientsPerfCollector(fbclient)
 		registry.MustRegister(clientsPerfCollector)
 	}
+	if metrics == "all" || metrics == "objectstore" {
+		bucketsPerfCollector := NewBucketsPerfCollector(fbclient, buckets)
+		buckestS3PerfCollector := NewBucketsS3PerfCollector(fbclient, buckets)
+		bucketsSpaceCollector := NewBucketsSpaceCollector(buckets)
+		objstoreacctsCollector := NewObjectStoreAccountsCollector(fbclient)
+		registry.MustRegister(
+			bucketsPerfCollector,
+			buckestS3PerfCollector,
+			bucketsSpaceCollector,
+			objstoreacctsCollector,
+		)
+	}
 	if metrics == "all" || metrics == "usage" {
 		usageCollector := NewUsageCollector(fbclient, filesystems)
 		registry.MustRegister(usageCollector)

specification/metrics/purefb-metrics.md

@@ -23,13 +23,15 @@ This document describes the semantic conventions for Pure FlashBlade Metrics.
 
 ## Collections by Endpoint
 
-| Endpoint          | Description             | Metrics Instruments collected                                                                                                     |
-| ----------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
-| /metrics          | Full array metrics      | all                                                                                                                               |
-| /metrics/array    | Array only metrics      | `purefb_info`, `purefb_alerts`, `purefb_array`, `purefb_buckets`, `purefb_file_systems`, `purefb_hardware`, `purefb_object_store` |
-| /metrics/clients  | Client only metrics     | `purefb_info`, `purefb_clients`                                                                                                   |
-| /metrics/usage    | Quota only metrics      | `purefb_info`, `purefb_file_system_usage`                                                                                         |
-| /metrics/policies | NFS policy related info | `purefb_info`, `purefb_nfs_export_rule`                                                                                           |
+| Endpoint             | Description              | Metrics Instruments collected                                     |
+| -------------------- | ------------------------ | ----------------------------------------------------------------- |
+| /metrics             | Full array metrics       | all                                                               |
+| /metrics/array       | Array metrics            | `purefb_info`, `purefb_alerts`, `purefb_array`, `purefb_hardware` |
+| /metrics/clients     | Client metrics           | `purefb_info`, `purefb_clients`                                   |
+| /metrics/filesystems | File System metrics      | `purefb_info`, `purefb_file_systems`                              |
+| /metrics/objectstore | Object Store metrics     | `purefb_info`, `purefb_buckets`, `purefb_object_store`            |
+| /metrics/policies    | NFS policy related info  | `purefb_info`, `purefb_nfs_export_rule`                           |
+| /metrics/usage       | Quota metrics            | `purefb_info`, `purefb_file_system_usage`                         |
 
 ## Metric Statuses