From a5105518fc429fc4b5cc455dbcb91567fadd13ed Mon Sep 17 00:00:00 2001 From: Igor Lukanin Date: Tue, 8 Oct 2024 01:10:39 +0200 Subject: [PATCH 1/5] docs: Data-at-rest encryption in Cube Store and customer-provided keys (CMK) --- .../product/caching/running-in-production.mdx | 52 ++++++++--- docs/pages/product/workspace.mdx | 4 + docs/pages/product/workspace/_meta.js | 1 + .../product/workspace/encryption-keys.mdx | 89 +++++++++++++++++++ 4 files changed, 135 insertions(+), 11 deletions(-) create mode 100644 docs/pages/product/workspace/encryption-keys.mdx diff --git a/docs/pages/product/caching/running-in-production.mdx b/docs/pages/product/caching/running-in-production.mdx index 3e76d5540f742..eab7151fd8d40 100644 --- a/docs/pages/product/caching/running-in-production.mdx +++ b/docs/pages/product/caching/running-in-production.mdx @@ -221,27 +221,33 @@ Cube Store cluster uses both persistent and scratch storage. Cube Store makes use of a separate storage layer for storing metadata as well as for persisting pre-aggregations as Parquet files. -Cube Store [can be configured][ref-config-env] to use either AWS S3 or -Google Cloud Storage (GCS) as persistent storage. If desired, local path on +Cube Store can be configured to use either AWS S3, Google Cloud Storage (GCS), or +Azure Blob Storage as persistent storage. If desired, a local path on the server can also be used in case all Cube Store cluster nodes are co-located on a single machine. -Cube Store can only use one type of remote storage at runtime. +Cube Store can only use one type of remote storage at the same time. -Cube Store requires strong consistency guarantees from underlying distributed -storage. AWS S3, Google Cloud Storage, and Azure Blob Storage (Cube Cloud only) -are the only known implementations that provide strong consistency. Using other -implementations in production is discouraged and can lead to consistency and -data corruption errors. +Cube Store requires strong consistency guarantees from an underlying distributed +storage. AWS S3, Google Cloud Storage, and Azure Blob Storage are the only known +implementations that provide them. Using other implementations in production is +discouraged and can lead to consistency and data corruption errors. + + +Using Azure Blob Storage with Cube Store is only supported in Cube Cloud on +[Enterprise and above plans](https://cube.dev/pricing). + + + A simplified example using AWS S3 might look like: ```yaml @@ -313,10 +319,34 @@ should be built before any tables are removed. ## Security -Cube Store currently does not have any in-built authentication mechanisms. For -this reason, we recommend running your Cube Store cluster on a network that only -allows requests from the Cube deployment. +### Authentication + +Cube Store does not have any in-built authentication mechanisms. For this reason, +we recommend running your Cube Store cluster with a network configuration that +only allows access from the Cube deployment. + +### Data-at-rest encryption + +Cube Store provides data-at-rest protection by utilizing the [modular encryption +mechanism][link-parquet-encryption] of Parquet files in its [persistent +storage](#persistent-storage). Pre-aggregation data is secured using the +[AES cipher][link-aes] with 256-bit keys. Data encyption and decryption are +completely seamless to Cube Store operations. + + + +Data-at-rest encryption in Cube Store is only available in Cube Cloud on +[Enterprise and above plans](https://cube.dev/pricing). + + + +You can provide, rotate, or drop your own [customer-managed keys][ref-cmk] (CMK) +for Cube Store via the Encryption Keys page in Cube Cloud. + [link-wsl2]: https://docs.microsoft.com/en-us/windows/wsl/install-win10 [ref-caching-partitioning]: /product/caching/using-pre-aggregations#partitioning [ref-config-env]: /reference/configuration/environment-variables +[link-parquet-encryption]: https://parquet.apache.org/docs/file-format/data-pages/encryption/ +[link-aes]: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard +[ref-cmk]: /product/workspace/encryption-keys \ No newline at end of file diff --git a/docs/pages/product/workspace.mdx b/docs/pages/product/workspace.mdx index 825646d2ef56f..7538865e44ced 100644 --- a/docs/pages/product/workspace.mdx +++ b/docs/pages/product/workspace.mdx @@ -37,6 +37,8 @@ metrics to external monitoring tools. Cube Cloud account and [single sign-on][ref-sso]. - Use [Audit Log][ref-audit-log] to review security-related events in your Cube Cloud account. +- Use the [encryption keys][ref-encryption-keys] page to manage [data-at-rest +encryption in Cube Store][ref-cube-store-encryption]. - Use [Budgets][ref-budgets] to control the usage and spend of your Cube Cloud account. - Use [Preferences][ref-prefs] to adjust the workspace to your liking. @@ -74,3 +76,5 @@ With Cube Core, you can: [ref-cli]: /product/workspace/cli [ref-ai-assistant]: /product/workspace/ai-assistant [ref-semantic-catalog]: /product/workspace/semantic-catalog +[ref-encryption-keys]: /product/workspace/encryption-keys +[ref-cube-store-encryption]: /product/caching/running-in-production#data-at-rest-encryption \ No newline at end of file diff --git a/docs/pages/product/workspace/_meta.js b/docs/pages/product/workspace/_meta.js index 8981c36ea275a..6f154b4bed8bc 100644 --- a/docs/pages/product/workspace/_meta.js +++ b/docs/pages/product/workspace/_meta.js @@ -14,6 +14,7 @@ module.exports = { "access-control": "Access Control", "sso": "Single Sign-on", "audit-log": "Audit Log", + "encryption-keys": "Encryption keys", "budgets": "Budgets", "preferences": "Preferences", "cli": "CLI", diff --git a/docs/pages/product/workspace/encryption-keys.mdx b/docs/pages/product/workspace/encryption-keys.mdx new file mode 100644 index 0000000000000..fed83f8d476a1 --- /dev/null +++ b/docs/pages/product/workspace/encryption-keys.mdx @@ -0,0 +1,89 @@ +# Encryption keys + +The Encryption Keys page in Cube Cloud allows to manage [data-at-rest +encryption in Cube Store][ref-cube-store-encryption]. + + + +Data-at-rest encryption in Cube Store is only available in Cube Cloud on +[Enterprise and above plans](https://cube.dev/pricing). + + + +Navigate to Settings → Encryption Keys in your Cube Cloud deployment +to [provide](#add-a-key), [rotate](#rotate-a-key), or [drop](#drop-a-key) +your own customer-managed keys (CMK) for Cube Store. + +## Customer-managed keys for Cube Store + +On the Encryption Keys page, you can see all previously provided keys: + + + +### Add a key + +To add an encryption key, click Create to open a modal window. +Provide any string as the key name and paste a [Base64-encoded][link-base64] +string as the 256-bit key value. + + + +**Once the first encryption key is added, Cube Store will assume that data-at-rest +encryption is enabled.** After that, querying unencrypted pre-aggregation partitions +will yield the following error: `Invalid Parquet file in encrypted mode. File (or +at least the Parquet footer) is not encrypted`. + + + +It may take a few minutes for any changes to encryption keys to take effect. + + + +After the refresh worker builds or rebuilds pre-aggregation partitions with +respect to their [refresh strategy][ref-pre-aggs-refresh-strategy] or after they +are [built manually][ref-pre-aggs-build-manually], their data will be encrypted. + +**For encryption, the most recently added encryption key is used.** For decryption, +all previously provided keys can be used, if there are still any pre-aggregation +partitions encrypted with those keys. + +### Rotate a key + +To rotate an encryption key, you have to [add a new key](#add-a-key) and then +rebuild pre-aggregation partitions using this key, either by the means of the +refresh worker, or manually. + +You can check which encryption key is used by any pre-aggregation partition by +querying `system.tables` in Cube Store via [SQL Runner][ref-sql-runner]: + + + + + +Only newly built or rebuilt pre-aggregation partitions will use the newly addded +encryption key. Previously built partitions will still be encrypted using +previously provided keys. If you [drop a key](#drop-a-key) before these partitions +are rebuilt, querying them will yield an error. + + + + + +If you're using [incremental pre-aggregations][ref-pre-aggs-incremental], the +refresh worker will likely only rebuild some of their partitions. You have to [rebuild +them manually][ref-pre-aggs-build-manually] to ensure that the new encryption key +is used. + + + +### Drop a key + +To drop an encryption key, click Delete next to it. + + +[ref-cube-store-encryption]: /product/caching/running-in-production#data-at-rest-encryption +[link-base64]: https://en.wikipedia.org/wiki/Base64 +[ref-pre-aggs-refresh-strategy]: /product/caching/using-pre-aggregations#refresh-strategy +[ref-pre-aggs-build-manually]: /product/workspace/pre-aggregations +[ref-pre-aggs-incremental]: /reference/data-model/pre-aggregations#incremental +[ref-sql-runner]: /product/workspace/sql-runner \ No newline at end of file From 54023f5cc788a7ca9c70b97057df46273007a22b Mon Sep 17 00:00:00 2001 From: Igor Lukanin Date: Tue, 8 Oct 2024 01:16:14 +0200 Subject: [PATCH 2/5] . --- docs/pages/product/caching/running-in-production.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/pages/product/caching/running-in-production.mdx b/docs/pages/product/caching/running-in-production.mdx index eab7151fd8d40..917e3fe1447f5 100644 --- a/docs/pages/product/caching/running-in-production.mdx +++ b/docs/pages/product/caching/running-in-production.mdx @@ -221,6 +221,13 @@ Cube Store cluster uses both persistent and scratch storage. Cube Store makes use of a separate storage layer for storing metadata as well as for persisting pre-aggregations as Parquet files. + + +Persistent storage can be [encrypted](#data-at-rest-encryption) to ensure +data-at-rest protection. + + + Cube Store can be configured to use either AWS S3, Google Cloud Storage (GCS), or Azure Blob Storage as persistent storage. If desired, a local path on the server can also be used in case all Cube Store cluster nodes are From 522109268051d455295573ea8103c9281df178e6 Mon Sep 17 00:00:00 2001 From: Igor Lukanin Date: Tue, 8 Oct 2024 12:13:11 +0200 Subject: [PATCH 3/5] Update docs/pages/product/workspace/encryption-keys.mdx Co-authored-by: Sam Hughes --- docs/pages/product/workspace/encryption-keys.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/pages/product/workspace/encryption-keys.mdx b/docs/pages/product/workspace/encryption-keys.mdx index fed83f8d476a1..950b6fdad26de 100644 --- a/docs/pages/product/workspace/encryption-keys.mdx +++ b/docs/pages/product/workspace/encryption-keys.mdx @@ -60,7 +60,7 @@ querying `system.tables` in Cube Store via [SQL Runner][ref-sql-runner]: -Only newly built or rebuilt pre-aggregation partitions will use the newly addded +Only newly built or rebuilt pre-aggregation partitions will use the newly added encryption key. Previously built partitions will still be encrypted using previously provided keys. If you [drop a key](#drop-a-key) before these partitions are rebuilt, querying them will yield an error. From 33d515234702877dc33c242d827a6b5e4f19e9b5 Mon Sep 17 00:00:00 2001 From: Igor Lukanin Date: Tue, 8 Oct 2024 12:27:46 +0200 Subject: [PATCH 4/5] Describe layers --- .../product/caching/running-in-production.mdx | 28 +++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/pages/product/caching/running-in-production.mdx b/docs/pages/product/caching/running-in-production.mdx index 917e3fe1447f5..c56fb1f57eb4e 100644 --- a/docs/pages/product/caching/running-in-production.mdx +++ b/docs/pages/product/caching/running-in-production.mdx @@ -221,13 +221,6 @@ Cube Store cluster uses both persistent and scratch storage. Cube Store makes use of a separate storage layer for storing metadata as well as for persisting pre-aggregations as Parquet files. - - -Persistent storage can be [encrypted](#data-at-rest-encryption) to ensure -data-at-rest protection. - - - Cube Store can be configured to use either AWS S3, Google Cloud Storage (GCS), or Azure Blob Storage as persistent storage. If desired, a local path on the server can also be used in case all Cube Store cluster nodes are @@ -255,6 +248,14 @@ Using Azure Blob Storage with Cube Store is only supported in Cube Cloud on + + +As an additional layer on top of standard AWS S3, Google Cloud Storage (GCS), or +Azure Blob Storage encryption, persistent storage can optionally use [Parquet +encryption](#data-at-rest-encryption) for data-at-rest protection. + + + A simplified example using AWS S3 might look like: ```yaml @@ -334,11 +335,14 @@ only allows access from the Cube deployment. ### Data-at-rest encryption -Cube Store provides data-at-rest protection by utilizing the [modular encryption -mechanism][link-parquet-encryption] of Parquet files in its [persistent -storage](#persistent-storage). Pre-aggregation data is secured using the -[AES cipher][link-aes] with 256-bit keys. Data encyption and decryption are -completely seamless to Cube Store operations. +[Persistent storage](#persistent-storage) is secured using the standard AWS S3, +Google Cloud Storage (GCS), or Azure Blob Storage encryption. + +Cube Store also provides optional data-at-rest protection by utilizing the +[modular encryption mechanism][link-parquet-encryption] of Parquet files in its +persistent storage. Pre-aggregation data is secured using the [AES cipher][link-aes] +with 256-bit keys. Data encyption and decryption are completely seamless to Cube +Store operations. From d9692315096187cdee2d8db93655841dc0288633 Mon Sep 17 00:00:00 2001 From: Igor Lukanin Date: Tue, 8 Oct 2024 12:34:15 +0200 Subject: [PATCH 5/5] . --- docs/pages/product/workspace/encryption-keys.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/pages/product/workspace/encryption-keys.mdx b/docs/pages/product/workspace/encryption-keys.mdx index 950b6fdad26de..e20532ba09538 100644 --- a/docs/pages/product/workspace/encryption-keys.mdx +++ b/docs/pages/product/workspace/encryption-keys.mdx @@ -23,8 +23,8 @@ On the Encryption Keys page, you can see all previously provided keys ### Add a key To add an encryption key, click Create to open a modal window. -Provide any string as the key name and paste a [Base64-encoded][link-base64] -string as the 256-bit key value. +Provide the key name and the key value: an 256-bit AES encryption key, encoded +in [standard Base64][link-base64] in its canonical representation. @@ -60,10 +60,10 @@ querying `system.tables` in Cube Store via [SQL Runner][ref-sql-runner]: -Only newly built or rebuilt pre-aggregation partitions will use the newly added -encryption key. Previously built partitions will still be encrypted using -previously provided keys. If you [drop a key](#drop-a-key) before these partitions -are rebuilt, querying them will yield an error. +Only newly built or rebuilt pre-aggregation partitions will be encrypted using the +newly added encryption key. Previously built partitions will still be encrypted +using previously provided keys. If you [drop a key](#drop-a-key) before these +partitions are rebuilt, querying them will yield an error. @@ -82,7 +82,7 @@ To drop an encryption key, click Delete next to it. [ref-cube-store-encryption]: /product/caching/running-in-production#data-at-rest-encryption -[link-base64]: https://en.wikipedia.org/wiki/Base64 +[link-base64]: https://datatracker.ietf.org/doc/html/rfc4648#section-4 [ref-pre-aggs-refresh-strategy]: /product/caching/using-pre-aggregations#refresh-strategy [ref-pre-aggs-build-manually]: /product/workspace/pre-aggregations [ref-pre-aggs-incremental]: /reference/data-model/pre-aggregations#incremental