Docs11112/storage monitoring restructure (#29706)

* [DOCS-11112] Add cloud storage monitoring folder

* [DOCS-11112] Add tile partials

* [DOCS-11112] Add tile partials, edit style

* [DOCS-11112] Revert header changes

* [DOCS-11112] Incorporate feedback

* [DOCS-11112] Add AWS doc link

* Apply suggestions from code review

Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>

* Incorporate feedback

Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>

* Fix link

* [DOCS-11112] Add existing inventory tab

* [DOCS-11112] Fix link

---------

Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>

Author: buraizu

content/en/integrations/guide/storage-monitoring-setup.md

@@ -9,51 +9,76 @@ private: true
 
 Storage Monitoring for Amazon S3, Google Cloud Storage, and Azure Blob Storage provides deep, prefix-level analytics to help you understand exactly how your storage is being used. It detects potential issues before they impact operations, and helps you make data-driven decisions about storage optimization. Use these insights to track storage growth, investigate access patterns, and optimize costs.
 
-This guide explains how to configure Storage Monitoring in Datadog for your S3 buckets, GCS buckets, and Azure Storage Accounts.
+This guide explains how to configure Storage Monitoring in Datadog for your Amazon S3 buckets, Google Cloud Storage buckets, and Azure storage accounts. Access your Storage Monitoring data by navigating to **Infrastructure > Storage Monitoring**.
 
-Access your Storage Monitoring data by navigating to **Infrastructure > Storage Monitoring**.
+Select your cloud storage service to access setup instructions.
 
-## Setup for Amazon S3
+{{< partial name="cloud_storage_monitoring/storage-monitoring-setup.html" >}}
 
-### Installation
+## Setup for Amazon S3
 
 {{< tabs >}}
 {{% tab "Recommended: Storage Monitoring UI" %}}
 
-The fastest way to set up Storage Monitoring is going to **Infrastructure > Storage Monitoring > [Add Buckets][1]**. On the Add Buckets page, you can configure multiple S3 buckets for Storage Monitoring in one go.
+The fastest way to configure Storage Monitoring is through the [Add Buckets][501] page in Datadog, where you can set up multiple S3 buckets at the same time.
 
-1. Go to Datadog > Infrastructure > Storage Monitoring.
-
-2. Click [Add Buckets][1].
+1. Go to Datadog > **Infrastructure** > **Storage Monitoring**.
+2. Click [Add Buckets][501].
 
 {{< img src="integrations/guide/storage_monitoring/add-buckets.png" alt="Select buckets for enabling Storage Monitoring" responsive="true">}}
 
 3. Enable Amazon S3 Integration and Resource collection for all the AWS accounts you want to monitor.
 
-  **Note**: For each AWS account that has the S3 buckets you want to monitor, make sure your Datadog IAM roles include the following permissions: `s3:GetObject`, `s3:ListObjects`, and `s3:PutInventoryConfiguration`.
+   1. **Allow Datadog to read from your destination buckets.** Add the following permissions to the Datadog IAM integration role for the account that owns the destination buckets:
+      - `s3:GetObject`
+      - `s3:ListBucket`
+      
+      Scope these read-only permissions to only the destination buckets containing your S3 inventory files.
+      
+   1. **Allow source buckets to write to destination buckets.** The destination buckets must include a policy that allows the source buckets to write inventory data. See [Creating a destination bucket policy][502] in the AWS documentation for details.
+   
+   Example source-bucket policy:
+
+      ```json
+        {
+          "Version": "2012-10-17",
+          "Statement": [
+            {
+              "Sid": "AllowListInventoryBucket",
+              "Effect": "Allow",
+              "Action": "s3:ListBucket",
+              "Resource": "arn:aws:s3:::storage-monitoring-s3-inventory-destination"
+            },
+            {
+              "Sid": "AllowGetInventoryObjects",
+              "Effect": "Allow",
+              "Action": "s3:GetObject",
+              "Resource": "arn:aws:s3:::storage-monitoring-s3-inventory-destination/*"
+            }
+          ]
+        }
+
+      ```
 
 4. Select the S3 buckets you want to monitor with Storage Monitoring. You can select buckets from multiple AWS accounts at once.
 
 {{< img src="integrations/guide/storage_monitoring/step-2.png" alt="Select buckets for enabling Storage Monitoring" responsive="true">}}
 
 5. Assign a destination bucket per region to store S3 inventory reports from the source buckets. This can be an existing AWS bucket or a new one.
 
-- Source bucket: The S3 bucket you want to monitor with Storage Monitoring
-- Destination bucket: Used to store inventory reports (one per AWS region, can be reused)
-
+   - Source bucket: The S3 bucket you want to monitor with Storage Monitoring
+   - Destination bucket: Used to store inventory reports (one per AWS region, can be reused)
 6. Complete the configuration. The inventory generation process will start within AWS within 24 hours of the first report.
-
 7. Return to **Infrastructure > Storage Monitoring** to see your bucket(s) appear.
 
-[1]: https://app.datadoghq.com/storage-monitoring?mConfigure=true
-
+[501]: https://app.datadoghq.com/storage-monitoring?mConfigure=true
+[502]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-inventory.html#configure-inventory-destination-bucket-policy
 {{% /tab %}}
 {{% tab "CloudFormation" %}}
 
 You can also set up Storage Monitoring using the provided CloudFormation templates. This process involves two steps:
 
-#### Step 1: Configure inventory generation
-
+### Step 1: Configure inventory generation
 
 This template configures your existing S3 bucket to generate inventory reports, which Datadog uses to generate detailed metrics about your bucket prefixes.
 
@@ -81,7 +106,7 @@ This template configures your existing S3 bucket to generate inventory reports,
 **Note:** This CloudFormation template can be rolled back, but rolling back doesn't delete the created resources. This is to ensure the existing bucket doesn't get deleted. You can manually delete the inventory configurations by going on the **Management** tab in the bucket view.
 
 **Note:** Review [Amazon S3 pricing][106] for costs related to inventory generation.
-#### Step 2: Configure required permissions
+### Step 2: Configure required permissions
 
 This template creates two IAM policies:
   - A policy to allow Datadog to read inventory files from the destination bucket
@@ -100,12 +125,11 @@ This template creates two IAM policies:
    - **SourceBucketPrefix**: This parameter limits the inventory generation to a specific prefix in the source bucket
    - **DestinationBucketPrefix**: If you want to reuse an existing bucket as the destination, this parameter allows the inventory files to be shipped to a specific prefix in that bucket. Ensure that any prefixes do not include trailing slashes (`/`)
 
-
 {{< img src="integrations/guide/storage_monitoring/bucket_policy_stack_details.png" alt="Stack parameters for bucket policy" responsive="true" style="width:90%;" >}}
 
 6. On the **Review and create** step, verify the parameters have been entered correctly, and click **Submit**.
 
-#### Post-setup steps
+### Post-setup steps
 
 After completing the CloudFormation setup, fill out the [post-setup form][105] with the following required information:
 1. Name of the destination bucket holding the inventory files.
@@ -122,22 +146,91 @@ After completing the CloudFormation setup, fill out the [post-setup form][105] w
 [105]: https://forms.gle/L97Ndxr2XLen1GBs7
 [106]: https://aws.amazon.com/s3/pricing/
 {{% /tab %}}
+
+{{% tab "Terraform" %}}
+
+You can use the Terraform [aws_s3_bucket_inventory][403] resource to set up Storage Monitoring. 
+
+The following example shows how to enable daily inventory on an S3 bucket for Datadog monitoring. To use this example:
+
+   - Replace `<MY_MONITORED_BUCKET>` with the name of the bucket to be monitored.
+   - Replace `<MY_INVENTORY_DESTINATION>` with the name of the bucket that receives your inventory files.
+   - Replace `<DESTINATION_ACCOUNT_ID>` with the AWS account ID that owns the destination bucket.
+
+```tf
+resource "aws_s3_bucket" "monitored" {
+  bucket = "<MY_MONITORED_BUCKET>"
+}
+
+resource "aws_s3_bucket" "inventory_destination" {
+  bucket = "<MY_INVENTORY_DESTINATION>"
+}
+
+resource "aws_s3_bucket_inventory" "daily_inventory" {
+  bucket = aws_s3_bucket.monitored.id
+  name   = "datadog-daily-inventory"
+
+
+  included_object_versions = "All"
+  schedule {
+    frequency = "Daily"
+  }
+  destination {
+    bucket {
+      account_id = "<DESTINATION_ACCOUNT_ID>"
+      bucket_arn = aws_s3_bucket.inventory_destination.arn
+      format     = "CSV"
+      prefix     = "datadog-inventory/"
+    }
+  }
+  optional_fields = [
+    "Size",
+    "StorageClass",
+    "LastModifiedDate"
+  ]
+}
+```
+
+**Notes**:
+
+   - The destination bucket can be your source bucket, but for security and logical separation, many organizations use a separate bucket.
+   - The `optional_fields` section is recommended for Datadog prefix metrics.
+
+### Post-setup steps
+
+Once the inventory configuration is set up and your inventory files begin appearing in the destination bucket, fill out [this form][401] to provide your S3 configuration details. This allows Datadog to begin generating prefix metrics for your storage.
+
+### Use modules for complex setups
+
+If you need to manage multiple buckets, complex inventory policies, encryption, or cross-account setups, you can use the [terraform-aws-s3-bucket module][402].
+
+### Troubleshooting
+
+- S3 Inventory files are delivered daily, and may take up to 24 hours to appear after setup.
+- Ensure IAM permissions allow S3 to write inventory files to your destination bucket.
+- If cross-account access is needed, confirm that the inventory destination prefix (`datadog-inventory/` in the example) is correct and accessible to Datadog.
+
+[401]: https://docs.google.com/forms/d/e/1FAIpQLScd0xor8RQ76N6BemvvMzg9UU7Q90svFrNGY8n83sMF2JXhkA/viewform
+[402]: https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/tree/master/examples/s3-inventory
+[403]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_inventory
+{{% /tab %}}
+
 {{% tab "AWS Console" %}}
 
 To manually set up the required [Amazon S3 Inventory][206] and related configuration, follow these steps:
 
-#### Step 1: Create a destination bucket
+### Step 1: Create a destination bucket
 
 1. [Create an S3 bucket][201] to store your inventory files. This bucket acts as the central location for inventory reports. **Note**: You must only use one destination bucket for all inventory files generated in an AWS account.
 2. Create a prefix within the destination bucket (optional).
 
-#### Step 2: Configure the bucket and integration role policies
+### Step 2: Configure the bucket and integration role policies
 
 1. Follow the steps in the [Amazon S3 user guide][202] to add a bucket policy to your destination bucket allowing write access (`s3:PutObject`) from your source buckets.
 
 2. Ensure the Datadog AWS integration role has `s3:GetObject` and `s3:ListObjects` permissions on the destination bucket. These permissions allow Datadog to read the generated inventory files.
 
-#### Step 3: Configure Inventory generation
+### Step 3: Configure inventory generation
 
 For each bucket you want to monitor:
 1. Go to the [Amazon S3 buckets page][203] in the AWS console, and select the bucket.
@@ -160,7 +253,7 @@ For each bucket you want to monitor:
 
 **Note**: Review [Amazon S3 pricing][204] for costs related to inventory generation.
 
-#### Post-setup steps
+### Post-setup steps
 
 After completing the above steps, fill out the [post-setup form][205] with the following required information:
 
@@ -178,6 +271,18 @@ After completing the above steps, fill out the [post-setup form][205] with the f
 [205]: https://forms.gle/L97Ndxr2XLen1GBs7
 [206]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-inventory.html
 {{% /tab %}}
+
+{{% tab "Existing S3 inventory" %}}
+
+If you have already configured S3 inventory for the buckets you want to monitor, choose **one** of the following options:
+
+- Fill out [this form][601] to share your configurations with Datadog
+- [Reach out to us][602] to use an API for setting up multiple buckets
+
+[601]: https://forms.gle/dhDbSxTvCUDXg1QR7
+[602]: mailto:storage-monitoring@datadoghq.com
+{{% /tab %}}
+
 {{< /tabs >}}
 
 ### Validation
@@ -188,28 +293,26 @@ To verify your setup:
    - Confirm the Datadog integration can access the files:
        - Navigate to **Infrastructure -> Storage Monitoring -> Installation Recommendations** to see if the bucket you configured is showing in the list
 
-
 ### Troubleshooting
+
 If you encounter any issues or need assistance:
 - Make sure to use only one destination bucket for all inventory files per AWS account
 - Verify all permissions are correctly configured
 - If you're still encountering issues, [reach out][1] with your bucket details, AWS account ID, and Datadog org ID
 
 ## Setup for Google Cloud Storage
 
-### Installation
-
 The process involves the following steps:
 
-#### Step 1: Install the GCP integration and enable resource collection
+#### Step 1: Install the Google Cloud integration and enable resource collection
 
-To collect GCP Storage metrics from your GCP project, install the GCP integration in Datadog. Enable Resource Collection for the project containing the buckets you want to monitor. Resource Collection allows Datadog to associate your buckets' labels with the metrics collected through storage monitoring.
+To collect Google Cloud Storage metrics from your Google Cloud project, install the Google Cloud integration in Datadog. Enable Resource Collection for the project containing the buckets you want to monitor. Resource Collection allows Datadog to associate your buckets' labels with the metrics collected through storage monitoring.
 
 **Note**: While you can disable specific metric namespaces, keep the Cloud Storage namespace (gcp.storage) enabled.
 
 #### Step 2: Enable the Storage Insights API
 
-Enable the [Storage Insights][2] API in your GCP project.
+Enable the [Storage Insights][2] API in your Google Cloud project.
 
 #### Step 3: Grant service agent permissions
 
@@ -226,21 +329,21 @@ You can create an inventory report configuration in multiple ways. The quickest
 
 1. Includes these metadata fields: `"bucket", "name", "project", "size", "updated", "storageClass"`
 2. Generates CSV reports with `'\n'` as the delimiter and `','` as the separator
-3. Uses this destination path format: `<Bucket>/{{date}}`, where `<Bucket>` is the monitored bucket-name
+3. Uses this destination path format: `<BUCKET>/{{date}}`, where `<BUCKET>` is the monitored bucket-name
 
 {{< tabs >}}
 {{% tab "Google Cloud CLI" %}}
 
 Use the [Google Cloud CLI][301] to run the following command:
 
 ```
-gcloud storage insights inventory-reports create <source_bucket_url> \
+gcloud storage insights inventory-reports create <SOURCE_BUCKET_URL> \
   --no-csv-header \
   --display-name=datadog-storage-monitoring \
-  --destination=<gs://my_example_destination_bucket/source_bucket_name/{{date}}> \
+  --destination=gs://<DESTINATION_BUCKET>/<SOURCE_BUCKET>/{{date}}> \
   --metadata-fields=project,bucket,name,size,updated,storageClass \
   --schedule-starts=<YYYY-MM-DD> \
-  --schedule-repeats=<daily|weekly> \
+  --schedule-repeats=<DAILY|WEEKLY> \
   --schedule-repeats-until=<YYYY-MM-DD>
 ```
 
@@ -249,7 +352,7 @@ gcloud storage insights inventory-reports create <source_bucket_url> \
 {{% /tab %}}
 {{% tab "Terraform" %}}
 
-Copy the following Terraform template, substitute the necessary arguments, and apply it in the GCP project that contains your bucket.
+Copy the following Terraform template, substitute the necessary arguments, and apply it in the Google Cloud project that contains your bucket.
 
 <!-- vale off -->
 {{% collapse-content title="Terraform configuration for inventory reports" level="h4" expanded=true %}}
@@ -372,8 +475,8 @@ After completing the setup steps, fill out the [post-setup][3] form with the fol
 2. Name of the service account with the granted permissions
 3. Prefix where the files are stored in the destination bucket (if any)
 4. Name of the source bucket you want to monitor (the bucket producing inventory files)
-5. GCP location of the destination bucket holding the inventory files
-6. GCP ProjectID containing the buckets
+5. Google Cloud location of the destination bucket holding the inventory files
+6. Google Cloud ProjectID containing the buckets
 7. Datadog org ID
 
 ### Validation
@@ -385,38 +488,32 @@ To verify your setup:
 4. Navigate to Infrastructure -> Storage Monitoring -> Installation Recommendations to see if your configured bucket appears in the list
 
 ### Troubleshooting
+
 If you encounter any issues or need assistance:
-- Use only one destination bucket for all inventory files per GCP project
+- Use only one destination bucket for all inventory files per Google Cloud project
 - Verify all permissions are correctly configured
-- If issues persist, [reach out][1] with your bucket details, GCP Project ID, and Datadog org ID
+- If issues persist, [reach out][1] with your bucket details, Google Cloud Project ID, and Datadog org ID
 
 [1]: mailto:storage-monitoring@datadoghq.com
 [2]: https://cloud.google.com/storage/docs/insights/using-inventory-reports#enable_the_api
 [3]: https://forms.gle/c7b8JiLENDaUEqGk8
 
-
-
 ## Setup for Azure Blob Storage
 
-### Installation
-
-To set up Storage Monitoring for Azure Blob Storage, follow these steps:
-
 {{< tabs >}}
 {{% tab "Azure CLI" %}}
 
-To enable inventories for the selected storage accounts in each subscription, run the following script in your [Azure Cloud Shell][301]:
+Enable inventories for the selected storage accounts in each subscription by running the following script in your [Azure Cloud Shell][301]:
 
 ```shell
 curl https://datadogstoragemonitoring.blob.core.windows.net/scripts/install.sh \
-  | bash -s -- <client_id> <subscription_id> <comma_separated_storage_account_names>
+  | bash -s -- <CLIENT_ID> <SUBSCRIPTION_ID> <COMMA_SEPARATED_STORAGE_ACCOUNT_NAMES>
 ```
 
 Before running the script, set your [shell environment][302] to Bash and replace the various placeholder inputs with the correct values:
-- `<client_id>`: The client ID of an App Registration already set up using the [Datadog Azure integration][302]
-- `<subscription_id>`: The subscription ID of the Azure subscription containing the storage accounts
-- `<comma_separated_storage_account_names>`: A comma-separated list of the storage accounts you want to monitor. For example, `storageaccount1,storageaccount2`
-
+- `<CLIENT_ID>`: The client ID of an App Registration already set up using the [Datadog Azure integration][302]
+- `<SUBSCRIPTION_ID>`: The subscription ID of the Azure subscription containing the storage accounts
+- `<COMMA_SEPARATED_STORAGE_ACCOUNT_NAMES>`: A comma-separated list of the storage accounts you want to monitor (for example, `storageaccount1,storageaccount2`)
 
 [301]: https://shell.azure.com
 [302]: /integrations/azure/#setup
@@ -427,8 +524,7 @@ Before running the script, set your [shell environment][302] to Bash and replace
 
 For Each Storage Account you wish to monitor, follow all of the steps here:
 
-
-#### Create a blob inventory policy
+### Create a blob inventory policy
 1. In the Azure portal, navigate to your Storage Account.
 2. Go to **Data management** > **Blob inventory**.
 3. Click **Add**.
@@ -451,7 +547,7 @@ For Each Storage Account you wish to monitor, follow all of the steps here:
    - Exclude prefix: datadog-storage-monitoring
 5. Click **Add**.
 
-#### Add the role assignment
+### Add the role assignment
 1. In the Azure portal, navigate to your Storage Account.
 2. Go to **Data storage** > **Containers**.
 3. Click on the **datadog-storage-monitoring** container.
@@ -472,6 +568,3 @@ For Each Storage Account you wish to monitor, follow all of the steps here:
 After you finish with the above steps, fill out the [post-setup form][310].
 
 [310]: https://forms.gle/WXFbGyBwWfEo3gbM7
-
-
-