[FEATURE] Add support for databricks_access_control_rule_set on the workspace

[mingue]

Use-cases

Hi, I have a terraform service principal that is admin on the workspace level, I would like to be able to define permissions at the workspace level over who can access and use the Service Principals created by terraform to run jobs using the databricks_access_control_rule_set resource.

Currently databricks_access_control_rule_set can only be used if you are an account admin in Databricks. Our terraform identity is only limited to workspace admin to manage permissions over a particular workspace. So this is not a viable option for us.

However, this is not a security restriction as this is allowed using the UI, which seems to invoking a different API that allows workspace admins to define permissions over the workspace https://adb-***.azuredatabricks.net/ajax-api/2.0/preview/accounts/access-control/rule-sets

Attempted Solutions

No alternative to automate management of Service Principals with permissions only on a workspace level.

Proposal

Add support for workspace admin to define databricks_access_control_rule_set or similar resource at the workspace level using the api indicated above /ajax-api/2.0/preview/accounts/access-control/rule-sets

thanks!

[alexott]

https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/access_control_rule_set can be used in the workspace context (see note in the docs)

[mingue]

Yes @alexott the documentation says that, but it seems the underlying implementation uses account level APIs. As these Service Principals (SPs) are created using our terraform identity which becomes owner of the SPs, however when trying to define permissions it does not work through the databricks provider, although the same thing works on the UI.

"This resource allows you to manage access rules on Databricks account level resources. For convenience we allow accessing this resource through the Databricks account and workspace."

As I'm getting the following error when trying to deploy that resource

│ Error: cannot create access control rule set: Not authorized to perform this operation
│ 
│   with databricks_access_control_rule_set.automation_sp_rule_set["sp_***"],
│   on access.tf line 55, in resource "databricks_access_control_rule_set" "automation_sp_rule_set":
│   55: resource "databricks_access_control_rule_set" "automation_sp_rule_set" {

[Roymprog]

@alexott, I'm running into a similar issue, however slightly different.
I was able to create the resource through the workspace provider, yet after creation my terraform plans fail complaining the access control rule set cannot be read

│ Error: cannot read access control rule set: Not authorized to perform this operation
│ 
│   with module.databricks_sp_automation.databricks_access_control_rule_set.this[0],
│   on main.tf line 6, in resource "databricks_access_control_rule_set" "this":
│    6: resource "databricks_access_control_rule_set" "this" {

This makes it a bug rather than a feature request, as the misleading documentation got me stuck. Only way out I see is acquiring account admin permissions to fix this.
FYI, using provider version 1.83.0.

EDIT:
My error originates from the authorative behaviour of the resource, my. bad. In my case I assigned roles/servicePrincipal.user to another user, the authorative behaviour of the resource removed the permissions on the tf service principal.
I thought the following only applied to roles/servicePrincipal.user, but apparently type is broader, as it also affected roles/servicePrincipal.manager.

Configuring this resource for an object will OVERWRITE any existing permissions of the same type

EDIT 2:
In my first EDIT I drew the wrong conclusion. The nature of the databricks service principal is the reason why I could create the resource, but not update is. I created the ruleset with only one grant_rules block assigning permissions to a Databricks managed service principal. The error above happened when I tried to assign a second grant_rules to an External service principal from Azure. Thus, you can use the resource with the workspace provider, but only on databricks managed service principals.