feat: add support for creating deployments by project name by mieliespoor · Pull Request #8611 · apache/incubator-devlake

mieliespoor · 2025-10-13T18:37:04Z

⚠️ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

I have read through the Contributing Documentation.
I have added relevant tests.
I have added relevant documentation.
I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.

Summary

Add a new endpoint for posting to a webhook using the project name. This will allow for a generic webhook to be created and added to multiple projects, and the webhook retrieved using the project name.

Does this close any open issues?

No

Screenshots

Include any relevant screenshots here.

Other Information

relates to discussion: #6348

…ving connections by project and plugin name

mieliespoor · 2025-10-13T22:18:35Z

There are no unit tests for this code and no documentation in this repository related to this.

…ved clarity and maintainability

mieliespoor · 2025-10-20T16:49:32Z

Testing this, we are able to push deployments and see these deployment data in the cicd_deployments and cicd_deployment_commits tables. Deployment data from webhooks, from what I can see is not present on many dashboards, but one dashboard in particular has an issue.

Because we now use one single webhook, which is then added to multiple projects, the query for the DORA Details - Deployment Frequency dashboard, specifically the Deployment list in the selected project(s) panel is incorrectly attributing deployments to projects.

Looking into it, we can see that the project_mapping table has the cicd_scope row for the same webook id for the two projects we used to test. This is still expected, because we will be reusing the webhook.

Looking at this part of the query:

with _deployment_commit_rank as(
  SELECT
  	pm.project_name,
  	IF(cdc._raw_data_table != '', cdc._raw_data_table, cdc.cicd_scope_id) as _raw_data_table,
  	cdc.id,
    cdc.display_title,
    cdc.url,
  	cdc.cicd_deployment_id,
  	cdc.cicd_scope_id,
  	result,
  	environment,
    finished_date,
    row_number() over(partition by cdc.cicd_deployment_id order by finished_date desc) as _deployment_commit_rank
  FROM cicd_deployment_commits cdc
  left join project_mapping pm on cdc.cicd_scope_id = pm.row_id and pm.`table` = 'cicd_scopes'
  WHERE
    pm.project_name in ($project)
  	and result = 'SUCCESS'
  	and environment = 'PRODUCTION'
)

I believe the issue to be around the line where the row_number() is determined. Because two rows, but for different projects exist, it does not treat these as distinctly different. What I did try and do, was add pm.project_name to the partition statement. This resulted in both projects showing up, but one of them shouldn't as it is not the project with the deployment data.

I'm not a db dev, nor a dba, so I'm out of my depth to try and solve this issue. Any suggestions here, would be welcomed.

klesh · 2025-10-21T07:09:31Z

Okay, based on the discussion.
I think the missing part is to "create the missing webhook connection" because the current implementation merely finds an existing connection, then calls PostDeploymentCicdTask.

Check if there is a webhook connection existing for the project.
Yes: call PostDeploymentCicdTask with it
No: create one, and update project/project to include it. then call PostDeploymentCicdTask with it.

mieliespoor · 2025-10-22T21:52:01Z

config-ui/src/features/connections/utils.ts

    postPipelineDeployTaskEndpoint: connection.postPipelineDeployTaskEndpoint,
    postPullRequestsEndpoint: connection.postPullRequestsEndpoint,
-    apiKeyId: connection.apiKey.id,
+    apiKeyId: connection.apiKey?.id,


apiKey is optional for a webhook, so we won't always have an apiKey id.

klesh · 2025-10-28T02:56:29Z

backend/plugins/webhook/api/deployments.go

+	connection := &models.WebhookConnection{}
+	projectName := input.Params["projectName"]
+	webhookName := fmt.Sprintf("%s_deployments", projectName)
+	err := findByProjectName(connection, input.Params, pluginName)


It would be better to pass the webhookName to the findByProjectName function instead of redefining.

klesh · 2025-10-28T03:00:29Z

backend/server/api/router.go

 	}
 	// mount all api resources for all plugins
 	for pluginName, apiResources := range resources {
+		if pluginName == "webhook" {


This is not the way to handle route conflict.
What was the conflict exactly?

Not exactly a conflict, but more a duplicate route not wanted. The way routing is done, this route, because how it is defined on the plugin, would have resulted in the route /plugins/webhook/projects/:projectName/deployments instead of only /projects/:projectName/deployments. With this we now only have the latter as we remove the registration of the route on the webhook plugin route. Best way I could think of achieving this, unless the code is moved elsewhere and not in the plugin itself.

What is wrong with /plugins/webhook/projects/:projectName/deployments?
The API is offered by the webhook plugin, it should be prefixed by /plugins/webhook by convention.

mieliespoor · 2025-11-10T21:33:00Z

@klesh I have made the requested changes. Please let me know if I need to make any further changes.

d4x1 · 2025-11-30T12:36:10Z

backend/plugins/webhook/impl/impl.go

 		"connections/by-name/:connectionName/issue/:issueKey/close": {
 			"POST": api.CloseIssueByName,
 		},
+		"projects/:projectName/deployments": {


I think projects/by-name/:projectName/deployments is better, just like onnections/by-name/***.

d4x1 · 2025-11-30T12:39:04Z

backend/plugins/webhook/api/deployments.go

+	if err != nil {
+		// if not found, we will attempt to create a new connection
+		// Use direct comparison against the package sentinel; only treat other errors as fatal.
+		if !errors.Is(err, gorm.ErrRecordNotFound) {


Maybe we can use https://github.com/apache/incubator-devlake/blob/920bc624f1c04bc01e944410d413fa113ed25554/backend/impls/dalgorm/dalgorm.go#L490 to keep a unified style.

I hope my change is acceptable.

backend/plugins/webhook/api/deployments.go

d4x1 · 2025-11-30T12:50:31Z

backend/plugins/webhook/api/deployments.go

+	// find or create the connection for this project
+	connection := &models.WebhookConnection{}
+	projectName := input.Params["projectName"]
+	webhookName := fmt.Sprintf("%s_deployments", projectName)


According to previous discussion, I think this means current project's default(or missing) webhook connection.
I prefer something that contains the default (or missing) meaning. We can add more items later under the default webhook.
This is just an advice.

I used the _deployments suffix because I see can how I end up adding a similar webhook for incidents at one point. We are using an ITSM system which isn't supported, and to be honest, I won't even bother to look at supporting it. I would however consider adding a webhook like this so that system can call to open/close incidents to complete the DORA loop.

I could go with _deployments_default for the webhook name? Open to suggestions here.

Thanks for your response. It makes sense.

d4x1 · 2025-11-30T12:58:06Z

backend/plugins/webhook/api/deployments.go

+// @Failure 403  {string} errcode.Error "Forbidden"
+// @Failure 500  {string} errcode.Error "Internal Error"
+// @Router /projects/:projectName/deployments [POST]
+func PostDeploymentsByProjectName(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {


Try to make this handler looks simple and clear.
For example:

func PostDeploymentsByProjectName(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) { GetOrCreateConnection postDeployments } func GetOrCreateConnection(){ // part 1: check project's missing webhook connection, create or update. // part 2: if the connection is created newly, updatte project's blueprint. }

Let me know if the changes are satisfactory

…ctName variable

…hook connection lookup

… function

d4x1 · 2025-12-01T17:49:30Z

backend/plugins/webhook/api/deployments.go

+	// find or create the connection for this project
+	connection := &models.WebhookConnection{}
+	projectName := input.Params["projectName"]
+	webhookName := fmt.Sprintf("%s_deployments", projectName)


Thanks for your response. It makes sense.

feat: add support for creating deployments by project name and retrie…

f9b3b1a

…ving connections by project and plugin name

refactor: replace FirstByProjectName with findByProjectName for impro…

6fe7d4e

…ved clarity and maintainability

mieliespoor marked this pull request as ready for review October 14, 2025 08:20

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. component/plugins This issue or PR relates to plugins pr-type/feature-development This PR is to develop a new feature labels Oct 14, 2025

mieliespoor and others added 3 commits October 16, 2025 21:38

Merge branch 'apache:main' into main

ad2ebc3

chore: add gitattributes to help windows developers

234df77

fix(webhook): add project deployments endpoint for webhook plugin

b79d3ac

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Oct 19, 2025

chore(build): add .gitattributes to paths-ignore in .licenserc.yaml

2aef41c

mieliespoor added 3 commits October 21, 2025 23:11

fix: create connection if not exist

30434c2

fix: allow for null apiKeys

0ddce2c

feat: create webhook and blueprint connection if not exist

e13f552

mieliespoor commented Oct 22, 2025

View reviewed changes

refactor: create specific name for project deployment webhooks

463f094

klesh reviewed Oct 28, 2025

View reviewed changes

mieliespoor added 2 commits November 10, 2025 21:28

chore: revert change to webhook routing

7a05719

chore: pass in webhook name

d816e12

d4x1 reviewed Nov 30, 2025

View reviewed changes

d4x1 and others added 4 commits November 30, 2025 20:58

Merge branch 'main' into main

a95456b

chore: improve logging for webhook connection creation by using proje…

cf34aaa

…ctName variable

refactor: replace gorm error check with custom error handling for web…

b6969d9

…hook connection lookup

refactor: enhance connection handling in PostDeploymentsByProjectName…

5bdc13f

… function

d4x1 approved these changes Dec 1, 2025

View reviewed changes

d4x1 merged commit de7be95 into apache:main Dec 2, 2025
11 checks passed

Conversation

mieliespoor commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Pre Checklist

Summary

Does this close any open issues?

Screenshots

Other Information

Uh oh!

mieliespoor commented Oct 13, 2025

Uh oh!

mieliespoor commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klesh commented Oct 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mieliespoor commented Nov 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mieliespoor commented Oct 13, 2025 •

edited

Loading

mieliespoor commented Oct 20, 2025 •

edited

Loading