docs: add proposal of Karmada Self-Signed Certificate Content Standardization #6189

tiansuo114 · 2025-03-07T07:33:53Z

What type of PR is this?
/kind documentation

What this PR does / why we need it:
Design documentation for supporting karmada's new self-signed certificate system
Which issue(s) this PR fixes:
Part of 6091

Special notes for your reviewer:
@XiShanYongYe-Chang
@chaosi-zju
@zhzhuang-zju
Does this PR introduce a user-facing change?:

codecov-commenter · 2025-03-07T07:49:24Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.34%. Comparing base (824bd8d) to head (7993912).
Report is 163 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6189      +/-   ##
==========================================
+ Coverage   47.97%   49.34%   +1.37%     
==========================================
  Files         674      677       +3     
  Lines       55841    55004     -837     
==========================================
+ Hits        26788    27143     +355     
+ Misses      27305    26097    -1208     
- Partials     1748     1764      +16

Flag	Coverage Δ
unittests	`49.34% <ø> (+1.37%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tiansuo114 · 2025-03-07T07:57:26Z

Preliminary Design Document Discussion

This document is an initial design draft. However, there are some unresolved questions regarding the overall system design that I have raised in the document. For clarity, I will describe the issues again here.

1. Secret Generation During Cluster Deployment

When deploying the cluster using karmadactl init, a karmada-apiserver.config file is generated locally.
This file is used by client components to obtain the certificates required to connect to the server.
However, after deploying the operator with /karmadactl init, I did not find any secret in the cluster that corresponds to this configuration.
Question: What is the design intention behind this part?

2. Generating Individual Certificate Files for Each Client

I have encountered some design issues when considering generating independent, usable certificate files for each client. I currently have two approaches:

Option 1:

Generate an independent authentication certificate file for each client component.
Modify the client component command flags so that each client reads its own certificate file for authentication.

Option 2:

Continue using the karmada-apiserver.config file for client components.
In the karmada-apiserver.config file, store different certificate contents for different client components using a multi-user approach.
Modify the related mechanisms (it's unclear whether this is feasible at the moment) so that each client loads its corresponding certificate data and uses that for authentication.

3. Using a Consistent Certificate Across Different Servers for the Same Client

This may require adjustments to the apiserver implementation logic.
First: Decouple the generated certificates for client components from binding to a specific server.
Second: Different Karmada workspaces should have different CA root certificates.
Challenge: How to allow client components to use a consistent certificate for authentication when the CA root certificates differ.
Proposed Solution:
During the karmadactl init and the deployment of a new Karmada operator on a new workspace, add a configuration option that allows the new workspace to use an existing CA root certificate as its own. This would help ensure the possibility of mutual communication.

@XiShanYongYe-Chang @chaosi-zju

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

zhzhuang-zju · 2025-03-08T08:25:05Z

@tiansuo114 Thank you for your proposal as it is valuable for the security of the project. In fact, the changes in the certificate configuration will ultimately be reflected in the fields of the certificate. And the proposal you put forward in your proposal mainly affects the Subject and Subject Alternative Name of the certificate. So, could you provide a comparison of the changes before and after for both the server certificate and the client certificate? Like:

Certificate:
    Data:
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: CN = karmada
        Subject: O = system:masters, CN = system:admin
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:*.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc, DNS:localhost, IP Address:127.0.0.1, IP

tiansuo114 · 2025-03-08T09:19:30Z

@tiansuo114 Thank you for your proposal as it is valuable for the security of the project. In fact, the changes in the certificate configuration will ultimately be reflected in the fields of the certificate. And the proposal you put forward in your proposal mainly affects the Subject and Subject Alternative Name of the certificate. So, could you provide a comparison of the changes before and after for both the server certificate and the client certificate? Like:
Certificate:
    Data:
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: CN = karmada
        Subject: O = system:masters, CN = system:admin
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:*.karmada-system.svc.cluster.local, DNS:*.karmada-system.svc, DNS:localhost, IP Address:127.0.0.1, IP 

Thank you for the detailed description. The certificate updates you mentioned primarily affect the Subject and Subject Alternative Name (SAN) fields. Below are the key differences between the certificates before and after the changes:

Before Changes (Current Implementation)

Server Certificate (`karmada-apiserver`)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3120819415934316465 (0x2b4f60b956e82fb1)
        Signature Algorithm: sha256WithRSAEncryption
    Issuer: CN = karmada
    Validity
        Not Before: Mar 6 01:57:50 2025 GMT
        Not After : Mar 6 01:57:48 2026 GMT
    Subject: O = , CN = karmada-apiserver
    X509v3 extensions:
        X509v3 Subject Alternative Name:
            DNS:, DNS:.karmada-system.svc,
            DNS:.karmada-system.svc.cluster.local,
            DNS:karmada-aggregated-apiserver,
            DNS:karmada-aggregated-apiserver.karmada-system.svc.cluster.local,
            DNS:karmada-apiserver,
            DNS:karmada-apiserver.karmada-system.svc.cluster.local,
            DNS:karmada-webhook,
            DNS:karmada-webhook.karmada-system.svc,
            DNS:karmada-webhook.karmada-system.svc.cluster.local,
            DNS:kubernetes, DNS:kubernetes.default,
            DNS:kubernetes.default.svc,
            DNS:localhost,
            IP Address:127.0.0.1, IP Address:172.18.0.3, IP Address:2407:D840:51:1:0:0:0:6

Client Certificate (`client-certificate-data`)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6967587319794734003 (0x60b1d7c061e193b3)
        Signature Algorithm: sha256WithRSAEncryption
    Issuer: CN = karmada
    Validity
        Not Before: Mar 6 01:57:50 2025 GMT
        Not After : Mar 6 01:57:48 2026 GMT
    Subject: O = system:masters, CN = system:admin
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Extended Key Usage:
            TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Basic Constraints: critical
            CA:FALSE
        X509v3 Authority Key Identifier:
            87:3D:86:17:F6:A0:A8:D2:5B:01:95:A5:F9:BB:C9:CD:A3:88:3A:3C
        X509v3 Subject Alternative Name:
            DNS:, DNS:.karmada-system.svc,
            DNS:.karmada-system.svc.cluster.local,
            DNS:karmada-aggregated-apiserver,
            DNS:karmada-aggregated-apiserver.karmada-system.svc.cluster.local,
            DNS:karmada-apiserver,
            DNS:karmada-apiserver.karmada-system.svc.cluster.local,
            DNS:karmada-webhook,
            DNS:karmada-webhook.karmada-system.svc,
            DNS:karmada-webhook.karmada-system.svc.cluster.local,
            DNS:kubernetes, DNS:kubernetes.default,
            DNS:kubernetes.default.svc,
            DNS:localhost,
            IP Address:127.0.0.1, IP Address:172.18.0.3, IP Address:2407:D840:51:1:0:0:0:6

After Changes

Server Certificate (`karmada-apiserver`)

Certificate:
    Data:
        Signature Algorithm: sha256WithRSAEncryption
    Issuer: CN = karmada-apiserver-ca  # Dedicated CA per component
    Subject: CN = karmada-apiserver
    X509v3 extensions:
        X509v3 Subject Alternative Name:
            DNS:karmada-apiserver,
            DNS:karmada-apiserver.karmada-system.svc,
            DNS:karmada-apiserver.karmada-system.svc.cluster.local,
            DNS:localhost,
            IP Address:127.0.0.1
#Contains only the domain name required by the component

Client Certificate (`karmada-scheduler`)

Certificate:
    Data:
        Signature Algorithm: sha256WithRSAEncryption
    Issuer: CN = karmada-scheduler-ca  # Dedicated CA per component
    Subject: O = system:karmada-components, CN = system:karmada-scheduler
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
# In order to complete the function of connecting different servers, it does not contain any domain name
# Because I don't know much about the functional design that allows the same client to use consistent certificates for different servers, this could be a mistake

tiansuo114 · 2025-03-20T14:23:51Z

The content of the document has been updated according to the results of the meeting. How does it look now?
@zhzhuang-zju @XiShanYongYe-Chang

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

XiShanYongYe-Chang · 2025-04-15T09:01:42Z

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

+
+Special Cases:
+1. For the two client components connecting to karmada-scheduler-estimator (karmada-scheduler and karmada-descheduler), since they use the gRPC protocol when connecting to karmada-scheduler-estimator, and they are the only components using gRPC protocol for this connection, the certificate is only used for TLS connection and does not contain identity verification attributes.
+2. Regarding the karmada-interpreter-webhook component, since it is only used for cluster e2e testing, it is considered a test component and does not play a role in actual production clusters. Therefore, it is unnecessary to change its certificate to a more granular one.


This component is not only used by E2E but also a demo component. Users can refer to this component to implement their own components and apply them to the production environment.

XiShanYongYe-Chang · 2025-04-15T09:06:50Z

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

+util::create_signing_certkey "" "${CERT_DIR}" ca karmada '"client auth","server auth"'
+
+# 2. Create common certificates for servers using root CA
+util::create_certkey "" "${CERT_DIR}" "ca" server server "" "${karmadaAltNames[@]}"


How do you set the parameter karmadaAltNames?

tiansuo114 · 2025-04-16T05:30:23Z

I tried to modify the document content according to yesterday's meeting, how does it look now?
@XiShanYongYe-Chang
@zhzhuang-zju

XiShanYongYe-Chang

/cc @zhzhuang-zju

XiShanYongYe-Chang · 2025-04-25T06:54:35Z

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

+- Identity Confusion Risk: All components share the same certificate content, making it impossible to distinguish component identities through certificate Subject
+- Security Vulnerabilities: A single certificate compromise would jeopardize the entire system's security, violating the principle of least privilege
+- Operational Complexity: Certificate rotation requires global operations, lacking fine-grained control capabilities
+- Standards Compliance Gap: Does not meet cloud-native security standards (such as NIST SP 800-204B) requirements for mTLS communication between microservices


After this proposal, are these shortcomings gone?

I made some mistakes in writing this part,After these modifications are applied to the system:

- Identity Confusion Risk: All components share the same certificate content, making it impossible to distinguish component identities through certificate Subject - Security Vulnerabilities: A single certificate compromise would jeopardize the entire system's security, violating the principle of least privilege

These two issues will be completely resolved.

- Operational Complexity: Certificate rotation requires global operations, lacking fine-grained control capabilities - Standards Compliance Gap: Does not meet cloud-native security standards (such as NIST SP 800-204B) requirements for mTLS communication between microservices

However, these two issues will not be completely resolved and will still require further modifications. I have now modified this content to only retain the issues that have been completely resolved.

XiShanYongYe-Chang · 2025-04-25T07:04:37Z

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

+util::create_key_pair "" "${CERT_DIR}" "sa"
+```
+
+### 2. Certificate Content Change Examples(有没有必要将证书内容修改的实际方式列出来())


中文部分是不是可以删除掉了。

I'm very sorry. 😭 This was my mistake. Now this part of the incorrect content has been deleted

zhzhuang-zju · 2025-04-29T07:37:48Z

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

+```
+Certificate:
+    Issuer: CN = karmada
+    Subject: CN = karmada-apiserver


Suggested change

Subject: CN = karmada-apiserver

Subject: CN = system:karmada:karmada-apiserver

Be consistent with the following content.

zhzhuang-zju · 2025-04-29T07:39:49Z

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md

+
+### 1. Certificate Generation Script Modification
+
+Current certificate generation process in `hack/deploy-karmada.sh`:


The following script modifications are only for some components. You can add a note indicating that this is just an example and not a full set of changes

Ok. The above problems have been fixed. How does this document look now?

…dization Signed-off-by: tiansuo114 <zhaoyi_114@outlook.com>

zhzhuang-zju · 2025-05-07T06:22:07Z

/retest

zhzhuang-zju · 2025-05-07T06:32:20Z

thanks~
/lgtm

XiShanYongYe-Chang · 2025-05-07T06:48:04Z

/approve

karmada-bot · 2025-05-07T06:48:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XiShanYongYe-Chang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [XiShanYongYe-Chang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot requested review from RainbowMango and Tingtal March 7, 2025 07:33

karmada-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 7, 2025

zhzhuang-zju reviewed Mar 8, 2025

View reviewed changes

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md Outdated Show resolved Hide resolved

zhzhuang-zju reviewed Mar 8, 2025

View reviewed changes

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md Outdated Show resolved Hide resolved

zhzhuang-zju reviewed Mar 8, 2025

View reviewed changes

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md Outdated Show resolved Hide resolved

karmada-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 20, 2025

zhzhuang-zju reviewed Mar 21, 2025

View reviewed changes

docs/proposals/cert/Self-Signed_Certificate_Content_Standardization.md Outdated Show resolved Hide resolved

tiansuo114 force-pushed the self-signed_cert_proposal branch from 597712c to 80a295f Compare April 3, 2025 01:20

tiansuo114 force-pushed the self-signed_cert_proposal branch from 80a295f to 2a3f45a Compare April 10, 2025 01:25

tiansuo114 changed the title ~~docs: add proposal of Karmada Self-Signed Certificate Content Standar…~~ docs: add proposal of Karmada Self-Signed Certificate Content Standardization Apr 15, 2025

XiShanYongYe-Chang reviewed Apr 15, 2025

View reviewed changes

tiansuo114 force-pushed the self-signed_cert_proposal branch from 2a3f45a to fa574db Compare April 16, 2025 05:21

XiShanYongYe-Chang reviewed Apr 25, 2025

View reviewed changes

karmada-bot requested a review from zhzhuang-zju April 25, 2025 07:06

zhzhuang-zju reviewed Apr 29, 2025

View reviewed changes

tiansuo114 force-pushed the self-signed_cert_proposal branch 2 times, most recently from 5858392 to 6ae46de Compare April 29, 2025 07:46

docs: add proposal of Karmada Self-Signed Certificate Content Standar…

7993912

…dization Signed-off-by: tiansuo114 <zhaoyi_114@outlook.com>

tiansuo114 force-pushed the self-signed_cert_proposal branch from 6ae46de to 7993912 Compare April 29, 2025 07:49

karmada-bot assigned zhzhuang-zju May 7, 2025

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label May 7, 2025

karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 7, 2025

karmada-bot merged commit cad6ad5 into karmada-io:master May 7, 2025
26 of 27 checks passed

	Subject: CN = karmada-apiserver
	Subject: CN = system:karmada:karmada-apiserver


		### 1. Certificate Generation Script Modification

		Current certificate generation process in `hack/deploy-karmada.sh`:

docs: add proposal of Karmada Self-Signed Certificate Content Standardization #6189

docs: add proposal of Karmada Self-Signed Certificate Content Standardization #6189

Uh oh!

Conversation

tiansuo114 commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tiansuo114 commented Mar 7, 2025

Preliminary Design Document Discussion

1. Secret Generation During Cluster Deployment

2. Generating Individual Certificate Files for Each Client

Option 1:

Option 2:

3. Using a Consistent Certificate Across Different Servers for the Same Client

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhzhuang-zju commented Mar 8, 2025

Uh oh!

tiansuo114 commented Mar 8, 2025

Before Changes (Current Implementation)

Server Certificate (karmada-apiserver)

Client Certificate (client-certificate-data)

After Changes

Server Certificate (karmada-apiserver)

Client Certificate (karmada-scheduler)

Uh oh!

tiansuo114 commented Mar 20, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tiansuo114 commented Apr 16, 2025

Uh oh!

XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhzhuang-zju commented May 7, 2025

Uh oh!

zhzhuang-zju commented May 7, 2025

Uh oh!

XiShanYongYe-Chang commented May 7, 2025

Uh oh!

karmada-bot commented May 7, 2025

Uh oh!

Uh oh!

Uh oh!

tiansuo114 commented Mar 7, 2025 •

edited

Loading

codecov-commenter commented Mar 7, 2025 •

edited

Loading

Server Certificate (`karmada-apiserver`)

Client Certificate (`client-certificate-data`)

Server Certificate (`karmada-apiserver`)

Client Certificate (`karmada-scheduler`)