Certificate Renewal & Rotation Logic

[jmgilman]

Certificate Renewal & Rotation Logic

Overview

Implement the core certificate renewal and rotation logic, including the scheduler, renewal process, Certificate API client, and validation. This handles the automatic renewal of certificates at 30% lifetime remaining, with proper retry logic and error handling.

Requirements

Module Structure

Implement the following modules:

internal/
├── api/
│   ├── certificate.go         # Certificate API client
│   └── client.go              # HTTP client with retry logic
├── rotation/
│   ├── scheduler.go          # Certificate rotation scheduler
│   ├── renewer.go            # Certificate renewal logic
│   └── validator.go          # Certificate validation
pkg/
├── crypto/
│   └── utils.go              # Certificate utilities

Certificate API Client

Implement API client in internal/api/certificate.go:

type CertificateAPIClient struct {
    baseURL    string
    endpoints  EndpointConfig
    httpClient *http.Client
    jwtManager *auth.JWTManager
    logger     *slog.Logger
}

type EndpointConfig struct {
    Issue    string  // "/api/v1/certificates/issue"
    Renew    string  // "/api/v1/certificates/renew"
    Status   string  // "/api/v1/certificates/{serial}"
    CAChain  string  // "/api/v1/certificates/ca"
}

// Issue request/response types
type IssueRequest struct {
    Profile      string   `json:"profile"`
    CommonName   string   `json:"common_name"`
    SAN          []string `json:"san"`
    ValidityDays int      `json:"validity_days"`
    CSR          string   `json:"csr,omitempty"`
}

type IssueResponse struct {
    Certificate     string    `json:"certificate"`
    CertificateChain string   `json:"certificate_chain"`
    PrivateKey      string    `json:"private_key,omitempty"`
    SerialNumber    string    `json:"serial_number"`
    ExpiresAt       time.Time `json:"expires_at"`
    IssuerCA        string    `json:"issuer_ca"`
}

// Renewal request/response types
type RenewRequest struct {
    SerialNumber string `json:"serial_number"`
    CSR          string `json:"csr,omitempty"`
}

type RenewResponse struct {
    Certificate     string    `json:"certificate"`
    CertificateChain string   `json:"certificate_chain"`
    SerialNumber    string    `json:"serial_number"`
    ExpiresAt       time.Time `json:"expires_at"`
    PreviousSerial  string    `json:"previous_serial"`
}

// Core functions to implement:
func (c *CertificateAPIClient) IssueCertificate(req *IssueRequest) (*IssueResponse, error)
func (c *CertificateAPIClient) RenewCertificate(req *RenewRequest) (*RenewResponse, error)
func (c *CertificateAPIClient) GetCertificateStatus(serial string) (*CertificateStatus, error)
func (c *CertificateAPIClient) GetCAChain() (string, error)

HTTP Client with Retry Logic

Implement retry client in internal/api/client.go:

type RetryClient struct {
    client         *http.Client
    maxAttempts    int
    backoffIntervals []time.Duration  // [1m, 5m, 15m]
    logger         *slog.Logger
}

func (r *RetryClient) DoWithRetry(req *http.Request) (*http.Response, error) {
    var lastErr error
    
    for attempt := 0; attempt < r.maxAttempts; attempt++ {
        if attempt > 0 {
            backoff := r.getBackoffDuration(attempt)
            r.logger.Info("Retrying request after backoff",
                slog.Int("attempt", attempt),
                slog.Duration("backoff", backoff))
            time.Sleep(backoff)
        }
        
        resp, err := r.client.Do(req)
        if err != nil {
            lastErr = err
            if !r.isRetryableError(err) {
                return nil, err
            }
            continue
        }
        
        if r.isRetryableStatus(resp.StatusCode) {
            lastErr = fmt.Errorf("received status %d", resp.StatusCode)
            resp.Body.Close()
            continue
        }
        
        return resp, nil
    }
    
    return nil, fmt.Errorf("max retries exceeded: %w", lastErr)
}

func (r *RetryClient) getBackoffDuration(attempt int) time.Duration {
    if attempt-1 < len(r.backoffIntervals) {
        return r.backoffIntervals[attempt-1]
    }
    return r.backoffIntervals[len(r.backoffIntervals)-1]
}

Certificate Renewal Scheduler

Implement scheduler in internal/rotation/scheduler.go:

type Scheduler struct {
    config         *RotationConfig
    renewer        *Renewer
    storage        *storage.Manager
    checkInterval  time.Duration
    renewalThreshold float64  // 0.3 (30% of lifetime)
    ticker         *time.Ticker
    stopChan       chan struct{}
    logger         *slog.Logger
}

type RotationConfig struct {
    CheckInterval    time.Duration  // 1 hour
    RenewalThreshold float64        // 0.3
    RenewalWindow    time.Duration  // 72 hours
    MaxAttempts      int            // 5
}

// Core functions to implement:
func (s *Scheduler) Start()
func (s *Scheduler) Stop()
func (s *Scheduler) TriggerManualRenewal() error
func (s *Scheduler) checkAndRenew() error
func (s *Scheduler) shouldRenew(cert *x509.Certificate) bool

Renewal Decision Logic

func (s *Scheduler) shouldRenew(cert *x509.Certificate) bool {
    now := time.Now()
    expiry := cert.NotAfter
    totalLifetime := expiry.Sub(cert.NotBefore)
    remainingLifetime := expiry.Sub(now)
    
    // Renew if less than 30% lifetime remaining
    thresholdDuration := time.Duration(float64(totalLifetime) * s.renewalThreshold)
    if remainingLifetime <= thresholdDuration {
        return true
    }
    
    // Also renew if within renewal window (72 hours)
    if remainingLifetime <= s.config.RenewalWindow {
        return true
    }
    
    return false
}

Certificate Renewer

Implement renewer in internal/rotation/renewer.go:

type Renewer struct {
    apiClient      *api.CertificateAPIClient
    storage        *storage.Manager
    bootstrap      *bootstrap.Manager
    validator      *Validator
    serviceReloader *services.ReloadManager
    logger         *slog.Logger
    attemptCounter int
    maxAttempts    int
}

type RenewalResult struct {
    Success      bool
    NewSerial    string
    OldSerial    string
    Error        error
    Duration     time.Duration
}

// Core functions to implement:
func (r *Renewer) RenewCertificate() (*RenewalResult, error)
func (r *Renewer) performRenewal() (*RenewalResult, error)
func (r *Renewer) determineRenewalType() (RenewalType, error)
func (r *Renewer) generateCSR(commonName string, sans []string) (string, []byte, error)

Renewal Process Flow

func (r *Renewer) performRenewal() (*RenewalResult, error) {
    start := time.Now()
    
    // 1. Determine renewal type (bootstrap -> issue, or renew)
    renewalType, err := r.determineRenewalType()
    if err != nil {
        return nil, err
    }
    
    // 2. Get current certificate details
    current, err := r.storage.ReadCurrentCertificate()
    if err != nil && renewalType != RenewalTypeBootstrap {
        return nil, err
    }
    
    // 3. Generate new private key and CSR (optional)
    privateKey, csr, err := r.generateCSR(r.config.CommonName, r.config.SANs)
    if err != nil {
        return nil, err
    }
    
    // 4. Request new certificate
    var newCert *storage.Certificate
    if renewalType == RenewalTypeBootstrap {
        newCert, err = r.issueInitialCertificate(csr)
    } else {
        newCert, err = r.renewExistingCertificate(current.SerialNumber, csr)
    }
    if err != nil {
        r.attemptCounter++
        return &RenewalResult{
            Success: false,
            Error:   err,
            Duration: time.Since(start),
        }, err
    }
    
    // 5. Validate new certificate
    if err := r.validator.ValidateCertificate(newCert); err != nil {
        return nil, fmt.Errorf("new certificate validation failed: %w", err)
    }
    
    // 6. Store new certificate
    if err := r.storage.WritePendingCertificate(newCert); err != nil {
        return nil, err
    }
    
    // 7. Backup current certificate
    if current != nil {
        if err := r.storage.BackupCurrentCertificate(); err != nil {
            r.logger.Warn("Failed to backup current certificate", slog.Any("error", err))
        }
    }
    
    // 8. Atomically replace certificate
    if err := r.storage.PromotePendingToCurrent(); err != nil {
        return nil, fmt.Errorf("failed to promote certificate: %w", err)
    }
    
    // 9. Reload dependent services
    if err := r.serviceReloader.ReloadAll(); err != nil {
        r.logger.Error("Failed to reload some services", slog.Any("error", err))
        // Non-fatal: certificate is already replaced
    }
    
    // 10. Mark bootstrap transition complete if applicable
    if renewalType == RenewalTypeBootstrap {
        if err := r.bootstrap.MarkTransitionComplete(); err != nil {
            r.logger.Warn("Failed to mark bootstrap transition", slog.Any("error", err))
        }
    }
    
    r.attemptCounter = 0 // Reset on success
    
    return &RenewalResult{
        Success:   true,
        NewSerial: newCert.SerialNumber,
        OldSerial: current?.SerialNumber,
        Duration:  time.Since(start),
    }, nil
}

Certificate Validator

Implement validator in internal/rotation/validator.go:

type Validator struct {
    requiredKeyUsage    x509.KeyUsage
    requiredExtKeyUsage []x509.ExtKeyUsage
    minKeySize          int
    logger              *slog.Logger
}

func NewValidator() *Validator {
    return &Validator{
        requiredKeyUsage: x509.KeyUsageDigitalSignature | 
                         x509.KeyUsageKeyEncipherment | 
                         x509.KeyUsageKeyAgreement,
        requiredExtKeyUsage: []x509.ExtKeyUsage{
            x509.ExtKeyUsageServerAuth,
            x509.ExtKeyUsageClientAuth,
        },
        minKeySize: 2048, // RSA minimum, or P-256 for ECDSA
    }
}

// Core validation functions:
func (v *Validator) ValidateCertificate(cert *storage.Certificate) error
func (v *Validator) validateExpiry(cert *x509.Certificate) error
func (v *Validator) validateKeyUsage(cert *x509.Certificate) error
func (v *Validator) validateSANs(cert *x509.Certificate, expected []string) error
func (v *Validator) validateChain(cert *x509.Certificate, chain []*x509.Certificate) error
func (v *Validator) validateKeyStrength(cert *x509.Certificate) error

Certificate Utilities

Implement utilities in pkg/crypto/utils.go:

// Certificate parsing and generation utilities
func ParseCertificatePEM(pemData []byte) (*x509.Certificate, error)
func ParsePrivateKeyPEM(pemData []byte) (crypto.PrivateKey, error)
func GeneratePrivateKey(keyType string) (crypto.PrivateKey, error)
func GenerateCSR(privateKey crypto.PrivateKey, cn string, sans []string) ([]byte, error)
func EncodeCertificatePEM(cert *x509.Certificate) []byte
func EncodePrivateKeyPEM(key crypto.PrivateKey) ([]byte, error)
func GetCertificateSerialNumber(cert *x509.Certificate) string
func CalculateCertificateFingerprint(cert *x509.Certificate) string

Acceptance Criteria

1. Scheduler Operation

   • Checks certificates every hour
   • Correctly calculates renewal threshold (30% lifetime)
   • Triggers renewal at appropriate time
   • Handles manual renewal triggers via SIGUSR1

2. Renewal Process

   • Distinguishes between bootstrap and renewal cases
   • Uses correct API endpoint for each case
   • Generates valid CSRs with proper parameters
   • Successfully renews certificates before expiry

3. API Integration

   • Authenticates with JWT tokens
   • Handles API errors appropriately
   • Implements exponential backoff (1m, 5m, 15m)
   • Respects configured timeouts

4. Certificate Validation

   • Validates all required certificate attributes
   • Checks key usage extensions
   • Verifies certificate chain
   • Ensures minimum key strength

5. Error Handling

   • Alerts after max attempts (5) reached
   • Maintains current certificate on failure
   • Provides detailed error information
   • Recovers from transient failures

Testing Requirements

• Unit tests for renewal logic
• Unit tests for scheduler calculations
• Mock tests for API client
• Test CSR generation
• Test certificate validation
• Test retry logic with backoff
• Integration tests with mock Certificate API
• Test bootstrap to renewal transition
• Test failure scenarios and recovery
• Performance tests for scheduler overhead

Dependencies

• Standard crypto/x509 package
• Standard crypto/rsa and crypto/ecdsa packages
• Standard encoding/pem package
• HTTP client with timeout support