Enhance Crossplane XRD to Backstage Catalog Metadata Flow

[felixboehm]

Enhance Crossplane XRD to Backstage Catalog Metadata Flow

🎯 Objective

Implement comprehensive metadata mapping from Crossplane XRDs to Backstage catalog entities through the kubernetes-ingestor plugin to enable rich discovery, relationship tracking, and operational context.

📋 Background

Currently, our Crossplane templates (XRDs) are ingested into Backstage but lack rich metadata that would enable:

• Better categorization and discovery
• Relationship tracking between templates
• Cost and compliance awareness
• Direct links to documentation and source code
• Operational context (SLA, support channels)

🔄 Proposed Metadata Flow

1. Source: Enhanced XRD Metadata Structure

Each XRD should include comprehensive labels and annotations:

# template-*/configuration/xrd.yaml
apiVersion: apiextensions.crossplane.io/v2
kind: CompositeResourceDefinition
metadata:
  name: <resource>.<domain>
  labels:
    # Backstage discovery
    backstage.io/kubernetes-id: <unique-template-id>
    backstage.io/kubernetes-namespace: crossplane-system
    
    # Template categorization
    openportal.dev/template-type: [infrastructure|application|data|network]
    openportal.dev/template-category: <specific-category>
    openportal.dev/template-tier: [platform|team|user]
    openportal.dev/cost-level: [low|medium|high]
    openportal.dev/compliance-required: ["true"|"false"]
    
    # Versioning
    openportal.dev/version: <semver>
    openportal.dev/api-version: v1alpha1
    
    # Form generation hint
    terasky.backstage.io/generate-form: "true"
    
  annotations:
    # Core Backstage annotations
    backstage.io/source-location: "url:https://github.com/open-service-portal/<repo>"
    backstage.io/techdocs-ref: "url:https://github.com/open-service-portal/<repo>"
    backstage.io/view-url: "<github-view-url>"
    backstage.io/edit-url: "<github-edit-url>"
    
    # Descriptive metadata
    openportal.dev/description: "<human-readable-description>"
    openportal.dev/icon: "<icon-name>"
    openportal.dev/tags: "<comma-separated-tags>"
    
    # Relationships
    openportal.dev/provides-api: "<api-identifier>"
    openportal.dev/depends-on: "<comma-separated-dependencies>"
    openportal.dev/part-of-system: "crossplane-templates"
    openportal.dev/domain: "infrastructure"
    
    # Operational metadata
    openportal.dev/sla: [best-effort|business-hours|24x7]
    openportal.dev/support-channel: "#<slack-channel>"
    openportal.dev/documentation: "<wiki-url>"
    
    # Publishing configuration
    terasky.backstage.io/publish-phase: |
      gitRepo: "github.com?owner=open-service-portal&repo=catalog-orders"
      gitBranch: "main"
      gitLayout: "cluster-scoped"
      basePath: "system/<ResourceKind>"
      createPr: true

2. Kubernetes Ingestor Configuration

Update app-portal/app-config.yaml with comprehensive mapping rules:

kubernetesIngestor:
  customResources:
    - group: apiextensions.crossplane.io
      apiVersion: v2
      plural: compositresourcedefinitions
      objectType: crossplane-template
      
      mappings:
        - target: catalog
          entity:
            apiVersion: backstage.io/v1alpha1
            kind: Component
            metadata:
              name: '{{ .metadata.name | replace "." "-" }}'
              namespace: default
              title: '{{ .spec.names.kind }} Template'
              description: '{{ .metadata.annotations["openportal.dev/description"] }}'
              
              labels:
                template-type: '{{ .metadata.labels["openportal.dev/template-type"] }}'
                category: '{{ .metadata.labels["openportal.dev/template-category"] }}'
                tier: '{{ .metadata.labels["openportal.dev/template-tier"] }}'
                cost-level: '{{ .metadata.labels["openportal.dev/cost-level"] }}'
                compliance: '{{ .metadata.labels["openportal.dev/compliance-required"] }}'
                version: '{{ .metadata.labels["openportal.dev/version"] }}'
                
              annotations:
                # Kubernetes references
                backstage.io/kubernetes-id: '{{ .metadata.name }}'
                backstage.io/kubernetes-namespace: '{{ .metadata.namespace | default "crossplane-system" }}'
                
                # Source and docs
                backstage.io/source-location: '{{ .metadata.annotations["backstage.io/source-location"] }}'
                backstage.io/techdocs-ref: '{{ .metadata.annotations["backstage.io/techdocs-ref"] }}'
                backstage.io/view-url: '{{ .metadata.annotations["backstage.io/view-url"] }}'
                backstage.io/edit-url: '{{ .metadata.annotations["backstage.io/edit-url"] }}'
                
                # Crossplane specifics
                openportal.dev/xrd-name: '{{ .metadata.name }}'
                openportal.dev/api-group: '{{ .spec.group }}'
                openportal.dev/api-version: '{{ .spec.versions[0].name }}'
                openportal.dev/api-kind: '{{ .spec.names.kind }}'
                openportal.dev/icon: '{{ .metadata.annotations["openportal.dev/icon"] }}'
                
                # Relationships
                openportal.dev/provides-api: '{{ .metadata.annotations["openportal.dev/provides-api"] }}'
                openportal.dev/depends-on: '{{ .metadata.annotations["openportal.dev/depends-on"] }}'
                
                # Operational
                openportal.dev/sla: '{{ .metadata.annotations["openportal.dev/sla"] }}'
                openportal.dev/support-channel: '{{ .metadata.annotations["openportal.dev/support-channel"] }}'
                
              tags: '{{ .metadata.annotations["openportal.dev/tags"] | split "," }}'
              
              links:
                - url: '{{ .metadata.annotations["backstage.io/source-location"] }}'
                  title: Source Code
                  icon: github
                - url: '{{ .metadata.annotations["openportal.dev/documentation"] }}'
                  title: Documentation
                  icon: docs
                  
            spec:
              type: crossplane-template
              lifecycle: '{{ .metadata.labels["openportal.dev/lifecycle"] | default "production" }}'
              owner: '{{ .metadata.labels["openportal.dev/owner"] | default "platform-team" }}'
              system: '{{ .metadata.annotations["openportal.dev/part-of-system"] }}'
              
              dependsOn: '{{ .metadata.annotations["openportal.dev/depends-on"] | split "," | prefix "resource:default/" }}'
              providesApis:
                - '{{ .metadata.name | replace "." "-" }}-api'

3. Supporting Entity Definitions

Create organizational structure in app-portal/examples/entities/:

# org.yaml - Organizational structure
---
apiVersion: backstage.io/v1alpha1
kind: Domain
metadata:
  name: infrastructure
  description: Infrastructure and platform services
spec:
  owner: platform-team

---
apiVersion: backstage.io/v1alpha1
kind: System
metadata:
  name: crossplane-templates
  description: Crossplane infrastructure provisioning templates
spec:
  domain: infrastructure
  owner: platform-team

---
apiVersion: backstage.io/v1alpha1
kind: Group
metadata:
  name: platform-team
  description: Platform engineering team
spec:
  type: team
  children: []
  members:
    - <user-entities>

# resources.yaml - Infrastructure resources
---
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: crossplane
  description: Crossplane control plane
spec:
  type: platform
  owner: platform-team
  system: crossplane-templates

---
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: provider-kubernetes
  description: Kubernetes provider for Crossplane
spec:
  type: crossplane-provider
  owner: platform-team
  system: crossplane-templates
  dependsOn:
    - resource:default/crossplane

---
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: provider-helm
  description: Helm provider for Crossplane
spec:
  type: crossplane-provider
  owner: platform-team
  system: crossplane-templates
  dependsOn:
    - resource:default/crossplane

📊 Implementation Tasks

Phase 1: Core Metadata (Priority: High)

• Update all XRDs with essential labels and annotations
• Configure kubernetes-ingestor mapping rules in app-config.yaml
• Create organizational entities (Domain, System, Group)
• Test ingestion and verify entity creation

Phase 2: Relationships (Priority: Medium)

• Add dependency annotations to all XRDs
• Create Resource entities for providers
• Map API relationships
• Implement system/domain associations

Phase 3: Operational Context (Priority: Medium)

• Add SLA and support channel annotations
• Include cost-level indicators
• Add compliance flags
• Link to documentation and runbooks

Phase 4: Advanced Features (Priority: Low)

• Custom processor for XRD → API entity generation
• Usage metrics integration
• Cost tracking annotations
• Health status indicators

🎯 Success Criteria

1. Discovery: All Crossplane templates appear in Backstage catalog with rich metadata
2. Categorization: Templates are properly categorized by type, tier, and cost
3. Relationships: Dependencies between templates and providers are visible
4. Documentation: Direct links to TechDocs, source code, and external docs
5. Operational: SLA, support channels, and compliance info readily available

📈 Benefits

• Improved Discovery: Developers can easily find the right template
• Cost Awareness: Cost levels help with resource planning
• Compliance Tracking: Compliance requirements are visible upfront
• Better Support: Direct links to support channels and documentation
• Relationship Visibility: Clear understanding of template dependencies

🔗 References

• Backstage Software Catalog
• Well-known Annotations
• Well-known Relations
• Descriptor Format
• System Model

📝 Example Implementation

See template-namespace as reference implementation:

• Enhanced XRD: template-namespace/configuration/xrd.yaml
• Catalog info: template-namespace/catalog-info.yaml
• TechDocs: template-namespace/mkdocs.yml

---

Labels: enhancement, documentation, backstage, crossplane, metadata, catalog
Assignees: @platform-team
Projects: Backstage Integration
Milestone: Q1 2025 - Catalog Enhancements

[felixboehm]

Suggest to develop like this:

Compare:

• template-export.sh: fetch generated backstage templates and store the yaml files in given folder
• template-ingest.sh: ingest given template and store the generated backstage template & api yaml files in given folder

Refactor:
template-ingest.sh uses ingestor.js which is a new CLI entrypoint for the kubernetes-ingestor plugin. during refactoring of the plugin we can compare the outputs and add unit tests. Refactor goal: smaller structures, reusable code.
This way we can see the ingestor results and understand the metadata information flow much better.

Improve:

• Add better metadata mappings: tech-docs, entity relations, ... --> Improves catalog UI
• Add options for creating steps (direct apply, github PR, ...) --> Options for how XRs should be created

[felixboehm]

Using the new ingestor plugin, this should be as easy as customizing the ingestor-templates.