MDS Data Loading and Caching

MDS Data Loading and Caching System

Table of Contents

1. Introduction
2. System Architecture Overview
3. Lazy Loading Mechanism
4. Two-Phase Loading Process
5. Background Processing System
6. Caching Strategy
7. Certificate Information Management
8. Performance Optimization
9. Implementation Details
10. Usage Examples
11. Troubleshooting

Introduction

The MDS (Metadata Statement) data loading and caching system is a sophisticated mechanism designed to handle large-scale authenticator metadata efficiently while maintaining a responsive user interface. The system implements a lazy loading architecture that optimizes performance by deferring expensive operations until they are actually needed.

The core challenge addressed by this system is the efficient handling of large MDS datasets (containing thousands of authenticator entries) while providing immediate UI responsiveness. This is achieved through a combination of lightweight initial parsing, background processing, and intelligent caching strategies.

System Architecture Overview

The MDS data loading system consists of several interconnected components that work together to provide efficient metadata management:

graph TB
subgraph "Client-Side Components"
MdsLazyLoader["MdsLazyLoader<br/>Core Lazy Loader"]
TransformUtils["Transform Utilities<br/>Entry Processing"]
BackgroundProcessor["Background Processor<br/>Batch Operations"]
CertificateCache["Certificate Cache<br/>Decoded Certificates"]
end
subgraph "Server-Side Components"
MetadataServer["Metadata Server<br/>Data Serving"]
MdsVerifier["MDS Verifier<br/>Signature Validation"]
CertificateDecoder["Certificate Decoder<br/>X.509 Parsing"]
end
subgraph "Data Flow"
RawData["Raw MDS Entries<br/>JSON Payload"]
LightEntries["Lightweight Entries<br/>UI-Ready"]
FullEntries["Full Entries<br/>Complete Metadata"]
CertificateDetails["Certificate Details<br/>Algorithm Info"]
end
RawData --> MdsLazyLoader
MdsLazyLoader --> TransformUtils
TransformUtils --> LightEntries
LightEntries --> BackgroundProcessor
BackgroundProcessor --> FullEntries
FullEntries --> CertificateCache
CertificateCache --> CertificateDetails
MetadataServer --> MdsLazyLoader
MdsVerifier --> MetadataServer
CertificateDecoder --> CertificateCache

Loading

Diagram sources

• mds-lazy-loader.js
• mds.js

Lazy Loading Mechanism

The lazy loading mechanism is implemented through the MdsLazyLoader class, which serves as the central orchestrator for deferred metadata processing. This system optimizes performance by initially parsing only the essential fields needed for UI display, deferring full parameter parsing until absolutely necessary.

Core Lazy Loader Features

The MdsLazyLoader class provides several key capabilities:

• Deferred Parsing: Initial lightweight parsing using transformEntryLightweight
• Background Processing: Asynchronous full parsing in batches
• Progress Tracking: Real-time progress monitoring during background loading
• Caching Management: Intelligent caching of processed entries
• Memory Optimization: Efficient memory usage through selective parsing

Initialization and Setup

The lazy loader is initialized with the complete metadata object and maintains internal state for tracking parsed entries:

sequenceDiagram
participant Client as "Client Application"
participant Loader as "MdsLazyLoader"
participant Background as "Background Processor"
participant Utils as "Transform Utils"
Client->>Loader : createMdsLazyLoader()
Client->>Loader : initialize(metadata)
Loader->>Loader : setup internal state
Note over Loader : Initial lightweight parsing
loop For each entry
Loader->>Utils : transformEntryLightweight(entry)
Utils-->>Loader : lightweight entry
end
Note over Loader,Background : Background processing starts
Loader->>Background : startBackgroundLoading()
Background->>Background : process in batches
loop Batch Processing
Background->>Utils : upgradeEntryToFull(entry)
Utils-->>Background : full entry
Background->>Loader : markEntryFullyParsed()
end

Loading

Diagram sources

• mds-lazy-loader.js
• mds-utils.js

Section sources

• mds-lazy-loader.js
• mds-utils.js

Two-Phase Loading Process

The system implements a sophisticated two-phase loading process that balances initial UI responsiveness with complete metadata availability:

Phase 1: Lightweight Parsing (Initial Load)

During the first phase, the system performs minimal parsing to create entries suitable for immediate UI display:

flowchart TD
Start([Start Initial Load]) --> ParseRaw["Parse Raw Metadata"]
ParseRaw --> CreateLightweight["Create Lightweight Entries"]
CreateLightweight --> ExtractBasic["Extract Basic Fields:<br/>• Name<br/>• Protocol<br/>• Certification<br/>• Identifier<br/>• Icon"]
ExtractBasic --> DeferComplex["Defer Complex Fields:<br/>• Certificate Details<br/>• Algorithm Info<br/>• Status Reports"]
DeferComplex --> StoreReferences["Store Raw References"]
StoreReferences --> BuildIndex["Build AAGUID Index"]
BuildIndex --> ReadyUI["UI Ready"]
ReadyUI --> StartBackground["Start Background Processing"]
StartBackground --> End([Phase Complete])

Loading

Diagram sources

• mds-utils.js

The lightweight transformation process focuses on extracting only the essential fields required for the main authenticator list view. Complex fields like certificate details and algorithm information are deferred to later processing phases.

Phase 2: On-Demand Full Parsing

The second phase involves upgrading lightweight entries to full entries when specific information is requested:

flowchart TD
Request[User Requests Details] --> CheckLightweight{"Is Entry Lightweight?"}
CheckLightweight --> |Yes| UpgradeEntry["Upgrade to Full Entry"]
CheckLightweight --> |No| ReturnCached["Return Cached Full Entry"]
UpgradeEntry --> ParseComplex["Parse Complex Fields:<br/>• Certificate Details<br/>• Algorithm Information<br/>• Status Reports<br/>• Attestation Keys"]
ParseComplex --> UpdateCache["Update Entry Cache"]
UpdateCache --> MarkParsed["Mark as Fully Parsed"]
MarkParsed --> ReturnFull["Return Full Entry"]
ReturnCached --> End([Complete])
ReturnFull --> End

Loading

Diagram sources

• mds-utils.js

Section sources

• mds-utils.js

Background Processing System

The background processing system manages the asynchronous parsing of metadata entries in batches to maintain UI responsiveness while ensuring complete data availability.

Batch Processing Configuration

The system uses configurable batch sizes and delays to optimize performance:

Parameter	Value	Purpose
BACKGROUND_BATCH_SIZE	200	Number of entries processed per batch
BACKGROUND_BATCH_DELAY_MS	16	Delay between batches (approximates 1 frame)

Background Processing Workflow

sequenceDiagram
participant UI as "User Interface"
participant Loader as "MdsLazyLoader"
participant BatchProcessor as "Batch Processor"
participant CertificateDecoder as "Certificate Decoder"
participant Cache as "Certificate Cache"
UI->>Loader : startBackgroundLoading()
Loader->>BatchProcessor : processBatch(indices)
loop For Each Batch
BatchProcessor->>BatchProcessor : filter valid indices
BatchProcessor->>BatchProcessor : extract entries
par Certificate Processing
BatchProcessor->>CertificateDecoder : decodeCertificate(certs)
CertificateDecoder->>Cache : cache decoded results
Cache-->>CertificateDecoder : cached details
end
BatchProcessor->>Loader : markEntryFullyParsed()
Loader->>UI : progress callback
end
BatchProcessor->>Loader : onComplete()
Loader->>UI : completion callback

Loading

Diagram sources

• mds-lazy-loader.js
• mds.js

Progress Monitoring and Control

The background processing system provides comprehensive progress monitoring and control mechanisms:

• Progress Callbacks: Real-time progress updates during batch processing
• Completion Callbacks: Finalization notifications when processing completes
• Abort Support: Graceful termination of background operations
• Error Handling: Robust error recovery and reporting

Section sources

• mds-lazy-loader.js
• mds.js

Caching Strategy

The system implements a multi-layered caching strategy to optimize performance and reduce redundant processing:

Certificate Information Caching

The certificate cache stores decoded X.509 certificate information to avoid repeated network requests and parsing operations:

graph LR
subgraph "Certificate Cache Flow"
Request[Certificate Request] --> CheckCache{"Cache Hit?"}
CheckCache --> |Yes| ReturnCached[Return Cached Details]
CheckCache --> |No| DecodeCert[Decode Certificate]
DecodeCert --> StoreCache[Store in Cache]
StoreCache --> ReturnNew[Return New Details]
end
subgraph "Cache Storage"
Base64Cert[Base64 Certificate] --> NormalizedCert[Normalized Certificate]
NormalizedCert --> DecodedDetails[Decoded Certificate Details]
DecodedDetails --> CacheEntry[Cache Entry]
end

Loading

Diagram sources

• mds.js

Entry Caching Strategy

The system maintains separate caches for different entry types:

• Lightweight Entries: Cached immediately after initial parsing
• Full Entries: Cached after complete processing
• Certificate Details: Cached separately for reuse across entries

Memory Management

The caching system implements intelligent memory management:

• LRU Eviction: Least recently used entries are evicted when memory limits are reached
• Selective Caching: Only frequently accessed entries are kept in memory
• Cleanup Triggers: Automatic cleanup based on memory pressure and access patterns

Section sources

• mds.js

Certificate Information Management

The system provides comprehensive certificate information management, including decoding, caching, and presentation of X.509 certificate details.

Certificate Decoding Process

flowchart TD
Input[Certificate Input] --> Normalize[Normalize Base64]
Normalize --> CheckCache{"Check Certificate Cache"}
CheckCache --> |Hit| ReturnCached[Return Cached Details]
CheckCache --> |Miss| SendRequest[Send Decode Request]
SendRequest --> ServerDecode[Server-Side Decode]
ServerDecode --> ParseX509[Parse X.509 Structure]
ParseX509 --> ExtractInfo[Extract Certificate Info]
ExtractInfo --> CacheResult[Cache Result]
CacheResult --> ReturnDetails[Return Details]
ReturnCached --> End([Complete])
ReturnDetails --> End

Loading

Diagram sources

• mds.js

Certificate Information Extraction

The system extracts comprehensive information from X.509 certificates:

Field	Description	Usage
Algorithm Information	Public key algorithm and parameters	Security assessment
Subject Common Names	Certificate subject information	Identity verification
Validity Period	Certificate expiration dates	Trust assessment
Signature Algorithm	Digital signature algorithm	Cryptographic strength
Key Usage	Permitted certificate usages	Access control

Certificate Summary Generation

The system generates human-readable summaries of certificate information:

graph TB
subgraph "Certificate Summary Components"
Algorithm[Algorithm Info<br/>RSA-2048, ECDSA-P256]
Subject[Subject Names<br/>DigiCert, Google]
Validity[Validity Period<br/>2023-01-01 to 2028-01-01]
Extensions[Extensions<br/>Key Usage, Extended Key Usage]
end
subgraph "Summary Output"
FormattedSummary[Formatted Summary Text]
InteractiveDetails[Interactive Details Panel]
end
Algorithm --> FormattedSummary
Subject --> FormattedSummary
Validity --> FormattedSummary
Extensions --> InteractiveDetails

Loading

Diagram sources

• mds.js

Section sources

• mds.js
• mds.js

Performance Optimization

The system implements several performance optimization strategies to handle large MDS datasets efficiently:

Browser Responsiveness

The system ensures browser responsiveness through several mechanisms:

• Yielding to Browser: Uses requestIdleCallback and requestAnimationFrame for optimal timing
• Batch Processing: Processes entries in small batches to prevent UI blocking
• Progress Updates: Provides frequent progress updates to maintain user feedback

Memory Efficiency

Memory usage is optimized through:

• Lazy Loading: Only loads data when needed
• Selective Parsing: Parses only required fields initially
• Garbage Collection: Proper cleanup of unused data structures

Network Optimization

Network requests are optimized through:

• Certificate Caching: Reuses decoded certificate information
• Batch Requests: Groups related operations to reduce overhead
• Compression: Uses compressed data transmission

Section sources

• mds-lazy-loader.js

Implementation Details

Core Classes and Functions

The system is built around several key classes and functions:

MdsLazyLoader Class

The MdsLazyLoader class serves as the primary interface for lazy loading operations:

// Constructor initializes internal state
constructor() {
    this.allEntries = [];
    this.fullyParsedIndices = new Set();
    this.fullyParsedKeys = new Map();
    this.isBackgroundLoading = false;
    this.backgroundLoadComplete = false;
    this.backgroundLoadAborted = false;
    this.backgroundLoadPromise = null;
    this.onProgressCallback = null;
    this.onCompleteCallback = null;
    this.metadata = null;
}

Transformation Functions

The system provides specialized transformation functions for different processing stages:

• transformEntryLightweight(): Creates lightweight entries for initial UI display
• upgradeEntryToFull(): Completes parsing of deferred fields
• transformEntry(): Performs complete entry transformation

Background Processing

Background processing is handled through configurable batch operations:

// Batch processing configuration
const BACKGROUND_BATCH_SIZE = 200;
const BACKGROUND_BATCH_DELAY_MS = 16;

// Batch processing function
const processBatch = async (batchIndices) => {
    const validIndices = batchIndices.filter(i => 
        typeof i === 'number' && i >= 0 && i < mdsData.length
    );
    
    const batchEntries = validIndices.map(i => mdsData[i]).filter(entry => entry != null);
    await populateCertificateDerivedInfoForBatch(batchEntries);
};

Section sources

• mds-lazy-loader.js
• mds-utils.js

Usage Examples

Basic Lazy Loading Setup

// Create lazy loader instance
const lazyLoader = createMdsLazyLoader();

// Initialize with metadata
lazyLoader.initialize(metadata);

// Start background loading
lazyLoader.startBackgroundLoading({
    signal: abortController.signal,
    onBatchProcessed: processBatch
}).then(() => {
    console.log('Background loading complete');
});

Certificate-Based Entry Lookup

// Find entries with specific certificate
const matchingEntries = lazyLoader.findEntriesWithCertificate(certificateBase64);

// Upgrade entries to full version when needed
matchingEntries.forEach(rawEntry => {
    const key = normaliseAaguid(rawEntry?.aaguid);
    const existingEntry = mdsState.byAaguid.get(key);
    
    if (existingEntry && existingEntry.isLightweightEntry) {
        const fullEntry = upgradeEntryToFull(existingEntry);
        Object.assign(existingEntry, fullEntry);
        lazyLoader.markEntryFullyParsed(existingEntry.index, key);
    }
});

Progress Monitoring

// Set up progress callbacks
lazyLoader.onProgress((parsed, total, percent) => {
    updateProgressBar(parsed, total, percent);
});

lazyLoader.onComplete(() => {
    console.log('All entries fully parsed');
});

Troubleshooting

Common Issues and Solutions

Background Loading Performance

Issue: Background loading is slow or unresponsive Solution: Check browser performance and adjust batch sizes if necessary

Issue: Memory usage increases rapidly Solution: Monitor cache sizes and implement cache eviction policies

Certificate Decoding Failures

Issue: Certificate decoding fails consistently Solution: Verify certificate format and check server-side certificate decoding service

Issue: Certificate cache becomes stale Solution: Implement cache invalidation based on metadata updates

UI Responsiveness

Issue: UI becomes unresponsive during loading Solution: Ensure proper yielding to browser and check batch processing intervals

Debugging Tools

The system provides several debugging capabilities:

• Progress Tracking: Real-time progress monitoring
• Error Logging: Comprehensive error reporting
• Statistics: Loading statistics and performance metrics
• Abort Support: Graceful termination of operations

Section sources

• mds-lazy-loader.js
• mds.js