Event-Driven Pipeline with Azure Functions

A complete event-driven data pipeline that processes JSON files uploaded to Azure Blob Storage and saves the data to Azure Cosmos DB using Azure Functions.

🏗️ Architecture

• Azure Blob Storage: Triggers when JSON files are uploaded to the "data" container
• Azure Functions: Processes the uploaded files and extracts data
• Azure Cosmos DB: Stores the processed data in a NoSQL database
• Terraform: Infrastructure as Code for Azure resource provisioning

📋 Prerequisites

Before running this project, ensure you have:

• Node.js (v18 or higher)
• Azure CLI
• Azure Functions Core Tools
• Terraform (optional, for infrastructure setup)
• An active Azure subscription

🚀 Quick Start

Step 1: Clone the Repository

git clone https://github.com/gopalepic/Event-driven-pipeline.git
cd Event-driven-pipeline

Step 2: Set Up Azure Resources

Using Terraform

1. Navigate to the root directory and initialize Terraform:

terraform init
terraform plan
terraform apply

2. Note down the output values:
   • Storage Account connection string
   • Cosmos DB endpoint and key

Step 3: Configure the Function App

1. Navigate to the function app directory:

cd process_data

2. Install dependencies:

npm install

3. Create local.settings.json file with your Azure credentials:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=YOUR_STORAGE_ACCOUNT;AccountKey=YOUR_STORAGE_KEY;EndpointSuffix=core.windows.net",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "COSMOSDB_ENDPOINT": "",
    "COSMOSDB_KEY": "YOUR_COSMOS_PRIMARY_KEY"
  }
}

Step 4: Build and Run

1. Build the TypeScript code:

npm run build

2. Start the Azure Functions runtime:

func start

You should see output similar to:

Azure Functions Core Tools
Core Tools Version: 4.x.x
Function Runtime Version: 4.x.x

Functions:
  blobTrigger: blobTrigger

Host lock lease acquired by instance ID 'xxxxx'.

📝 Testing the Pipeline

Test with Sample Data

1. Create a test JSON file (test-data.json):

[
  {"id": "1", "value": "test data"},
  {"id": "2", "value": "more test data"},
  {"id": "3", "value": "even more test data"}
]

2. Upload the file to your Azure Storage blob container named "data"

3. Watch the function logs - you should see:

Processing blob: test-data.json
Saved item with id: 1
Saved item with id: 2
Saved item with id: 3
Data saved to Cosmos DB - 3 items processed

🔧 Configuration

Environment Variables

Variable	Description	Example
AzureWebJobsStorage	Storage account connection string	DefaultEndpointsProtocol=https;AccountName=...
COSMOSDB_ENDPOINT	Cosmos DB endpoint URL	https://myaccount.documents.azure.com:443/
COSMOSDB_KEY	Cosmos DB primary key	AccountKey=xxxxx

Supported Data Formats

The function supports both:

• Single JSON object: {"id": "1", "value": "data"}
• Array of JSON objects: [{"id": "1", "value": "data"}, {"id": "2", "value": "more data"}]

Each object must have an id field for Cosmos DB partitioning.

📁 Project Structure

Event-driven-pipeline/
├── main.tf                      # Terraform infrastructure
├── process_data/
│   ├── src/
│   │   └── functions/
│   │       └── ProcessBlob.ts   # Main function code
│   ├── dist/                    # Compiled JavaScript
│   ├── package.json             # Dependencies
│   ├── host.json               # Function app configuration
│   ├── local.settings.json     # Local environment variables
│   └── tsconfig.json           # TypeScript configuration
└── README.md                   # This file

🛠️ Development

Making Changes

1. Switch to the problems branch:

git checkout problems

2. Make your changes to process_data/src/functions/ProcessBlob.ts

3. Build and test:

cd process_data
npm run build
func start

4. Commit and push:

git add .
git commit -m "Your changes"
git push origin problems

5. Create a Pull Request to merge into main

Local Development

For local development, you can use Azure Storage Emulator:

# Install Azurite (Azure Storage Emulator)
npm install -g azurite

# Start Azurite
azurite --silent --location c:\azurite --debug c:\azurite\debug.log

Update local.settings.json:

{
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    ...
  }
}

🔍 Troubleshooting

Common Issues

1. Function not starting:

   • Ensure you're in the process_data directory
   • Check that host.json and local.settings.json exist

2. Cosmos DB connection errors:

   • Verify your COSMOSDB_ENDPOINT and COSMOSDB_KEY values
   • Ensure the database "pipeline-db" and container "data" exist

3. Blob trigger not firing:

   • Check your storage account connection string
   • Ensure the "data" container exists
   • Verify blob files are being uploaded correctly

4. TypeScript compilation errors:

   • Run npm install to ensure all dependencies are installed
   • Check tsconfig.json configuration

Debugging

Enable verbose logging by starting with:

func start --verbose

📊 Monitoring

Azure Portal

Monitor your pipeline through:

• Function App: View execution logs and metrics
• Storage Account: Monitor blob uploads and triggers
• Cosmos DB: Check data insertion and query performance

Local Monitoring

The function logs will show:

• Blob processing events
• Individual item saves to Cosmos DB
• Error messages and stack traces

🔒 Security

Production Deployment

For production:

1. Use Azure Key Vault for secrets
2. Enable managed identity
3. Configure network security rules
4. Set up monitoring and alerts

Environment Variables

Never commit local.settings.json to version control. It's already included in .gitignore.

📄 License

This project is licensed under the MIT License.

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

📞 Support

If you encounter any issues:

1. Check the troubleshooting section above
2. Review Azure Functions documentation
3. Open an issue in this repository

---

Happy coding! 🚀