Signature Initiative

Customer Service Hub — a production Databricks Lakehouse, deployed as code

Outcome 6 production workflows · 3 workspaces · 140 tests Notebooks, jobs, permissions & schema shipped atomically through CI/CD

Built and operate the Lakehouse that consolidates customer-support and engagement data — voice, chat, order events, cases and customer master data — into Unity Catalog for operational reporting, AI-driven interaction classification and PII-safe analytics. Every schema change, job and permission is version-controlled and promoted Dev → Stage → Prod by pipeline, never by hand.

Problem

  • Support and engagement data was spread across voice, chat, order-event and case systems, with no single governed place to analyse it.
  • Notebook and job deployment depended on a separate Git-sync process, so code and job definitions could drift apart between environments.
  • Unity Catalog schema changes were applied manually — a risk in Production and a bottleneck with several developers working in parallel.
  • Customer transcripts contain PII and had to be safe to analyse before anyone could classify or report on them.

Approach

  • CI/CD as code: a 4-stage Azure DevOps pipeline (Validate → Dev → Stage → Prod) deploying Databricks Asset Bundles with per-environment service principals, approval gates and a single validated artefact reused everywhere.
  • Bundle generator: Python tooling that assembles databricks.yml from per-job YAML, enforces naming rules, catches duplicates and missing notebooks at build time, and applies job permissions consistently.
  • Schema reconciliation engine: diffs version-controlled DDL against the live catalog, auto-applies safe changes (new columns, type widening) and routes destructive or ambiguous drift to a conditional review gate that blocks only in Production.
  • Ingestion & AI: metadata-driven, watermark-based ingestion; Azure Service Bus → Auto Loader every 15 minutes; GPU PII masking (XLM-RoBERTa + Presidio) followed by AI_CLASSIFY interaction categorisation.

Outcome

  • 6 production Databricks Workflows live across 3 workspaces, running every 15 minutes to daily.
  • Atomic deployments: notebooks, jobs and permissions ship together; the separate Git-sync dependency is gone.
  • 1,042 → 228 lines of pipeline YAML through reusable templates; 140 pytest tests run on every build.
  • Governance enforced in CI: six DDL rules; 40 tables, 20 routines and 2 views remediated in one pass.
  • Safe schema evolution without blocking parallel work in Dev/Stage, and human review where it matters in Prod.

Databricks · Unity Catalog · Delta Lake · Asset Bundles · Auto Loader · Workflows · PySpark · Azure DevOps · Azure Service Bus · pytest · Presidio · XLM-RoBERTa · AI_CLASSIFY

Also at Ashley

AI-integrated BigQuery SDLC — a repo-first operating model

Exported BigQuery into a Git repo via INFORMATION_SCHEMA, wrote RFC-2119 standards for AI coding agents, and designed a human-gated workflow (feature branch → PR review → generated change script → owner deployment) so AI pair-programming could be used safely on the warehouse. Representative delivery went from ~5 days to 1–2 days, and other Ashley teams now use it as the reference pattern.

BigQuery · GCS · Augment Code · Git · GitHub PRs · INFORMATION_SCHEMA · PowerShell

Reference Architecture

Customer Service Hub data flow — ingest, protect, classify, serve

Problem. Voice, chat, order-event, case and customer master data lived in separate systems, and interaction transcripts carried PII that made them unsafe to analyse as-is.

Approach. Land order events from Azure Service Bus as parquet and pick them up with Auto Loader; drive every other source through a config-table ingestion framework with watermarks; layer data raw → staging → processed in Unity Catalog; mask PII on GPU before any LLM classification touches a transcript.

Outcome. A governed Lakehouse serving operational reporting, engagement insight and AI-classified interactions — with new sources onboarded by configuration rather than new code, and the whole stack promoted Dev → Stage → Prod by pipeline.

  • FreshnessOrder events every 15 minutes
  • ReliabilityWatermark loads · idempotent reprocessing
  • PrivacyPII masked before classification
  • GovernanceUnity Catalog · DDL rules enforced in CI
Source Systems Voice · Chat · Orders Sources Azure Service Bus Order events → parquet Capture Auto Loader Raw zone · watermarks Ingest Stage → Processed PySpark · SQL procedures PII · Classify Presidio · AI_CLASSIFY Transform Unity Catalog Delta · 5 domains Reporting Job failure alerts Serve order events every 15 min metadata-driven GPU · LLM governed consumer surface

Azure Service Bus · Auto Loader · Delta Lake · PySpark · Unity Catalog · Presidio · XLM-RoBERTa · AI_CLASSIFY · Databricks Workflows

Simplified view of the Customer Service Hub platform at Ashley. Earlier programs used the same capture → transform → serve → observe contract on ADF, Synapse, Dataflow and BigQuery.

Experience

Delivery timeline with outcomes

Total experience: -- since June 2022.

Dec 2024 – Present

Data Engineer · Ashley Global Capability Center

  • Built the Customer Service Hub Databricks Lakehouse end-to-end — a 4-stage Azure DevOps CI/CD framework on Databricks Asset Bundles deploying notebooks, jobs, permissions and config atomically across Dev, Stage and Prod. See the initiative →
  • Engineered a Python/PySpark Unity Catalog reconciliation framework that auto-applies safe schema evolution and routes destructive drift through conditional review; backed by 140 pytest tests and six CI-enforced DDL rules.
  • Built and operate 6 production Databricks Workflows with metadata-driven, watermark-based ingestion, including a 15-minute Azure Service Bus → Auto Loader order-event flow.
  • Delivered a GPU-accelerated PII-masking and interaction-classification pipeline using XLM-RoBERTa Large, Microsoft Presidio and Databricks AI_CLASSIFY.
  • Re-engineered an encrypted-file processing workload on Google Cloud Workflows and Cloud Run Jobs with parallel processing — runtime cut from ~4 hours to ~10 minutes (≈20×).
  • Designed the repo-first, AI-integrated BigQuery SDLC now used as the reference pattern by other Ashley teams (~5 days → 1–2 days per change).
Jul 2023 – Dec 2024

Data Engineer · Ashvad Tech Private Limited (formerly Ashtbit Technologies)

  • Built scalable ETL pipelines using Azure Data Factory and Synapse for enterprise reporting and analytics.
  • Delivered .NET Web API integration services and Hangfire-based orchestration for automated workflows.
  • Enhanced Power BI reporting and implemented the migration to Entra-only authentication to strengthen access governance.
Jul 2022 – Jul 2023

Data Engineer · Provana

  • Developed automated ETL frameworks in ADF and Synapse for repeatable onboarding and operations.
  • Configured Logic Apps and KQL-driven monitoring patterns for proactive incident handling.
  • Improved operational throughput through workflow standardization and automation.

Personal repos

Sandbox & side work

A selection of my personal GitHub repos — sandboxes where I prototype patterns and explore tools. Employer work is covered in Experience and the Signature Initiative.

Analytics Pipeline

TripAdvisor Reviews Analysis

Problem: Build robust sentiment analytics with business-ready reporting outputs.

Built: End-to-end Azure analytics pipeline using Synapse, Data Lake, Data Warehouse, Power BI, and Azure ML.

Result: Personal sandbox — produced a working end-to-end pattern (ingest → warehouse → ML → Power BI) on sample TripAdvisor data; useful as a reference layout, not a production deployment.

Synapse · Data Lake · Data Warehouse · Power BI · Azure ML

Open Repository

Near Real-Time Signals

Azure Real-Time Twitter Data

Problem: Capture social signals and process them continuously for near real-time analysis.

Built: Streaming workflow with Azure Functions, Event Hub, and Stream Analytics orchestrated through Python.

Result: Personal sandbox — a runnable proof of an Azure streaming wiring (Function → Event Hub → Stream Analytics) built to learn the moving parts, not to operate continuously.

Python · Azure Functions · Event Hub · Stream Analytics

Open Repository

Cloud-Native API

Cosmos DB Books Library API

Problem: Demonstrate a practical REST API backed by a globally distributed NoSQL store with clean data-access patterns.

Built: Python service exposing CRUD endpoints over an Azure Cosmos DB container with environment-driven configuration.

Result: Personal sandbox — a compact, deployable reference for Cosmos-backed CRUD and document modelling; intended as a starting template, not a live service.

Python · REST · Azure Cosmos DB

Open Repository

Now

Currently building & learning

Building

Hardening the Unity Catalog reconciliation engine — richer drift classification and a reviewed change-script path for destructive schema changes.

Building

A vendor-agnostic customer-service data model that treats human, automated and AI-assisted resolution as first-class — ready for new conversational-AI sources without re-architecture.

Learning

Microsoft Fabric Real-Time Intelligence — eventstreams, KQL DB, and Activator patterns for low-latency analytics.

Week 1

What I'd build for your team in week 1

  • Run a schema-drift audit comparing your Unity Catalog (or warehouse) against what's in source control, and ship a one-page risk note on what would break a deploy.
  • Put one Databricks job under Asset Bundles with a Dev → Prod pipeline, service-principal auth and an approval gate — a working template the rest can copy.
  • Document one repo-first standard for AI-assisted change (naming, idempotent DDL, PR gate) so the next 10 changes follow the same shape.

Concrete, low-blast-radius, designed to surface what to invest in next.

Credentials & contact

Let us build something meaningful

Awards

Dirty Fingernail Award · 2025 Above & Beyond · Q4 2025

Internal recognition for hands-on, deep-dive problem solving on complex production issues.

Certifications

Microsoft Azure Data Scientist Associate · Recert Nov 2025 DeepLearning.AI TensorFlow Developer

Education

Bachelor’s in Computer Science · Karunya Institute of Technology and Sciences · 2014–2018

Open to

Data Engineer · Cloud Data Platform Engineer · Chennai, India · remote & hybrid welcome

If your team is building Databricks Lakehouse platforms, automating data-platform delivery, or modernizing with reliable AI-assisted engineering, I am open to collaborating.

Email LinkedIn GitHub