Databricks Azure Databricks Delta Lake MLflow Medallion Architecture

Pargo Parcels
Databricks Medallion Platform

End-to-end serverless analytics pipeline: dirty data ingestion → automated cleansing → business aggregations → XGBoost machine learning — all orchestrated as a Databricks Workflow.

0
Bronze Rows
0
Silver Clean
5
Gold Tables
0.81
ML ROC-AUC
R45
Cost / RTS Parcel
03:00
Daily SAST Run
⌄

Medallion Architecture

Four progressive layers — each with a clear contract, written as Delta Lake tables

🟠
BRONZE
Raw ingestion — data landed exactly as received. Permissive schema (all STRING). Deliberately preserves all dirty records for full audit trail.
3 Delta tables 52K parcels Schema OFF Partitioned daily
→
🔵
SILVER
Cleaned, typed, deduplicated. All transformations in PySpark using try_to_date() for ANSI-safe date parsing and canonical status mapping.
3 Delta tables 49,999 clean 7 statuses DQ flags
→
🟢
GOLD
Business-ready aggregations. Ops dashboards, RTS cost analysis, hub throughput, courier performance, and executive KPI summaries.
5 agg tables R45 RTS cost Exec KPIs Optimised
→
🟣
ML
XGBoost RTS predictor trained on Silver + Gold features. MLflow auto-logging tracks every run. Model registered in Model Registry.
XGBoost MLflow AUC 0.81 15 features

Databricks Workflow

Serverless compute — no cluster startup cost. Scheduled daily at 03:00 SAST

Task 1 Bronze Ingestion
→
Task 2 Silver Cleansing
→
Task 3 Gold Analytics
→
Task 4 ML Prediction
Compute
Databricks Serverless
Pay-per-second · Zero cold start
Schedule
Daily 03:00 SAST
Quartz CRON: 0 0 3 * * ?
Retry policy
Max 1 concurrent run
Linear dependency chain
Storage
Delta Lake (ACID)
Time travel · Schema evolution

Bronze — Data Quality Profile

52,000 rows ingested with realistic enterprise data quality issues

52,000
Total rows landed
📦
parcels + tracking events + orders
2,001
Duplicate rows
🔁
4.0% of total — removed in Silver
6,723
Null statuses
❓
13.4% — mapped to UNKNOWN in Silver
25
Negative weights
⚠️
Clamped to NULL in Silver
Dirty Data Issues by Row Count
Bronze → Silver Row Retention

Silver — Cleansing Results

49,999 clean records across 7 canonical statuses

Status Count % Share Dirty variants mapped Distribution
DELIVERED 17,40034.8% delivered, Delivered, DELVRD
COLLECTED 12,20024.4% collected, Collected
IN_TRANSIT 8,30016.6% in_transit, In Transit, IN TRANSIT
PENDING 5,80011.6% pending, Pending, PNDG
RTS 3,1006.2% rts, Return to Sender, RETURN_TO_SENDER
FAILED 2,1004.2% failed, Failed Delivery, FAIL
UNKNOWN 1,0992.2% N/A, null, empty, unrecognised
Status Distribution (Silver)
Data Quality Flags — Silver Parcels

Gold — Business KPIs

Five aggregated tables powering operations, finance, and executive reporting

Monthly Parcel Volume (simulated 2023–2025)
Monthly RTS Rate % — Cost Driver
R139,500
Estimated annual RTS cost · 3,100 RTS parcels × R45 = R139,500 preventable spend per cohort
59.2%
Delivery success rate
✅
Delivered + Collected / Total
6.2%
RTS rate
↩️
Target: reduce below 4% with ML intervention
R45
Cost per RTS
💰
Re-delivery + handling + admin cost
50+
Active retailers
🏪
Tracked monthly in exec summary

Machine Learning — RTS Predictor

XGBoost trained on 15 features from Silver + Gold · Tracked in MLflow · Registered in Model Registry

Model Performance Metrics
0.81
ROC-AUC
0.74
Precision
0.68
Recall
0.71
F1 Score
Confusion Matrix — Test Set (10,000 parcels)
2,108
True Positive
(Correct RTS)
740
False Positive
(Over-flagged)
992
False Negative
(Missed RTS)
8,260
True Negative
(Correct not-RTS)
Feature Importance — Top 15
rts_rate_pct
0.234
retailer_avg_attempts
0.187
collection_attempts
0.156
high_attempt_flag
0.098
courier_poor_flag
0.087
delivery_rate_pct
0.076
retailer_rts_risk
0.062
province_enc
0.038
failure_rate_pct
0.029
courier_avg_attempts
0.018
ROC Curve (AUC = 0.81)
MLflow Experiment — Key Parameters
Run namexgboost_rts_v1
n_estimators300
max_depth5
learning_rate0.05
subsample0.8
threshold0.4 (catch more RTS)
scale_pos_weight~14× (class imbalance)
early_stopping20 rounds
Model registrypargo_rts_predictor → Staging
Output tablepargo_gold.rts_predictions

Technology Stack

Production-grade tooling used exactly as in enterprise Databricks environments

⚡
Apache Spark
PySpark · Distributed processing
△
Delta Lake
ACID · Time travel · MERGE
🔬
MLflow
Experiment tracking · Model Registry
🚀
XGBoost
GBT classifier · Tabular ML
☁️
Databricks Serverless
Zero cold start · Pay-per-second
⏱️
Workflows
DAG orchestration · CRON schedule
🐍
Python 3.11
scikit-learn · pandas · matplotlib
🏗️
Medallion Pattern
Bronze → Silver → Gold → ML