🗄️
Large-Scale Freight Data Migration Pipeline (1.38M Files)
Architected an asynchronous data extraction and sanitization pipeline transferring over 1.38 million freight records from legacy eLS into Rail-Flow TMS with zero loss.
Overview
As CargoBeamer prepared to go live with the Rail-Flow Transport Management System (TMS), years of freight, customer, and trailer records were locked inside the legacy eLS portal without standard public export APIs.
I served as the sole architect and engineer responsible for designing, building, and executing the complete asynchronous extraction and migration pipeline.
Engineering Architecture
-
Resilient Asynchronous Scraping:
- Engineered an async Python pipeline using Playwright and FastAPI with intelligent retry logic and session management.
- Designed custom indicators and verification passes to detect missed records and automatically trigger targeted re-scraping.
-
Multi-Model Ingestion:
- Extracted and normalized ~26 distinct data object types across Customer, Disposition, and Terminal operational modules.
- Handled complex multi-part forms and binary document attachments.
-
Cloud Storage & Database Seeding:
- Streamed extracted documents and structured metadata directly to AWS S3 and Cloudflare R2 object storage.
- Built sanitization scripts to merge and seed multiple source databases into unified production schemas.
Impact
- Successfully migrated over 1.38 million documents in the largest category alone (Trailer Reports).
- Achieved zero data loss and enabled on-schedule SaaS cutover to Rail-Flow TMS.