Forty analysts spent most of every day copying rates, schedules, and capacity data from 300+ carrier portals into spreadsheets. Errors compounded downstream; pricing decisions ran on week-old numbers. Two prior automation attempts had collapsed under portal changes and IP blocks.
We built a distributed scraping fleet with per-portal adapters, automatic schema-drift detection, and rotating residential egress. Extracted data flows through a cleaning and enrichment layer into their warehouse, with anomaly alerts before bad data reaches a dashboard.
Analysts moved from data entry to data review: a triage UI surfaces only the records the pipeline isn't confident about.
Manual data operations dropped 80% within the first quarter. Pricing now runs on same-day numbers, and the pipeline has held 99.98% uptime through hundreds of portal changes — each absorbed by adapters, not analysts.
"They rebuilt in four months what our previous vendor couldn't stabilize in two years."