FreightSight-Ingest

Turn any messy freight file into normalized, canonical loads — with per-row confidence and cross-border crossing inference.

Every logistics team fights the same problem: a customer emails a spreadsheet, and no two look alike. Different column names, junk header rows, dates in five formats, weights in lbs or kg, cities glued to states. freightsight-ingest reads that file, figures out what every column means, normalizes the values, and — for cross-border lanes — infers which commercial border crossing the lane most likely uses.

• Zero runtime dependencies. The whole core is hand-rolled TypeScript — no CSV library, no framework, no cloud SDK.
• Deterministic and offline-first. Parsing, column mapping, normalization, confidence scoring, and crossing inference all run synchronously with no network. Geocoding is the only optional async step.
• Never invents data. Mapping decides what a column means; values are copied verbatim from the file. Nothing is hallucinated.
• Confidence, not blocking. Every row gets a 0–1 confidence score so low-quality rows can be softly flagged for review instead of silently dropped or wrongly placed.

This is the open-source engine extracted from FreightSight, a live Canada–US border-intelligence platform.

Install

npm install freightsight-ingest

Requires Node 18+ (for the built-in fetch used by the optional geocoder) or any modern browser.

Quick start

import { runIngest } from 'freightsight-ingest';

const csv = `
Load #,Ship From,Deliver To,Pickup Appt,Trailer,Wt (lbs)
L-88231,"Toronto, ON","Detroit, MI",7/3/2026 08:00,Reefer,42150
`;

const result = runIngest(csv);

result.mapping;      // [{ header: 'Ship From', field: 'origin', confidence: 0.85, via: 'keyword' }, ...]
result.loads;        // normalized CanonicalLoad[] — ISO dates, kg weights, canonical equipment
result.fingerprint;  // stable hash of the header set — use it as a template-cache key
result.stats;        // { rowsTotal, rowsLoaded, rowsFlagged, fieldsMapped, geocodePending }

Run the full example against a deliberately messy manifest (junk preamble rows, quirky headers, mixed date and weight formats):

npm install
npm run example

What it does

The pipeline runs in these stages:

1. Parse. Auto-detects the delimiter (, \t ; |), honors quoted fields spanning newlines, and finds the real header row — skipping preamble and junk.
2. Map columns. A synonym + regex ruleset maps arbitrary headers (PU City, ship from, Del Appt, Wt (lbs)) to canonical fields, most-specific-first, with collision resolution so no field is double-assigned.
3. Fingerprint. An order-independent hash of the header set gives you a template-cache key — remember a mapping once, apply it instantly to every future file from that sender.
4. Normalize. Dates → ISO 8601 (handles 7/3/2026, 03-Jul-2026, 2026-07-05 14:30, Jul 4 2026); weights → kg (defaults lbs when unit-less); equipment → canonical buckets (reefer, dry van, flatbed, …); country inferred from state/province code.
5. Infer crossing. Given origin + destination coordinates, snaps a cross-border lane to the most "on-the-way" commercial gateway and infers direction (US-bound / CA-bound). Confidence falls as the required detour grows, so a domestic lane scores low.
6. Score confidence. Per-row 0–1 score drives a soft review nudge rather than a hard failure.

For files without coordinates, geocode first, then re-infer:

import { runIngest, enrichWithGeocode } from 'freightsight-ingest';

const result = runIngest(csv);
const enriched = await enrichWithGeocode(result.loads, process.env.MAPBOX_TOKEN!);
// each load now has origin/dest lat-lng and a resolved crossing

The geocoder uses Mapbox and caches by address. Supply your own token — none is bundled.

API

Export	Description
runIngest(text, opts?)	The synchronous pipeline: parse → map → normalize → (crossing if coords present) → confidence. Returns an IngestResult.
parseDelimited(text)	Delimiter-detecting, quote-aware parser → { headers, rows, delimiter, skippedPreamble }.
mapColumns(headers)	Header → canonical-field mapping with confidence and collision resolution.
fingerprintHeaders(headers)	Order-independent template-cache key for a header set.
normalizeDate / normalizeWeight / normalizeEquipment / countryFromState	Standalone value normalizers.
inferCrossing(origin, dest, originCountry?, destCountry?)	Lane → nearest-on-route commercial gateway + direction + confidence.
GATEWAYS	The commercial border-crossing anchor list (public coordinates).
haversineKm(...)	Great-circle distance helper.
geocodeAddress(address, token) / enrichWithGeocode(loads, token)	Optional Mapbox geocoding + crossing re-inference.

Full types (CanonicalLoad, IngestResult, ColumnMapping, CrossingInference, …) are exported and documented inline in src/types.ts.

Scope — what's here and what isn't

This package is the generic ingest engine plus the border-crossing geometry (which is just public gateway coordinates and haversine math). It is intentionally self-contained and useful on its own.

It does not include, and is not required by, FreightSight's proprietary layer: the live CBSA/CBP border-wait binding, the template-learning cache that accumulates across customers, the inbound-email ingest path, and — obviously — any customer data. The engine markets the approach; the live data and the accumulated mapping library stay in the product. See ARCHITECTURE.md for the full design and where this fits.

Beyond freight

The freight canonical schema is one target. The parse → fingerprint → map → normalize → confidence spine is schema-agnostic — every B2B product eventually fights the "a customer sent us a weird CSV" problem. Swap the field set and synonym rules in src/mapping.ts and src/types.ts to retarget it.

Contributing

Issues and PRs welcome — especially new header synonyms, date/weight formats, and additional crossings. Keep the core dependency-free.

License

MIT © FreightSight. See LICENSE.