Data Pipeline Workflows

The HTS MCP data pipeline transforms raw tariff data into a searchable, enriched knowledge base. This guide walks through each step.

Pipeline Overview

Source Data → Bronze (Load) → Enrich → Embed → Graph → Classify

The full pipeline runs in 9 ordered steps. Each step is idempotent and can be re-run safely.

Step 1: Load Tariff Data

uv run hts load general_tariff_data.txt

Loads ~13,000 tariff records from a text/CSV file into the tariffs table. The loader:

• Auto-detects numeric columns (rates, ad valorem percentages)
• Parses date columns to YYYY-MM-DD format
• Validates HTS8 codes are exactly 8 digits
• Uses ON CONFLICT DO UPDATE for idempotent upserts

Step 2: Load Sections

uv run hts load-sections hts_sections.csv

Loads the hierarchical section structure and extracts 4-digit headings:

• Populates hts_sections with the raw hierarchical data
• Extracts hts_headings (indent=0 rows become 4-digit headings)
• Returns count of sections loaded and headings extracted

Step 3: Load Chapters

uv run hts load-chapters

Fetches chapter data from the USITC API (hts.usitc.gov):

• Downloads all 98 chapters with descriptions and section groupings
• Parses interpretive notes at chapter and section levels
• Stores chapters, sections (Roman numeral I–XXII), notes, and raw HTML
• Uses concurrent HTTP requests for speed

Step 4: Extract Code References

uv run hts extract-code-references

Scans all tariff descriptions for HTS code mentions:

• Regex pattern matches codes like 0201.30, 0201, 0201.30.80
• Validates against known tariff prefixes in the database
• Expands 5-digit codes to matching 6-digit codes
• Stores in tariff_code_references with source and target

Step 5: Load Policy Resources

uv run hts load-tariff-resources

Loads curated trade policy resources from a JSON file:

• Sources: Wikipedia, USTR, CBP, WTO, trade associations
• Resource types: reference, guidance, legal, educational, news, tool
• Full refresh: deletes existing and reloads all

Step 6: Enrich HTS6 Codes

uv run hts enrich-hts6

Generates AI-powered descriptions for 6-digit codes:

• For each HTS6 code, analyzes all child HTS8 descriptions
• GPT-5.4 Nano generates: enriched description (max 100 chars), 5-7 keywords, exclusionary terms, common attributes
• Concurrent processing with ThreadPoolExecutor
• Idempotent: skips already-enriched codes on re-run
• Stores in hts6_enrichments

Step 7: Generate Multi-Level Embeddings

uv run hts generate-multilevel-embeddings

Creates OpenAI embeddings at multiple hierarchy levels:

• Chapter: Section and chapter context
• HTS4: Section, chapter, and heading description
• HTS6: Full hierarchy context plus enriched description and keywords
• HTS8: Category, subheading, keywords, and specific description

Each level supports full and short text variants. Embeddings are stored in hts_embeddings with 1536 dimensions (text-embedding-3-small).

Step 8: Generate Graph Edges

uv run hts generate-graph-edges

Finds semantic neighbors using pgvector KNN:

• Uses CROSS JOIN LATERAL with <=> cosine distance operator
• Default: 30 nearest neighbors per code, minimum 0.65 similarity
• Stores candidates in hts_edge_candidates with similarity scores
• Typically generates 100,000+ candidate edges

Step 9: Classify Graph Edges

uv run hts classify-graph-edges

Classifies edge candidates using an LLM:

• Async pipeline with semaphore-controlled concurrency (200 concurrent)
• GPT-5.4 Nano classifies into 10 relationship types with confidence scores
• Processes in chunks with rate-limit pacing
• Resumable: picks up where it left off if interrupted
• Stores classified edges in hts_semantic_edges

Timing Expectations