Enrichment Pipeline

The enrichment pipeline uses AI to generate structured descriptions for HTS6 codes and multi-level embeddings for search.

HTS6 AI Enrichment

Why HTS6?

The 6-digit level is the international standard (Harmonized System). Original HTS descriptions at this level are often terse or hierarchical. AI enrichment generates:

• Enriched description: Concise, standalone summary (max 100 chars)
• Keywords: 5-7 product-specific search terms
• Exclusionary terms: Terms that should NOT match this code
• Common attributes: Properties shared by all child HTS8 codes

Process

For each HTS6 code:

1. Collect all child HTS8 descriptions
2. Send to GPT-5.4 Nano with a structured prompt
3. Parse the JSON response
4. Store in hts6_enrichments

Concurrency

Uses ThreadPoolExecutor with configurable workers (default: 25):

with ThreadPoolExecutor(max_workers=max_workers) as executor:
    futures = {executor.submit(enrich_one, code): code for code in batch}
    for future in as_completed(futures):
        result = future.result()

Idempotency

The enrichment checks for existing records before processing. Re-running skips already-enriched codes, making the pipeline safe to restart.

Text Builder

The text builder constructs embedding text for each level/variant combination. The text format is critical for search quality: it determines what the embedding captures.

Text Formats

Chapter (full):

section {roman}: {section_title}; chapter {num}: {chapter_title}

HTS4 (full):

section {roman}: {section_title}; chapter {num}: {chapter_title}; category: {heading_description}

HTS6 (full):

section {roman}: {section_title}; chapter {num}: {chapter_title};
category: {hts4_description}; hts 6-digit: {code} - {enriched_description};
keywords: {keyword1}, {keyword2}, ...

HTS6 (short):

hts 6-digit: {code} - {enriched_description}; keywords: {keyword1}, {keyword2}, ...

HTS8 (full):

category: {hts4_description}; hts 6-digit: {hts6_code};
keywords: {keywords}; hts 8-digit: {code} - {brief_description}

HTS8 (short):

hts 8-digit: {code} - {brief_description}

Design Decisions

• Full variants include hierarchical context for disambiguation. A query about “steel pipes” matches better when the embedding knows the section (Base Metals) and chapter (Iron and Steel).
• Short variants are pure description-to-description similarity. Useful when you want direct product comparison without hierarchical bias.
• Keywords in HTS6 boost searchability by adding terms the original description may lack.

Embedding Generation

Process

1. Text builder generates texts for all requested level/variant combinations
2. Texts are batched (default: 500 per API call) and sent to OpenAI
3. Embeddings (1536D vectors) are upserted to hts_embeddings

Configuration

Volume

Storage

Embeddings are stored in the hts_embeddings table:

CREATE TABLE hts_embeddings (
    code TEXT,
    level TEXT,          -- chapter, hts4, hts6, hts8
    variant TEXT,        -- full, short
    embedding vector(1536),
    embedding_text TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (code, level, variant)
);

An IVFFlat or HNSW index on the embedding column enables fast KNN queries.