Semantic Graph

The knowledge graph connects tariff codes with classified semantic relationships. It’s built in two phases: edge generation (KNN) and edge classification (LLM).

Phase 1: Edge Generation

KNN with pgvector

Uses CROSS JOIN LATERAL for efficient batch KNN:


SELECT sb.code AS source_code, nn.code AS target_code,
       1 - (sb.embedding <=> nn.embedding) AS similarity,
       ROW_NUMBER() OVER (
         PARTITION BY sb.code
         ORDER BY sb.embedding <=> nn.embedding
       ) AS rank
FROM source_batch sb
CROSS JOIN LATERAL (
    SELECT e.code
    FROM hts_embeddings e
    WHERE e.code != sb.code
    ORDER BY e.embedding <=> sb.embedding
    LIMIT :k
) nn
WHERE 1 - (sb.embedding <=> nn.embedding) >= :min_similarity

Configuration

Parameter	Default	Description
`k`	30	Nearest neighbors per code
`min_similarity`	0.65	Cosine similarity threshold
`level`	hts8	Embedding level
`variant`	full	Embedding variant

Output

Candidates are stored in hts_edge_candidates with:

Source/target codes
Similarity score
Source/target chapters and headings (for filtering)
Run ID and timestamp

Typically generates 100,000–150,000 candidates for ~13,000 HTS8 codes.

Phase 2: Edge Classification

Async LLM Pipeline

The classifier processes candidates through an async pipeline:


Fetch unclassified → Chunk → Semaphore-controlled async LLM calls → Validate → Batch flush

Fetch: LEFT JOIN to hts_semantic_edges finds candidates not yet classified (resume support)
Chunk: Candidates are processed in chunks of 1,000 with configurable delay between chunks
LLM call: Each candidate pair is sent to GPT-5.4 Nano with a structured JSON schema
Validate: Response is parsed and validated (type must be in allowed set, confidence 0-1)
Flush: Classified edges are batch-inserted to hts_semantic_edges

Concurrency Control


semaphore = asyncio.Semaphore(concurrency)  # default: 200
 
async with semaphore:
    response = await openai.chat.completions.create(...)

Between chunks, a configurable delay (default: 1s) provides rate-limit pacing.

Resume Behavior

If the pipeline is interrupted:

Completed classifications are already flushed to the database
Re-running picks up from where it stopped
The LEFT JOIN ensures only unclassified candidates are fetched

Relationship Types

The LLM classifies each edge into one of 10 types:

Type	Definition	Example
`material_affinity`	Same material, different forms	Raw steel → Steel bars
`functional_similarity`	Same purpose, different materials	Plastic containers → Glass containers
`manufacturing_process`	Similar production methods	Hot-rolled steel → Cold-rolled steel
`end_use`	Complementary products	Screws → Screw drivers
`substitution`	Borderline classification	Fresh beef → Chilled beef
`component_assembly`	Part-to-whole	Engine parts → Complete engines
`cross_category_bridge`	Unexpected cross-chapter link	Textile machinery → Textiles
`abstraction`	Hierarchical parent-child	Heading → Subheading
`weak_association`	Tangentially related	Low-value edges
`no_meaningful_relationship`	False positive from KNN	Filtered out

Classification Output

Each classified edge includes rich metadata:

Field	Purpose
`confidence`	How certain the LLM is (0-1)
`reasoning`	2 sentences explaining the relationship
`key_differentiator`	What distinguishes the two codes (max 8 words)
`classification_clarity`	How clear-cut the classification was (0-1)
`inclusionary/exclusionary terms`	Terms that define/exclude each code
`relationship_haiku`	5-7-5 syllable creative summary
`bidirectional`	Whether the relationship is symmetric

Graph Statistics

After a full run, typical distribution:

Category	Percentage	Description
Meaningful relationships	~40%	Material, functional, manufacturing, end use, substitution, component
Cross-category bridges	~2%	High-value cross-chapter connections
Weak/none	~55%	Filtered out in most queries
Errors	Less than 0.1%	Parse failures, timeouts