Search Engine

The search engine supports three strategies with automatic fallback and multi-level search.

Search Strategies

Semantic Search

Uses OpenAI embeddings and pgvector cosine similarity:

1. Embed the query using text-embedding-3-small
2. Query hts_embeddings with the <=> cosine distance operator
3. JOIN to tariffs for descriptions and rates
4. Return results sorted by similarity score

SELECT e.code, t.brief_description,
       1 - (e.embedding <=> query_embedding) AS score
FROM hts_embeddings e
JOIN tariffs t ON e.code = t.hts8
WHERE e.level = 'hts8' AND e.variant = 'full'
ORDER BY e.embedding <=> query_embedding
LIMIT 10

Lexical Search

Uses PostgreSQL ILIKE pattern matching on brief_description:

SELECT hts8, brief_description, mfn_text_rate
FROM tariffs
WHERE brief_description ILIKE '%keyword%'
ORDER BY LENGTH(brief_description)
LIMIT 10

Results are scored by inverse length: shorter (more specific) descriptions rank higher.

Hybrid Search (Default)

Combines both strategies with weighted scoring:

score = 0.7 × semantic_score + 0.3 × lexical_score

Steps:

1. Run semantic search (limit × 2 results)
2. Run lexical search (limit × 2 results)
3. Merge by code, combining scores with the 0.7/0.3 weighting
4. Sort by combined score, return top-K

Automatic Fallback

If embeddings are unavailable (not yet generated), hybrid and semantic strategies automatically fall back to lexical. The response metadata.strategy field reflects the actual strategy used.

Multi-Level Search

Search can target any hierarchy level:

Each level has two variants:

• full: Includes hierarchical context (section, chapter, heading) for better disambiguation
• short: Code and description only for direct similarity matching

Relationship Enrichment

When include_related=true, search results are enriched with:

• References: Codes mentioned in the tariff’s description
• Referenced-by count: How many other tariffs reference this code
• Has relationships: Whether graph edges exist for this code

This is done via batch queries to tariff_code_references for efficiency.

Performance

Embedding generation (one-time query overhead) adds ~100ms. Subsequent queries reuse the embedded query vector.