Enrichment Pipeline

Documentation Index

Fetch the complete documentation index at: https://podium.build/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

What It Does

​

Why It Matters

• Rich product attributes — texture, effectiveness, sentiment, ingredient analysis, certifications, and more — extracted from real-world data
• Provenance on every signal — agents and users always know where a data point came from and how confident it is
• Cold-start intelligence — new platforms have useful product data from day one, without waiting for user interactions
• Multi-level baselines — understand a product’s attributes in context: how it compares to its category and the broader market
• Continuous discovery — Podium automatically discovers and enriches new products from multiple sources, keeping your catalog fresh

​

Schema Overview

​

EnrichmentSource

​

RawEnrichmentRecord

​

EnrichmentSignal

​

AttributeBaseline

​

ProductEmbedding

​

Multi-Level Baselines

​

Scoring Formula

score = average_confidence × log1p(signal_count)

​

Recomputation Modes

​

Attribute Taxonomy

​

Domain Taxonomy

​

Data Sources

​

Structured APIs

• Certifications from label tags (organic, cruelty-free, vegan) — confidence 0.95
• Categories from category tags — confidence 0.9
• Key ingredients from ingredient text via pattern matching — confidence 0.9
• Scent profile from fragrance-free/unscented patterns — confidence 0.85

1. Structured product APIs — retailer URL resolution via direct API lookups
2. Web search — discovering retailer URLs for products without direct API coverage
3. Affiliate platform resolution — resolving intermediate redirect URLs (e.g., affiliate tracking links) to real retailer pages

​

Web Intelligence

• Product descriptions → AI-powered attribute extraction
• Individual reviews → AI extraction with per-review confidence
• Product images → stored as image signals for catalog display

​

Community Sources

• Searches configured communities with product-name queries
• Fetches qualifying posts and comments
• Extracts attributes from real user discussions
• Provides a sentiment layer that complements structured data

​

Product Discovery

1. Matched against your existing catalog via entity resolution
2. Added as new ProductCatalogItem records if they’re novel
3. Enriched with the full attribute extraction pipeline

​

Creator Storefronts

• Resolved product IDs linked to your catalog
• Creator metadata (handle, display name, platform, avatar)
• Product count and resolution status

​

Wellness Vertical

​

Attribute Extraction

​

Rule-Based Path

​

AI-Powered Path

• Target: 12 attribute types with expected value formats
• Input: product name + text content + text type
• Output: JSON array of { attributeType, attributeValue, confidence, evidence }
• Hard confidence floor: signals below 0.6 are discarded
• Batch processing available for multiple texts per raw record

​

Entity Resolution

​

Tier 1: Fuzzy Text Match

• Compares external names against all product names using trigram similarity
• Similarity > 0.75 → immediate match
• Similarity 0.4–0.75 → candidate for Tier 2
• Cost: zero external API calls

​

Tier 2: Semantic Embedding Match

• Triggered only when Tier 1 doesn’t find a strong match
• Cosine similarity search against pre-computed product embeddings
• Threshold: 0.82 cosine similarity for acceptance

​

Normalization

1. Type canonicalization: Maps variant names to canonical types (e.g., fragrance → scent, skin_type → skin_type_suitability, shade → color)
2. Value canonicalization: Maps common aliases to standard values (e.g., light weight → lightweight, cruelty free → cruelty-free)
3. Confidence clamping: Forces all values to [0, 1]
4. Deduplication: Removes exact type:value duplicates within a batch

​

Consuming Enrichment Data

import { createPodiumClient } from '@podium-sdk/node-sdk';
const client = createPodiumClient({ apiKey: process.env.PODIUM_API_KEY });

const feed = await client.agentic.listProductsFeed({
  limit: 20,
  categories: 'skincare',
});

for (const product of feed.products) {
  console.log(product.name, product.intentScore);
  if (product.attributes?.baselines) {
    for (const baseline of product.attributes.baselines) {
      console.log(`  ${baseline.attributeType}: ${baseline.topValues[0]?.value} (${baseline.scope})`);
    }
  }
}

const recs = await client.companion.listRecommendations({
  userId: 'user_123',
  count: 5,
  category: 'skincare',
});

​

Async Pipeline Orchestration

​

Admin Endpoints

curl -X POST https://api.podium.build/api/v1/admin/enrichment/run \
  -H "Authorization: Bearer $PODIUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "sourceId": "clsource_sephora" }'

curl https://api.podium.build/api/v1/admin/enrichment/status \
  -H "Authorization: Bearer $PODIUM_API_KEY"

​

Building Your Own Pipeline

1. Define your attribute taxonomy — what product attributes matter in your vertical?
2. Register data sources — structured APIs for your domain, review platforms, community forums
3. Configure extraction — rule-based for structured sources, AI-powered for unstructured text
4. Generate product embeddings — pre-compute embeddings for your catalog to enable entity resolution
5. Set baseline thresholds — minimum signal count, confidence floor, and TTL for your data freshness requirements

CREATE TABLE enrichment_source (
  id         TEXT PRIMARY KEY,
  name       TEXT NOT NULL,
  source_type TEXT NOT NULL,  -- STRUCTURED_API, WEB_PAGE, COMMUNITY, DISCOVERY
  config     JSONB,
  last_run_at TIMESTAMPTZ
);

CREATE TABLE enrichment_signal (
  id              TEXT PRIMARY KEY,
  source_id       TEXT REFERENCES enrichment_source(id),
  product_id      TEXT,  -- nullable for unresolved signals
  attribute_type  TEXT NOT NULL,
  attribute_value TEXT NOT NULL,
  confidence      REAL NOT NULL CHECK (confidence BETWEEN 0 AND 1),
  signal_source   TEXT DEFAULT 'MARKET_DERIVED',
  metadata        JSONB,
  UNIQUE (source_id, attribute_type, attribute_value)
);

CREATE TABLE attribute_baseline (
  id             TEXT PRIMARY KEY,
  product_id     TEXT,  -- nullable for category/market baselines
  scope          TEXT NOT NULL,  -- PRODUCT, CATEGORY, MARKET
  attribute_type TEXT NOT NULL,
  top_values     JSONB NOT NULL,
  signal_count   INTEGER NOT NULL,
  computed_at    TIMESTAMPTZ DEFAULT NOW(),
  valid_until    TIMESTAMPTZ NOT NULL,
  UNIQUE (product_id, scope, attribute_type)
);