ARCHITECTURE
System Design
Social Inference Engine is a layered pipeline. Every observation passes through five stages: Ingestion → Normalisation → Retrieval → Classification → Action. Each stage is independently testable and has documented performance characteristics.
Data flow (top-level)
Platform API→BloomFilter→ReservoirSampler→NormalizationEngine→DataResidencyGuard→pgvector / BFS→LLMRouter→LLMAdjudicator→ConfidenceCalibrator→ActionRanker→SSE / REST
Ingestion Layer
Platform Connectors (×13)Celery workers; 15-min fetch schedule; BaseConnector ABC: authenticate(), fetch(), validate_credentials()
BloomFilterO(1) deduplication; 12–13 µs per URL; false-positive rate 1% (configurable); false-negative rate 0%
ReservoirSamplerVitter's Algorithm R; O(n); ~1,000 items/ms; statistically unbiased even at 500k-item streams
Normalisation Layer
NormalizationEngineUnified NormalizedObservation schema; handles JSON, HTML, plain text, and truncated UTF-8
DataResidencyGuardPII scrubbing (email, phone, URLs with identifying params); SHA-256 author pseudonymisation; immutable audit log
Retrieval Layer
pgvector / HNSW1,536-dim embeddings; approximate nearest-neighbour search; top-k few-shot context assembly
ContextMemoryStoreBFS graph traversal O(V+E); degree-4 ring topology in benchmark; complements ANN with structural similarity
Classification Layer
LLMRouterDeterministic two-tier routing: churn_risk / misinformation_risk / support_escalation → Tier 1 (frontier); remaining 7 types → Tier 2 (fine-tuned / local)
LLMAdjudicatorStructured output with evidence spans, signal_type, confidence float, and abstention reason code
ConfidenceCalibratorPer-type temperature scaling; online gradient-descent; 6–8 µs per update; state persisted to calibration_state.json
Action Layer
ActionRankerComposite score: Opportunity 35% + Urgency 30% + Risk 35%; 14–18 µs per signal; confidence gate at 0.5; CRITICAL / HIGH / MEDIUM / LOW tiers
SSE StreamGET /api/v1/signals/stream; push on classify; no polling; no third-party push service required
REST APIFastAPI; OpenAPI 3.1 spec; JWT auth; RBAC (VIEWER / ANALYST / MANAGER); 60 req/min rate limit
Technology stack
Language
Python 3.11
Web framework
FastAPI
Task queue
Celery + Redis
Database
PostgreSQL 15 + pgvector
Object storage
MinIO (S3-compatible)
Containerisation
Docker Compose
Auth
JWT (python-jose)
Migrations
Alembic
LLM abstraction
Custom LLMRouter
Testing
pytest + pytest-asyncio