The Data Engine
Technical Architecture
How HQ and OPOL work under the hood
Open Politics combines powerful data processing capabilities with intuitive user interfaces to democratise political intelligence. This page explains how our system is built and how the components work together.
This separation of concerns allows us to evolve each component independently whilst maintaining seamless integration between them.
We use Prefect for workflow orchestration:
Additional storage components:
Interactive 3D representation of global political events
Interface for defining and applying analytical frameworks
The following is a very technical section about our tools and methods. If you are interested in the app (HQ) and what you can do with it as a user, please click here
Overview
The platform consists of two primary components that operate together:HQ
Web application that visualises and provides analytical tools for the data
OPOL
Data engine that ingests, processes, and enriches political content
OPOL: The Data Engine
OPOL serves as the foundation of Open Politics, transforming diverse political data into structured, enriched information ready for analysis.OPOL Stack Architecture with Workflow Orchestration
Key Capabilities
Data Ingestion
Collection of data from diverse sources through configurable scrapers
Vector Processing
Conversion of text into numerical vectors for semantic analysis
Entity Recognition
Identification of people, organisations, locations, and other entities
Geocoding
Translation of location mentions into geographical coordinates
LLM Classification
Application of user-defined analytical frameworks using language models
Semantic Search
Finding content based on meaning rather than just keywords
Processing Pipeline
OPOL processes content through a series of interconnected microservices, each handling a specific aspect of the data enrichment pipeline:Content Ingestion
Scrapers collect articles, documents, and other content from configured sources
Custom sources can be added by creating Python scripts that return data in the format:
URL | Headline | Paragraphs | SourceVectorisation
Text content is converted into vector embeddings for semantic operations
Vectors enable similarity search, clustering, and other advanced operations
Entity Extraction
Named entities are identified and categorised using NLP techniques
Geocoding
Location entities are enriched with geographical coordinates
This enables geospatial visualisation and analysis of political content
Classification
User-defined analytical frameworks are applied using LLMs
Classifications can be customised for specific analytical needs
Technical Implementation
Microservice Architecture
Microservice Architecture
Each component of the data pipeline is implemented as an independent microservice:This modularity allows for independent scaling and development of each component.
Workflow Orchestration
Workflow Orchestration
- Batch processing of content through the pipeline
- Monitoring and observability
- Error handling and retries
- Scheduling and triggering
localhost:4200Storage Layer
Storage Layer
PostgreSQL
Primary storage for content and metadata
- Content storage (full text, headlines, etc.)
- Metadata (publication dates, sources)
- Entity relationships
- Classification results
PGVector
Vector storage for semantic operations
- Document embeddings
- Similarity search
- Nearest neighbour queries
- Semantic clustering
- Redis for job queues and caching
- File storage for document management
LLM Integration
LLM Integration
We provide flexible options for language model integration:
- OpenAI API for cloud-based processing
- Google Vertex AI for alternative cloud options
- Ollama for local LLM deployment
Set
LOCAL_LLM=True in your environment to use Ollama for local llm usageHQ: The User Interface
HQ is a modern web application that provides intuitive access to the processed data and analytical capabilities of OPOL.Key Components
Globe Visualisation
- Event clustering and filtering
- Location-based exploration
- Temporal playback
- Entity highlighting
Classification Runner
- Schema definition
- Document selection
- Model configuration
- Results visualisation
Search Interface
Unified search across multiple data types
- Semantic search
- Faceted filtering
- Relevance scoring
- Search history
Workspace System
Customisable environments for different analytical needs
- Layout customisation
- Dashboard creation
- Tool integration
- Result sharing
Technical Implementation
- Framework: NextJS-based web application
- Frontend: TypeScript with React components
- Styling: Tailwind CSS for responsive design
- Visualisation: Three.js for 3D globe and graph visualisations
- UI Components: ShadCN for accessible interface elements
- Authentication: Secure user authentication and authorisation
Deployment Options
Open Politics supports multiple deployment scenarios to accommodate different needs:Self-Hosted
Deploy the entire stack on your own infrastructure
- Complete data sovereignty
- Full customisation options
- Higher resource requirements
- Technical expertise needed
Hybrid
Run HQ locally with a hosted OPOL backend
- Reduced infrastructure requirements
- Data processing handled externally
- Simplified setup and maintenance
- Balance of control and convenience
Fully Hosted
Access through our public hosted instance
- No local infrastructure needed
- Immediate access to features
- Regular updates
- Community data sharing
Custom Deployment
Tailored deployment for specific organisational needs
- Optimised for particular use cases
- Integration with existing systems
- Specialised data sources
- Custom security configurations
System Requirements
For local deployments of the complete stack:RAM
32GB recommended
Storage
20GB minimum
Processor
4+ cores recommended
GPU is optional but can significantly improve performance for LLM operations and vector processing.