# App Setup
Source: https://docs.open-politics.org/pages/app/app-setup
Configure providers and foundation services
HQ uses foundation services to power different capabilities. Configure them in the **Provider Configuration** panel on the [home screen](https://open-politics.org/hq).
## Foundation Services
Chat, classification, structured extraction. The core AI capability.
Vector search, similarity matching, semantic discovery within infospaces.
Live web results for chat - ground responses in current information.
Extract and map locations from your documents. Powers geographic visualizations.
Each service type supports multiple providers. Pick what fits your needs — cloud APIs, local inference, or a mix.
***
## API Key Storage
Two storage modes for each provider:
| Mode | What it does | Use when |
| ------------------ | -------------------------- | ------------------------------------------------ |
| **Runtime Only** | Keys stay in your browser | Quick testing, interactive use |
| **Save for Tasks** | Keys encrypted in database | Background jobs, large analysis runs, automation |
**Background tasks require saved keys.** The Celery worker that processes large analysis runs can't access your browser. If jobs hang or fail, check that you've saved keys to the backend.
***
## Providers
### Language Models
| Provider | Get Key |
| --------- | ------------------------------------------------------ |
| Anthropic | [console.anthropic.com](https://console.anthropic.com) |
| OpenAI | [platform.openai.com](https://platform.openai.com) |
| Google | [aistudio.google.com](https://aistudio.google.com) |
| Ollama | Local — no key needed |
### Ollama
Ollama is a local language model server that has many models available and is a great way to get started with local inference.
### Embeddings
Used for vectorizing content in your infospaces. Configure separately from language models.
| Provider | Notes |
| -------- | ---------------------------------- |
| OpenAI | Reliable, widely used |
| Ollama | Local embeddings, complete privacy |
### Web Search
Enables the chat to pull in live search results.
| Provider | Get Key |
| -------- | ----------------------------------- |
| Tavily | [tavily.com](https://tavily.com) |
| SearXNG | (soon back integrated in the stack) |
### Geocoding
Location extraction uses Nominatim. Self-hosted deployments include it by default. No API key needed for local Nominatim.
***
## Verifying Setup
1. Open **Chat** and send a message — confirms language model works
2. Upload an asset and check if it is ingested properly
3. Ask chat to search the web — confirms web search
4. Run an analysis with location fields — confirms geocoding
***
## Next Steps
Create workspaces for your projects
Define what to extract from documents
# Assets & Bundles
Source: https://docs.open-politics.org/pages/app/assets
Flexible data management & composition
Assets are composable - they have parent-child relationships. A CSV is one file, but it's also individual rows you can analyse separately. A PDF is one document, but it's also pages. Most data you'd want to work with - mbox archives, article feeds, survey exports - can be broken down this way.
**For example:**
```mermaid theme={null}
graph TD
PDF["Report.pdf"] --> P1["Page 1"]
PDF --> P2["Page 2"]
PDF --> P3["Page 3"]
```
```mermaid theme={null}
graph TD
CSV["Survey.csv"] --> R1["Row 1"]
CSV --> R2["Row 2"]
CSV --> R3["Row 3"]
```
If an adapter for your format doesn't exist yet, it's straightforward to build one.
## Formats & Uploading
The main asset is the PDF with metadata and text content. The pages are sub-assets.
CSV Parent + Rows as sub-assets
Complex content, like articles with images and links. Soon with our own compositor for writing.
Scraped websources with metadata as Articles
Simplest Asset
Images in PNG & JPG.
## Streaming Sources
Many sources stream data continuously. Set up ongoing discovery and ingestion from:
Tavily, SearXNG, and other search APIs
RSS & RSS-style XML feeds
Scrape all items from a page (e.g. cnn.com/economy)
Ingest URLs, detect changes on refresh
## Bundles
Bundles organise your assets. Think folders, but an asset can live in multiple bundles without duplicating.
"Climate Policy Research", "Election Coverage 2024", "Grant Applications Q1"
"Government Documents", "News Articles", "Academic Papers"
Bundles can nest. Assets can belong to several at once. When you run analysis, you pick which bundles to include.
# Chat
Source: https://docs.open-politics.org/pages/app/chat
Work with your data through conversational AI
Chat brings all HQ tools into one interface. Instead of clicking through menus, you talk to your data directly.
## What You Can Do
Query your uploaded content directly. The system pulls relevant context from your infospace's vector store.
Describe what you want to extract in conversation. Get a working schema back.
"Apply my bias schema to the Reuters bundle" - natural language commands for annotation runs.
Drill down, filter, compare across runs. Ask follow-up questions about your findings.
# Dashboards
Source: https://docs.open-politics.org/pages/app/dashboards
Visualise analysis results through charts, maps, and tables
Dashboards turn annotation results into visual insights. After running a schema on your assets, explore the data through charts, maps, and tables.
## Available Panels
Filter, sort, and export individual results. Good for reviewing quality and finding specific items.
Track how values change over time. Line charts, bar charts, area charts.
Plot locations mentioned in documents. Requires geocoding enabled.
See proportions across categories. Pie charts, histograms, bar comparisons.
## Default Dashboard
Every annotation run gets a default dashboard with:
* Data table showing all results
## Adding Panels
Click **Add Panel** to include additional visualizations. Each panel can be:
* Filtered to show specific subsets
* Configured for colors, labels, scales
* Exported as image or data
## Export Options
* **CSV/JSON** - The table can export as CSV, the whole run can be exported as a zip of JSON & media files.
* **Share Link** - Share a public static link to the dashboard. Logged in users can import the analysis.
**In Development:** Knowledge graphs, cross-panel filtering, and interactive drill-down are planned features.
# Infospaces
Source: https://docs.open-politics.org/pages/app/infospaces
Organise projects into isolated workspaces with built-in vector search
## What Are Infospaces?
Each infospace is a separate information environment - your assets, schemas, and analysis results stay isolated. Think of them as separate research projects, departments, or clients.
## Managing Infospaces
### Creating and Switching
**Default Infospace:**
You start with "Default Infospace" containing example assets and schemas to help you get started.
**Switching Between Infospaces:**
Use the Infospace Switcher in the top left sidebar to see all your infospaces. Click any infospace to switch to it.
**Creating New Infospaces:**
* Click "Create Infospace" to add a new workspace
* Choose a descriptive name for your project
* Set up initial bundles and schemas
**Managing Existing Infospaces:**
* Select "Manage Infospaces" to rename or delete
* Archive old projects to keep your workspace clean
* Export infospace data for backup or sharing
* Import infospace configurations from other users
## Example Infospace Organisation
* "Election Coverage 2024"
* "Grant Applications"
* "Academic Papers"
* "Immigration Policy Analysis"
* "Media Bias Study"
* "Legislative Tracking"
* "Taxes"
* "Personal Mail"
* "Old documents"
* "Marketing Research"
* "Market signals"
* "Current Projects"
* "Publications"
## Vector Storage & Search
Every infospace is also a [vector store](https://en.wikipedia.org/wiki/Vector_database). Your documents get embedded - enabling semantic search, similarity matching and better tooling for your LLM model to work with.
When you enable vectorisation in your infospace settings:
* Background task creates vectors for all content
* Continuous processing of new uploads
* Automatic updates when content changes
* Find documents with similar content or themes
* Deduplicate documents
* Provide relevant context for AI analysis
* Enhance chat interface with document knowledge
### Vectorization Process
## Data Storage & Backups
### Storage Locations
* Data stored locally in `./store/minio`
* Full control over data location and access
* No external dependencies for storage
* Suitable for sensitive or confidential data
* Data stored in Hetzner Object Storage
* Managed infrastructure and maintenance
* Automatic scaling and redundancy
* Professional backup and recovery services
### Backup Management
**Creating Backups:**
1. **Navigate to Backup Settings** in your infospace
2. **Choose Backup Scope**: Full infospace or specific data
3. **Select Backup Destination**: Local storage or cloud
4. **Schedule Regular Backups**: Daily, weekly, or monthly
5. **Monitor Backup Status**: Ensure successful completion
**Backup Types:**
* **Full Backups**: Complete infospace data and configuration
* **Incremental Backups**: Only changed data since last backup
* **Schema Backups**: Analysis frameworks and configurations
* **Asset Backups**: Document content and metadata
**Restoring from Backups:**
1. **Select Backup to Restore** from available backups
2. **Choose Restore Scope**: Full restore or selective data
3. **Verify Data Integrity**: Check restored content
4. **Update Permissions**: Ensure proper access controls
5. **Test Functionality**: Verify all features work correctly
# Installation & Architecture
Source: https://docs.open-politics.org/pages/app/installation-self-hosted
Self-hosting guide and platform architecture
## How to set up your HQ
The simplest way to use Open Politics is through our hosted platform:
1. Register at [open-politics.org](https://open-politics.org/accounts/register)
2. Log in at [open-politics.org/](https://open-politics.org/accounts/login)
3. Begin using the platform with pre-loaded data and continue to:
Configure your API keys and providers
Our preferred way to self-host on a single machine or develop with is Docker Compose. For Multi-machine setups, we recommend Kubernetes.
But in both the cases you can choose to outsource services to managed providers for a more lightweight (but less private) setup.
| Service | Purpose | Example Provider | |
| ---------------- | --------------- | ---------------- | --------------------------------- |
| Postgres | Main database | Hetzner | |
| Redis | Cache and queue | Upstash | |
| MinIO / S3 | Object storage | Hetzner | |
| Nominatim | Geocoding | Nominatim | - uses lot of disk space (170gb+) |
| LLM API | Language model | OpenAI | - high gpu and/or cpu usage |
| Tavily / SerpAPI | Web search | Tavily | |
Run the complete platform locally:
* Docker and Docker Compose
* 32GB RAM recommended
* 20GB free disk space
* Git
```bash theme={null}
git clone https://github.com/open-politics/open-politics-hq.git
cd open-politics-hq
```
```bash theme={null}
cp .env.example .env
# Edit .env with your configuration
# All values with changeThis need to be changed before the app will start.
# for example with 'openssl rand -base64 14' in your terminal.
```
```bash theme={null}
docker compose up
```
Open your browser and navigate to:
```
http://localhost:3000
```
Initial setup may take time depending on your system configuration.
If you are here, we assume you know what you are doing.
## Base Helm Chart
We have a base helm chart available [here](https://github.com/open-politics/open-politics-hq/tree/main/.deployments)
```bash theme={null}
git clone https://github.com/open-politics/open-politics-hq
cd .deployments/end-to-end-hetzner-k3s-terraform-helm/open-politics-hq-deployment/hq-cluster-chart
cp values.example.yaml values.yaml
# Edit values.yaml with your configuration
```
## Hetzner Kubernetes Stack
Our easiest way to deploy HQ via Kubernetes is to use Hetzner. We provide an end-to-end script that provisions a master and worker nodes via Terraform and installs K3s. We suggest using K9s alongside our scripts to manage and scale the cluster.
# HQ on Hetzner (K3s) — End-to-End Deployment Guide
This repository delivers a batteries-included, reproducible way to provision a Hetzner K3s cluster and deploy your application stack with one command.
## What This Gives You
* **3 Hetzner Cloud servers by default**: 1 control plane, 2 workers (configurable)
* **K3s** with sensible flags for Hetzner CCM integration
* **Traefik** Ingress with automatic TLS via Let's Encrypt (ACME)
* **Persistent storage** using Hetzner CSI
* **Application stack via Helm**: frontend, backend API, Celery workers, Redis, optional Postgres/MinIO templates
***
## Quick Start
```bash theme={null}
cd .deployments/end-to-end-hetzner-k3s-terraform-helm/open-politics-hq-deployment/
cp .tfvars.example .tfvars
cp hq-cluster-chart/values.example.yaml hq-cluster-chart/values.yaml
# Edit .tfvars and values.yaml with your configuration (see below)
```
```bash theme={null}
./deploy.sh
```
The script is idempotent. Re-running applies updates safely. Typical first deploy time: \~10–15 minutes.
## Configuration Checklist
Before running `./deploy.sh`, ensure you've completed:
* [ ] **Hetzner API Token**: Set `hcloud_token` in `.tfvars`
* [ ] **Domain**: Set `domain` in `.tfvars` (must match `env.config.DOMAIN` in `values.yaml`)
* [ ] **Application Values**: Copy and customize `hq-cluster-chart/values.yaml`
* [ ] **Secrets**: Replace all `xxx` placeholders with real values
* [ ] **SSH Keys**: Run `./scripts/manage-ssh-keys.sh generate`
* [ ] **DNS Ready**: Have your domain ready for A record creation
***
## Prerequisites
* Hetzner Cloud account and API token (Read/Write)
* Local tools: `terraform`, `kubectl`, `helm`, `jq`
* An SSH key uploaded to Hetzner; the scripts can rotate/manage keys automatically
***
## Initial Configuration
```bash theme={null}
cp .tfvars.example .tfvars
```
**Required:**
* `hcloud_token`: Hetzner API token ([get from console](https://console.hetzner.com/projects))
* `domain`: your public domain for TLS/ingress
**Optional (with defaults):**
* `worker_count`: number of workers (default: 2)
* `master_server_type` / `worker_server_type`: server types (default: `cx22`)
* `location`: Hetzner location (default: `fsn1`)
* `cluster_slug`: resource naming prefix (default: `hq`)
```bash theme={null}
cp hq-cluster-chart/values.example.yaml hq-cluster-chart/values.yaml
```
**Required customizations:**
* `email`: admin email for Let's Encrypt
* `env.config.DOMAIN`: your domain (must match `.tfvars`)
* `env.config.BACKEND_CORS_ORIGINS`: update with your domain
* Docker image repositories for backend, frontend, and celery-worker
**Secrets (replace all `xxx` placeholders):**
* `SECRET_KEY`, `FIRST_SUPERUSER_PASSWORD`, `POSTGRES_PASSWORD`
* `MINIO_ROOT_PASSWORD`, `MINIO_SECRET_KEY`
* `SMTP_USER`, `SMTP_PASSWORD`, `EMAILS_FROM_EMAIL`
* API keys: `OPENAI_API_KEY`, `TAVILY_API_KEY`, `GOOGLE_API_KEY`, `MAPBOX_ACCESS_TOKEN`
```bash theme={null}
./scripts/manage-ssh-keys.sh generate
```
This generates key pairs, uploads to Hetzner, and updates cloud-init templates.
**Manual alternative:** Upload your SSH key to Hetzner Console and set `ssh_key_name` in `.tfvars`.
Have your domain ready. After deployment, create A records pointing to the Load Balancer IP for both `your-domain.com` and `www.your-domain.com`.
***
## How It Works
The `./deploy.sh` orchestrator performs two phases:
1. **Provision infrastructure**: Terraform creates network, firewall, servers; cloud-init installs K3s
2. **Deploy stack**: Configures Traefik, installs Hetzner CCM/CSI, deploys the Helm chart
**Core commands:**
| Command | Description |
| ---------------------- | ---------------------------------------- |
| `./deploy.sh` | Full deploy (infra → app) |
| `./scripts/connect.sh` | Fetch kubeconfig and set kubectl context |
| `./scripts/status.sh` | Summary of nodes, services, ingress, PVs |
| `./scripts/logs.sh` | Tail app logs |
| `./scripts/monitor.sh` | Open k9s |
| `./scripts/destroy.sh` | Tear everything down |
***
## DNS and TLS
```bash theme={null}
kubectl get svc traefik -n kube-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```
Point your domain (and `www`) to the Load Balancer IP. Traefik will obtain Let's Encrypt certificates automatically (typically 2-5 minutes after DNS propagation).
```bash theme={null}
./scripts/test-deployment.sh # Test DNS, HTTP/HTTPS, and TLS
./scripts/status.sh # View pods, services, ingress
./scripts/check-ssl.sh # Check ACME progress
```
***
## Troubleshooting
| Issue | Solution |
| --------------------------- | ----------------------------------------------------------------- |
| Workers don't schedule pods | CCM removes taints automatically; wait or remove manually |
| No Load Balancer IP | Confirm Traefik is `LoadBalancer` and CCM is Ready |
| Domain mismatch | Ensure `.tfvars` domain matches `values.yaml` |
| CORS errors | Verify `BACKEND_CORS_ORIGINS` includes your domain |
| ACME stuck | Verify DNS A records point to LB IP; run `./scripts/check-ssl.sh` |
| SSH failures | Run `./scripts/manage-ssh-keys.sh generate` |
## Cost
Default: 3× `cx22` + 1 Load Balancer ≈ €40–45/month. Reduce costs by lowering `worker_count` or using smaller server types.
# Overview
Source: https://docs.open-politics.org/pages/app/overview
How HQ is organised and where to find what
We aim to provide few but powerful core concepts to allow for flexible data ingestion, analysis and management.
Foundation services like AI Models, a Geocoder or a Websearch tool can extend the functionality and allow for hybrid setups. From cloud to fully local on your own hardware.
Before you start: make sure your providers for the foundation services are properly installed and configured. See [App Setup](/pages/app/app-setup) for more information.
## Infospaces
Think of an infospace as a project workspace. Each one keeps your data separate from other projects and has its own database and vector index to help you find relevant content.
```mermaid theme={null}
flowchart TB
I[Infospace]
I --> A[Assets]
I --> S[Schemas]
```
Your data. PDFs, CSVs, articles, web feeds - anything you want to analyse. Composable: a document breaks into pages, a spreadsheet into rows. Bundles group assets for organisation and batch analysis.
Your lens and instructions for the AI. Natural language tasks with strict output definitions. Get structured, comparable data across all your documents.
***
## Analysis
This is where it all comes together. With the building blocks above you are set to work through your data by configuring an analysis.
```mermaid theme={null}
flowchart TB
A[Assets] --> AN[Annotations]
S[Schemas] --> AN
AN --> D[Dashboards]
```
What questions do you want to answer? What data do you have?
Also set up parametrs like parallelisation and if you want justifications for fields.
[More information on creating an analysis](/pages/concepts/running-analysis)
Your results. Tables, charts, maps. Export or share.
Every run produces structured data. Or as we call them: annotations. Filter, visualise and drill down into your findings.
[More information on dashboards](/pages/app/dashboards)
## Chat & Tools
Your data management interface. Query, build schemas, run analysis, explore, find similar items. Deduplicate and synthesise - through dialogue.
\| [More information on chat](/pages/app/chat) | [Advanced Chatting](/pages/concepts/advanced-chatting) |
```mermaid theme={null}
flowchart TB
C[Chat] --> A[Assets]
C[Chat] --> S[Schemas]
C[Chat] --> D[Dashboards]
C[Chat] --> W[Websearch]
C[Chat] --> V[Vector search]
```
## Want to know what do with the results?
# Schemas
Source: https://docs.open-politics.org/pages/app/schemas
Tell the AI what to extract from your documents
A schema is a set of instructions for extracting structured data from documents. You describe what you want in plain English, the AI applies it consistently across hundreds or thousands of items.
Schemas bridge natural language and structured data. You express your framework in plain English, and get back consistent, well-defined results ready for analysis, data management and export.
## What You Get
Each speech gets analysed for the same fields: position summary, security stance (1-10), location mentions, talking point topics, migration attitude, governance focus, rhetoric style. The schema defines what to extract - the AI does the extraction.
With structured output, you can filter, compare, and visualise. Which speakers emphasise humanitarian concerns vs. security? How does rhetoric differ by country? That's the point.
## Building a Schema
Start with your actual questions. What do you want to know about your documents?
* What positions are being taken?
* How is the issue framed?
* Who's mentioned, and in what context?
* What's the emotional temperature?
Then turn each question into an extraction field.
### Good Instructions vs. Vague Instructions
The difference between useful output and noise is specificity.
**Type:** Number (1-10)\
**Extract:** Position on border security. 1 = open borders, 10 = strict enforcement
**Type:** List\
**Extract:** Main topics discussed - migration policy, demographic change, EU solidarity, public services, etc.
*Clear scales, concrete categories, examples of valid values.*
**Type:** Text\
**Extract:** What is their position on the issue?
*Too vague. Which issue? What aspects of their position? The AI will guess, and guesses vary.*
### Field Types
| Type | Use for | Example |
| ---------- | --------------------------------------------- | ---------------------------------------- |
| **String** | Freeform text, summaries, answers, categories | `summary: "Speaker argues for..."` |
| **Number** | Ratings, scales, counts, amounts | `security_stance: 7` |
| **List** | Multiple values, tags, topics | `topics: ["migration", "EU solidarity"]` |
**Binary fields for filtering:** It's often useful to include 0/1 or yes/no fields for filtering results. Something like `is_spam: 0` or `is_relevant: 1` lets you quickly filter out noise in dashboards.
### Special Field Names
Certain field names unlock specific features when you view results:
| Field name | What it enables | How to prompt |
| ----------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| `location` | Geographic maps - locations get geocoded and plotted | Ask for single most relevant location or region relevant |
| `timestamp` | Time series charts - combine with number fields to track changes over time | Ask for a ISO timestamp of the article's main event or publication date |
| `summary` | Highlighted in result views as the main description | Ask for a summary with your preferred style and length |
| `tags` | Counted and aggregated across results | Either freeform or just from a few given tags |
It is always good to provide a fallback choice like "None" or "Unapplicable".
Other list fields also get counted automatically. If you extract `talking_point_topics` as a list, you'll see frequency distributions in dashboards.
Numbers with defined scales (1-10, 1-5) work well for comparative analysis - they enable meaningful aggregation and time series when combined with timestamps.
***
## Tips
**Include examples in your instructions.** "Source type: government, activist, expert, journalist, or anonymous" gives the AI a bounded set to work with.
**Use scales for subjective measures.** "Emotional intensity (1-10)" is more useful than "how emotional is it?" because you can aggregate and compare.
**Start simple, then add.** Begin with basic extraction (who, what, where), confirm it works, then layer in analysis fields (sentiment, framing, stance).
**Look at failures.** When extraction is wrong or inconsistent, the instructions usually need tightening. What did the AI misunderstand?
***
## Sharing Schemas
Schemas can be uploaded to the library for others to use. They can see your methodology, critique it, replicate it, build on it. Transparency about how analysis is done matters - it's what separates systematic research from vibes.
***
## Related pages
Apply schemas to your documents
Promote extractions to persistent metadata
Upload and organise documents for analysis
Visualise results with charts and tables
# Curating Annotations
Source: https://docs.open-politics.org/pages/concepts/curating-fragments
Turn analysis results into persistent document metadata
Annotations are scoped to a run. \
Curating them makes them permanent qualities of our data.
Curated results become **persistent fragments** - metadata pinned to the asset. Subsequent runs see them as context.
* Temporary analysis results
* Tied to a specific schema run
* Can be deleted or re-run
* Not visible to subsequent analysis
* Permanent asset metadata
* Survives schema changes
* Included in future analysis context
* Builds cumulative knowledge
## How to
From the results table of a run dashboard:
1. **Review the extraction** - Check it's accurate and useful, set filters
2. **Click "Curate"** - Converts annotations to persistent fragments
3. **Confirm** - Fragment is now attached to the asset (see above)
***
## Related
Apply schemas to your documents
Define what to extract from documents
Upload and organise documents
Visualise results with charts and tables
# Running Analysis
Source: https://docs.open-politics.org/pages/concepts/running-analysis
Apply schemas to your documents and extract structured data
Once you have a schema and some assets, you can run analysis. Each document gets processed by your chosen model, and results are stored as structured annotations.
## The Process
Pick individual assets or entire bundles
Select which schema to apply
OpenAI, Anthropic, Google, or local via Ollama
Each document gets processed, results stored as structured JSON
## Choosing a Model
Different models have different strengths:
| Provider | Best for | Notes |
| ------------- | ----------------------------------- | ---------------------------- |
| **OpenAI** | General extraction, high throughput | Reliable, fast, good default |
| **Anthropic** | Nuanced analysis, long documents | Better at complex reasoning |
| **Google** | Large context windows | Good for lengthy documents |
| **Ollama** | Privacy, local processing | No data leaves your machine |
For more recent references check: [https://artificialanalysis.ai/models](https://artificialanalysis.ai/models)
When analysing large collections
1. **Start small** - Test on 2-3 documents first
2. **Check outputs** - Do they match expectations?
3. **Refine instructions** - Tighten schema if needed
4. **Scale up** - Run on full collection
## Results
After analysis completes, results appear as **annotations** on your assets. Each annotation contains the structured data your schema extracted.
View results:
* **On the asset** - See all annotations for a specific document
* **Fragments** - If curated from a table (see [curation](/pages/concepts/curating-fragments)), you can see the persistent fragments on the asset detail view
* **In dashboards** - Aggregate and visualise across the entire run
* **Via export** - Download as CSV or JSON for external analysis
***
## Related
Define what to extract from documents
Promote good extractions to persistent metadata
Visualise results with charts and tables
Configure API keys and providers
# Manifest
Source: https://docs.open-politics.org/pages/project/manifest
Dive into our mission and the ideas behind HQ and the Open Politics Project
## Why we are building HQ
The modern information landscape is a chaotic mess that is hard to navigate.
It has become hard, almost impossible, to stay up-to-date with what is happening around the world, your bubble or inside your organisation. Many are lost and unable to make sense of the information they are presented with. Understandably so, as the amount of information is growing exponentially and the quality of the information is decreasing.
**So how can you make sense of and present the information in a way that is easy to understand and use?**
This is not a problem of knowledge. It's a problem of not having the right tools.
## The Idea
A journalist knows how to identify "security framing" in news coverage. \
A policy analyst knows what counts as "meaningful stakeholder engagement." \
An accountant knows which line on a scanned invoice is the total. \
A legal assistant knows which clause in a contract actually matters.
Any document you work with - you know what to look for. You have your own "lens" on things.
**Our goal is to enable you in applying your lens on your data.**
> (Methodically, reproducible, and at scale)
You inherited 10 years of internal documents - nobody knows what's in there. \
Your inbox has 3,000 unread emails from a project that ended badly. \
You have a folder called "misc" with 400 PDFs.
You'd know it if you saw it. You've done this before, manually, on a handful of documents.
That recognition - you've done it. On ten documents. Maybe fifty. But not five hundred. Not five thousand. At some point you stop. Not because you don't know what you're looking for. Because doing it again, and again, and again - that's not a knowledge problem. It's a labor problem. And until now, solving it meant hiring engineers or learning to code yourself.
HQ lets you write down what you're looking for - in plain language - and apply it across everything.
**Your questions, your methods and results.**
Your question becomes a schema. The schema becomes a method. The outputs structured and reproducible. The method runs at scale. And because schemas are just text, they're shareable, transparent, improvable. Others can see exactly how you defined your framework - critique it, refine it, or apply it to their own data.
**For example: analysing 1,000 speeches from the European Parliament**
In this case a csv file of 1,000 speeches
Define your questions, methods and scores.
View the results
***
## A note about software as public infrastructure
This capability of analysing large amounts of data that is generalizable and works at scale should not be locked behind wealthy private companies or institutional walls. Yet it mostly is.
Schemas, geocoding, vector search, local AI - basic components when you list them out. But that's the point. These are intelligence capabilities an open society needs. Like libraries or archives, they should be equally accessible.
Open source. Self-hostable. Or bring your own LLM keys and use our hosted variant. Share your analytical frameworks publicly if you want transparency. Use it for journalism, research, advocacy, governance - anything that serves the public interest.
None of this would exist without the many people dedicated to open source and the countless open technologies we build upon. We are standing on the shoulders of giants and a massive collaborative ecosystem. We are grateful for their work and proud to take part of it.
## Project Origins & Funding
**CCCB Datengarten Presentation**
We started this at Freie Universität Berlin's Political Science department, working on a research paper. At some point we looked at what we'd built and thought - this is useful beyond just our project. And the capability to do this kind of work exists, but it's locked behind expensive platforms or requires engineering teams - and always written in proprietary code.
So we felt to urge to build HQ, start the Open Politics Project and make this available for everyone.
Our first official funding came from the European Horizon NGI Search project. Before that, we were lucky to have a warm circle of friends and early supporters who believed in the idea before there was much to show. No venture capital - just journalists and neighbours who got it.
The business model is simple: we'll keep a free hosted version running with as much free storage space as we can shoulder. Hosting HQ is cheap - most of the cost comes from LLMs which on the hosted variant you bring yourself via API keys. We focus on helping organisations deploy and use the platform effectively: infrastructure orchestration, custom implementations, training. The platform itself always stays open source and self-hostable. No tracking or telemetry.
Our marketing is the research that gets done with these tools. If people publish interesting work using HQ, that's worth more than any advertising campaign.
## Contact & Contributing
We're building this in the open. Code, analytical methods, documentation - all public and improvable.
[engage@open-politics.org](mailto:engage@open-politics.org)
Community discussions
Come by on Thursdays to the Chaos Computer Club Berlin - we might be around :)
Report bugs, suggest features, contribute or have a look inside.
## License
**AGPLv3** - Use, modify, distribute freely. Services built on it must also be open source. Enterprise licenses available for private modifications under strict ethical guidelines.
Find the full license [here](https://github.com/open-politics/open-politics-hq/blob/main/LICENSE.md).