Data Sources

Comprehensive documentation of all data sources, feed tiers, and collection methods used by World Monitor.

---

Data Layers

Real-Time Data Layers

---

Intelligence Feeds

Telegram OSINT Intelligence Feed

26 curated Telegram channels provide a raw, low-latency intelligence feed covering conflict zones, OSINT analysis, and breaking news — sources that are often minutes ahead of traditional wire services during fast-moving events.

Tier	Channels
Tier 1	VahidOnline (Iran politics)
Tier 2	Abu Ali Express, Aurora Intel, BNO News, Clash Report, DeepState, Defender Dome, Iran International, LiveUAMap, OSINTdefender, OSINT Updates, Ukraine Air Force (kpszsu), Povitryani Tryvoha
Tier 3	Bellingcat, CyberDetective, GeopoliticalCenter, Middle East Spectator, Middle East Now Breaking, NEXTA, OSINT Industries, OsintOps News, OSINT Live, OsintTV, The Spectator Index, War Monitor, WFWitness

Architecture: A GramJS MTProto client running on the Railway relay polls all channels sequentially on a 60-second cycle. Each channel has a 15-second timeout (GramJS getEntity/getMessages can hang indefinitely on FLOOD_WAIT or MTProto stalls), and the entire cycle has a 3-minute hard timeout. A stuck-poll guard force-clears the mutex after 3.5 minutes, and FLOOD_WAIT errors from Telegram’s API stop the cycle early rather than propagating to every remaining channel.

Messages are deduplicated by ID, filtered to exclude media-only posts (images without text), truncated to 800 characters, and stored in a rolling 200-item buffer. The relay connects with a 60-second startup delay to prevent AUTH_KEY_DUPLICATED errors during Railway container restarts (the old container must fully disconnect before the new one authenticates). Topic classification (breaking, conflict, alerts, osint, politics, middleeast) and channel-based filtering happen at query time via the /telegram/feed relay endpoint.

OREF Rocket Alert Integration

The dashboard monitors Israel’s Home Front Command (Pikud HaOref) alert system for incoming rocket, missile, and drone sirens — a real-time signal that is difficult to obtain programmatically due to Akamai WAF protection.

Data flow: The Railway relay polls oref.org.il using curl (not Node.js fetch, which is JA3-blocked) through a residential proxy with an Israeli exit IP. On startup, the relay bootstraps history via a two-phase strategy: Phase 1 loads from Redis (filtering entries older than 7 days); if Redis is empty, Phase 2 fetches from the upstream OREF API with exponential backoff retry (up to 3 attempts, delays of 3s/6s/12s + jitter). Alert history is persisted to Redis with dirty-flag deduplication to prevent redundant writes. Live alerts are polled every 5 minutes. Wave detection groups individual siren records by timestamp to identify distinct attack waves. Israel-local timestamps are converted to UTC with DST-aware offset calculation. 1,480 Hebrew→English location translations — an auto-generated dictionary (from the pikud-haoref-api cities.json source) enables automatic translation of Hebrew city names in alert data. Unicode bidirectional control characters are stripped via sanitizeHebrew() before translation lookups to prevent mismatches.

CII integration: Active OREF alerts boost Israel’s CII conflict component by up to 50 points (25 + min(25, alertCount × 5)). Rolling 24-hour history adds a secondary boost: 3–9 alerts in the window contribute +5, 10+ contribute +10 to the blended score. This means a sustained multi-wave rocket barrage drives Israel’s CII significantly higher than a single isolated alert.

GPS/GNSS Interference Detection

GPS jamming and spoofing — increasingly used as electronic warfare in conflict zones — is detected by analyzing ADS-B transponder data from aircraft that report GPS anomalies. Data is sourced from gpsjam.org, which aggregates ADS-B Exchange data into H3 resolution-4 hexagonal grid cells.

Classification: Each H3 cell reports the ratio of aircraft with GPS anomalies vs. total aircraft. Cells with fewer than 3 aircraft are excluded as statistically noisy. The remaining cells are classified:

Interference Level	Bad Aircraft %	Map Color
Low	0–2%	Hidden
Medium	2–10%	Amber
High	> 10%	Red

Region tagging: Each hex cell is tagged to one of 12 named conflict regions via bounding-box classification (Iran-Iraq, Levant, Ukraine-Russia, Baltic, Mediterranean, Black Sea, Arctic, Caucasus, Central Asia, Horn of Africa, Korean Peninsula, South China Sea) for filtered region views.

CII integration: ingestGpsJammingForCII maps each H3 hex centroid to a country via the local geometry service, then accumulates per-country interference counts. In the CII security component, GPS jamming contributes up to 35 points: min(35, highCount × 5 + mediumCount × 2).

Security Advisory Aggregation

Government travel advisories serve as expert risk assessments from national intelligence agencies — when the US State Department issues a “Do Not Travel” advisory, it reflects classified threat intelligence that no open-source algorithm can replicate.

Sources: 4 government advisory feeds (US State Dept, Australia DFAT Smartraveller, UK FCDO, New Zealand MFAT), 13 US Embassy country-specific alert feeds (Thailand, UAE, Germany, Ukraine, Mexico, India, Pakistan, Colombia, Poland, Bangladesh, Italy, Dominican Republic, Myanmar), and health agency feeds (CDC Travel Notices, ECDC epidemiological updates, WHO News, WHO Africa Emergencies).

Advisory levels (ranked): Do-Not-Travel (4) → Reconsider Travel (3) → Exercise Caution (2) → Normal (1) → Info (0). Both RSS (item) and Atom (entry) formats are parsed. Country extraction uses a 265-entry country name map (generated from GeoJSON + aliases) for accurate matching of compound names and alternative spellings.

Architecture: Advisories follow the gold standard pattern. A Railway cron seed (seed-security-advisories.mjs) fetches all 24 feeds hourly via the relay RSS proxy, deduplicates by title, builds a byCountry risk-level map, and writes to Redis. The Vercel ListSecurityAdvisories RPC reads Redis only (no external calls). Bootstrap hydration provides instant advisory data on page load.

CII integration — advisories feed into instability scores through two mechanisms:

• Score boost: Do-Not-Travel → +15 points, Reconsider → +10, Caution → +5. Multi-source agreement adds a consensus bonus: ≥3 governments concur → +5, ≥2 → +3
• Score floor: Do-Not-Travel from any government forces a minimum CII score of 60; Reconsider forces minimum 50. This prevents a country with low event data but active DNT warnings from showing an artificially calm score

The Security Advisories panel displays advisories with colored level badges and source country flags, filterable by severity (Critical, All) and issuing government (US, AU, UK, NZ, Health).

Airport Delay & NOTAM Monitoring

111 airports across 5 regions (Americas, Europe, Asia-Pacific, MENA, Africa) are continuously monitored for delays, ground stops, and closures through three independent data sources:

Source	Coverage	Method
FAA ASWS	14 US hub airports	Real-time XML feed from nasstatus.faa.gov — ground delays, ground stops, arrival/departure delays, closures
AviationStack	40 international airports	Last 100 flights per airport — cancellation rate and average delay duration computed from flight records
ICAO NOTAM API	46 MENA airports	Real-time NOTAM (Notice to Air Missions) query for active airport/airspace closures

NOTAM closure detection targets MENA airports where airspace closures due to military activity or security events carry strategic significance. Detection uses two methods: ICAO Q-code matching (aerodrome/airspace closure codes FA, AH, AL, AW, AC, AM combined with closure qualifiers LC, AS, AU, XX, AW) and free-text regex scanning for closure keywords (AD CLSD, AIRPORT CLOSED, AIRSPACE CLOSED). When a NOTAM closure is detected, it overrides any existing delay alert for that airport with a severe/closure classification.

Severity thresholds: Average delay ≥15min or ≥15% delayed flights = minor; ≥30min/30% = moderate; ≥45min/45% = major; ≥60min/60% = severe. Cancellation rate ≥80% with ≥10 flights = closure. All results are cached for 30 minutes in Redis. When no AviationStack API key is configured, the system generates probabilistic simulated delays for demonstration — rush-hour windows and high-traffic airports receive higher delay probability.

Aviation Intelligence Panel

The Airline Intel panel provides a comprehensive 6-tab aviation monitoring interface covering operations, flights, carriers, tracking, news, and pricing:

Tab	Data	Source
Ops	Delay percentages, cancellation rates, NOTAM closures, ground stops	FAA ASWS, AviationStack, ICAO
Flights	Specific flight status with scheduled vs estimated times, divert/cancel flags	AviationStack flight lookup
Airlines	Per-carrier statistics at monitored airports (delay %, cancellation rate, flight count)	AviationStack carrier ops
Tracking	Live ADS-B aircraft positions with altitude, ground speed, heading, on-ground status	OpenSky / Wingbits
News	20+ recent aviation news items with entity matching (airlines, airports, aircraft types)	RSS feeds tagged aviation
Prices	Multi-carrier price quotes with cabin selection (Economy/Business), currency conversion, non-stop filters	Travelpayouts cached data

Aviation watchlist — users customize a personal list of airports (IATA codes), airlines, and routes persisted to localStorage as aviation:watchlist:v1. The default watchlist includes IST, ESB, SAW, LHR, FRA, CDG airports and TK airline, reflecting common monitoring needs. The watchlist drives which airports appear in the Ops tab and which carriers are tracked in the Airlines tab.

Delay severity is classified across five levels — normal, minor, moderate, major, severe — with each alert carrying a severity source (FAA, Eurocontrol, or computed from flight data). The panel auto-refreshes on a 5-minute polling cycle with a live indicator badge, and uses a SmartPollLoop with adaptive backoff on failures.

---

Cyber & Natural Events

Cyber Threat Intelligence Layer

Six threat intelligence feeds provide indicators of compromise (IOCs) for active command-and-control servers, malware distribution hosts, phishing campaigns, malicious URLs, and ransomware operations:

Feed	IOC Type	Coverage
Feodo Tracker (abuse.ch)	C2 servers	Botnet C&C infrastructure
URLhaus (abuse.ch)	Malware hosts	Malware distribution URLs
C2IntelFeeds	C2 servers	Community-sourced C2 indicators
AlienVault OTX	Mixed	Open threat exchange pulse IOCs
AbuseIPDB	Malicious IPs	Crowd-sourced abuse reports
Ransomware.live	Ransomware	Active ransomware group feeds

Each IP-based IOC is geo-enriched using ipinfo.io with freeipapi.com as fallback. Geolocation results are Redis-cached for 24 hours. Enrichment runs concurrently — 16 parallel lookups with a 12-second timeout, processing up to 250 IPs per collection run.

IOCs are classified into four types (c2_server, malware_host, phishing, malicious_url) with four severity levels, rendered as color-coded scatter dots on the globe. The layer uses a 10-minute cache, a 14-day rolling window, and caps display at 500 IOCs to maintain rendering performance.

Natural Disaster Monitoring

Three independent sources are merged into a unified disaster picture, then deduplicated on a 0.1° geographic grid:

Source	Coverage	Types	Update Frequency
USGS	Global earthquakes M4.5+	Earthquakes	5 minutes
GDACS	UN-coordinated disaster alerts	Earthquakes, floods, cyclones, volcanoes, wildfires, droughts	Real-time
NASA EONET	Earth observation events	13 natural event categories (30-day open events)	Real-time

GDACS events carry color-coded alert levels (Red = critical, Orange = high) and are filtered to exclude low-severity Green alerts. EONET wildfires are filtered to events within 48 hours to prevent stale data. Earthquakes from EONET are excluded since USGS provides higher-quality seismological data.

The merged output feeds into the signal aggregator for geographic convergence detection — e.g., an earthquake near a pipeline triggers an infrastructure cascade alert.

Dual-Source Protest Tracking

Protest data is sourced from two independent providers to reduce single-source bias:

1. ACLED (Armed Conflict Location & Event Data) — 30-day window, tokenized API with Redis caching (10-minute TTL). Covers protests, riots, strikes, and demonstrations with actor attribution and fatality counts.
2. GDELT (Global Database of Events, Language, and Tone) — 7-day geospatial event feed filtered to protest keywords. Events with mention count ≥5 are included; those above 30 are marked as validated.

Events from both sources are Haversine-deduplicated on a 0.1° grid (~10km) with same-day matching. ACLED events take priority due to higher editorial confidence. Severity is classified as:

• High — fatalities present or riot/clash keywords
• Medium — standard protest/demonstration
• Low — default

Protest scoring is regime-aware: democratic countries use logarithmic scaling (routine protests don’t trigger instability), while authoritarian states use linear scoring (every protest is significant). Fatalities and concurrent internet outages apply severity boosts.

Climate Anomaly Detection

15 conflict-prone and disaster-prone zones are continuously monitored for temperature and precipitation anomalies using Open-Meteo ERA5 reanalysis data. A 30-day baseline is computed, and current conditions are compared against it to determine severity:

Severity	Temperature Deviation	Precipitation Deviation
Extreme	> 5°C above baseline	> 80mm/day above baseline
Moderate	> 3°C above baseline	> 40mm/day above baseline
Normal	Within expected range	Within expected range

Anomalies feed into the signal aggregator, where they amplify CII scores for affected countries (climate stress is a recognized conflict accelerant). The Climate Anomaly panel surfaces these deviations in a severity-sorted list.

Displacement Tracking

Refugee and displacement data is sourced from the UN OCHA Humanitarian API (HAPI), providing population-level counts for refugees, asylum seekers, and internally displaced persons (IDPs). The Displacement panel offers two perspectives:

• Origins — countries people are fleeing from, ranked by outflow volume
• Hosts — countries absorbing displaced populations, ranked by intake

Crisis badges flag countries with extreme displacement: > 1 million displaced (red), > 500,000 (orange). Displacement outflow feeds into the CII as a component signal — high displacement is a lagging indicator of instability that persists even when headlines move on.

Population Exposure Estimation

Population exposure estimation calculates affected populations within event-specific radii. See Algorithms - Population Exposure for the full methodology.

---

Infrastructure Monitoring

Strategic Port Infrastructure

62 strategic ports are cataloged across six types, reflecting their role in global trade and military posture:

Type	Count	Examples
Container	21	Shanghai (#1, 47M+ TEU), Singapore, Ningbo, Shenzhen
Oil/LNG	8	Ras Tanura (Saudi), Sabine Pass (US), Fujairah (UAE)
Chokepoint	9	Suez Canal, Panama Canal, Strait of Malacca, Gibraltar, Bosphorus, Dardanelles
Naval	6	Zhanjiang, Yulin (China), Vladivostok (Russia)
Mixed	15+	Ports serving multiple roles (trade + military)
Bulk	20+	Regional commodity ports

Ports are ranked by throughput and weighted by strategic importance in the infrastructure cascade model: oil/LNG terminals carry 0.9 criticality, container ports 0.7, and naval bases 0.4. Port proximity appears in the Country Brief infrastructure exposure section.

Live Webcam Surveillance Grid

22 YouTube live streams from geopolitical hotspots across 5 regions provide continuous visual situational awareness:

Region	Cities
Iran / Attacks	Tehran, Tel Aviv, Jerusalem (Western Wall)
Middle East	Jerusalem (Western Wall), Tehran, Tel Aviv, Mecca (Grand Mosque)
Europe	Kyiv, Odessa, Paris, St. Petersburg, London
Americas	Washington DC, New York, Los Angeles, Miami
Asia-Pacific	Taipei, Shanghai, Tokyo, Seoul, Sydney

The webcam panel supports two viewing modes: a 4-feed grid (default strategic selection: Jerusalem, Tehran, Kyiv, Washington DC) and a single-feed expanded view. Region tabs (ALL/IRAN/MIDEAST/EUROPE/AMERICAS/ASIA) filter the available feeds. The Iran/Attacks tab provides a dedicated 2×2 grid for real-time visual monitoring during escalation events between Iran and Israel.

Resource management is aggressive — iframes are lazy-loaded via Intersection Observer (only rendered when the panel scrolls into view), paused after 5 minutes of user inactivity, and destroyed from the DOM entirely when the browser tab is hidden. On Tauri desktop, YouTube embeds route through a cloud proxy to bypass WKWebView autoplay restrictions. Each feed carries a fallback video ID in case the primary stream goes offline.

---

Server-Side Aggregation

Server-Side Feed Aggregation

Rather than each client browser independently fetching dozens of RSS feeds through individual edge function invocations, the listFeedDigest RPC endpoint aggregates all feeds server-side into a single categorized response.

Architecture:

Client (1 RPC call) → listFeedDigest → Redis check (digest:v1:{variant}:{lang})
                                              │
                                    ┌─────────┴─── HIT → return cached digest
                                    │
                                    ▼ MISS
                           ┌─────────────────────────┐
                           │  buildDigest()           │
                           │  20 concurrent fetches   │
                           │  8s per-feed timeout     │
                           │  25s overall deadline    │
                           └────────┬────────────────┘
                                    │
                              ┌─────┴─────┐
                              │ Per-feed   │ ← cached 600s per URL
                              │ Redis      │
                              └─────┬─────┘
                                    │
                                    ▼
                           ┌─────────────────────────┐
                           │  Categorized digest      │
                           │  Cached 900s (15 min)    │
                           │  Per-item keyword class.  │
                           └─────────────────────────┘

The digest cache key is news:digest:v1:{variant}:{lang} with a 900-second TTL. Individual feed results are separately cached per URL for 600 seconds. Items per feed are capped at 5, categories at 20 items each. XML parsing is edge-runtime-compatible (regex-based, no DOM parser), handling both RSS item and Atom entry formats. Each item is keyword-classified at aggregation time. An in-memory fallback cache (capped at 50 entries) provides last-known-good data if Redis fails.

This eliminates per-client feed fan-out — 1,000 concurrent users each polling 25 feed categories would have generated 25,000 edge invocations per poll cycle. With server-side aggregation, they generate exactly 1 (or 0 if the digest is cached).

---

Source Credibility & Feed Tiering

Every RSS feed is assigned a source tier reflecting editorial reliability:

Tier	Description	Examples
Tier 1	Wire services, official government sources	Reuters, AP, BBC, DOD
Tier 2	Major established outlets	CNN, NYT, The Guardian, Al Jazeera
Tier 3	Specialized/niche outlets	Defense One, Breaking Defense, The War Zone
Tier 4	Aggregators and blogs	Google News, individual analyst blogs

Feeds also carry a propaganda risk rating and state affiliation flag. State-affiliated sources (RT, Xinhua, IRNA) are included for completeness but visually tagged so analysts can factor in editorial bias. Threat classification confidence is weighted by source tier — a Tier 1 breaking alert carries more weight than a Tier 4 blog post in the focal point detection algorithm.

---

Data Freshness & Intelligence Gaps

Data Freshness & Intelligence Gaps

A singleton tracker monitors 31 data sources (GDELT, GDELT Doc, RSS, AIS, OpenSky, Wingbits, USGS, weather, outages, ACLED, ACLED conflict, Polymarket, predictions, PizzINT, economic, oil, spending, NASA FIRMS, cyber threats, UCDP, UCDP events, HAPI, UNHCR, climate, WorldPop, giving, BIS, WTO trade, supply chain, security advisories, GPS jamming) with status categorization: fresh (<15 min), stale (2h), very_stale (6h), no_data, error, disabled. Two sources (GDELT, RSS) are flagged as requiredForRisk — their absence directly impacts CII scoring quality. The tracker explicitly reports intelligence gaps — what analysts can’t see — preventing false confidence when critical data sources are down or degraded.

---

Prediction Markets

Prediction Markets as Leading Indicators

Polymarket geopolitical markets are queried using tag-based filters (Ukraine, Iran, China, Taiwan, etc.) with 5-minute caching. Market probability shifts are correlated with news volume: if a prediction market moves significantly before matching news arrives, this is flagged as a potential early-warning signal.

4-tier fetch strategy — prediction markets use a cascading fetch chain to maximize data availability:

1. Bootstrap hydration — zero-network, page-load-embedded data from the Redis-cached predictions key. If fresh (<20 min), the panel renders instantly without any API call
2. Sebuf RPC — POST /api/prediction/v1/list-prediction-markets queries Redis for the seed-script-maintained cache. Single request, sub-100ms cold start
3. Browser-direct Polymarket — the browser fetches Polymarket’s Gamma API directly, bypassing JA3 fingerprinting (browser TLS passes Cloudflare)
4. Sidecar native TLS — on Tauri desktop, Rust’s reqwest TLS fingerprint differs from Node.js, providing another bypass vector

Country-specific markets — fetchCountryMarkets(country) maps 40+ countries to Polymarket tag variants (e.g., “Russia” matches [“russia”, “russian”, “moscow”, “kremlin”, “putin”]), enabling the Country Brief to display prediction contracts relevant to any nation.

Smart filtering — markets are ranked by 24h trading volume, filtered to exclude sports and entertainment (100+ exclusion keywords: NBA, NFL, Oscar, Grammy, etc.), and require meaningful price divergence from 50% or volume above $50K to suppress noise. Each variant gets different tag sets — geopolitical queries politics/world/ukraine/middle-east tags, tech queries ai/crypto/business tags.

Cloudflare JA3 bypass — Polymarket’s API is protected by Cloudflare TLS fingerprinting (JA3) that blocks all server-side requests. The system uses a 3-tier fallback:

Tier	Method	When It Works
1	Browser-direct fetch	Always (browser TLS passes Cloudflare)
2	Tauri native TLS (reqwest)	Desktop app (Rust TLS fingerprint differs from Node.js)
3	Vercel edge proxy	Rarely (edge runtime sometimes passes)

Once browser-direct succeeds, the system caches this state and skips fallback tiers on subsequent requests. Country-specific markets are fetched by mapping countries to Polymarket tags with name-variant matching (e.g., “Russia” matches titles containing “Russian”, “Moscow”, “Kremlin”, “Putin”).

Markets are filtered to exclude sports and entertainment (100+ exclusion keywords), require meaningful price divergence from 50% or volume above $50K, and are ranked by trading volume. Each variant gets different tag sets — geopolitical focus queries politics/world/ukraine/middle-east tags, while tech focus queries ai/crypto/business tags.