System Architecture

The Swarm Nexus production system is designed for resilience, scalability, and maintainability. This document provides a high-level overview of the production topology, data flow, and how the core components interact.

Production Topology

Our infrastructure runs on a dedicated host and is orchestrated to ensure a clear separation of concerns between data collection, data presentation, and API access.

• Host: Ubuntu 23.04

• Web Server: Nginx, which terminates TLS and acts as a reverse proxy.

• Data Store: A single SQLite database running in Write-Ahead Logging (WAL) mode for concurrent read/write access.

• Services:

  • Dashboard: A Flask/Gunicorn application for the user interface.

  • Agent: A Node.js (Hono) application providing a REST API.

  • Collector: A Node.js (Puppeteer) service responsible for data ingestion.

• Orchestration: Services and recurring tasks (like backups and health checks) are managed by systemd.

System Diagram

flowchart LR
  subgraph "Host (Ubuntu 23.04)"
    Nginx[Nginx: TLS + Reverse Proxy]
    Dashboard[Dashboard Service:3010]
    Agent[Agent REST API:3011]
    Collector[Collector Service]
    DB[(SQLite Database<br>/opt/swarmnexus/data/swarm.db)]
    Heartbeat[(collector_heartbeat.json)]

    Nginx --> Dashboard
    Nginx --> Agent
    Dashboard --> DB
    Agent --> DB
    Collector --> DB
    Collector --> Heartbeat
  end
  Internet[Users & External Services] --> Nginx

End-to-End Data Flow

1. Collection: The Collector service actively scrapes Twitter (X) for mentions of @SwarmNexus. It fetches the full context of each mention, parses it for metadata (like $TICKERs), and upserts the structured data into the SQLite database. After each successful run, it writes a "heartbeat" file to confirm it is operational.

2. Storage: All data resides in a central SQLite database in WAL mode. This allows the Collector to write new data without blocking the read-only access required by the other services.

3. Presentation & Access:

   • The Dashboard (Flask app) reads from the database to provide the user interface, leaderboards, and content feeds.

   • The Agent (Node.js API) reads from the database to expose structured data via a public REST API for developers and third-party integrations.

4. Proxy & Security: All services listen on the local loopback address (127.0.0.1). Nginx is the only component exposed to the public internet. It handles all incoming traffic, terminates SSL/TLS, and reverse-proxies requests to the appropriate backend service based on the URL path (/ for the Dashboard, /agent/ for the API).