> For the complete documentation index, see [llms.txt](https://docs.k9.io/key9-identity/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.k9.io/key9-identity/highvolt/getting-started/architecture.md).

# Architecture Overview

## System Diagram

```
┌──────────────────────────────────────────────────────────────────────────┐
│                              Data Sources                                │
│                                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────────────────┐    │
│  │ voltage  │  │ suricata │  │  aws-s3  │  │    aws-s3-lambda       │    │
│  │(endpoint)│  │(network) │  │  (batch) │  │  (real-time / Lambda)  │    │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └──────────┬─────────────┘    │
└───────┼─────────────┼─────────────┼───────────────────┼──────────────────┘
        │             │             │                   │
        │        ┌────┴─────┐       │           S3 event → CloudTrail
        │        │  Redis   │       │           → EventBridge (StackSet)
        │        │ Pub/Sub  │       │           → Central event bus
        │        └────┬─────┘       │                   │
        │             │             │          STS AssumeRole (member acct)
        │             │             │          S3 GetObject (cross-account)
        └─────────────┴─────────────┴───────────────────┘
                               │
                               │  HTTPS (JWT)
                               ▼
                   ┌────────────────────────────────┐
                   │      highvolt-server           │
                   │                                │
                   │  POST /submit  ──►  Queue      │
                   │  POST /query   ──►  OpenSearch │
                   │                                │
                   │       Workers                  │
                   │     (goroutines)               │
                   └──────────┬─────────────────────┘
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
       ┌──────────┐    ┌──────────┐    ┌──────────────┐
       │   LLM    │    │OpenSearch│    │  JSONAir     │
       │(OpenAI   │    │(results  │    │(config store)│
       │ compat.) │    │ storage) │    │              │
       └──────────┘    └──────────┘    └──────────────┘
```

## Configuration Management (JSONAir)

All components fetch their runtime configuration from **JSONAir**, an external configuration service. This separates configuration from deployment. On startup each component:

1. Authenticates to JSONAir using a Personal Access Token (PAT) from the environment.
2. Downloads its JSON configuration blob.
3. Starts a background goroutine (`Monitor_Config`) that polls JSONAir on a configurable interval and hot-reloads if the config changes.

## Request Lifecycle

1. A client (Voltage, Suricata, AWS-S3) discovers a file matching a configured MIME type.
2. The client computes MD5, SHA1, and SHA256 hashes of the file.
3. The client **queries** `highvolt-server` with the SHA256 — if the file has already been analyzed, it is skipped.
4. The client base64-encodes the file and **submits** it to `highvolt-server` as a JSON payload.
5. `highvolt-server` validates the payload, places it on an internal in-memory queue, and returns immediately (async).
6. A worker goroutine dequeues the work, classifies the MIME type, and routes it to the appropriate processor (IMAGE, TEXT, PDF, OFFICE, or ARCHIVE).
7. The processor sends the file to the LLM via an OpenAI-compatible `POST /chat/completions` request.
8. The LLM returns a JSON verdict. The worker merges the verdict into the original submission JSON and indexes it to OpenSearch, using the SHA256 as the document ID.
9. The raw `file_data` field is stripped before storage — only metadata and LLM results are saved.

## Deduplication

Most clients maintain a local **GOB binary registry** (a `map[string]bool` serialized to disk) keyed by SHA256 hash (Voltage) or S3 path (aws-s3). Before submitting a file, the client checks this local cache first, then confirms with the server via the `/query` endpoint. This two-layer approach avoids re-downloading large files unnecessarily.

The `aws-s3-lambda` client is stateless and has no local registry. It relies solely on the server-side SHA256 query for deduplication. Because the Lambda is only invoked for newly created objects, re-processing existing objects is not a concern.

## Concurrency Model

`highvolt-server` uses a bounded worker pool. The pool size (`max_workers`) and queue depth (`max_queue_size`) are both configurable. Submissions that arrive when the queue is full are rejected with HTTP 503.

## Authentication Flow

```
Client                     highvolt-server
  │                               │
  │  POST /auth/token {token:PAT} │
  ├──────────────────────────────►│
  │                               │  constant-time compare
  │  {access_token: JWT, ...}     │  JWT signed w/ HMAC-SHA256
  │◄──────────────────────────────│
  │                               │
  │  POST /submit  Bearer <JWT>   │
  ├──────────────────────────────►│
  │                               │  jwtMiddleware validates
  │  200 OK                       │
  │◄──────────────────────────────│
```

JWT tokens are short-lived (configurable expiry). Clients detect 401 responses and automatically re-authenticate.