# Architecture Overview

## System Diagram

```
┌──────────────────────────────────────────────────────────────────────────┐
│                              Data Sources                                │
│                                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────────────────┐    │
│  │ voltage  │  │ suricata │  │  aws-s3  │  │    aws-s3-lambda       │    │
│  │(endpoint)│  │(network) │  │  (batch) │  │  (real-time / Lambda)  │    │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └──────────┬─────────────┘    │
└───────┼─────────────┼─────────────┼───────────────────┼──────────────────┘
        │             │             │                   │
        │        ┌────┴─────┐       │           S3 event → CloudTrail
        │        │  Redis   │       │           → EventBridge (StackSet)
        │        │ Pub/Sub  │       │           → Central event bus
        │        └────┬─────┘       │                   │
        │             │             │          STS AssumeRole (member acct)
        │             │             │          S3 GetObject (cross-account)
        └─────────────┴─────────────┴───────────────────┘
                               │
                               │  HTTPS (JWT)
                               ▼
                   ┌────────────────────────────────┐
                   │      highvolt-server           │
                   │                                │
                   │  POST /submit  ──►  Queue      │
                   │  POST /query   ──►  OpenSearch │
                   │                                │
                   │       Workers                  │
                   │     (goroutines)               │
                   └──────────┬─────────────────────┘
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
       ┌──────────┐    ┌──────────┐    ┌──────────────┐
       │   LLM    │    │OpenSearch│    │  JSONAir     │
       │(OpenAI   │    │(results  │    │(config store)│
       │ compat.) │    │ storage) │    │              │
       └──────────┘    └──────────┘    └──────────────┘
```

## Configuration Management (JSONAir)

All components fetch their runtime configuration from **JSONAir**, an external configuration service. This separates configuration from deployment. On startup each component:

1. Authenticates to JSONAir using a Personal Access Token (PAT) from the environment.
2. Downloads its JSON configuration blob.
3. Starts a background goroutine (`Monitor_Config`) that polls JSONAir on a configurable interval and hot-reloads if the config changes.

## Request Lifecycle

1. A client (Voltage, Suricata, AWS-S3) discovers a file matching a configured MIME type.
2. The client computes MD5, SHA1, and SHA256 hashes of the file.
3. The client **queries** `highvolt-server` with the SHA256 — if the file has already been analyzed, it is skipped.
4. The client base64-encodes the file and **submits** it to `highvolt-server` as a JSON payload.
5. `highvolt-server` validates the payload, places it on an internal in-memory queue, and returns immediately (async).
6. A worker goroutine dequeues the work, classifies the MIME type, and routes it to the appropriate processor (IMAGE, TEXT, PDF, OFFICE, or ARCHIVE).
7. The processor sends the file to the LLM via an OpenAI-compatible `POST /chat/completions` request.
8. The LLM returns a JSON verdict. The worker merges the verdict into the original submission JSON and indexes it to OpenSearch, using the SHA256 as the document ID.
9. The raw `file_data` field is stripped before storage — only metadata and LLM results are saved.

## Deduplication

Most clients maintain a local **GOB binary registry** (a `map[string]bool` serialized to disk) keyed by SHA256 hash (Voltage) or S3 path (aws-s3). Before submitting a file, the client checks this local cache first, then confirms with the server via the `/query` endpoint. This two-layer approach avoids re-downloading large files unnecessarily.

The `aws-s3-lambda` client is stateless and has no local registry. It relies solely on the server-side SHA256 query for deduplication. Because the Lambda is only invoked for newly created objects, re-processing existing objects is not a concern.

## Concurrency Model

`highvolt-server` uses a bounded worker pool. The pool size (`max_workers`) and queue depth (`max_queue_size`) are both configurable. Submissions that arrive when the queue is full are rejected with HTTP 503.

## Authentication Flow

```
Client                     highvolt-server
  │                               │
  │  POST /auth/token {token:PAT} │
  ├──────────────────────────────►│
  │                               │  constant-time compare
  │  {access_token: JWT, ...}     │  JWT signed w/ HMAC-SHA256
  │◄──────────────────────────────│
  │                               │
  │  POST /submit  Bearer <JWT>   │
  ├──────────────────────────────►│
  │                               │  jwtMiddleware validates
  │  200 OK                       │
  │◄──────────────────────────────│
```

JWT tokens are short-lived (configurable expiry). Clients detect 401 responses and automatically re-authenticate.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.k9.io/key9-identity/highvolt/getting-started/architecture.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.