Commit eec37f
2026-08-07 11:27:07 Anonymous: init| /dev/null .. 02-system-architecture.md | |
| @@ 0,0 1,174 @@ | |
| + | --- |
| + | title: System Architecture |
| + | aliases: [3netra Architecture, Meta App Architecture] |
| + | tags: [3netra/wiki, architecture, diagrams] |
| + | product: 3netra Meta Android App |
| + | source_repository: 3Netra-ai/3n_meta_android |
| + | source_ref: origin/prod |
| + | source_revision: b48aada8fd5427b5db0c02945569963c4ae543c8 |
| + | last_verified: 2026-08-06 |
| + | up: "[[Home]]" |
| + | --- |
| + | |
| + | # System Architecture |
| + | |
| + | ## High-Level System Diagram |
| + | |
| + | ```mermaid |
| + | flowchart TB |
| + | subgraph HW["Meta Ray-Ban Smart Glasses"] |
| + | CAM[12MP Camera] |
| + | MIC[Mic Array] |
| + | SPK[Open-ear Speakers] |
| + | TAP[Temple Touch Controls] |
| + | end |
| + | |
| + | subgraph PHONE["3netra Companion App (Kotlin/Compose)"] |
| + | UI[UI Layer — Jetpack Compose] |
| + | BT[BluetoothService — orchestrator] |
| + | WCS[WearableCameraService — Meta MWDAT] |
| + | VLM[Vision Pipeline — SupabaseAiService] |
| + | FACE[FaceNet TFLite + Supabase face_embeddings] |
| + | TTS[TTS — Android TTS; Sarvam path conditional] |
| + | VOICE[VoiceCommandService / AudioTranscriptionService] |
| + | end |
| + | |
| + | subgraph CLOUD["Cloud"] |
| + | SB[(Supabase — Auth/Postgres/Realtime)] |
| + | EDGE[Supabase Edge Function chat-vision] |
| + | OAI[OpenAI gpt-4o] |
| + | SARVAM[Sarvam.ai — translate/TTS/STT] |
| + | OLLAMA[Ollama LAN — moondream/llava — dev only] |
| + | end |
| + | |
| + | CAM -->|MWDAT camera session| WCS |
| + | TAP -.gesture callback bridge not found.- BT |
| + | MIC -->|BT SCO| VOICE |
| + | BT --> WCS --> VLM |
| + | VLM -->|Functions invoke| EDGE --> OAI |
| + | BT --> TTS --> SARVAM |
| + | SPK -.audio.- TTS |
| + | UI --> SB |
| + | FACE --> SB |
| + | VLM -.legacy/dev only.- OLLAMA |
| + | ``` |
| + | |
| + | ## Service-Oriented App Architecture |
| + | |
| + | ```mermaid |
| + | flowchart LR |
| + | subgraph UI["UI Layer (Compose screens)"] |
| + | S[Screens + ViewModels] |
| + | end |
| + | subgraph SVC["Service Layer (singletons)"] |
| + | SUP[SupabaseService — auth+DB] |
| + | BLS[BluetoothService — gestures/pipeline/chat] |
| + | AI[SupabaseAiService — Edge Function client] |
| + | SAR[SarvamService — Indic voice] |
| + | WCSV[WearableCameraService — Meta camera] |
| + | FNM[FaceNetModel + Supabase face embeddings] |
| + | EMS[EmergencyAlertManager — SOS] |
| + | MISC[Hallway/Recess/Emotion services — prompt wrappers] |
| + | end |
| + | subgraph EXT["External"] |
| + | SB[Supabase] |
| + | EDGE[Supabase Edge Function] |
| + | SA[Sarvam API] |
| + | META[Meta MWDAT SDK / BLE] |
| + | AND[Android TTS / SpeechRecognizer / CameraX] |
| + | end |
| + | S --> SVC |
| + | SUP --> SB |
| + | BLS --> META |
| + | AI --> EDGE |
| + | SAR --> SA |
| + | WCSV --> META |
| + | EMS --> SB |
| + | MISC --> AWSS |
| + | BLS --> AND |
| + | ``` |
| + | |
| + | ## Repository Layout (polyglot) |
| + | |
| + | ``` |
| + | 3nApp-Mobile/ |
| + | ├── src/frontend/ # ★ PRIMARY — native Kotlin/Compose app (own Gradle build) |
| + | │ └── app/src/main/kotlin/com/threenetra/app/ |
| + | │ ├── MainActivity.kt # entry: Meta SDK init, auth deep links, start destination |
| + | │ ├── MainViewModel.kt # Supabase auth StateFlow |
| + | │ ├── services/ (18) # Supabase, Bluetooth, AwsBackend, Sarvam, WearableCamera, |
| + | │ │ # VoiceCommand, AudioTranscription, Emotion, Hallway, Recess, |
| + | │ │ # EmergencyAlert, PinManager, SmartCapture, VisionService(Ollama)… |
| + | │ ├── ml/FaceNetModel.kt # TFLite FaceNet embeddings (facenet.tflite asset) |
| + | │ ├── ui/ # Navigation, auth, onboarding, student screens, NetraTheme |
| + | │ └── utils/ # EncryptedFaceStorageManager, MediaCompressor, PdfExportManager |
| + | ├── src/backend/ # AWS SAM prototype — not referenced by the active app path |
| + | ├── android/ # Expo/React Native generated shell (vestigial; no JS sources) |
| + | ├── app/ # root-level Kotlin prototype (camera stub, returns null) |
| + | ├── functions/ # Firebase Cloud Functions — Resend welcome/notification emails (legacy) |
| + | ├── api/send-welcome.js # Vercel serverless — Resend welcome email (legacy) |
| + | ├── supabase/ # Supabase CLI local dev config (no migrations/seed committed) |
| + | ├── docs/ # 16-file doc suite (architecture, API, VLM, setup, debugging) |
| + | ├── .agent_docs/ # PRD (md+pdf), architecture/design prompts, assets |
| + | └── skills/ # agent skills: prod-local-deploy, sync-agent-skills, vlm-providers |
| + | ``` |
| + | |
| + | > [!warning] Two Android codebases coexist |
| + | > The maintained native app lives in `src/frontend/`; the Expo/RN shell at root/`android/` is missing its JS entry files (RN build would fail). All feature work happens in `src/frontend/`. See ADR-001 in [[12-Risks-Decisions]]. |
| + | |
| + | ## Backends |
| + | |
| + | ### 1. Supabase (primary data/auth) |
| + | |
| + | GoTrue auth (email/password, email OTP, Google IDToken), PostgREST, Realtime channel on `student_settings`. Kotlin SDK `io.github.jan-tennert.supabase` BOM 2.5.0 + Ktor. See [[08-Data-Model-Supabase]]. |
| + | |
| + | ### 2. Supabase Edge Function (active app call target) |
| + | |
| + | `SupabaseAiService` invokes `chat-vision` through the Supabase Functions client. The deployed function source calls OpenAI `gpt-4o`, but the Android client discards the function response and returns the literal success text `"AI response received"`. Thus capture/chat/summary UI wiring is present, but meaningful model narration is **not** verified at this ref. |
| + | |
| + | ### 3. AWS SAM "Routing Logic" prototype (not active) |
| + | |
| + | One Lambda (`nodejs20.x`, 30s/512MB) behind API Gateway catch-all `/{proxy+}` is implemented, including routes below, but `AwsBackendService` has no production call sites (only its minimal test). |
| + | |
| + | | Route | Function | |
| + | |---|---| |
| + | | `POST /api/chat` | OpenAI proxy — `{prompt, mediaType: NONE/IMAGE/VIDEO, frames[]}`; **gpt-4o** for video, **gpt-4o-mini** otherwise; max_tokens 300; system prompt "helpful and concise assistant for a visually impaired student" | |
| + | | `POST /api/vision` | Rekognition `DetectLabels` (max 10, min conf 75) | |
| + | | `POST /api/upload` | base64 image → S3 `uploads/{ts}.jpg` (no lifecycle rule is configured; a comment merely suggests one) | |
| + | | `POST /api/auth` | Supabase token verification | |
| + | |
| + | ### 4. Legacy email paths |
| + | |
| + | Firebase Cloud Functions (`sendWelcomeEmail` on auth.user onCreate, `sendNotificationEmail` callable) and Vercel `api/send-welcome.js` — all via **Resend**. No `firebase.json`; Supabase replaced Firebase auth/data. |
| + | |
| + | ## AI Pipeline (capture → speech) |
| + | |
| + | 1. Trigger: UI button, glasses gesture, or voice command |
| + | 2. `WearableCameraService.capturePhoto()` — MWDAT device session → 15fps LOW stream → still photo (Bitmap/HEIC → bytes; mutex-guarded) |
| + | 3. Optional on-device FaceNet calculation followed by Supabase embedding lookup (People/Emotions modes) |
| + | 4. `SupabaseAiService` invokes `chat-vision`; at this ref it replaces the response with a fixed success string |
| + | 5. `logLocalSession()` → in-memory history and an attempted Supabase `sessions` insert |
| + | 6. Speech is Android TTS in `SceneDescriptionScreen`; `BluetoothService.speak()` has a conditional Sarvam path. End-to-end voice/vendor behavior is untested. |
| + | |
| + | See the sequence diagram in [[03-Frontend-Navigation#Frontend Flow Chart (core capture loop)]]. |
| + | |
| + | ## Tech Stack |
| + | |
| + | | Layer | Technology | |
| + | |---|---| |
| + | | UI | Jetpack Compose (BOM 2024.10.00) + Material3 + Navigation-Compose 2.7.7 | |
| + | | Language/Async | Kotlin 2.1.10, Coroutines, StateFlow, kotlinx-serialization | |
| + | | Auth/DB | Supabase (gotrue-kt, postgrest-kt, realtime-kt) + Ktor client | |
| + | | Identity | Google Credential Manager + play-services-auth 21.0.0 | |
| + | | Glasses | Meta MWDAT SDK 0.7.0 (GitHub Packages Maven) | |
| + | | Vision AI | Supabase `chat-vision` Edge Function → OpenAI gpt-4o; AWS and Ollama clients are unreferenced/legacy paths | |
| + | | Face ID | LiteRT 1.4.0 FaceNet; active flow fetches/stores embeddings in Supabase | |
| + | | Voice | Sarvam.ai + Android TextToSpeech/SpeechRecognizer | |
| + | | Camera (phone) | CameraX 1.4.2 (face registration) | |
| + | | Security | androidx.security-crypto (EncryptedSharedPreferences), jbcrypt | |
| + | | Build/CI | Gradle 8.x, GitHub Actions debug-APK build on pushes to `prod`; it does not run tests or lint | |
| + | |
| + | --- |
| + | |
| + | > [!tip] Navigation |
| + | > ⬅️ [[01-Product-Overview|Product Overview]] · 🏠 [[Home]] · ➡️ [[03-Frontend-Navigation|Frontend & Navigation]] |