Skip to content

Offline Architecture

Overview

Formbird's offline mode enables continuous data access in environments with unreliable connectivity. The system caches documents to the browser's IndexedDB database and static assets via a Service Worker, allowing the application to function fully offline.

This document describes the technical architecture for developers implementing or maintaining offline functionality.

High-Level Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                              Browser                                     │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────────────┐  │
│  │   Main Thread   │  │  Service Worker │  │  Shared/Web Worker      │  │
│  │                 │  │                 │  │  (Offline Poller)       │  │
│  │  ┌───────────┐  │  │  - Workbox      │  │                         │  │
│  │  │DataService│◄─┼──┼──┤ caching       │  │  - Polls IndexedDB     │  │
│  │  └─────┬─────┘  │  │  - Static files │  │  - Sends to server      │  │
│  │        │        │  │  - Vendor libs  │  │  - Updates sync status  │  │
│  │  ┌─────▼─────┐  │  └────────┬────────┘  └────────────┬────────────┘  │
│  │  │IndexedDB  │  │           │                        │               │
│  │  │Service    │  │           │                        │               │
│  │  └─────┬─────┘  │           │                        │               │
│  │        │        │           │                        │               │
│  │  ┌─────▼─────────────────────────────────────────────▼───────────┐   │
│  │  │                     IndexedDB (Dexie)                         │   │
│  │  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐ │   │
│  │  │  │  documents   │  │    queue     │  │    keyValueStorage   │ │   │
│  │  │  └──────────────┘  └──────────────┘  └──────────────────────┘ │   │
│  │  └───────────────────────────────────────────────────────────────┘   │
│  └─────────────────┘                                                    │
└────────────────────────────────────────────────────────────────────────┬┘
                                                                         │
                              Network                                    │
                                                                         │
┌────────────────────────────────────────────────────────────────────────▼┐
│                           Server Infrastructure                          │
│  ┌──────────────────┐  ┌───────────────────┐                            │
│  │ Formbird Web     │  │  Rust Syncher     │                            │
│  │ Services         │  │  (offline-server) │                            │
│  │  - REST API      │◄─┤                   │                            │
│  │  - WebSocket     │  │  - BSON protocol  │                            │
│  │                  │  │  - Redis session  │                            │
│  └────────┬─────────┘  └─────────┬─────────┘                            │
│           │                      │                                       │
│  ┌────────▼──────────────────────▼──────────────────────────────────┐   │
│  │                     MongoDB + ElasticSearch                       │   │
│  └───────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────┘

Core Components

1. IndexedDB with Dexie

IndexedDB is the browser-based database used for offline data storage. Formbird uses Dexie.js as a wrapper library because it provides:

  • Promise-based API
  • Query translation from ElasticSearch syntax
  • Schema versioning and upgrades
  • Multi-key index support

Key Collections: | Collection | Purpose | |------------|---------| | documents | Cached documents, templates, rulesets | | queue | Documents waiting to be synced to server | | keyValueStorage | User config, session data, offline state |

Index Management: - Indexes are defined in a single offlineIndex document (systemType: "offlineIndex") - Schema upgrades occur automatically when indexes are added/removed - The following default indexes are always present:

DOCUMENT_INDEX_PENDING_OPERATION: 'systemHeader.offline.status',
DOCUMENT_INDEX_CURRENT_VERSION_OPERATION: 'systemHeader.offline.currentVersionClient',
DOCUMENT_INDEX_VERSION_ID: 'systemHeader.versionId',
DOCUMENT_INDEX_DOCUMENT_ID: 'documentId',
DOCUMENT_INDEX_SYSTEM_TYPE: 'systemHeader.systemType',
DOCUMENT_INDEX_SUMMARY_NAME: 'systemHeader.summaryName',
DOCUMENT_INDEX_SUMMARY_DESCRIPTION: 'systemHeader.summaryDescription',
DOCUMENT_INDEX_NAME: 'name',

2. Service Worker

The Service Worker handles caching of static assets (HTML, CSS, JS, images) using Workbox. It:

  • Caches the application shell for offline loading
  • Handles vendor libraries configured via vendorLibrariesRel
  • Survives page refresh and continues running when tabs are closed
  • Built with Webpack (see webpack.offline.config.js)

Configuration:

// Service worker injects manifest during build
injectManifest({
    "swDest": swDest,
    "globDirectory": "server/public",
    "globPatterns": ["**/*.{gif,eot,svg,ttf,woff,woff2,ico,png,jpg,js,css,html}"],
    "maximumFileSizeToCacheInBytes": 10 * 1024 * 1024  // 10MB max
})

3. Rust Syncher (offline-server)

The Rust-based offline syncher handles efficient synchronization of large datasets. It:

  • Uses BSON binary protocol for efficient data transfer
  • Reads session data from Redis
  • Handles document pagination during initial sync
  • Runs as a separate service, proxied via fb-proxy

Starting the syncher:

FB_CONFIG_FILE=server/config/production.json cargo run --package offline-server

4. Offline Poller

The Offline Poller runs in a Shared Worker (or Web Worker on Android Chrome) to:

  • Monitor the IndexedDB queue for unsynced documents
  • Send documents to the server in order
  • Post status updates to the main thread
  • Continue running across page refreshes

Note: Shared Workers on Android Chrome are currently being trialled but are not a standard Chrome feature yet. By default, the system uses Web Workers on Android devices. This is configured via the offline.android.enableSharedWorkers setting (see 012-Offline-Setup.md).

Data Flow

Online Mode (Caching Enabled)

  1. Read Operations:
  2. Documents are fetched from the server
  3. If the network fails mid-request, the system falls back to the local cache

  4. The user receives data from whichever source succeeds

  5. Write Operations:

  6. Documents are saved locally first to maintain order
  7. The document is added to the sync queue
  8. The Offline Poller sends queued items to the server in the background
  9. Queue status is updated on success or error

Offline Mode

  1. Read Operations:
  2. Documents are retrieved directly from the local IndexedDB cache
  3. No network requests are attempted

  4. Returns cached data or an error if the document is not cached

  5. Write Operations:

  6. Documents are saved to the local cache
  7. Documents are added to the sync queue
  8. Changes are visible immediately to the user
  9. Queued documents sync to the server when connectivity is restored

Data Service Switching

The system automatically switches between online and offline data sources based on the following conditions:

  • Caching not enabled: All operations use the online server only
  • Caching enabled, no connection: All operations use the local offline cache
  • Caching enabled, connected: Operations attempt to use the server first. If a network error occurs, the system automatically falls back to the local cache

Initial Caching Process

When a user enables caching:

  1. Navigate to caching template (if configured via cachingEnableTemplate)
  2. Load IndexedDB indexes from offlineIndex document
  3. Initialize DataService with offline mode
  4. Sync documents via Rust syncher:
  5. Fetch documents in pages (configurable via syncPageSize)
  6. Insert into IndexedDB documents collection
  7. Track progress in OfflineStatusService
  8. Cache static assets via Service Worker
  9. Cache map tiles (if configured)
  10. Mark caching complete

WebSocket Integration

Documents are pushed to cached clients via WebSocket:

  1. Server saves document to MongoDB
  2. If document has offline keys, push via WebSocket
  3. WebSocketService receives push notification
  4. If document matches user's offline keys, save to IndexedDB
  5. UI updates automatically with new data

Technology Stack

Component Technology Purpose
Browser Database IndexedDB + Dexie Offline document storage
Dexie Addon dexie-addon Extends Dexie with additional offline capabilities
Static Caching Service Worker + Workbox App shell, assets, vendor libs
Data Sync Rust (offline-server) Efficient binary sync protocol
Query Translation dexie-elasticsearch-addon Formbird-developed translator that converts ElasticSearch queries to IndexedDB queries
Session Management Redis Share sessions between services