IndexedDB and Dexie

Overview

Formbird uses IndexedDB as the browser-based database for offline data storage, with Dexie.js as a wrapper library. The dexie-addon-elasticsearch addon translates ElasticSearch queries to Dexie queries, enabling the same query syntax to work both online and offline.

IndexedDB Fundamentals

IndexedDB is a low-level API for client-side storage of structured data. Key characteristics:

• Asynchronous - All operations are non-blocking
• Transactional - Supports ACID transactions
• Index-based - Uses indexes for efficient querying
• Large capacity - Can store hundreds of megabytes
• Same-origin policy - Data isolated per domain

Dexie.js

Dexie.js provides a cleaner API over IndexedDB with:

• Promise-based interface
• Schema versioning and migration
• Bulk operations support
• Addon system for extensions
• Query builder syntax

Database Schema

Collections

The Formbird IndexedDB database contains these collections:

Default Indexes

The documents collection always includes these default indexes:

DOCUMENT_INDEX_PENDING_OPERATION: 'systemHeader.offline.status',
DOCUMENT_INDEX_CURRENT_VERSION_OPERATION: 'systemHeader.offline.currentVersionClient',
DOCUMENT_INDEX_VERSION_ID: 'systemHeader.versionId',
DOCUMENT_INDEX_DOCUMENT_ID: 'documentId',
DOCUMENT_INDEX_SYSTEM_TYPE: 'systemHeader.systemType',
DOCUMENT_INDEX_SUMMARY_NAME: 'systemHeader.summaryName',
DOCUMENT_INDEX_SUMMARY_DESCRIPTION: 'systemHeader.summaryDescription',
DOCUMENT_INDEX_NAME: 'name',

Index Configuration

The offlineIndex Document

Indexes are configured in a single offlineIndex document rather than scattered across templates:

{
    "documentId": "9425d82f-b25f-4dba-a370-d2b7c7fc289f",
    "systemHeader": {
        "systemType": "offlineIndex",
        "summaryName": "Offline Index Configuration"
    },
    "indexedDBIndexes": [
        "systemHeader.versionId",
        "documentId",
        "systemHeader.systemType",
        "systemHeader.summaryName",
        "name",
        "*geohash",
        "*appTags",
        "systemHeader.templateId",
        "status",
        "assetNumber"
    ]
}

Index Types

Multi-Key (Compound) Indexes

For queries filtering on multiple fields, use comma-separated compound indexes:

{
    "indexedDBIndexes": [
        "systemHeader.systemType",
        "systemHeader.templateId, systemHeader.systemType",  // Compound index
        "*appTags"
    ]
}

Creating the offlineIndex Document

For most use cases, you simply need to create a document with systemType: "offlineIndex" and define an array of indexedDBIndexes:

{
    "systemHeader": {
        "systemType": "offlineIndex",
        "summaryName": "Offline Index Configuration",
        "keyIds": ["your-offline-key"]
    },
    "indexedDBIndexes": [
        "documentId",
        "systemHeader.templateId",
        "status",
        "*appTags"
    ]
}

For consolidating indexes from multiple sources:

If you need to consolidate indexes from many existing templates, you can run the MongoDB script:

// db-scripts/Formbird/00001-00050/00019 - 12575 - offlineIndex/12575-offlineIndex.js
// Aggregates indexedDBIndexes from all templates into single offlineIndex document

After running the script, save the document in Formbird to index it in ElasticSearch.

Schema Versioning

IndexedDB requires schema upgrades when indexes change:

1. Adding indexes - Triggers automatic schema upgrade
2. Removing indexes - Also triggers upgrade (as of version 4.2.57)
3. Version tracking - Stored in keyValueStorage as idbCurrentVersion

// Dexie handles versioning automatically
db.version(1).stores({ documents: "++id,name" });
db.version(2).stores({ documents: "++id,name,status" });  // Added index

Limitation: You cannot change the primary key of a table. Clearing cache and re-syncing is required.

Dexie ElasticSearch Addon

The @formbird/dexie-addon-elasticsearch addon translates ElasticSearch queries to Dexie:

Installation

The addon is loaded dynamically (no core rebuild needed):

{
    "clientConfiguration": {
        "offline": {
            "dexieAddons": [
                {
                    "fileName": "/vendor/offline/dexie-elasticsearch-addon/dexie-elasticsearch-addon.js",
                    "exportName": "ElasticIndexedDBAddon"
                }
            ]
        }
    }
}

Supported Query Types

Query Flow

ElasticSearch Query → Addon Parser → Indexed Fields Check
                                           ↓
                    Table.where() ← Uses highest priority index (fast)
                                           ↓
                    Collection.filter() ← Applies remaining terms

Query execution process:

1. The system identifies the highest priority indexed term based on indexedDBIndexesPriority (if set), otherwise uses the order of indexedDBIndexes
2. Uses Table.where() on the highest priority available index
3. Uses Collection.filter() on the result for all remaining query terms
4. If no indexes match any query terms, uses Collection.filter() on all terms (full table scan)

Geospatial Queries

Geohashing

Geospatial queries use geohashing for indexed lookups:

// Libraries used
import geohash from 'ngeohash';
import geohashPoly from 'geohash-poly';

How it works: 1. GeoJSON coordinates are converted to geohashes during caching 2. Bounding box queries generate array of geohashes 3. Table.where().inAnyRange() efficiently queries the index 4. Neighbors are included to handle edge cases

Geohash Index Configuration

{
    "indexedDBIndexes": [
        "*geohash",
        "*locationGeo.features____geohash"
    ]
}

Geohash Precision

Geohash precision determines the size of each cell:

Configure precision based on your map zoom requirements.

Flattened Values for Array Indexing

IndexedDB cannot index object properties within arrays. Formbird uses flattened values:

Problem

{
    "parentsRel": [
        { "documentId": "abc-123", "name": "Parent 1" },
        { "documentId": "def-456", "name": "Parent 2" }
    ]
}

Cannot create index on parentsRel.documentId directly.

Solution

The client preprocessor creates flattened values:

{
    "parentsRel": [...],
    "parentsRel____documentId": ["abc-123", "def-456"]
}

Now *parentsRel____documentId can be indexed.

Enabling Flattening

For sc-related-document components:

{
    "componentName": "sc-related-document",
    "name": "parentsRel",
    "saveFlattenedValues": true
}

The core automatically flattens values if the field has an entry in indexedDBIndexes.

Query Execution

Table vs Collection Queries

// Table query (uses index - fast)
await db.documents
    .where("systemHeader.templateId")
    .equals("abc-123")
    .toArray();

// Collection query (no index - slower but more flexible)
await db.documents
    .filter(doc => doc.status === "Active")
    .toArray();

Query Priority

The addon prioritizes indexed fields in Table.where():

1. Check if queried field has an index
2. If indexed: use Table.where()
3. If not indexed: use Collection.filter()
4. Log warning if performance could be improved with index

Query Timeout

Large queries can block the UI. Configure timeout:

// Queries have configurable timeout to prevent blocking
const QUERY_TIMEOUT_MS = 30000;

Workers

Dexie Shared Worker

The dexieSharedWorker handles queries across tabs:

• Single worker shared between all tabs
• Prevents concurrent access conflicts
• Falls back to Web Worker on unsupported browsers

Query Queueing

Large queries are queued to prevent blocking:

// High priority queries (user interactions) run first
// Background queries (prefetching) run when idle

Performance Considerations

Best Practices

1. Index high-value fields that return small result sets
2. Prioritize fields that return the smallest subset of documents
3. documentId should be high priority (returns single document)

4. *appTags and systemHeader.templateId should be low priority (many documents share these values)

5. Use compound indexes for multi-field queries

6. Combine fields that are always queried together

7. Limit result sets

8. Use pagination in queries

9. Avoid toArray() on large collections

10. Batch inserts

11. Use bulk operations during caching
12. Avoid inserting documents one-by-one

Troubleshooting

Common Issues

Troubleshooting Slow Queries

1. Check if the field is indexed - Missing indexes cause full table scans
2. Review index priority - A high priority index returning large subsets (e.g., querying on systemHeader.templateId first) will return many results before the Collection.filter() step can narrow them down
3. Add new indexes - Consider adding indexes for frequently filtered fields that return small result sets
4. Check for redundant query terms - Unnecessary terms add overhead

Debug Logging

Enable verbose logging:

// In browser console
Dexie.debug = true;

Related Documentation

• 101-Offline-Architecture.md - System overview
• 109-Configuration-Reference.md - Configuration options
• 112-Performance-Optimization.md - Performance tuning