Hot data store

Keep a rolling window of recent data in a fast-query database for significantly faster queries on the last few hours or days of captured data. All data continues to be written to blob storage regardless of hot data store settings. The hot data store is an additional copy, not a replacement.

Configure

The hot data store is configured per capture method on each component. You choose which components send data to it and how many hours of data to retain.

  1. Go to app.viam.com and navigate to your machine’s CONFIGURE tab.
  2. Find the component you want to enable hot storage for (for example, your sensor).
  3. Click Advanced to expand the advanced configuration.
  4. Enable Sync to Hot Data Store.
  5. Set the Time frame to the number of hours of data to retain (for example, 24 for the last 24 hours).
  6. Click Save.

Add the recent_data_store configuration to your component’s data capture settings. Set stored_hours to the number of hours of recent data to store.

{
  "components": [
    {
      "name": "sensor-1",
      "api": "rdk:component:sensor",
      "model": "rdk:builtin:fake",
      "attributes": {},
      "service_configs": [
        {
          "type": "data_manager",
          "attributes": {
            "capture_methods": [
              {
                "method": "Readings",
                "capture_frequency_hz": 0.5,
                "additional_params": {},
                "recent_data_store": {
                  "stored_hours": 24
                }
              }
            ]
          }
        }
      ]
    }
  ]
}

The stored_hours field controls how many hours of data the hot data store retains. A scheduled cleanup job runs hourly and removes documents with a time_received timestamp older than the configured window.

Query

Queries execute against blob storage by default. To query the hot data store, specify it as the data source in your query.

Use DataClient.TabularDataByMQL with data_source set to TabularDataSourceType.TABULAR_DATA_SOURCE_TYPE_HOT_STORAGE:

from viam.gen.app.data.v1.data_pb2 import TabularDataSourceType

results = await data_client.tabular_data_by_mql(
    organization_id=ORG_ID,
    query=[
        {"$match": {"component_name": "temperature-sensor"}},
        {"$sort": {"time_received": -1}},
        {"$limit": 10},
    ],
    tabular_data_source_type=TabularDataSourceType.TABULAR_DATA_SOURCE_TYPE_HOT_STORAGE,
)

for entry in results:
    print(entry)

Use DataClient.TabularDataByMQL with TabularDataSourceType set to hot storage:

hotQueryStages := []map[string]interface{}{
    {"$match": map[string]interface{}{
        "component_name": "temperature-sensor",
    }},
    {"$sort": map[string]interface{}{"time_received": -1}},
    {"$limit": 10},
}

hotData, err := dataClient.TabularDataByMQL(ctx, orgID, hotQueryStages,
    &app.TabularDataByMQLOptions{
        TabularDataSourceType: app.TabularDataSourceTypeHotStorage,
    },
)
if err != nil {
    logger.Fatal(err)
}

for _, entry := range hotData {
    fmt.Printf("%v\n", entry)
}

Use dataClient.TabularDataByMQL with dataSourceType set to TabularDataSourceType.TABULAR_DATA_SOURCE_TYPE_HOT_STORAGE:

import { createViamClient } from "@viamrobotics/sdk";

// Configuration constants – replace with your actual values
let API_KEY = "";  // API key, find or create in your organization settings
let API_KEY_ID = "";  // API key ID, find or create in your organization settings
let ORG_ID = "";  // Organization ID, find or create in your organization settings

async function main(): Promise<void> {
    // Create Viam client
    const client = await createViamClient({
        credentials: {
            type: "api-key",
            authEntity: API_KEY_ID,
            payload: API_KEY,
        },
    });

    const tabularData = await client.dataClient.tabularDataByMQL(
        ORG_ID,
        [
            { "$match": { "component_name": "sensor-1" } },
            { "$limit": 10 }
        ],
        {
            tabularDataSourceType: 2,
        }
    );
    console.log(tabularData);
}

// Run the script
main().catch((error) => {
    console.error("Script failed:", error);
    process.exit(1);
});

The hot data store returns the same document schema as the standard data store. The only difference is the data is limited to the configured time window and queries execute faster because the dataset is smaller.

Previous

Reference

Next

Export data