Filter at the edge

Reduce the volume of data your robot captures and syncs. Robots can generate gigabytes per day from cameras and sensors, but most of that data is redundant. Edge filtering means the machine decides what is worth recording or syncing, so you save bandwidth, storage costs, and noise in your datasets.

This is especially important for machines on cellular connections, metered networks, or with limited local storage.

This page covers three approaches, from simplest to most powerful:

1. Reduce capture frequency – capture less often.
2. Use a filtered camera – use an ML model to decide frame-by-frame what to capture.
3. Conditional sync – capture locally but only sync when conditions are met.

Reduce capture frequency (time-based sampling)

The simplest filter is capturing less often. If you configured your camera at 1 Hz (one frame per second), consider whether you actually need that rate.

1. In the Viam app, navigate to your machine’s CONFIGURE tab.
2. Find the component you are capturing from (for example, my-camera).
3. In the Data capture section, find the capture method you configured.
4. Change the frequency to a lower value:
   • 1 Hz = 1 capture per second = ~2.5 GB/day for camera images
   • 0.1 Hz = 1 capture every 10 seconds = ~250 MB/day
   • 0.0167 Hz = 1 capture per minute = ~25 MB/day
   • 0.00028 Hz = 1 capture per hour = ~0.4 MB/day
5. Click Save.

The change takes effect immediately. No restart required.

This approach works well when you need periodic snapshots but not continuous monitoring. It does not help when you need to capture specific events – for that, continue to the next techniques.

Configure conditional sync

Conditional sync lets you capture data locally at full frequency but only upload it to the cloud when certain conditions are met. This is useful when you want a local buffer of recent data but only care about syncing data that meets specific criteria.

You configure conditional sync through the data management service in your machine’s configuration. The sync configuration supports conditions based on sensor readings, component states, or other data sources on your machine.

1. In the Viam app, go to your machine’s CONFIGURE tab.
2. Find the data management service in your configuration. If you do not see it, add it by clicking +, selecting Service, and choosing data management.
3. In the data management service configuration, set the Sync interval to control how frequently synced data is uploaded. A longer interval means data accumulates locally before being sent in batches.
4. To add sync conditions, you can use the selective_syncer_name attribute to specify a custom module that controls when sync occurs. This module programmatically decides whether accumulated data should be synced based on any logic you define – sensor thresholds, time of day, connectivity status, or external triggers.

For example, you might write a selective sync module that checks a temperature sensor and only triggers sync when the reading exceeds a threshold. The data management service calls into your module to determine whether to proceed with each sync cycle.

Use a filtered camera with ML

You can use the filtered_camera registry module to selectively capture only images that contain certain objects or people, using a machine learning (ML) model. The filtered camera wraps an existing camera and only outputs frames that match your ML model’s criteria, so only interesting frames are ever written to disk.

Prerequisites

• A configured camera component on your machine. See Configure a camera if you need to set one up.
• The data management service configured. See Capture and sync edge data for instructions.

Instructions

{
  "camera": "camera-1",
  "vision_services": [
    {
      "vision": "vision-1",
      "objects": {
        "Person": 0.8
      }
    }
  ],
  "window_seconds": 0
}

Build a custom filter module

For filtering needs that go beyond what the filtered-camera module provides, you can write your own module. Common examples:

• A sensor wrapper that only reports readings above a threshold (for example, temperature above 50 degrees C)
• A camera filter with custom logic specific to your application
• A filter that combines data from multiple sensors before deciding what to capture

The pattern is: write a module that wraps an existing component, evaluates its data against your criteria, and only returns data worth capturing. Configure data capture on the wrapper component instead of the raw component.

See Write a module for the general module development guide. The key technique is accessing the source component through the dependencies parameter in your module’s reconfigure method.

Manage local storage

On constrained machines (Raspberry Pi, Jetson Nano, single-board computers), local storage is limited. Understanding how Viam manages the capture directory helps you avoid filling the disk.

How local storage works

• Captured data is written to ~/.viam/capture by default.
• Each capture creates a file (JPEG for images, JSON for tabular data).
• The sync process uploads files and deletes them after successful upload.
• If sync is disabled or the network is down, files accumulate locally.

Configure storage limits

You can configure the maximum storage that data capture will use on disk. In your data management service configuration, set the maximum_capture_file_size_bytes attribute to limit the size of individual capture files, and monitor the overall capture directory size.

To check current disk usage of the capture directory:

du -sh ~/.viam/capture

To monitor it over time:

watch -n 60 du -sh ~/.viam/capture

Best practices for constrained machines

• Use the lowest capture frequency that meets your needs. Every reduction in frequency directly reduces storage pressure.
• Enable sync. Without sync, local storage fills indefinitely.
• Use filtered capture. The filtered camera and threshold sensor modules above reduce what gets written to disk, not just what gets synced.
• Monitor disk space. Set up a simple cron job or health check that alerts you when the capture directory exceeds a size threshold.
• Adjust sync interval. A shorter sync interval (for example, every 30 seconds) keeps the local buffer small. A longer interval (for example, every 5 minutes) reduces network requests but requires more local storage.

Try it

Verify your filtering is working

1. Configure one of the filtering techniques above on your machine.
2. Open the DATA tab in the Viam app.
3. Compare data volume before and after filtering:
   • Check how many new entries appear per minute with your filter active.
   • Compare to the rate you saw in Capture and sync data before filtering.
4. For the filtered camera, verify that captured images show meaningful variation between consecutive frames.

Troubleshooting

What’s next

• Query data – write SQL and MQL queries against your filtered, high-signal dataset.
• Create a dataset – use filtered captures to build cleaner training datasets for ML models.
• Write a module – learn more about building and deploying custom modules on Viam.