Act on detections

You have a vision service detecting or classifying objects, but you need your machine to respond automatically – stop an arm when a person is nearby, sort items by color, or trigger an action when an anomaly appears. This guide shows you how to build a module that reads vision results and controls other resources based on what it sees.

Concepts

The wrapper pattern

The most common approach is to create a module that wraps an existing resource. The wrapper intercepts API calls, checks vision results, and decides whether to pass the call through or block it.

For example, a “safe arm” module wraps a real arm. When your code calls move_to_position, the wrapper first checks the vision service for people in the frame. If no one is detected, it passes the command to the real arm. If a person is detected, it raises an error.

This pattern works with any resource type: arms, bases, motors, or even other services.

Choosing a resource type

Your module must implement a resource API. Pick the type that matches what you are controlling:

The choice determines which API methods you must implement. A wrapper around an arm implements the arm API. A standalone logic service might implement the generic service API.

Dependencies

Your module needs access to the vision service, a camera, and whatever resource it controls. Viam’s dependency system handles this: you declare required dependencies in validate_config, and viam-server ensures they are available before your module starts.

Steps

1. Generate the module scaffold

Install the Viam CLI and generate a module template. Replace <ORGANIZATION-ID> with your organization ID.

viam module generate --language python --model-name safe-arm \
  --name my-vision-module --public-namespace <ORGANIZATION-ID> --public

The CLI creates a project directory with the files you need. The only file you need to modify is the model file in src/models/.

2. Add imports

Open the generated model file (for example, src/models/safe-arm.py) and add imports for the vision service and any resources you will control:

from typing import cast
from viam.services.vision import *

Import additional resource types as needed. For the safe arm example:

from viam.components.arm import Arm

3. Validate configuration

The validate_config method parses your module’s configuration and returns a list of required dependencies. This tells viam-server to wait until all dependencies are available before starting your module.

Your module needs at minimum a camera name and a vision service name:

@classmethod
def validate_config(
    cls, config: ComponentConfig
) -> Tuple[Sequence[str], Sequence[str]]:
    req_deps = []
    fields = config.attributes.fields
    if "camera_name" not in fields:
        raise Exception("missing required camera_name attribute")
    elif not fields["camera_name"].HasField("string_value"):
        raise Exception("camera_name must be a string")
    camera_name = fields["camera_name"].string_value
    if not camera_name:
        raise ValueError("camera_name cannot be empty")
    req_deps.append(camera_name)
    if "vision_name" not in fields:
        raise Exception("missing required vision_name attribute")
    elif not fields["vision_name"].HasField("string_value"):
        raise Exception("vision_name must be a string")
    vision_name = fields["vision_name"].string_value
    if not vision_name:
        raise ValueError("vision_name cannot be empty")
    req_deps.append(vision_name)
    return req_deps, []

Add validation for any other resources your module wraps. For the safe arm, add arm_name to the required dependencies the same way.

4. Initialize dependencies in reconfigure

The reconfigure method runs when the module starts and whenever configuration changes. Use it to get references to your dependencies:

def reconfigure(
    self, config: ComponentConfig,
    dependencies: Mapping[ResourceName, ResourceBase]
):
    camera_name = config.attributes.fields["camera_name"].string_value
    vision_name = config.attributes.fields["vision_name"].string_value

    vision_resource_name = VisionClient.get_resource_name(vision_name)
    if vision_resource_name not in dependencies:
        raise KeyError(f"Vision service '{vision_name}' not found in "
                       f"dependencies. Available: "
                       f"{list(dependencies.keys())}")

    self.vision_service = cast(VisionClient,
                               dependencies[vision_resource_name])
    self.camera_name = camera_name

    return super().reconfigure(config, dependencies)

For the safe arm, also initialize the arm reference:

    arm_name = config.attributes.fields["arm_name"].string_value
    arm_resource_name = Arm.get_resource_name(arm_name)
    self.arm = cast(Arm, dependencies[arm_resource_name])

5. Implement the vision check

Create a helper method that queries the vision service and returns a decision:

async def _is_safe(self):
    detections = await self.vision_service.get_detections_from_camera(
        self.camera_name)
    for d in detections:
        if d.confidence > 0.4 and d.class_name == "Person":
            self.logger.warn(
                f"Detected {d.class_name} "
                f"with confidence {d.confidence}.")
            return False
    return True

async def _is_safe(self):
    classifications = (
        await self.vision_service.get_classifications_from_camera(
            self.camera_name, 4))
    for c in classifications:
        if c.confidence > 0.6 and c.class_name == "UNSAFE":
            self.logger.warn(
                f"Classification {c.class_name} "
                f"with confidence {c.confidence}.")
            return False
    return True

6. Wire the check into API methods

Override the API methods where you want vision-based gating. For the safe arm:

async def move_to_position(
    self,
    pose: Pose,
    *,
    extra: Optional[Dict[str, Any]] = None,
    timeout: Optional[float] = None,
    **kwargs
):
    if await self._is_safe():
        await self.arm.move_to_position(
            pose, extra=extra, timeout=timeout)
    else:
        raise ValueError(
            "Person detected. Safe arm will not move.")


async def move_to_joint_positions(
    self,
    positions: JointPositions,
    *,
    extra: Optional[Dict[str, Any]] = None,
    timeout: Optional[float] = None,
    **kwargs
):
    if await self._is_safe():
        await self.arm.move_to_joint_positions(
            positions, extra=extra, timeout=timeout)
    else:
        raise ValueError(
            "Person detected. Safe arm will not move.")

Pass through all other methods to the underlying resource:

async def do_command(
    self,
    command: Mapping[str, ValueTypes],
    *,
    timeout: Optional[float] = None,
    **kwargs
) -> Mapping[str, ValueTypes]:
    return await self.arm.do_command(
        command, timeout, **kwargs)

7. Test locally

Configure the module as a local module on your machine:

1. Navigate to your machine’s CONFIGURE tab.
2. Click +, select Local module, then Local module again.
3. Enter the path to run.sh (for example, /home/user/my-vision-module/run.sh).
4. Click Create, then Save.

Add your resource as a local component:

1. Click +, select Local module, then Local component.

2. Fill in:

   • Model namespace triplet: check your module’s meta.json
   • Type: the resource type (for example, arm)
   • Name: a descriptive name (for example, safe-arm-1)

3. Add the configuration attributes:

   {
     "camera_name": "my-camera",
     "vision_name": "my-detector",
     "arm_name": "my-arm"
   }
   

4. Save and use the TEST panel to verify behavior.

8. Upload to the registry

Once your module works locally:

1. Commit and push your code to a GitHub repository.
2. Follow the steps to upload your module using cloud build.
3. After upload, remove the local module and add the resource from the registry instead.

9. Update references

If your module wraps another resource, update any services or processes that reference the original. For example, if motion planning used my-arm, update it to use safe-arm-1 so all movement commands go through the vision check.

Try It

1. Configure a safe arm module with a person detection model and point the camera at yourself. Attempt to move the arm and verify it refuses.
2. Move out of frame and try again – the arm should move normally.
3. Adjust the confidence threshold in _is_safe and observe how it affects sensitivity.
4. Try swapping detections for classifications to see how the two approaches differ.

Troubleshooting

What’s Next

• Alert on Detections – send email or webhook notifications when specific objects are detected.
• Deploy a Module – package and upload your module to the Viam Registry.
• Vision Service API Reference – full API documentation for detections, classifications, and more.