Hand-rolled Claw (3): All-around Perception — Multi-modal & Dynamic Skill Tree

Fri, 20 Mar 2026 10:45:00 +0800

[!TIP] Objective: Enable multi-modal (vision/files) perception via CLI parameters and build a dynamically-loaded plugin-based skill system.

1. Letting the CLI “See” the World

Multi-modal capabilities don’t necessarily require complex SDKs. In VISAGENT, we leverage the native support for file paths in the gemini CLI (@path) to achieve sensory integration at the RoleEngine layer:

def _do_raw_invoke(self, message, files=None):
    # Construct multi-modal suffix
    mm_suffix = ""
    if files:
        mm_suffix = "\n" + "\n".join([f"@{f}" for f in files])
    
    # Append to final Prompt
    full_input = f"{message}{mm_suffix}"
    # ... execute subprocess

Field Experience: To handle complex visual tasks, we encapsulated a dedicated vision_expert skill. By using the DEEP reasoning mode, we guide the AI through Chain-of-Thought thinking, enabling precise identification of screenshots and UI components.

Multi-Modal on Square Uncle

Hand-rolled Claw (3): All-around Perception — Multi-modal & Dynamic Skill Tree

1. Letting the CLI “See” the World