Core Action Position: Detected Object

The following are the customizable properties specific to the Detected Object method of determining core action position:

Object

When a wizard is recorded, Recorder captures a series of images of the click position. The images are captured at very short intervals, to capture all possible variations in the object appearance. Each image represents a different state of the object to detect, whether it is idle, hovered or clicked. By default, the object state captured immediately after the click is the selected image (called mouse down); however, you have the option to select the object variation the robot will use.

Example

In Microsoft Outlook 365, the Paste button changes appearance depending on whether it is idle, hovered over, or clicked. The selected object below (indicated by a check mark and blue border) is the mouse down version of the button – as it appears just after it is clicked:

Selecting an object image variation

To select a variation of the object image:

  1. Access the core action’s Position tab

  2. In the Position dropdown list, verify that Detected Object is selected

  3. In the Object section, click on the image variation you want the robot to detect

Roll your cursor over an image to view a tooltip about the object state that the image represents.

The selected variation will appear with a check mark and a blue border. When the wizard is run, the robot will search for an image on the user's screen that matches the wizard’s object image in order to detect the object.

Customizing the detected object

Alternatively, you can customize the detected object completely by selecting a different object in the step's Display Pane.

To customize the selected step's detected object:

  1. In the Display Pane, the step's current detected object is highlighted by a blue bounding box and label, and contains a blue dot that represents the mouse click position

  2. Drag and resize the bounding box around the object you want the robot to detect, and move the blue dot within the box to reposition the mouse click

While you are dragging the bounding box, a highlighted square marks the area of the window within which you can select a new detected object.

The image of the detected object you selected and any available variations appear in the Object section, along with the Use recorded object link.

To remove the customized object and revert to the originally recorded object, click the Use recorded object link.

Wait for object to appear

If you want the robot to wait some extra time for the core action's detected object to appear before failing object detection, tick the Wait for object to appear checkbox and specify the maximum wait time. If the object does not appear by the time specified, the core action's fallback command will be executed. If the window appears before the maximum time is up, the wizard will immediately continue without waiting the remaining time.

Utilizing this option can prevent wizard failure in case of slow systems or network/internet connections.

Detection match

When a wizard is run and the robot looks for the click position, it analyzes the screen for image/text that match the captured image/text selected in the Object section. The detection is based on the minimum match threshold for images and text, which appear in the Detection match section. For full details on how to customize the default thresholds, see Detection Match Configuration.

Docking

Object docking tells the robot to which window border (or center) the detected object is docked or pinned. Docking focuses the area in which the robot searches for the detected object, so it can be helpful when performance speed or accuracy needs to be improved.

An object can be docked:

  • Horizontally: The object is located at a fixed distance from the left or right window border or from the window's horizontal center

  • Vertically: The object is located at a fixed distance from one of the top or bottom window borders or from window's vertical center

  • Both horizontally & vertically

By default, no specific docking position is selected.

If the object is not found within the expected area, the robot searches the entire screen.

How do you know if an object is docked?

To discover if an object is docked within an application window, go to the application window itself (outside of Studio) and resize the window by dragging it's borders. Pay attention to how resizing the window affects the distance between the detected object and the window's borders and center.

Horizontal docking:

  • If the distance between the object and the left border does not change when dragging the side borders, the object is docked left

  • If the distance between the object and the right border does not change when dragging the side borders, the object is docked right

  • If the distance between the object and the window's horizontal center does not change when dragging the side borders, the object is docked to center

Vertical docking:

  • If the distance between the object and the top border does not change when dragging the top/bottom borders, the object is docked to top

  • If the distance between the object and the bottom border does not change when dragging the top/bottom borders, the object is docked to bottom

  • If the distance between the object and the window's vertical center does not change when dragging the top/bottom borders, the object is docked to center

Editing an object's docking settings

To edit an object's docking settings:

  1. Access the core action’s Position tab

  2. In the Position dropdown list, verify that Detected Object is selected

  3. In the Docking section:

    1. Select the desired horizontal docking from the horizontal docking dropdown list

    2. Select the desired vertical docking from the vertical docking dropdown list