Block reference / UI Interaction

Image search

Find a picture on the phone's screen, then tap it, wait for it or say whether it is there.

Only inWarpX Phone

Image search finds something by how it looks: an icon in a game, a button drawn into a picture, a badge the elements do not describe. You cut the picture from the phone's screen once, and the block finds it again, even on a phone with a different screen width. A picture breaks when the app's look changes, so prefer an element block such as Touch whenever there is an element to name.

Image search finds the gift box by its picture and taps it.
The Image search block's settings, filled in as in the example below.

Settings

Picture

The picture to find, shown as a preview. Capture opens the phone's screen to draw a box round it; Clear removes it, and Edit as text shows what it is stored as, which can also be a {{variable}}.

Example{ "template": "iVBOR…", "sourceWidth": 1080 }
Then

Tap it finds the picture and taps it. Wait until it shows waits for it without tapping. Check whether it is there (true / false) looks once.

ExampleTap it
Only look in (area)

Part of the screen to search, as [left, top, right, bottom] shares of the screen between 0 and 1; empty searches the whole screen. Pick opens the phone's screen to draw it.

Example[0.7, 0.1, 1, 0.3]
How alike (0.5 – 1)

How closely the screen must match the picture, from 0.5 to 1. The default 0.8 allows for small differences; raise it when the block finds the wrong thing.

Example0.8
Wait up to (ms)when Then is Tap it or Wait until it shows

How long to keep looking, in milliseconds, up to 120000.

Example10000
Fail when the time runs outwhen Then is Wait until it shows

On (the default), the block fails when the picture has not shown in time. Off, it answers false and goes on.

ExampleChecked
Save whether it was found (true / false) to a variable

Also stores the answer, true or false, in a workflow variable.

ExampleChecked
Variable namewhen Save whether it was found (true / false) to a variable is on

The variable to write. Later blocks read it as {{variables.giftFound}}, for example in a Conditions path.

ExamplegiftFound
Delay (ms)

A pause after the block finishes, before the next block starts. Write one number such as 1000, or a range such as 1000,3000 for a random pause in that range on each run.

Example1000

Every block also has the shared block settings — a description, Disable block and what to do on an error. Fields that take text accept data from earlier blocks.

What it hands on

Hands on true or false, whether the picture was found, as {{prevBlockData}} and in the variable when asked. Leaves by its single output.

Example: Collect a game's daily gift

  1. Start
  2. Start app
  3. Image search
  4. Delay
Where Image search sits in the example workflow.
  1. Pick the phone in the editor's toolbar and open the game where the gift box shows.
  2. Add an Image search block and click Capture. On the phone's screen, draw a box round the gift box icon.
  3. Leave Then on Tap it and How alike (0.5 – 1) on 0.8.
  4. Click Pick next to Only look in (area) and draw round the corner where the icon sits.
  5. Connect the block's fallback output to the rest of the flow, so a day with no gift carries on, and press Run.

Good to know

  • The search runs on this computer with OpenCV, on a picture of the screen, so no agent is needed. The picture is looked for at the size it would have on this phone, scaled from the screen it was cut from, and a step either side.
  • Tap it touches a random point in the middle half of what it found, not the same pixel every time.
  • With Tap it, a picture that does not show fails the block with IMAGE_NOT_FOUND and how alike the closest match was, such as the closest match was 62% alike. Use that figure to judge How alike (0.5 – 1).
  • Check whether it is there (true / false) looks once and never fails for a missing picture.
  • A picture of almost one colour is refused, since it would match somewhere at random. Cut something with a clear shape.
  • When the block fails, it leaves by its fallback output if one is connected: a phone block comes with On error set to Execute fallback. With nothing connected there, the workflow's On workflow error setting decides.

See also

  • Find text (OCR) — Look for text in the picture of the screen, and say whether it is there or tap it.
  • Touch — Tap, hold or double-tap an element on the phone's screen, or a point on it.
  • Swipe & check screen — Swipe until something shows, or swipe once and say whether the list has ended.