fortrace.utility.image_processing package

Submodules

fortrace.utility.image_processing.image_similarity module

fortrace.utility.image_processing.image_similarity.detect_newly_opened_window(vm_image: bytes | PathLike, target: bytes | PathLike, coordinates: tuple[int, int, int, int] | None = None) → tuple[int, int, int, int] | None

Obtain the coordinates of a newly opened window via image difference.

A newly opened window is considered as the biggest rectangle identified in the delta of two images.

Parameters:

• vm_image – screenshot before the window has opened

• target – screenshot after the window has opened (has to be the same size as vm_image)

• coordinates – coordinates in vm_image of ‘parent’ window that will be scanned for a new window (x, y, x + w, y + h)

Returns:

image coordinates of the newly opened window (x, y, x + w, y + h) or can return

None, if the two images are the same

fortrace.utility.image_processing.image_similarity.image_difference(vm_image: bytes | PathLike | Mat, target: bytes | PathLike | Mat, min_area: int = 40) → tuple[list[tuple[int, int, int, int]], list[float]]

Compute the difference between two images, using structural similarity.

Parameters:

• vm_image – image before change took place

• target – image after change took place

• min_area – minimal area in pixels to be considered as difference

Returns:

list of all differences between the two images with bounding boxes and their area. A list entry looks as follows:

((x, y, x + w, y + h), <area of bounding box>)

fortrace.utility.image_processing.image_similarity.image_similarity(vm_image: bytes | PathLike | Mat, target: bytes | PathLike | Mat) → float

Compare two images to each other using SSIM

Parameters:

• vm_image – image captured with a domain

• target – image captured with a domain or a path to a reference image

Returns:

structural similarity score between the two pictures

fortrace.utility.image_processing.image_similarity.nrmse(vm_image: bytes | PathLike | ndarray, target: bytes | PathLike | ndarray) → float

Compute ‘Normalized Root Mean Squared Error’ between the two images.

Parameters:

• vm_image – image captured with a domain

• target – image captured with a domain or a path to a reference image

Returns:

normalized error between [0,1], where a lower score means more similar images

fortrace.utility.image_processing.opencv_utils module

class fortrace.utility.image_processing.opencv_utils.OpenCVRectangle(x0: int, y0: int, x1: int, y1: int, coordination_format: Literal['xyxy', 'xywh'] = 'xyxy')

Bases: object

__getitem__(item: int | slice) → int | list[int]

Enable to use class like tuple or list and access elements with [<item>].

Parameters:

item – index of the element to access (0 <= item <= 4) or slice

Returns:

Coordinate(s) at given index.

centroid() → tuple[int, int]

Centroid of the rectangle.

property h

Height of the rectangle.

p0() → tuple[int, int]

Upper left point of the rectangle.

Note

(0,0)———————–+ | image | | p0——-+ | | | rect | | | +——–+ | | | +—————————+

Returns:

(x0, y0) of the rectangle

Return type:

The point p0

p1() → tuple[int, int]

Lower right point of the rectangle.

Note

(0,0)———————–+ | image | | +——–+ | | | rect | | | +——-p1 | | | +—————————+

Returns:

The point p1 (x1, y1) of the rectangle

property w

Width of the rectangle.

fortrace.utility.image_processing.opencv_utils.draw_rectangle(image: Mat | ndarray, rect: OpenCVRectangle, inplace: bool = True) → Mat | ndarray

Draw a rectangle on the image.

Parameters:

• image – captured image, readable by openCV

• rect – OpenCVRectangle to draw on the image

• inplace – draw the rectangle on the original image, altering it permanently

Returns:

openCV image with a drawn on rectangle

fortrace.utility.image_processing.opencv_utils.extract_rectangle(image: Mat | ndarray, rect: OpenCVRectangle) → Mat | ndarray

Extract the area of the rectangle from the image.

Parameters:

• image – OpenCV image

• rect – rectangle to be extracted

Returns:

extracted sub-area of the image

fortrace.utility.image_processing.opencv_utils.read_image(image: bytes | PathLike | ndarray, color_code: int = 1) → Mat

Read an image with OpenCV

Parameters:

• image – image to be read (can be bytes, path to image or numpy array)

• color_code – see OpenCV ImreadModes

Returns:

OpenCV image

fortrace.utility.image_processing.opencv_utils.read_image_gray_scale(image: bytes | PathLike | ndarray) → Mat

Read image in gray scale using opencv.

Parameters:

image – image object or path ot image

Returns:

the matrix containing the image

fortrace.utility.image_processing.opencv_utils.show_image(image: ndarray, title: str = 'title', rectangle: tuple[int, int, int, int] | None = None)

Shows an image with optionally drawn in rectangle.

Parameters:

• image – image read in with read_image_gray_scale

• title – title of the window

• rectangle – rectangle to draw on the image

Returns:

Opens a window to view the image. Close it with ‘Space’ to resume the scenario.

fortrace.utility.image_processing.text_detection module

fortrace.utility.image_processing.text_detection.detect_and_recognize_text(vm_image: bytes | PathLike[str], coordinates: tuple[int, int, int, int] | None = None, language: str = 'eng') → tuple[list[tuple[int, int, int, int]], list[str]]

Advanced method for text detection with OpenCV’s EAST detector and eventual OCR with Tesseract

Parameters:

• vm_image – screenshot of the VM

• coordinates – OpenCV coordinates of the area to be processed in this function, e.g., window coordinates

• language – a list of languages that are expected in the picture. Use ISO 3-digit language codes, separated by ‘+’ Example: ‘eng+fra’

Returns:

coordinates (start_x, start_y, end_x, end_y) of text and text itself

Return type:

tuple containing two lists

fortrace.utility.image_processing.text_detection.jaro(recognized_text: list[str], target: str, threshold: float = 0.6) → bool

Calculates the Jaro-Winkler distance for the provided inputs.

This function iterates over the list of detected strings (most likely OCR-ed screenshot of a VM) and tries to match the provided target string to one of them. Immediately stops, once target is matched.

Parameters:

• recognized_text – list of detected strings, e.g., from OCR-ed screenshot of a VM

• target – string that is searched for

• threshold – threshold which should trigger a True result

Returns:

True, if target could be found in recognized_text. False otherwise.

fortrace.utility.image_processing.text_detection.text_line_contains(recognized_text: list[str] | str, substrings: list[str] | str, compare_method: Literal['plain', 'ignore_case', 'jaro', 'sequence'] = 'ignore_case', escape_ansi_characters: bool = True, **kwargs) → bool

Scans the recognized text for provided substrings and immediately returns True for a match.

Parameters:

• recognized_text – list of strings with recognized text

• substrings – substrings to scan for

• compare_method – Literal, describing the supported compare methods

• escape_ansi_characters – escape ANSI characters in detected text

• **kwargs – provide additional information for some string matching methods, e.g., a threshold

Returns:

True, if any substring is in recognized text.

fortrace.utility.image_processing.text_detection.texts_similar(recognized_text: list[str], target_text: list[str], threshold: float = 0.6)

Compare two texts with each other using Jaro-Winkler distance.

The texts are aligned with each other and target text can be longer than the text to be recognized.

Parameters:

• recognized_text – presumably text obtained with perform_ocr, thus starting with gibberish

• target_text – the text to be compared against

• threshold – for comparison

Returns:

if both texts are somewhat similar False: otherwise

Return type:

True

Module contents