Vision Classes

Mappers

Proxymow Server initialises multiple mappers for various tasks:

Image Mappers

  • Distortion - used to distort the virtual camera to make it more realistic
  • Undistortion - used to provide undistorted camera image to feed calibration
  • Unwarp - used to unwarp the undistorted camera image during calibration
  • Undistortion Unwarp - used to undistort and unwarp the camera image for the arena view

Data Mappers

  • Undistortion Unwarp - master mapper for converting pixels to metres
  • Grid Undistortion Unwarp - on-the-fly mapper for converting pixels to metres

Mappers have to be populated before they can be used. A pipeline is passed to the constructor, a list of operations in specific order. These can include: * transform * unbarrel * unbarrel_inv

transform which performs unwarping is a linear matrix operation that can be reversed using the inverse matrix.

unbarrel which performs undistortion is a non-linear operation which requires its own interpolator in order to obtain the inverse unbarrel_inv.

Data Mappers are primarily focussed on transforming contours, i.e. lists of 1d numpy arrays which represent the x,y coordinates in numpy row, column order. The DataMapper transform_contour() method accepts pixels in row, column order and returns metres in cartesian x, y order. Image Mappers implement the transform_image() method, which calls skimage.transform.warp() passing the _map as a lookup table.

The undistortion algorithm is based on the algorithm in this article.

Projection

This class is responsible for determining the best isosceles triangle that encloses a contour. It is based on the algorithm described in this paper.

The following operations are performed on the contour:

  • Sub-sampling
  • Convex Hull
  • Morph Contour to Triangle
  • Identify the Vertices

Sub-sampling reduces the number of points that need to be processed. Convex Hull ensures the shape is dictated by the outline of the target, and overcomes issues with contours collapsing in. morph_contour_to_polygon and reduce in the contour_lib do the main work. Having obtained the best triangle the final task is to identify which vertex is the tip.

Viewport

The Viewport class is responsible for maintaining a window into the scene. By tracking the target with a viewport we can shrink the size of image arrays that need to be processed to find the target. When the vision system starts, or resets, then the viewport is nulled and represents the full scene. For full scene a lower resolution image is used to mine prospects - basically places of interest that warrant further investigation.

One of the pillars of the Proxymow philosophy is to keep Vision and Navigation separate. The expectation is that this will result in a more robust system. The upshot of this is that the viewport has no knowledge of where the robot is going next, or its speed, and therefore cannot track it. The best that we can do is centre the viewport over the last known location, and oversize it appropriately. What we are trying to prevent is the target escaping the window, and more seriously clipping. The constant RESIZE_POSE_TO_VIEWPORT can be adjusted to tune this behaviour.

The Viewport class has a find_contours() class method, that is passed a prepared array that has been sliced using the viewport parameters.

Linux Camera Classes

OpticalPi and OpticalLusb are classes that manage Pi Cameras and USB Cameras on Linux platforms. They both use Picamera2 objects to set the camera properties and obtain the image. They each have a settings object, from the camera settings hierarchy, which is used to customise their behaviour.

Windows Cameras

OpticalWusb does the same job for USB Webcams on Windows platforms, but uses OpenCV instead of Picamera2.

Snapshot

The Snapshot class represents a snapshot of the vision system for a particular frame. The UI needs to display images alongside corresponding data, all whilst new frames are being processed. Four snapshots are buffered on the server to enable the UI to display a coherent view of activity.