Working with EyeGestures engines can be complex, so I decided to write this post to clarify the process.

This guide will walk you through the necessary components, using the example provided in examples/simple_example_v2.py.

Imports and Main Object + Helper

First, let's focus on the essential imports. From the EyeGestures library, EyeGestures_v2 is the core object that contains the primary algorithm.

from eyeGestures.utils import VideoCapture
from eyeGestures.eyegestures import EyeGestures_v2

gestures = EyeGestures_v2()
cap = VideoCapture(0)

To streamline operations, I created a wrapper for the OpenCV camera feed, called VideoCapture. While you can use other sources for images (e.g., from a web or frontend client), using VideoCapture on a local machine can be advantageous. Unlike the native OpenCV camera feed, it is bufferless.

Bufferless operation means that frames are not stored in a buffer, allowing the frame rate to be dictated by the speed of the gaze tracking processing algorithm. This design ensures that if gaze tracking takes more time, the camera does not store frames, thus avoiding latency and delays in the algorithm.

For instance, if the camera operates at 30fps but the gaze tracking processes at 15fps, the camera feed will be capped at 15fps. If the camera maintained a faster rate without discarding unused frames, the gaze tracking would process historical data, leading to increasing delays in the application.

Algorithm Execution

After setting up the object and capturing a frame, the next step is to obtain gaze estimates. This function, called within a loop, returns a tracking event and a calibration data event.

event, calibration = gestures.step(frame, calibrate, screen_width, screen_height)

The step function requires the current screen width and height (as these can change if the window or display size is adjusted). It also requires a calibrate flag, which indicates whether to run the calibration algorithm and produce a proper calibration event (if false, the calibration event will have dummy 0,0 positions).

Calibration Event

Calibration is critical. The calibration event is an object containing a point with x, y coordinates on the screen and an acceptance_radius for that point. This helps display a visual cue for the user during calibration, guiding their gaze.

calibration.point # x, y positions of the calibration point
calibration.acceptance_radius # radius of that point

The point's radius decreases as calibration progresses and tracking becomes more precise.

Standard Event - Application Control

The core of EyeGestures is the event, which allows you to control your application.

Similar to the calibration event, the standard event stores the position of the point adjusted to the given screen width and height. This indicates where the point should be displayed on the screen.

Additionally, the event includes two parameters: fixation, which measures whether the user is fixating on a particular component, and blink, which detects if the user is blinking (requiring them to cross the fixation threshold).

event.point # x, y positions
event.fixation # fixation level from 0.0 - 1.0 
event.blink # true or false - true indicates a blink was detected

Conclusion

This guide provides the fundamental steps to effectively use the EyeGestures_v2 engine. By following these instructions, you can streamline your application development and ensure precise gaze tracking.