A Python system controlling a computer with hand gestures via a webcam, tracking 21 hand landmarks with MediaPipe to adjust brightness and emulate the mouse via PyAutoGUI.
Remote control of computer applications with hand gestures is an academic project carried out by a team of students with technical assistance from Techno Enjaz, to build a touchless interaction system that replaces the mouse and traditional controls with hand gestures in front of an ordinary webcam, without gloves, sensors, or depth cameras. The system uses MediaPipe Hands to extract and track 21 hand landmarks in real time, OpenCV to capture and process video, and PyAutoGUI to execute commands on the operating system, all in Python.
The project documents the implementation of screen brightness control using the distance between thumb and index finger, with an on-screen bar showing the brightness level, plus a six-gesture dictionary for mouse control (move, left, right, and double click, drag and drop, and a safety gesture). It belongs to the field of human–computer interaction (HCI) and natural user interfaces (NUI), and the reported results are descriptive, from practical trials, without quantitative accuracy or speed measurements.
| Item | Details |
|---|---|
| Project type | Computer-vision system for controlling a computer with hand gestures |
| Field | Human–computer interaction (HCI), computer vision, hand tracking |
| Project status | Working prototype on a personal computer with a webcam, tested with different hand poses |
| Techno Enjaz role | Technical support and assistance to the students during project development |
| Core technologies | Python, MediaPipe Hands, OpenCV, NumPy, PyAutoGUI, Anaconda, VS Code |
| Outputs | Gesture-based brightness control, six-gesture mouse emulation, visual feedback, stop and debug keys |
Most control systems still rely on the mouse and keyboard, which requires the user to be near the device and touch it, limits use in settings that require sterility or free hands, and makes use difficult for some people with motor impairments. At the same time:
The project therefore aimed for a solution that combines tracking accuracy, low cost with an ordinary webcam, and real-time operation on the CPU without a powerful GPU.
The project was carried out academically by the students, with technical assistance and support from Techno Enjaz during development. It is presented among the office's work as an example of its support for computer-vision projects that turn hand tracking into real control commands on a computer.
The system locates the thumb tip and index fingertip, continuously computes the distance between them, and converts it to a brightness percentage sent to the operating system immediately. The screen shows the detected hand model, the computed distance, and a bar indicating the current brightness level.
| Gesture | Finger position | Result |
|---|---|---|
| Lower brightness | Thumb and index close together | A low value in the lower brightness range |
| Raise brightness | Thumb moved away from index | The value updates upward as the hand moves |
| Maximum brightness | Largest possible gap between the two fingers | 100% brightness |
A dictionary of six gestures was designed across the range of hand openness, inspired by familiar everyday signs to reduce cognitive load:
| Gesture | Shape | Function |
|---|---|---|
| Fist | All fingers closed | Continuous cursor movement |
| Left click | Index and middle fingers open (victory sign) | Single click |
| Right click | Three fingers open | Right click |
| Double click | Index finger only | Double click |
| Drag and drop | Thumb and index open as if holding something | Keeps the mouse button pressed for the duration of the gesture |
| Open palm | Four or five fingers open | Safety key that returns the system to a neutral state and cancels any command in progress |
The system uses a four-layer architecture:
| Layer | Components |
|---|---|
| Presentation and interaction | User, webcam, screen |
| Capture and processing | Capture module, color conversion (RGB to HSV), mirroring |
| Analysis and interpretation | Hand tracking with MediaPipe Hands, distance and angle extraction, gesture classification, command mapping |
| Execution | Cursor smoothing filter, input simulation with PyAutoGUI, visual feedback |
System parameters (camera resolution, mouse sensitivity, number of gesture-confirmation frames, detection thresholds) are set from a central configuration file, with diagnostic tools to test the camera, a debug mode showing each finger's state, and the option to build an executable (EXE). The project was developed in Python within an Anaconda environment for package management and VS Code for writing and testing code.
The system was tested in practice with several hand poses in front of the camera. The trials showed successful hand tracking and landmark extraction, and conversion of the thumb–index distance into gradual brightness values from very low levels up to 100%, with good tracking stability and no noticeable delay between hand movement and the brightness update. The conclusion states that the system also executed cursor control, some system commands, and volume control through predefined gestures.
These results are descriptive: the project reports no actual measurements of recognition accuracy, frame rate, or response time. The figures in the analysis (such as 30 frames per second, response time under 150 ms, and accuracy above 90%) are targets, requirements, and expected estimates, not measured results. The virtual-keyboard screenshots in the project come from previous studies reviewed for comparison and are not outputs of this system.
According to the outlook in the project:
The system processes the camera stream locally on the computer without cloud servers, and one of its non-functional requirements is not to store or transmit the live video.
If you are working on a system based on computer vision and gesture control and need technical help developing the idea or building a prototype, you can contact the Techno Enjaz office to discuss your project's requirements and the appropriate scope of support.
It is an academic project built by a team of students with technical assistance from Techno Enjaz that lets users control a computer with hand gestures in front of an ordinary webcam, without a mouse or sensors. It relies on MediaPipe Hands to track 21 hand landmarks, OpenCV for video processing, and PyAutoGUI to execute commands on the operating system.
Each finger's state is computed from the landmarks with geometric rules, smoothed by majority vote over the last 5 frames, and a gesture is only accepted after repeating for 3 consecutive frames. It is then mapped to a command, with exponential cursor smoothing (α = 0.4), movements under 8 pixels ignored, and a 0.3-second click cooldown.
The system controls screen brightness with the thumb–index distance from low up to 100%, with an on-screen brightness bar. It also includes a six-gesture mouse dictionary: fist to move, index and middle for left click, three fingers for right click, index alone for double click, thumb and index for drag and drop, and open palm as a safety key.
No. CNN and LSTM networks appear in the theory chapter as gesture-recognition topics, while the documented implementation relies on geometric rules for finger states. The virtual-keyboard screenshots come from previous studies reviewed for comparison, and integrating deep learning is proposed as future work.
Practical trials showed successful hand tracking and gradual brightness control with good stability and no noticeable delay, but the project reports no actual measurements of accuracy or frame rate. Figures such as 30 FPS and accuracy above 90% are expected targets, and limits include single-hand support and sensitivity to lighting and complex backgrounds.
Reach out to Techno Enjaz engineering team to discuss technical feasibility, prototype hardware, and development scope.
Discover more engineering prototypes and academic implementations
A virtual board for writing, drawing, picking colors and erasing with hand gestures in front of a camera, built in Python with MediaPipe hand tracking and OpenCV.
A prototype that detects people with YOLOv8 and tracks them with DeepSORT in defined work areas, counts presence time in seconds, and exports records to Excel.
An exam-monitoring prototype using YOLO, MediaPipe and OpenCV to analyze a student's head direction, flagging head turns in red for review and forward gaze in green.
