A smart board turning a webcam into an air writing and drawing space via MediaPipe Hands and OpenCV gestures, solving handwritten equations step by step with Gemini.
The Smart Board is an interactive system that turns an ordinary webcam into an input tool for writing and drawing in the air with a finger inside a digital workspace, with no pen, colored marker, or extra hardware. It was carried out by a team of students with technical assistance from Techno Enjaz, in Python, using MediaPipe Hands to track 21 hand landmarks and OpenCV for video processing and the interface.
The system recognizes gestures from the number of raised fingers and the geometric distances between them: the index finger to write, two fingers to choose colors and tools, and the thumb to save a snapshot of the board. It adds an educational layer by sending a handwritten math equation to the Google Gemini API, which returns the solution with its steps inside the same interface, making it a low-cost alternative to smart boards and digital drawing tablets that runs on mid-range computers without a GPU.
| Item | Details |
|---|---|
| Project type | Interactive system for writing and drawing in the air with hand gestures |
| Field | Computer vision, human–computer interaction, hand tracking, educational technology |
| Project status | Working system tested in practice with a webcam on a mid-range computer |
| Techno Enjaz role | Technical support and assistance to the students during project development |
| Core technologies | Python, MediaPipe Hands, OpenCV, NumPy, PIL, threading, Google Gemini API |
| Outputs | Finger drawing and writing, color selection, eraser and full clear, snapshot saving, solving handwritten equations with steps |
Human–computer interfaces are moving toward motion and gestures instead of traditional input devices, but well-known interactive teaching tools such as smart boards and digital drawing tablets are expensive and not widespread in schools with modest resources, and they come with their own physical constraints.
The project reviewed earlier approaches and found limits in each: Air Canvas systems that track a colored pen by converting RGB to HSV are sensitive to lighting and need an external marker; the MediaPipe Hands framework offers accurate tracking but is a developer framework rather than a complete educational application; gesture-based virtual-mouse apps lack the path stability needed for writing; and depth-sensor systems such as Kinect are accurate but need costly hardware. The Smart Board was designed to combine accurate hand tracking, stable strokes for writing and drawing, and clear gesture logic, using only an ordinary camera.
The project was carried out academically by the students, with technical assistance and support from Techno Enjaz during development. It is presented among the office's work as an example of its support for projects that combine computer vision, gesture interaction, and generative AI in educational applications.
| Gesture | Function |
|---|---|
| Index finger only raised | Writing and drawing mode on the board |
| Two fingers raised | Selection mode: drawing stops and colors or tools can be chosen from the interface |
| Thumb only raised | Special commands such as saving a snapshot of the board |
Gestures are distinguished by the number of raised fingers and by the distances between fingertips, or between them and the wrist. Where gestures looked alike because of hand angle or overlapping fingers, extra conditions were added to reduce unintended mode switches. The gestures are not fixed; they can be changed or replaced by editing the program's conditions. The interface tools include changing the drawing color, an eraser, and a full clear.
Gemini is used as a dedicated engine for analyzing and solving handwritten equations, not as a general text assistant:
| Library | Role in the system |
|---|---|
| OpenCV (cv2) | Capturing the camera feed, displaying frames, drawing lines, buttons, and text |
| MediaPipe | Extracting hand landmark coordinates with a ready, pre-trained model |
| NumPy and math | Handling images as arrays, cropping screen regions, geometric calculations between coordinates |
| threading | Running the model call in the background without stopping the feed |
| requests, json, and base64 | Sending requests to Gemini, structuring data, and encoding the image for transmission |
| PIL, os, and dotenv | Rendering text with special encodings, managing files, keeping the API key outside the source code |
The project uses the ready-made MediaPipe Hands model without retraining or changing its weights; its output is adapted through mathematical conditions and geometric relationships between landmarks, so no additional training data was needed.
The practical implementation showed that the system:
The book does not report quantitative measurements such as frames per second, gesture-recognition accuracy, the share of equations solved correctly, or the specifications of the test machine.
According to the outlook in the project:
If you are working on an interactive system based on computer vision and tracking and need technical help developing the idea or building a prototype, you can contact the Techno Enjaz office to discuss your project's requirements and the appropriate scope of support.
It is an interactive system that turns an ordinary webcam into an input tool for writing and drawing with a finger in the air, with no pen or extra hardware, built by a team of students with technical assistance from Techno Enjaz. It aims to offer a low-cost alternative to smart boards and digital drawing tablets in educational settings.
The ready-made MediaPipe Hands model extracts the coordinates of 21 hand landmarks, and the system identifies the gesture from the number of raised fingers and the distances between fingertips and the wrist. A raised index finger means writing, two fingers select colors and tools, and a raised thumb saves a snapshot of the board.
The user selects the equation and confirms sending; the system crops it, converts it to grayscale with inverted colors and sends it to the Google Gemini API with math-only instructions. The model returns the solution with its steps, the system shows a warning instead if the equation is unclear, and the call runs in a separate thread so the camera feed does not freeze.
The system is built in Python with MediaPipe Hands for hand tracking, OpenCV for video capture, drawing and the interface, NumPy and math for geometric calculations, threading for parallel processing, requests, json and base64 for calling Gemini, and PIL, os and dotenv for text rendering, file management and protecting the API key.
The system ran in real time with a webcam on a mid-range computer without a GPU, but the book reports no quantitative accuracy or frame-rate figures. It tracks one hand only and is affected by landmark jitter, lighting, finger occlusion and lack of depth, and some symbols and fractions in air-written equations can be misread.
Reach out to Techno Enjaz engineering team to discuss technical feasibility, prototype hardware, and development scope.
Discover more engineering prototypes and academic implementations
A virtual board for writing, drawing, picking colors and erasing with hand gestures in front of a camera, built in Python with MediaPipe hand tracking and OpenCV.
A prototype that detects people with YOLOv8 and tracks them with DeepSORT in defined work areas, counts presence time in seconds, and exports records to Excel.
An exam-monitoring prototype using YOLO, MediaPipe and OpenCV to analyze a student's head direction, flagging head turns in red for review and forward gaze in green.
