From Hand to Throttle: Building a Gesture- Controlled Drone the Right Way

[vishwa vimukthi gammuduwaththage]

I'm a founder who builds computer vision into physical systems, and gesture control is one of those ideas that looks finished the moment you see hand landmarks tracking on screen, and then isn't. Drawing a skeleton on your hand feels like the hard part. It's not. The hard part is everything between "I can see your hand" and "the aircraft did what you meant, and nothing you didn't."


This piece walks the whole pipeline. If you only want detection, Stage 1 stands alone. If you actually want to fly something, you need all three.

What hand pose estimation actually is​

Human pose estimation detects and tracks the positions of body parts such as joints like elbows, shoulders, knees, in images or video, using computer vision to identify key landmarks and infer posture from them. One clarification worth making, because it trips people up: the model isn't reasoning about anatomy. It was trained on labelled keypoints, and this pixel is a knuckle, that one is a wrist, and it predicts those points directly. The "skeleton" you see drawn on screen is just lines connecting predicted points; there's no bone model underneath.


Hands are an especially good control surface. They're the most articulate part of the body; we already use them for fine, complex tasks, and they can be tracked independently of the rest of you, and you don't need your whole body in frame. MediaPipe's hand model tracks 21 landmarks per hand: knuckles, finger joints, fingertips, and the wrist, which is more than enough resolution to distinguish a fist from an open palm from a pointing finger.


That 21-point vector is the raw material for everything downstream.

Stage 1: Detect the landmarks.​

MediaPipe is Google's open-source framework for building real-time ML pipelines over audio and video. The classic hand-tracking entry point is one pip install away:

pip install mediapipe opencv-python

Here's a clean webcam loop that detects and draws hand landmarks, the original browser/asyncio scaffolding stripped out, because this runs on your machine, not in Pyodide:

import cv2
import mediapipe as mp

mp_hands = mp.solutions.hands
mp_drawing = mp.solutions.drawing_utils

hands = mp_hands.Hands(
    max_num_hands=1,            # one hand for control avoids ambiguity
    min_detection_confidence=0.7,
    min_tracking_confidence=0.5,
)

cap = cv2.VideoCapture(0)

while cap.isOpened():
    ok, frame = cap.read()
    if not ok:
        break

    frame = cv2.flip(frame, 1)  # mirror, so left/right feel natural
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgb)

    if results.multi_hand_landmarks:
        for lm in results.multi_hand_landmarks:
            mp_drawing.draw_landmarks(frame, lm, mp_hands.HAND_CONNECTIONS)

    cv2.imshow("Hand Pose", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()
hands.close()

One important caveat. mp.solutions.hands is the legacy Solutions API. Google ended active support for the legacy MediaPipe Solutions on March 1, 2023, and moved everything to the MediaPipe Tasks API, where the equivalent is the HandLandmarker task loaded from a downloadable .task model file.


The legacy code above still runs; the prebuilt binaries are maintained on an as-is basis, so it's fine for a quick prototype. But if you're starting something you intend to maintain, target mediapipe.tasks.python.vision.HandLandmarker instead. Same 21 landmarks, supported path forward.


Either way, at the end of Stage 1 you have, every frame, a set of 21 (x, y, z) landmarks. That's a picture of a hand. It is not a command. That's the gap the original version of this article, and most others, quietly leave open.

Stage 2: From landmarks to gestures (the part everyone skips).​

A drone doesn't take a point cloud of your knuckles. It takes discrete commands: take off, land, forward, up. So you need a classifier that collapses 21 fuzzy landmarks into one of a handful of clean gesture labels. Two approaches:

Rule-based finger states. The pragmatic starting point. For each finger, decide whether it's extended or curled by comparing the fingertip landmark's position to the joint below it. Count and combine, and you get robust, explainable gestures with zero training:

# landmark indices for the four fingertip points (MediaPipe convention)
TIPS = [8, 12, 16, 20]  # index, middle, ring, pinky

def fingers_up(lm):
    """Return a list of 4 booleans: is each finger extended?"""
    return [lm.landmark[t].y < lm.landmark[t - 2].y for t in TIPS]

def classify(lm):
    up = fingers_up(lm)
    if all(up):            return "OPEN_PALM"   # e.g. take off / hover
    if not any(up):        return "FIST"        # e.g. land
    if up == [True, False, False, False]:  return "POINT"  # e.g. forward
    return "UNKNOWN"

A small neural net. When rules get brittle, such as subtle gestures, rotation, and custom vocabularies that feed the normalized 21-point vector into a small multilayer perceptron and train it on a few hundred labelled examples per gesture. This is exactly what the well-known open-source tello-gesture-control project does (it was featured on Google's own developer blog): MediaPipe keypoints into a lightweight NN that emits a gesture ID.


The non-negotiable bit: debouncing. A raw per-frame classifier will flicker, and one bad frame reads FIST and your drone lands mid-flight. The standard fix is a gesture buffer: only act when the last N frames agree. The community projects converge on exactly this pattern: buffer recent gesture IDs and fire a command only when one clearly dominates and precisely to eliminate false triggers before they reach the aircraft.

from collections import deque, Counter

buffer = deque(maxlen=8)

def stable_gesture(g):
    buffer.append(g)
    label, count = Counter(buffer).most_common(1)[0]
    return label if count >= 6 else None   # 6 of last 8 frames must agree

Only a gesture that survives the buffer earns the right to become a command.

Stage 3: From gestures to drone commands.​

Now, and only now, do we touch the drone. The DJI Tello is the standard learning platform here for good reason: it exposes a rich Python API, so you never have to speak to the flight controller directly. The djitellopy wrapper turns commands into one-liners:

from djitellopy import Tello

tello = Tello()
tello.connect()
print(tello.get_battery(), "% battery")

COMMANDS = {
    "OPEN_PALM": lambda: tello.takeoff(),
    "FIST":      lambda: tello.land(),
    "POINT":     lambda: tello.move_forward(30),   # cm
}

def act(gesture):
    action = COMMANDS.get(gesture)
    if action:
        action()

Wire the three stages together from detect → stable_gesture() → act() — and you have a hand that actually flies a drone, rather than a hand that merely lights up on a screen.


Safety is part of the design, not an afterthought. Every serious version of this includes a manual override. The convention the reference projects settled on is a keyboard fall-through: a key to toggle gesture mode off, dedicated keys for manual flight, and an escape key that lands immediately regardless of what your hands are doing. Build that first. A gesture pipeline with no kill switch is a liability, not a demo.

The Counterargument: "Gesture control is a gimmick."​

It's worth steelmanning the skeptic, because on today's hardware they have a point.


The case against: a physical controller is more precise, has lower latency, and doesn't fail in bad lighting or when your hand leaves frame. Vision adds a whole failure surface, including misclassification, dropped tracking, and variable light, that in exchange for a "look ma, no controller" demo that's strictly worse for actually flying accurately. And the safety story is genuinely harder: a stick has a spring return to neutral; a hand that drifts out of frame has no natural "do nothing" state unless you engineer one.


Where that holds: precision flight, racing, cinematography, anything where milliseconds and centimetres matter. Nobody should fly a mapping mission by waving.


Where it breaks down: the value of gesture control was never precision, and it's hands-free, equipment-free intent in contexts where a controller is impractical or unavailable. A field technician whose hands are busy. A first responder directing a drone while carrying gear. An inspection or search task where you want to redirect an aircraft with a glance and a gesture, not by looking down at sticks.


This is really a human-machine interface question, and the same logic that's pushing HMIs toward more natural, embodied inputs applies here: the controller isn't going away, but it stops being the only way in. Frame it as one input modality among several, with a hard manual fallback, and the gimmick objection mostly dissolves.

Where this goes next​

The interesting extensions all live above Stage 3: motion gestures (trajectories, not static poses) for richer vocabularies; running detection on the drone's own camera feed rather than a laptop webcam, so the operator doesn't need to be at a base station; and fusing gesture intent with onboard autonomy so a gesture sets a goal and the drone figures out the path itself, which connects directly to the perception-and-navigation stack I've written about in VSLAM for autonomous UAVs, and to the hands-on control work in my RC drone reverse-engineering series.

The Takeaway​

Hand landmark detection is a solved, one-import problem. That's exactly why it's misleading: the tracking overlay looks like the finished product when it's really the first of three stages. The engineering that matters - turning noisy landmarks into stable gestures, debouncing them, mapping them to commands, and wrapping the whole thing in a safety layer, and it is invisible in a screenshot but is the entire difference between a cool visualisation and a drone that flies. Build all three stages, put the kill switch in first, and treat gesture control as one modality rather than a replacement for the sticks