Question 3 of 4: Measures for Human Factors — Drone Operation

Question 3: Measures for Human Factors — Drone Operation (20 marks: a–10, b–10)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Part (a) — Metrics and Their Purpose

The scenario names three distinct performance demands – timely maneuvering, low error rate, and accurate/timely response to in-flight direction – so the measurement strategy below covers speed, accuracy and cognitive-load dimensions rather than a single catch-all score, since a drone operator who is fast but error-prone, or accurate but overloaded, both represent real failure modes for the production company.

• Task completion time / response latency (time from a called direction or an on-screen hazard appearing to the corrective control input). Useful to the operator as direct, actionable training feedback (are reaction times improving with practice), and to the production company as the metric that most directly predicts costly re-shoots – the scenario states delays cause additional filming expense, so this is the metric tied most tightly to cost.
• Error rate (near-misses with birds/objects, hard/unstable landings, off-target framing requiring a re-take). Useful to the operator to identify specific recurring failure patterns (e.g. consistently late on left-side obstacles) for targeted retraining, and to the company as a safety/quality/insurance metric and as an objective basis for operator certification or currency requirements.
• Control smoothness / input variability (joystick reversal rate, over-correction magnitude, jerk in the commanded flight path). Useful to the operator as an early indicator of a still-developing skill (novices show more corrective reversals than experts for the same manoeuvre) before it shows up as an outright error, and to the company as a leading indicator for training progression decisions (ready for solo/live shoots vs. needs more supervised practice).
• Situation awareness (SA) (accuracy recalling/predicting drone position, nearby hazards and remaining battery/flight-time when probed mid-task). Useful to the operator because SA loss is a leading precursor to the exact hazards named in the scenario (birds, moving objects, uneven landing surfaces), and to the company as a predictor of near-miss risk before an actual incident occurs.
• Subjective mental workload (e.g. NASA-TLX rating taken immediately after each flight). Useful to the operator/trainer to distinguish "slow because of skill" from "slow because the task combination (filming direction + obstacle avoidance + framing) is overloading" – two different training fixes – and to the company for scheduling decisions (shift length, task pairing) that keep workload sustainable across a filming day.

Part (b) — Measurement Methods and Hardware/Software

• Task completion time / latency: logged automatically from the drone's own flight-controller telemetry (timestamped GPS/IMU stream and command log, standard on commercial platforms such as the pictured Yuneec controller) cross-referenced against timestamped verbal cues from a synced production-audio or scenario-marker log; analysed offline with simple timestamp-differencing scripts – no added hardware beyond what the drone/controller already record.
• Error rate: a combination of the same telemetry log (automatically flags altitude/attitude excursions, hard-landing accelerometer spikes) and human-rated video review of the recorded camera feed by a trained observer using a structured checklist/scoring rubric (near-miss, off-target framing, unstable landing), since some errors, e.g. "wrong shot framing," are a judgment call telemetry alone cannot capture.
• Control smoothness: computed directly from the joystick/controller's own raw input log (sampling rate, reversal count, input-derivative/jerk) – a software-only metric requiring no extra hardware, calculable with the same logging software used for latency.
• Situation awareness: the SAGAT method – periodically freezing a training simulation and querying the operator on drone position/heading/remaining battery/known hazards, scored against ground truth – requires a drone flight simulator (safer and repeatable for probe-based testing than pausing a live flight) with a built-in query/scoring module; for live/field assessment, a structured post-flight recall interview is a lower-fidelity substitute.
• Subjective workload: a paper or tablet-based NASA-TLX questionnaire administered immediately post-flight (six-dimension rating: mental demand, physical demand, temporal demand, performance, effort, frustration), requiring no special hardware beyond a tablet/form and standard scoring software; can be supplemented with a wearable heart-rate-variability monitor for a continuous physiological workload proxy during flight if a fully instrumented assessment is warranted.

Together these metrics let the training program separate "needs more repetitions" (smoothness/error-rate trending down with practice) from "task design is the problem" (workload consistently high, SA consistently low regardless of experience) – the latter would point the company toward redesigning the task (e.g. a second crew member handling framing/direction while the pilot flies) rather than more individual training.