* feat(motor_failure_detector): add current-residual detection library
Per-motor model-based motor-failure detector for multirotors. Predicts the
expected ESC current from the commanded actuator signal
(I_exp = current_slope_a*u + current_idle_a) and trips on a low-pass-filtered
residual held past a trip band for a persistence time. The command (not measured
rpm) is the fault-independent reference: a dead motor that draws no current while
still commanded produces a large residual, whereas a current-vs-rpm model would
mask a clean stop.
Band and persistence are configurable per residual direction. Undercurrent is the
fault-consistent side -- an open circuit, a shed propeller or a desync makes a
motor draw less than commanded -- while overcurrent covers a winding short,
mechanical jam or demagnetization. They are separate because the benign residual
is far from symmetric: a fast throttle increase draws acceleration current the
static model cannot predict, so overcurrent excursions are large and unbounded,
whereas an undercurrent residual is floored at -I_exp. A single band sized to
reject the overcurrent transients can therefore end up wider than the largest
undercurrent fault a motor can physically produce, which makes a full current
collapse undetectable rather than merely slow. Set undercurrent_threshold_a <= 0
to keep one symmetric band on the residual magnitude.
Built on PX4 primitives rather than hand-rolled equivalents: update() takes a
timestamp (esc_status.timestamp in flight, the log timestamp in replay) and
derives dt internally; the residual filter is a mathlib AlphaFilter; the
persistence debounce is a systemlib::Hysteresis per direction (with a sticky latch
on top, and reset across data gaps so a dropout can't complete a pending trip).
The data-gap threshold (kMaxGap = 0.3 s) is sized above the ~10 Hz arming-check
feed interval and below the 400 ms ESC-offline timeout: a live feed is never
mistaken for a gap (a smaller value equal to the feed interval reset the filter
and debounce on the majority of jittered ticks, so a sustained fault never
survived the persistence window -- offline replay: 0/100 sampling phases latched
vs 100/100 gap-free), while a genuine outage still resets and is handled by the
separate ESC liveness guard.
Includes an 18-case gtest spanning the failure-mode spectrum and edge cases:
healthy-never-trips (incl. high throttle), open-circuit and overcurrent trips,
severe prop-loss undercurrent, the missed ~50% partial loss (documents the
detection floor), the per-direction bands (a tighter undercurrent band catching a
collapse one symmetric band misses, band independence, per-side persistence),
persistence (short spike crosses-threshold-but-recovers), gap reset, the 10 Hz
feed vs. real-gap boundary, dropout-holds, multi-motor, the exclusion gate,
monitor-only, and reset/latch.
* refactor(motor_failure_detector): resolve the undercurrent band at configure time
An unset undercurrent threshold now mirrors the overcurrent band in
configure(), so update() runs the same two bands in every configuration and
no longer carries a second threshold test that reads like a repeat of the
disable gate. A symmetric band is the two-band case with equal bands.
Also name esc_status CONNECTED_ESC_MAX at kMaxMotors so the constant is
findable, use the PX4 time literals for the persistence conversions, and
drop the excluded temporary.
* feat(commander): use current-residual detector for the ESC motor-failure check
Replace the flat, instantaneous current check in EscChecks::checkMotorStatus with
the MotorFailureDetector library (added in the parent PR). Per motor it predicts
the expected ESC current from the commanded thrust
(I_exp = MOTFAIL_C2T*u + MOTFAIL_IDLE) and trips on the low-pass-filtered
residual, with an independent band and hold time per direction: MOTFAIL_UNDER held
for MOTFAIL_UND_TIME below the expectation, MOTFAIL_OVER held for MOTFAIL_OVR_TIME
above it.
Because the two directions need separate bands, the single band and hold time are
renamed to name the direction they act on: MOTFAIL_OFF -> MOTFAIL_OVER and
MOTFAIL_TIME -> MOTFAIL_OVR_TIME, migrated in param_translation.cpp. The old band
applied to both directions, so it carries over to the overcurrent side, and
MOTFAIL_UNDER defaults to 0, which keeps a single symmetric band -- an existing
vehicle behaves as before. MOTFAIL_IDLE also defaults to 0, and the whole check
stays gated by FD_ACT_EN (off by default).
The detector is driven off esc_status.timestamp, so a telemetry stall freezes the
filter instead of integrating stale current; the separate 400 ms ESC telemetry
timeout stays in checkEscOnline(). It is reconfigured from parameters while
disarmed, so a parameter change never resets a latched failure mid-flight. The
motor_failure_mask still feeds failure_detector_status via
FailureDetector::publishStatus, leaving the ControlAllocator reaction path
untouched.
The mask reports the detector's sticky latch alone, not whether the motor is
currently evaluable. That preserves the invariant the previous check spelled out
-- "Only set, never clear mid-air: stopping the motor in response could make it
appear healthy again" -- because handling a failure makes exactly that motor
unevaluable: the control allocator drops it, which publishes a NAN command,
which excludes it from the detector. Re-deriving the mask from the exclusion
state clears it, the allocator restores all motors, and the next check re-trips.
Measured in SIH on a hexarotor with a missing propeller: 83 mask transitions in
nine seconds, with a reversed recovery motor swinging between idle and near-full
reverse at the 10 Hz check rate. An ESC going offline after a real failure
retracts the report the same way. The latch is only ever set on a tick where the
motor was evaluable, so reporting it alone cannot report a motor that was never
judged.
* feat(motor_failure_detector): add offline ulog replay tool
Add a host tool that replays a recorded .ulg through the real MotorFailureDetector
and prints the per-motor trip verdict, for validating and tuning the detector
offline (EKF2-replay style: the real library drives recorded data, not a
re-implementation).
Built on ulog_cpp (PX4's C++ ULog library, added as a git submodule and built for
the posix host target only), so it reads ESC current, motor command and arming
state by field name from the log's own format -- one binary replays logs of any
PX4 message version with no rebuild. The feed mirrors the runtime in escCheck:
evaluated only while armed, at the runtime's 10 Hz cadence, with the per-ESC
current-report startup latch and reset-on-disarm. Config comes from the log's
MOTFAIL_* parameters with a built-in calibration fallback, and --set
MOTFAIL_NAME=value overrides any knob for what-if retuning. Because the runtime's
exact 10 Hz sample phase cannot be reconstructed offline, a single pass over the log
drives several detector instances on staggered grid offsets and reports a per-motor
peak band and trip fraction rather than a single phase's verdict, flagging when the
result is phase-dependent.
batch_replay.sh runs the tool over a folder of logs and summarizes each verdict
(OK / FAIL / SKIP) from its exit code.
Off by default so it never enters the firmware build or CI: it is a posix-only,
EXCLUDE_FROM_ALL host executable gated behind the MFD_REPLAY_TOOL cache option.
Build it with:
cmake -DMFD_REPLAY_TOOL=ON build/px4_sitl_default
ninja -C build/px4_sitl_default mfd_replay
* docs(motor_failure_detector): document the current-residual motor-failure check
Add a configuration page for motor failure detection, next to the other
actuator topics, and a developer page for the offline replay tool:
- what the check compares and why the under/overcurrent bands are separate
- the new MOTFAIL_* parameters, including the rename migration
- how to fit the current model and pick the bands from the vehicle's own
logs with mfd_replay / batch_replay.sh
The "Motor Failure Trigger" section in the failsafe page described the
pre-rename parameters, the unfiltered symmetric comparison and claimed the
failure was non-latching, so it is reduced to a summary of the trigger and
its action, pointing at the new page for the thresholds.
* docs(docs): Subedit for simpler introductions and terminology
* docs(docs): Improve upgrading from v1.18 section
* feat(motor_failure_detector): add current-model fitting tool
mfd_fit derives MOTFAIL_C2T and MOTFAIL_IDLE from recorded logs, so the model
and the thresholds are taken from the same data.
Like mfd_replay it reads ESC current, motor command and arming state through
ulog_cpp by field name, so one binary handles any log version. Each log is a
single streaming pass that accumulates only the sums a least-squares line needs,
so memory does not grow with the log or with the number of logs, and a whole
corpus can be fitted in one invocation.
Alongside the parameters it reports the residual spread the trip bands have to
clear, and each motor's bias against the shared model.
* docs(docs): document motor failure detection tuning and the fitting tool
Follow-up to review of the detection and replay pages:
- fix the link left behind by the Failure Response -> Failure Action rename
- drop the Multicopter badge: the check has no vehicle-type gate, so a VTOL's
multirotor motors and its forward-flight motor are both checked
- describe the model-fitting step without the statistics jargon
- state that only healthy logs may be used, for the fit and the thresholds
- document mfd_fit, and reduce the release note to a pointer
* feat(motor_failure_detector): report residual extremes and add a band sweep
The replay tool printed the largest residual magnitude, which hid whether an
excursion was above or below the model. Since the two trip bands are chosen per
direction, that left the reader running one band at a time to find out. It now
prints the signed minimum and maximum instead, so each band's headroom is
readable straight from the table.
Add sweep_matrix.sh alongside batch_replay.sh: it sweeps one band against its
hold time over a set of logs and prints how many logs false-trip at each pair,
so the lowest usable band per hold time is the first zero in a row.
Both tools now number motors from 1, matching the "Motor N ..." the vehicle
reports, rather than from 0.
* docs(docs): rework the motor failure calibration walkthrough
Follow a real calibration through both pages and fix what it ran into:
- the replay page's worked example is one vehicle end to end, with every
number and pasted output taken from that corpus
- the fitting tool, the single-log run and the band sweeps are ordered the way
the work is actually done, with replaying a whole fleet last
- the sweeps are shown as a matrix of false trips per (hold time, band), and
the grids are derived from the residual extremes rather than guessed
- commands match the output they produce, and say what they need: mfd_replay
without --set uses a fallback calibration that belongs to another vehicle
- vehicle_status is optional, not required, for the replay
- drop the repetition: batch_replay.sh, the hover-current ceiling and the
phase-sensitivity note were each explained two or three times
* fix(motor_failure_detector): separate configure and reset with the appropriate triggers
* refactor(motor_failure_detector): clarify leading edge hysteresis as trigger time compared to "persistence"
* fix(escCheck): encapsulate _motor_failure_direction handling into MotorFailureDetector
* fix(astyle): exclude ulog_cpp submodule form check
* fix(motor_failure_detector_tools): fail log replay when no log is present
* refactor(motor_failure_detector): Improve documentation
* fix(escCheck): check load per ESC not per motor function
This was already done in my original implementation. It supports checking multiple ESCs mapped to the same motor function individually. It's also consistent with other ESC checks that run for every ESC that reports telemetry.
* fix(motor_failure_detector): run update() per motor and consider individual telemetry timestamps
* fix(motor_failure_detector): simplify output interface to getters of actually used results
* fix(motor_failure_detector): inline what doesn't need to be separate
* fix(motor_failure_detector): make the bands independent and drop special cases
- Each trip band is now independent: a threshold <= 0 disables only that band. The undercurrent band no longer copies the overcurrent one.
- Drop the MOTFAIL_OFF/MOTFAIL_TIME parameter migration. The new parameters start at their defaults.
- Treat a non-finite current like a non-finite command (not evaluated, filter and debounce reset). No driver reports NaN current, so the path only existed for its test.
- Compute the residual as max(current, 0) - I_expected. Same as before, but written such that it's obvious: regeneration counts as 0A.
- Drop unnecessary trigger time parameter clamp to >= 0 before converting to unsigned microseconds, metadate already decalres that as out of range.
- Count each ESC sample once by its own timestamp in mfd_fit, like the motor_failure_detector.
* fix(motor_failure_detector): check all still considered online ESC telemetry
* fix(motor_failure_detector): declare all link dependencies
* fix(motor_failure_detector_tools): reuse log reading logic to reduce duplication
* fix(motor_failure_detector): replace reversible flag getting passed into the detector for separate handling
Having a NAN command already handles that.
* test(escChecks): add unit tests for load based failure detection
---------
Co-authored-by: Hamish Willee <hamishwillee@gmail.com>
Co-authored-by: Matthias Grob <maetugr@gmail.com>
Co-authored-by: Andrea Bernasconi <andrea.bernasconi@auterion.com>
21 KiB
Offline Motor Failure Replay
Two command line tools calibrate the motor failure detector against a vehicle's own flight logs.
mfd_fit derives the current model from a set of logs, and mfd_replay runs the real detector over a log and prints what it would have decided for each motor.
They exist because the threshold parameters (MOTFAIL_*) are specific to an airframe, so the only place to get them is that airframe's own flights.
mfd_replay links the detector library itself, so a replay runs the same code that runs in flight.
One flight is rarely enough to characterise a vehicle, so usually we run the tools over a directory of logs.
The tools are used to:
- Confirm a candidate configuration never trips on a set of healthy flights.
- Find the tightest thresholds a vehicle's data supports, by sweeping values and re-running.
- Check after the fact whether the detector would have flagged a motor in a flight where something went wrong.
Building
To build the tools:
-
Build PX4 SITL (to populate the build directory)
make px4_sitl_default # once, if the build directory does not exist yet -
Build the tool binaries:
cmake -DMFD_REPLAY_TOOL=ON build/px4_sitl_default ninja -C build/px4_sitl_default mfd_replay mfd_fit
This produces build/px4_sitl_default/mfd_replay and build/px4_sitl_default/mfd_fit.
mfd_fit derives the current model from logs; mfd_replay then runs the detector with it.
Fitting the Current Model
The first step is to fit the current model: find the straight line that best predicts each motor's current from its command, since the detector flags a motor whose current strays too far from it.
To do this, run mfd_fit once over every healthy log you have for the airframe.
It fits the expected current I_expected = MOTFAIL_C2T*u + MOTFAIL_IDLE to all of them together, where u is the motor command, and prints the two parameter values plus the residual sigma you need later to choose the bands.
The more of the vehicle's flying the logs cover, the better the fit, and there is no limit on how many you pass: a hundred logs take under a second.
::: warning Fit on healthy flights only. The fit cannot tell a faulty motor from a healthy one, so a single failure-test log pulls the model toward that fault and shifts the residual of every healthy motor with it. :::
./build/px4_sitl_default/mfd_fit ~/logs/healthy/*.ulg
Typical output, for three healthy flights:
# 3 logs, 80697 armed samples
# I_expected = 6.91*u + 0.37 A (residual sigma 0.50 A)
# mean command 0.387, mean current 3.04 A
MOTFAIL_C2T 6.91
MOTFAIL_IDLE 0.37
# per-motor mean residual against this model [A]
# motor 1: -0.12 (sigma 0.64, n 13484)
# ...
# motor 6: -0.10 (sigma 0.46, n 13360)
# worst per-motor bias 0.12 A -- the trip bands have to clear this before any fault.
It contains:
- The fitted model,
I_expected = 6.91*u + 0.37 A.6.91is the slope (expected current per unit of command, in A) and0.37is the offset (expected current at zero command, in A). - The same slope and offset as the two parameters to set, MOTFAIL_C2T
6.91and MOTFAIL_IDLE0.37. - The residual sigma,
0.50 A: how far the samples scatter around the model. This is the healthy noise the trip bands have to clear, so it caps how tight they can be. - For each motor, its mean residual against the model (
-0.12 Afor motor 1), with that motor's own sigma and sample countn. - The worst per-motor bias,
0.12 A: the largest of the per-motor mean residuals, ignoring sign. The detector uses one model for all motors, so this is error the bands absorb before any fault does.
To use it:
-
Set
MOTFAIL_C2TandMOTFAIL_IDLEto the printed values.On some vehicles the fitted offset comes out below zero.
MOTFAIL_IDLEcannot be set below 0, so the tool adds aNOTEsaying so: use 0 in that case. Nothing is lost by it, because the bands are then chosen against the residual that results, so they absorb the difference. -
Keep the residual sigma for Choosing the Bands.
Use the fitted offset, not one copied from another airframe. An offset that is too high by some amount lowers every healthy motor's residual by that same amount, which the undercurrent band then has to absorb. For example, replaying the healthy flight shown in Running the Detector on One Log with another airframe's offset of 2.88 A, instead of the fitted 0.37 A, raises the expected current by 2.51 A. Every residual drops by the same 2.51 A (motor 1's lowest goes from -1.79 A to -4.30 A), past the 2.25 A undercurrent band, so all six healthy motors are reported as failed:
# config (...): I_exp = 6.91*u + 2.88 A, trip if LPF(I-I_exp) < -2.25 for 0.70 s or > 1.00 for 2.50 s
# motor min[A] max[A] trips verdict
1 -4.30 +1.69 5/5 FAILED (all phases)
...
6 -4.75 +1.43 5/5 FAILED (all phases)
Running the Detector on One Log
./build/px4_sitl_default/mfd_replay flight.ulg \
--set MOTFAIL_C2T=6.91 --set MOTFAIL_IDLE=0.37 \
--set MOTFAIL_UNDER=2.25 --set MOTFAIL_UND_TIME=0.7 \
--set MOTFAIL_OVER=1.0 --set MOTFAIL_OVR_TIME=2.5
Run without the --set flags it uses a built-in fallback calibration, which belongs to a different vehicle and will report failures on healthy flights.
The log must contain esc_status (per-motor current) and actuator_motors (the motor commands).
vehicle_status is used to evaluate only the armed part of the flight; without it the tool says so and evaluates everything.
Fields are resolved by name from the log's own format, so one binary reads logs from any PX4 version, with 8 or 12 ESCs, without rebuilding.
A log that is not a ULog, or that is missing a needed field, is reported as an error rather than silently misread.
Output for a healthy flight with a calibrated configuration:
# flight.ulg (5 phases, 728 armed samples each)
# config (built-in calibration, overrides: MOTFAIL_C2T MOTFAIL_IDLE MOTFAIL_OVER MOTFAIL_OVR_TIME MOTFAIL_UNDER MOTFAIL_UND_TIME): I_exp = 6.91*u + 0.37 A, trip if LPF(I-I_exp) < -2.25 for 0.70 s or > 1.00 for 2.50 s
# motor min[A] max[A] trips verdict
1 -1.79 +4.20 0/5 ok
2 -3.09 +2.11 0/5 ok
3 -1.79 +2.67 0/5 ok
4 -2.13 +2.38 0/5 ok
5 -2.35 +2.78 0/5 ok
6 -2.24 +3.94 0/5 ok
| Column | Meaning |
|---|---|
motor |
Motor index, counting from 1, as mapped from each ESC's actuator function. |
min[A] |
Deepest the filtered residual went below the model, over the whole flight and every sampling phase. This is what the undercurrent band has to clear. |
max[A] |
Highest it went above the model, likewise. This is what the overcurrent band has to clear. |
trips |
How many of the sampling phases flagged this motor. |
verdict |
ok, FAILED (all phases), or FAILED (sampling-dependent!) when only some phases flagged it. |
Motors are numbered from 1, the same as in the Motor 1 undercurrent detected message the vehicle reports.
Exit codes make the tool usable as a check in a script: 0 if every motor stayed healthy, 1 if any motor was flagged, 2 if the log could not be used.
The Sampling Phase Sweep
Every replay runs five detectors side by side over the same log (the 5 phases in the header and the /5 in trips), because a single one can miss a trip that would happen on the vehicle.
On the vehicle the check runs at 10 Hz, so it sees only a subset of the ESC samples, and which subset depends on the alignment between the check and the telemetry stream. That alignment cannot be reconstructed from a log.
Each detector therefore starts its 10 Hz grid at a different offset, and the tool reports the spread. A single-phase answer would be a lower bound on the trip risk, not a verdict.
Read the result as follows:
0/5: the configuration did not trip on this flight, andmin/maxsay by how much it cleared each band.0/5but with a note printed below the table: one of the extremes moved by more than 30% between phases, so this run happened not to trip and the same flight could trip on the vehicle.1/5…4/5(sampling-dependent!): treat this as a false positive, not as a pass.5/5: it would have tripped.
Where the Configuration Comes From
The configuration in the # config header line is the set of MOTFAIL_* values the replay ran the detector with.
For each parameter, in order of precedence:
--set NAME=valueon the command line.- The value recorded in the log, if the log contains the
MOTFAIL_*parameters. - A built-in fallback.
The header line always prints the configuration that was used, and names any parameter that came from --set.
::: warning The built-in fallback is an example fit for one particular vehicle, not a recommended default. On another airframe it will report failures on perfectly healthy flights. Always replay with the configuration you are actually evaluating. :::
Overriding one knob at a time is how the band sweeps below are done:
./build/px4_sitl_default/mfd_replay flight.ulg --set MOTFAIL_UNDER=1.75 --set MOTFAIL_UND_TIME=0.7
The accepted names are MOTFAIL_C2T, MOTFAIL_IDLE, MOTFAIL_UNDER, MOTFAIL_UND_TIME, MOTFAIL_OVER and MOTFAIL_OVR_TIME.
Choosing the Bands
After fitting the current model you have MOTFAIL_C2T, MOTFAIL_IDLE and the residual sigma, which you use with mfd_replay to choose the remaining four parameters: the undercurrent and overcurrent bands and their trigger times.
The goal is the tightest bands that never trip on healthy flight, so the logs have to cover the throttle range the vehicle really flies, including climbs and aggressive manoeuvres: the largest deviation across them is what sets the achievable threshold. As with the fit, every log has to come from a flight where all motors were healthy.
The steps below use batch_replay.sh to replay a whole set of logs at once.
1. Read the healthy residual.
Replay each log with the fitted model and both bands disabled (0), so nothing trips and the min/max columns report the full excursion.
Each log prints its own header and table; one of them:
for log in ~/logs/healthy/*.ulg; do
./build/px4_sitl_default/mfd_replay "$log" \
--set MOTFAIL_C2T=6.91 --set MOTFAIL_IDLE=0.37 --set MOTFAIL_OVER=0 --set MOTFAIL_UNDER=0
done
# motor min[A] max[A] trips verdict
1 -1.36 +7.84 0/5 ok
2 -1.32 +2.72 0/5 ok
3 -1.85 +1.93 0/5 ok
4 -1.37 +7.72 0/5 ok
5 -2.17 +1.11 0/5 ok
6 -1.55 +1.96 0/5 ok
Read the two columns separately, because they set the two sweeps.
The deepest min across all logs and motors is what the undercurrent band has to clear, and the highest max is what the overcurrent band has to clear.
On this vehicle they are very different, about −2.2 A against +7.8 A, which is the whole reason for two bands.
2. Sweep each band against its trigger time.
sweep_matrix.sh sweeps one direction while disabling the other, and prints
how many logs false-trip at each (trigger time, band) pair.
The first 0 in a row is the lowest band that trigger time supports.
export C2T=6.91 IDLE=0.37 # the model you fitted
src/lib/motor_failure_detector/tools/sweep_matrix.sh ~/logs/healthy under
src/lib/motor_failure_detector/tools/sweep_matrix.sh ~/logs/healthy over
Set the grid from the min/max you just read, with TRIGGER_TIMES and BANDS:
- Bands run from the healthy noise up to that side's extreme.
Roughly twice the residual sigma makes a sensible bottom, since below that every log trips.
On the undercurrent side the top is the expected current at hover: a band above it cannot be reached even by a rotor that has stopped completely.
The overcurrent side has no such ceiling, so start at its
maxand come down by spending trigger time instead of amplitude. - Trigger times come from the latency you can accept.
A latch takes about
0.1 sof sampling plus0.2 sof filter plus the trigger time, so a fault that destabilises the vehicle wants well under a second, while a thermal fault can wait seconds.
TRIGGER_TIMES="0.1 0.3 0.5 0.7 1.0" BANDS="1.0 1.25 1.5 1.75 2.0 2.5 3.0" \
src/lib/motor_failure_detector/tools/sweep_matrix.sh ~/logs/healthy under
trigger[s] 1.0 1.25 1.5 1.75 2.0 2.5 3.0 <- MOTFAIL_UNDER [A] (logs tripped)
0.1 3 2 2 2 2 1 0
0.3 2 2 2 2 1 1 0
0.5 2 2 1 1 1 0 0
0.7 2 1 1 0 0 0 0
1.0 1 1 1 0 0 0 0
Each row gives one answer.
Walk it from left to right and stop at the first 0:
| trigger time | lowest band with no trips |
|---|---|
| 0.1 s | 3.0 A |
| 0.3 s | 3.0 A |
| 0.5 s | 2.5 A |
| 0.7 s | 1.75 A |
| 1.0 s | 1.75 A |
Now compare each floor against this side's ceiling, the expected current at hover, which is about 3.0 A here.
That makes the top two rows unusable rather than merely cautious: the smallest band that avoids false positives is already the band a total failure at hover cannot reach. The 0.5 s row sits just under the ceiling, which leaves almost nothing for a partial loss. By 0.7 s the floor has dropped to 1.75 A, which leaves real room below the ceiling, and a longer trigger time gains nothing. That is what picks 0.7 s here, rather than taste.
Sweep coarse, then refine. The first grid only has to show which zone is interesting; a second, finer pass finds the edge.
The overcurrent side is the example.
Run on the default grid, sweep_matrix.sh ~/logs/healthy over puts every zero in the 2.5 s row, which says the floor is 0.5 A or lower and the trigger time that reaches it is 2.5 s or shorter, but not what either actually is.
Re-running with finer trigger times and smaller bands:
TRIGGER_TIMES="1.5 1.75 2.0 2.25 2.5" BANDS="0.2 0.3 0.5 0.75 1.0 1.5 2.0 3.0" \
src/lib/motor_failure_detector/tools/sweep_matrix.sh ~/logs/healthy over
trigger[s] 0.2 0.3 0.5 0.75 1.0 1.5 2.0 3.0 <- MOTFAIL_OVER [A] (logs tripped)
1.5 3 3 2 1 1 1 1 1
1.75 2 1 0 0 0 0 0 0
2.0 2 1 0 0 0 0 0 0
2.25 2 1 0 0 0 0 0 0
2.5 2 1 0 0 0 0 0 0
Two things are readable now. The floor is 0.5 A, since 0.3 A still trips a log. And it arrives abruptly between 1.5 s and 1.75 s: at 1.5 s even a 3.0 A band trips, at 1.75 s a 0.5 A band is clean, and every longer trigger time is identical.
That second part is physics rather than a coincidence of the grid. The benign excursion that sets this side must last between 1.5 s and 1.75 s, so a trigger time past it rejects the excursion completely while a trigger time just short of it rejects nothing.
Choosing from a pair of tables like these:
- On the fault side, latency decides the trigger time, and the band takes some margin above the floor. How much is your call, and it depends on how much data the floor came from. A floor measured on three flights is a sample rather than the worst case, so something like 30% hedges against the next flight going deeper, turning a 1.75 A floor into 2.25 A. With hundreds of flights behind it, the measured floor is already close to the real one and you can sit nearer to it.
- On the nuisance side there is no ceiling from physics, so the band is set by what you want to catch rather than by what a fault can reach. Pick the smallest over-draw worth flagging, find the shortest trigger time whose floor is below it, then go a little longer than that. The floor drops at whatever trigger time outlasts the benign excursion, so the shortest trigger time that works is sitting on its edge. The extra trigger time costs only latency, and these faults are thermal.
::: warning
If you write your own sweep instead of using the script, write it in bash and pass the --set flags
literally or through an array.
In zsh an unquoted variable holding --set A=1 --set B=2 arrives as a single argument, so the tool
prints a usage error and exits without a verdict table, and a loop that only counts failures reads that
as "nothing tripped".
Assert that a table was produced before trusting a zero, as the script's grep -q guard does.
:::
3. Verify.
Re-run the whole set of flight logs with the chosen configuration, including logs that were not used for the fit, and confirm every motor is ok with no phase-dependent verdicts.
Then check the detection floor the configuration implies: the undercurrent band divided by the per-motor hover current is the smallest current loss that can be seen at hover.
Replaying a Fleet of Logs
A single flight doesn't usually contain enough information to calculate thresholds, so the usual unit of work is a set of flight logs.
batch_replay.sh runs the tool over every .ulg in a directory and summarizes the verdicts:
src/lib/motor_failure_detector/tools/batch_replay.sh ~/logs/healthy --set MOTFAIL_UNDER=1.75 --set MOTFAIL_UND_TIME=0.7
OK flight1.ulg
OK flight2.ulg
OK flight3.ulg
----
total=3 OK=3 FAIL=0 SKIP=0
Anything after the directory is passed straight through to mfd_replay.
SKIP counts logs with no usable ESC data.
Run it from the repository root, or set MFD_REPLAY to the path of the binary.
::: details Replaying thousands of logs
batch_replay.sh works through the logs one at a time, which is fine for tens of them and slow for thousands.
For a large set, run mfd_replay itself with one process per log, and give each its own output file so the verdict tables do not interleave:
mkdir -p /tmp/verdicts
ls ~/logs/healthy/*.ulg | xargs -P"$(nproc)" -I{} \
sh -c './build/px4_sitl_default/mfd_replay "$1" > /tmp/verdicts/$(basename "$1").txt' _ {}
grep -l FAILED /tmp/verdicts/*.txt # the logs that tripped
This gives the full per-motor table for every log instead of the one-line summary, which is what you want when you then have to look at why something tripped. :::
What the Replay Does Not Reproduce
- The sampling phase, which is why the sweep exists.
- Real faults, unless the log contains one. There is no fault injection: to study the true-positive side, replay a log from a flight where a motor really failed, or modify a log so that the current of one motor reflects the fault you want to study.
- Parameter changes during the log. On the vehicle the detector is reset whenever it is disarmed and reconfigured on parameter change; the replay configures it once, resets it on disarm and evaluates only the armed samples.
- Anything outside the current check. The ESC telemetry timeout, the arming checks and the failure response all live elsewhere in the firmware and are not part of this tool.
See Also
- Motor Failure Detection — what the detector does and how it is configured on the vehicle.
- Flight Log Analysis — general tooling for working with ULog files.