Files
PX4-Autopilot/docs/en/debug/failure_injection.md
T
Andrea BernasconiandHamish Willee db94ca49db feat(failure_injection): add wrong, garbage and drift IMU failures (#29601)
* feat(failure_injection): add wrong, garbage and drift injection for gyro and accel

Corrupt the addressed IMU's samples on all three axes: wrong adds a
constant bias (SYS_FAIL_{G,A}_BIAS), garbage uniform noise within
+-SYS_FAIL_{G,A}_NOISE and drift a bias growing by SYS_FAIL_{G,A}_DRIFT
per second since injection. Applied to both the plain and FIFO driver paths.

* refactor(failure_injection): handle IMU failures in process_imu()

Mirror process_gnss(): resolve the mode once, run the generic Off / Stuck
first and only then apply the Wrong / Garbage / Drift offset, so the Stuck
cache keeps the uncorrupted sample and a later Stuck never replays a
drifted one. Drop the second process() on the integrated FIFO report, which
cached the already-corrupted report. The offset is still computed before
Off so drift_start is reset while the sensor is off.

* docs(failure_injection): align IMU failure docs with code

The Syntax section did not list drift and described garbage as
uninitialised-memory output, while gyro/accel garbage adds bounded noise
to the real measurement. Also match the new parameter descriptions to
the wording of the existing SYS_FAIL_* parameters.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Hamish Willee <hamishwillee@gmail.com>

* fix(failure_injection): add drift RC mode and clarify bias param text

* refactor(drivers): Drop extra comment

* test(mavsdk_tests): add SITL test for IMU failure injection

Fly the SIH quad through gyro and accel bias, noise and drift with the
default SYS_FAIL_{G,A}_* magnitudes and check it holds altitude.

Holding altitude alone also passes when the injection never reaches the
sensor data, so the test also checks that the IMU noise level jumps
while noise is injected and returns once cleared. Bias and drift are
not checked in the data: the rate controller and the estimator react
to them and partly hide them in flight.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Andrea Bernasconi <andrea.bernasconi@auterion.com>

* test(mavsdk_tests): inject IMU failures through MAVSDK 4.2.0

MAVSDK-Python 4.2.0 adds FailureType.DRIFT (mavlink/MAVSDK#3211), so the
IMU failure test no longer needs to send a raw MAV_CMD_INJECT_FAILURE
with its own copies of the unit and type values. Use the tester's
inject_failure() like the motor failure tests, and bump the mavsdk pin
so the new type is available.

Assisted-by: Claude:claude-fable-5-2
Signed-off-by: Andrea Bernasconi <andrea.bernasconi@auterion.com>

---------

Signed-off-by: Hamish Willee <hamishwillee@gmail.com>
Signed-off-by: Andrea Bernasconi <andrea.bernasconi@auterion.com>
Co-authored-by: Hamish Willee <hamishwillee@gmail.com>
2026-11-13 11:24:05 +01:00

205 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# System Failure Injection
System failure injection allows you to induce different types of sensor and system failures, either via MAVLink (using [MAV_CMD_INJECT_FAILURE](https://mavlink.io/en/messages/common.html#MAV_CMD_INJECT_FAILURE) or via the [MAVSDK failure plugin](https://mavsdk.mavlink.io/main/en/cpp/api_reference/classmavsdk_1_1_failure.html)), or "manually" via a PX4 console like the [MAVLink shell](../debug/mavlink_shell.md#mavlink-shell).
This enables easier testing of [safety failsafe](../config/safety.md) behaviour, and more generally, of how PX4 behaves when systems and sensors stop working correctly.
Failure injection is disabled by default, and can be enabled using the [SYS_FAILURE_EN](../advanced_config/parameter_reference.md#SYS_FAILURE_EN) parameter.
Failures can be injected both in simulation and on real hardware; this requires firmware built with the failure-injection module.
The command always goes through the same firmware failure-injection module — whether it arrives over MAVLink or from the console, the accepted combinations are identical.
What differs is whether a _consumer_ applies the failure, and that depends on the environment.
## Supported Failure Types
The table lists the failure types that actually take effect per environment: `off`, `stuck`, `wrong`, `garbage`, `drift` (`ok` is not listed, but clears an active injection in all environments).
A `—` means the module still accepts the command, but no consumer applies it in that environment.
| Component | [Gazebo] (gz) | [SIH] | `simulator_mavlink` (Gazebo Classic) | Hardware |
| ----------------- | ------------------------------------------- | ------------------------------------------- | ------------------------------------------- | ------------------------------------------- |
| `gyro` | `off`, `stuck`, `wrong`, `garbage`, `drift` | `off`, `stuck`, `wrong`, `garbage`, `drift` | `off`, `stuck`, `wrong`, `garbage`, `drift` | `off`, `stuck`, `wrong`, `garbage`, `drift` |
| `accel` | `off`, `stuck`, `wrong`, `garbage`, `drift` | `off`, `stuck`, `wrong`, `garbage`, `drift` | `off`, `stuck`, `wrong`, `garbage`, `drift` | `off`, `stuck`, `wrong`, `garbage`, `drift` |
| `mag` | `off`, `stuck` | `off`, `stuck` | `off`, `stuck` | `off`, `stuck` |
| `baro` | `off`, `stuck` | `off`, `stuck` | `off`, `stuck` | `off`, `stuck` |
| `distance_sensor` | `off`, `stuck` | `off`, `stuck` | `off`, `stuck` | `off`, `stuck` |
| `gps` | `off`, `stuck`, `wrong` | `off`, `stuck`, `wrong` | `off`, `stuck`, `wrong` | `off`, `stuck`, `wrong` |
| `airspeed` | `off`, `stuck`, `wrong` | — | `off`, `wrong` | — |
| `vio` | `off`, `stuck` | — | `off` | — |
| `battery` | `off`, `wrong` | `off`, `wrong` | `off`, `wrong` | `off`, `wrong` |
| `traffic` | `off` | `off` | `off` | `off` |
| `motor` | `off`, `wrong` | `off`, `stuck`, `wrong` | `off`, `wrong` | `off`, `wrong` |
| `esc` | `off`, `wrong` | `off`, `wrong` | `off`, `wrong` | `off`, `wrong` |
| `can` | — | — | — | `off` |
[SIH]: ../sim_sih/index.md
[Gazebo]: ../sim_gazebo_gz/index.md
::: info
- `gps off | stuck | wrong` on Gazebo (Gz): applied to the GNSS data from the simulator's own sensor as well as to the simulated-GPS module ([SIM_GZ_EN_GPS](../advanced_config/parameter_reference.md#SIM_GZ_EN_GPS) `0`).
- `gyro wrong | garbage | drift` and `accel wrong | garbage | drift` <Badge type="tip" text="main (PX4 v2.0)" /> corrupt the addressed sensor's samples on all three axes: `wrong` adds a constant bias ([SYS_FAIL_G_BIAS](../advanced_config/parameter_reference.md#SYS_FAIL_G_BIAS) / [SYS_FAIL_A_BIAS](../advanced_config/parameter_reference.md#SYS_FAIL_A_BIAS)), `garbage` adds uniform noise within ±[SYS_FAIL_G_NOISE](../advanced_config/parameter_reference.md#SYS_FAIL_G_NOISE) / ±[SYS_FAIL_A_NOISE](../advanced_config/parameter_reference.md#SYS_FAIL_A_NOISE), and `drift` adds a bias that grows by [SYS_FAIL_G_DRIFT](../advanced_config/parameter_reference.md#SYS_FAIL_G_DRIFT) / [SYS_FAIL_A_DRIFT](../advanced_config/parameter_reference.md#SYS_FAIL_A_DRIFT) per second since injection.
- `airspeed off | stuck | wrong` on Gazebo (Gz): only injectable when airspeed is provided by the simulated-airspeed module ([SENS_EN_ARSPDSIM](../advanced_config/parameter_reference.md#SENS_EN_ARSPDSIM)); worlds that model an airspeed sensor directly are not injected.
- `vio off | stuck` on Gazebo (Gz): applied to the external-vision odometry that the bridge publishes from the model's odometry topic (for example `x500_vision`); `stuck` keeps publishing the last good sample with fresh timestamps.
- `battery wrong` reports the remaining charge just below the [SYS_FAIL_BAT_LVL](../advanced_config/parameter_reference.md#SYS_FAIL_BAT_LVL) warning threshold to trigger the battery failsafe; `off` stops publishing the battery status entirely.
- `traffic off` suppresses incoming reports and marks the ADS-B/FLARM link unhealthy.
- `motor off` and `motor wrong` are the detected and undetected motor failures, and both require [CA_FAILURE_MODE](../advanced_config/parameter_reference.md#CA_FAILURE_MODE). `motor off` is the _detected_ failure: the motor is reported as failed, so the control allocator removes it from the allocation and handles the opposite motor per `CA_FAILURE_MODE`. Only a single failed motor can be recovered from, so an instance mask naming several motors has no effect. `motor wrong` is the _undetected_ failure: the output is stopped without informing the allocator, which models a failure with no feedback path (such as PWM ESCs), and any number of motors can be addressed.
- `motor stuck` (SIH only) models a stuck rotor with a healthy ESC: the motor is still commanded but produces no thrust, and the ESC stays online, reporting zero RPM.
Neither the failure detector nor the control allocator is informed; use it to test the [motor not spinning after arming](../config/safety.md#motor-not-spinning-after-arming) check.
Motors are addressed by motor instance, so `-i 1` targets motor 1.
- `esc off` reports the addressed ESC as offline and blanks its telemetry; `esc wrong` keeps it online but reports implausible telemetry (voltage and current at 10% of the real value, RPM 10x). ESCs are addressed by motor instance (the ESC's actuator function), so `-i 1` targets the ESC driving motor 1.
- On hardware, ESC injection is applied only by the UAVCAN (DroneCAN) ESC driver; the other ESC drivers (DShot, Cyphal, VOXL, TAP ESC) publish their telemetry unmodified.
- `can off` takes the addressed CAN bus offline entirely, so every node on it stops responding; the instance selects the bus. Only applied on fmu-v6x-class hardware.
:::
Sensors delivered through the shared driver layer (IMU, magnetometer, barometer, rangefinder via the `PX4*` sensor wrappers) support `off`/`stuck` in every environment that uses that layer — including the Gazebo and SIH sensor simulators, which feed synthesized measurements through the same wrappers.
The IMU (`gyro`, `accel`) also supports `wrong`/`garbage`/`drift` through the same layer.
The remaining gaps are backend-specific: GPS and airspeed are handled by dedicated simulator code (see the GPS and airspeed notes in the info box above), SIH does not simulate an injectable airspeed.
Components not listed (`optical_flow`, `servo`, `avoidance`, `rc_signal`, `mavlink_signal`) are rejected everywhere (`MAV_RESULT_UNSUPPORTED`); see the note below on NACK behaviour.
::: info
PX4 may accept a command to set a particular failure mode even it that mode is not supported by your simulator.
All [MAV_CMD_INJECT_FAILURE](https://mavlink.io/en/messages/common.html#MAV_CMD_INJECT_FAILURE) commands are handled internally by the failure-injection module, which acknowledges each command and republishes the active failures for the sensor/actuator simulators to apply.
The failure-injection module will NACK the command with [MAV_RESULT_UNSUPPORTED](https://mavlink.io/en/messages/common.html#MAV_RESULT_UNSUPPORTED) for failure combinations that are not implemented by PX4 or any simulator.
However it the module will accept (respond with [MAV_MISSION_ACCEPTED](https://mavlink.io/en/messages/common.html#MAV_MISSION_ACCEPTED)) for any other failure-type, even if it is not supported by your _particular_ simulator.
:::
## Failure System Command
Failures can be injected using the [failure system command](../modules/modules_command.md#failure) from any PX4 [console/shell](../debug/consoles.md) (such as the [QGC MAVLink Console](../debug/mavlink_shell.md#qgroundcontrol-mavlink-console) or SITL _pxh shell_), specifying both the target and type of the failure.
### Syntax
The full syntax of the [failure](../modules/modules_command.md#failure) command is:
```sh
failure <component> <failure_type> [-i <instance_number>] [-m <instance_bitmask>]
```
where:
- _component_:
- Sensors:
- `gyro`: Gyroscope
- `accel`: Accelerometer
- `mag`: Magnetometer
- `baro`: Barometer
- `gps`: Global navigation satellite system
- `optical_flow`: Optical flow.
- `vio`: Visual inertial odometry
- `distance_sensor`: Distance sensor (rangefinder).
- `airspeed`: Airspeed sensor
- Systems:
- `battery`: Battery
- `motor`: Motor
- `esc`: ESC telemetry
- `servo`: Servo
- `avoidance`: Obstacle/collision avoidance system
- `traffic`: Traffic avoidance (ADS-B/transponder)
- `rc_signal`: RC Signal
- `mavlink_signal`: MAVLink data telemetry connection
- `can`: CAN bus. The instance selects the bus. Supported on fmu-v6x-class hardware.
- _failure_type_:
- `ok`: Publish as normal (Disable failure injection)
- `off`: Stop publishing
- `stuck`: Constantly report the same value, which _can_ happen on a malfunctioning sensor.
For `motor` (SIH only), `stuck` stops the rotor while its ESC keeps reporting zero RPM.
- `garbage`: Publish random noise (which looks like reading uninitialised memory). For `gyro`/`accel` the noise is instead added to the real measurement.
- `wrong`: Publish invalid values that still look reasonable/aren't "garbage". For an actuator such as `motor` there is nothing to publish, so `wrong` means the motor stops without anything reporting it
- `slow`: Publish at a reduced rate
- `delayed`: Publish valid data with a significant delay
- `intermittent`: Publish intermittently
- `drift`: Publish values that drift increasingly away from the true value
- _instance number_ (optional): Instance number of affected sensor.
0 (default) indicates all sensors of specified type.
- _instance bitmask_ (optional): <Badge type="tip" text="main (PX4 v2.0)" />
address several instances at once (bit 0 = first instance, bit 1 = second, …; decimal or `0x` hex). Used only when `-i` is omitted.
Example: `-m 0x5` targets instances 1 and 3.
::: info
GPS implements only the `off`, `stuck`, and `wrong` failure modes; the other failure types have no effect on it.
`gps wrong` makes the addressed receiver report the fix type selected by [SYS_FAIL_GPS_WRG](../advanced_config/parameter_reference.md#SYS_FAIL_GPS_WRG) and the jamming state selected by [SYS_FAIL_GPS_JAM](../advanced_config/parameter_reference.md#SYS_FAIL_GPS_JAM) (`Unchanged` keeps the receiver's own state), and leaves the reported position untouched.
:::
## RC Switch Trigger
<Badge type="tip" text="main (PX4 v2.0)" />
A failure can also be injected from an RC switch, without a console or telemetry link. This is useful for in-flight hardware testing. It is configured with the following parameters:
- [SYS_FAIL_RC_SRC](../advanced_config/parameter_reference.md#SYS_FAIL_RC_SRC): the auxiliary RC input that triggers the failure — `0` disables it, `1`–`6` select AUX1–AUX6 (mapped via `RC_MAP_AUXn`).
- [SYS_FAIL_RC_UNIT](../advanced_config/parameter_reference.md#SYS_FAIL_RC_UNIT): the affected component (the `FAILURE_UNIT` value; e.g. `101` = motor).
- [SYS_FAIL_RC_MODE](../advanced_config/parameter_reference.md#SYS_FAIL_RC_MODE): the failure type (the `FAILURE_TYPE` value; e.g. `1` = off).
- [SYS_FAIL_RC_INST](../advanced_config/parameter_reference.md#SYS_FAIL_RC_INST): the affected instances, as a bitmask (bit 0 = instance 1, e.g. `5` = instances 1 and 3; `0` = all instances).
While the selected aux switch is on, the configured failure is injected; switching it back off clears the failure.
The injection goes through the same path as the console/MAVLink commands, so for a motor it fails the motor exactly as `failure motor off` (`SYS_FAIL_RC_MODE` = `1`) or `failure motor wrong` (`SYS_FAIL_RC_MODE` = `4`) does, with the same [CA_FAILURE_MODE](../advanced_config/parameter_reference.md#CA_FAILURE_MODE) requirement.
A `SYS_FAIL_RC_INST` mask addressing several motors only takes effect for `wrong`, the undetected failure.
In SIH, `SYS_FAIL_RC_MODE` = `2` injects `motor stuck`, which does not need `CA_FAILURE_MODE`.
## MAVSDK Failure Plugin
The [MAVSDK failure plugin](https://mavsdk.mavlink.io/main/en/cpp/api_reference/classmavsdk_1_1_failure.html) can be used to programmatically inject failures.
It is used in [PX4 Integration Testing](../test_and_ci/integration_testing_mavsdk.md) to simulate failure cases (for example, see [PX4-Autopilot/test/mavsdk_tests/test_multicopter_failsafe.py](https://github.com/PX4/PX4-Autopilot/blob/main/test/mavsdk_tests/test_multicopter_failsafe.py)).
The plugin API is a direct mapping of the failure command shown above, with a few additional error signals related to the connection.
## Example: GPS
To test the GPS failsafe by stopping GPS:
1. Enable the [SYS_FAILURE_EN](../advanced_config/parameter_reference.md#SYS_FAILURE_EN) parameter.
2. Enter the following commands on the MAVLink console or SITL _pxh shell_:
```sh
# Stop GPS publishing
failure gps off
# Restart GPS publishing
failure gps ok
```
## Example: Motor
To fail a motor mid-flight:
1. Enable the [SYS_FAILURE_EN](../advanced_config/parameter_reference.md#SYS_FAILURE_EN) parameter.
2. Enable [CA_FAILURE_MODE](../advanced_config/parameter_reference.md#CA_FAILURE_MODE) parameter to allow turning off motors.
3. Pick the failure type for the behavior you want to test:
- `off` — detected: the motor is flagged as failed and removed from control allocation, so the allocator compensates using the remaining motors (per `CA_FAILURE_MODE`).
Only one failed motor can be recovered from, so an instance mask naming several motors has no effect.
- `wrong` — undetected: the motor output is stopped without notifying the allocator, so its effectiveness is not excluded.
This models a failure with no feedback path, such as PWM ESCs.
Any number of motors can be addressed.
4. Enter the following commands on the MAVLink console or SITL _pxh shell_:
```sh
# Detected failure: motor 1 is reported as failed and removed from the allocation
failure motor off -i 1
# Undetected failure: motor 1 stops, the allocator is not informed
failure motor wrong -i 1
# Turn it back on
failure motor ok -i 1
```
## Example: Battery
To trigger the battery failsafe by reporting a depleted pack:
1. Enable the [SYS_FAILURE_EN](../advanced_config/parameter_reference.md#SYS_FAILURE_EN) parameter.
2. Optionally select the injected warning level with [SYS_FAIL_BAT_LVL](../advanced_config/parameter_reference.md#SYS_FAIL_BAT_LVL): Warn, Critical or Emergency. The reported remaining charge is set just below the matching threshold ([BAT_LOW_THR](../advanced_config/parameter_reference.md#BAT_LOW_THR), [BAT_CRIT_THR](../advanced_config/parameter_reference.md#BAT_CRIT_THR) or [BAT_EMERGEN_THR](../advanced_config/parameter_reference.md#BAT_EMERGEN_THR)).
3. Enter the following commands on the MAVLink console or SITL _pxh shell_:
```sh
# Report the battery as depleted at the SYS_FAIL_BAT_LVL warning level -> battery failsafe
failure battery wrong
# Stop publishing the battery status entirely
failure battery off
# Stop injecting the failure
failure battery ok
```