fix(various): issues found running SITL tests with sanitizers (#28943)

* test(mavsdk): fail test cases on sanitizer reports

The runner only looked at the mavsdk_tests exit code, so ASan, TSan and
UBSan reports from px4 ended up in the logs without failing anything.
Scan the output of all processes for sanitizer report markers and fail
the test case if any appear, including leak reports printed at exit.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(mavlink): allow more boot time when sim speed factor is set

In lockstep SITL the boot timeout is measured in sim time, but the
remainder of rcS after mavlink start runs in real time. At high speed
factors (e.g. 30x in mavsdk_tests) the 20 s sim-time timeout expires
before rcS reaches "mavlink boot_complete", triggering a spurious
error. Use 40 s when PX4_SIM_SPEED_FACTOR is set.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(mavlink): don't share vehicle_status subscription across threads

send_autopilot_capabilities() (also called from the receiver thread on
REQUEST_MESSAGE) and the SETUP_SIGNING armed check (receiver thread)
used the same _vehicle_status_sub as handleStatus() on the main thread.
Besides being a data race flagged by TSan, advancing the subscription's
generation from another thread can make handleStatus() miss a
vehicle_status update. Use local subscriptions for those two callers.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(gyro_fft): do initial sensor selection on the work queue

init() ran SensorSelectionUpdate() on the shell thread, which registers
the gyro callback before setting _selected_sensor_device_id, _gyro_fifo
and the gap perf counter. A gyro sample can then start Run() on the
work queue while those are still being written, which TSan flagged.
Schedule Run() instead and let it do the first selection.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(logger): copy events into an aligned struct before access

The event was copied into _msg_buffer at an offset that isn't aligned
for event_s and then accessed through an event_s pointer, which is
undefined behaviour (flagged by UBSan) even for single-byte fields.
Copy into a local event_s, adjust it there and memcpy it into the
buffer. The bytes written to the log are unchanged.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(manual_control): don't convert negative kill gesture time to unsigned

MAN_KILL_GEST_T is negative by default to disable the gesture, which
made the hysteresis time a negative float converted to hrt_abstime,
undefined behaviour flagged by UBSan. Clamp to 0; the gesture is only
evaluated when the parameter is positive.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(mavlink): define mavlink_channel_t with all 6 channels

The library's mavlink_channel_t ends at MAVLINK_COMM_3, and channels 4
and 5 were created by casting out-of-range values to the enum, which
UBSan flags. Use the library's HAVE_MAVLINK_CHANNEL_T hook to define
the enum with all channels when MAVLINK_COMM_NUM_BUFFERS is 6.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(mavlink): clamp DISTANCE_SENSOR fields to their integer ranges

A slightly negative range reading (seen with SIH near the ground) was
converted to the uint16_t distance field, which is undefined behaviour
flagged by UBSan. Clamp distances and covariance to their field ranges.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(sensors): skip baro GNSS offset without baro samples

With no baro samples accumulated, the average pressure was 0/0 = NaN.
A NaN delta altitude poisons the offset low-pass filter, so the baro
GNSS autocal could silently never complete. Skip the run instead, and
leave the GNSS sample for the next one. Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(mavlink): don't divide by zero rate without variable rate streams

Without variable rate streams the bandwidth multiplier was a division by
zero, and the resulting inf was only clamped away later. There is
nothing to scale in that case, so use 1. Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(gyro_fft): skip peak interpolation for a zero or flat spectrum

Quinn's estimator divides by the peak bin magnitude and by 1 - ap and
1 - am, which are zero for a zero or flat spectrum (e.g. noise-free SIH
gyro data). Return NAN, the existing no-estimate value, instead.
Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(logger): don't compute write rate before time has passed

Right after a log file is opened the elapsed time is zero, so the write
rate was a division by zero. Report 0 until time has passed. Flagged by
UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(battery): check cell voltage before computing scale

Check the filtered cell voltage before dividing by it rather than
relying on the NaN/inf result of the division. Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(perf): report 0 rms for a single event

With one event the rms was computed as 0/0. Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(matrix): only normalize axis for non-zero axis-angle rotations

The axis was normalized before checking for a zero angle, dividing by a
zero norm (e.g. SIH with no rotation). The result was already unused in
that case. Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* fix(mathlib): return infinite cutoff for unconfigured AlphaFilter

A time constant of zero means no filtering; return an infinite cutoff
frequency explicitly instead of dividing by zero. Flagged by UBSan.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

---------

Signed-off-by: Julian Oes <julian@oes.ch>
This commit is contained in:
Julian Oes authored and GitHub committed 2026-10-02 10:39:54 -06:00
1 parent b798249a61
commit e6db1b3a14
13 files changed
+105 -45

No files matched your search

+3 -3
View File
@@ -379,10 +379,10 @@ uint16_t Battery::determineFaults()
void Battery::computeScale()
{
_scale = _params.v_charged / _cell_voltage_filter_v.getState();
const float cell_voltage = _cell_voltage_filter_v.getState();
if (PX4_ISFINITE(_scale)) {
_scale = math::constrain(_scale, 1.f, 1.3f); // Allow at most 30% compensation
if (cell_voltage > FLT_EPSILON) {
_scale = math::constrain(_params.v_charged / cell_voltage, 1.f, 1.3f); // Allow at most 30% compensation
} else {
_scale = 1.f;
+2 -1
View File
@@ -148,7 +148,8 @@ public:
}
const T &getState() const { return _filter_state; }
float getCutoffFreq() const { return 1.f / (M_TWOPI_F * _time_constant); }
// a time constant of 0 means no filtering, i.e. an infinite cutoff frequency
float getCutoffFreq() const { return (_time_constant > FLT_EPSILON) ? 1.f / (M_TWOPI_F * _time_constant) : INFINITY; }
// delete mixing seconds with microseconds explicitly -> better compiler error
AlphaFilter(uint64_t sample_interval_us, float time_constant_s) = delete;
+2 -1
View File
@@ -171,13 +171,14 @@ public:
{
Quaternion &q = *this;
Type angle = aa.norm();
Vector<Type, 3> axis = aa.unit();
if (angle < Type(1e-10)) {
q(0) = Type(1);
q(1) = q(2) = q(3) = 0;
} else {
// only normalize once the angle is known to be non-zero
Vector<Type, 3> axis = aa.unit();
Type magnitude = std::sin(angle / Type(2));
q(0) = std::cos(angle / Type(2));
q(1) = axis(0) * magnitude;
+4 -4
View File
@@ -546,7 +546,7 @@ perf_print_counter(perf_counter_t handle)
uint32_t tl = pce->time_least.load();
uint32_t tm = pce->time_most.load();
float m2 = pce->M2.load();
float rms = sqrtf(m2 / (ec - 1));
float rms = (ec > 1) ? sqrtf(m2 / (ec - 1)) : 0.f;
PX4_INFO_RAW("%s: %" PRIu64 " events, %" PRIu64 "us elapsed, %.2fus avg, min %" PRIu32 "us max %" PRIu32
"us %5.3fus rms\n",
handle->name, ec, tt,
@@ -563,7 +563,7 @@ perf_print_counter(perf_counter_t handle)
uint32_t tl = pci->time_least.load();
uint32_t tm = pci->time_most.load();
float m2 = pci->M2.load();
float rms = sqrtf(m2 / (ec - 1));
float rms = (ec > 1) ? sqrtf(m2 / (ec - 1)) : 0.f;
PX4_INFO_RAW("%s: %" PRIu64 " events, %.2fus avg, min %" PRIu32 "us max %" PRIu32 "us %5.3fus rms\n",
handle->name, ec,
@@ -601,7 +601,7 @@ perf_print_counter_buffer(char *buffer, int length, perf_counter_t handle)
uint32_t tl = pce->time_least.load();
uint32_t tm = pce->time_most.load();
float m2 = pce->M2.load();
float rms = sqrtf(m2 / (ec - 1));
float rms = (ec > 1) ? sqrtf(m2 / (ec - 1)) : 0.f;
num_written = snprintf(buffer, length,
"%s: %" PRIu64 " events, %" PRIu64 "us elapsed, %.2fus avg, min %" PRIu32 "us max %" PRIu32 "us %5.3fus rms",
handle->name,
@@ -620,7 +620,7 @@ perf_print_counter_buffer(char *buffer, int length, perf_counter_t handle)
uint32_t tl = pci->time_least.load();
uint32_t tm = pci->time_most.load();
float m2 = pci->M2.load();
float rms = sqrtf(m2 / (ec - 1));
float rms = (ec > 1) ? sqrtf(m2 / (ec - 1)) : 0.f;
num_written = snprintf(buffer, length,
"%s: %" PRIu64 " events, %.2f avg, min %" PRIu32 "us max %" PRIu32 "us %5.3fus rms",
+16 -6
View File
@@ -149,9 +149,10 @@ bool GyroFFT::init()
arm_float_to_q15(&hanning_value, &_hanning_window[n], 1);
}
if (!SensorSelectionUpdate(true)) {
ScheduleDelayed(500_ms);
}
// Do the initial sensor selection in Run() on the work queue. Selecting here
// registers a callback that can trigger Run() while the selection is
// still being written from this thread.
ScheduleNow();
return true;
}
@@ -279,15 +280,24 @@ float GyroFFT::EstimatePeakFrequencyBin(q15_t fft[], int peak_index)
const float divider = (real[k] * real[k] + imag[k] * imag[k]);
if (divider < FLT_EPSILON) {
return NAN;
}
// ap = (X[k + 1].r * X[k].r + X[k+1].i * X[k].i) / (X[k].r * X[k].r + X[k].i * X[k].i)
float ap = (real[k + 1] * real[k] + imag[k + 1] * imag[k]) / divider;
// dp = -ap / (1 – ap)
float dp = -ap / (1.f - ap);
// am = (X[k - 1].r * X[k].r + X[k – 1].i * X[k].i) / (X[k].r * X[k].r + X[k].i * X[k].i)
float am = (real[k - 1] * real[k] + imag[k - 1] * imag[k]) / divider;
// dp and dm below are undefined for ap or am of 1 (e.g. a flat spectrum)
if ((fabsf(1.f - ap) < FLT_EPSILON) || (fabsf(1.f - am) < FLT_EPSILON)) {
return NAN;
}
// dp = -ap / (1 – ap)
float dp = -ap / (1.f - ap);
// dm = am / (1 – am)
float dm = am / (1.f - am);
+12 -13
View File
@@ -996,19 +996,19 @@ bool Logger::handle_event_updates(uint32_t &total_bytes)
bool data_written = false;
while (_event_subscription.updated()) {
event_s *orb_event = (event_s *)(_msg_buffer + sizeof(ulog_message_data_s));
_event_subscription.copy(orb_event);
// Copy into an aligned struct first: the message data in _msg_buffer is
// not necessarily aligned for event_s.
event_s orb_event;
_event_subscription.copy(&orb_event);
uint8_t *const msg_data = _msg_buffer + sizeof(ulog_message_data_s);
// Important: we can only access single-byte values in orb_event (it's not necessarily aligned)
if (events::internalLogLevel(orb_event->log_levels) == events::LogLevelInternal::Disabled) {
if (events::internalLogLevel(orb_event.log_levels) == events::LogLevelInternal::Disabled) {
++_event_sequence_offset; // skip this event
} else {
// adjust sequence number
uint16_t updated_sequence;
memcpy(&updated_sequence, &orb_event->event_sequence, sizeof(updated_sequence));
updated_sequence -= _event_sequence_offset;
memcpy(&orb_event->event_sequence, &updated_sequence, sizeof(updated_sequence));
orb_event.event_sequence -= _event_sequence_offset;
memcpy(msg_data, &orb_event, sizeof(orb_event));
size_t msg_size = sizeof(ulog_message_data_s) + _event_subscription.get_topic()->o_size_no_padding;
uint16_t write_msg_size = static_cast<uint16_t>(msg_size - ULOG_MSG_HEADER_LEN);
@@ -1031,11 +1031,10 @@ bool Logger::handle_event_updates(uint32_t &total_bytes)
}
// mission log: only warnings or higher
if (events::internalLogLevel(orb_event->log_levels) <= events::LogLevelInternal::Warning) {
if (events::internalLogLevel(orb_event.log_levels) <= events::LogLevelInternal::Warning) {
if (_writer.is_started(LogType::Mission)) {
memcpy(&updated_sequence, &orb_event->event_sequence, sizeof(updated_sequence));
updated_sequence -= _event_sequence_offset_mission;
memcpy(&orb_event->event_sequence, &updated_sequence, sizeof(updated_sequence));
orb_event.event_sequence -= _event_sequence_offset_mission;
memcpy(msg_data, &orb_event, sizeof(orb_event));
if (write_message(LogType::Mission, _msg_buffer, msg_size)) {
data_written = true;
@@ -1073,7 +1072,7 @@ void Logger::publish_logger_status()
status.is_logging = true;
status.total_written_kb = kb_written;
status.write_rate_kb_s = kb_written / seconds;
status.write_rate_kb_s = (seconds > FLT_EPSILON) ? kb_written / seconds : 0.f;
status.dropouts = _statistics[i].write_dropouts;
status.message_gaps = _message_gaps;
status.buffer_used_bytes = buffer_fill_count_file;
+3 -1
View File
@@ -34,6 +34,7 @@
#include "ManualControl.hpp"
#include <px4_platform_common/events.h>
#include <lib/mathlib/mathlib.h>
#include <lib/systemlib/mavlink_log.h>
#include <uORB/topics/vehicle_command.h>
@@ -355,7 +356,8 @@ void ManualControl::updateParams()
_stick_arm_hysteresis.set_hysteresis_time_from(false, 1_s);
_stick_disarm_hysteresis.set_hysteresis_time_from(false, 1_s);
_button_arm_hysteresis.set_hysteresis_time_from(false, 1_s);
_stick_kill_hysteresis.set_hysteresis_time_from(false, _param_man_kill_gest_t.get() * 1_s);
// A negative MAN_KILL_GEST_T disables the gesture, don't convert it to an unsigned time
_stick_kill_hysteresis.set_hysteresis_time_from(false, math::max(_param_man_kill_gest_t.get(), 0.f) * 1_s);
_selector.setRcInMode(_param_com_rc_in_mode.get());
_selector.setTimeout(_param_com_rc_loss_t.get() * 1_s);
+11 -2
View File
@@ -57,8 +57,17 @@
#if !defined(CONSTRAINED_MEMORY)
# define MAVLINK_COMM_NUM_BUFFERS 6
# define MAVLINK_COMM_4 static_cast<mavlink_channel_t>(4)
# define MAVLINK_COMM_5 static_cast<mavlink_channel_t>(5)
// The library's mavlink_channel_t only goes up to MAVLINK_COMM_3, so casting 4 or 5
// to it would be an out-of-range enum value. Provide our own with all channels.
# define HAVE_MAVLINK_CHANNEL_T
typedef enum {
MAVLINK_COMM_0,
MAVLINK_COMM_1,
MAVLINK_COMM_2,
MAVLINK_COMM_3,
MAVLINK_COMM_4,
MAVLINK_COMM_5
} mavlink_channel_t;
#endif
#include <mavlink_types.h>
+16 -7
View File
@@ -326,13 +326,13 @@ bool Mavlink::set_channel()
case 3:
_channel = MAVLINK_COMM_3;
return true;
#ifdef MAVLINK_COMM_4
#if MAVLINK_COMM_NUM_BUFFERS > 4
case 4:
_channel = MAVLINK_COMM_4;
return true;
#endif
#ifdef MAVLINK_COMM_5
#if MAVLINK_COMM_NUM_BUFFERS > 5
case 5:
_channel = MAVLINK_COMM_5;
@@ -1207,10 +1207,12 @@ Mavlink::handle_message(const mavlink_message_t *msg)
mavlink_msg_setup_signing_decode(msg, &setup_signing);
if (target_is_us_or_broadcast(setup_signing.target_system, setup_signing.target_component)) {
// Reject signing changes while armed
// Reject signing changes while armed. This runs on the receiver
// thread, so use a local subscription instead of _vehicle_status_sub.
uORB::Subscription vehicle_status_sub{ORB_ID(vehicle_status)};
vehicle_status_s vehicle_status{};
if (_vehicle_status_sub.copy(&vehicle_status)
if (vehicle_status_sub.copy(&vehicle_status)
&& vehicle_status.arming_state == vehicle_status_s::ARMING_STATE_ARMED) {
send_statustext_critical("MAVLink signing: rejected while armed");
return;
@@ -1292,9 +1294,12 @@ Mavlink::send_statustext_emergency(const char *string)
bool
Mavlink::send_autopilot_capabilities()
{
// Called from both the main and the receiver thread (REQUEST_MESSAGE),
// so use a local subscription instead of _vehicle_status_sub.
uORB::Subscription vehicle_status_sub{ORB_ID(vehicle_status)};
vehicle_status_s status;
if (_vehicle_status_sub.copy(&status)) {
if (vehicle_status_sub.copy(&status)) {
mavlink_autopilot_version_t msg{};
msg.capabilities = MAV_PROTOCOL_CAPABILITY_MISSION_FLOAT;
@@ -1586,8 +1591,12 @@ Mavlink::update_rate_mult()
mavlink_ulog_streaming_rate_inv = 1.0f - _mavlink_ulog->current_data_rate();
}
/* scale up and down as the link permits */
float bandwidth_mult = (float)(_datarate * mavlink_ulog_streaming_rate_inv - const_rate) / rate;
/* scale up and down as the link permits, nothing to scale without variable rate streams */
float bandwidth_mult = 1.0f;
if (rate > 0.0f) {
bandwidth_mult = (float)(_datarate * mavlink_ulog_streaming_rate_inv - const_rate) / rate;
}
/* Reduce rate while sending parameters in low bandwidth mode */
if (sending_parameters() && _mode == Mavlink::MAVLINK_MODE_LOW_BANDWIDTH) {
+9 -2
View File
@@ -438,8 +438,15 @@ MavlinkReceiver::handle_message(mavlink_message_t *msg)
_parameters_manager.handle_message(msg);
} else {
if (hrt_elapsed_time(&_mavlink.get_first_start_time()) > 20_s) {
PX4_ERR("system boot did not complete in 20 seconds");
#if defined(__PX4_POSIX)
// startup runs in real time while lockstep sim time can run faster, so allow more time when sped up
static const hrt_abstime boot_timeout = getenv("PX4_SIM_SPEED_FACTOR") ? 40_s : 20_s;
#else
static constexpr hrt_abstime boot_timeout = 20_s;
#endif
if (hrt_elapsed_time(&_mavlink.get_first_start_time()) > boot_timeout) {
PX4_ERR("system boot did not complete in %d seconds", (int)(boot_timeout / 1_s));
_mavlink.set_boot_complete();
}
}
@@ -88,12 +88,13 @@ private:
break;
}
msg.current_distance = dist_sensor.current_distance * 1e2f; // m to cm
// clamp to the field ranges, converting an out-of-range float is undefined
msg.current_distance = math::constrain(dist_sensor.current_distance * 1e2f, 0.f, (float)UINT16_MAX); // m to cm
msg.id = i;
msg.max_distance = dist_sensor.max_distance * 1e2f; // m to cm
msg.min_distance = dist_sensor.min_distance * 1e2f; // m to cm
msg.max_distance = math::constrain(dist_sensor.max_distance * 1e2f, 0.f, (float)UINT16_MAX); // m to cm
msg.min_distance = math::constrain(dist_sensor.min_distance * 1e2f, 0.f, (float)UINT16_MAX); // m to cm
msg.orientation = dist_sensor.orientation;
msg.covariance = dist_sensor.variance * 1e4f; // m^2 to cm^2
msg.covariance = math::constrain(dist_sensor.variance * 1e4f, 0.f, (float)UINT8_MAX); // m^2 to cm^2
msg.horizontal_fov = dist_sensor.h_fov;
msg.vertical_fov = dist_sensor.v_fov;
msg.quaternion[0] = dist_sensor.q[0];
@@ -519,7 +519,8 @@ bool VehicleAirData::BaroGNSSAltitudeOffset()
vehicle_gnss_s vehicle_gnss;
if (!_vehicle_gnss_sub.update(&vehicle_gnss)) {
// no baro samples accumulated yet, leave the GNSS sample for the next run
if ((_data_sum_count[_selected_sensor_sub_index] == 0) || !_vehicle_gnss_sub.update(&vehicle_gnss)) {
return false;
}
@@ -45,6 +45,12 @@ class TesterInterface:
class Tester:
# Markers of reports from ASan, LSan, TSan and UBSan. A test case fails
# if any process prints one, even if the test itself passed.
SANITIZER_REPORT_PATTERN = re.compile(
r'(ERROR: AddressSanitizer:|ERROR: LeakSanitizer:|'
r'WARNING: ThreadSanitizer:|:\d+:\d+: runtime error: )')
def __init__(self,
config: Dict[str, Any],
iterations: int,
@@ -76,6 +82,7 @@ class Tester:
self.tests = self.determine_tests(config['tests'], model, case)
self.active_runners = []
self.pre_test_hook: Optional[Callable[[str, str], None]] = None
self.sanitizer_reports: List[str] = []
@staticmethod
def wildcard_match(pattern: str, potential_match: str) -> bool:
@@ -254,6 +261,7 @@ class Tester:
logfile_path = self.determine_logfile_path(log_dir, 'combined')
self.start_combined_log(logfile_path)
self.sanitizer_reports = []
self.start_runners(log_dir, test, case)
@@ -279,6 +287,16 @@ class Tester:
self.collect_runner_output()
self.stop_combined_log()
if self.sanitizer_reports:
print(colorize(
"Sanitizer reported {} issue{}:".format(
len(self.sanitizer_reports),
self.plural_s(len(self.sanitizer_reports))),
color.BOLD))
for report in self.sanitizer_reports:
print(" {}".format(report))
is_success = False
result = {'success': is_success,
'logfiles': [runner.get_log_filename()
for runner in self.active_runners]}
@@ -413,6 +431,8 @@ class Tester:
break
self.add_to_combined_log(line)
if self.SANITIZER_REPORT_PATTERN.search(line):
self.sanitizer_reports.append(line.strip())
if self.verbose:
print(line, end="")