Files
PX4-Autopilot/test/mavsdk_tests/integration_test_runner/process_helper.py
T
Julian Oes 1c7a07a411 test(mavsdk_tests): rewrite the integration tests in Python (#29332)
* build(mavsdk_tests): stop building the C++ integration tests

The integration tests are moving to the native MAVSDK Python package, so
the C++ test binary and its pinned libmavsdk-dev package are no longer
needed. The C++ sources stay in the tree as a reference until all tests
are ported.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): run the integration tests with MAVSDK Python

The MAVSDK pip package is now a native binding of the C++ library, so
the tests can be written in Python without a C++ build or a matching
libmavsdk-dev package, and the version is pinned in requirements.txt.

The existing runner keeps starting PX4 for each case; only the test
executable changes to pytest. Cases are pytest node IDs and the Catch2
tags become pytest markers.

The tester is a port of the C++ AutopilotTester core. Checks done in
telemetry callbacks are collected and fail the test at teardown, since
MAVSDK Python swallows exceptions raised in callbacks.

The first ported test is takeoff and hold position.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): port the multicopter tests to Python

Ports missions, offboard, manual flight, failsafes, failure injection,
RTL with rally points and geofence avoidance, home yaw and parameter FTP.
The tester gains the raw mission, manual control, failure, FTP and
MavlinkDirect helpers these need, plus an RTL subclass for rally points
and the geofence breach monitor.

The failsafe tests are parametrized over the height source instead of
being duplicated.

The parameter FTP download test now waits for the baro GNSS calibration
before comparing, since the offset it writes could otherwise change
between the PARAM_VALUE and the FTP download.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): port the VTOL and control allocation tests to Python

Ports the VTOL missions, the VTOL return tests, figure eight, loiter
reposition, airspeed failure, the SIH XVert transition and the
gazebo-classic control allocation tests.

The loiter and figure-eight testers are merged into one loiter tester
that sends its commands as COMMAND_INT through MavlinkDirect. NaN fields
are sent as JSON null, since NaN is not valid JSON and MAVSDK reads null
as NaN.

Timeouts that were host time in C++ (approaches, figure eight) are now
sim time with values that fit the speed factors used in CI.

The gazebo runner can now take the world from the config, since pytest
node IDs no longer carry "(name.world)".

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): settle the hover before injecting motor failures

The altitude and speed waits both pass while the vehicle is still
slowing down at the top of the climb, so the altitude reference for the
failure was taken mid-climb. At 30x speed the following overshoot was
just over the 1 m tolerance.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): port the follow-me tests to Python

The C++ follow-me tests were never built: they were disabled pending a
MAVSDK change that has long since been merged, and the RC adjustment
test did not even compile. They now run with the other multicopter
tests.

Running them for the first time showed a few test bugs: target
locations were sent before follow-me was active, manual control was
dropped after 20 s, and the angle check ignored the rate limit from
FLW_TGT_MAX_VEL. The RC checks now measure the change from right before
each stick input.

The follow distance adjusts at about 40% of the nominal speed, since
PX4 only grows the setpoint while the drone is within 1 m of it. The
test accepts that for now.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): remove the C++ integration tests

All tests now run with MAVSDK Python, so the C++ tests, the tester
classes and the bundled Catch2 are no longer needed.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* build(setup): install the integration test requirements

The MAVSDK and pytest pins move to Tools/setup/test-requirements.txt,
which the setup scripts and the dev container now install, so the
integration tests work after the normal environment setup. The mavsdk
pin allows 4.0.x patch releases only.

The file lives in Tools/setup because that is the dev container's build
context. Container builds that stage only requirements.txt skip it.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* docs(test_and_ci): describe the Python MAVSDK integration tests

Covers the setup, running test cases by pytest ID, running a test
against a PX4 instance started by hand, and writing tests with the
testers and fixtures. Drops the Catch2 notes and the outdated list of
test cases.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* refactor(packaging): port the SIH container mission test to MAVSDK v4

Since v4, the mavsdk package is the native binding with a different API,
so the script failed to import with the version the setup scripts now
install. It now uses the synchronous API.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): work around MAVSDK-Python unsubscribe crash

Up to MAVSDK-Python 4.0.6, unsubscribe_*() releases the callback
trampoline before the C unsubscribe returns, so a callback that is
running or waiting for the GIL can jump into freed memory. This crashed
the hex test in CI. Keep the subscription entries until the tester is
closed, until the pin includes the fix.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): retry the param FTP comparison until values settle

Waiting for CAL_BARO0_OFF to change does not work everywhere: at the
CI home location the baro already matches GNSS and the calibration
writes an offset of 0. Instead, compare the PARAM_VALUE and FTP
downloads and retry a few times while parameters still settle.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): port the vision target tests to Python

Port the precision landing tests from #29265: the pad is simulated with
TARGET_RELATIVE and TARGET_ABSOLUTE sent through MavlinkDirect, and the
estimate is checked from LANDING_TARGET.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): fail when a filter or case selects nothing

A config filter that matched no test, for example because of a typo,
ran zero cases and passed. Also, --case used fnmatch, which reads the
brackets of parametrized test IDs as a character class, so such a test
could not be selected by its ID. Treat brackets literally and fail when
nothing is selected.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): check the COMMAND_ACK of sent commands

Commands sent through MavlinkDirect were sent once and the ACK was
ignored, so a dropped or denied command only showed up later as an
unrelated timeout. Retry until acked like MAVSDK does, and check the
result.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): check the condition once more on poll timeout

Also make the comment of check_rtl_approaches say that it can't fail.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

* test(mavsdk_tests): disable baro auto-calibration in the param FTP test

Port #29359 to the Python test: the baro GNSS calibration can save a
new CAL_BARO0_OFF between the PARAM_VALUE and FTP downloads. Disable it
instead of retrying the comparison.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Julian Oes <julian@oes.ch>

---------

Signed-off-by: Julian Oes <julian@oes.ch>
2026-10-26 08:50:51 -06:00

410 lines
14 KiB
Python

import queue
import time
import os
import atexit
import subprocess
import shutil
import threading
import errno
import sys
from typing import Any, Dict, List, TextIO, Optional
from .logger_helper import colorize, color
PX4_SITL_GAZEBO_PATH = "Tools/simulation/gazebo-classic/sitl_gazebo-classic"
PX4_GAZEBO_MODELS = PX4_SITL_GAZEBO_PATH + "/models"
PX4_GAZEBO_WORLDS = PX4_SITL_GAZEBO_PATH + "/worlds"
class Runner:
def __init__(self,
log_dir: str,
model: str,
case: str,
verbose: bool):
self.name = ""
self.cmd = ""
self.cwd = ""
self.args: List[str]
self.env: Dict[str, str] = os.environ.copy()
self.model = model
self.case = case
self.log_filename = ""
self.log_fd: TextIO
self.verbose = verbose
self.output_queue: queue.Queue[str] = queue.Queue()
self.start_time = time.time()
self.log_dir = log_dir
self.log_filename = ""
self.stop_thread: Any[threading.Event] = None
def set_log_filename(self, log_filename: str) -> None:
self.log_filename = log_filename
def get_log_filename(self) -> str:
return self.log_filename
def start(self) -> None:
if self.verbose:
print("Running: {}".format(" ".join([self.cmd] + self.args)))
atexit.register(self.stop)
if self.verbose:
print("Logging to {}".format(self.log_filename))
self.log_fd = open(self.log_filename, 'w')
self.process = subprocess.Popen(
[self.cmd] + self.args,
cwd=self.cwd,
env=self.env,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True
)
self.stop_thread = threading.Event()
self.thread = threading.Thread(target=self.process_output)
self.thread.start()
def has_started_ok(self) -> bool:
return True
def process_output(self) -> None:
assert self.process.stdout is not None
while True:
line = self.process.stdout.readline()
if not line and \
(self.stop_thread.is_set() or self.poll is not None):
break
if not line or line == "\n":
continue
line = self.add_prefix(10, self.name, line)
self.output_queue.put(line)
self.log_fd.write(line)
self.log_fd.flush()
def add_prefix(self, width: int, name: str, text: str) -> str:
return "[" + self.seconds() + "|" + name.ljust(width) + "] " + text
def seconds(self) -> str:
dt = time.time() - self.start_time
return "{: 8.03f}".format(dt)
def poll(self) -> Optional[int]:
return self.process.poll()
def wait(self, timeout_min: float) -> Optional[int]:
try:
return self.process.wait(timeout=timeout_min * 60)
except subprocess.TimeoutExpired:
print("Timeout of {} min{} reached, stopping...".
format(timeout_min, "s" if timeout_min > 1 else ""))
self.stop()
print("stopped.")
return errno.ETIMEDOUT
def get_output_line(self) -> Optional[str]:
while True:
try:
return self.output_queue.get(block=True, timeout=0.1)
except queue.Empty:
return None
def stop(self) -> int:
atexit.unregister(self.stop)
if not self.stop_thread:
return 0
returncode = self.process.poll()
if returncode is None:
if self.verbose:
print("Terminating {}".format(self.name))
self.process.terminate()
try:
returncode = self.process.wait(timeout=1)
except subprocess.TimeoutExpired:
pass
if returncode is None:
if self.verbose:
print("Killing {}".format(self.name))
self.process.kill()
returncode = self.process.poll()
if self.verbose:
print("{} exited with {}".format(
self.name, self.process.returncode))
self.stop_thread.set()
self.thread.join()
self.log_fd.close()
return self.process.returncode
def time_elapsed_s(self) -> float:
return time.time() - self.start_time
class Px4Runner(Runner):
def __init__(self, workspace_dir: str, log_dir: str,
model: str, case: str, speed_factor: float,
debugger: str, verbose: bool, build_dir: str,
rootfs_base_dirname: str):
super().__init__(log_dir, model, case, verbose)
self.name = "px4"
self.cwd = os.path.join(workspace_dir, build_dir,
rootfs_base_dirname, "rootfs")
self.cmd = "nice"
self.args = [
"--20",
os.path.join(workspace_dir, build_dir, "bin/px4"),
os.path.join(workspace_dir, build_dir, "etc"),
"-s",
"etc/init.d-posix/rcS",
"-t",
os.path.join(workspace_dir, "test_data"),
"-d"
]
self.env["PX4_SIM_MODEL"] = model
self.env["PX4_SIM_SPEED_FACTOR"] = str(speed_factor)
self.debugger = debugger
self.clear_rootfs()
self.create_rootfs()
if not self.debugger:
pass
elif self.debugger == "valgrind":
self.args = ["--track-origins=yes", "--leak-check=full", "-v",
self.cmd] + self.args
self.cmd = "valgrind"
elif self.debugger == "callgrind":
self.args = ["--tool=callgrind", "-v", self.cmd] + self.args
self.cmd = "valgrind"
elif self.debugger == "gdb":
self.args = ["--args", self.cmd] + self.args
self.cmd = "gdb"
else:
print("Using custom debugger " + self.debugger)
self.args = [self.cmd] + self.args
self.cmd = self.debugger
def clear_rootfs(self) -> None:
rootfs_path = self.cwd
if self.verbose:
print("Clearing rootfs (except logs): {}".format(rootfs_path))
if os.path.isdir(rootfs_path):
for item in os.listdir(rootfs_path):
if item == 'log':
continue
path = os.path.join(rootfs_path, item)
if os.path.isfile(path) or os.path.islink(path):
os.remove(path)
else:
shutil.rmtree(path)
def create_rootfs(self) -> None:
rootfs_path = self.cwd
if self.verbose:
print("Creating rootfs: {}".format(rootfs_path))
try:
os.makedirs(rootfs_path)
except FileExistsError:
pass
def get_output_line(self) -> Optional[str]:
line = super().get_output_line()
if line is not None:
# colorize warnings and errors
if 'ERROR' in line:
line = colorize(line, color.RED)
elif 'WARN' in line:
line = colorize(line, color.YELLOW)
return line
class GzserverRunner(Runner):
def __init__(self,
workspace_dir: str,
log_dir: str,
model: str,
case: str,
speed_factor: float,
verbose: bool,
build_dir: str,
world_name: str):
super().__init__(log_dir, model, case, verbose)
self.name = "gzserver"
self.cwd = workspace_dir
self.env["GAZEBO_PLUGIN_PATH"] = \
os.path.join(workspace_dir, build_dir, "build_gazebo-classic")
self.env["GAZEBO_MODEL_PATH"] = \
os.path.join(workspace_dir, PX4_GAZEBO_MODELS)
self.env["PX4_SIM_SPEED_FACTOR"] = str(speed_factor)
self.cmd = "stdbuf"
self.args = ["-o0", "-e0", "gzserver", "--verbose",
os.path.join(workspace_dir,
PX4_GAZEBO_WORLDS,
world_name)]
def has_started_ok(self) -> bool:
# Wait until gzerver has started and connected to gazebo master.
timeout_s = 20
steps = 10
for step in range(steps):
with open(self.log_filename, 'r') as f:
for line in f.readlines():
if 'Connected to gazebo master' in line:
return True
time.sleep(float(timeout_s) / float(steps))
print("gzserver did not connect within {}s"
.format(timeout_s))
return False
def get_output_line(self) -> Optional[str]:
line = super().get_output_line()
# Some gazebo versions don't seem to reset the color, so always add a RESET
if line is not None:
line = line + color.RESET.value
return line
class GzmodelspawnRunner(Runner):
def __init__(self,
workspace_dir: str,
log_dir: str,
model: str,
case: str,
verbose: bool,
build_dir: str):
super().__init__(log_dir, model, case, verbose)
self.name = "gzmodelspawn"
self.cwd = workspace_dir
self.env["GAZEBO_PLUGIN_PATH"] = \
os.path.join(workspace_dir, build_dir, "build_gazebo-classic")
self.env["GAZEBO_MODEL_PATH"] = \
os.path.join(workspace_dir, PX4_GAZEBO_MODELS)
self.cmd = "gz"
if os.path.isfile(os.path.join(workspace_dir,
PX4_GAZEBO_MODELS,
self.model, self.model + ".sdf")):
model_path = os.path.join(workspace_dir,
PX4_GAZEBO_MODELS,
self.model, self.model + ".sdf")
else:
raise Exception("Model not found")
self.cmd = "stdbuf"
self.args = ["-o0", "-e0",
"gz", "model",
"--verbose",
"--spawn-file", model_path,
"--model-name", self.model,
"-x", "1.01", "-y", "0.98", "-z", "0.83"]
def has_started_ok(self) -> bool:
# The problem is that sometimes gzserver does not seem to start
# quickly enough and gz model spawn fails with the error:
# "An instance of Gazebo is not running." but still returns 0
# as a result.
# We work around this by trying to start and then check whether
# using has_started_ok() whether it was successful or not.
timeout_s = 20
steps = 10
for _ in range(steps):
returncode = self.process.poll()
if returncode is None:
time.sleep(float(timeout_s) / float(steps))
continue
with open(self.log_filename, 'r') as f:
for line in f.readlines():
if 'An instance of Gazebo is not running' in line:
return False
else:
return True
print("gzmodelspawn did not return within {}s"
.format(timeout_s))
return False
class GzclientRunner(Runner):
def __init__(self,
workspace_dir: str,
log_dir: str,
model: str,
case: str,
verbose: bool):
super().__init__(log_dir, model, case, verbose)
self.name = "gzclient"
self.cwd = workspace_dir
self.env = dict(os.environ, **{
"GAZEBO_MODEL_PATH":
os.path.join(workspace_dir, PX4_GAZEBO_MODELS)})
self.cmd = "nice"
self.args = ["--19",
"gzclient",
"--verbose"]
class TestRunnerMavsdk(Runner):
def __init__(self,
test_dir: str,
log_dir: str,
model: str,
case: str,
mavlink_connection: str,
speed_factor: float,
verbose: bool,
build_dir: str):
super().__init__(log_dir, model, case, verbose)
self.name = "mavsdk_tests"
# pytest picks up pytest.ini and conftest.py from here, and the
# case is a node ID relative to it.
self.cwd = test_dir
self.cmd = "nice"
self.args = ["--18",
sys.executable, "-m", "pytest",
"-v",
# Print output live instead of only on failure.
"-s",
"--url", mavlink_connection,
"--speed-factor", str(speed_factor),
case]
self.env["PYTHONUNBUFFERED"] = "1"
class TestRunnerRos(Runner):
def __init__(self,
workspace_dir: str,
log_dir: str,
model: str,
case: str,
verbose: bool,
ros_package_build_dir: str):
super().__init__(log_dir, model, case, verbose)
self.name = "integration_tests"
self.cwd = workspace_dir
self.cmd = "nice"
self.args = ["-17",
os.path.join(
ros_package_build_dir,
"integration_tests"),
"--gtest_filter="+case, "--gtest_color=yes"]
def get_output_line(self) -> Optional[str]:
line = super().get_output_line()
if line is not None:
# colorize assertion failures & errors
if 'Failure' in line or '[ERROR]' in line or '[FATAL]' in line:
line = colorize(line, color.RED)
return line