* feat(mavlink): serve packed parameters as @PARAM/param.pck
QGC already downloads ArduPilot parameters as a packed FTP file, which
is much more reliable on lossy SiK links than the PARAM_VALUE firehose.
Serve the same format from PX4 so a hash miss can use that path.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(mavlink): serialize the full message when resending a cached FTP reply
_last_reply caches only the header, PayloadHeader and four data bytes,
but the resend path handed that 19-byte array to the serializer as a
complete mavlink_file_transfer_protocol_t, which reads 254 bytes and
transmits whatever follows the cache. The decoded request is dead once
the resend decision is made, so expand the cache into it instead of
adding another message to the receiver thread's stack. Clear the unused
payload before caching so a resent NAK matches the original.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* test(mavsdk_tests): download parameters via MAVLink FTP
Fetch @PARAM/param.pck through MAVSDK's burst download on SIH and check
every entry against the PARAM_VALUE stream, so the packed format and the
FTP server's session, size and EOF handling stay covered by the SITL
tests that QGC and MAVSDK both rely on.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(mavlink): cache FTP Open ACKs whose size LSB is 5
_reply skipped the compact cache whenever data[0] was kErrNoSessionsAvailable. On a NAK that is the error code; on an Open ACK it is the file size LSB. A lost Open ACK for param.pck of that length was retried as a second Open and failed.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* feat(tools): download packed parameters over MAVFTP
Host-side check of @PARAM/param.pck against the PARAM_VALUE stream.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(mavlink): pad so a defaulted value does not straddle an FTP block
pack() kept the last 4 bytes of an entry in one chunk. With withdefaults those bytes are the default, so the value can still split. QGC always requests defaults; a hole-fill then retries one chunk with a live param_get().
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(mavlink): snapshot param.pck at open so a param_set cannot corrupt it
Pack the used set once into a heap buffer. Reads memcpy that snapshot, so a value change during the download cannot shift later entries or splice two generations across a retried FTP block.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(mavlink): freeze param.pck membership instead of snapshotting the file
Keep which parameters appear and whether each includes a default in bitsets (~1 KB). Pack values live. Layout cannot shift mid-download; a retried block cannot splice a number.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* style(mavlink): format MavlinkFtpParamTest.cpp
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* refactor(mavlink): keep FTP work buffers as members
@PARAM/param.pck is a connect-time path, not the rare path the lazy new[] was for. send() could free the buffers after 2 s of no request while a burst was still running. Fold them into the instance (495 B) and always idle-close the session at 30 s.
A new Open takes over the single session so a lost Terminate on a slow link does not NAK until that close.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(tools): make mavftp_param.py reliable on a radio link
pymavlink applied FTP timings after ResetSessions, wrote param.pck?withdefaults=1 into cwd, and bumped the session id on Open retry (PX4 only ACKs session 0). Set radio timings at construction, pin session 0, retry a missed Open, and treat a complete decode as success.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* feat(mavlink): accept packed parameter FTP uploads
CreateFile/Write/Terminate of @PARAM/param.pck applies the ArduPilot
packed stream so a GCS can bulk-load without a PARAM_SET round trip per
value. Unknown and read-only names are skipped; Terminate NAKs a
truncated file or the with-defaults magic.
* test(mavsdk_tests): upload MPC_XY_P via packed param FTP
Pin the SIH upload case to a known float instead of whatever
get_all_params() returns first.
---------
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
On NuttX these are two things: / is the filesystem root, and /fs/microsd is the
SD card mounted under it. MAVLink FTP serves the root and confines writes to the
SD card, so the read-only ROMFS at /etc is visible but cannot be written.
On POSIX they were the same directory. PX4_ROOTFSDIR and PX4_STORAGEDIR both
resolved to CONFIG_BOARD_ROOT_PATH, which on SITL is ".", so the working
directory was simultaneously the FTP root and the only writable area. The
consequences were that the ROMFS symlink sat inside the FTP root, and that
_validatePathIsWritable() had nothing meaningful to check against and so was
compiled out on POSIX entirely, leaving no write restriction at all.
Give POSIX the same split. CONFIG_BOARD_ROOT_PATH keeps its meaning as the
storage directory, and a new CONFIG_BOARD_FS_ROOT_PATH names the root FTP
serves, defaulting to the storage path so every existing board is unchanged.
SITL sets the root to "." and storage to "./fs", which mirrors NuttX: logs,
parameters, dataman and eeprom move under ./fs, and etc/ stays in the root as
read-only data.
With storage distinct from the root, the write restriction now applies on every
platform rather than NuttX only, and no longer compares against a hardcoded
prefix length that was wrong for any board not using /fs/microsd.
Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Julian Oes <julian@oes.ch>
The bash version spent most of its lines on plumbing: parsing a report
back out of a per-commit `brew ruby` child, awk and herestrings for set
logic, bash 3.2 workarounds, and a trap to restore the tap. The job
itself is small and lives inside Homebrew anyway, so run the whole thing
under Homebrew's Ruby and call its tap, formula and bottle APIs directly.
One process instead of one Homebrew startup per candidate commit, the
tap checkout restored by ensure, tarballs checked with Net::HTTP, and no
second language embedded in a string. Formulary.clear_cache and
Tap#clear_cache between checkouts keep each candidate's formulae fresh.
GITHUB_OUTPUT still arrives because brew forwards GITHUB_* whenever CI
is set. Same walk, same exemption for formulae unbottled at the pin,
same outputs; verified to pick the same commit as the bash version.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
homebrew-core moved to protobuf 36.0 and abseil 20260817.0, and the
macos-15 runner image (20260828) now ships that snapshot. The gz bottles
at the pinned commit c90f81ac were built against protobuf 35.1, so the
gz_bridge build fails on every run since today with
gz/msgs/details/discovery.pb.h:17:2: fatal error: "Protobuf C++
gencode is built with an incompatible version of"
on main as well as on this PR. 43aee9cc is tap HEAD, where OSRF has
rebuilt every gz-harmonic dependency for protobuf 36.0 and abseil
20260817.0 (the gz-msgs10 10.4.0_2 bottle's INSTALL_RECEIPT records
both). Tools/ci/refresh_gz_tap_pin.sh picked this commit.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The weekly refresh workflow carried its logic inline, compared only the
tap SHA against the pin, and duplicated the --sim-tools package list.
`brew install --dry-run` never prints "from source", so its bottle check
could not fire.
Tools/ci/refresh_gz_tap_pin.sh now owns the job and the workflow calls
it. It reads the osrf/simulation formulae from macos.sh, walks the tap
from HEAD back to the pin, asks Homebrew at each commit whether every
formula in the runtime closure has a bottle for the host, HEAD-requests
each tarball, and rewrites the pin to the first commit that passes.
Formulae with no bottle at the current pin (the gz-harmonic
meta-formula, which OSRF never bottles) are not required.
peter-evans/create-pull-request opens or updates the bot PR from the
script's outputs, so an unmerged bump is refreshed instead of duplicated.
The job runs on macos-15, the oldest macOS in compile_macos.yml, because
Homebrew pours an older macOS bottle onto a newer one, not the reverse.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
macos.sh only pinned osrf/simulation when --ci was passed, so dev
machines still hit OSRF's bottle-gap window and built Gazebo from
source. Apply it whenever --sim-tools runs.
The pin is a point-in-time SHA with nothing to advance it, so add a
weekly workflow that checks whether osrf/simulation HEAD is fully
bottled and opens a PR bumping gz-tap-pin.txt when it is.
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: FARHANG <farhang.nba@gmail.com>
* fix(uploader): shorten the wait for a bootloader after reboot
Reboot-to-bootloader is immediate, so this wait only has to cover USB
re-enumeration, not the reboot. Five seconds of it meant a port that was
never going to answer held up every other port on the list, and it could
outlast the window it was trying to catch: a board sits in its bootloader
for BOOTLOADER_DELAY, 3s on some boards and 5s on most, before it jumps to
the application.
With an FMU-v6C and a Black Magic probe attached, a pass over the probe's
two CDC nodes drops from 14.75s to 5.45s, and a running application is in
its bootloader 3.52s after the uploader starts.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Julian Oes <julian@oes.ch>
* fix(uploader): report ports by their /dev/serial/by-id name
A ttyACM number is handed out in plug order and says nothing about which
device it belongs to, which is awkward as soon as there is more than one
CDC device on the bus -- "Attempting reboot on /dev/ttyACM0" gives no hint
that it is a debug probe rather than the board.
Resolve each detected port back to its by-id symlink when one exists, so
every message names the device. This also collapses a duplicate: a board
is matched both by a by-id pattern and by the /dev/ttyACM* catch-all, so
it was being probed twice per pass. With an FMU-v6C and a Black Magic
probe attached, detection goes from four entries with two naming the same
board to three, each identifying itself.
Linux only; elsewhere, and for ports with no symlink, paths are unchanged.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Julian Oes <julian@oes.ch>
---------
Signed-off-by: Julian Oes <julian@oes.ch>
* fix(ci): attach FMU bootloader .bin to GitHub releases
Releases only shipped `_bootloader.px4`, the USB uploader envelope. SWD recovery needs the raw `.bin` at 0x08000000.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(ci): do not attach unused bootloader .px4
USB flashing would write it into the application slot. SWD uses the raw .bin.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(ci): omit bootloader SBOMs from GitHub releases
GitHub Releases glob artifacts/**/*.sbom.spdx.json. Bootloader metadata dirs are not a recovery artifact.
*_bootloader_* variants (e.g. bootloader_secureboot) stay omitted: those images are baked with in-tree test keys.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* docs(bootloader): point SWD recovery at the release .bin
Match the cannode pre-built blurb. The .px4 envelope is not the SWD image.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* docs(bootloader): scope release .bin to SWD programmers
gdb load needs ELF program headers. The release image is a raw .bin for ST-Link / CubeProgrammer / OpenOCD, and only in-tree *_bootloader targets are attached.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
---------
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(ci): publish flashable CAN node firmware
Release packaging only collected .px4 files, which the DroneCAN bootloader cannot flash.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(ci): attach canbootloader .bin as the SWD image
The cannode bootloader is a raw .bin for st-flash, not a .px4 envelope.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(ci): drop cannode.zip from release assets
The target-named .uavcan.bin and _canbootloader.bin files are the flashable images; the zip only duplicated them.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
---------
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* feat(sensors): decouple sensor internal/external classification from the bus
A sensor's internal/external classification was derived from its bus
(px4_i2c_bus_external / px4_spi_bus_external), but a bus is a wire: one bus
routinely serves both chips soldered on the FC and a pinned-out connector, and
any onboard sensor on such a shared bus was classified external. That hands an
onboard mag an operator-settable rotation instead of the board rotation, feeds
an onboard baro's self-heated die temperature into air density as if it were
ambient, and inverts the 75/50 default priority. FMU-v6C and AirBrainH743
worked around it with hand-rolled device_id whitelists behind
BOARD_OVERRIDE_I2C_DEVICE_EXTERNAL; FMU-v6XRT's second onboard baro was simply
misclassified.
Classify the device instead of the bus: drivers publish is_external in
sensor_accel/gyro/mag/baro, derived from the device id by default and
overridden by the new -O start flag ("onboard") for onboard sensors that share
a bus with an external connector. -I/-X stay pure bus probe filters.
The question "is this sensor external" had four answers. It now has one, with
a single override point:
-O -> I2CSPIDriverConfig::external -> Device::set_external() and the
PX4* wrappers -> is_external in the sensor topic
px4_i2c_device_external() was px4_i2c_bus_external() with a device id decode in
front, and calibration::DeviceExternal() forwarded to device_is_external();
both are gone. Device::external() stops being a bus query: it is non-virtual,
defaults to device_is_external(), and takes the declared value through
set_external(), so the five I2C/SPI overrides that used to answer it from the
bus are deleted and cannot diverge again. device::I2C and device::SPI set it
from the config, so every driver built on the bus framework follows -O without
doing anything. px4_i2c_bus_external() and px4_spi_bus_external() survive as
what they honestly are - bus predicates - reachable only through the fallback.
-O is opt-in per driver (BusCLIArguments::support_onboard), the same way -k is:
a flag that every driver advertised but only a handful honoured would be a
silent no-op on the rest. This also drops the special case for the mcp23009 and
mcp23017 GPIO expanders, which use -O for their output state and simply do not
opt in.
sensor_gyro_fifo carries is_external too. VehicleAngularVelocity prefers the
FIFO topic for any IMU that publishes one, so without it the rate controller
would take its rotation from the bus while VehicleIMU took it from the topic -
the same chip, two classifications.
The BOARD_OVERRIDE_I2C_DEVICE_EXTERNAL hook is removed along with both board
implementations, replaced by -O on the affected rc.board_sensors lines.
FMU-v5x, MR-CANHUBK3, KakuteF7, NXT-Dual and MicoAir H743-Lite each start an
onboard barometer on an external bus and get the flag as well.
Assisted-by: Claude:claude-fable-5, Claude:claude-opus-5[1m]
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* feat(sensors): treat shared buses as first-class topology
A boolean internal/external flag cannot describe a bus that carries both
hard-mounted chips and a connector. -O was an opt-in override for that
case, so most drivers silently ignored it and classification still
followed the bus.
Declare Internal / External / Shared on the bus, probe External and
Shared, and classify each sensor from -I/-s vs -X/-S.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(ci): drop stale ITCM symbols and clang-tidy errors
ITCM lists still named calibration::DeviceExternal and
px4_spi_bus_external after both were removed. The host test stubs
forwarded varargs in a way the analyzer rejected, and stripped I2C
headers kept extra trailing newlines.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* fix(sensors): inline device_is_external into the header
Reaching it through drivers__device pulled the kernel-only library into
the userspace image of a protected build; giving it a library of its own
put an archive referencing px4_i2c_buses/px4_spi_buses after the board
library that defines them. Inlining sidesteps both.
* fix(sensors): compile tcbp001ta and probe canhubk3 GPS mag
tcbp001ta is not an I2CSPIDriver, so config.external does not exist; it
only ever starts on internal SPI. -X on canhubk3 I2C2 never ran because
that bus is Internal.
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
---------
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
ensure_env() sent the child's stdout to DEVNULL to keep this process's stdout
parseable, but the generator reports missing imports with print(), so that was
exactly the output being discarded. What survived was a CalledProcessError
whose message repeats the argv, and the argv carries every .msg path: 9901
bytes on one line, naming no cause and no remedy.
Capture both streams and, on failure, exit with the child's own output. The
message names the command that failed and its exit code rather than asserting
what kind of failure it was, so it stays accurate whatever the child is or why
it failed. Missing empy now reports 167 bytes:
loadconfig: px_generate_zenoh_topic_files.py failed (exit 1)
Failed to import em: No module named 'em'
You may need to install it using:
pip3 install --user empy
stdout stays clean on success, so callers that parse it are unaffected.
Smoke tested on Ubuntu 24.04, matching the runner base: happy path returns
valid JSON for --group and --group --seeders; missing empy and missing
pyros-genmsg each report the cause and remedy; an unrunnable generator reports
the interpreter's own "can't open file"; a child exiting non-zero with no
output reports "(no output)" rather than an empty message; and a preset
ZENOH_KCONFIG_TOPICS still skips the generator entirely.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
A .px4board line whose value does not fit its symbol's type is not an error to
kconfiglib. It warns, drops the assignment, and leaves the symbol at its
default, so the loaded config quietly differs from the file and no consumer
downstream can tell. corvon 743v2 built with the wrong UAVCAN interface count
this way, and the only trace was a warning in a job that exited 0.
load_target_config() now compares kconf.warnings before and after the load and
exits on any "assignment ignored", naming the file and quoting the warning.
Checked against the whole tree: 293 targets load with none rejected, so this
does not fail existing boards. Reintroducing the quoted value makes it exit 1
with the file path and the reason.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* feat(gps): add u-blox u-center mode on UART2
GPS_UBX_BAUD2 alone never reached the receiver: outside the RTCM and GCS
modes the driver leaves UART2 untouched, so attaching a USB-serial adapter
to it produced no traffic. GPS_UBX_MODE 7 configures UART2 as a UBX
diagnostic port while the autopilot keeps its session on UART1.
* chore(gps): bump PX4-GPSDrivers submodule
Pulls in the u-center UART2 mode from PX4-GPSDrivers#228.
* docs(docs): prettier and cross link
* docs(docs): subedit
* chore(gps): bump PX4-GPSDrivers submodule
Picks up the info-level log for the u-center UART2 configuration, on top
of the CFG-UART1/2-ENABLED key ID fix now merged upstream.
* feat(gps): log what each UART carries after configuration
The line reported the link speed but not what the receiver was actually
configured to do with the port. Bumps PX4-GPSDrivers for the UART1 role
lookup and the matching per-port lines on UART2.
* feat(tools): add ubx_monitor.py, a live view of a u-blox serial link
Autobauds, then reports what the receiver is actually sending: link
utilisation, MON-VER identity, NAV-PVT fix state and per-message rates.
Written to check the u-center diagnostic port, useful for any GPS UART.
* chore(gps): point PX4-GPSDrivers at main
The u-center UART2 mode merged upstream (#228); track main instead of the
development branch.
---------
Co-authored-by: Hamish Willee <hamishwillee@gmail.com>
When homebrew-core bumps a shared dependency, OSRF's bot bumps each gz
formula's revision and deletes its bottle block within minutes of the upstream
merge. The matching rebuild is a separate, unautomated step: across 2026 the
gap ran 0.5 to 11 days, median 2.5, leaving gz unbottled roughly 18% of the
time. In that window macos.sh compiles gz-sim8, gz-msgs10 and five others from
source and the setup step goes from ~8min to 20-30min.
The removal only edits the formula. The bottle tarballs are never deleted from
OSRF's S3 bucket, so pinning to the last fully bottled revision keeps installs
binary while upstream catches up.
Gated behind a new --ci flag rather than applied unconditionally. Automation
wants a reproducible, fast install; development machines want the current
formulae and can absorb a source build, and should not have their tap left on
a detached HEAD as a side effect of running the setup script.
Only the gz tap is pinned. opencv@4, protobuf, gstreamer and the rest still
track homebrew-core, so CI keeps catching upstream formula breakage on every
PR, which is the reason --sim-tools runs there in the first place.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The Dockerfile only ever copied requirements.txt, so optional-requirements.txt
was never installed into the image even though it sits in the build context.
Every CI job needing pyelftools therefore fetched it from PyPI at job time and
failed whenever files.pythonhosted.org returned 502s.
symforce publishes no Linux aarch64 wheel and no sdist, so installing the file
unguarded would break the linux/arm64 image build. Gate it with an environment
marker so aarch64 installs pyelftools only.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* ci(build-all): seed ccache only for chip families with cold caches
Measured across recent runs, the seeder stage contributes nothing in
steady state: all 264 build-job cache restores in the 8 most recent
successful runs came from group-level caches, zero from seeder caches,
yet the stage gates the build matrix for ~3.4 min of every run's
critical path and burns ~22 8cpu-runner-minutes warming caches nobody
reads. Its fail-fast value is similarly thin: 2 of the last 40 failed
runs failed at seed stage; the other 34 passed seeding and failed in
the matrix anyway.
Probe each family's ccache namespace in the RunsOn cache bucket from
the group_targets job and emit only cold families into the seed
matrix. Warm runs skip the stage entirely (the matrix starts right
after the scan); a cold family still seeds first and gates the matrix,
which is the one case the warmup pays for. The probe fails open: if
the bucket env, the listing, or the assumed object layout is wrong,
every family reads as cold and the workflow behaves exactly as before.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* ci(build-all): fail open when seeder probe output is unparseable
The has_cold computation ran unguarded under bash -e: malformed probe
output would fail the whole workflow instead of falling back to the
full seeder matrix. Guard it so every failure path degrades to seeding
everything, on RunsOn and on downstream forks without it alike.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* ci(build-all): move seeder probe logic into the helper script
The probe step carried fail-open shell logic inline in the workflow.
Move output writing, has_cold computation, and all fail-open handling
into filter_cold_seeders.py; the workflow step is now a single script
invocation. The script always exits zero and degrades every failure
(missing bucket env, aws error, unparseable input) to the full seeder
matrix.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* ci(build-all): probe the magic cache's real object layout
The first probe guessed cache/{key} for the magic-cache object layout
and read every family COLD, degrading to unconditional seeding. The
real layout is cache/v1/{org}/{repo}/{ref}/{version-hash}/{cache-key};
the version hash is the actions/cache digest and not knowable a
priori, so list the default-branch scope once (main-scope caches are
visible to every ref) and match cache-key basenames. Verified locally
against the live bucket: all 10 families read warm and the seed stage
skips.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* ci(build-all): single-source ccache key roots in the matrix generator
The ccache key convention was encoded three times: hand-assembled in
the workflow's restore/save steps, again in the seeder's
cache-key-prefix argument, and a third time in the probe script. Drift
between them is silent thanks to the probe's fail-open design: a
renamed key scheme would read every family COLD forever and quietly
revert to unconditional seeding.
Mint the namespace root once in generate_board_targets_json.py as
cache_prefix on every group and seeder matrix entry, with seeders
occupying the reserved 'seeder' group inside their family namespace.
The workflow composes {cache_prefix}-{group}-{ref}-{sha} from matrix
fields only, and the probe greps the root it is handed, knowing
nothing of the format. Composed keys are byte-identical to the old
scheme (asserted for all 33 groups and 10 seeders), so no cache is
invalidated.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
---------
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The pynacl install step from #27237 was a stopgap until the build
container shipped pynacl preinstalled. That happened in #27237 itself:
it added pynacl to Tools/setup/requirements.txt, which dev_container.yml
bakes into the image, so v1.18.0-beta1 and later already ship it. Only
the v1.17.0-rc2 pin here kept the step alive: 30+ unpinned PyPI pulls
per workflow run that already cost us a matrix job to PyPI 502s.
Verified by building px4_fmu-v6x_secureboot locally in
px4io/px4-dev:v1.18.0-beta2: the signing step runs against the baked-in
pynacl with no pip install. The voxl2 container keeps its own pin and
needs no fallback since only CONFIG_BOARD_SECUREBOOT targets (NuttX
platform only) invoke the signing script.
Expect one cold ccache run after merge: the new container's compilers
invalidate all caches at once via compiler_check = content.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The voxl2 build group carried chip_family 'native' while its seeder
carried 'voxl2', so the group restored/saved under
ccache-native-x64-voxl2-0-* while the seeder saved under
ccache-voxl2-x64-seeder-*. The restore-keys chain can never cross that
namespace gap: the seeder's warm cache is discarded every run, and cold
builds fall back to the SITL seeder cache (host gcc objects, zero
possible hits against aarch64-linux-gnu-gcc or hexagon-clang). Present
since #27050.
Give the voxl2 group chip_family 'voxl2' so it shares the seeder's key
namespace, drop the now-redundant voxl2 special-casing in seeder
generation, and size the voxl2 cache at 800M since it holds objects for
two toolchains (aarch64-gnu + hexagon-clang) and already sits near the
400M limit.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
macos.sh --sim-tools installs the unversioned Homebrew opencv formula, which
is now 5.0.0, and PX4-OpticalFlow does not build against it:
klt_feature_tracker/src/trackFeatures.cpp:43:10:
fatal error: 'opencv2/core/types_c.h' file not found
PX4-OpticalFlow/src/flow_opencv.cpp:110:7:
fatal error: no member named 'undistortPoints' in namespace 'cv'
types_c.h was removed in OpenCV 5, and undistortPoints moved when calib3d was
split into 3d/calib. The failure lands about a thousand targets into
make px4_sitl and points at submodule sources rather than at the dependency.
ubuntu.sh takes libopencv-dev from apt, which is still 4.x, so CI never sees
this.
Install opencv@4 instead. It is keg-only, so find_package still resolves to
5.0.0 without a prefix hint. Add one alongside the qt@5 hint from 9c2e634325,
which is keg-only for the same reason.
Assisted-by: Claude:claude-opus-5[1m]
Signed-off-by: Jackson Korba <jackson.korba@gmail.com>
Terminal emulators send DEL (0x7f) for the Backspace key, but only BS
(0x08) was handled, so the keypress fell through to the catch-all branch
and got appended to the line buffer instead of erasing. This made the
shell effectively append-only.
Also drop the stray echo of the backspace character: erase_last_n_chars()
already emits ESC[1D + ESC[K and leaves the cursor in the right column,
so writing 0x08 on top of it moved the cursor one column too far left and
the next keystroke overwrote the preceding character.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Julian Oes <julian@oes.ch>
* fix(tools/uploader): rediscover CDC port after reboot-to-bootloader
After MAVLink/NSH reboot-to-bootloader the USB CDC node re-enumerates
under a different product string (application vs bootloader). The
uploader kept reopening the original path, so identify raced a vanished
node and only recovered on a later outer-loop rescan — when it recovered
at all.
Close the flight-stack port after sending reboot, wait for re-enumeration,
rebind to whichever matching port appears (preferring anything other than
the path we left), and only then run identify. Exact --port paths that do
not currently exist are no longer treated as live candidates.
* fix(tools/uploader): harden port rediscovery after reboot-to-bootloader
Send the reboot sequence at every configured flight-stack baud rate:
writes at a wrong baud succeed silently on real UARTs, so stopping at
the first "successful" baud never reached the second one. Rank
post-reboot candidates instead of excluding the previous path (the OS
often re-uses the tty number for the bootloader), try newly appeared
nodes first so a pinned app path that vanished with the product string
can still be found, collapse by-id symlinks onto their targets, and
spend one identify attempt per node per pass so a silent wrong device
cannot eat the rediscovery window. Exact-port existence checks only
apply on POSIX (os.path.exists("COM10") is false even when present),
and reboot commands honor --use-protocol-splitter-format again, which
the v2 rewrite had left wired to nothing.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
* perf(tools/uploader): load firmware once instead of every retry cycle
The wait-for-bootloader loop re-parsed and re-decompressed the full
image on every ~50 ms retry.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
---------
Signed-off-by: Jacob Dahl <dahl.jakejacob@gmail.com>
Add a host-side utility that reads strong undefined dynamic symbols
from an ELF. Validate imports against symbols supplied by provider
files and ELFs, and reject imports matching configured ownership
prefixes.
Assisted-by: OpenAI:Codex
Since b11e615810 src/modules/zenoh/Kconfig sources the generated topic
catalog through ZENOH_KCONFIG_TOPICS, which only cmake exports, so every
standalone Kconfig parse crashed and build_all_targets produced an empty
build matrix on every run while the scan job still reported success.
Add Tools/kconfig/loadconfig.py as the single entry point for parsing
the Kconfig tree outside cmake: it generates the topic catalog exactly
like cmake/kconfig.cmake and provides shared board/target enumeration
and config loading. Port generate_board_targets_json.py (CI matrix) and
updateconfig.py (make updateconfig) to it; both were independently
broken and each reimplemented board scanning on its own.
Harden the workflow so the next failure of this class is loud: run the
generator as a standalone assignment, so bash -e fails the step (the
previous echo "matrix=$(...)" form masked the exit code), and write
the JSON outputs through the heredoc GITHUB_OUTPUT form.
Matrix output verified byte-identical to the pre-b11e6158101 baseline
in all three modes; updateconfig output verified identical against the
same baseline.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Homebrew 6.0+ refuses to load formulae from untrusted third-party taps. This adds a trusted block for OSRF taps.
Signed-off-by: Cade Andersen <cadecandersen@gmail.com>
* feat(navigator): extend detect and avoid module to follow regulatory standards such as ASTM F3442
* docs(docs): minor subedit
* refactor(navigator): reduce flash by grouping notif into same events
* docs(daa): Improve docs readability with ::: details blocks
* fix(navigator): single event when on ground with conflict
* refactor(boards): increase flash length from 4M - 128k to 5M - 128k
* rework(general): define DAA standard at build with new CONFIG_NAVIGATOR_ADSB_F3442
* clean(navigator): minor changes to clean the PR
* feat(boards): add CONFIG_NAVIGATOR_ADSB_FAKE_TRAFFIC in visionTargetEstStatic.px4board
* rework(daa): reduce amount of abstractions and minor cleaning
* refactor(boards): allyes revert flash length to 4M-128K
* refactor(boards): allyes increase flash length to 5M-128K
* rework(daa): rework event notifications to improve clarity
* docs(docs): move details inside of ::: details block
* docs(docs): run npx prettier
* refactor(daa): avoid void mutator functions
* docs(docs): Improve Python helper to decode daa unique id
* rework(daa): move encoded id handling to the adsb lib and refactor on_active
* refactor(daa): naming and zero init
* fix(daa): move dataman dep from daa level to unit test level
* refactor(daa): define common daa_input and rework process_transponder_report
* fix(daa): remove stale todo
* refactor(daa): rename crosstrack_based_daa to crosstrack_standard
* refactor(daa): rename F34_ params to DAA_
* docs(docs): revert changes to autogenerated docs
* docs(docs): Add Detect And Avoid in index.md
* refactor(daa): update message on init failed
* refactor(daa): only publish most_urgent conflict once in on_active()
* refactor(daa): move conflict buffer handling in the adsb lib
* refactor(daa): move notifications into new class ConflictNotifier and only notify once per cycle
* refactor(daa): uninit _cycle_changes to store into .bss (and save flash)
* docs(docs): remove unused image link
* refactor(daa): move automated action policy to the adsb lib
* refactor(daa): move self detection into adsb lib as DaaTrafficFilter
* refactor(daa): minor changes in unit tests
* refactor(daa): merge DaaTrafficFilter and DaaEncoding
* fix(daa): fix CI by removing navigator from the DAA deps
* refactor(daa): reduce stack size
* fix(daa): advertise _fake_traffic_pub in constructor
* refactor(daa): Comments only, simplify and reduce comments
* fix(daa): brief comment missing end star
* Disable CONFIG_NAVIGATOR_ADSB in ark_fpv_default board
* docs(docs): General clarifications and remove the 1.18 badge
* refactor(daa): cleaning
* refactor(daa): minor cleaning
* refactor(adsb): simplify conflict tracker assuming push_back cannot fail
* build(cmake): cleaner fix to protect regex alternations
* refactor(daa): Crosstrack, remove unit test requiring finite yaw (yaw not used by standard)
* refactor(daa): minor cleaning and fix callsign to uint64
---------
Co-authored-by: jonas <jonas.perolini@rigi.tech>
Co-authored-by: Hamish Willee <hamishwillee@gmail.com>
The --viewer path (tee MAVLink frames to UDP and enable the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams so Hawkeye could
render an SIH flight live) was never validated end to end: Hawkeye
never actually rendered from it. Rather than ship an unproven feature,
drop it.
Removes px4bench.attach_viewer_tee (and its function-local socket use),
the flight_mission --viewer/--viewer-port args, the viewer tee call and
the stream-enable block, and the now-unused --board-dev arg that only
fed those stream commands. The SIH flight logic, firmware gate,
storage/serial tests, and the arming gate are unchanged. README and the
bench testing docs page drop the Hawkeye section and all --viewer
mentions.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The full-suite persistence check failed on hardware with 'after reboot
SDLOG_UTC_OFFSET = 740 (expected 777)'. Firmware persistence is solid;
this was a test race. A PARAM_SET propagates through the param system
asynchronously, and phase_persistence ran 'param save' on the strength
of the PARAM_VALUE echo alone. When save raced ahead of the commit it
persisted the PRIOR value (740, the last value Phase 2 wrote). The echo
was not even proof of the new value: wait_param_echo returned
seen=[740, 777], a stale queued PARAM_VALUE from Phase 2 having slipped
past drain_param_values.
Make persistence deterministic with two gates instead of timing:
1. After setting the marker, read it back from the board until it reports
MARKER_VALUE (read_until) before saving. This confirms the value is
committed to RAM and trusts the board's read over any stale echo.
2. After 'param save', confirm the saved state with 'param show <name>'
and require the '+' (saved) flag AND the marker value before rebooting
(retry save once first). '*' means unsaved; the test never reboots on
an unsaved marker. Flag columns per src/systemcmds/param/param.cpp:822
(x used, + saved, * unsaved, l locked).
phase_set_readback now also trusts read_until over the echo, so the same
stale-queued-echo does not fail it either; its pass criteria stay the
readback match. Restores in phase_persistence and the final cleanup use
read_until too.
New shared helpers in px4bench.params: read_until(mav, name, expected,
timeout) -> (ok, last_seen), and param_is_saved(shell, name) plus its
parse_param_show() parser for the 'param show' saved flag and value.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
A hardware run exhausted the board's task/fd table after a few probe
cycles: shell open failed while heartbeat still worked, and continued
writes eventually stalled the link. Not a firmware hang, a resource leak
the tooling drove.
The firmware spawns a new nsh task plus two pipes (four fds) on the
first SERIAL_CONTROL write of a session (Mavlink::get_shell) and only
frees them when a SERIAL_CONTROL message arrives with the RESPOND flag
cleared (Mavlink::close_shell). Two tooling bugs combined to leak them:
- MavlinkShell.open() returned True on any buffered byte, including
leftover output from a prior session, so it reported a live shell
without one. It now confirms an actual 'nsh>' prompt or a completed
echo-sentinel round-trip before returning True, keeping the same
timeout-and-report-failure behavior.
- open/close cycles reopened before the single flags=0 teardown was
processed, spawning a second shell before the first was freed. close()
now sends the flags=0 teardown and then briefly drains incoming
SERIAL_CONTROL until quiet (bounded, never a hang) so the firmware
runs close_shell before a caller can reopen.
Lifecycle: every MavlinkShell user now opens one session, runs all its
work on it, and closes it in a finally so no error path leaks a shell.
The SIH flight and storage tests run all their probes on a single shell
(shell_command_exists already takes an open shell); flight_mission no
longer opens a throwaway probe shell separate from the flight session
beyond the unavoidable reboot boundary in enter_sih.
Folds in the sentinel fix (run() sends the echo sentinels on their own
input line, because nsh aborts a ';' chain when a command fails, which
previously lost the sentinel for a failing probe and read as a stall).
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The flight test is now part of the default suite; both the bench README
and the bench testing guide said it ran separately. Update the sequence
description, the risk map and suite tables (storage_stress and
flight_mission rows, serial_loopback for fixtures), the SIH sections
(probe-and-skip, the typed 'arm' confirmation, --allow-arming for
automation), the production section (--allow-arming on the line,
serial_loopback with a jumper on the fixture), and the firmware gate
--build description (bench capability report with variant and config
line recommendations). Storage threshold defaults are justified where
the flags are documented.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The firmware already ships sd_bench, sd_stress, and serial_test; the
suite was not exploiting them.
bench/storage_stress.py joins the default sequence before log_transfer
(storage health first, then the test that depends on storage). It runs
sd_bench with data verification and parses write/read throughput and
fsync latency into named checks with generous FAIL floors (defaults
justified in the README: 100 KB/s write, 200 KB/s read, 500 ms fsync;
a working-but-slow card prints a WARNING instead of flapping), then
sd_stress for file churn with a short default iteration count. Both
phases probe-and-skip independently: a firmware without a command or a
board without an SD card records SKIP with a warning, never FAIL.
bench/serial_loopback.py is a standalone operator/fixture test for
manufacturing: with a TX-RX loopback jumper installed on a chosen UART,
serial_test transmits a known pattern and the session summary
(rx/tx/rx err counts) plus pattern-error lines decide the verdict. Not
in the default sequence because it needs a physical fixture.
The firmware gate's --build path now prints a bench capability report
from one shared table: after reading default.px4board plus the label
overlay (labels merge both, cmake/kconfig.cmake:47-54) it lists whether
the image will contain simulator_sih, sd_bench, sd_stress, and
serial_test and which suite tests will run or skip. For anything
missing it names a sibling variant that has it (exact --target) and
the exact config line, and interactively offers to append the line to
the local board config, stating that the edit is local and uncommitted.
All four options are 'default n' in Kconfig, so the merged .px4board
lines decide presence.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The simulated flight is the only test that exercises commander,
navigator, and land-detector flight logic on real NuttX scheduling;
leaving it out of the default suite meant the most valuable coverage
was routinely skipped. It now runs last in the sequence.
Arming safety gets two explicit paths: on a TTY the operator sees a
prominent warning that the flight controller WILL ARM (simulated
flight, pwm_out_sim replaces the real outputs, board must be bare) and
must type 'arm'; declining records a skip, not a failure. For
automation and manufacturing lines --allow-arming bypasses the prompt,
and a non-TTY run without it skips the flight with a warning naming
the flag.
Before touching any configuration the test probes the live firmware
for the SIH module ('simulator_sih status'; NuttX nsh replies
'nsh: <cmd>: command not found', apps/nshlib/nsh_parse.c) and skips
with a warning when the module is absent.
Shared plumbing in px4bench: EXIT_SKIP (75) recorded as SKIP by the
orchestrator, shell_command_exists() for probe-and-skip, and
arming_gate() used by both the orchestrator flow and the standalone
flight test. Config mutation by the flight test is expected; it
restores the original airframe and SYS_HITL afterwards.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
--target px4_fmu-v6xrt_bench previously skipped the early board_id
check because only bare vendor_model targets were known; a label
selects a .px4board config within the same board, so strip it and
resolve to the board dir. The uploader still enforces board_id against
the bootloader at flash time.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Today's runs executed against whatever firmware happened to be on the
board (an old April build at first) and nothing checked. For a
qualification tool that is unacceptable: before any test starts the suite
now establishes, and can control, exactly which build it is testing.
New px4bench/firmware.py: .px4 metadata parsing and validation (fields
verified against Tools/px_mkfw.py and px4_uploader.py), board identity
via ver all over the nsh shell, flashing through Tools/px4_uploader.py
with streamed output and a hard timeout, HW-arch to build-target
inference by enumerating boards/, an early wrong-board check against
firmware.prototype board_id, and the named firmware_identity check
(git-hash prefix match against the artifact).
run_bench_suite.py preflight supports four firmware sources, all
converging on the same flash + verify path: keep what is on the board
(--any-firmware), flash a local file (--firmware), build the inferred
target from this source tree (--build, --target, --build-timeout), or
download a GitHub release artifact (--release TAG|latest, via gh;
asset naming verified against the v1.17.0 release). --expect-hash
verifies without flashing. Flags are mutually exclusive; with none
given, a TTY gets an operator menu covering the same four sources, and
automation exits with an error before touching the board. The detected
identity is printed, written to firmware.json in the suite report dir,
and stamped into every test's report dir via PX4BENCH_FIRMWARE_INFO.
If the board's mavlink is wedged the uploader cannot soft-reboot it into
the bootloader (hit live today); after repeated reboot attempts the gate
prints an operator instruction to replug USB so the uploader catches the
bootloader at power-on.
sih/flight_mission.py gains a verify-only --expect-hash and stamps the
firmware identity into its report dir; it never flashes.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The README opened by framing the whole suite around one merged PR
series. The value is general: CI never boots NuttX and SITL runs on a
host OS, so defects in boot ordering, link lifecycle, storage, loop
rates, and RTOS-only concurrency are invisible until a board is on a
bench. Reword the introduction around that gap and the timeout-first
design rule instead of the originating incident.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Turn the flat script pile into a standalone project so contributors can
navigate and extend it:
- px4bench/ shared library package: core primitives in __init__ (Reporter,
connect, MavlinkShell, reboot/replug, viewer tee, mavlink status parsers,
pymavlink add_message workaround) plus protocol modules params.py,
missions.py, and ftp.py extracted from the tests. Zero helper duplication
remains across scripts.
- bench/ holds the real-firmware tests (boot_health, reboot_loop,
usb_replug, link_forwarding, param_stress, mission_stress, log_transfer);
sih/ holds the simulated flight (flight_mission), making the
simulation/no-simulation boundary explicit.
- pyproject.toml (px4bench 0.1.0, BSD-3-Clause, pymavlink/pyserial deps,
pyulog extra) so pip install -e Tools/bench_test works; every script
remains directly runnable without installation via a parent-dir path shim.
- README rewritten contributor-first: architecture, per-test justification
tied to the v1.18 risk areas, why pymavlink over MAVSDK, how to add a
test, baseline workflow.
- Consistent CLI surface (shared connection args; --report-dir replaces
log_transfer's --outdir); decorative section banners removed.
All hardware-learned behavior is preserved exactly: param echo drain and
match-by-value, shell sentinel strip-all, the add_message workaround,
explicit param save before reboot, mission clear before upload, RTL in
MAV_FRAME_MISSION, and the post-flight ULog download.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
sih_flight.py switches the board to a SIH airframe (SYS_HITL=2, physics
on the FMU), flies takeoff, a 3-waypoint square, and RTL as an auto
mission, asserting arming, takeoff, waypoint progression, landing, and
auto-disarm with per-phase timeouts, then restores the original config.
Covers the flight-logic paths the bench tests cannot reach; pwm_out_sim
replaces real outputs so nothing is driven on the rails.
--viewer tees every received MAVLink frame to UDP (frame per datagram,
same framing as the SITL viewer channel) and enables the
HIL_STATE_QUATERNION/HIL_ACTUATOR_CONTROLS streams on the board, so
Hawkeye renders the flight live from a serial-connected board.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
First hardware run (fmu-v6xrt over USB, macOS) surfaced three issues:
- param_torture consumed PARAM_VALUE echoes positionally; one duplicate
broadcast desynced every subsequent check. Drain stale messages before
each set and match the echo against the expected value, reporting all
observed values so a genuine wrong-echo bug remains visible.
- the shell sentinel's second safety echo leaked into the next command's
captured output; strip any sentinel line, not just the current one.
- pymavlink 2.4.49 add_message() intermittently crashes with TypeError
when the first message of an instanced type arrives with its instance
field unset; wrap it defensively until fixed upstream.
Full suite now passes 5/5 against fmu-v6xrt.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Semi-automated bench tests for qualifying v1.18 on real Pixhawk-class
hardware after the TSAN/concurrency series (#27606, #27809, #27813).
CI builds NuttX but never boots it, and the failure mode of this class
of regression is a silent hang, not a crash: every operation here has a
hard timeout and a timeout is reported as FAIL naming what stalled.
Tests map to the reworked subsystems: boot_health (WorkQueue/uORB
snapshot with baseline diffing), dual_link_forwarding (nested-send lock
path under simultaneous two-link load), param_torture (set/readback and
reboot persistence on SDLOG_UTC_OFFSET), mission_torture (dataman and
the mission shared-state mutex, 220 items, link alternation),
mavftp_log (logger on/off plus MAVFTP download and ULog verification),
reboot_loop and the operator-assisted usb_replug (mavlink instance and
RAM lifecycle across re-enumeration). run_bench_suite.py orchestrates
the non-interactive tests with a per-test watchdog.
Requires pymavlink/pyserial, pyulog optional. Nothing arms the vehicle.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
* refactor(ROMFS): remove trailing zeros from all airframes
* refactor(ROMFS): remove allignment whitespace from all startup scripts
* feat(romfs_pruner): strip end of line comments in startup scripts
and airframes to not waste any flash.
* feat(romfs_pruner): strip inline multi-whitespace
* fix typo
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Jacob Dahl <37091262+dakejahl@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Homebrew 6.0 (2026-06-11) makes tap-trust mandatory and refuses to load
formulae from untrusted third-party taps. This broke every macOS build:
brew install aborted with "Refusing to load formula px4/px4/fastdds from
untrusted tap px4/px4" before pouring any package, so ccache (and the
rest of the toolchain) was never installed and setup-ccache failed with
"ccache: command not found".
Trust the three required taps non-interactively before installing from
them. Guarded behind a brew trust capability check so older Homebrew,
which has no trust gate, skips it.
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
