This guide will cover integrating the Memfault Linux SDK into systems built with
Yocto.
When you're done, you will have a basic integration setup that can talk to the
Memfault cloud, enabling one or more of the following features:
Useful links
Assumptions and dependencies
Add memfaultd to your system
Keep meta-memfault-example open as a reference
implementation. Your integration should look similar to it once you're done
following the steps in this tutorial.
The memfaultd service is the main orchestrator of the subsystems that conform
and are used by the Memfault Linux SDK. As with all our SDKs, its source code
is available on GitHub.
Among its responsibilities are:
- Keeping a queue of items to be uploaded to the Memfault cloud.
- Uploading that data at a specified interval, and recovering from network
failures using an exponential back-off system, independent of the data
collection interval.
- Controlling whether data is allowed to be collected (e.g. by user consent)
from a device or not.
- Adjusting which data is collected based on the fleet
sampling configuration of this device.
- Configuring SWUpdate dynamically to perform OTA updates.
- Providing configuration access via memfaultctl
(
memfaultctl is a symbolic link to memfaultd).
memfaultd is required for integration.
Include the meta-memfault layer in your
bblayers.conf file, alongside dependencies:
# Apart from other layers you may already depend on:
BBLAYERS ?= " \
${YOCTOROOT}/sources/memfault-linux-sdk/meta-memfault \
${YOCTOROOT}/sources/memfault-linux-sdk/meta-rust-bin \
${YOCTOROOT}/sources/meta-openembedded/meta-oe \
"
Rust in Yocto for the Memfault SDK
memfaultd requires the Rust compiler version 1.65.0 or higher. Dunfell does
not ship with Rust support and Kirkstone ships with 1.59.0 so we provide Rust
recipes with the SDK.
The meta-rust-bin layer included in the memfault-linux-sdk is a slightly
modified version of rust-embedded/meta-rust-bin . This
version differs from upstream by providing a new class cargo_bin instead of
overriding the cargo class provided by meta-oe.
We found that overriding cargo often caused problems with some Poky packages,
notably python3-cryptography-native. The modified version that we provide does
not have this issue.
Authorize Memfault License
Since memfaultd has a commercial license, you'll need to add an exception for
it, for example in your local.conf:
LICENSE_FLAGS_ACCEPTED:append = " commercial_memfaultd"
(Optional) Use systemd as your init system
memfaultd will work with or without systemd. When systemd is available, it
will automatically be used to detect normal reboots and shutdowns (refer to our
reboot reason tracking guide).
To use systemd as an init system (using poky), add to your conf/local.conf
file:
DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES:remove = "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED:append = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""
(Optional) Use OpenSSL instead of rust-tls
By default, memfaultd will use the rustls TLS library,
which is a full TLS implementation in Rust. We recommend this option for most
users.
In some scenario you may want to use OpenSSL:
- To avoid having two SSL implementations in the system and minimize the size of
the final image.
- For architectures which are not supported by
rustls (notably, mips
variants).
To use OpenSSL, set the openssl-tls feature on the memfaultd recipe.
For example, in your Yocto build/local.conf:
PACKAGECONFIG:pn-memfaultd := "swupdate coredump logging openssl-tls"
If you are planning on using swupdate for OTA, you will also need to add the
meta-swupdate layer:
# Apart from other layers you may already depend on:
BBLAYERS ?= " \
${YOCTOROOT}/sources/memfault-linux-sdk/meta-memfault \
${YOCTOROOT}/sources/meta-openembedded/meta-oe \
${YOCTOROOT}/sources/meta-swupdate \
"
You can grab a copy of meta-swupdate from the source
repository. Note that meta-oe is a dependency of
meta-swupdate.
And add the necessary dependencies to your image file (swupdate will be
required automatically by the memfaultd recipe as long as the swupdate
feature is active):
IMAGE_INSTALL:append = " \
u-boot-env \
u-boot-fw-utils \
"
Minimal Configuration of memfaultd
(Optional) Opt out of memfaultd default features
The memfaultd daemon is modular and you can control which features are
built-in to reduce code size, dependencies and resource usage. By default, all
features are enabled.
The available features are:
Set PACKAGECONFIG for the memfaultd package to control which features are
included in the memfaultd build. For example, you can add a
memfaultd.bbappend recipe with the following contents:
PACKAGECONFIG := "swupdate coredump logging"
Alternatively, in your local.conf file:
PACKAGECONFIG:pn-memfaultd := "swupdate coredump logging"
Add a memfault-device-info executable to your build
The memfaultd daemon expects to find a memfault-device-info executable in
$PATH. The memfault-device-info executable is created by you during
integration and is meant to provide device-specific information in order to
identify it in Memfault services. The expected output is in the following
format:
MEMFAULT_DEVICE_ID=<device-id>
MEMFAULT_HARDWARE_VERSION=<hardware-version>
Read more about Device ID, Software Versions, and Hardware
Versions to better understand these
core Memfault platform concepts.
Here's an example memfault-device-info shell script one might add to
/usr/bin/memfault-device-info:
#!/bin/sh
echo MEMFAULT_DEVICE_ID=$(cat /etc/machine-id)
echo MEMFAULT_HARDWARE_VERSION=$(dmidecode -s system-product-name)
The script would output something like this:
$ memfault-device-info
MEMFAULT_DEVICE_ID=4a7d5d74-e8f0-471f-9f8c-23e3dd5ce18c
MEMFAULT_HARDWARE_VERSION=some-board-rev-1.3
If you added a recipe named memfault-device-info, don't forget to add it to
your image dependencies:
IMAGE_INSTALL:append = " \
memfault-device-info \
u-boot-env \
u-boot-fw-utils \
"
Add /etc/memfaultd.conf to your build
A default memfaultd.conf (called builtin.conf in the
source files) is embedded within the memfaultd binary, and the daemon will
fall back to it if /etc/memfaultd.conf does not provide an override for a
specific key. All values in it can be overwritten by adding an
/etc/memfaultd.conf file to your build, e.g. using a bbappends recipe for
memfaultd (see an example). You may
copy parts of the built-in config (or the whole file) to customize configuration
on your target device.
Note that while the sample file has reasonable defaults for most keys, it cannot
guess a value for some fields such as software_version, software_type or
project_key. Hence, a minimal /etc/memfaultd.conf file should look like
this:
{
"software_version": "1.0.0",
"software_type": "main",
"project_key": "<YOUR_PROJECT_KEY>"
}
Note that /etc/memfaultd.conf is designed to be in a read-only filesystem.
Read more about Software Versions and Hardware
Versions to better understand these
core Memfault platform concepts.
Persistent storage
In /etc/memfaultd.conf:
{
...
"persist_dir": "<YOUR_PERSISTENT_STORAGE_DIR>"
}
By default it will be a directory in /media. You will most likely need to
change this to wherever your device stores data in a persistent manner.
Requirements for this directory are:
- That it is writable to by
memfaultd.
- That its data persists after firmware upgrades.
Your persist_dir will be written to for the following reasons:
- Save persistent configuration options (
enable_data_collection and
enable_dev_mode).
- Save the internal reboot reason file when memfaultd restarts.
- Save the Fleet Sampling configuration (only when it changes).
Temporary storage
You can configure memfaultd to use a different folder for data that will be
written more frequently and does not necessarily needs to be persisted across
reboots. You can even decide to store this temporary storage directory on a
temporary filesystem in RAM (tmpfs), although in this case, the data will be
lost if a reboot or crash occurs before the data was uploaded.
{
...
"tmp_dir": "<YOUR_TEMPORARY_STORAGE_DIR>"
}
By default the temporary storage directory is the persistent storage directory.
This is the recommended configuration.
This directory will be used for:
- Coredumps
- Heartbeats (metrics)
- Logs
- Storing the boot id of the device across memfaultd restarts
- Storing the Memfault reboot events queue
You can control how much space memfaultd will use as temporary storage. See
our configuration reference for more context.
Set enable_data_collection
By default, enable_data_collection is false (see the default
configuration). This is to enable asking end users for
consent before collecting or transmitting any data to Memfault services.
Once the end user has given their consent, you can enable data collection like
so:
$ memfaultctl enable-data-collection
To disable it:
$ memfaultctl disable-data-collection
The memfaultd service will restart automatically whenever you run either of
those commands if called with a value different from the current configuration.
Take a look at the /etc/memfaultd.conf reference for
more information.
To always enable data collection you can include the key
enable_data_collection with the value true in your configuration file.
Test the integration
You can test your integration by rebooting your test device. Then, on the
Memfault app, open Fleet → Devices and check that your device shows up. If you open
the specific device, you should be able to find its reboot events under the
Reboots tab.
If you encounter any issues in your data transfer implementation, Memfault has tools to help debug!
- To troubleshoot data not getting uploaded or processed correctly by the Memfault cloud, take a look at the Integration Hub → Processing Log view. This provides a filterable, chronological view of recent errors that have occurred while processing received data. See this documentation page for more information on the Integration Hub.
- Server-side rate limiting will apply to the device you're using to work on the integration process. Once you can see the device on the Memfault Web App, consider enabling Server-Side Developer Mode for it on the Memfault Web App to temporarily bypass these limits.
Enabling device-side developer mode and lowering
upload_interval_seconds and heartbeat_interval_seconds in
/etc/memfaultd.conf can also be helpful during
integration and testing to see reporting from your device as quickly as
possible. The memfaultctl sync command can also be used to manually sync data
between your device and Memfault.
Integration next-steps
Congratulations! 🎉 Memfault is running in your product. You will start getting
useful insights automatically.
To get more value from Memfault, we recommend the following steps.
Reboot reasons
Memfault will automatically classify some reboots such as kernel panics, user
triggered reboot and OTA updates.
For other reboot reasons that are specific to your product, you can inform
memfaultd before restarting the system. Memfault Dashboard will then give you
an accurate picture of why your devices are restarting.
Our Reboot Reason Tracking Guide presents several
mechanisms to do this.
Coredumps
memfaultd will automatically configure the kernel to capture coredumps and
upload them to Memfault.
Follow our Coredumps Guide to upload debug symbols to
Memfault during your build process. This is required for Memfault coredump
analysis to work.
Python Exceptions
Python programs do not generate coredumps and their errors are therefore not
captured by default. Instead, they dump a stack trace when an exception is
encountered.
To capture Python exceptions, we provide the
pyfault library. This library embeds into
your python program installing a custom handler in the sys.excepthook. This
converts the typical python stacktrace into the custom trace format expected by
memfaultd and sends it to memfaultd over http. See the
readme for more in
depth information on installation and usage.
Metrics
By default, memfaultd will capture system metrics and provide an overview of
the performance of the system.
Our Metrics Guide explains how to collect custom
application-specific metrics from your system.
Logging
If you are using systemd, log collection will be handled automatically by
memfaultd. Refer to our Linux Logging Guide for details
on how this feature can be configured and how to collect logs on non-systemd
systems.
OTA
A working OTA setup will require collaboration between the bootloader, the
updater (for example swupdate) and memfaultd. You will also need to
configure your build process to generate update images in a format compatible
with the updater.
Follow our Embedded Linux OTA Integration Guide to configure
and use OTA.
]]>
