Release Testing#

During the release freeze we rely on community testing to catch regressions before a new release ships. If you’re able to run the nightly builds for a few days and report your findings, you’ll directly help improve the quality of the upcoming release.

You can help with that by testing the nightly builds on your device or in a virtual machine. You can file any findings by reporting a bug. For release dates see the release calendar.

What to Test#

Explorative Testing#

You don’t need to test everything. Pick a few things you use regularly and verify that they still work with the nightly build:

Booting and unlocking the device
Calls and SMS
Mobile data and Wi-Fi
Suspend and resume
Notifications
Quick settings
Camera and audio
Applications you use frequently

Even a short test session is useful. For example, booting the nightly build, placing a call, connecting to Wi-Fi and confirming notifications work already provides valuable feedback.

Targeted Testing#

For targeted testing, review the merge requests that were included in the upcoming release. Merge request descriptions often contain testing instructions or screenshots showing the expected behaviour.

To find the merge requests that got merged during a release cycle use the associated Gitlab milestone:

https://gitlab.gnome.org/groups/World/Phosh/-/merge_requests/?state=merged&milestone_title=Phosh%20<releasenumber>

Replace <releasenumber> with the release version you want to check, e.g. for 0.56.0: https://gitlab.gnome.org/groups/World/Phosh/-/merge_requests/?state=merged&milestone_title=Phosh%200.56.0

This gives an overview of the changes in most components and you can click on a particular merge request to find more information.

If it’s unclear what should be tested just add a comment in the merge request asking for steps on how to test.

Reporting Your Test Results#

If you tested the effects of a particular merge request successfully you can add a comment about what you tested, on which device and on which distro as a new comment in the merge request.

Success reports are just as important as failure reports. Knowing that a feature works correctly on different devices and distributions gives developers confidence that the release is ready. If you found a problem you can add a comment or (preferably) open a new issue with information on how to reproduce the issue.

If you performed broader tests you can report your results in the per-release tracking bug in the meta-phosh repository or (more casually) in developer matrix room.

What to Report#

When reporting your results, please include:

The device
The distribution you’re running
Nightly build date (or package version)
What you tested
Whether it worked as expected
Steps to reproduce any issues
Whether this is a regression or is also present in the current stable release

How to Test#

If you’re using a Debian-based distribution or postmarketOS on your device, testing doesn’t require any compilation as you can use nightly packages. For many things you can also use a prebuilt BengalOS virtual machine image.

Every Test Helps#

You don’t need to be a developer to help with release testing. Running a nightly build for a day and reporting what worked (or didn’t) on your device provides valuable information that helps us ship a more reliable release.