Skip to content

Update buildbot.rst doc #1738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 7 commits into from
Feb 19, 2026
Merged

Update buildbot.rst doc #1738

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7c72e02
Update buildbot doc
StanFromIreland Feb 18, 2026
e1b82d1
Itamar's review
StanFromIreland Feb 19, 2026
d52de6a
Victor's review
StanFromIreland Feb 19, 2026
9cca1a6
Apply suggestions from code review
StanFromIreland Feb 19, 2026
de20eb1
Fix more issues!
StanFromIreland Feb 19, 2026
cff6f9e
Hugo's other suggestions
StanFromIreland Feb 19, 2026
7c50fb2
Update testing/buildbots.rst
StanFromIreland Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 72 additions & 69 deletions testing/buildbots.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,35 +11,38 @@ branches <devcycle>`, Python has a set of dedicated machines (called *buildbots*
or *build workers*) used for continuous integration. They span a number of
hardware/operating system combinations. Furthermore, each machine hosts
several *builders*, one per active branch: when a new change is pushed
to this branch on the public `GitHub repository <https://github.com/python/cpython>`__, all corresponding builders
will schedule a new build to be run as soon as possible.
to this branch on the public `GitHub repository <https://github.com/python/cpython>`__,
all corresponding builders will schedule a new build to be run as soon as possible.

The build steps run by the buildbots are the following:
The build steps run by the buildbots are:

* Check out the source tree for the change which triggered the build
* Compile Python
* Run the test suite using :ref:`strenuous settings <strenuous_testing>`
* Clean up the build tree

It is your responsibility, as a core developer, to check the automatic
build results after you push a change to the repository. It is therefore
important that you get acquainted with the way these results are presented,
It is the responsibility of core team members to check the automatic
Copy link
Member Author

@StanFromIreland StanFromIreland Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guide is not in the core team section.

Copy link
Contributor

@itamaro itamaro Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the intent, and it seems to preserve the original meaning, but I wonder if this is actually what we want?
in particular, do we expect non-core-team members to monitor and react to buildbot failures on their own PRs? or is it always the responsibility of the core-team-member that merged the PR?

Copy link
Member Author

@StanFromIreland StanFromIreland Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we expect non-core-team members to monitor and react to buildbot failures on their own PRs

IMO, we shouldn't rely on them. If they want to, I don't think anyone is stopping them, but ultimately I think it should fall on the core team.

Copy link
Member

@encukou encukou Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in particular, do we expect non-core-team members to monitor and react to buildbot failures on their own PRs?

No. If you're not familiar with buildbot quirks, the failures aren't easy to diagnose.

build results after they push a change to the repository. It is therefore
important that they are acquainted with the way these results are presented,
and how various kinds of failures can be explained and diagnosed.


In case of trouble
==================

Please read this page in full. If your questions aren't answered here and you
need assistance with the buildbots, a good way to get help is to either:

* contact the ``python-buildbots@python.org`` mailing list where all buildbot
worker owners are subscribed; or
* contact the `python-buildbots@python.org
<https://mail.python.org/archives/list/python-buildbots@python.org/>`__
mailing list where all buildbot worker owners are subscribed; or
* contact the release manager of the branch you have issues with.


Buildbot failures on pull requests
==================================

The ``bedevere-bot`` on GitHub will put a message on your merged Pull Request
The Bedevere bot on GitHub will put a message on your merged pull request
if building your commit on a stable buildbot worker fails. Take care to
evaluate the failure, even if it looks unrelated at first glance.

Expand All @@ -48,6 +51,7 @@ after each commit. In particular, reference leaks builds take several hours to
complete so they are done periodically. This is why it's important for you to
be able to check the results yourself, too.


Triggering on pull requests
===========================

Expand All @@ -63,58 +67,50 @@ buildbots again on a later commit, you'll have to remove the label and add it
again.

If you want to test a pull request against specific platforms, you can trigger
one or more build bots by posting a comment that begins with:
one or more buildbots by posting a comment that begins with:

.. code-block:: none

!buildbot regex-matching-target

For example to run both the iOS and Android build bot, you can use:
For example to run both the iOS and Android buildbots, you can use:

.. code-block:: none

!buildbot ios|android

bedevere-bot will post a comment indicating which build bots, if
Bedevere will post a comment indicating which buildbots, if
any, were matched. If none were matched, or you do not have the
necessary permissions to trigger a request, it will tell you that too.

The ``!buildbot`` comment will also only run buildbots on the most recent
commit. To trigger the buildbots again on a later commit, you will have to
repeat the comment.


Checking results of automatic builds
====================================

There are three ways of visualizing recent build results:
The Web interface at https://buildbot.python.org/#/ has several ways of
visualizing recent build results:

* The Web interface for each branch at https://www.python.org/dev/buildbot/,
where the so-called "waterfall" view presents a vertical rundown of recent
builds for each builder. When interested in one build, you'll have to
click on it to know which commits it corresponds to. Note that
the buildbot web pages are often slow to load, be patient.
* A `release status dashboard <https://buildbot.python.org/#/release_status>`__
that shows the status of buildbots for each active branch,
summarizing whether the builds are ready for release.

* The command-line ``bbreport.py`` client, which you can get from
Copy link
Member Author

@StanFromIreland StanFromIreland Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This project has not been updated for ten years, I tried it with a few versions but it no longer works.

Copy link
Member Author

@StanFromIreland StanFromIreland Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC @ezio-melotti, I think you are the maintainer, based on the commits?

https://code.google.com/archive/p/bbreport. Installing it is trivial: just add
the directory containing ``bbreport.py`` to your system path so that
you can run it from any filesystem location. For example, if you want
to display the latest build results on the development ("main") branch,
type::
* A `waterfall view <https://buildbot.python.org/#/waterfall>`__
that presents a vertical rundown of recent builds for each builder.
When interested in one build, click on it for more detail such as logs and
which commits it corresponds to.

bbreport.py -q 3.x

* The buildbot "console" interface at https://buildbot.python.org/all/
This works best on a wide, high resolution
* A `console view <https://buildbot.python.org/#/console>`__,
which works best on a wide, high resolution
monitor. Clicking on the colored circles will allow you to open a new page
containing whatever information about that particular build is of interest to
you. You can also access builder information by clicking on the builder
status bubbles in the top line.

If you like IRC, having an IRC client open to the #python-dev-notifs channel on
Copy link
Member Author

@StanFromIreland StanFromIreland Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I watched the channel for a week, no updates (whereas in the CPython discord there were several), so I assume it is defunct.

irc.libera.chat is useful. Any time a builder changes state (last build
passed and this one didn't, or vice versa), a message is posted to the channel.
Keeping an eye on the channel after pushing a commits is a simple way to get
notified that there is something you should look in to.
Note that the buildbot web pages are often slow to load, be patient.

Some buildbots are much faster than others. Over time, you will learn which
ones produce the quickest results after a build, and which ones take the
Expand All @@ -124,20 +120,24 @@ Also, when several commits are pushed in a quick succession in the same
branch, it often happens that a single build is scheduled for all these
commits.


Stability
=========

A subset of the buildbots are marked "stable". They are taken into account
when making a new release. The rule is that all stable builders must be free of
persistent failures when the release is cut. It is absolutely **vital**
that core developers fix any issue they introduce on the stable buildbots,
as soon as possible.
A subset of the buildbots are marked as
`"stable" <https://buildbot.python.org/#/builders?tags=%2Bstable>`__.
They are taken into account when making a new release.
The rule is that all tier 1 and 2 stable builders must be free of
persistent failures when the release is cut (see :pep:`11` for more information).
It is absolutely **vital** that core team members fix or revert any issue they
introduce on the stable buildbots, as soon as possible.

This does not mean that other builders' test results can be taken lightly,
either. Some of them are known for having platform-specific issues that
prevent some tests from succeeding (or even terminating at all), but
introducing additional failures should generally not be an option.


Flags-dependent failures
========================

Expand All @@ -148,11 +148,8 @@ or to Python itself. To reproduce, make sure you use the same flags as the
buildbots: they can be found out simply by clicking the **stdio** link for
the failing build's tests. For example::

./python.exe -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW
./python -E -m test --slow-ci --timeout=1200 -j2 --junit-xml test-results.xml -j10

.. note::
Running ``Lib/test/regrtest.py`` is exactly equivalent to running
``-m test``.

Ordering-dependent failures
===========================
Expand All @@ -170,58 +167,64 @@ run, and check the beginning of the test output proper.
Let's assume, for the sake of example, that the output starts with:

.. code-block:: none
:emphasize-lines: 6
:emphasize-lines: 9

./python -Wd -E -bb Lib/test/regrtest.py -uall -rwW
== CPython 3.3a0 (default:22ae2b002865, Mar 30 2011, 13:58:40) [GCC 4.4.5]
== Linux-2.6.36-gentoo-r5-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4400+-with-gentoo-1.12.14 little-endian
== /home/buildbot/buildarea/3.x.ochtman-gentoo-amd64/build/build/test_python_29628
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0)
Copy link
Member Author

@StanFromIreland StanFromIreland Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the example to a more recent one, but this change is not too important, I can revert it.

Copy link
Contributor

@itamaro itamaro Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems reasonable!

Using random seed 2613169
[ 1/353] test_augassign
[ 2/353] test_functools
./python -E -m test --slow-ci --timeout=2400 -j2 -u-cpu,-urlfetch,-network --junit-xml test-results.xml -j4
== CPython 3.15.0a6+ (heads/main:d625f7da33b, Feb 13 2026, 17:27:29) [GCC 12.2.0]
== Linux-6.12.20+rpt-rpi-v8-aarch64-with-glibc2.36 little-endian
== Python build: release
== cwd: /home/stan/buildarea/3.x.stan-raspbian.nondebug/build/build/test_python_worker_181905æ
== CPU count: 4
== encodings: locale=ISO-8859-1 FS=utf-8
== resources: all,-cpu,-network,-urlfetch
Using random seed: 1000348774
0:00:00 load avg: 3.34 Run 500 tests in parallel using 4 worker processes (timeout: 40 min, worker timeout: 45 min)
0:00:01 load avg: 3.34 [ 1/500] test_colorsys passed
0:00:01 load avg: 3.34 [ 2/500] test_float passed

You can reproduce the exact same order using::
You can reproduce the exact same order by adding the ``--randseed 1000348774`` option::

./python -Wd -E -bb -m test -uall -rwW --randseed 2613169
./python -E -m test --slow-ci --timeout=2400 -j2 -u-cpu,-urlfetch,-network --junit-xml test-results.xml -j4 --randseed 1000348774

It will run the following sequence (trimmed for brevity):

.. code-block:: none

[ 1/353] test_augassign
[ 2/353] test_functools
[ 3/353] test_bool
[ 4/353] test_contains
[ 5/353] test_compileall
[ 6/353] test_unicode
[ 1/500] test_colorsys
[ 2/500] test_float
[ 3/500] test.test_io.test_memoryio
[ 4/500] test_profile
[ 5/500] test_picklebuffer
[ 6/500] test_zipimport
[ 7/500] test_devpoll
...

If this is enough to reproduce the failure on your setup, you can then
bisect the test sequence to look for the specific interference causing the
failure. Copy and paste the test sequence in a text file, then use the
``--fromfile`` (or ``-f``) option of the test runner to run the exact
sequence recorded in that text file::

./python -Wd -E -bb -m test -uall -rwW --fromfile mytestsequence.txt
./python -E -m test --slow-ci --timeout=2400 -j2 -u-cpu,-urlfetch,-network --junit-xml test-results.xml -j4 --fromfile mytestsequence.txt

In the example sequence above, if ``test_unicode`` had failed, you would
In the example sequence above, if ``test_zipimport`` had failed, you would
first test the following sequence:

.. code-block:: none

[ 1/353] test_augassign
[ 2/353] test_functools
[ 3/353] test_bool
[ 6/353] test_unicode
[ 1/500] test_colorsys
[ 2/500] test_float
[ 3/500] test.test_io.test_memoryio
[ 6/500] test_zipimport

And, if it succeeds, the following one instead (which, hopefully, shall
fail):

.. code-block:: none

[ 4/353] test_contains
[ 5/353] test_compileall
[ 6/353] test_unicode
[ 4/500] test_profile
[ 5/500] test_picklebuffer
[ 6/500] test_zipimport

Then, recursively, narrow down the search until you get a single pair of
tests which triggers the failure. It is very rare that such an interference
Expand All @@ -243,7 +246,7 @@ not reach a perfect level of reproducibility. Some of them will sometimes
display spurious failures, depending on various conditions. Here are common
offenders:

* Network-related tests, such as ``test_poplib``, ``test_urllibnet``, etc.
* Network-related tests, such as ``test_poplib``, ``test_urllibnet``, and so on.
Their failures can stem from adverse network conditions, or imperfect
thread synchronization in the test code, which often has to run a
server in a separate thread.
Expand Down
Loading