doc: Explain briefly how to write new tests
Add a second on writing tests, covering when to use Python and C, where to put the tests, etc. Add a link to the existing Python test documentation. Signed-off-by: Simon Glass <sjg@chromium.org>
This commit is contained in:
Simon Glass
Tom Rini
parent
e1b12e3945
commit
fc32833145
|
|
@ -33,4 +33,5 @@ Testing
|
|||
coccinelle
|
||||
testing
|
||||
py_testing
|
||||
tests_writing
|
||||
tests_sandbox
|
||||
|
|
|
|||
|
|
@ -13,7 +13,8 @@ results. Advantages of this approach are:
|
|||
U-Boot; there can be no disconnect.
|
||||
- There is no need to write or embed test-related code into U-Boot itself.
|
||||
It is asserted that writing test-related code in Python is simpler and more
|
||||
flexible than writing it all in C.
|
||||
flexible than writing it all in C. But see :doc:`tests_writing` for caveats
|
||||
and more discussion / analysis.
|
||||
- It is reasonably simple to interact with U-Boot in this way.
|
||||
|
||||
Requirements
|
||||
|
|
|
|||
|
|
@ -117,6 +117,8 @@ or is covered sparingly. So here are some suggestions:
|
|||
is much easier to add onto a test - writing a new large test can seem
|
||||
daunting to most contributors.
|
||||
|
||||
See doc:`tests_writing` for how to write tests.
|
||||
|
||||
|
||||
Future work
|
||||
-----------
|
||||
|
|
|
|||
|
|
@ -200,3 +200,10 @@ linker_list::
|
|||
000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent
|
||||
000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle
|
||||
000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props
|
||||
|
||||
|
||||
Writing tests
|
||||
-------------
|
||||
|
||||
See :doc:`tests_writing` for how to write new tests.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,346 @@
|
|||
.. SPDX-License-Identifier: GPL-2.0+
|
||||
.. Copyright 2021 Google LLC
|
||||
.. sectionauthor:: Simon Glass <sjg@chromium.org>
|
||||
|
||||
Writing Tests
|
||||
=============
|
||||
|
||||
This describes how to write tests in U-Boot and describes the possible options.
|
||||
|
||||
Test types
|
||||
----------
|
||||
|
||||
There are two basic types of test in U-Boot:
|
||||
|
||||
- Python tests, in test/py/tests
|
||||
- C tests, in test/ and its subdirectories
|
||||
|
||||
(there are also UEFI tests in lib/efi_selftest/ not considered here.)
|
||||
|
||||
Python tests talk to U-Boot via the command line. They support both sandbox and
|
||||
real hardware. They typically do not require building test code into U-Boot
|
||||
itself. They are fairly slow to run, due to the command-line interface and there
|
||||
being two separate processes. Python tests are fairly easy to write. They can
|
||||
be a little tricky to debug sometimes due to the voluminous output of pytest.
|
||||
|
||||
C tests are written directly in U-Boot. While they can be used on boards, they
|
||||
are more commonly used with sandbox, as they obviously add to U-Boot code size.
|
||||
C tests are easy to write so long as the required facilities exist. Where they
|
||||
do not it can involve refactoring or adding new features to sandbox. They are
|
||||
fast to run and easy to debug.
|
||||
|
||||
Regardless of which test type is used, all tests are collected and run by the
|
||||
pytest framework, so there is typically no need to run them separately. This
|
||||
means that C tests can be used when it makes sense, and Python tests when it
|
||||
doesn't.
|
||||
|
||||
|
||||
This table shows how to decide whether to write a C or Python test:
|
||||
|
||||
===================== =========================== =============================
|
||||
Attribute C test Python test
|
||||
===================== =========================== =============================
|
||||
Fast to run? Yes No (two separate processes)
|
||||
Easy to write? Yes, if required test Yes
|
||||
features exist in sandbox
|
||||
or the target system
|
||||
Needs code in U-Boot? Yes No, provided the test can be
|
||||
executed and the result
|
||||
determined using the command
|
||||
line
|
||||
Easy to debug? Yes No, since access to the U-Boot
|
||||
state is not available and the
|
||||
amount of output can
|
||||
sometimes require a bit of
|
||||
digging
|
||||
Can use gdb? Yes, directly Yes, with --gdbserver
|
||||
Can run on boards? Some can, but only if Some
|
||||
compiled in and not
|
||||
dependent on sandboxau
|
||||
===================== =========================== =============================
|
||||
|
||||
|
||||
Python or C
|
||||
-----------
|
||||
|
||||
Typically in U-Boot we encourage C test using sandbox for all features. This
|
||||
allows fast testing, easy development and allows contributors to make changes
|
||||
without needing dozens of boards to test with.
|
||||
|
||||
When a test requires setup or interaction with the running host (such as to
|
||||
generate images and then running U-Boot to check that they can be loaded), or
|
||||
cannot be run on sandbox, Python tests should be used. These should typically
|
||||
NOT rely on running with sandbox, but instead should function correctly on any
|
||||
board supported by U-Boot.
|
||||
|
||||
|
||||
How slow are Python tests?
|
||||
--------------------------
|
||||
|
||||
Under the hood, when running on sandbox, Python tests work by starting a sandbox
|
||||
test and connecting to it via a pipe. Each interaction with the U-Boot process
|
||||
requires at least a context switch to handle the pipe interaction. The test
|
||||
sends a command to U-Boot, which then reacts and shows some output, then the
|
||||
test sees that and continues. Of course on real hardware, communications delays
|
||||
(e.g. with a serial console) make this slower.
|
||||
|
||||
For comparison, consider a test that checks the 'md' (memory dump). All times
|
||||
below are approximate, as measured on an AMD 2950X system. Here is is the test
|
||||
in Python::
|
||||
|
||||
@pytest.mark.buildconfigspec('cmd_memory')
|
||||
def test_md(u_boot_console):
|
||||
"""Test that md reads memory as expected, and that memory can be modified
|
||||
using the mw command."""
|
||||
|
||||
ram_base = u_boot_utils.find_ram_base(u_boot_console)
|
||||
addr = '%08x' % ram_base
|
||||
val = 'a5f09876'
|
||||
expected_response = addr + ': ' + val
|
||||
u_boot_console.run_command('mw ' + addr + ' 0 10')
|
||||
response = u_boot_console.run_command('md ' + addr + ' 10')
|
||||
assert(not (expected_response in response))
|
||||
u_boot_console.run_command('mw ' + addr + ' ' + val)
|
||||
response = u_boot_console.run_command('md ' + addr + ' 10')
|
||||
assert(expected_response in response)
|
||||
|
||||
This runs a few commands and checks the output. Note that it runs a command,
|
||||
waits for the response and then checks it agains what is expected. If run by
|
||||
itself it takes around 800ms, including test collection. For 1000 runs it takes
|
||||
19 seconds, or 19ms per run. Of course 1000 runs it not that useful since we
|
||||
only want to run it once.
|
||||
|
||||
There is no exactly equivalent C test, but here is a similar one that tests 'ms'
|
||||
(memory search)::
|
||||
|
||||
/* Test 'ms' command with bytes */
|
||||
static int mem_test_ms_b(struct unit_test_state *uts)
|
||||
{
|
||||
u8 *buf;
|
||||
|
||||
buf = map_sysmem(0, BUF_SIZE + 1);
|
||||
memset(buf, '\0', BUF_SIZE);
|
||||
buf[0x0] = 0x12;
|
||||
buf[0x31] = 0x12;
|
||||
buf[0xff] = 0x12;
|
||||
buf[0x100] = 0x12;
|
||||
ut_assertok(console_record_reset_enable());
|
||||
run_command("ms.b 1 ff 12", 0);
|
||||
ut_assert_nextline("00000030: 00 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................");
|
||||
ut_assert_nextline("--");
|
||||
ut_assert_nextline("000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 12 ................");
|
||||
ut_assert_nextline("2 matches");
|
||||
ut_assert_console_end();
|
||||
|
||||
ut_asserteq(2, env_get_hex("memmatches", 0));
|
||||
ut_asserteq(0xff, env_get_hex("memaddr", 0));
|
||||
ut_asserteq(0xfe, env_get_hex("mempos", 0));
|
||||
|
||||
unmap_sysmem(buf);
|
||||
|
||||
return 0;
|
||||
}
|
||||
MEM_TEST(mem_test_ms_b, UT_TESTF_CONSOLE_REC);
|
||||
|
||||
This runs the command directly in U-Boot, then checks the console output, also
|
||||
directly in U-Boot. If run by itself this takes 100ms. For 1000 runs it takes
|
||||
660ms, or 0.66ms per run.
|
||||
|
||||
So overall running a C test is perhaps 8 times faster individually and the
|
||||
interactions are perhaps 25 times faster.
|
||||
|
||||
It should also be noted that the C test is fairly easy to debug. You can set a
|
||||
breakpoint on do_mem_search(), which is what implements the 'ms' command,
|
||||
single step to see what might be wrong, etc. That is also possible with the
|
||||
pytest, but requires two terminals and --gdbserver.
|
||||
|
||||
|
||||
Why does speed matter?
|
||||
----------------------
|
||||
|
||||
Many development activities rely on running tests:
|
||||
|
||||
- 'git bisect run make qcheck' can be used to find a failing commit
|
||||
- test-driven development relies on quick iteration of build/test
|
||||
- U-Boot's continuous integration (CI) systems make use of tests. Running
|
||||
all sandbox tests typically takes 90 seconds and running each qemu test
|
||||
takes about 30 seconds. This is currently dwarfed by the time taken to
|
||||
build all boards
|
||||
|
||||
As U-Boot continues to grow its feature set, fast and reliable tests are a
|
||||
critical factor factor in developer productivity and happiness.
|
||||
|
||||
|
||||
Writing C tests
|
||||
---------------
|
||||
|
||||
C tests are arranged into suites which are typically executed by the 'ut'
|
||||
command. Each suite is in its own file. This section describes how to accomplish
|
||||
some common test tasks.
|
||||
|
||||
(there are also UEFI C tests in lib/efi_selftest/ not considered here.)
|
||||
|
||||
Add a new driver model test
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use this when adding a test for a new or existing uclass, adding new operations
|
||||
or features to a uclass, adding new ofnode or dev_read_() functions, or anything
|
||||
else related to driver model.
|
||||
|
||||
Find a suitable place for your test, perhaps near other test functions in
|
||||
existing code, or in a new file. Each uclass should have its own test file.
|
||||
|
||||
Declare the test with::
|
||||
|
||||
/* Test that ... */
|
||||
static int dm_test_uclassname_what(struct unit_test_state *uts)
|
||||
{
|
||||
/* test code here */
|
||||
|
||||
return 0;
|
||||
}
|
||||
DM_TEST(dm_test_uclassname_what, UT_TESTF_SCAN_FDT);
|
||||
|
||||
Replace 'uclassname' with the name of your uclass, if applicable. Replace 'what'
|
||||
with what you are testing.
|
||||
|
||||
The flags for DM_TEST() are defined in test/test.h and you typically want
|
||||
UT_TESTF_SCAN_FDT so that the devicetree is scanned and all devices are bound
|
||||
and ready for use. The DM_TEST macro adds UT_TESTF_DM automatically so that
|
||||
the test runner knows it is a driver model test.
|
||||
|
||||
Driver model tests are special in that the entire driver model state is
|
||||
recreated anew for each test. This ensures that if a previous test deletes a
|
||||
device, for example, it does not affect subsequent tests. Driver model tests
|
||||
also run both with livetree and flattree, to ensure that both devicetree
|
||||
implementations work as expected.
|
||||
|
||||
Example commit: c48cb7ebfb4 ("sandbox: add ADC unit tests") [1]
|
||||
|
||||
[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/c48cb7ebfb4
|
||||
|
||||
|
||||
Add a C test to an existing suite
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use this when you are adding to or modifying an existing feature outside driver
|
||||
model. An example is bloblist.
|
||||
|
||||
Add a new function in the same file as the rest of the suite and register it
|
||||
with the suite. For example, to add a new mem_search test::
|
||||
|
||||
/* Test 'ms' command with 32-bit values */
|
||||
static int mem_test_ms_new_thing(struct unit_test_state *uts)
|
||||
{
|
||||
/* test code here*/
|
||||
|
||||
return 0;
|
||||
}
|
||||
MEM_TEST(mem_test_ms_new_thing, UT_TESTF_CONSOLE_REC);
|
||||
|
||||
Note that the MEM_TEST() macros is defined at the top of the file.
|
||||
|
||||
Example commit: 9fe064646d2 ("bloblist: Support relocating to a larger space") [1]
|
||||
|
||||
[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/9fe064646d2
|
||||
|
||||
|
||||
Add a new test suite
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each suite should focus on one feature or subsystem, so if you are writing a
|
||||
new one of those, you should add a new suite.
|
||||
|
||||
Create a new file in test/ or a subdirectory and define a macro to register the
|
||||
suite. For example::
|
||||
|
||||
#include <common.h>
|
||||
#include <console.h>
|
||||
#include <mapmem.h>
|
||||
#include <dm/test.h>
|
||||
#include <test/ut.h>
|
||||
|
||||
/* Declare a new wibble test */
|
||||
#define WIBBLE_TEST(_name, _flags) UNIT_TEST(_name, _flags, wibble_test)
|
||||
|
||||
/* Tetss go here */
|
||||
|
||||
/* At the bottom of the file: */
|
||||
|
||||
int do_ut_wibble(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[])
|
||||
{
|
||||
struct unit_test *tests = UNIT_TEST_SUITE_START(wibble_test);
|
||||
const int n_ents = UNIT_TEST_SUITE_COUNT(wibble_test);
|
||||
|
||||
return cmd_ut_category("cmd_wibble", "wibble_test_", tests, n_ents, argc, argv);
|
||||
}
|
||||
|
||||
Then add new tests to it as above.
|
||||
|
||||
Register this new suite in test/cmd_ut.c by adding to cmd_ut_sub[]::
|
||||
|
||||
/* Within cmd_ut_sub[]... */
|
||||
|
||||
U_BOOT_CMD_MKENT(wibble, CONFIG_SYS_MAXARGS, 1, do_ut_wibble, "", ""),
|
||||
|
||||
and adding new help to ut_help_text[]::
|
||||
|
||||
"ut wibble - Test the wibble feature\n"
|
||||
|
||||
If your feature is conditional on a particular Kconfig, then you can use #ifdef
|
||||
to control that.
|
||||
|
||||
Finally, add the test to the build by adding to the Makefile in the same
|
||||
directory::
|
||||
|
||||
obj-$(CONFIG_$(SPL_)CMDLINE) += wibble.o
|
||||
|
||||
Note that CMDLINE is never enabled in SPL, so this test will only be present in
|
||||
U-Boot proper. See below for how to do SPL tests.
|
||||
|
||||
As before, you can add an extra Kconfig check if needed::
|
||||
|
||||
ifneq ($(CONFIG_$(SPL_)WIBBLE),)
|
||||
obj-$(CONFIG_$(SPL_)CMDLINE) += wibble.o
|
||||
endif
|
||||
|
||||
|
||||
Example commit: 919e7a8fb64 ("test: Add a simple test for bloblist") [1]
|
||||
|
||||
[1] https://gitlab.denx.de/u-boot/u-boot/-/commit/919e7a8fb64
|
||||
|
||||
|
||||
Making the test run from pytest
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
All C tests must run from pytest. Typically this is automatic, since pytest
|
||||
scans the U-Boot executable for available tests to run. So long as you have a
|
||||
'ut' subcommand for your test suite, it will run. The same applies for driver
|
||||
model tests since they use the 'ut dm' subcommand.
|
||||
|
||||
See test/py/tests/test_ut.py for how unit tests are run.
|
||||
|
||||
|
||||
Add a C test for SPL
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Note: C tests are only available for sandbox_spl at present. There is currently
|
||||
no mechanism in other boards to existing SPL tests even if they are built into
|
||||
the image.
|
||||
|
||||
SPL tests cannot be run from the 'ut' command since there are no commands
|
||||
available in SPL. Instead, sandbox (only) calls ut_run_list() on start-up, when
|
||||
the -u flag is given. This runs the available unit tests, no matter what suite
|
||||
they are in.
|
||||
|
||||
To create a new SPL test, follow the same rules as above, either adding to an
|
||||
existing suite or creating a new one.
|
||||
|
||||
An example SPL test is spl_test_load().
|
||||
|
||||
|
||||
Writing Python tests
|
||||
--------------------
|
||||
|
||||
See :doc:`py_testing` for brief notes how to write Python tests. You
|
||||
should be able to use the existing tests in test/py/tests as examples.
|
||||
Loading…
Reference in New Issue