Commit Graph

2246 Commits

Author SHA1 Message Date
Simon Glass
347e0f00e8 binman: Re-enable concurrent tests
With the change to absolute imports the concurrent tests feature
unfortunately broke. Fix it.

We cannot easy add a warning, since the output messes up tests which check
the output.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-26 19:59:57 -06:00
Simon Glass
52b10dd7de binman: Don't change the descriptor in tests
At present testPackX86RomMeNoDesc removes the contents of the
descriptor.bin file and testPackX86RomMeMissingDesc removes the file
completely.

If a test that relies on this file happens to run after it is removed, it
will not work. Since we have no control over the selecting of tests that
run in parallel and series, we must avoid changing the files.

Update this tests to use separate files instead.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-26 19:59:57 -06:00
Nicolas Boichat
e1db5c9b08 patman: When no tracking branch is provided, tell the user
The user can either count the number of patches, or provide a
tracking branch.

Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Nicolas Boichat
949775689e patman: Make sure sendemail.suppresscc is (un)set correctly
Setting sendemail.suppresscc to all or cccmd leads to --cc-cmd
parameter being ignored, and emails going either nowhere, or
just to the To: line maintainer.

Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Patrick Delaunay
e5ff9ab70a Add information for skipped commit options
The unsupported Commit-xxx option are silently skipped
and removed as 're_remove=Commit-\w*', this patch adds
warning message in this case to detect misspelled issue
for the 2 supported options:
  Commit-notes:
  Commit-changes:

For example: the final 's' is missing (Commit-note:)

NB: no issue for Series-xxx option as only the supported
    options are accepted (see valid_series in series.py)

Signed-off-by: Patrick Delaunay <patrick.delaunay@st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Patrick Delaunay
7058dd071a patman: Detect unexpected END
Detect unexpected 'END' line when a section is not detected.

This patch detect issue when tag name for section start is misspelled,
for example 'Commit-note:' for 'Commit-notes:'

  Commit-note:
  ....
  END

Then 'Commit-note:' is removed silently by re_remove = "Commit-\w*:"
but 'END' is kept in commit message.

Signed-off-by: Patrick Delaunay <patrick.delaunay@st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Simon Glass
fdc34368dd binman: Add support for generating a FIT
FIT (Flat Image Tree) is the main image format used by U-Boot. In some
cases scripts are used to create FITs within the U-Boot build system. This
is not ideal for various reasons:

- Each architecture has its own slightly different script
- There are no tests
- Some are written in shell, some in Python

To help address this, add support for FIT generation to binman. This works
by putting the FIT source directly in the binman definition, with the
ability to adjust parameters, etc. The contents of each FIT image come
from sub-entries of the image, as is normal with binman.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Simon Glass
c063917906 dtoc: Allow adding variable-sized data to a dtb
Add a method for adding a property containing arbitrary bytes. Make sure
that the tree can expand as needed in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Simon Glass
152b246298 mkimage: Allow updating the FIT timestamp
Normally the FIT timestamp is created the first time mkimage is run on a
FIT, when converting the source .its to the binary .fit file. This
corresponds to using the -f flag. But if the original input to mkimage is
a binary file (already compiled) then the timestamp is assumed to have
been set previously.

Add a -t flag to allow setting the timestamp in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
b3295fd4e3 binman: Allow zero-length entries to overlap
Some binary blobs unfortunately obtain their position in the image from
other binary blobs, such as Intel's 'descriptor'. In this case we cannot
rely on packing to work. It is not possible to produce a valid image in
any case, due to the missing blobs.

Allow zero-length overlaps so that this does not cause any problems.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
0ba4b3dfee binman: Allow missing Intel blobs
Update the Intel blob entries to support missing binaries.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
b1cca9552c binman: Detect when valid images are not produced
When external blobs are missing, show a message indicating that the images
are not functional.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Simon Glass
38fdb4cb35 patman: Update errors and warnings to use stderr
When warnings and errors are produced by tools they should be written to
stderr. Update the tout implementation to handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
4f9f1056ec binman: Allow external binaries to be missing
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.

U-Boot does with with ATF (ARM Trusted Firmware) today.

Add a new flag to binman to request this behaviour.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
04e6a6b9ec binman: Convert existing binary blobs to blob_ext
Many of the existing blobs rely on external binaries which may not be
available. Move them over to use blob_ext to indicate this.

Unfortunately cros-ec-rw cannot use this class because it inherits
another. So set the 'external' value for that class.

While we are here, drop the import of Entry since it is not used (and
pylint3 complains).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-25 14:46:57 -06:00
Simon Glass
ce867ad7c8 binman: Add an etype for external binary blobs
It is useful to be able to distinguish between ordinary blobs such as
u-boot.bin and external blobs that cannot be build by the U-Boot build
system. If the external blobs are not available for some reason, then we
know that a value image cannot be built.

Introduce a new 'blob-ext' entry type for that.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
34861d506c binman: Use super() instead of specifying parent type
It is easier and less error-prone to use super() when the parent type is
needed. Update binman to remove the type names.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
c6162cf87c binman: Fix a few typos in the entry docs
Some typos have been fixed in the generated entry docs but the code was
not updated. Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
0dc706fe54 binman: Add support for calling mkimage
As a first step to integrating mkimage into binman, add a new entry type
that feeds data into mkimage for processing and incorporates that output
into the image.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
b5287c4126 binman: Set a default toolpath
When binman is run from 'make check' it is given a toolpath so that the
latest tools (e.g. mkimage) are used. When run manually with no toolpath,
it relies on the system mkimage. But this may be missing or old.

Make some effort to find the built-from-soruce version by looking in the
current directory and in the builds created by 'make check'.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
32eb66d2d4 binman: Specify the toolpath when running test coverage
At present binman's test coverage runs without a toolpath set. This means
that the system tools will be used. That may not be correct if they are
out of date or missing and this can result in a reduction in test coverage
below 100%.

Provide the toolpath to binman in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
fdb3040e96 binman: Correct the search patch for pylibfdt
Now that binman uses tools/ as its base directory for importing modules,
the path to the pylibfdt build by U-Boot is incorrect. Fix it with a new
path.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-24 19:25:15 -06:00
Simon Glass
5de9b9c03c binman: cbfs: Fix IFWI typo
This comment references the wrong thing. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-24 19:25:15 -06:00
Simon Glass
6c8e0bfe6e binman: Output errors to stderr
At present binman outputs errors to stdout which means that fails are
effectively silent when printed by buildman, for example. Fix this by
outputing errors to stderr.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-24 19:25:15 -06:00
Simon Glass
c9360f1626 patman: Add a -D option to enable debugging
Most users don't want to see traceback errors. Add an option to enable
them for debugging. Disable them by default.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
7207e2b984 patman: Support collecting response tags in Patchstream
Collect response tags such as 'Reviewed-by' while parsing the stream.
This allows us to see what tags are present.

Add a new 'Fixes' tag also, since this is now quite common.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
3c541c0836 patman: Allow disabling 'bright' mode with Print output
At present all text is marked bright, which makes it stand out on the
terminal. Add a way to disable that, as is done with the Color class.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
6bb74de7ed patman: Add a 'test' subcommand
At present we use --test to indicate that tests should be run. It is
better to use a subcommand for list, like binman. Change it and adjust
the existing code to fit under a 'send' subcommand, the default.

Give this subcommand the same default arguments as the others.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
c4e79029e2 patman: Allow different commands
At present patman only does one thing so does not have any comments. We
want to add a few more command, so create a sub-parser for the default
command ('send').

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
fda1e372d3 patman: Convert to ArgumentParser
Convert from OptionParser to ArgumentParser to match binman. With this we
can easily add sub-commands.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
137947e05b patman: Allow skipping patches at the end
The -s option allows skipping patches at the top of the branch. Sometimes
there are commits at the bottom that need to be skipped. At present it is
necessary to count the number of commits and then use -c to tell patman
how many to process.

Add a -e option to easily skip a number of commits at the bottom of the
branch.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
262130f57c patman: Allow creating patches for another branch
Add a -b option to allow patches to be created from a branch other than
the current one.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
fd70986a62 patman: Add a test that uses gitpython
It is convenient to use gitpython to create a real git repo for testing
patman's operation. Add a test for this. So far it just checks that patman
produces the right number of patches for a branch.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
7d5b04e8a5 patman: Move main code out to a control module
To make testing easier, move the code out from main into a separate
'control' module and split it into four parts: setup, preparing patches,
checking patches and emailing patches.

Add comments and fix a few code-style issues while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Simon Glass
0b3d24a779 patman: Use test_util to show test results
This handles skipped tests correctly, so use it instead of the existing
code.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Tom Rini
7208396bbf Revert "Merge tag 'dm-pull-20jul20' of git://git.denx.de/u-boot-dm"
This reverts commit 5d3a21df66, reversing
changes made to 56d37f1c56.

Unfortunately this is causing CI failures:
https://travis-ci.org/github/trini/u-boot/jobs/711313649

Signed-off-by: Tom Rini <trini@konsulko.com>
2020-07-24 08:42:06 -04:00
Nicolas Boichat
28e0367fc8 patman: When no tracking branch is provided, tell the user
The user can either count the number of patches, or provide a
tracking branch.

Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Nicolas Boichat
754c05caf3 patman: Make sure sendemail.suppresscc is (un)set correctly
Setting sendemail.suppresscc to all or cccmd leads to --cc-cmd
parameter being ignored, and emails going either nowhere, or
just to the To: line maintainer.

Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Patrick Delaunay
3ce7f75f78 Add information for skipped commit options
The unsupported Commit-xxx option are silently skipped
and removed as 're_remove=Commit-\w*', this patch adds
warning message in this case to detect misspelled issue
for the 2 supported options:
  Commit-notes:
  Commit-changes:

For example: the final 's' is missing (Commit-note:)

NB: no issue for Series-xxx option as only the supported
    options are accepted (see valid_series in series.py)

Signed-off-by: Patrick Delaunay <patrick.delaunay@st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Patrick Delaunay
df1fa4b223 patman: Detect unexpected END
Detect unexpected 'END' line when a section is not detected.

This patch detect issue when tag name for section start is misspelled,
for example 'Commit-note:' for 'Commit-notes:'

  Commit-note:
  ....
  END

Then 'Commit-note:' is removed silently by re_remove = "Commit-\w*:"
but 'END' is kept in commit message.

Signed-off-by: Patrick Delaunay <patrick.delaunay@st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Simon Glass
3b9a87321c binman: Add support for generating a FIT
FIT (Flat Image Tree) is the main image format used by U-Boot. In some
cases scripts are used to create FITs within the U-Boot build system. This
is not ideal for various reasons:

- Each architecture has its own slightly different script
- There are no tests
- Some are written in shell, some in Python

To help address this, add support for FIT generation to binman. This works
by putting the FIT source directly in the binman definition, with the
ability to adjust parameters, etc. The contents of each FIT image come
from sub-entries of the image, as is normal with binman.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Simon Glass
1f238bd5bd dtoc: Allow adding variable-sized data to a dtb
Add a method for adding a property containing arbitrary bytes. Make sure
that the tree can expand as needed in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Simon Glass
4c63d21754 mkimage: Allow updating the FIT timestamp
Normally the FIT timestamp is created the first time mkimage is run on a
FIT, when converting the source .its to the binary .fit file. This
corresponds to using the -f flag. But if the original input to mkimage is
a binary file (already compiled) then the timestamp is assumed to have
been set previously.

Add a -t flag to allow setting the timestamp in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00
Simon Glass
8200d8871a binman: Allow zero-length entries to overlap
Some binary blobs unfortunately obtain their position in the image from
other binary blobs, such as Intel's 'descriptor'. In this case we cannot
rely on packing to work. It is not possible to produce a valid image in
any case, due to the missing blobs.

Allow zero-length overlaps so that this does not cause any problems.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00
Simon Glass
2f5c3a4d1d binman: Allow missing Intel blobs
Update the Intel blob entries to support missing binaries.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00
Simon Glass
13262c9362 binman: Detect when valid images are not produced
When external blobs are missing, show a message indicating that the images
are not functional.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Simon Glass
f9793a12c5 patman: Update errors and warnings to use stderr
When warnings and errors are produced by tools they should be written to
stderr. Update the tout implementation to handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00
Simon Glass
5f850fb9a6 binman: Allow external binaries to be missing
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.

U-Boot does with with ATF (ARM Trusted Firmware) today.

Add a new flag to binman to request this behaviour.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00
Simon Glass
894f635755 binman: Convert existing binary blobs to blob_ext
Many of the existing blobs rely on external binaries which may not be
available. Move them over to use blob_ext to indicate this.

Unfortunately cros-ec-rw cannot use this class because it inherits
another. So set the 'external' value for that class.

While we are here, drop the import of Entry since it is not used (and
pylint3 complains).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:47 -06:00
Simon Glass
d498630ea9 binman: Add an etype for external binary blobs
It is useful to be able to distinguish between ordinary blobs such as
u-boot.bin and external blobs that cannot be build by the U-Boot build
system. If the external blobs are not available for some reason, then we
know that a value image cannot be built.

Introduce a new 'blob-ext' entry type for that.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00