Python does not like the module name being the same as the module
directory. To allow buildman modules to be used from other tools, rename
it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Python does not like the module name being the same as the module
directory. To allow patman modules to be used from other tools, rename
it.
Signed-off-by: Simon Glass <sjg@chromium.org>
When outputing a progress line we don't want it to go past the end of the
current terminal line, or it will not be possible to erase it. Add an
option to Print() which allows limiting the output to the terminal width.
Since ANSI sequences do not take up space on the terminal, these are
removed.
Signed-off-by: Simon Glass <sjg@chromium.org>
When printing progress it is useful to print a message and leave the
cursor at the end of the line until the operation is finished. When it is
finished, the line needs to be erased so a new line can start in its place.
Add a function to handle clearing a line previously written by
terminal.Print()
Signed-off-by: Simon Glass <sjg@chromium.org>
Quite often on a series that has clean-up patches, the individual patches
may fit within the cc limit but the cover letter does not. Apply the same
limit to the cover letter.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Chris Packham <judge.packham@gmail.com>
fixes: 8ab452d587
When compiling list of cover letter cc addresses, using null as a
separater, then encoding to utf-8 results in lots of "\x00" as
separators. patman then doesnt understand that when it comes to
repoting the list to send-email.
Fix this by not encoding to utf-8, as done for the other patch files.
Signed-off-by: Robert Beckett <bob.beckett@collabora.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
At present patman test fail in some environments which don't use utf-8
as the default file encoding. Add this explicitly.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present all the 'command' methods return bytes. Most of the time we
actually want strings, so change this. We still need to keep the internal
representation as bytes since otherwise unicode strings might break over
a read() boundary (e.g. 4KB), causing errors. But we can convert the end
result to strings.
Add a 'binary' parameter to cover the few cases where bytes are needed.
Signed-off-by: Simon Glass <sjg@chromium.org>
There is a contributor in Linux kernel with a comma in their name, which
confuses patman and results in invalid to- or cc- addresses on some
patches. To avoid this, let's use \0 as a separator when generating cc
file.
Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
As per the centithread on ksummit-discuss [1], there are folks who
feel that if a Change-Id is present in a developer's local commit that
said Change-Id could be interesting to include in upstream posts.
Specifically if two commits are posted with the same Change-Id there's
a reasonable chance that they are either the same commit or a newer
version of the same commit. Specifically this is because that's how
gerrit has trained people to work.
There is much angst about Change-Id in upstream Linux, but one thing
that seems safe and non-controversial is to include the Change-Id as
part of the string of crud that makes up a Message-Id.
Let's give that a try.
In theory (if there is enough adoption) this could help a tool more
reliably find various versions of a commit. This actually might work
pretty well for U-Boot where (I believe) quite a number of developers
use patman, so there could be critical mass (assuming that enough of
these people also use a git hook that adds Change-Id to their
commits). I was able to find this git hook by searching for "gerrit
change id git hook" in my favorite search engine.
In theory one could imagine something like this could be integrated
into other tools, possibly even git-send-email. Getting it into
patman seems like a sane first step, though.
NOTE: this patch is being posted using a patman containing this patch,
so you should be able to see the Message-Id of this patch and see that
it contains my local Change-Id, which ends in 2b9 if you want to
check.
[1] https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2019-August/006739.html
Signed-off-by: Douglas Anderson <dianders@chromium.org>
Remove this file from git and instead build it using the Makefile.
Update tools.GetInputFilename() to support reading files from an absolute
path, so that we can read the Elf test files easily. Also make sure that
the temp directory is report in ELF tests as this was commented out.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present tools are not expected to fail. If they do an exception is
raised but there is no detail about what went wrong. This makes it hard
to debug if something does actually go wrong.
Fix this by outputting both stderr and stdout on failure.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present outdir remains set ever after the output directory has been
removed. Fix this to avoid trying to access it when it is not present.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present binman cannot replace data within a CBFS since it does not
allow rewriting of the files in that CBFS. Implement this by using the
new WriteData() method to handle the case.
Add a header to compressed data so that the amount of compressed data can
be determined without reference to the size of the containing entry. This
allows the entry to be larger that the contents, without causing errors in
decompression. This is necessary to cope with a compressed device tree
being updated in such a way that it shrinks after the entry size is
already set (an obscure case). It is not used with CBFS since it has its
own metadata for this. Increase the number of passes allowed to resolve
the position of entries, to handle this case.
Add a test for this new logic.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new logging feature to log information about progress with
packing. This is useful to see how binman is figuring things out.
Also update elf.py to use the same feature.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present tout writes directly to stdout. This is not necessary and it
prevents tests from redirecting output. Change it to use print() for the
non-progress output.
Signed-off-by: Simon Glass <sjg@chromium.org>
Binman generally operates silently but in some cases it is useful to see
what Binman is actually doing at each step. Enable some logging output
with different logging levels selectable via the -v flag.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present entry modules can only be accessed using Entry.Lookup() or
Entry.Create(). Most of the time this is fine, but sometimes a module
needs to provide constants or helper functions useful to other modules.
It is easier in this case to use 'import'.
Add an __init__ file to permit this.
Signed-off-by: Simon Glass <sjg@chromium.org>
This class is the new way to handle arguments in Python. Convert binman
over to use it. At the same time, introduce commands so that we can
separate out the different parts of binman functionality.
Signed-off-by: Simon Glass <sjg@chromium.org>
This tool has quite a few arguments and options, so put the functionality
in a function so that we call it from one place and hopefully get it
right.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add utility functions to compress and decompress using lz4 and lzma
algorithms. In the latter case these use the legacy lzma support favoured
by coreboot's CBFS.
No tests are provided as these functions will be tested by the CBFS
tests in a separate patch.
Signed-off-by: Simon Glass <sjg@chromium.org>
If kwargs contains raise_on_error then this function generates an error
due to a duplicate argument. Fix this.
Signed-off-by: Simon Glass <sjg@chromium.org>
We need slightly different commands to run code coverage with Python 3.
Update the RunTestCoverage() function to handle this.
Signed-off-by: Simon Glass <sjg@chromium.org>
While reading files in binary mode is the norm, sometimes we want to use
text mode. Add an optional parameter to handle this.
Signed-off-by: Simon Glass <sjg@chromium.org>
In Python 3 bytes and str are separate types. Use bytes to ensure that
the code functions correctly with Python 3.
Signed-off-by: Simon Glass <sjg@chromium.org>
The difference between the bytes and str types in Python 3 requires a
number of minor changes to this function. Update it to handle the input
data using the 'bytes' type. Create two useful helper functions which can
be used by other modules too.
Signed-off-by: Simon Glass <sjg@chromium.org>
Change the code so that it works on both Python 2 and Python 3. This works
by using unicode instead of latin1 for the test input, and ensuring that
the output is converted to a string rather than a unicode object on
Python 2.
Signed-off-by: Simon Glass <sjg@chromium.org>
The unicode type does not exist in Python 3 and when displaying strings
they do not have the 'u' prefix. Adjusts the settings unit tests to deal
with this difference, by converting the comparison value to a string, thus
dropping the 'u'.
Signed-off-by: Simon Glass <sjg@chromium.org>
We use sets to produce the list of To and Cc lines for a series. This does
not result in stable ordering of the recipients. Sort each list to ensure
that the output is repeatable. This is necessary for tests.
Signed-off-by: Simon Glass <sjg@chromium.org>
Create helper functions in the tools module to deal with the differences
between unicode in Python 2 (where we use the 'unicode' type) and Python 3
(where we use the 'str' type).
Signed-off-by: Simon Glass <sjg@chromium.org>
The method of multiplying a character by a number works well for creating
a repeated string in Python 2. But in Python 3 we need to use bytes()
instead, to avoid unicode problems, since 'bytes' is no-longer just an
alias of 'str'.
Create a function to handle this detail and call it from the relevant
places in binman.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present cros_subprocess and the tools library use a string to obtain
stdout from a program. This works fine on Python 2. With Python 3 we end
up with unicode errors in some cases. Fix this by providing a binary mode,
which returns the data as bytes() instead of a string.
Signed-off-by: Simon Glass <sjg@chromium.org>
Python 3 requires this, and Python 2 allows it. Convert the code over to
ensure compatibility with Python 3.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present this function uses lists and strings. This does not work so
well with Python 3, and testing against '' does not work for a bytearray.
Update the code to fix these issues.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present the tests run one after the other using a single CPU. This is
not very efficient. Bring in the concurrencytest module and run the tests
concurrently, using one process for each CPU by default. A -P option
allows this to be overridden, which is necessary for code-coverage to
function correctly.
This requires fixing a few tests which are currently not fully
independent.
At some point we might consider doing this across all pytests in U-Boot.
There is a pytest version that supports specifying the number of processes
to use, but it did not work for me.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present calling Uninit() always called ClearProgress() which outputs
a \r character as well as spaces to remove any progress information on the
line. This can mess up the normal output of binman and other tools. Fix
this by outputing this only when progress information has actually been
previous written.
Signed-off-by: Simon Glass <sjg@chromium.org>
With Python 2.7.15rc1, ConfigParser.SafeConfigParser has unfortunately
started returning unicode, for unknown reasons. Adjust the code to handle
this by converting everything to unicode. We cannot convert things to
ASCII since email addresses may be encoded with UTF-8.
Signed-off-by: Simon Glass <sjg@chromium.org>
In some cases it is useful to add a group of files to the image and be
able to access them at run-time. Of course it is possible to generate
the binman config file with a set of blobs each with a filename. But for
convenience, add an entry type which can do this.
Add required support (for adding nodes and string properties) into the
state module.
Signed-off-by: Simon Glass <sjg@chromium.org>
When tools are needed but not present, at present we just get an error
which can be confusing for the user. Try to be helpful by reporting the
tool as missing and suggesting a possible remedy.
Also update the Run() method to support this.
Signed-off-by: Simon Glass <sjg@chromium.org>
A recent rename of the function did not rename the test file. Fix this.
Fixes: 12308b128f (lib: fdtdec: Rename routine fdtdec_setup_memory_size())
Signed-off-by: Simon Glass <sjg@chromium.org>
When this fails it is useful to see the current directory, since U-Boot's
build system will typically change into the output directory during the
build. Add this information to the error.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tests use the 'test_result' feature to return a predetermined command
result for particular commands. The avoids needing to have the real
command available just to run a test. It works by calling the function
provided by the test, to get the value.
However sometimes the test does need to run the real command. Allow it to
fall back to do this when the function does not return a result.
Signed-off-by: Simon Glass <sjg@chromium.org>
Previously the first three words in a git-mailrc alias entry could only
be separated by spaces. git-send-email and Mutt both allow arbitrary
whitespace here.
Signed-off-by: Adam Sampson <ats@offog.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
This patch renames the routine fdtdec_setup_memory_size()
to fdtdec_setup_mem_size_base() as it now fills the
mem base as well along with size.
Signed-off-by: Siva Durga Prasad Paladugu <siva.durga.paladugu@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
At present only binman has the logic for determining Python test coverage
but this is useful for other tools also. Move it out into a separate file
so it can be used by other tools.
Signed-off-by: Simon Glass <sjg@chromium.org>
Some environments require providing the '--smtp-server' argument to
'git send-email'. Add support for this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a test to exercise the check for a valid SPDX license.
Signed-off-by: Chris Packham <judge.packham@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Many mailing-lists consider a long Cc list a sign of spam and will
either drop the message or mark it for moderation. Because patman
automatically invokes get_maintainer.pl the Cc list can expand
unexpectedly. Allow the user to specify a limit for the Cc list.
This limit is applied after removing any known bouncing addresses. By
default no limit is applied.
Signed-off-by: Chris Packham <judge.packham@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
The format of this line has changed. Update the patman test to suit.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Alex Kiernan <alex.kiernan@gmail.com>
When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.
In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.
This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.
Signed-off-by: Tom Rini <trini@konsulko.com>
All of these host tools are apparently written for Python2,
not Python3.
Use 'python2' in the shebang line according to PEP 394
(https://www.python.org/dev/peps/pep-0394/).
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
get_maintainer.pl quotes names which it considers unsafe, i.e. anything
containing [^a-zA-Z0-9_ \-]. This confuses patman, it will duplicate
addresses which are also in Series-to/cc. Strip the quotes.
Signed-off-by: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
This section of the settings file may be missing. Handle that gracefully
rather than emitting an error.
Also update patman to write this section when a new settings file is
created.
Fixes: e11aa602 (patman: add support for omitting bouncing addresses)
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Chris Packham <judge.pckham@gmail.com>
Tested-by: Bin Meng <bmeng.cn@gmail.com>
Add support for reading a list of bouncing addresses from a in-tree file
(doc/bounces) and from the ~/.patman config file. These addresses are
stripped from the Cc list.
Signed-off-by: Chris Packham <judge.packham@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com <mailto:philipp.tomsich@theobroma-systems.com>>
The existing test (patman --test) only covers basic checkpatch output.
We have had some problems with unicode processing and could use test
coverage for the various tags patman supports.
Add a new functional test which runs most of the patman flow on a few
test commits and checks that the results are correct.
See the documentation in the test for a description of what it does.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
This is not a good variable name in Python because 'list' is a type. It
shows up highlighted in some editors. Rename it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Allow the add_maintainers parameter to be a list of maintainers, thus
allowing us to simulate calling the script in tests without actually
needing it to work.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Add some unicode to the test patches to make sure that patman does the
right thing.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
There is no need for this function to return the same object that was
passed in. Drop the return value.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Unicode characters may appear in input patches so we should not warn about
them. Drop this warning.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
This is not a good variable name in Python because 'str' is a type. It
shows up highlighted in some editors. Rename it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
The communication filter reads data in blocks and converts each block to
unicode (if necessary) one at a time. In the unlikely event that a unicode
character in the input spans a block this will not work. We get an error
like:
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1022-1023:
unexpected end of data
There is no need to change the input to unicode, so the easiest fix is to
drop this feature.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Don't mess with the email address when outputting them. Just make sure
they are encoded with utf-8.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
This change encodes the CC list to UTF-8 to avoid failures on
maintainer-addresses that include non-ASCII characters (observed on
Debian 7.11 with Python 2.7.3).
Without this, I get the following failure:
Traceback (most recent call last):
File "tools/patman/patman", line 159, in <module>
options.add_maintainers)
File "[snip]/u-boot/tools/patman/series.py", line 234, in MakeCcFile
print(commit.patch, ', '.join(set(list)), file=fd)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 81: ordinal not in range(128)
from Heiko's email address:
[..., u'"Heiko St\xfcbner" <heiko@sntech.de>', ...]
While with this change added this encodes to:
"=?UTF-8?q?Heiko=20St=C3=BCbner?= <heiko@sntech.de>"
Signed-off-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>
Tested-by: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
os.read() returns a byte array in Python 3.5.2 and needs to be converted
into a string. Check if the returned value is an instance of bytes and
if it is decode it as a utf-8 string. If it is not a utf-8 encoded string
the decoding may fail with an exception.
Prior to this fix the comparisions check data == "" would fail when data
was b'' and would cause an infinite memory leaking loop. joins would
also fail with an exception below but due to the infinite loop it never
made it that far.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>
When gathering addresses for the Cc list patman would encounter a
UnicodeDecodeError due to non-ascii characters in the author name.
Address this by explicitly using utf-8 when building the Cc list.
Signed-off-by: Chris Packham <judge.packham@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>
It is convenient to be able to deal with checkpatch warnings in the same
way as build warnings. Tools such as emacs and kate can quickly locate
the source file and line automatically.
To achieve this, adjust the format to match the C compiler, and output to
stderr.
Signed-off-by: Simon Glass <sjg@chromium.org>
In python 3.x StringIO is no longer a module, and the class can instead
be found in the io module. Adjust the code in the doctest input to
account for both.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
In python 3.x the iteritems() method has been removed from dictionaries,
and the items() method does effectively the same thing. On python 2.x
using items() is a little less efficient since it involves copying data,
but as speed isn't a concern in this code switch to using items() anyway
for simplicity.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
In python 3.x module names used in import statements are case sensitive,
and the configparser module is named in all lower-case. Import it as such
in order to avoid errors when running with python 3.x.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
Syntax for exception handling is a little more strict in python 3.x.
Convert all uses to a form accepted by both python 2.x & python 3.x.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
In python 3.x, print must be used as a function call. Convert all print
statements to the function call style, importing from __future__ where
we print with no trailing newline or print to a file object.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
In preparation for running on python 3.x, which will refuse to run
scripts which mix tabs & spaces for indentation, replace 2 tab
characters present in series.py with spaces.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Simon Glass <sjg@chromium.org>
Output which does not include a newline will not be displayed unless
flushed. Add a flush to ensure that it becomes visible.
Signed-off-by: Simon Glass <sjg@chromium.org>
When tools want to display information of varying levels of importance, it
helps to provide the user with control over the verbosity of these messages.
Progress messages work best if they are displayed and then removed from the
display when no-longer relevant.
Add a new tout library (terminal out) to handle these tasks.
Signed-off-by: Simon Glass <sjg@chromium.org>
For tools which want to use input files and temporary output, it is useful
to have the handling of these dealt with in one place. Add a new library
which allows input files to be read, and output files to be written, all
based on a common directory structure.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is more useful to have this method raise an error when something goes
wrong. Make this the default and adjust the few callers that don't want to
use it this way.
Signed-off-by: Simon Glass <sjg@chromium.org>
The following python error:
Traceback (most recent call last):
File "./tools/patman/patman", line 144, in <module>
series = patchstream.FixPatches(series, args)
File "./tools/patman/patchstream.py", line 477, in FixPatches
commit = series.commits[count]
IndexError: list index out of range
is seen when:
- 'END' is missing in those tags
- those tags are put in the last part in a commit message
- the commit is not the last commit of the series
Add testing logic to see if a new commit starts.
Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>
'Series-changes' uses blank line to indicate its end. If that is
missing, series internal state variable 'in_change' may be wrong.
Correct its state.
Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>
If 'END' is missing in a 'Cover-letter' section, and that section
happens to show up at the very end of the commit message, and the
commit is the last commit of the series, patman fails to generate
cover letter for us. Handle this in CloseCommit of patchstream.
Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>
'Cover-letter', 'Series-notes' and 'Commit-notes' tags require an
'END' to be put at the end of its section. If we forget to put an
'END' in those sections, and these sections are followed by another
patman tag, patman generates incorrect patches. This adds codes to
handle such scenario.
Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>
Like other patman tags, use a new variable cover_match to indicate
a match for 'Cover-letter'.
Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Acked-by: Simon Glass <sjg@chromium.org>