Refactor the implementation slightly so that section data is not
rebuilt when it is already available.
We still have GetData() set up to rebuild the section, since we don't
currently track when things change that might affect a section. For
example, if a blob is updated within a section, we must rebuild it.
Tracking that would be possible but is more complex, so it left for
another time.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present this function adds up the total size of entries to work out the
size of a section's contents. With compression this is no-longer enough.
We may as well bite the bullet and build the section contents instead.
Call _BuildSectionData() to get the (possibly compressed) contents and
GetPaddedData() to get the same but with padding added.
Note that this is inefficient since the section contents is calculated
twice. Future work will improve this.
This affects testPackOverlapMap() since the error is reported with a
different section size now (enough to hold the contents). Update that at
the same time.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present this function assumes that the size of a section is at least as
large as its contents. With compression this is often not the case. Relax
this constraint by using the uncompressed size, if available.
Signed-off-by: Simon Glass <sjg@chromium.org>
This method introduces a separation between packing and checking that is
different for sections. In order to handle compression properly, we need
to be able to deal with a section's size being smaller than the
uncompressed size of its contents. It is easier to make this work if
everything happens in the Pack() method.
The only real user of CheckEntries() is entry_Section and it can call it
directly. Drop the call from 'control' and handle it locally.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present CheckSize() is called from the function that packs the entries.
Move it up to the main Pack() function so that _PackEntries() can just
do the packing.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present sorting and expanding entries are side-effects of the
CheckEntries() function. This is a bit confusing, as 'checking' would
not normally involve making changes.
Move these steps into the Pack() function instead.
Signed-off-by: Simon Glass <sjg@chromium.org>
This function just calls CheckEntries() in the only non-trivial
implementation. Drop it and use CheckEntries() directly.
Signed-off-by: Simon Glass <sjg@chromium.org>
When a section is compressed, all entries within it are grouped together
into a compressed block of data. This obscures the start of each
individual child entry.
Avoid reporting bogus 'image-pos' properties in this case, since it is
not possible to access the entry at the location provided. The entire
section must be decompressed first.
CBFS does not support compressing whole sections, only individual files,
so needs no special handling here.
Signed-off-by: Simon Glass <sjg@chromium.org>
Section contents is not set up when ObtainContents() is called, since
packing often changes the layout of the contents. Ensure that the contents
are correctly recorded by making this function regenerate the section. It
is normally only called by the parent section (when packing) or by the
top-level image code, when writing out the image. So the performance
impact is fairly small.
Now that sections have their contents in their 'data' property, update
testSkipAtStartSectionPad() to check it.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present padding of sections is inconsistent with other entry types, in
that different pad bytes are used.
When a normal entry is padded by its parent, the parent's pad byte is
used. But for sections, the section's pad byte is used.
Adjust logic to always do this the same way.
Note there is still a special case in entry_Section.GetPaddedData() where
an image is padded with the pad byte of the top-level section. This is
necessary since otherwise there would be no way to set the pad byte of
the image, without adding a top-level section to every image.
Signed-off-by: Simon Glass <sjg@chromium.org>
Each section is padded up to its size, if the contents are not large
enough. Move this logic from _BuildSectionData() to
GetPaddedDataForEntry() so that all the padding is in one place.
With this, the testDual test is working again, so enable it.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present this function does the padding needed around an entry. It is
easier to understand what is going on if we have a function that returns
the contents of an entry, with padding included.
Refactor the code accordingly, adding a new GetPaddedData() method.
Signed-off-by: Simon Glass <sjg@chromium.org>
Create a new _BuildSectionData() to hold the code that is now in
GetData(), so that it is clearly separated from entry.GetData() base
function.
Separate out the 'pad-before' processing to make this easier to
understand.
Unfortunately this breaks the testDual test. Rather than squash several
patches into an un-reviewable glob, disable the test for now.
This also affects testSkipAtStartSectionPad(), although it still not
quite what it should be. Update that temporarily for now.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we use 'compress' as the property to set the compression of
a 'files' entry. But this conflicts with the same property for entries,
of which Entry_section is a subclass.
Strictly speaking, since Entry_files is in fact a subclass of
Entry_section, the files can be compressed individually but also the
section (that contains all the files) can itself be compressed. With this
change, it is possible to express that.
Signed-off-by: Simon Glass <sjg@chromium.org>
While a section is the base class of Image, it is more correct to refer
to sections in most places in this file. Fix these comments.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we have an Entry_blob_ext which implement a blob which holds an
external binary. We need to support other entry types that hold external
binaries, e.g. Entry_blob_named_by_arg. Move the support into the base
Entry class to allow this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Other relevant properties (pad-after, offset, size, align, align-size,
align-end) already work since Pack() sets correct ranges for subentries'
data (.offset, .size variables), but some padding here is necessary to
align the data within this range to match the pad-before property.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Switch to str.startswith for matching like the FIT etype does since the
current version doesn't ignore 'hash-1', 'hash-2', etc.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Some binary blobs unfortunately obtain their position in the image from
other binary blobs, such as Intel's 'descriptor'. In this case we cannot
rely on packing to work. It is not possible to produce a valid image in
any case, due to the missing blobs.
Allow zero-length overlaps so that this does not cause any problems.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.
U-Boot does with with ATF (ARM Trusted Firmware) today.
Add a new flag to binman to request this behaviour.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
It is easier and less error-prone to use super() when the parent type is
needed. Update binman to remove the type names.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Some binary blobs unfortunately obtain their position in the image from
other binary blobs, such as Intel's 'descriptor'. In this case we cannot
rely on packing to work. It is not possible to produce a valid image in
any case, due to the missing blobs.
Allow zero-length overlaps so that this does not cause any problems.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.
U-Boot does with with ATF (ARM Trusted Firmware) today.
Add a new flag to binman to request this behaviour.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
It is easier and less error-prone to use super() when the parent type is
needed. Update binman to remove the type names.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.
Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).
Signed-off-by: Simon Glass <sjg@chromium.org>
At present binman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.
Move binman to use absolute imports. This enables removable of the path
adjusting in Entry also.
Signed-off-by: Simon Glass <sjg@chromium.org>
A recent change adjusted the symbol calculation to work on x86 but broke
it for Tegra. In fact this is because they have different needs.
On x86 devices the code is linked to a ROM address and the end-at-4gb
property is used for the image. In this case there is no need to add the
base address of the image, since the base address is already built into
the offset and image-pos properties.
On other devices we must add the base address since the offsets start at
zero.
In addition the base address is currently added to the 'offset' and 'size'
values. It should in fact only be added to 'image-pos', since 'offset' is
relative to its parent and 'size' is not actually an address. This code
should have been adjusted when support for 'image-pos' and 'size' was
added, but it was not.
To correct these problems:
- move the code that handles adding the base address to section.py, which
can check the end-at-4gb property and which property
(offset/size/image-pos) is being read
- add the base address only when needed (only for image-pos and not if the
image uses end-at-4gb)
- add a note to the documentation
- add a separate test to cover x86 behaviour
Fixes: 15c981cc (binman: Correct symbol calculation with non-zero image base)
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>
It is useful to be able to access the size of an image in SPL, with
something like:
binman_sym_declare(unsigned long, u_boot_any, size);
...
ulong u_boot_size = binman_sym(ulong, u_boot_any, size);
Add support for this and update the tests.
Signed-off-by: Simon Glass <sjg@chromium.org>
Entries which include a section and need to obtain its contents call
GetData(), as with any other entry. But the current implementation of this
method in entry_Section requires the size of the section to be known. If
it is unknown, an error is produced, since size is None:
TypeError: can't multiply sequence by non-int of type 'NoneType'
There is no need to know the size in advance since the code can be
adjusted to build up the section piece by piece, instead of patching each
entry into an existing bytearray.
Update the code to handle this and add a test.
Signed-off-by: Simon Glass <sjg@chromium.org>
Some x86 sections have special offsets which currently result in empty
data being returned from the 'extract' command. Fix this by taking account
of the skip-at-start property.
Add a little more debugging while we are here.
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Bin Meng <bmeng.cn@gmail.com>
At present this method assumes that the parent section does not need
to recalculate its position or adjust any metadata it may contain. But
when the entry changes size this may not be true. Also if the parent
section is more than just a container (e.g. it is a CBFS) then the
section may need to regenerate its output.
Add a new WriteChildData() method to sections and call this from the
WriteData() method, to handle this situation.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we simply extract the data directly from entries using the
image_pos information. This happens to work on current entry types, but
cannot work if the entry type encodes the data in some way. Update the
ReadData() method to provide the data by calling a new ReadChildData()
method in the parent. This allows the entry_Section class, or possibly
any other container class, to return the correct data in all cases.
Signed-off-by: Simon Glass <sjg@chromium.org>
The positioning does not currently work correctly if at the end of an
image with no fixed size. Also if the header is in the middle of an image
it can cause a gap in the image since the header position is normally at
the image end, so entries after it are placed after the end of the image.
Fix these problems and add more tests to cover these cases.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we have an 'image' property in the entry for this purpose, but
this is not necessary and seems error-prone in the presence of
inheritance. Add a function instead. The Entry_section class overrides
this with a special version, since top-level sections are in fact images,
since Image inherits Entry_section.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present the Entry constructor sets up the object and then immediately
reads its device-tree node to obtain its properties.
This breaks a convention that constructors should not do any processing.
A consequence is that we must pass all arguments to the constructor and
cannot have the node-reading proceed in a different way unless we pass
flags to that constructor. We already have a 'test' flag in a few cases,
and now need to control whether the 'orig_offset' and 'orig_size'
properties are set or not.
Adjust the code to require a separate call to ReadNode() after
construction. The Image class remains as it was.
Signed-off-by: Simon Glass <sjg@chromium.org>
When modifying an image it is convenient to load the data from the file
into each entry so that it can be reprocessed. Add a new LoadData() method
to handle this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Use the new logging feature to log information about progress with
packing. This is useful to see how binman is figuring things out.
Also update elf.py to use the same feature.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present this function returns a set of device-tree filenames. It has no
way of returning the actual device-tree object. Change it to a dictionary
so that we can add this feature in a future patch.
Also drop fdt_set since it is no-longer used.
Signed-off-by: Simon Glass <sjg@chromium.org>
When support for sections (and thus hierarchical images) was added to
binman, the decision was made to create a new Section class which could
be used by both Image and an Entry_section class. The decision between
using inheritance and composition was tricky to make, but in the end it
was decided that Image was different enough from Entry that it made sense
to put the implementation of sections in an entirely separate class. It
also has the advantage that core Image code does have to rely on an entry
class in the etype directory.
This work was mostly completed in commit:
8f1da50ccc "binman: Refactor much of the image code into 'section'
As a result of this, the Section class has its own version of things like
offset and size and these must be kept in sync with the parent
Entry_section class in some cases.
In the last year it has become apparent that the cost of keeping things in
sync is larger than expected, since more and more code wants to access
these properties.
An alternative approach, previously considered and rejected, now seems
better.
Adjust Image to be a subclass of Entry_section. Move the code from Section
(in bsection.py) to Entry_section and delete Section. Update all tests
accordingly.
This requires substantial changes to Image. Overall the changes reduce
code size by about 240 lines. While much of that is just boilerplate from
Section, there are quite a few functions in Entry_section which now do not
need to be overiden from Entry. This suggests the change is beneficial
even without further functionality being added.
A side benefit is that the properties of sections are now consistent with
other entries. This fixes a problem in testListCmd() where some properties
are missing for sections.
Unfortunately this is a very large commit since it is not feasible to do
the migration piecemeal. Given the substantial tests available and the
100% code coverage of binman, we should be able to do this safely.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is useful to be able to summarise all the entries in an image, e.g. to
display this to this user. Add a new ListEntries() method to Entry, and
set up a way to call it through the Image class.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add support for detecting entries that change size after they have already
been packed, and re-running packing when it happens.
This removes the limitation that entry size cannot change after
PackEntries() is called.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present if this function tries to update the contents such that the
size changes, it raises an error. We plan to add the ability to change
the size of entries after packing is completed, since in some cases it is
not possible to determine the size in advance.
An example of this is with a compressed device tree, where the values
of the device tree change in SetCalculatedProperties() or
ProcessEntryContents(). While the device tree itself does not change size,
since placeholders for any new properties have already bee added by
AddMissingProperties(), we cannot predict the size of the device tree
after compression. If a value changes from 0 to 0x1234 (say), then the
compressed device tree may expand.
As a first step towards supporting this, make ProcessContentsUpdate()
return a value indicating whether the content size is OK. For now this is
always True (since otherwise binman raises an error), but later patches
will adjust this.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present sections are always placed automatically. Even if an 'offset'
property is provided it is ignored. Update the logic to support an offset
for sections.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
