fix(media): stream large files and honour size settings (#3387)

2026-10-17 21:15:32 +09:00 · 2026-10-11 13:28:24 +02:00
1 parent 6acd7f3d22
commit 45b78a13c2
@@ -435,7 +435,7 @@ Updates the bot account an application owns and returns the resulting [bot profi

 <sup>2</sup> Accepted only when it equals the bot's current discriminator. Any other value returns the field code `BOT_DISCRIMINATOR_CANNOT_BE_CHANGED`

-<sup>3</sup> The value can include a `data:...;base64,` prefix, which Fluxer strips before it measures the payload. The remaining base64 payload is bounded at 13981016 characters for both fields, the encoded length of a full 10485760-byte image, and a longer value is rejected with the field code `BASE64_LENGTH_INVALID`
+<sup>3</sup> The value can include a `data:...;base64,` prefix, which Fluxer strips before it measures the payload. The remaining base64 payload is bounded at 139810136 characters for both fields, the encoded length of a 104857600-byte image, and a longer value is rejected with the field code `BASE64_LENGTH_INVALID`. The decoded image must fit the resolved `avatar_max_size` [limit](/http-api/instance/#limit-keys), 10485760 bytes by default, or it is rejected with `IMAGE_SIZE_EXCEEDS_LIMIT`

 <sup>4</sup> Rejected with 403 `CONTENT_BLOCKED` when content moderation or the profile substring blocklist blocks the value

@@ -531,12 +531,12 @@ Every field is optional, and an omitted field preserves its current value.
 | Field | Type | Description |
 | --- | --- | --- |
 | name? | ?string | The name of the group (1-100 characters after normalisation), or null to clear it |
-| icon?<sup>1</sup> | ?base64 string | The base64-encoded icon image (at most 13,981,014 encoded characters), or null to clear it |
+| icon?<sup>1</sup> | ?base64 string | The base64-encoded icon image (at most 139,810,136 encoded characters), or null to clear it |
 | owner_id?<sup>2</sup> | ?snowflake | The ID of the new owner |
 | nicks?<sup>3</sup> | ?map[snowflake, ?string] | The nicknames keyed by the decimal user ID (each 0-32 characters, or null) |
 | nsfw?<sup>4</sup> | ?boolean | Whether the group is marked for mature content |

-<sup>1</sup> Raw base64 or a data URL, with 1 through 13,981,014 characters after the prefix
+<sup>1</sup> Raw base64 or a data URL, with 1 through 139,810,136 characters after the prefix

 <sup>2</sup> An explicit null requests no transfer. The caller must be the current owner and the target a current recipient

@@ -58,9 +58,9 @@ The image and name submitted for one new emoji.

 <sup>3</sup> The decoded ceiling is the resolved `emoji_max_size` [limit key](/http-api/instance/#limit-keys) in guild scope, 524288 bytes by default

-The remaining encoded part is bounded to 699052 characters, the base64 expansion of the 524288 byte default. A longer value returns the validation code `BASE64_LENGTH_INVALID` at the `image` path. A value that is not canonical base64 with correct padding returns `INVALID_BASE64_FORMAT`. An image over the resolved ceiling returns `IMAGE_SIZE_EXCEEDS_LIMIT` at the same path, and `maxSize` is the ceiling that applied.
+The remaining encoded part is bounded to 139810136 characters, the base64 expansion of a 104857600 byte image. A longer value returns the validation code `BASE64_LENGTH_INVALID` at the `image` path. A value that is not canonical base64 with correct padding returns `INVALID_BASE64_FORMAT`. An image over the resolved ceiling returns `IMAGE_SIZE_EXCEEDS_LIMIT` at the same path, and `maxSize` is the ceiling that applied.

-The 699052 character bound does not move with the limit key, so a resolved ceiling above 524288 bytes admits no more than 524289 decoded bytes.
+The character bound does not move with the limit key, so a resolved ceiling above 104857600 bytes admits no more than 104857602 decoded bytes.

 Accepted upload formats are PNG, JPEG, WebP, GIF, APNG, AVIF, HEIC, HEIF, JXL, and SVG. Fluxer takes the format and the animation state from the decoded image. An animated AVIF is rejected.

@@ -127,7 +127,7 @@ The request body shared by [Modify current guild member](#modify-current-guild-m

 A `roles` entry that does not resolve to an existing role of the guild is dropped from the replacement, so a request naming only unknown roles clears the member's role set.

-An `avatar` or `banner` base64 payload longer than 13981016 characters returns `BASE64_LENGTH_INVALID`, and a malformed one returns `INVALID_BASE64_FORMAT`. Fluxer then checks the decoded bytes against the instance-configured `avatar_max_size` [limit](/http-api/instance/#limit-keys), whose stock value is 10485760. The same limit applies to both fields. The decoded image must also be in a format the field accepts, and an animated AVIF is rejected. Pixel dimensions are not enforced. A value that is too large returns `IMAGE_SIZE_EXCEEDS_LIMIT`, and one whose format or animation is not allowed returns `INVALID_IMAGE_FORMAT`.
+An `avatar` or `banner` base64 payload longer than 139810136 characters returns `BASE64_LENGTH_INVALID`, and a malformed one returns `INVALID_BASE64_FORMAT`. Fluxer then checks the decoded bytes against the instance-configured `avatar_max_size` [limit](/http-api/instance/#limit-keys), whose stock value is 10485760. The same limit applies to both fields. The decoded image must also be in a format the field accepts, and an animated AVIF is rejected. Pixel dimensions are not enforced. A value that is too large returns `IMAGE_SIZE_EXCEEDS_LIMIT`, and one whose format or animation is not allowed returns `INVALID_IMAGE_FORMAT`.

 Null and any `communication_disabled_until` that is not in the future both clear the timeout. A time more than 365.25 days ahead returns `TIMEOUT_CANNOT_EXCEED_365_DAYS`, and a value that passes the schema but is not a real instant returns `INVALID_TIMEOUT_VALUE`. `timeout_reason` has no effect unless `communication_disabled_until` is also supplied.

@@ -68,7 +68,7 @@ The item shape shared by [Create guild sticker](#create-guild-sticker) and each
 | name<sup>1</sup> | string | The name of the sticker (2-30 characters) |
 | description?<sup>2</sup> | ?string | The sticker description (1-500 characters), or null for none |
 | tags? | array[string] | [Sticker tags](#sticker-tags), defaulting to an empty array |
-| image<sup>3</sup> | base64 string | The image data, with at most 524288 decoded bytes |
+| image<sup>3</sup> | base64 string | The image data (at most the resolved `sticker_max_size` limit in decoded bytes, 524288 by default) |

 <sup>1</sup> The value is trimmed before the bound applies, as it is for `description` and each tag

@@ -76,9 +76,9 @@ The item shape shared by [Create guild sticker](#create-guild-sticker) and each

 <sup>3</sup> A data URL prefix is accepted and everything up to the first comma is removed before any bound applies

-The remaining encoded part is bounded to 699052 characters, which encodes a full 524288 byte image, and a longer value returns the validation code `BASE64_LENGTH_INVALID` at the `image` path. A value that is not canonical base64 with correct padding returns `INVALID_BASE64_FORMAT`.
+The remaining encoded part is bounded to 139810136 characters, which encodes a 104857600 byte image, and a longer value returns the validation code `BASE64_LENGTH_INVALID` at the `image` path. A value that is not canonical base64 with correct padding returns `INVALID_BASE64_FORMAT`.

-Decoded bytes are at most the operator-configured `sticker_max_size` [limit key](/http-api/instance/#limit-keys) value, resolved against the guild's complete feature set and defaulting to 524288. A larger image is rejected on the `image` path with `IMAGE_SIZE_EXCEEDS_LIMIT`, whose message names the resolved ceiling. An image over 524289 bytes exceeds the 699052 character bound and returns `BASE64_LENGTH_INVALID`, so a configured value above 524289 never applies.
+Decoded bytes are at most the operator-configured `sticker_max_size` [limit key](/http-api/instance/#limit-keys) value, resolved against the guild's complete feature set and defaulting to 524288. A larger image is rejected on the `image` path with `IMAGE_SIZE_EXCEEDS_LIMIT`, whose message names the resolved ceiling. The character bound does not move with the limit key, so a configured value above 104857600 admits no more than 104857602 decoded bytes.

 Accepted upload formats are PNG, JPEG, APNG, GIF, WebP, AVIF, and SVG. Fluxer takes the format from the decoded image. An animated AVIF is rejected. Fluxer detects animation state from the image, so this object has no animation field.

@@ -522,7 +522,7 @@ Creates a guild owned by the caller. Requires a user session credential. Returns

 A `name` outside 1 to 100 characters after normalisation returns `STRING_LENGTH_INVALID`. A blocked name returns 403 `CONTENT_BLOCKED`.

-The `icon` base64 payload is bounded to 1 to 13981016 characters and is otherwise rejected with `BASE64_LENGTH_INVALID`, and a payload that is not valid base64 with `INVALID_BASE64_FORMAT`. The decoded image must fit within the instance `avatar_max_size` limit, which defaults to the 10 MiB ceiling, and a larger image is rejected with `IMAGE_SIZE_EXCEEDS_LIMIT`. PNG, JPEG, WebP, GIF, APNG, AVIF, HEIC, HEIF, JXL, and SVG are accepted. An animated AVIF is rejected with `INVALID_IMAGE_FORMAT`.
+The `icon` base64 payload is bounded to 1 to 139810136 characters and is otherwise rejected with `BASE64_LENGTH_INVALID`, and a payload that is not valid base64 with `INVALID_BASE64_FORMAT`. The decoded image must fit within the instance `avatar_max_size` limit, which defaults to the 10 MiB ceiling, and a larger image is rejected with `IMAGE_SIZE_EXCEEDS_LIMIT`. PNG, JPEG, WebP, GIF, APNG, AVIF, HEIC, HEIF, JXL, and SVG are accepted. An animated AVIF is rejected with `INVALID_IMAGE_FORMAT`.

 Without a template the guild is created with a `Text Channels` category holding a text channel named `general`, a `Voice Channels` category holding a voice channel named `General`, and an everyone role with the default permission set plus `CREATE_PUBLIC_THREADS`, `CREATE_PRIVATE_THREADS`, and `SEND_MESSAGES_IN_THREADS`. The `general` channel becomes the system channel.

@@ -346,8 +346,8 @@ Each key names one limit. A key whose name begins with `feature_` is a feature g

 | Value | Description |
 | --- | --- |
-| avatar_max_size | Maximum file size for avatar uploads in bytes, in user scope |
-| emoji_max_size | Maximum file size for emoji uploads in bytes, in guild scope |
+| avatar_max_size | Maximum file size for avatar uploads in bytes, effective up to 104857600, in user scope |
+| emoji_max_size | Maximum file size for emoji uploads in bytes, effective up to 104857600, in guild scope |
 | feature_animated_avatar | Allow animated avatar uploads, in user scope |
 | feature_animated_banner | Allow animated banner uploads, in user scope |
 | feature_custom_discriminator | Allow custom discriminator selection, in user scope. Ignored for people on an instance with the `none` tag style |
@@ -389,7 +389,7 @@ Each key names one limit. A key whose name begins with `feature_` is a feature g
 | max_voice_message_duration | Maximum voice message duration in seconds, in user and guild scope |
 | max_webhooks_per_channel | Maximum webhooks per channel, in guild scope |
 | max_webhooks_per_guild | Maximum webhooks per guild, in guild scope |
-| sticker_max_size | Maximum file size for sticker uploads in bytes, in guild scope |
+| sticker_max_size | Maximum file size for sticker uploads in bytes, effective up to 104857600, in guild scope |

 <sup>1</sup> Emoji limits are enforced as one shared total against `max_guild_emojis`, and the aliases exist only for older clients. The default of each alias equals the `max_guild_emojis` default

@@ -98,7 +98,7 @@ Modifies the current account and returns the resulting [user](/http-api/users/#u

 <sup>5</sup> A claimed account also supplies `password`, and a body that omits it is rejected with `PASSWORD_NOT_SET`. When the instance checks for breached passwords, a breached one is rejected with [`PASSWORD_IS_TOO_COMMON`](/http-api/errors/)

-<sup>6</sup> The value may have a `data:...;base64,` prefix. The remaining payload runs 1 to 13981016 characters
+<sup>6</sup> The value may have a `data:...;base64,` prefix. The remaining payload runs 1 to 139810136 characters

 <sup>7</sup> Every non-null value requires the animated banner entitlement, which gates every profile banner, and is otherwise rejected with [`BANNERS_REQUIRE_PREMIUM`](/http-api/errors/)

@@ -839,7 +839,7 @@ The guild allowance is the resolved [max_webhooks_per_guild](/http-api/instance/
 | name | string | Webhook name (1-80 characters) |
 | avatar?<sup>1</sup> | ?base64 string | Base64-encoded avatar image, or null |

-<sup>1</sup> A leading data URI header is stripped at the first comma before decoding, and the base64 payload is 1 through 13981016 characters
+<sup>1</sup> A leading data URI header is stripped at the first comma before decoding, and the base64 payload is 1 through 139810136 characters

 The decoded bytes must be at most the resolved [avatar_max_size](/http-api/instance/#limit-keys) value, which is the 10 MiB ceiling of 10485760 bytes by default. The image must be an accepted avatar upload format, and an animated AVIF is rejected.

@@ -23,9 +23,11 @@ Every media read route can produce these statuses in addition to the ones in its
 | 403 | Forbidden | The [cross-origin read policy](/media-proxy/overview/#cross-origin-reads) refuses the request `Origin` on an `mp` or `upload` endpoint |
 | 404 | Not found | The path is not a media route, the endpoint runs in `relay` mode, or the object does not exist |
 | 405 | Method not allowed | The method is neither `GET` nor `HEAD` |
-| 413 | Payload too large | The stored object exceeds the 500 MiB media bound |
+| 413 | Payload too large | The stored object exceeds the 500 MiB media bound and the route neither streams it unchanged nor spools it |
 | 502 | Bad gateway | The object store could not be read |

+The 500 MiB media bound applies where a route holds an object in memory. A read that streams the stored object unchanged, such as an attachment read without a transformation parameter, has no size bound. An attachment read that asks for a frame of a video above the bound copies the video to the [spool directory](/operator/configuration/#media-proxy-settings) first, one video at a time, and returns 413 when the video exceeds the spool ceiling. Any other transformation of an object above the bound returns 413.
+
 ## Get attachment

 <RouteHeader method="GET" path="/attachments/{path}" unauthenticated />
@@ -1379,7 +1379,7 @@ Default `536870912`, 512 MiB. The memory all buffered relay uploads may hold at

 #### `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_SPOOL_DIR`

-Defaults to the system temporary directory. Where relay bodies spool. Must be writable. Compose forwards it from `.env`.
+Defaults to the system temporary directory. Where relay bodies spool, and where `media-proxy` copies a stored audio or video object above 500 MiB when it has to read inside the file. That happens when the API asks for the media metadata or a frame of an upload or a stored object, and when an attachment read asks for a frame of a video. Must be writable. Compose forwards it from `.env`.

 #### `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_SPOOL_CHUNK_BYTES`

@@ -1387,7 +1387,11 @@ Default `1048576`. Spool chunk size. Accepts 64 KiB to 64 MiB. Compose forwards

 #### `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_SPOOL_MAX_TOTAL_BYTES`

-Default `8589934592`. Total spool ceiling. Accepts 0 to 256 GiB, and must be at or above `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_MAX_BODY_BYTES`, which puts the usable floor at 500 MiB by default. Compose forwards it from `.env`.
+Default `8589934592`. Total spool ceiling, shared by relay bodies and the copies of stored audio and video above 500 MiB. A relay upload that does not fit beside the copies in progress fails. An upload larger than the ceiling is stored as a plain file without media metadata, and a stored video larger than the ceiling has no poster frame. Accepts 0 to 256 GiB, and must be at or above `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_MAX_BODY_BYTES`, which puts the usable floor at 500 MiB by default. Compose forwards it from `.env`.
+
+Sending an audio or video attachment above 500 MiB holds up to three copies of it on local disk at once. `media-proxy` holds one in the spool directory, and `api` holds the download and the copy it rewrites without metadata in its temporary directory. Leave free space of three times the attachment size limit for each send you expect at the same moment.
+
+`api` waits 30 seconds for the metadata of an upload up to 500 MiB. Above that it waits 30 seconds plus one second for each 16 MiB of the upload, about 94 seconds for 1 GiB and 542 seconds for 8 GiB. In that time `media-proxy` copies the upload out of object storage, reads it, and hashes it. A deployment that cannot sustain 16 MiB per second through those steps stores the upload as a plain file.

 #### `FLUXER_MEDIA_PROXY_MAX_NATIVE_TRANSFORMS`

@@ -1765,6 +1769,12 @@ The premium tier name and information link, the billing switch, the Stripe secre

 Size-based attachment lifetimes. The built-in default is on.

+Turning expiry off hides every expiry date at once and stops deletion. The stored dates are kept.
+
+On a self-hosted instance, changes also apply to attachments that were uploaded earlier. Turning expiry on, or changing a size or lifetime rule, starts a background job that works through existing messages in chunks and recalculates the expiry of their attachments. No attachment is deleted while that job runs. An attachment whose date passed while expiry was off, or that the new settings would expire sooner, is kept for at least the renew window first. Until the job reaches a message, its attachments show the state they had before, so one whose date passed while expiry was off shows as expired until then.
+
+The official hosted instance does not run that job. There, a change applies to attachments uploaded from then on, and deletion of expired attachments is never paused.
+
 #### Instance Config, Gateway Rollout Configuration

 Session and guild rollout percentages, RPC timeouts, and Gateway concurrency.