pepepper/fluxer
1
0 Fork 0
mirror of https://github.com/fluxerapp/fluxer synced 2026-10-18 21:45:31 +09:00
Code Issues Packages Projects Releases Wiki Activity

feat(media-proxy): sign attachment URLs and gate origins (#2830)

Browse Source
This commit is contained in:
Hampus authored and GitHub committed 2026-09-18 15:57:32 +02:00
1 parent 025c01ab13
commit 522cf08e61
108 files changed
+10148 -441
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

fluxer_docs/src/content/docs/http-api/memes.mdx
+1 -1
View File
@@ -54,7 +54,7 @@ One media asset owned by one account. Fluxer resolves every media field at creat
<sup>2</sup> [Save meme from message](#save-meme-from-message) falls back to the selected attachment's own description when the body supplies no non-empty alt text
<sup>3</sup> `url` is the media endpoint followed by `attachments/{user_id}/{attachment_id}/{filename}`
<sup>3</sup> `url` is the media endpoint followed by `attachments/{user_id}/{attachment_id}/{filename}`, and Fluxer signs it on every read as a [signed attachment URL](/http-api/messages/#signed-attachment-urls)
<sup>4</sup> Set when the source attachment or embed was already animated, and when the Media Proxy reports the stored copy as animated
fluxer_docs/src/content/docs/http-api/messages.mdx
+111 -7
View File
@@ -150,8 +150,8 @@ An attachment is one file stored against a message and served from media deliver
| content_type<sup>1</sup> | string | The media type detected for the attachment |
| content_hash | ?string | The hash of the attachment's stored bytes, or null when they were never hashed |
| size | integer | The stored size of the attachment in bytes |
| url<sup>2</sup> | ?string | Public attachment URL, or null once the attachment has expired |
| proxy_url<sup>2</sup> | ?string | Media Proxy URL, or null once the attachment has expired |
| url<sup>2</sup> | ?string | Signed public attachment URL, or null once the attachment has expired |
| proxy_url<sup>2</sup> | ?string | Signed Media Proxy URL, or null once the attachment has expired |
| width?<sup>3</sup> | integer | Pixel width for visual media |
| height?<sup>3</sup> | integer | Pixel height for visual media |
| placeholder? | string | Encoded media placeholder |
@@ -164,14 +164,47 @@ An attachment is one file stored against a message and served from media deliver
<sup>1</sup> Always present. It falls back to a type guessed from the stored filename when the attachment has no recorded media type, and to `application/octet-stream` when the filename yields no guess
<sup>2</sup> Both fields are the same canonical media endpoint URL, and both become null together once the attachment has expired
<sup>2</sup> Both fields are the same media endpoint URL with the same [signature](#signed-attachment-urls), and both become null together once the attachment has expired
<sup>3</sup> Omitted for an audio media type even when a stored dimension exists
<sup>4</sup> The key is always present. Its value is the recorded decay deadline, or null when the attachment has none
<sup>4</sup> The key is always present. Its value is the recorded decay deadline, or null when the attachment has none. It is unrelated to the `ex` time of a signed URL
<sup>5</sup> Present and true only when the attachment has already expired, so an absent key must be read as false
A new attachment starts with the media type derived from the filename given when its upload was requested. For a pre-uploaded attachment that is the `filename` of its [attachment upload request item](#attachment-upload-request-item-object), which can differ from the `filename` the message sends. When the starting type is not a supported image, video, or audio type, Fluxer checks the first 8 KiB of the file for one. For an image, video, or audio file the Media Proxy can read, `content_type` is then the type detected from its bytes. When the starting type is not a media type, SVG bytes keep it. So does a file Fluxer cannot check or the Media Proxy cannot read.
### Signed attachment URLs
An instance configured with attachment URL secrets signs `url` and `proxy_url`. Fluxer adds the signature parameters to the attachment path, before any other parameter. An instance without those secrets returns the URL with none of them.
| Field | Type | Description |
| --- | --- | --- |
| ex | string | The Unix time in seconds at which the URL expires, as 8 lowercase hexadecimal digits, or `0` on a data package URL |
| is | string | The Unix time in seconds at which the URL was issued, as 8 lowercase hexadecimal digits |
| hm | string | The signature over `ex`, `is`, `uc`, and the attachment path, as 64 lowercase hexadecimal digits |
| uc | string | Present only on a data package URL, where the value is always `dp` |
An ordinary URL has `ex`, `is`, and `hm`, and no `uc`. It is valid for 24 hours from `is`. `is` starts at the attachment's creation time and moves forward in steps of 12 hours, so a URL Fluxer returns has more than 12 and at most 24 hours left. Fluxer signs the URL each time it returns the attachment, so a later read of the message returns a URL for the current step. [Refresh attachment URLs](#refresh-attachment-urls) signs a URL again without reading the message.
A client MUST keep `ex`, `is`, `hm`, and `uc` exactly as issued. It MAY add any other query parameter, such as a [transformation](/media-proxy/transformations/), because the signature covers no other parameter. A client SHOULD refresh a URL whose `ex` is absent or less than one hour away. Once the operator enforces signatures, a read with a missing, altered, or expired signature returns 404 from the Media Proxy, as [Signed attachment URLs](/media-proxy/overview/#signed-attachment-urls) defines.
Fluxer never rewrites message `content` or embed text. An attachment URL pasted into either keeps the parameters it was sent with, and it stops working when its own `ex` passes.
### Data package attachment URLs
A [data harvest](/http-api/users/data-harvest/) archive records a data package URL for each attachment instead of an ordinary one. It has `uc=dp`, and its `ex` is the single character `0`:
```text
?ex=0&is={issued}&hm={signature}&uc=dp
```
A data package URL does not expire. The Media Proxy checks its signature against every configured secret and skips the expiry check, so it reads its object for as long as the secret that signed it stays in the instance's secret list. Removing that secret is the only thing that ends it, and it ends every data package URL that secret signed at once.
`is` is set the way it is on an ordinary URL, and the signature covers `uc`, so an ordinary URL relabelled `uc=dp` does not verify, and neither does a data package URL with `uc` removed.
A client treats a data package URL as never expiring and does not refresh it before use. A refused read still triggers the same forced refresh as any other attachment URL, and that returns an ordinary URL.
### Example
```json
@@ -183,8 +216,8 @@ An attachment is one file stored against a message and served from media deliver
"content_type": "image/png",
"content_hash": "9f86d081884c7d659a2feaa0c55ad015",
"size": 18422,
"url": "https://media.example.com/attachments/1501314428688998182/1501320000000000001/diagram.png",
"proxy_url": "https://media.example.com/attachments/1501314428688998182/1501320000000000001/diagram.png",
"url": "https://media.example.com/attachments/1501314428688998182/1501320000000000001/diagram.png?ex=69fba4b5&is=69fa5335&hm=5586be3ae70411fe6582431649e2f28abc6c201e4f12a4f43a3256b72b90d3de",
"proxy_url": "https://media.example.com/attachments/1501314428688998182/1501320000000000001/diagram.png?ex=69fba4b5&is=69fa5335&hm=5586be3ae70411fe6582431649e2f28abc6c201e4f12a4f43a3256b72b90d3de",
"width": 1280,
"height": 720,
"flags": 0,
@@ -440,6 +473,10 @@ Every optional field above is omitted entirely.
<sup>1</sup> Always derived from `url`, so every serialised media object has it
When `url` is an attachment URL of this instance, Fluxer signs both fields, so `url` and `proxy_url` are the same URL with the same fresh [signature](#signed-attachment-urls). A rich embed can name such a URL with or without `ex`, `is`, `hm`, and `uc`. Fluxer removes them before it stores the URL, and signs both fields again each time it returns the embed.
Fluxer signs every other embed field that can hold an attachment URL of this instance the same way. That is `url` on the embed itself, `url`, `icon_url`, and `proxy_icon_url` on an [embed author](#embed-author-object), `icon_url` and `proxy_icon_url` on an [embed footer](#embed-footer-object), and `url` on an [embed provider](#embed-provider-object). An icon URL and its proxy twin are then the same signed URL. A field whose URL is not an attachment URL of this instance is returned as it was stored, and a proxy field of an external URL is the [signed external media](/media-proxy/routes/#get-signed-external-media) path instead.
## Embed footer object
### Structure
@@ -660,7 +697,7 @@ One declared upload inside a [Request attachment upload URLs](#request-attachmen
<sup>1</sup> A decimal string is converted to the integer
<sup>2</sup> The issued upload URL and the stored attachment both use the media type derived from `filename`
<sup>2</sup> The issued upload URL uses the media type derived from `filename`. The [message attachment object](#message-attachment-object) states how the stored attachment is typed
### Singlepart attachment upload object
@@ -726,6 +763,28 @@ One planned upload named in a [Complete attachment upload](#complete-attachment-
| upload_filename | string | Opaque upload key returned when the upload was planned |
| upload_id | string | Multipart upload identifier returned when the upload was planned |
## Refreshed attachment URL object
One result of [Refresh attachment URLs](#refresh-attachment-urls).
### Structure
| Field | Type | Description |
| --- | --- | --- |
| original | string | The URL exactly as it was sent |
| refreshed<sup>1</sup> | string | The URL with a fresh [signature](#signed-attachment-urls), or `original` unchanged |
<sup>1</sup> Equal to `original` when the URL is not an attachment URL of this instance, or when the instance has no attachment URL secrets
A URL is an attachment URL of this instance when all of these hold:
- It parses as an absolute URL whose scheme, host, and port equal those of the media endpoint. The host is compared without regard to case, and a default port equals no port.
- Its path, as written in the string, starts with the media endpoint path followed by `/attachments/`.
- The rest of the path, percent-decoded, is valid UTF-8 and a safe storage key with no empty, `.`, or `..` segment.
- That key is `attachments/{channel_id}/{attachment_id}/{filename}`, where both IDs are decimal digits and the filename is not empty.
To sign a URL, Fluxer removes every `ex`, `is`, `hm`, and `uc` parameter and every empty pair, then places the fresh `ex`, `is`, and `hm` first. Every other parameter follows in its original order and bytes, and a fragment is kept. A [data package URL](#data-package-attachment-urls) loses its `uc` this way, so the result is an ordinary URL valid for 24 hours.
## Channel pin object
A channel pin pairs one pinned message with the time it was pinned.
@@ -1019,6 +1078,51 @@ The operation makes each completed upload available for a message. It creates no
10 requests per 10 seconds for each authenticated user and channel ID, on the `channel:attachment:upload::channel_id` bucket, which is shared with [Request attachment upload URLs](#request-attachment-upload-urls).
## Refresh attachment URLs
<RouteHeader method="POST" path="/v1/attachments/refresh-urls" bot />
Signs from 1 through 50 attachment URLs again. Returns one [refreshed attachment URL](#refreshed-attachment-url-object) object for each URL sent. A [data package URL](#data-package-attachment-urls) is accepted like any other, and its result is an ordinary URL valid for 24 hours.
### Limitations
- An OAuth2 bearer token is refused with 403 `ACCESS_DENIED`.
- Fluxer checks no channel membership, no permission, and no attachment existence, and reads no stored data. A URL is signed whether or not its object exists or the caller can see it.
### JSON body
| Field | Type | Description |
| --- | --- | --- |
| attachment_urls<sup>1</sup> | array[string] | URLs to sign again (1-50 entries, each at most 2,048 characters) |
<sup>1</sup> Each entry is read exactly as sent, with no trimming or normalisation
### Response body
| Field | Type | Description |
| --- | --- | --- |
| refreshed_urls<sup>1</sup> | array[[refreshed attachment URL](#refreshed-attachment-url-object) object] | Results in request order |
<sup>1</sup> Contains exactly one entry for each requested URL, in the order the entries were sent, and a duplicated URL therefore produces a duplicated entry
### Response
| Status | Body | Condition |
| --- | --- | --- |
| 200 | response body | Every URL was processed |
| 400 | [error response](/http-api/#error-response) | Body, URL count, or a URL length is invalid and the request returns `INVALID_FORM_BODY` |
| 401 | [error response](/http-api/#error-response) | The credential is missing or invalid and the request returns `UNAUTHORIZED` |
| 403 | [error response](/http-api/#error-response) | Caller presents a bearer credential and the request returns `ACCESS_DENIED` |
| 403 | [error response](/http-api/#error-response) | The account has an outstanding required action and the request returns `ACCOUNT_SUSPICIOUS_ACTIVITY` |
:::caution[A refreshed URL is not an access check]
Anyone who can authenticate can sign any attachment URL of the instance whose path they know. A signature limits how long a copied URL works. It does not decide who can read the file.
:::
### Rate limit
20 requests per 10 seconds for each authenticated user, on the `attachment:refresh_urls` bucket.
## Create message
<RouteHeader method="POST" path="/v1/channels/{channel_id}/messages" bot />
fluxer_docs/src/content/docs/http-api/users/data-harvest.mdx
+1 -1
View File
@@ -189,7 +189,7 @@ Fluxer prepares the archive in the background. It contains:
- `account/security.json`
- the account avatar and banner under `assets/user/` when present
Attachment metadata appears with its message and has the attachment ID, filename, size, content type, CDN URL, and pixel dimensions. The attachment files themselves are never included.
Attachment metadata appears with its message and has the attachment ID, filename, size, content type, CDN URL, and pixel dimensions. The CDN URL is a [data package URL](/http-api/messages/#data-package-attachment-urls), so it does not expire and still reads its attachment after the archive's own seven day deadline passes. Removing the secret that signed it is the only thing that ends it. The attachment files themselves are never included.
Every authored message is collected, with no ceiling on the count. Messages that no longer exist are omitted. Other read failures fail the attempt. After completion, Fluxer attempts to email a download URL if the account has an email address and the instance has email delivery enabled. That URL expires seven days after it was issued. Email failure does not prevent downloading a completed archive.
fluxer_docs/src/content/docs/media-proxy/overview.md
+94 -19
View File
@@ -17,7 +17,7 @@ The Media Proxy serves Fluxer attachments, image assets, themes, entrance sound
| Image asset<sup>1</sup> | `/avatars`, `/icons`, `/branding`, `/banners`, `/splashes`, `/embed-splashes`, `/emojis`, `/stickers` | [Image asset contract](/media-proxy/routes/#image-asset-contract) |
| Static object<sup>2</sup> | `/{key}` | [Get static object](/media-proxy/routes/#get-static-object) |
| Upload relay | `/v1/relay/{key}` | [Upload relay](/media-proxy/upload-relay/) |
| Operator and internal | `/_health`, `/_metrics`, `/_metadata`, `/_thumbnail`, `/_frames` | [Operator and internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) |
| Operator and internal | `/_health`, `/_metrics`, `/_metadata`, `/_sniff`, `/_thumbnail`, `/_frames` | [Operator and internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) |
<sup>1</sup> Guild member avatars and banners are image assets too, at `/guilds/{guild_id}/users/{user_id}/avatars` and `/guilds/{guild_id}/users/{user_id}/banners`
@@ -33,10 +33,10 @@ The upload relay is the exception in both respects. It is served below `/v1/rela
## Authorisation
No public Media Proxy route reads the HTTP API `Authorization` header. A stored object is authorised by its path, signed external media by its path signature, and an upload by the capability in its URL. Only the internal `POST` endpoints read `Authorization`, and they require the exact value `Bearer {secret}` built from the deployment secret.
No public Media Proxy route reads the HTTP API `Authorization` header. An attachment is authorised by its path and, under the [signed attachment URL policy](#signed-attachment-urls), by the signature in its query string. Any other stored object is authorised by its path, signed external media by its path signature, and an upload by the capability in its URL. Only the internal `POST` endpoints read `Authorization`, and they require the exact value `Bearer {secret}` built from the deployment secret.
:::caution[A media URL is a bearer capability]
An attachment path has no signature and no expiry, so anyone holding the URL reads the object for as long as it exists. Leaving the channel does not revoke a URL already issued.
Anyone holding a URL reads the object without an account. Under `enforce`, an ordinary signed attachment URL works until its `ex` time, which is at most one day after it was signed, and a [data package URL](/http-api/messages/#data-package-attachment-urls) works until the secret that signed it is removed. Under `off` or `report`, an attachment path works for as long as the object exists. Leaving the channel does not revoke a URL already issued.
:::
The Media Proxy has no request-count rate limit, so no response has the [rate limit headers](/topics/rate-limits/#rate-limit-headers) of the versioned HTTP API. A failure returns a short `text/plain` body, and [media error response](/media-proxy/responses-and-limits/#media-error-response) defines that shape.
@@ -50,10 +50,11 @@ One Media Proxy process serves exactly one mode. The mode is fixed at startup an
| `mp` | Attachments, signed external media, themes, entrance sounds, and every image asset route |
| `static` | Every read route as a raw object read from the static bucket, with no transformation and no SVG rasterisation<sup>1</sup> |
| `upload` | The [upload relay](/media-proxy/upload-relay/), plus every `mp` read route from the same buckets |
| `relay` | The [upload relay](/media-proxy/upload-relay/) alone. Every read path returns 404 |
<sup>1</sup> A `static` mode endpoint also strips `X-Robots-Tag` from every response and sends no `Content-Disposition`
The relay `PUT` is the only route with a mode gate, and it returns 404 outside `upload` mode. On a read that requests no transformation, only an `mp` endpoint rasterises SVG, so an `upload` endpoint returns the original SVG bytes. The [operator and internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) behave the same in every mode.
The relay `PUT` returns 404 outside `upload` and `relay` mode. A `relay` endpoint returns 404 for a `GET` or `HEAD` of any read path and reads no object for it. On a read that requests no transformation, only an `mp` endpoint rasterises SVG, so an `upload` endpoint returns the original SVG bytes. The [operator and internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) behave the same in every mode.
The operator chooses which mode serves each published base URL. The reference self-hosted deployment serves `endpoints.media` from an `upload` mode process.
@@ -67,32 +68,102 @@ On the [signed external route](/media-proxy/routes/#get-signed-external-media),
## Request headers
Only `Range` affects the representation a public read returns. `X-Forwarded-For` decides which address the [media access allowlist](#access-restrictions) evaluates, so it can turn a 200 into a 403 without changing the representation. Every other request header, including `Authorization`, `Cookie`, `Accept`, `Origin`, `If-Range`, `If-None-Match`, `If-Modified-Since`, `If-Match`, and `If-Unmodified-Since`, is ignored on a public read route.
Only `Range` affects the representation a public read returns. The [cross-origin read policy](#cross-origin-reads) reads `Origin`, and under `enforce` that header decides `Access-Control-Allow-Origin` and can turn a 200 into a 403. Every other request header, including `Authorization`, `Cookie`, `Accept`, `If-Range`, `If-None-Match`, `If-Modified-Since`, `If-Match`, and `If-Unmodified-Since`, is ignored on a public read route.
### Common request headers
| Field | Type | Description |
| --- | --- | --- |
| Range?<sup>1</sup> | string | One byte range under the [byte-range contract](#byte-ranges) |
| X-Forwarded-For?<sup>2</sup> | string | Client address chain read by the [media access allowlist](#access-restrictions) |
| Origin?<sup>2</sup> | string | Serialised request origin read by the [cross-origin read policy](#cross-origin-reads) |
<sup>1</sup> A value whose unit is not the exact lowercase `bytes`, or that names more than one range, is ignored, and the response has the complete representation
<sup>2</sup> Read only when the allowlist gate is enabled and the peer address is a configured trusted proxy. A value from any other peer is ignored
<sup>2</sup> Read only when the cross-origin read policy is `report` or `enforce`
## Access restrictions
## Cross-origin reads
An operator can restrict public reads to a CDN edge address allowlist. This restriction is disabled by default.
An operator can limit which web origins read media through CORS. `FLUXER_MEDIA_PROXY_CORS_MODE` sets this cross-origin read policy, and it defaults to `off`.
A request from an address outside the list returns 403. `/_health`, `/_metrics`, `/_metadata`, `/_thumbnail`, `/_frames`, and every path below `/v1/relay/` are exempt.
| Value | Description |
| --- | --- |
| `off` | A media representation has `Access-Control-Allow-Origin: *`, and no `Origin` is read |
| `report` | Every response is the same as under `off`, and each read `enforce` would refuse is logged |
| `enforce` | A read from an origin outside the allowlist returns 403 |
When the peer address is a configured trusted proxy, the client address is the rightmost `X-Forwarded-For` hop that is not itself a trusted proxy. Otherwise the Media Proxy uses the peer address and ignores `X-Forwarded-For`. It does the same when there is no `X-Forwarded-For`, or when every hop in it is unparseable or is itself a trusted proxy.
Under `enforce`, an `Origin` is allowed when it equals the serialised form of an entry in `FLUXER_MEDIA_PROXY_CORS_ALLOWED_ORIGINS` byte for byte. Anything else is refused, including `null`, a repeated `Origin` header, and a value that is not ASCII. A refused read returns 403 on every read path, and the Media Proxy reads no object and fetches no external URL for it. [Media Proxy settings](/operator/configuration/#media-proxy-settings) defines how an entry is parsed and serialised.
A read with no `Origin` is served without `Access-Control-Allow-Origin`. A browser sends no `Origin` for an `<img>` or a `<video>` without `crossorigin`, so those elements still load. An allowed read gets its own origin back in `Access-Control-Allow-Origin` on a media representation and on a 416. No error has CORS headers.
Under `enforce`, every response to a `GET` or `HEAD` on a read path has `Vary: Accept-Encoding, Origin`, the 403 for a refused `Origin` included.
`report` changes no response. The Media Proxy logs a warning with the reason `cors_origin_would_deny` for each read `enforce` would refuse.
The policy applies to a response the Media Proxy generates, and a shared cache in front of it answers a hit on its own. An entry stored while the policy was `off` holds `Access-Control-Allow-Origin: *` and a year of `Cache-Control`, and a foreign origin reading that URL gets both back until the entry goes. Purge `/attachments/` when you move the policy to `enforce`. [Caching in front of Fluxer](/operator/reverse-proxy/#caching-in-front-of-fluxer) states how.
A `static` or `relay` endpoint ignores the policy. The [upload relay](/media-proxy/upload-relay/) and the [operator and internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) are exempt from it. A method other than `GET` or `HEAD` on a read path returns 405 whatever its `Origin`.
The reference self-hosted deployment leaves the policy `off`. An operator who turns it on gets the instance public origin as the only entry, and [Media Proxy settings](/operator/configuration/#media-proxy-settings) states how to allow `https://web.fluxer.app` as well.
## Signed attachment URLs
An operator can require a signature on every attachment read. `FLUXER_MEDIA_PROXY_ATTACHMENT_SIGNATURE_MODE` sets this signed attachment URL policy, and it defaults to `off`. The reference self-hosted deployment leaves it `off`, and turning it on takes a secret and the mode. The Media Proxy reads the mode once at start, and [Turning the signature policy on](/operator/reverse-proxy/#turning-the-signature-policy-on) gives the order to move it and a cache in front of it, and what each pairing of the two serves.
| Value | Description |
| --- | --- |
| `off` | No signature is read, and an attachment read is served whatever its query string |
| `report` | Every attachment read is served, and each read `enforce` would refuse is logged |
| `enforce` | An attachment read without a valid signature returns 404 |
An instance with attachment URL secrets signs every attachment URL it returns, and [Refresh attachment URLs](/http-api/messages/#refresh-attachment-urls) signs one again. An ordinary signed URL has three query parameters, and Fluxer places them first in this order:
```text
?ex={expires}&is={issued}&hm={signature}
```
`ex` and `is` are each exactly 8 lowercase hexadecimal digits, the Unix time in seconds at which the URL expires and at which it was issued. `hm` is exactly 64 lowercase hexadecimal digits. Any other parameter follows them after a single `&`. A client MUST keep them exactly as issued and MAY add or change any other parameter.
A [data package URL](/http-api/messages/#data-package-attachment-urls) has a fourth parameter, `uc`, and its `ex` is the single character `0`:
```text
?ex=0&is={issued}&hm={signature}&uc=dp
```
`uc` is exactly `dp` and nothing else. `is` and `hm` have the same shape as on an ordinary URL. The 8-digit rule for `ex`, and the rule that `is` is no later than `ex`, hold for an ordinary URL alone. A data package URL never expires.
The Media Proxy reads the raw query string. It splits it on `&`, skips empty fields, and splits each field at its first `=`. It form-decodes each name and takes each value exactly as sent. A read is refused when any of these holds:
- There is no query string, or none of `ex`, `is`, `hm`, and `uc` is present.
- One or two of `ex`, `is`, and `hm` are present, so `uc` alone is refused too.
- Any of the four occurs more than once.
- `uc` is present with a value other than `dp`.
- `ex` is not 8 lowercase hexadecimal digits on an ordinary URL, or not the single character `0` on a data package URL.
- `is` is not 8 lowercase hexadecimal digits, or `hm` is not 64 lowercase hexadecimal digits.
- `is` is later than `ex` on an ordinary URL.
- `hm` does not verify against any configured secret.
- The current time is at or after `ex` on an ordinary URL.
A shared cache in front of the Media Proxy must leave the signature parameters `ex`, `is`, `hm`, and `uc` out of its cache key, and must decode each parameter name before it compares it. A cache that keeps them stores a new entry every time the same object is signed again. [Caching in front of Fluxer](/operator/reverse-proxy/#caching-in-front-of-fluxer) states the rest of the rules.
Every other parameter is ignored, and so is the order of the parameters. The signature is over `ex`, `is`, `uc`, and the percent-decoded storage key, and never over the host, the scheme, the port, or another parameter. Because `uc` is signed, an ordinary URL relabelled `uc=dp` does not verify, and neither does a data package URL with `uc` removed. One signed URL therefore reads every [transformation](/media-proxy/transformations/) and `download` variant of its object. Every percent-encoded spelling of the same key verifies the same way, and a key whose decoded bytes are not valid UTF-8 is refused.
Under `enforce`, a refused read returns 404 with the body `This content is no longer available.`, `Cache-Control: no-store`, and no `Access-Control-Allow-Origin`. The Media Proxy reads no object for it. The response is the same for every reason, for `GET` and `HEAD`, with or without a `Range`, and whatever transformation or `download` the request names. A read whose object does not exist returns that same 404 under `enforce`, even with a valid signature, so a response never tells a missing object apart from a refused signature. `off` and `report` return the ordinary `Not found` body for a missing object.
The Media Proxy checks the method first, then the `relay` mode, then the [cross-origin read policy](#cross-origin-reads), then the storage key, and only then the signature. A refused `Origin` returns 403 and an invalid storage key returns 400, and neither reads a signature.
`report` refuses nothing. Every status and body under `report` is the one `off` returns, and so is every header apart from the cache policy. A read whose signature is valid gets the same capped `Cache-Control` and `CDN-Cache-Control` that `enforce` sets, so the entries a cache stores during the dry run do not outlive the URLs that made them. A read `report` would refuse keeps the one-year policy `off` returns. The Media Proxy logs a warning with the reason `attachment_signature_would_deny` for each read `enforce` would refuse. The warning has the verdict `missing`, `malformed`, `mismatch`, or `expired`, the request identifier, and the user agent. Under every mode, the request log writes the values of `ex`, `is`, `hm`, and `uc` as `[redacted]`.
A `static` or `relay` endpoint ignores the policy. The policy covers `/attachments/` paths alone, so signed external media, image assets, themes, and entrance sounds need no signature. The [operator and internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) are exempt, and they read an attachment URL of this instance without a signature.
:::caution[A signature is not access control]
Refresh attachment URLs signs any well-formed attachment URL of the instance for any signed-in user or bot, with no membership check. A signature limits how long a copied URL works without a refresh. It does not decide who reads the object.
:::
## Selector parsing
A read route names its representation in its query string. The attachment, signed external, and image asset routes accept selectors. The theme and entrance sound routes accept none and ignore any query string.
Query names and values use URL form decoding, so `+` decodes to a space and a percent escape decodes to its byte. When a name occurs more than once, the final value wins. An unknown name is ignored.
Query names and values use URL form decoding, so `+` decodes to a space and a percent escape decodes to its byte. When a name occurs more than once, the final value wins. An unknown name is ignored. The [signed attachment URL policy](#signed-attachment-urls) reads `ex`, `is`, `hm`, and `uc` under its own rules.
The Media Proxy does not redirect alternative spellings to a canonical URL.
@@ -126,18 +197,18 @@ Content disposition follows the media type the origin declared, so SVG mislabell
| Field | Type | Description |
| --- | --- | --- |
| Content-Type<sup>1</sup> | string | The detected or selected media type |
| Content-Type<sup>1</sup> | string | The detected or selected media type, or `text/plain; charset=utf-8` for a JavaScript type outside `static` mode |
| Content-Length?<sup>2</sup> | integer | The selected body length, including on HEAD |
| Content-Range? | string | Present on 206 as `bytes {start}-{end}/{size}` and on 416 as `bytes */{size}` |
| Content-Disposition?<sup>3</sup> | string | The route-selected inline or attachment disposition |
| Accept-Ranges | string | The literal `bytes` on every media representation |
| Access-Control-Allow-Origin | string | The literal `*` on media responses |
| Access-Control-Allow-Origin? | string | The literal `*` on media responses. Under `enforce` it is the request origin, or absent with no `Origin` |
| Cache-Control<sup>4</sup> | string | The browser cache policy for the representation |
| CDN-Cache-Control | string | The matching shared cache policy |
| Vary | string | The literal `Accept-Encoding` |
| Vary | string | The literal `Accept-Encoding`, or `Accept-Encoding, Origin` under `enforce` |
| X-Content-Type-Options | string | The literal `nosniff` |
| X-Robots-Tag<sup>5</sup> | string | The indexing policy |
| X-Fluxer-Version<sup>6</sup> | string | The serving Fluxer build identifier, or `dev` when none is configured |
| X-Fluxer-Version | string | The serving Fluxer build identifier, or `dev` when none is configured |
<sup>1</sup> Fixed at `text/css; charset=utf-8` on the theme route. A 416 response has none
@@ -149,8 +220,6 @@ Content disposition follows the media type the origin declared, so SVG mislabell
<sup>5</sup> The literal `noindex, nofollow, nosnippet, noimageindex, notranslate, max-snippet:0, max-image-preview:none, max-video-preview:0`. Present in `mp` and `upload` mode, and a `static` mode endpoint removes it
<sup>6</sup> Absent from a media access allowlist rejection
Every successful media representation uses `Cache-Control: public, max-age=31536000` and `CDN-Cache-Control: public, max-age=31536000`. An audio or video representation adds `no-transform` to `Cache-Control` only. No response repeats its policy in an `Expires` header. [Cache policies](/media-proxy/responses-and-limits/#cache-policies) lists every cache policy this surface sets, including the responses that set none.
The upload relay is the only route that returns an `ETag`, and it relays the object storage value for the stored object. No read route sends an `ETag` or a `Last-Modified`, so a cache revalidates a representation by fetching it again.
@@ -159,12 +228,18 @@ The upload relay is the only route that returns an `ETag`, and it relays the obj
The signed path is derived from the target URL, so the bytes behind one unchanged target are cached for a year by browsers under `Cache-Control` and by shared caches under `CDN-Cache-Control`.
:::
A 416 response has `Accept-Ranges`, `Content-Range`, `Access-Control-Allow-Origin`, `Vary`, and `X-Robots-Tag`, with no `Content-Type` and no cache policy. Its body is empty.
A 416 response has `Accept-Ranges`, `Content-Range`, `Access-Control-Allow-Origin`, `Vary`, and `X-Robots-Tag`, with no `Content-Type` and no cache policy. Its body is empty. The [cross-origin read policy](#cross-origin-reads) sets its `Access-Control-Allow-Origin` and `Vary` the same way it does on a 200.
## Content detection
Use the response's `Content-Type`. It can differ from the filename extension or the origin's declared type.
An endpoint in any mode other than `static` replaces a `Content-Type` that names a JavaScript media type with `text/plain; charset=utf-8`, at any status. It splits the value at every comma and reads each part on its own. In each part it skips leading spaces and tabs, then reads the type up to the first space, tab, semicolon, or opening parenthesis. The header is replaced when that type in any part equals one of these, ignoring ASCII case: `application/ecmascript`, `application/javascript`, `application/x-ecmascript`, `application/x-javascript`, `text/ecmascript`, `text/javascript`, `text/javascript1.0` through `text/javascript1.5`, `text/jscript`, `text/livescript`, `text/x-ecmascript`, and `text/x-javascript`. So `text/javascript x`, `TEXT/JAVASCRIPT (x)`, and `video/mp4, text/javascript` are replaced, and `text/javascript1.6` and `text/javascriptx` are left as they are.
The endpoint also replaces a `Content-Type` that is not visible ASCII, at any status. A 200 or 206 with no `Content-Type`, or with one that is empty or only spaces and tabs, gets `text/plain; charset=utf-8` too. A response with any other status and no `Content-Type` keeps none, so a 416 and the empty-body method rejection of a registered path still have no `Content-Type`. `Content-Disposition` is unchanged, so it still names the original filename.
A `static` endpoint is the one endpoint that changes no `Content-Type`. An `upload` or `relay` endpoint applies the same replacement an `mp` endpoint does, which is what makes the reference self-hosted deployment safe, because it serves `endpoints.media` from an `upload` endpoint.
## Content disposition
Disposition follows the resolved media type. An image other than SVG and a video are served inline. Every other type, including SVG and PDF, is served as an attachment. An explicit `download` request forces attachment disposition on every route that accepts the parameter.
fluxer_docs/src/content/docs/media-proxy/responses-and-limits.md
+18 -12
View File
@@ -14,9 +14,9 @@ An unsuccessful response body is an English reason phrase under the content type
| --- | --- | --- |
| 400 | Bad request | An invalid storage key, dimension, format, or external path, an unparsable relay `partNumber`, a failed attachment or external transformation, or a relay upload body the endpoint failed to read from the client<sup>1</sup> |
| 401 | Unauthorized | An invalid signed external path signature, a missing, malformed, or expired relay capability, or a missing or invalid internal bearer token |
| 403 | Media access denied | The media access allowlist rejected the client address |
| 403 | Forbidden | A relay capability presented for another bucket, key, method, `uploadId`, or `partNumber`, or `/_metrics` requested from a non-loopback address |
| 404 | Not found | A `GET` or `HEAD` of an unrouted path, an object that does not exist, or a relay `PUT` outside `upload` mode |
| 403 | Forbidden | A relay capability presented for another bucket, key, method, `uploadId`, or `partNumber`, `/_metrics` requested from a non-loopback address, or a public read whose `Origin` the [cross-origin read policy](/media-proxy/overview/#cross-origin-reads) refuses |
| 404 | Not found | A `GET` or `HEAD` of an unrouted path or of any read path on a `relay` endpoint, an object that does not exist, or a relay `PUT` outside `upload` and `relay` mode |
| 404 | This content is no longer available.<sup>4</sup> | An attachment read `enforce` refuses, or a missing attachment object under `enforce` |
| 405 | Method not allowed<sup>2</sup> | A method other than `GET` or `HEAD` on a read route or on an unrouted path |
| 413 | Payload too large | A stored object, external body, upload body, or internal request body beyond its bound |
| 500 | Transcode failed | An image asset transcode failed and the source is not directly displayable |
@@ -27,13 +27,15 @@ An unsuccessful response body is an English reason phrase under the content type
<sup>1</sup> The relay also answers 400 when the client connection fails part way through the body
<sup>2</sup> The registered paths, `/_health`, `/_metrics`, `/_metadata`, `/_thumbnail`, `/_frames`, and `/v1/relay/{key}`, answer an unaccepted method with an empty body, no `Content-Type`, and an `Allow` header
<sup>2</sup> The registered paths, `/_health`, `/_metrics`, `/_metadata`, `/_sniff`, `/_thumbnail`, `/_frames`, and `/v1/relay/{key}`, answer an unaccepted method with an empty body, no `Content-Type`, and an `Allow` header. All seven are registered in every mode, so the method rejection is the same in every mode, even where the path itself answers 404
<sup>3</sup> `/_metadata` is the only endpoint that remaps an origin 429. The signed external read route retains 429 as 429
<sup>4</sup> Sent under the `enforce` [signed attachment URL policy](/media-proxy/overview/#signed-attachment-urls) alone, where it also replaces `Not found` for an attachment object that does not exist. Under `report` every response is the one `off` returns, so a missing object keeps the `Not found` body there
When the Media Proxy passes an external origin status through to the client, the body is `Upstream fetch failed` on the signed external read route. On `/_metadata`, the body is the canonical reason phrase of that status. An object store error that maps to no case above uses the canonical reason phrase of its status.
Every error a route produces uses `Cache-Control: no-store` and the standard [security headers](/media-proxy/overview/#representation-headers). A 416 response, a media access allowlist rejection, and the empty-body method rejection of a registered path have the security headers and set no cache policy, and [Cache policies](#cache-policies) names them. No plain-text error has CORS headers unless it came from the [upload relay](/media-proxy/upload-relay/), and no error has `Retry-After` or a request identifier.
Every error a route produces uses `Cache-Control: no-store` and the standard [security headers](/media-proxy/overview/#representation-headers). The exceptions have the security headers and set no cache policy, and [Cache policies](#cache-policies) names them. No plain-text error has CORS headers unless it came from the [upload relay](/media-proxy/upload-relay/), and no error has `Retry-After` or a request identifier. Under the `enforce` [cross-origin read policy](/media-proxy/overview/#cross-origin-reads), every `GET` or `HEAD` error on a read path has `Vary: Accept-Encoding, Origin`. That includes the 403 for a refused `Origin`.
### Handling contract
@@ -49,8 +51,8 @@ Every `HEAD` response has an empty body, so the Body column describes `GET`, `PU
| 206 | Selected media bytes for `GET` and empty for `HEAD` | One byte range is satisfiable |
| 400 | [Media error response](#media-error-response) | A path, query, target, transformation, or relay request is invalid |
| 401 | [Media error response](#media-error-response) | An external path signature, a relay capability, or an internal token is invalid |
| 403 | [Media error response](#media-error-response) | The media access policy denied the request, a relay capability does not match the request, or an external origin returned 403 |
| 404 | [Media error response](#media-error-response) | The route or object does not exist, the relay is not served here, or an external origin returned 404 |
| 403 | [Media error response](#media-error-response) | The cross-origin read policy denied the request, a relay capability does not match the request, or an external origin returned 403 |
| 404 | [Media error response](#media-error-response) | The route or object does not exist, a `relay` endpoint received a read, the relay is not served here, an attachment signature was refused, or an external origin returned 404 |
| 405<sup>2</sup> | [Media error response](#media-error-response) | The route rejects the method, or an external origin returned 405 |
| 406, 408, 409, 410, 411, 412, 414, 415, 428, 429<sup>3</sup> | [Media error response](#media-error-response) | An external origin returned that status |
| 413 | [Media error response](#media-error-response) | A source, external response, or upload exceeds its applicable bound |
@@ -60,7 +62,7 @@ Every `HEAD` response has an empty body, so the Body column describes `GET`, `PU
| 503 | [Media error response](#media-error-response) | Upload spool capacity is exhausted, an external buffer reservation or allocation failed, or an external origin answered `/_metadata` with 429<sup>5</sup> |
| 504 | [Media error response](#media-error-response) | Transformation capacity was unavailable or a transformation deadline expired |
<sup>1</sup> The [internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) `/_metadata` and `/_frames` answer 200 with JSON, `/_health` with plain text, and `/_metrics` with the Prometheus text exposition
<sup>1</sup> The [internal endpoints](/media-proxy/routes/#operator-and-internal-endpoints) `/_metadata`, `/_sniff`, and `/_frames` answer 200 with JSON, `/_health` with plain text, and `/_metrics` with the Prometheus text exposition
<sup>2</sup> A read route path, or any other path outside the registered paths, answers with the [media error response](#media-error-response). Each registered path answers with an empty body, no `Content-Type`, and an `Allow` header
@@ -88,7 +90,7 @@ Decoded images are limited to 16,384 pixels on either edge and 268,435,456 pixel
The upload relay limits a body to the smaller of the authorised upload size and the endpoint body limit. That endpoint limit defaults to 500 MiB and can be configured from 1 byte through 5 GiB.
An internal `/_metadata`, `/_thumbnail`, or `/_frames` request body is limited to the base64 expansion of the 500 MiB media bound plus 1 MiB. All answer a larger body with 413.
An internal `/_metadata`, `/_sniff`, `/_thumbnail`, or `/_frames` request body is limited to the base64 expansion of the 500 MiB media bound plus 1 MiB. All answer a larger body with 413.
## Work admission
@@ -118,13 +120,17 @@ A status and its headers are chosen before the body is sent. A streamed object s
Every successful media representation uses `Cache-Control: public, max-age=31536000` and `CDN-Cache-Control: public, max-age=31536000`. An audio or video representation appends `no-transform` to `Cache-Control` only. Signed external media uses the same policy as stored media.
No route sets `immutable`, `Expires`, `ETag`, or `Last-Modified` on a read response, and no route selects a shorter cache policy. A cache revalidates a representation by fetching it again.
No route sets `immutable`, `Expires`, `ETag`, or `Last-Modified` on a read response. A cache revalidates a representation by fetching it again.
Every route-produced error response uses `Cache-Control: no-store`, and a successful `/_metrics` read uses it too. A 416 response, a successful upload relay response, a media access allowlist rejection, and the empty-body method rejection of a registered path set no cache policy at all.
One read selects a shorter policy. Under the `report` or `enforce` [signed attachment URL policy](/media-proxy/overview/#signed-attachment-urls), an attachment read whose signature is valid gets `public, max-age=` the seconds left on that signature in both headers, so no entry outlives the URL that made it. A data package URL never expires, so it gets the one-day ceiling instead. Every other read, and every attachment read under `off`, keeps the year.
A one-year `max-age` is only safe where a purge path reaches every cache that stores media. With no `ETag` and no `Last-Modified` there is nothing to revalidate against, so a deleted or replaced object is served until its entry is purged or drops out. A cache with no purge path must set a shorter lifetime of its own instead. [Caching in front of Fluxer](/operator/reverse-proxy/#caching-in-front-of-fluxer) states how.
Every route-produced error response uses `Cache-Control: no-store`, and a successful `/_metrics` read uses it too. A 416 response, a successful upload relay response, and the empty-body method rejection of a registered path set no cache policy at all.
## Range response headers
A complete media response has `Accept-Ranges: bytes`, the representation `Content-Type`, and an exact `Content-Length`<sup>1</sup>. A 206 also sets `Content-Range` to the selected interval over the complete size and `Content-Length` to the selected byte count. A 416 has `Content-Range: bytes */{size}` and `Accept-Ranges: bytes` with an empty body.
A complete media response has `Accept-Ranges: bytes`, the representation `Content-Type`, and an exact `Content-Length`<sup>1</sup>. Outside `static` mode that `Content-Type` is `text/plain; charset=utf-8` when it names a JavaScript media type, and a 200 or 206 with no `Content-Type` gets the same value. [Content detection](/media-proxy/overview/#content-detection) defines both rules. A 206 also sets `Content-Range` to the selected interval over the complete size and `Content-Length` to the selected byte count. A 416 has `Content-Range: bytes */{size}` and `Accept-Ranges: bytes` with an empty body.
<sup>1</sup> A streamed signed external response omits `Content-Length` when the origin declared none
fluxer_docs/src/content/docs/media-proxy/routes.mdx
+18 -8
View File
@@ -20,8 +20,8 @@ Every media read route can produce these statuses in addition to the ones in its
| Status | Body | Condition |
| --- | --- | --- |
| 403 | Media access denied | The request address is not permitted by the instance media access policy |
| 404 | Not found | The path is not a media route or the object does not exist |
| 403 | Forbidden | The [cross-origin read policy](/media-proxy/overview/#cross-origin-reads) refuses the request `Origin` on an `mp` or `upload` endpoint |
| 404 | Not found | The path is not a media route, the endpoint runs in `relay` mode, or the object does not exist |
| 405 | Method not allowed | The method is neither `GET` nor `HEAD` |
| 413 | Payload too large | The stored object exceeds the 500 MiB media bound |
| 502 | Bad gateway | The object store could not be read |
@@ -30,10 +30,10 @@ Every media read route can produce these statuses in addition to the ones in its
<RouteHeader method="GET" path="/attachments/{path}" unauthenticated />
Returns the attachment identified by the path, optionally transformed. The path comes from the attachment URL the HTTP API returned.
Returns the attachment identified by the path, optionally transformed. The path and its signature come from the attachment URL the HTTP API returned. Request that URL as issued. A transformation parameter can be added to its query string.
:::caution[An attachment URL is a bearer capability]
The path has no signature and no expiry, so anyone holding the URL reads the object from any origin for as long as it exists. Leaving the guild does not revoke a URL a member already holds.
Anyone holding the URL reads the object from any origin the [cross-origin read policy](/media-proxy/overview/#cross-origin-reads) allows. Under the `enforce` [signed attachment URL policy](/media-proxy/overview/#signed-attachment-urls) it works until `ex`, and otherwise for as long as the object exists. A [data package URL](/http-api/messages/#data-package-attachment-urls) has no `ex` and works until the secret that signed it is removed. Leaving the guild does not revoke a URL a member already holds.
:::
### Path parameters
@@ -55,9 +55,15 @@ The path has no signature and no expiry, so anyone holding the URL reads the obj
| animated? | boolean | Whether animated output is requested |
| effort?<sup>1</sup> | integer | The WebP encoder effort, clamped to 9 |
| download? | boolean | Whether attachment disposition is requested |
| ex?<sup>2</sup> | string | The expiry of the signed URL, as 8 lowercase hexadecimal digits of Unix seconds, or `0` on a data package URL |
| is?<sup>2</sup> | string | The issue time of the signed URL, as 8 lowercase hexadecimal digits of Unix seconds |
| hm?<sup>2</sup> | string | The signature, as 64 lowercase hexadecimal digits |
| uc?<sup>2</sup> | string | The data package marker, always `dp` when present |
<sup>1</sup> Read only on this route, and an empty or unparsable value is ignored. Only lossless animated WebP output uses a value above 6
<sup>2</sup> Read under the `report` or `enforce` [signed attachment URL policy](/media-proxy/overview/#signed-attachment-urls), which also defines how they are parsed. `enforce` requires `ex`, `is`, and `hm`, and reads `uc` on every URL, where its presence selects the [data package URL](/http-api/messages/#data-package-attachment-urls) grammar
[When a transformation runs](/media-proxy/transformations/#when-a-transformation-runs) defines the trigger.
When a transformation is requested, `Range` selects bytes from the transformed result.
@@ -73,7 +79,8 @@ A video source requires an explicit image `format`. A video request with `width`
| 200 | Complete media representation | The object and selected representation are available |
| 206 | Selected media bytes | One range is satisfiable |
| 400 | Bad request | The decoded key, a dimension, the format, or the transformation request is invalid, or the transformation failed |
| 404 | Not found | The attachment object does not exist |
| 404 | Not found | The attachment object does not exist and the signed attachment URL policy is `off` or `report` |
| 404 | This content is no longer available. | The signed attachment URL policy refuses the signature, or the object does not exist under `enforce` |
| 416 | empty | The range is unsatisfiable |
| 504 | Gateway timeout | Transformation capacity was unavailable or the transformation exceeded its deadline |
@@ -93,7 +100,7 @@ Fluxer constructs the signed path and exposes it on these fields:
- `proxy_url` of a [resolved GIF entry](/http-api/memes/#resolved-gif-entry-object)
- `proxy_src` of a [GIF media format object](/http-api/memes/#gif-media-format-object)
A field holds a signed path only when its source URL is external. A URL already under the Media Proxy endpoint, and any URL the deployment cannot sign, is returned verbatim, so a client MUST request the value exactly as issued and MUST NOT assume it has the signed shape.
A field holds a signed external path only when its source URL is external. A URL already under the Media Proxy endpoint keeps that endpoint, and gets an [attachment URL signature](/http-api/messages/#signed-attachment-urls) when it names an attachment of this instance. Any URL the deployment cannot sign is returned verbatim, so a client MUST request the value exactly as issued and MUST NOT assume it has the signed shape.
### Path parameters
@@ -488,14 +495,17 @@ The complete request path is the storage key after percent-decoding. The route a
## Operator and internal endpoints
These paths are not part of the public API surface. They are exempt from the media access allowlist and are not published by instance discovery. A client MUST NOT depend on them.
These paths are not part of the public API surface and are not published by instance discovery. A client MUST NOT depend on them.
| Path | Method | Description |
| --- | --- | --- |
| `/_health` | GET | Returns 200 while the process is running |
| `/_metrics` | GET | Returns the Prometheus text exposition of the process. A request from a non-loopback address returns 403 |
| `/_metadata` | POST | Extracts media metadata, a placeholder, and an optional NSFW verdict for the HTTP API |
| `/_sniff` | POST | Returns the media type detected in the first 8 KiB of a staged upload for the HTTP API |
| `/_thumbnail` | POST | Produces a WebP thumbnail of a staged upload for the HTTP API |
| `/_frames` | POST | Extracts one JPEG video frame for the HTTP API |
`/_metadata`, `/_thumbnail`, and `/_frames` require `Authorization: Bearer {deployment secret key}` and return 401 without it. Their JSON request bodies are bounded, and a body beyond the bound returns 413.
`/_metadata`, `/_sniff`, `/_thumbnail`, and `/_frames` require `Authorization: Bearer {deployment secret key}` and return 401 without it. Their JSON request bodies are bounded, and a body beyond the bound returns 413.
`/_sniff` returns 200 with a JSON `content_type`. It is the image, video, or audio media type the Media Proxy detects in the upload, or null when the bytes are SVG or a type it cannot process. An upload key that names no object returns 404.
fluxer_docs/src/content/docs/media-proxy/upload-relay.mdx
+1 -1
View File
@@ -12,7 +12,7 @@ The upload relay is where a client sends the bytes of a file. Each upload URL ha
Paths here are relative to the relay base in the issued upload URL. That base is a deployment setting.
The capability is the complete authorisation. The relay reads no HTTP API `Authorization` header and has no request-count rate limit. Every path below `/v1/relay/` is exempt from the [media access policy](/media-proxy/overview/#access-restrictions). The `PUT` has a [deployment mode](/media-proxy/overview/#deployment-modes) gate and returns 404 outside `upload` mode.
The capability is the complete authorisation. The relay reads no HTTP API `Authorization` header and has no request-count rate limit. The `PUT` has a [deployment mode](/media-proxy/overview/#deployment-modes) gate and returns 404 outside `upload` and `relay` mode.
:::note[Some upload URLs address object storage directly]
An upload URL points directly to object storage when the caller's IP address geolocates to a country the deployment lists in `keep_direct_countries`. Use the URL returned by the HTTP API. This page describes relay URLs only.
fluxer_docs/src/content/docs/operator/configuration.mdx
+33 -3
View File
@@ -58,7 +58,7 @@ Default `http`. The scheme for internal service URLs. Must be `http` or `https`,
## Secrets you must generate
Every value below ships as `CHANGE_ME`. Replace them all before the first start. Compose refuses to start when one is unset or empty, and names the variable. Generate each with `openssl rand -hex 32`, except the relay secret and the VAPID pair, whose own entries name the command. The installer in [Get started](/operator/get-started/) generates them all, so come here to rotate one value later or to set them up by hand.
Every value below ships as `CHANGE_ME`. Replace them all before the first start. Compose refuses to start when one is unset or empty, and names the variable. Generate each with `openssl rand -hex 32`, except the relay secret, the attachment URL secrets, and the VAPID pair, whose own entries name the command. The installer in [Get started](/operator/get-started/) generates them all, so come here to rotate one value later or to set them up by hand.
#### `POSTGRES_PASSWORD`
@@ -90,7 +90,21 @@ Media Proxy URLs. Must match across `api`, `worker`, `media-proxy`, `gifs`, and
#### `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_SECRET_BASE64`
Generate with `openssl rand -base64 32`. [Upload relay](/media-proxy/upload-relay/) capability tokens. Must be standard base64 decoding to at least 32 bytes. Hex fails. Must match on `api`, `worker`, and `media-proxy`. The API and the Media Proxy both check it at boot and refuse to start without it in `upload` mode.
Generate with `openssl rand -base64 32`. [Upload relay](/media-proxy/upload-relay/) capability tokens. Must be standard base64 decoding to at least 32 bytes. Must match on `api`, `worker`, and `media-proxy`. `api` and `worker` refuse to start without it in every mode. `media-proxy` refuses to start without it when `FLUXER_MEDIA_PROXY_MODE` is `upload` or `relay`.
#### `FLUXER_MEDIA_PROXY_ATTACHMENT_URL_SECRETS_BASE64`
Generate each entry with `openssl rand -base64 32`. Signs the [attachment URLs](/media-proxy/overview/#signed-attachment-urls) Fluxer returns. A comma-separated list, where each entry must be standard base64 decoding to at least 32 bytes. Spaces around an entry and empty entries are ignored. The first entry signs and every entry verifies. Must match on `api`, `worker`, and `media-proxy`, and must differ from `FLUXER_MEDIA_PROXY_SECRET_KEY`. `api` and `worker` refuse to start when an entry is invalid, and with no entry they return unsigned URLs. `media-proxy` reads it only when `FLUXER_MEDIA_PROXY_MODE` is `mp` or `upload`, and then refuses to start with an invalid entry, or with no entry when `FLUXER_MEDIA_PROXY_ATTACHMENT_SIGNATURE_MODE` is `report` or `enforce`.
The Compose stack leaves it empty, so a self-hosted instance returns unsigned URLs until an operator sets one. Setting it alone changes nothing on the read side, because `FLUXER_MEDIA_PROXY_ATTACHMENT_SIGNATURE_MODE` stays `off` until it is set too.
Rotate it in three steps, and apply each one with `docker compose up -d` before starting the next:
1. Add the new secret as the second entry. Every `media-proxy` then accepts it before any service signs with it.
2. Move the new secret to the first entry, so it signs.
3. Remove the old secret no sooner than one day after step 2, because an ordinary URL it signed stays valid for up to a day.
A [data package URL](/http-api/messages/#data-package-attachment-urls) inside a harvest export never expires, so removing a secret ends every data package URL it signed at once, including the links inside exports already delivered, and those exports have to be rebuilt. Step 3 is safe for ordinary URLs alone.
#### `FLUXER_ADMIN_SECRET_KEY_BASE`
@@ -1309,7 +1323,7 @@ The Media Proxy takes uploads, transforms images, and serves media back. All are
#### `FLUXER_MEDIA_PROXY_MODE`
Default `mp`. Which routes are served. `mp`, `static`, or `upload`. Anything else is a startup error. Compose sets `upload`.
Default `mp`. Which routes are served. `mp`, `static`, `upload`, or `relay`. `relay` serves the upload relay and the operator and internal endpoints, and returns 404 for every read. Anything else is a startup error. Also settable with `--mode`. Compose sets `upload`.
#### `FLUXER_MEDIA_PROXY_HOST`
@@ -1411,6 +1425,22 @@ Default empty. An external NSFW classifier. No such service ships with the stack
Default `0.85`. The media-side NSFW cutoff. Accepts 0.0 to 1.0 and finite. Distinct from the API threshold.
#### `FLUXER_MEDIA_PROXY_CORS_MODE`
Default `off`. The [cross-origin read policy](/media-proxy/overview/#cross-origin-reads). `off`, `report`, or `enforce`. Anything else is a startup error. `report` logs each read whose origin would be refused and changes no response. `enforce` refuses those reads with 403 and sets `Access-Control-Allow-Origin` to the request origin. When `FLUXER_MEDIA_PROXY_MODE` is `static` or `relay`, the Media Proxy ignores it, so a bad value is no startup error. Compose leaves it `off`.
#### `FLUXER_MEDIA_PROXY_CORS_ALLOWED_ORIGINS`
Default empty. Comma-separated origins that may read media through CORS, such as `https://web.fluxer.app`. Spaces around an entry and empty entries are ignored. Each entry is parsed as a URL and must be `http` or `https` with no credentials, query, fragment, or path beyond `/`. Its host must be an IPv4 address, a bracketed IPv6 address, or a domain name whose labels are ASCII letters, digits, and hyphens after parsing. So a wildcard such as `https://*.fluxer.app`, a brace pattern, an underscore, a trailing dot on a domain name, or any non-ASCII character fails. Write an internationalised domain name in its punycode form. Any failing entry makes the process exit. An entry is compared in the form a browser sends. `https://Web.Fluxer.App:443/` matches `https://web.fluxer.app`. Required when `FLUXER_MEDIA_PROXY_CORS_MODE` is `report` or `enforce`. When the policy is `report` or `enforce`, the Media Proxy logs the parsed list once at startup. When `FLUXER_MEDIA_PROXY_MODE` is `static` or `relay`, the Media Proxy ignores it.
Compose sets it to the public origin, built from `FLUXER_PUBLIC_ORIGIN`, or from `FLUXER_PUBLIC_SCHEME`, `FLUXER_DOMAIN`, and `FLUXER_PUBLIC_PORT`. That origin is the only entry, so the web app the instance serves reads its own media. The hosted web client and the desktop app run on `https://web.fluxer.app`. To let them read this instance's media through CORS, set the list in `.env` to the public origin and `https://web.fluxer.app`, such as `https://chat.example.com,https://web.fluxer.app`. A value set in `.env` replaces the Compose value, so it must name the public origin too.
#### `FLUXER_MEDIA_PROXY_ATTACHMENT_SIGNATURE_MODE`
Default `off`. The [signed attachment URL policy](/media-proxy/overview/#signed-attachment-urls). `off`, `report`, or `enforce`. Anything else is a startup error, an empty value included. `report` logs each attachment read `enforce` would refuse and serves it. `enforce` refuses those reads with 404. Under both, a read whose signature is valid gets a `Cache-Control` that ends with the signature instead of the usual year. `report` and `enforce` require `FLUXER_MEDIA_PROXY_ATTACHMENT_URL_SECRETS_BASE64`, and the Media Proxy logs the mode and the number of secrets once at startup. When `FLUXER_MEDIA_PROXY_MODE` is `static` or `relay`, the Media Proxy ignores it and the secrets, so a bad value is no startup error. Compose leaves it `off`, so a self-hosted instance signs nothing until an operator turns it on.
This variable is the whole switch, and moving between the three states needs nothing else changed. The Media Proxy reads it once at start, so apply a change with `docker compose up -d media-proxy`, which replaces the container. `docker compose restart media-proxy` reuses the old environment. A cache or proxy in front of the instance has a switch of its own, and [Turning the signature policy on](/operator/reverse-proxy/#turning-the-signature-policy-on) gives the order to move the two in and what each pairing serves.
`media-proxy` range-checks these values at startup and then reads them nowhere: `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_BUFFERED_RETRY_BYTES` and `FLUXER_MEDIA_PROXY_UPLOAD_RELAY_BUFFERED_RETRY_TOTAL_BYTES`. A bad value still fails the boot.
## Gateway settings
fluxer_docs/src/content/docs/operator/reverse-proxy.mdx
+76
View File
@@ -436,6 +436,82 @@ The edge reads the variable at container start, so apply a change with `docker c
`private_ranges` trusts the whole LAN. With `FLUXER_EDGE_BIND=0.0.0.0:8080`, any host on that network can impersonate any client address. Firewall the port to the proxy host.
:::
## Caching in front of Fluxer
Media reads are the only traffic worth caching, and the instance expects no cache in front of it. Every rule below is a correctness rule rather than a tuning choice, and a cache that cannot follow one should not store media at all.
#### Leave the signature parameters out of the cache key
`ex`, `is`, `hm`, and `uc` are the [signed attachment URL](/media-proxy/overview/#signed-attachment-urls) parameters. A key that keeps them stores a new entry every time the same object is signed again, which empties the cache of anything useful. Drop all four and keep every other parameter, because the rest select the representation.
Only drop them if your proxy also meets the next two rules. A key without them is the same key for a signed request and an unsigned one, so the signature check is the only thing between an unsigned request and the object.
#### Decode parameter names, and take the last value
Fluxer parses a query string with form URL decoding, so `%73ize` is the name `size` and `+` is a space. When a name repeats, the final value wins. A cache that compares raw bytes, or that keeps the first value, can be handed `%73ize=64` or `size=64&size=4096` and made to store one rendering under the key for another. Match both rules, or refuse to cache a request whose query string you cannot parse that way.
#### Check the signature before the cache lookup
A cache hit never reaches the instance, so the Media Proxy's own signature check never runs on one. Under the `enforce` [signed attachment URL policy](/media-proxy/overview/#signed-attachment-urls), verify the signature in the proxy before it looks in the cache. An nginx `access_by_lua_block` runs in the access phase, which is before the cache lookup. A proxy that cannot do that must leave `/attachments/` uncached, or it will serve an expired or forged URL from a warm entry.
#### Verify the same path the cache key is built from
Running before the cache lookup is not enough on its own. The check and the key must read the same spelling of the path, or a request the check skips still reaches the entry the key names.
In nginx that spelling is `$uri`, which nginx has already percent decoded, merged repeated slashes in, and resolved `.` and `..` out of. `$request_uri` is the raw bytes the client sent. A check that scopes itself on `$request_uri` never sees `//attachments/...`, `/attachments%2F...`, `/attachments/./...`, or `/x/../attachments/...` as an attachment read, while `$uri` and the cache key make all four the same object. Read `$uri` for the prefix test and for the storage key, take the query from `$request_uri`, and refuse a path whose `$uri` is not a usable storage key rather than passing it through.
The signature covers the decoded storage key, which is `$uri` without its leading slash.
#### Do not cache a range request without slice support
nginx removes `Range` from a request it may cache unless the slice module is built in, so a seek into a video pulls the whole object. Set `proxy_cache_max_range_offset 0`. A range request then reaches the instance untouched and is not stored, while an entry that is already warm still answers ranges from cache.
#### Wire up purge, and bound the entry lifetime anyway
Every media 200 is `public, max-age=31536000` with no `ETag` and no `Last-Modified`, so a cache never revalidates and a deleted object stays until its entry goes. Point `FLUXER_CACHE_PURGE_HTTP_ENDPOINT` at a receiver that clears your cache, and read the contract it must meet under [Cache purge](/operator/configuration/#feature-flags-and-development-switches). Give stored media a shorter lifetime of your own as well, because a purge that is lost is never sent again.
## Turning the signature policy on
`FLUXER_MEDIA_PROXY_ATTACHMENT_SIGNATURE_MODE` is the instance switch, and a proxy that verifies signatures has one of its own. Both sides name the same three states, `off`, `report`, and `enforce`. Both read their setting once at start, so apply an instance change with `docker compose up -d media-proxy`, which replaces the container. `docker compose restart media-proxy` reuses the old environment and changes nothing.
Turn the policy on in this order, and stop wherever a refusal appears that you did not expect:
1. Give `api`, `worker`, and `media-proxy` the same `FLUXER_MEDIA_PROXY_ATTACHMENT_URL_SECRETS_BASE64`, and give the proxy the same list. Every URL the instance returns is signed from here on, and nothing verifies yet.
2. Leave it there until the clients holding the unsigned URLs issued before step 1 have replaced them. A client refreshes a URL whose `ex` is absent or under an hour away, and one that has been open since before step 1 keeps its unsigned URLs until it reloads.
3. Set both sides to `report` and read the logs. Under `report` nothing is refused, and each read `enforce` would refuse is logged with its reason on the side that saw it.
4. Move the proxy to `enforce` once `report` is quiet.
5. Move the instance to `enforce`.
The proxy goes first because a cache hit never reaches the instance. Reversing steps 4 and 5 leaves every warm entry readable without a signature for as long as it lives.
From step 3 the instance caps `Cache-Control` on a read whose signature is valid at the time that signature has left, where `off` returns a year. `report` caps it too, so the entries stored during the dry run do not outlive the URLs that made them. A read `report` would refuse still returns a year, because there is no signature to cap it against. Entries stored before step 3 keep the year they were given.
### When the two sides disagree
| Proxy | Instance | What an attachment read gets |
| --- | --- | --- |
| `off` | `off` | Served. Neither side reads a signature |
| `off` | `report` | Served. The instance logs what `enforce` would refuse |
| `off` | `enforce` | Refused, unless a warm entry answers it without reaching the instance |
| `report` | `off` | Served. The proxy logs what its own `enforce` would refuse |
| `report` | `report` | Served. Both sides log |
| `report` | `enforce` | Refused, unless a warm entry answers it without reaching the instance |
| `enforce` | `off` | Refused at the proxy, before the cache lookup |
| `enforce` | `report` | Refused at the proxy. The instance logs nothing, because it is never asked |
| `enforce` | `enforce` | Refused at the proxy |
The two rows that serve a read the instance would refuse are the ones where the instance enforces alone. The instance is right and the cache is stale, and the entry answers until it goes, so pair an instance-first change with a purge of `/attachments/`.
A change to `FLUXER_MEDIA_PROXY_CORS_MODE` has the same problem and no step order that avoids it. An entry stored while the policy was `off` holds `Access-Control-Allow-Origin: *` and a year of `Cache-Control`, and the proxy replays both on every hit without asking the instance. The proxy's own signature check does not help, because it reads the signature and never reads `Origin`, so a foreign page holding one valid signed URL still reads the bytes cross-origin from a warm entry. Purge `/attachments/` when you move the CORS policy to `enforce`, the same purge step 5 needs.
`enforce` at the proxy alone covers every request that passes through it. Anything that can reach the instance another way is unchecked until the instance enforces too: a second cache, a health probe from outside the proxy, an internal service that reads an attachment URL directly, and a deployment running no proxy at all. Those are what step 5 covers, and what it can break.
Step 4 also ends the instance's own evidence. A read the proxy refuses never reaches the instance, so once the proxy enforces the instance logs nothing under `report` and its stream stays empty whatever it would have refused. Take step 5 on what step 3 showed, while both sides were still in `report`.
Both sides must hold the same secret list. A side missing a key refuses the URLs signed with it, so add a new key to both sides before the instance starts signing with it, and keep a retired key on both until the last URL it signed has expired. An ordinary URL expires within a day. A [data package URL](/http-api/messages/#data-package-attachment-urls) never does, and removing the key that signed it is what ends it.
With no secret list, `api` and `worker` return unsigned URLs, and a side in `enforce` refuses every attachment read. Never move a side to `enforce` before signing is on.
## Voice media does not use the proxy
LiveKit signalling goes through `/livekit/*` like everything else. WebRTC media does not touch the proxy at all: