mirror of
https://github.com/fluxerapp/fluxer
synced 2026-10-17 13:05:32 +09:00
fix(media-proxy): serve transforms whole, never as ranges (#3446)
This commit is contained in:
1 parent
0117605572
commit
83bb9d89c8
11 files changed
+250
-64
No files matched your search
@@ -183,11 +183,15 @@ This contract controls every range the Media Proxy resolves itself. A range is r
|
||||
|
||||
A reversed range, a zero-length suffix, a start outside the representation, or any range over an empty representation returns 416 with `Content-Range: bytes */{size}` and `Accept-Ranges: bytes`. A satisfiable range returns 206, `Accept-Ranges: bytes`, and exact `Content-Range` and `Content-Length` values.
|
||||
|
||||
### Transformed representations
|
||||
|
||||
A transformed representation is never served by range. Two Media Proxy instances can encode the same source to different bytes, so ranges answered by different instances do not join into one valid image. A response whose bytes a transformation produced ignores `Range`, returns the complete representation with 200, and has no `Accept-Ranges` header. `HEAD` does the same with an empty body. A rasterised SVG is a transformed representation, including the one an `mp` endpoint returns for a read that requests no transformation. A response that returns stored or origin bytes unchanged follows the contract above.
|
||||
|
||||
### Ranges on the signed external route
|
||||
|
||||
The route forwards a range to the origin only when no transformation is requested. It sends the range verbatim when the value after `bytes=` is non-empty and every byte of it is an ASCII graphic character. A multiple range therefore reaches the origin, and the origin decides how to answer it. A value with a space anywhere is dropped, and no range is sent. The route relays the origin partial response with the origin `Content-Range` unchanged.
|
||||
|
||||
A range on a transformed response selects bytes from the result and returns 206 or 416. Without a transformation, an origin that ignores the range can produce a complete 200 response. Always check the response status and range headers.
|
||||
A transformed response ignores the range and returns the complete result, as [Transformed representations](#transformed-representations) states. Without a transformation, an origin that ignores the range can produce a complete 200 response. Always check the response status and range headers.
|
||||
|
||||
:::caution[A mislabelled SVG reaches the client as bytes]
|
||||
An origin that answers a forwarded range with SVG bytes under another media type can produce a 206 containing raw SVG.
|
||||
@@ -203,7 +207,7 @@ Content disposition follows the media type the origin declared, so SVG mislabell
|
||||
| Content-Length?<sup>2</sup> | integer | The selected body length, including on HEAD |
|
||||
| Content-Range? | string | Present on 206 as `bytes {start}-{end}/{size}` and on 416 as `bytes */{size}` |
|
||||
| Content-Disposition?<sup>3</sup> | string | The route-selected inline or attachment disposition |
|
||||
| Accept-Ranges | string | The literal `bytes` on every media representation |
|
||||
| Accept-Ranges? | string | The literal `bytes`. Absent on a [transformed representation](#transformed-representations) |
|
||||
| Access-Control-Allow-Origin? | string | The literal `*` on media responses. Under `enforce` it is the request origin, or absent with no `Origin` |
|
||||
| Cache-Control<sup>4</sup> | string | The browser cache policy for the representation |
|
||||
| CDN-Cache-Control | string | The matching shared cache policy |
|
||||
|
||||
@@ -130,7 +130,7 @@ Every route-produced error response uses `Cache-Control: no-store`, and a succes
|
||||
|
||||
## Range response headers
|
||||
|
||||
A complete media response has `Accept-Ranges: bytes`, the representation `Content-Type`, and an exact `Content-Length`<sup>1</sup>. Outside `static` mode that `Content-Type` is `text/plain; charset=utf-8` when it names a JavaScript media type, and a 200 or 206 with no `Content-Type` gets the same value. [Content detection](/media-proxy/overview/#content-detection) defines both rules. A 206 also sets `Content-Range` to the selected interval over the complete size and `Content-Length` to the selected byte count. A 416 has `Content-Range: bytes */{size}` and `Accept-Ranges: bytes` with an empty body.
|
||||
A complete media response has `Accept-Ranges: bytes`, the representation `Content-Type`, and an exact `Content-Length`<sup>1</sup>. A [transformed representation](/media-proxy/overview/#transformed-representations) has no `Accept-Ranges` and is always complete. Outside `static` mode that `Content-Type` is `text/plain; charset=utf-8` when it names a JavaScript media type, and a 200 or 206 with no `Content-Type` gets the same value. [Content detection](/media-proxy/overview/#content-detection) defines both rules. A 206 also sets `Content-Range` to the selected interval over the complete size and `Content-Length` to the selected byte count. A 416 has `Content-Range: bytes */{size}` and `Accept-Ranges: bytes` with an empty body.
|
||||
|
||||
<sup>1</sup> A streamed signed external response omits `Content-Length` when the origin declared none
|
||||
|
||||
|
||||
@@ -68,7 +68,7 @@ Anyone holding the URL reads the object from any origin the [cross-origin read p
|
||||
|
||||
[When a transformation runs](/media-proxy/transformations/#when-a-transformation-runs) defines the trigger.
|
||||
|
||||
When a transformation is requested, `Range` selects bytes from the transformed result.
|
||||
A response with transformed bytes ignores `Range` and is always complete, as [Transformed representations](/media-proxy/overview/#transformed-representations) states.
|
||||
|
||||
An SVG attachment with a transformation parameter is rasterised, and `format` and `quality` select the output as they do for any other source, defaulting to WebP. Without a transformation parameter, only a Media Proxy in [`mp` deployment mode](/media-proxy/overview/#deployment-modes) rasterises it, always as lossless WebP. A Media Proxy in `upload` mode returns the stored SVG bytes.
|
||||
|
||||
@@ -79,11 +79,11 @@ A video source requires an explicit image `format`. A video request with `width`
|
||||
| Status | Body | Condition |
|
||||
| --- | --- | --- |
|
||||
| 200 | Complete media representation | The object and selected representation are available |
|
||||
| 206 | Selected media bytes | One range is satisfiable |
|
||||
| 206 | Selected media bytes | One range over the stored bytes is satisfiable |
|
||||
| 400 | Bad request | The decoded key, a dimension, the format, or the transformation request is invalid, or the transformation failed |
|
||||
| 404 | Not found | The attachment object does not exist and the signed attachment URL policy is `off` or `report` |
|
||||
| 404 | This content is no longer available. | The signed attachment URL policy refuses the signature, or the object does not exist under `enforce` |
|
||||
| 416 | empty | The range is unsatisfiable |
|
||||
| 416 | empty | A range over the stored bytes is unsatisfiable |
|
||||
| 504 | Gateway timeout | Transformation capacity was unavailable or the transformation exceeded its deadline |
|
||||
|
||||
[Common responses](#common-responses) also apply.
|
||||
@@ -140,7 +140,7 @@ A redirect beyond the [five-redirect bound](/media-proxy/responses-and-limits/#r
|
||||
|
||||
`effort` is read only on [Get attachment](#get-attachment) and is ignored here.
|
||||
|
||||
A transformation begins when `width`, `height`, `format`, or `quality` is present, when `animated` resolves to true, when the target filename ends in `.svg`, or when the origin body is SVG by media type or by its first bytes. A transforming request forwards no `Range` to the origin and applies the client range to the transformed bytes, so a satisfiable range returns 206 and an unsatisfiable range returns 416.
|
||||
A transformation begins when `width`, `height`, `format`, or `quality` is present, when `animated` resolves to true, when the target filename ends in `.svg`, or when the origin body is SVG by media type or by its first bytes. A transforming request forwards no `Range` to the origin. A response with transformed bytes ignores the client range and is always complete, as [Transformed representations](/media-proxy/overview/#transformed-representations) states. A transforming request that returns the origin bytes unchanged applies the client range to those buffered bytes, so a satisfiable range returns 206 and an unsatisfiable range returns 416.
|
||||
|
||||
A non-transforming request can forward the range to the origin. Handle both 200 and 206 responses and use their range headers, as described in [Byte ranges](/media-proxy/overview/#byte-ranges).
|
||||
|
||||
@@ -153,7 +153,7 @@ A video target requires an explicit image `format` to produce a thumbnail. Witho
|
||||
| Status | Body | Condition |
|
||||
| --- | --- | --- |
|
||||
| 200 | Complete external representation | The validated origin returned a complete representation |
|
||||
| 206<sup>1</sup> | Selected external bytes | The origin answered a forwarded range, or a range over buffered or transformed bytes is satisfiable |
|
||||
| 206<sup>1</sup> | Selected external bytes | The origin answered a forwarded range, or a range over buffered origin bytes is satisfiable |
|
||||
| 400 | Bad request | The path shape, decoded URL, redirect target, transformation request, or transformation is invalid or prohibited |
|
||||
| 401 | Unauthorized | The path signature is invalid |
|
||||
| 413 | Payload too large | The origin declared or delivered more than 500 MiB |
|
||||
@@ -162,7 +162,7 @@ A video target requires an explicit image `format` to produce a thumbnail. Witho
|
||||
| 503 | Service unavailable | Fluxer could not reserve or allocate a buffer for the origin body |
|
||||
| 504 | Gateway timeout | Transformation capacity was unavailable or the transformation exceeded its deadline |
|
||||
|
||||
<sup>1</sup> A forwarded range relays the origin `Content-Range` and `Content-Length` unchanged and omits either header the origin did not send, so a chunked origin 206 produces a 206 with no `Content-Length`. A local range over buffered or transformed bytes always has both
|
||||
<sup>1</sup> A forwarded range relays the origin `Content-Range` and `Content-Length` unchanged and omits either header the origin did not send, so a chunked origin 206 produces a 206 with no `Content-Length`. A local range over buffered origin bytes always has both
|
||||
|
||||
An origin status of 400, 401, 403, 404, 405, 406, 408, 409, 410, 411, 412, 413, 414, 415, 416, 428, or 429 reaches the client unchanged with the plain-text body `Upstream fetch failed`, and the origin body and headers are not returned. Any other unsuccessful origin status becomes 502. [Common responses](#common-responses) also apply.
|
||||
|
||||
@@ -258,7 +258,7 @@ An owner segment of the literal `.` or `..` parses as an asset path but produces
|
||||
|
||||
<sup>3</sup> An absent value keeps the hash-derived default, which is animated only for an `a_` hash. PNG and JPEG output has no animation
|
||||
|
||||
`width`, `height`, and `effort` are not read on an asset path, and no asset query value produces a 400. The route accepts a `Range` over the selected representation.
|
||||
`width`, `height`, and `effort` are not read on an asset path, and no asset query value produces a 400. The route accepts a `Range` when it returns the stored bytes unchanged. A transformed asset ignores `Range` and is always complete, as [Transformed representations](/media-proxy/overview/#transformed-representations) states.
|
||||
|
||||
<a id="image-asset-response"></a>
|
||||
|
||||
@@ -267,10 +267,10 @@ An owner segment of the literal `.` or `..` parses as an asset path but produces
|
||||
| Status | Body | Condition |
|
||||
| --- | --- | --- |
|
||||
| 200 | Complete asset representation | The original or transformed asset is available |
|
||||
| 206 | Selected asset bytes | One range is satisfiable |
|
||||
| 206 | Selected asset bytes | One range over the stored bytes is satisfiable |
|
||||
| 400 | Bad request | An owner segment of `.` or `..`, or a hash of `a_` on a path other than an emoji or sticker path, makes the storage key unsafe |
|
||||
| 404 | Not found | The path is not a valid asset path or the asset does not exist |
|
||||
| 416 | empty | The range is unsatisfiable |
|
||||
| 416 | empty | A range over the stored bytes is unsatisfiable |
|
||||
| 500<sup>1</sup> | Transcode failed | A required transcode failed and the source is not directly displayable |
|
||||
| 504 | Gateway timeout | Transformation capacity was unavailable or the transformation exceeded its deadline |
|
||||
|
||||
|
||||
+1
-3
@@ -95,7 +95,6 @@ fn external_cached_transform(
|
||||
url: &str,
|
||||
method: &Method,
|
||||
params: &HashMap<String, String>,
|
||||
headers: &HeaderMap,
|
||||
) -> Option<Response> {
|
||||
let hint = app.media.external_hints().get(url)?;
|
||||
let filename = url_filename(&hint.url);
|
||||
@@ -109,7 +108,6 @@ fn external_cached_transform(
|
||||
filename: &filename,
|
||||
route: TransformRoute::External,
|
||||
params,
|
||||
headers,
|
||||
},
|
||||
)
|
||||
}
|
||||
@@ -295,7 +293,7 @@ pub(in crate::server) async fn serve_external(
|
||||
};
|
||||
if prepared.wants_transform
|
||||
&& let Some(cached) =
|
||||
external_cached_transform(app, &prepared.url, &prepared.method, params, headers)
|
||||
external_cached_transform(app, &prepared.url, &prepared.method, params)
|
||||
{
|
||||
return cached;
|
||||
}
|
||||
|
||||
@@ -69,6 +69,41 @@ pub(in crate::server) fn media_response(response: MediaResponse<'_>) -> Response
|
||||
)
|
||||
}
|
||||
|
||||
pub(in crate::server) struct GeneratedMediaResponse<'a> {
|
||||
pub(in crate::server) method: Method,
|
||||
pub(in crate::server) data: BudgetedBytes,
|
||||
pub(in crate::server) content_type: &'a str,
|
||||
pub(in crate::server) disposition: Option<HeaderValue>,
|
||||
}
|
||||
|
||||
pub(in crate::server) fn generated_media_response(
|
||||
response: GeneratedMediaResponse<'_>,
|
||||
) -> Response {
|
||||
let GeneratedMediaResponse {
|
||||
method,
|
||||
data,
|
||||
content_type,
|
||||
disposition,
|
||||
} = response;
|
||||
let total_len = data.len();
|
||||
let body = if method == Method::HEAD {
|
||||
Body::empty()
|
||||
} else {
|
||||
Body::from(retained_response_bytes(data))
|
||||
};
|
||||
let mut response = media_body_response(
|
||||
body,
|
||||
StatusCode::OK,
|
||||
total_len,
|
||||
content_type,
|
||||
None,
|
||||
Some(total_len as u64),
|
||||
disposition,
|
||||
);
|
||||
response.headers_mut().remove(header::ACCEPT_RANGES);
|
||||
response
|
||||
}
|
||||
|
||||
pub(in crate::server) fn media_body_response(
|
||||
body: Body,
|
||||
status: StatusCode,
|
||||
|
||||
@@ -14,11 +14,11 @@ use crate::{
|
||||
load_media_input, read_sniff_prefix, resolve_metadata,
|
||||
},
|
||||
response::{
|
||||
MediaResponse,
|
||||
GeneratedMediaResponse,
|
||||
error::{
|
||||
canonical_reason_str, json_response, storage_error_response, text, text_with_source,
|
||||
},
|
||||
media_response,
|
||||
generated_media_response,
|
||||
},
|
||||
state::AppState,
|
||||
transform::execution::{
|
||||
@@ -231,11 +231,10 @@ pub(in crate::server) async fn thumbnail_handler(
|
||||
}
|
||||
}
|
||||
};
|
||||
media_response(MediaResponse {
|
||||
generated_media_response(GeneratedMediaResponse {
|
||||
method: Method::GET,
|
||||
data: BudgetedBytes::from(Bytes::from(media.bytes)),
|
||||
content_type: media.content_type,
|
||||
range_header: None,
|
||||
disposition: None,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -27,9 +27,9 @@ use crate::{
|
||||
filename_from_storage_key,
|
||||
},
|
||||
response::{
|
||||
MediaResponse, content_disposition_header,
|
||||
GeneratedMediaResponse, MediaResponse, content_disposition_header,
|
||||
error::{storage_error_response, text_with_source},
|
||||
media_response,
|
||||
generated_media_response, media_response,
|
||||
},
|
||||
state::AppState,
|
||||
transform::{
|
||||
@@ -146,11 +146,10 @@ pub(in crate::server) async fn serve_asset_image(
|
||||
resize_mode: Some(resize_mode),
|
||||
});
|
||||
if let Some(hit) = cached_transform_hit(runtime, &cache_key) {
|
||||
return media_response(MediaResponse {
|
||||
return generated_media_response(GeneratedMediaResponse {
|
||||
method,
|
||||
data: hit.data.as_bytes().clone().into(),
|
||||
content_type: hit.format.mime(),
|
||||
range_header,
|
||||
disposition: Some(content_disposition_header(
|
||||
hit.format.mime(),
|
||||
requested_download,
|
||||
@@ -202,11 +201,10 @@ pub(in crate::server) async fn serve_asset_image(
|
||||
});
|
||||
}
|
||||
};
|
||||
media_response(MediaResponse {
|
||||
generated_media_response(GeneratedMediaResponse {
|
||||
method,
|
||||
data: transformed.into(),
|
||||
content_type: out_ext.mime(),
|
||||
range_header,
|
||||
disposition: Some(content_disposition_header(
|
||||
out_ext.mime(),
|
||||
requested_download,
|
||||
@@ -390,7 +388,6 @@ async fn serve_spooled_attachment_video(
|
||||
filename,
|
||||
route: TransformRoute::Attachment,
|
||||
params,
|
||||
headers,
|
||||
},
|
||||
)
|
||||
};
|
||||
|
||||
@@ -18,8 +18,8 @@ use crate::{
|
||||
content_type_is_trustworthy, image_extension_from_filename, is_svg_content_type,
|
||||
},
|
||||
response::{
|
||||
MediaResponse, content_disposition_header, error::storage_error_response,
|
||||
error::text_with_source, media_response,
|
||||
GeneratedMediaResponse, content_disposition_header, error::storage_error_response,
|
||||
error::text_with_source, generated_media_response,
|
||||
},
|
||||
state::AppState,
|
||||
transform::{
|
||||
@@ -86,7 +86,7 @@ pub(in crate::server) async fn serve_stored_passthrough_stream(
|
||||
if app.cfg.mode == DeploymentMode::Mp
|
||||
&& image_extension_from_filename(key) == Some(AssetExtension::Svg)
|
||||
{
|
||||
return serve_stored_passthrough_svg(app, method, bucket, key, headers, &disposition).await;
|
||||
return serve_stored_passthrough_svg(app, method, bucket, key, &disposition).await;
|
||||
}
|
||||
let range_header = headers.get(header::RANGE).and_then(|v| v.to_str().ok());
|
||||
let forwarded_range = match range::classify_request_range(range_header) {
|
||||
@@ -105,7 +105,7 @@ pub(in crate::server) async fn serve_stored_passthrough_stream(
|
||||
}
|
||||
let content_type = passthrough_content_type(&object.content_type, key);
|
||||
if app.cfg.mode == DeploymentMode::Mp && is_svg_content_type(&content_type) {
|
||||
return serve_stored_passthrough_svg(app, method, bucket, key, headers, &disposition).await;
|
||||
return serve_stored_passthrough_svg(app, method, bucket, key, &disposition).await;
|
||||
}
|
||||
let total_len = match passthrough_total_len(app, bucket, key, object.total_length).await {
|
||||
Ok(value) => value,
|
||||
@@ -137,8 +137,7 @@ async fn serve_stored_passthrough_head(
|
||||
&& (is_svg_content_type(&content_type)
|
||||
|| image_extension_from_filename(key) == Some(AssetExtension::Svg))
|
||||
{
|
||||
return serve_stored_passthrough_svg(app, Method::HEAD, bucket, key, headers, disposition)
|
||||
.await;
|
||||
return serve_stored_passthrough_svg(app, Method::HEAD, bucket, key, disposition).await;
|
||||
}
|
||||
let total_len = match usize::try_from(head.content_length) {
|
||||
Ok(value) => value,
|
||||
@@ -163,7 +162,6 @@ async fn serve_stored_passthrough_svg(
|
||||
method: Method,
|
||||
bucket: &str,
|
||||
key: &str,
|
||||
headers: &HeaderMap,
|
||||
disposition: &PassthroughDisposition<'_>,
|
||||
) -> Response {
|
||||
let object = match app.store.read_object(bucket, key).await {
|
||||
@@ -171,15 +169,7 @@ async fn serve_stored_passthrough_svg(
|
||||
Err(err) => return storage_error_response(key, err),
|
||||
};
|
||||
let cache_identity = format!("{bucket}/{key}");
|
||||
serve_stored_svg_rasterized(
|
||||
app,
|
||||
method,
|
||||
object.data,
|
||||
&cache_identity,
|
||||
headers,
|
||||
disposition,
|
||||
)
|
||||
.await
|
||||
serve_stored_svg_rasterized(app, method, object.data, &cache_identity, disposition).await
|
||||
}
|
||||
|
||||
/// Resolves the object's full length, preferring what the store already reported on the streamed
|
||||
@@ -259,7 +249,6 @@ async fn serve_stored_svg_rasterized(
|
||||
method: Method,
|
||||
data: Bytes,
|
||||
cache_identity: &str,
|
||||
headers: &HeaderMap,
|
||||
disposition: &PassthroughDisposition<'_>,
|
||||
) -> Response {
|
||||
let runtime = app.media.transforms();
|
||||
@@ -287,11 +276,10 @@ async fn serve_stored_svg_rasterized(
|
||||
resize_mode: Some(options.resize_mode),
|
||||
});
|
||||
if let Some(hit) = cached_transform_hit(runtime, &cache_key) {
|
||||
return media_response(MediaResponse {
|
||||
return generated_media_response(GeneratedMediaResponse {
|
||||
method,
|
||||
data: hit.data.as_bytes().clone().into(),
|
||||
content_type: hit.format.mime(),
|
||||
range_header: headers.get(header::RANGE).and_then(|v| v.to_str().ok()),
|
||||
disposition: passthrough_disposition_header(disposition, hit.format.mime()),
|
||||
});
|
||||
}
|
||||
@@ -322,11 +310,10 @@ async fn serve_stored_svg_rasterized(
|
||||
);
|
||||
}
|
||||
};
|
||||
media_response(MediaResponse {
|
||||
generated_media_response(GeneratedMediaResponse {
|
||||
method,
|
||||
data: transformed.into(),
|
||||
content_type: format.mime(),
|
||||
range_header: headers.get(header::RANGE).and_then(|v| v.to_str().ok()),
|
||||
disposition: passthrough_disposition_header(disposition, format.mime()),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -230,6 +230,47 @@ async fn an_asset_image_transform_is_served_and_then_reused_from_its_content_ide
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_ranged_request_for_an_asset_transform_is_answered_with_the_whole_body() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let storage_root = tmp.path().canonicalize().unwrap();
|
||||
let app = test_app_state(asset_test_config(&storage_root));
|
||||
let asset = parse_standard_asset_path(ASSET_TEST_PATH).unwrap();
|
||||
app.store
|
||||
.write_object(
|
||||
&app.cfg.storage.bucket_cdn,
|
||||
&asset.storage_key,
|
||||
&crate::test_fixtures::synthetic_png(512, 512),
|
||||
"image/png",
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let params = HashMap::new();
|
||||
let mut range = HeaderMap::new();
|
||||
range.insert(header::RANGE, "bytes=0-3".parse().unwrap());
|
||||
let mut bodies = Vec::new();
|
||||
for headers in [&range, &range, &HeaderMap::new()] {
|
||||
let response = serve_asset_image(
|
||||
&app,
|
||||
Method::GET,
|
||||
parse_standard_asset_path(ASSET_TEST_PATH).unwrap(),
|
||||
¶ms,
|
||||
headers,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(StatusCode::OK, response.status());
|
||||
assert!(!response.headers().contains_key(header::ACCEPT_RANGES));
|
||||
assert!(!response.headers().contains_key(header::CONTENT_RANGE));
|
||||
bodies.push(response_body(response).await);
|
||||
}
|
||||
assert!(bodies[2].len() > 4);
|
||||
assert_eq!(bodies[2], bodies[0]);
|
||||
assert_eq!(bodies[2], bodies[1]);
|
||||
let rendered = app.metrics.render();
|
||||
assert!(rendered.contains("fluxer_media_proxy_transform_cache_misses_total 1\n"));
|
||||
assert!(rendered.contains("fluxer_media_proxy_transform_cache_hits_total 2\n"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_transform_failure_on_a_source_the_browser_cannot_show_stays_an_error() {
|
||||
let svg_failure = |error| {
|
||||
@@ -381,6 +422,70 @@ async fn a_stored_passthrough_serves_objects_larger_than_the_buffer_cap() {
|
||||
assert_eq!(&b"head"[..], &response_body(get).await[..]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_ranged_request_for_a_rasterised_stored_svg_is_answered_with_the_whole_body() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let storage_root = tmp.path().canonicalize().unwrap();
|
||||
let app = test_app_state(asset_test_config(&storage_root));
|
||||
let bucket = app.cfg.storage.bucket_cdn.clone();
|
||||
let key = "attachments/1/2/pic.svg";
|
||||
app.store
|
||||
.write_object(
|
||||
&bucket,
|
||||
key,
|
||||
br#"<svg xmlns="http://www.w3.org/2000/svg" width="8" height="8"><rect width="8" height="8" fill="red"/></svg>"#,
|
||||
"image/svg+xml",
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut range = HeaderMap::new();
|
||||
range.insert(header::RANGE, "bytes=0-3".parse().unwrap());
|
||||
let mut past_the_end = HeaderMap::new();
|
||||
past_the_end.insert(header::RANGE, "bytes=1048576-2097151".parse().unwrap());
|
||||
let mut bodies = Vec::new();
|
||||
for (method, headers) in [
|
||||
(Method::GET, &range),
|
||||
(Method::GET, &past_the_end),
|
||||
(Method::HEAD, &range),
|
||||
(Method::GET, &HeaderMap::new()),
|
||||
] {
|
||||
let response = passthrough::serve_stored_raw(&app, method, &bucket, key, headers).await;
|
||||
assert_eq!(StatusCode::OK, response.status());
|
||||
assert_eq!(
|
||||
Some("image/webp"),
|
||||
response
|
||||
.headers()
|
||||
.get(header::CONTENT_TYPE)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
);
|
||||
assert!(!response.headers().contains_key(header::ACCEPT_RANGES));
|
||||
assert!(!response.headers().contains_key(header::CONTENT_RANGE));
|
||||
let declared_len = response
|
||||
.headers()
|
||||
.get(header::CONTENT_LENGTH)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.and_then(|value| value.parse::<usize>().ok());
|
||||
bodies.push((declared_len, response_body(response).await));
|
||||
}
|
||||
let [
|
||||
(fresh_len, fresh_body),
|
||||
(cached_len, cached_body),
|
||||
(head_len, head_body),
|
||||
(whole_len, whole_body),
|
||||
] = bodies.try_into().expect("four responses");
|
||||
assert_eq!("image/webp", mime::sniff(&whole_body).mime);
|
||||
assert_eq!(Some(whole_body.len()), whole_len);
|
||||
assert_eq!(whole_len, fresh_len);
|
||||
assert_eq!(whole_len, cached_len);
|
||||
assert_eq!(whole_len, head_len);
|
||||
assert_eq!(whole_body, fresh_body);
|
||||
assert_eq!(whole_body, cached_body);
|
||||
assert!(head_body.is_empty());
|
||||
let rendered = app.metrics.render();
|
||||
assert!(rendered.contains("fluxer_media_proxy_transform_cache_misses_total 1\n"));
|
||||
assert!(rendered.contains("fluxer_media_proxy_transform_cache_hits_total 3\n"));
|
||||
}
|
||||
|
||||
fn poster_params() -> HashMap<String, String> {
|
||||
HashMap::from([
|
||||
("format".to_owned(), "webp".to_owned()),
|
||||
|
||||
@@ -31,9 +31,9 @@ use crate::{
|
||||
},
|
||||
native_task_executor::{NativeTaskExecutor, NativeTaskExecutorSettings},
|
||||
response::{
|
||||
MediaResponse, content_disposition_header,
|
||||
GeneratedMediaResponse, MediaResponse, content_disposition_header,
|
||||
error::{text_with_reason, text_with_source},
|
||||
media_response,
|
||||
generated_media_response, media_response,
|
||||
},
|
||||
transform::{
|
||||
cache::{
|
||||
@@ -350,7 +350,6 @@ pub(in crate::server) struct CachedVideoTransformRequest<'a> {
|
||||
pub(in crate::server) filename: &'a str,
|
||||
pub(in crate::server) route: TransformRoute,
|
||||
pub(in crate::server) params: &'a HashMap<String, String>,
|
||||
pub(in crate::server) headers: &'a HeaderMap,
|
||||
}
|
||||
|
||||
pub(in crate::server) fn cached_video_transform_response(
|
||||
@@ -371,10 +370,6 @@ pub(in crate::server) fn cached_video_transform_response(
|
||||
data: hit.data.as_bytes().clone(),
|
||||
content_type: hit.format.mime(),
|
||||
filename: request.filename,
|
||||
range_header: request
|
||||
.headers
|
||||
.get(header::RANGE)
|
||||
.and_then(|v| v.to_str().ok()),
|
||||
requested_download: transform.requested_download,
|
||||
}))
|
||||
}
|
||||
@@ -432,7 +427,6 @@ async fn serve_video_transform(
|
||||
data: hit.data.as_bytes().clone(),
|
||||
content_type: hit.format.mime(),
|
||||
filename,
|
||||
range_header,
|
||||
requested_download: transform.requested_download,
|
||||
});
|
||||
}
|
||||
@@ -486,7 +480,6 @@ async fn serve_video_transform(
|
||||
data: transformed,
|
||||
content_type: format.mime(),
|
||||
filename,
|
||||
range_header,
|
||||
requested_download: transform.requested_download,
|
||||
})
|
||||
}
|
||||
@@ -571,7 +564,6 @@ async fn serve_image_transform(
|
||||
data: hit.data.as_bytes().clone(),
|
||||
content_type: response_content_type,
|
||||
filename,
|
||||
range_header,
|
||||
requested_download: transform.requested_download,
|
||||
});
|
||||
}
|
||||
@@ -636,7 +628,6 @@ async fn serve_image_transform(
|
||||
data: transformed,
|
||||
content_type: response_content_type,
|
||||
filename,
|
||||
range_header,
|
||||
requested_download: transform.requested_download,
|
||||
})
|
||||
}
|
||||
@@ -725,7 +716,6 @@ pub(in crate::server) struct CachedTransformProbe<'a> {
|
||||
pub(in crate::server) filename: &'a str,
|
||||
pub(in crate::server) route: TransformRoute,
|
||||
pub(in crate::server) params: &'a HashMap<String, String>,
|
||||
pub(in crate::server) headers: &'a HeaderMap,
|
||||
}
|
||||
|
||||
// Answers a repeat transform from the cache without the caller fetching the source
|
||||
@@ -742,7 +732,6 @@ pub(in crate::server) fn cached_transform_response(
|
||||
filename,
|
||||
route,
|
||||
params,
|
||||
headers,
|
||||
} = probe;
|
||||
let transform = validate_transform_parameters(params, &runtime.limits(), route).ok()?;
|
||||
let source_is_svg = is_svg_content_type(content_type)
|
||||
@@ -787,7 +776,6 @@ pub(in crate::server) fn cached_transform_response(
|
||||
data: hit.data.as_bytes().clone(),
|
||||
content_type: &response_content_type,
|
||||
filename,
|
||||
range_header: headers.get(header::RANGE).and_then(|v| v.to_str().ok()),
|
||||
requested_download: transform.requested_download,
|
||||
}))
|
||||
}
|
||||
@@ -797,7 +785,6 @@ struct TransformedBytes<'a> {
|
||||
data: Bytes,
|
||||
content_type: &'a str,
|
||||
filename: &'a str,
|
||||
range_header: Option<&'a str>,
|
||||
requested_download: bool,
|
||||
}
|
||||
|
||||
@@ -807,11 +794,10 @@ fn transformed_response(request: TransformedBytes<'_>) -> Response {
|
||||
request.requested_download,
|
||||
Some(request.filename),
|
||||
));
|
||||
media_response(MediaResponse {
|
||||
generated_media_response(GeneratedMediaResponse {
|
||||
method: request.method,
|
||||
data: request.data.into(),
|
||||
content_type: request.content_type,
|
||||
range_header: request.range_header,
|
||||
disposition,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -335,6 +335,81 @@ async fn a_full_size_jxl_above_the_static_output_cap_is_downscaled() {
|
||||
);
|
||||
}
|
||||
|
||||
async fn jxl_attachment(
|
||||
runtime: &TransformRuntime,
|
||||
method: Method,
|
||||
params: &HashMap<String, String>,
|
||||
headers: &HeaderMap,
|
||||
) -> Response {
|
||||
serve_bytes_or_transform(
|
||||
runtime,
|
||||
ServeBytesRequest {
|
||||
method,
|
||||
data: Bytes::from_static(JXL_64_LOSSY_ALPHA),
|
||||
content_type: "image/jxl".to_owned(),
|
||||
cache_identity: "art.jxl",
|
||||
filename: "art.jxl",
|
||||
route: TransformRoute::Attachment,
|
||||
params,
|
||||
headers,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_ranged_request_for_a_transform_is_answered_with_the_whole_body() {
|
||||
let runtime = test_runtime();
|
||||
let transform = HashMap::from([("format".to_owned(), "webp".to_owned())]);
|
||||
let mut ranged = HeaderMap::new();
|
||||
ranged.insert(header::RANGE, "bytes=0-3".parse().unwrap());
|
||||
let mut bodies = Vec::new();
|
||||
for (method, headers) in [
|
||||
(Method::GET, &ranged),
|
||||
(Method::HEAD, &ranged),
|
||||
(Method::GET, &HeaderMap::new()),
|
||||
] {
|
||||
let response = jxl_attachment(&runtime, method, &transform, headers).await;
|
||||
assert_eq!(StatusCode::OK, response.status());
|
||||
assert!(!response.headers().contains_key(header::ACCEPT_RANGES));
|
||||
assert!(!response.headers().contains_key(header::CONTENT_RANGE));
|
||||
let declared_len = response
|
||||
.headers()
|
||||
.get(header::CONTENT_LENGTH)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.and_then(|value| value.parse::<usize>().ok());
|
||||
let body = to_bytes(response.into_body(), usize::MAX)
|
||||
.await
|
||||
.expect("body");
|
||||
bodies.push((declared_len, body));
|
||||
}
|
||||
let [
|
||||
(ranged_len, ranged_body),
|
||||
(head_len, head_body),
|
||||
(whole_len, whole_body),
|
||||
] = bodies.try_into().expect("three responses");
|
||||
assert!(whole_body.len() > 4);
|
||||
assert_eq!(Some(whole_body.len()), whole_len);
|
||||
assert_eq!(whole_len, ranged_len);
|
||||
assert_eq!(whole_len, head_len);
|
||||
assert_eq!(whole_body, ranged_body);
|
||||
assert!(head_body.is_empty());
|
||||
|
||||
let original = jxl_attachment(&runtime, Method::GET, &HashMap::new(), &ranged).await;
|
||||
assert_eq!(StatusCode::PARTIAL_CONTENT, original.status());
|
||||
assert_eq!(
|
||||
Some("bytes"),
|
||||
original
|
||||
.headers()
|
||||
.get(header::ACCEPT_RANGES)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
);
|
||||
let original_body = to_bytes(original.into_body(), usize::MAX)
|
||||
.await
|
||||
.expect("body");
|
||||
assert_eq!(&JXL_64_LOSSY_ALPHA[..4], &original_body[..]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_budget_leaves_room_for_the_cache_and_headroom() {
|
||||
assert_eq!(
|
||||
|
||||
Reference in new issue
Block a user