Formats & codecs

Depth behind the README’s “What it handles” matrix: codec routing, HDR signaling, audio bridging, subtitles, frame extraction, disc playback, and the documented edge cases. For the pipeline shapes these route through, see docs/architecture.md.

Containers and codecs

Containers (demux side): MKV, MP4, WebM, MPEG-TS, MPEG-PS (DVD VOB), AVI, OGG, FLV.

That list is the supported set, not the compiled set. The FFmpeg build also carries the HLS and DASH demuxers, and neither is a supported source shape: the build ships no network protocols (file, pipe and data only, since bytes come from the engine’s own AVIOReader over URLSession), so libavformat cannot fetch what a manifest points at. Remote HLS is handled a level up instead, by AVPlayer on the bypass or by the live ingest; a remote DASH manifest has no such path and is not playable.

Hardware decode (native AVPlayer path, VideoToolbox): H.264 (progressive), HEVC, HEVC Main10, on hardware whose VideoToolbox has a decoder for the specific profile. AV1 on devices with HW AV1 (M3+ Mac, iPhone 15 Pro+, future Apple TV chips) also routes natively.

Software decode (SoftwareVideoDecoder + AVSampleBufferDisplayLayer):

AV1 (libavcodec / dav1d) on devices without HW AV1 (currently all Apple TVs, M1 / M2 Macs, pre-A17-Pro iPhones).
VP9 and VP8 (libavcodec native) unconditionally, since AVPlayer’s HLS pipeline rejects the vp09 / vp08 CODECS attributes even where VideoToolbox can HW-decode them.
MPEG-4 Part 2 (XVID / DIVX / SP / ASP), MPEG-2 video, and VC-1, none of which AVPlayer’s HLS-fMP4 pipeline accepts; libavcodec ships native decoders for all three.
The legacy Microsoft tail (FFmpegBuild 2.4.3): MS-MPEG4 v1 / v2 / v3, the DivX 3.x that pre-2005 AVI rips carry, and WMV1 / WMV2 / WMV3 (WMV9). The routing was already correct before that release, so these reached SoftwareVideoDecoder and failed the load with unsupportedCodec for want of a compiled-in decoder. A native .wmv / .asf file plays whole since FFmpegBuild 3.1.0, which added the asf demuxer and the entire WMA decoder family (Standard, Pro, Lossless, Voice). That chain is all-or-nothing on purpose: the demuxer without a matching WMA decoder opens the file and then has nothing to decode its audio with, AudioBridge init fails, the cascade ends in droppedNoPipeline, and the session serves video-only and plays the file silently, which is a worse failure than an honest one. WMA is not fMP4-legal in any flavour, so it always takes the AudioBridge, like MP2 and Blu-ray LPCM. The video of such a file is usually WMV3 or VC-1, both of which were already here for their Matroska and MPEG-TS shapes. None of the six supports frame threading, so they decode single-threaded; the content that carries them is SD. Reported by cmcpherson274 (FFmpegBuild#3), whose library carries WMV only inside Matroska and MPEG-TS, which is why the native form waited for a second field report.
The Flash tail (FFmpegBuild 3.2.0): FLV1 (Sorenson Spark, the H.263 variant every pre-2008 .flv carries) and the On2 VP6 family (vp6, vp6f, vp6a) that Flash 8 brought. The flv demuxer has shipped from the start, so a modern Flash file (H.264 + AAC) always direct-played and only the legacy decoders were missing; that release also adds the era’s audio, Nellymoser Asao, ADPCM-SWF, Speex and FLV’s G.711 / big-endian PCM shapes, for the same all-or-nothing reason the WMA family shipped whole. Note where that reason lives: in the FFmpeg build, not in the routing table. AudioCodecCompat decides whether a codec is offered a stream-copy; the bridge cascade below it asks libavcodec for a decoder by id and never reads the table, so a codec the table does not name still plays when the build carries its decoder (measured on Nellymoser-in-FLV before its entry existed), and a codec the build lacks is silent whatever the table says. Flash Screen Video (flashsv / flashsv2) is deliberately absent: it needs zlib, which the build does not link, and screen recordings are not what a media library holds. Requested in the Sodalite Discord.
Every other video codec, by default. The native path is an allowlist of its own (HEVC, H.264, HW-decodable AV1) and HLSVideoEngine refuses the rest with unsupportedCodec, so a codec that is merely absent from the list above does not fall back to native, it fails the load. QuickTime RLE (qtrle, FFmpegBuild 2.4.1) is the case that surfaced it, and ProRes, MJPEG, Theora, Cinepak and rawvideo share the shape. Whether such a source then plays is a question for the FFmpeg build: with no decoder compiled in it fails at SoftwareVideoDecoder with Unsupported video codec, which is at least the honest error. AV_CODEC_ID_NONE is the one default-native case, since an audio-only source probes as NONE.
Interlaced H.264 (declared field order TT / BB / TB / BT), so the deinterlacer below can run; tvOS AVPlayer does not deinterlace, so 1080i / 576i broadcast combs on the native path (#107). On a seekable VOD source the declaration is verified against decoded frames first (#232). A field order describes how pictures are carried, not that any picture is interlaced: progressive-in-interlaced-carriage (PsF, which is what European 25 fps Blu-ray masters are, Blu-ray having no 1080p25) took this detour and then never deinterlaced, because the filter engages on AV_FRAME_FLAG_INTERLACED and on nothing else. InterlaceProbe decodes a short sample and applies that same predicate, so it measures the runtime’s own question rather than guessing at content: only a sample in which the flag never appears sends the stream back to the native path with hardware decode, an inconclusive one keeps the software route, and a flagged frame ends the sample at once (~29 ms on 1080i, ~100 ms for a full sample). Live sources are never probed.
H.264 High 4:2:2 / 4:4:4 / High-10 and HEVC Rext (4:2:2 / 4:4:4 / 12-bit) on hardware whose VideoToolbox has no decoder for the profile (Intel Macs, older Apple TV chips). AVPlayer accepts these at the HLS CODECS level and the item reaches readyToPlay, but the native path then renders nothing, so a per-format VTDecompressionSession probe at load (VTCapabilityProbe.canHardwareDecode) routes them to libavcodec, which decodes them (#2). Apple Silicon has HW decoders for all of these and keeps them native. The probe only judges a config record that actually carries out-of-band parameter sets: a source with in-band VPS/SPS/PPS (hev1, numOfArrays = 0) gives VideoToolbox no SPS to build a session from, which says nothing about hardware support, so those keep the native path (AetherPlayer#2).
H.264 that carries both stereo views inside one track, which is how a 3D Blu-ray MVC remux is muxed: Matroska StereoMode 13 / 14 (block_lr / block_rl, both eyes in one block), reported by libavformat as stream-level AV_PKT_DATA_STEREO3D of type AV_STEREO3D_FRAMESEQUENCE and spelled stereo_mode in the stream metadata (#435). The dependent view’s slices reference a subset SPS the base decoder never receives, so a plain H.264 decoder can only skip them; libavcodec does and decodes the base view, which is the left eye and the 2D fallback every non-3D player shows, while VideoToolbox is handed whole samples with both views’ NALs inside and renders nothing. The decision is the container’s declaration read at load, no probe. The frame-packed modes (side by side, top / bottom, checkerboard, row or column interleaved, anaglyph) are single self-contained pictures that keep the native path with hardware decode, and cropping an eye out of one of those is the host’s call, not the engine’s. MV-HEVC keeps the native path too: it is Apple’s own spatial-video format and AVPlayer plays its base layer. Real MVC 3D output is not offered on any path.

Interlaced sources (DVD-rip MPEG-2, SD / HD broadcast H.264) are deinterlaced through a persistent bwdif graph (yadif fallback) that engages on the first interlaced frame and costs nothing on progressive content. The dispatch decision lives in AetherEngine.load (VideoRoutingPolicy), gated per source on VTCapabilityProbe, codec id, declared field order, and on VOD the decode sample that verifies it.

MP4 without composition offsets

Some writers emit a sample table with no ctts while the H.264 bitstream still reorders pictures. Every sample then reports PTS == DTS, and since the native route stream-copies those timestamps into fMP4, AVPlayer is handed decode order as presentation order: each future reference picture is shown before the B pictures that precede it. Measured through AVFoundation’s own decoder on a twin pair (one encode muxed twice, composition offsets removed from one), 45 of 66 pictures landed at a time belonging to a different picture, with the content order stepping backwards 30 times (#409).

The container lost the information, but the bitstream did not: every slice header carries a picture order count, which is display order, and libavcodec’s H.264 parser reads it without decoding a pixel and takes MP4’s length-prefixed payload directly. H264CompositionOffsetRepair samples the head (twelve pictures at most, held rather than re-read, so no rewind and no second fetch) and repairs a confirmed source at the demuxer boundary:

PTS = (decode time of the picture that opened this coded video sequence) + shift + rank * step
DTS = DTS + shift - reorderDelay * step

Pulling decode time back by the reorder delay is what keeps PTS >= DTS; a healthy file carries the same negative head. shift is 0 or one reorder delay, depending on whether the writer left the sample ladder on the presentation axis or kept the edit list that trims the reorder head, and is clamped to that range so a malformed header cannot drag the picture off its audio. Because the rewrite happens once, in the demuxer, the fMP4 producer, the segment plan, the software decoder and the still extractor all read one axis, and the source keeps hardware decode: a missing table costs no route change. The container’s own index is folded onto the same ladder, since the segment plan is built from index entries and then filled with these packets.

Detection is fail-closed and costs a healthy file almost nothing: the first real PTS-DTS offset ends the sample (usually on the first packet, since a reordered file’s head sample sits one delay below zero). A source is only repaired when every sampled pair is equal, the decode ladder is uniform, the picture order regresses, and the ranks it produces are distinct and fill the sampled window. Anything short of that (variable frame timing, a picture order that does not advance one rank per picture, a sample that starts nowhere it can be anchored) is delivered exactly as the container wrote it. Reported by @orut34iop.

Matroska with presentation slots in coding order

Matroska block timestamps are presentation timestamps by specification, and the format has no composition-offset table to lose. A writer that fills them packet by packet while the bitstream reorders pictures has therefore not lost anything: every presentation slot is still in the file, each one just arrived attached to the picture that was decoded at that position rather than the one that is displayed there. On the reporting asset the first slots are 0, 40, 73, 107, 140 and the decoder emitted them as 0, 73, 107, 140, 40, one stepped-back presentation clock per mini-GOP for the length of the file. Measured here through the engine’s own software decoder on a generated twin, 15 of 30 frame times stepped backwards before the repair and 0 after (#511).

H264MatroskaSlotPermutation therefore permutes rather than reconstructs: a picture carries the slot its own display rank owns, and the slot is read from the file instead of being computed. Nothing fits a cadence, so a ladder quantized from a fractional frame rate is reproduced exactly rather than to within a tick, and a slot the writer clamped onto its cluster origin (the reporting asset has one, 7 ticks below the 1001/30 lattice its other 59 slots sit on) survives as written. Nothing moves the decode timestamps either: libavformat derives them from the rising slot ladder, which is the decode order the stream really has, and a picture at most its own reorder delay behind its slot cannot violate PTS >= DTS. The container index is untouched for the same reason, since it holds keyframe slots and a keyframe is the first picture of its own sequence.

The slot a picture needs is a packet away, not a plan away. A picture coded ahead of the slot it owns waits for the packet carrying that slot, which is the mini-GOP reorder created: three video packets on the reporting asset, in a 60-picture sequence. Nothing waits for the end of a sequence.

Detection is fail-closed and costs a healthy file almost nothing: one stepped-back slot is the container doing what the format says, and it ends the sample on what is normally the third packet. PTS != DTS is not an eligibility test here, because libavformat synthesizes a decode ladder from a rising presentation one just as readily as from a reordered one. A source is only repaired when the sampled slots rise strictly, the picture order regresses, the ranks are distinct, fill the sampled window and are a multiple of one measured step, and no picture sits further behind its own slot than the reorder delay the container declares. A stream that later stops being that shape, or a wait no mini-GOP explains, hands its packets back exactly as they arrived rather than permuting half a sequence. Diagnosed by @orut34iop on PR #511, whose numeric ladder is the regression fixture.

MP4 with composition offsets missing only in later regions

A healthy head does not establish a healthy table for the whole file. Some mixed MP4s retain valid offsets at the head, then give later reordered pictures zero offsets. Seeking into that region can produce persistent judder despite normal aggregate FPS and sufficient network buffering.

When a healthy origin picture corroborates the container’s edit/index lead, the demuxer keeps a zero-hold healthy path and watches for zero-offset IDRs. One bounded, complete IDR-to-IDR progressive sequence is parsed for picture order. If every selected packet has valid equal PTS/DTS and its distinct even POC fills the complete sequence, a proven permutation assigns its original DTS slots plus the corroborated presentation lead by display rank. Original DTS, audio and the already published keyframe index never move. Actual timestamp slots, rather than an average-FPS clock, preserve interval changes and quantization within a sequence.

The slot a picture needs is a packet away, not a plan away, exactly as in the Matroska policy above: a picture coded ahead of its own slot waits the mini-GOP the reorder created, so nothing waits for the end of a sequence and no sequence is too long to repair. The wait is bounded by the reorder delay the container declares, believed up to 16 pictures, and the packets held behind it by 1024 interleaved packets and 32 MiB, which is a ceiling on a container’s interleaving rather than on its GOP. A candidate sequence has to show its reordering within 64 pictures before a single picture is rewritten.

Healthy nonzero offsets resume unchanged delivery. Fields, missing timestamps, incomplete POC, a rank claimed twice, arithmetic overflow and insufficient lead are not guessed at. Every refusal hands the held packets back exactly as they arrived and lets the rest of that sequence stream through, so a shape this policy does not own costs the repair and never the session; the next IDR is a fresh candidate. Seek and teardown release both unpublished input and pending output. No decoder-route or host-UI change is needed. The existing whole-file missing-offset policy above remains separate.

See the partial-composition regression and reproduction for generated fixtures, original numeric evidence and verification limits.

HDR routing

Source	Wrapper signaling
H.264, HEVC (SDR)	BT.709
HEVC Main10 (HDR10)	BT.2020 / PQ
HEVC Main10 (HDR10+)	BT.2020 / PQ + per-frame ST 2094-40 SEI stream-copied
HEVC Main10 (DV P5)	`dvh1` track type (DV-only, IPT-PQ base; forced even on SDR panels)
HEVC Main10 (DV P8.1 / P8.4 / P7)	`hvc1` primary + `dvvC` box, DV engaged via `SUPPLEMENTAL-CODECS` on DV panels, plain HDR10 / HLG base elsewhere
HEVC Main10 (HLG)	BT.2020 / HLG
AV1 HDR	BT.2020 / PQ

HDR-to-SDR mapping is handled by AVPlayer and the system compositor according to the connected display. AetherEngine doesn’t tonemap on the host; it tells the system “this is BT.2020 PQ” (or DV, or HLG) via the HLS-fMP4 sample description and lets tvOS / iOS pick the right path.

An HDR master playlist is only served when the panel is ready for it. On tvOS the external panel must already be in HDR mode or Match Dynamic Range must be on (an SDR-parked panel rejects an HDR master with -11848). On iOS and macOS the built-in panel engages EDR on demand with no display mode switch, so AVPlayer.eligibleForHDRPlayback counts as readiness there; SDR-only devices read ineligible and stay media-direct. DisplayCriteriaController issues the HDMI content-frame-rate and dynamic-range hint via AVDisplayManager before the first segment is fetched, so the receiver-side handshake is in flight by the time AVPlayer is ready to render. (For why this ordering is mandatory on tvOS, see the README’s “Host setup on tvOS” section.) The per-mode capability split still comes from AVPlayer.availableHDRModes; the 26 SDKs deprecate it in favor of the eligibility Bool but ship no per-mode replacement, and the DV5 -11868 guard needs exactly that distinction, so the engine keeps the deprecated read until Apple obsoletes it. Since 6.82.0 that read may only ADD a mode, never subtract one: supportsHDR10 and supportsHLG take eligibleForHDRPlayback as their floor, because the table under-reports HLG over HDMI. Measured on a Samsung whose EDID advertises Hybrid Log-Gamma and which plays HLG in the TV’s own player, connected straight to an Apple TV, while the table reported .hlg absent; a second Apple TV on a different Samsung agreed, and an iPhone 17 Pro on its built-in panel reported it present, so the absence is about the HDMI path rather than the panel (AE#459). What that term reaches is one source: effectiveVideoFormat guards on Dolby Vision, so supportsHLG is consulted only for a DV Profile 8.4, which previously resolved to SDR on such a display while the manifest served VIDEO-RANGE=HLG. A plain HLG title was never clamped. Dolby Vision is left on the table alone, where it is measured correct in both directions (false on a Samsung without it, true on an iPhone 17 Pro the same day). That read does not exist on macOS at all (API_UNAVAILABLE(macos)), so the Mac table is derived from eligibility for HDR10 and HLG and leaves Dolby Vision unclaimed: eligibility proves EDR, not that AVFoundation will accept a DV variant on this display. A host that knows the hardware claims it with LoadOptions.panelPresentsDolbyVision, and a wrong claim costs one in-place media-playlist fallback rather than the item (AE#493) for the failures AVPlayer reports as an item failure. The claim no longer picks the packaging of a Profile 5, 8.1 or 8.4 source: 6.72.0 gave the non-DV branch its dvcC back and 6.73.0 its SUPPLEMENTAL-CODECS, so those three serve byte-identical manifests and segments with and without it, and what moves is the published videoFormat, the criteria request and the HDR readiness that rides along (Profile 7 and AV1 Dolby Vision are still gated on it). On tvOS that leaves a gap in the net: the -15628 an HDR10-only panel showed on the DV packaging in May 2026 (AE#4) is a stall rather than an item failure, it now reaches that panel with or without the claim, and it did not reproduce on tvOS 26.6. DV composition on a panel without Dolby Vision is what forceDolbyVisionOnNonDVDisplay is for there (AE#455). An HDR master additionally requires a known source frame rate (#130): AVPlayer filters a VIDEO-RANGE=PQ/HLG variant that carries no FRAME-RATE attribute out of the master at parse time and fails the item with -1002 without ever fetching the media playlist (SDR variants are accepted without it). The manifest frame rate uses the probe’s avg_frame_rate with an r_frame_rate fallback; a source where both are unset (some live MPEG-TS ingests) routes media-direct instead of serving a master AVPlayer provably rejects.

Non-DV HEVC derives its primary CODECS string from the source hvcC profile_tier_level (profile space, profile, tier, level, constraint bytes), so an 8-bit Main source is not mis-declared as Main10. The compatibility-flags element is the stored general_profile_compatibility_flags in REVERSE bit order per RFC 6381 / ISO 14496-15 Annex E: a real Main10 record stores 0x20000000 and prints hvc1.2.4..., matching MP4Box and Dolby’s own reference manifests. The declaration is checked against the init segment on device, so it has to be exact.

Whatever the profile, the hvcC shipped in init.mp4 must carry the parameter sets out of band. Sources authored with in-band VPS/SPS/PPS only (hev1, numOfArrays = 0, what MP4Box ...:xps_inband and the common Dolby Vision MP4 recipes produce) are normalized at load by scanning the head of the stream for the parameter sets and rebuilding the record; a record that arrives with extra non-parameter-set arrays (libx265’s user-data SEI) is stripped down to VPS/SPS/PPS instead. Shipping the source record verbatim in either case leaves AVPlayer with no usable format description: it buffers the whole forward window and never renders a frame.

Dolby Vision signaling

For DV streams the demuxer surfaces the source’s AVDOVIDecoderConfigurationRecord, and the route depends on the profile’s base-layer compatibility:

Profile 5 (DV-only, IPT-PQ, no base layer) emits a bare dvh1.05.<dvLevel> codec tag in the primary CODECS attribute with the dvcC box preserved. dvh1 is forced even on non-DV panels (AVPlayer’s system DV decoder tonemaps IPT-PQ internally; without dvh1 the IPT chroma reads as YCbCr and shows a green / purple cast), so P5-on-non-DV is routed through a media playlist to dodge the -11868 variant rejection.
Profiles 8.1 / 8.4 (HDR10- / HLG-compatible base) emit hvc1.2.4.L<level> as the primary CODECS tag. The muxer writes the dvvC box and the variant carries dvh1.08.<dvLevel>/db1p (8.1) or /db4h (8.4) in SUPPLEMENTAL-CODECS, which is what makes AVKit engage DV on a display that can present it. Both are emitted on every display, not only a DV-capable one: the pairing is what the authoring spec asks for, it is what lets a client that does not know dvh1 read the base layer instead of failing, and the loopback master reaches AirPlay receivers that are exactly that client. It used to be stripped, against an -11868 measured on tvOS 26.0 that no longer reproduces on 26.6, and keeping it is what makes AVPlayer put the RPU on the pixels wherever it has to convert the base layer, which on tvOS is every HDR source while the panel is not in HDR mode. On a panel that is already in HDR the base layer plays as plain HDR10 / HLG either way (AE#493).
Profile 8.1 on a non-DV display, with LoadOptions.forceDolbyVisionOnNonDVDisplay (experimental, default off, AE#455) is served the way Profile 5 is served instead: dvh1 sample entry, the container dvcC rewritten to profile 5 / compatibility 0, CODECS="dvh1.05.<dvLevel>", no supplemental. AVPlayer then runs its own Dolby Vision composition and applies the per-frame RPU to the pixels before they leave the device, where the default route gives the panel one static HDR10 grade. The bitstream never changes; only the container’s claim about it does, and what makes that hold together is that a Profile 8.1 RPU already carries the mapping out of its own HDR10 base layer, so the composer does not need the container to describe that layer. What makes it experimental is that this is not what the profile field means: a decoder that read the base layer’s colorimetry from the profile rather than from the RPU would decode IPT out of YCbCr, the green / violet cast of #4 and #176. Profile 8.1 only. Profile 8.4’s base layer is HLG, and a profile-5 dvcC over an HLG colr is a container that contradicts itself.
Any profile with a presentable base layer, with LoadOptions.dolbyVisionHandling = .baseLayerOnly is served as that base layer alone, on every display: plain hvc1 / av01 sample entry, the dvcC stripped, no supplemental, no Profile 7 conversion, and HDR10 / HLG display criteria rather than dvh1. The RPU NAL units stay in the samples and are ignored, which is the route a Profile 7 already takes on a display without Dolby Vision. It is a host option and not a routing decision because the case it exists for cannot be told from the container: a remux whose container record claims Profile 5 over a bitstream whose VUI declares BT.2020 YCbCr PQ, whose RPU carries the NLQ and residual fields only Profile 7 has, whose mapping is the identity and whose base layer is plain HDR10 with its own static metadata. Served as dvh1.05 the decoder reads YCbCr as IPT and the picture is green / violet; served as its base layer it is the HDR10 every player that ignores the record shows. The engine logs the contradiction on the default route (DV Profile 5 record over a BT.2020 YCbCr VUI) so a host knows to offer the option. What is admitted: HEVC Profile 7 / 8.1 / 8.4 and AV1 Profile 10.1 / 10.4, whose record names the base, and a Profile 5 / AV1 10.0 record whose VUI names one (matrix_coeffs BT.2020 with a PQ or HLG transfer; a genuine Profile 5 leaves both unspecified, IPT having no VUI code point). A Profile 5 whose VUI is unspecified has no base layer to present and keeps its route.
A Profile 5 record its own RPU contradicts is served as what the RPU says, on every display and with no option to set (#532). A remux whose container record claims Profile 5 over a bitstream whose VUI declares BT.2020 YCbCr with a PQ or HLG transfer states its own contradiction: IPT-PQ-c2 has no VUI code point, so a genuine Profile 5 leaves matrix_coeffs and transfer_characteristics unspecified. For that pairing, and only for it, the engine reads the first unspec62 RPU and takes libdovi’s guessed_profile over the record, which is proof rather than a heuristic because a Profile 5 RPU cannot carry a residual or an NLQ. An RPU that reads 7 takes the Profile 7 route above, an RPU that reads 8 takes the Profile 8.1 route and its compatibility rewrite, so the served container carries the record the source should have had. An RPU that agrees with the record, an RPU that cannot be read, and every source outside that pairing read no packets at all and keep their route, which is what makes the audit free for a genuine Profile 5. The read is a second open on the source, because the session’s probe demuxer is handed to the software path as it stands. LoadOptions.dolbyVisionHandling = .baseLayerOnly takes precedence over the correction, and the Profile 5 software-path refusal (#176) stands down for a corrected record, whose base layer is plain HEVC. AV1 Profile 10.0 is not covered: its RPU rides in an ITU-T T.35 metadata OBU rather than an unspec62 NAL, so the base-layer option remains the answer for that class.

AV1+DV emits a bare dav1.10.<dvLevel> primary for Profile 10.0 (DV-only, no base layer to fall back to), and an av01... primary plus a supplemental entry for the two cross-compatible profiles: dav1.10.<dvLevel>/db1p with VIDEO-RANGE=PQ for Profile 10.1 (HDR10-compat base) and dav1.10.<dvLevel>/db4h with VIDEO-RANGE=HLG for Profile 10.4 (HLG-compat base), on hardware-AV1 hosts. Same split as HEVC: the dav1 sample entry belongs to the profile without a base layer the way dvh1 belongs to Profile 5, and a cross-compatible profile carries its base layer’s own sample entry so a client that cannot read Dolby Vision still plays it.

Profile 7 (dual-layer, the common UHD-Blu-ray remux profile) has no decoder on any Apple platform, so the engine converts it to single-layer Profile 8.1 live during muxing: the RPU of each video packet is rewritten with libdovi (dovi_convert_rpu_with_mode, mode 2, the same transform as dovi_tool -m 2), the enhancement-layer NALs are dropped, and the container dvvC is set to Profile 8.1. On a DV-capable display this means real Dolby Vision (dvh1.08/db1p supplemental) instead of the plain HDR10 base; on a non-DV display Profile 7 still falls back to its HDR10 base, unchanged. The conversion is loss-free relative to what Apple could show before (the enhancement layer was never decodable on Apple hardware). MEL and FEL sources are both handled; a Full Enhancement Layer (FEL) source is logged, since its enhancement layer, which a native Profile 7 player would fold in, is discarded here while a Minimal (MEL) source loses nothing. Any per-packet conversion failure drops that RPU so the frame degrades to the clean HDR10 base, rather than shipping a Profile 7 RPU inside a container already declared 8.1. A session that converts publishes dolbyVisionConversion = .profile7ToProfile81, so a host label can name the profile actually served next to sourceDVProfile.

SDR-compatible-base profiles (HEVC Profile 8.2, AV1 Profile 10.2) carry a Rec.709 base layer that no Apple platform has a DV decoder for. The engine strips the dvcC / DV config and plays the base as plain hvc1 / av01 (logging not DV-routable, playing Rec.709 base); there is no Dolby Vision on any display for these, on DV panels and SDR panels alike.

HDR10+ dynamic metadata

ST 2094-40 metadata stays attached to the HEVC bitstream as user-data-registered ITU-T T.35 SEI NALs. The HLS-fMP4 stream-copy preserves the SEI through to AVPlayer, which forwards it to the system compositor. HDR10+-capable TVs apply the per-scene tone-mapping curves; HDR10-only TVs fall back to the static HDR10 base.

The published videoFormat starts at .hdr10 for any BT.2020 / PQ source and flips to .hdr10Plus the first time a packet’s T.35 SEI signature is seen in the producer’s scan. That includes a Dolby Vision source carrying an HDR10+ layer next to its RPU (Blu-ray Profile 7 does, and so does a Profile 8.1 remuxed from one) whenever the label resolved to its HDR10 base, while sourceVideoFormat keeps saying .dolbyVision; the evidence is latched per session, so a label republished later by the panel proof (AE#459) keeps it. Debounced across producer restarts so a scrub doesn’t re-fire. Hosts can drive an HDR10+ badge or analytics hook off the $videoFormat transition.

The label can also be taken back from the item itself, where the platform has no capability table to clamp it against (AE#515). A Dolby Vision source on macOS resolves to .hdr10, because supportsDolbyVision is unclaimable there without a host assertion, while AVFoundation goes on playing the dvh1 sample entry the engine served. Measured with the assertion off on a 16” XDR, a Profile 5 and a Profile 8.1 grade of Dolby’s reference content both strobe, so the RPU reaches the pixels with no claim set anywhere and the clamp was moving nothing but the label. When the item’s sample entry reads dvh1 / dvhe and the probe agrees the source is Dolby Vision, the label is upgraded from .hdr10 to .dolbyVision at readyToPlay. It is an upgrade and not a mirror of what AVFoundation parsed, for two reasons that both matter: an .sdr label is the clamp being right about a display presenting no HDR at all, and on tvOS and iOS the per-mode table answers the capability question, so the label follows it rather than a sample entry that a Profile 5 master carries on every panel. Profile 8.1 keeps .hdr10 on macOS: it reports hvc1 with the DV configuration alongside it, it composes on that display all the same, and nothing in the stack reports that.

Audio


Stream-copy (lossless into fMP4)	AAC-LC, AC3, EAC3, FLAC, ALAC. HE-AAC / HE-AACv2 stream-copy when the source carries an AudioSpecificConfig (any movie container) and bridge only without one (live ADTS / MPEG-TS, where a synthesized ASC would mis-signal SBR). LATM/LOAS-framed AAC (DVB broadcast framing) always bridges
Bridged (`AudioBridge`)	TrueHD, MLP, DTS, DTS-HD MA, MP3, MP2, Opus, Vorbis, PCM (raw shapes plus G.711 A-law / mu-law), WMA (Standard, Pro, Lossless, Voice), and the Flash tail Nellymoser Asao / ADPCM-SWF / Speex: decoded to PCM and re-encoded
Surround	5.1 / 7.1 with correct `AudioChannelLayout` preserved through the wrapper

Non-streamable codecs route through AudioBridge in one of two modes (LoadOptions.audioBridgeMode):

.surroundCompat (default): the encoder is chosen per source, not per mode. A source with more than two channels is re-encoded to lossy EAC3 at 128 kbps per channel (768 kbps 5.1). AVPlayer hands the encoded bitstream to HDMI and the sink decodes its own 5.1 mix, so surround works on essentially every modern AVR and soundbar (Sonos Arc, Samsung HW-Q, Bose). Two channels or fewer have no surround to carry and take the FLAC encoder instead, which is lossless where EAC3 would have been 256 kbps lossy, and LPCM the route decodes on the routes that can only pass a Dolby bitstream through (AE#395: an AirPlay 2 optical adapter played one MPEG-TS program’s stream-copied AC3 5.1 and was silent on the bridged EAC3 stereo of the same program, so which track av_find_best_stream happened to pick decided whether the viewer heard anything).
.lossless (opt-in): FLAC up to 7.1 lossless, which AVPlayer decodes to LPCM. Needs an AVR that accepts multichannel LPCM via HDMI (Denon, Marantz, NAD); on soundbars and basic AVRs that handle multichannel only via bitstream codecs the LPCM gets downmixed to stereo at the route.

Either encoder opens at a sample rate it actually has, which is not always the source’s (AE#548). E-AC-3 exists at 32 / 44.1 / 48 kHz only, so a 96 kHz TrueHD or DTS-HD master is resampled to 48 kHz on the .surroundCompat arm; opening the encoder at the source rate instead made avcodec_open2 refuse the context and took the whole session to silent video-only. The rates come from avcodec_get_supported_config, so an FFmpeg bump that widens them needs no change here, and an encoder that advertises no list (FLAC) keeps the source rate untouched, which is what keeps .lossless bit-perfect at 96 and 192 kHz. The resampler is in the path on every bridged packet either way, so the conversion costs nothing that was not already being paid.

.surroundCompat is the default because the soundbar / basic-AVR install base is the majority. Object metadata (Atmos / TrueHD-MA) is lost in either mode: FFmpeg’s EAC3 encoder doesn’t produce JOC, and FLAC has no object-channel concept. If a JOC source ever falls through to the bridge the engine logs a loud WARNING: Atmos downgrade, ....

Two bridge lifecycle invariants (issue #99): the encoder PTS counter re-bases onto the first fed packet’s (gate-shifted) source PTS on every session start and producer restart, so bridged audio always shares the video’s output timeline, including a load(startPosition:) resume that anchors mid-file (a 0-based bridge timeline puts the audio track a full resume-offset away from video inside the same fragments, which AVPlayer silently discards). And the EOF tail flush leaves the encoder in FFmpeg’s terminal draining state, so the bridge latches that and rebuilds the encoder on the next restart; a VOD pump that still dies with muxerFailed gets a bounded producer rebuild instead of stranding the session.

Audio a build has no decoder for

AC-4 (ATSC 3.0 / NextGen TV) and MPEG-H 3D Audio have no decoder in the bundled FFmpeg, and there is nothing to switch on: FFmpeg carries codec ids for both so a container can be demuxed, but libavcodec/allcodecs.c names neither. The AC-4 decoder patches have sat out of tree for years and the format is Dolby patent-encumbered; MPEG-H has an upstream wrapper around Fraunhofer’s mpeghdec, whose licence is not LGPL-redistributable. Apple’s own stack does not fill the gap either: there is no AC-4 format constant in CoreAudio.

Such a track is not merely silent, it is expensive. has_codec_parameters fails an audio stream with no sample rate, and that value can only come from the container or from opening a decoder, so find_stream_info reads to the full probe budget before failing open with the track missing anyway. On a live source that budget is spent at the wire rate, which is where Sodalite#100’s minute-long tuning indicator came from. The demuxer therefore parks a stream whose codec has no decoder AND whose parameters the container left unset out of the probe’s way, and restores it immediately after, so the open costs what it would have without the track and the caller still sees the track in audioTracks. A host with its own metadata (Jellyfin names a live channel’s audio codec in PlaybackInfo without opening a tuner) can do better still and refuse the channel with a real sentence.

Track language on the native path

AVFoundation reads a track’s language from the master playlist, not from the media, so the audio the engine muxes into its single variant is also declared there: one URI-less EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud",LANGUAGE="<iso 639-2/T>",DEFAULT=YES,AUTOSELECT=YES plus AUDIO="aud" on the variant (RFC 8216 4.3.4.2.1, the URI is absent precisely because the audio is inside the variant). AVPlayer then exposes it as a one-option audible AVMediaSelection group and labels it from LANGUAGE, which is what a system audio menu shows. Measured on macOS 26: the same fMP4 whose audio mdhd reads deu reports AVAssetTrack.languageCode == nil and no audible group at all when the master does not declare the rendition, while the identical mdhd read from a progressive .mp4 reports deu. The mdhd is written too (the track is that language whoever reads it), it just is not what the label comes from. The rendition is advertised only when the audio actually reached the variant, so an audio cascade that fell through to video-only never names a group its segments do not carry (AE#458).

Source labels are resolved to ISO 639-2/T through ICU, which covers every language it knows plus BCP-47 subtags (pt-BR becomes por) and rejects free text such as English or a track title, in front of a twenty-row table for the ISO 639-2/B bibliographic codes ICU does not resolve and Matroska routinely writes (ger, fre, cze). Two ISO 639-3 classes need more than that lookup. The members CLDR aliases to a macrolanguage (cmn, arb, pes, swh, uzn, kmr) have no alpha3 entry at all, so the tag is canonicalized where the direct route came back empty, which resolves the alias without moving any tag that already resolved (no stays nor, not nob). The members CLDR does NOT alias (cnr, prs, npi, ory, quz, crs and 50 more, 56 measured on macOS 26) have neither an alpha3 entry nor a canonical form, and there is nothing to convert: the tag already IS the ISO 639 code, so it passes through as itself. That last step is gated on ICU having a display NAME for the tag in a fixed reference locale, which is the validity signal canonicalization cannot give (canonicalLanguageIdentifier echoes dub and xyz back unchanged exactly as it echoes cnr). The locale is fixed rather than the device’s, or a file would resolve on an English Apple TV and not on a German one. A label that resolves to nothing writes nothing, so an untagged source keeps the master it had before.

Dolby Atmos

EAC3+JOC packets are stream-copied through the muxer untouched, on every output route. AVPlayer reads the segment, recognises JOC from the dec3 box (numDepSub=1, depChanLoc=0x0100), and lets the downstream renderer decide: over HDMI it tunnels out as Dolby MAT 2.0 and the AVR lights up the Atmos indicator; over AirPods it renders spatially; over plain Bluetooth A2DP / LE it downmixes the bed channels to stereo natively. The route never changes the engine’s decision (a JOC track is signaled in the playlist as ec-3, the same CODECS string as a non-JOC EAC3 5.1 track, so AVPlayer accepts it everywhere and the bitstream is never re-encoded for a route reason). The engine emits an explicit [HLSVideoEngine] EAC3+JOC Atmos: stream-copy engaged; ... diagnostic on every Atmos session.

Matroska CodecPrivate doesn’t usually carry the pre-parsed dec3 / dac3 box content the mov muxer needs at avformat_write_header time, so the muxer is configured with +delay_moov (alongside +empty_moov+default_base_moof+frag_custom). The moov atom is deferred until the first fragment-cut flush, by which point packets have flowed through mov_write_packet and libavformat’s handle_eac3 / handle_ac3 have populated the sample-entry boxes from the actual packet bitstream. The first cut emits the deferred ftyp+moov (routed by FragmentSplitter to init.mp4); subsequent cuts emit normal moof+mdat. Net effect: EAC3 / AC3 from matroska direct-play stream-copies cleanly with valid sample-entries, no manual bitstream parsing on the host side.

Subtitles

Subtitle cues come from one read: EVERY embedded subtitle stream (text and bitmap) stays in the session demuxer’s keep-set and is harvested by a tap on the host’s existing source read (each packet is observed then dropped, never muxed). Harvested packets are retained compressed in a per-session SubtitlePacketStore (300 s trailing window, byte-capped per stream) and a playhead-paced drainer decodes the selected stream near the playhead into the overlay. PGS streams in MPEG-TS (Blu-ray) arrive as one display set split across several PES packets (PCS|WDS|PDS|ODS|END, some without a PTS of their own); the store reassembles those chunks into one self-contained entry at the PCS presentation PTS before retention, so the drainer always decodes complete display sets (Matroska already carries one complete set per packet and is stored as-is), so enabling any embedded track is instant: the selection backfills synchronously from the store, with no positioning seek and no recovery machinery, even on remote disc images; the tap re-attaches with the producer across seeks and restarts. The tap’s forward coverage ends at the producer’s read position (its forward park, #102), which on direct play sits only a few seconds past the playhead; on VOD sessions a background subtitle-only side reader (the forward prefetcher, #151) therefore extends the store past the drainer’s 60 s lead window, by a 15 s margin so the set at the window’s forward edge still has its own successor stored (#362; raised to 270 s while a bitmap track’s OCR rendition is armed, which clears the OCR worker’s own 240 s window by 30 s so the store actually holds what the worker is about to read), so subtitleCues holds cues ahead of the playhead for a host-applied ADVANCE sync offset, text and bitmap alike (live sessions skip it; content past the edge does not exist). It is best-effort: if it cannot open or wedges, the tap-fed behavior above is unchanged. Its positioning seek is anchored on the subtitle stream’s own timestamp axis (#234): left to libavformat’s default reference the anchor follows whichever stream is not fully discarded, so the pacing stream that gives the park its control point (#230) would move it onto video keyframes, and on Matroska that lands in a later cluster and drops a landing cue that starts further back than the destination. The side reader shares the source link with the video pump, and on Matroska it is a second full copy of the stream (matroska_parse_cluster reads every block off the wire; only matroska_parse_block then honours the discard flag), so a subtitled session asks the link for about twice the media rate. Where the link has little headroom the two starve each other, so the video path has priority (#240): a side reader fetches while the pump is parked, yields while the pump is fetching and completely while a seek is in flight, keeps a bounded grace window after each anchor so a freshly selected track still fills, and takes the link back after a 60 s continuous yield so a pump that never parks cannot disable lookahead for the session. A playhead jump re-anchors the running session in place rather than rebuilding it, since a rebuild pays an open, the Matroska cue-index prewarm and a positioning seek, each a bounded range the origin delivers in full. Measured on a 1.4x-headroom bench, far-seek landings improved about 45% at the median and the side reader’s share of the link halved. Sidecars are fetched once. Each packet decodes through avcodec_decode_subtitle2 (except in-band CEA-608, which has an in-house line-21 decoder, see below), and the result lands in a single [SubtitleCue] published list:

Text codecs (SubRip / ASS / SSA / WebVTT / mov_text) → SubtitleCue.body = .text(String), or .richText([SubtitleTextRun]) when the cue asks for styling (#233). libavcodec converts every text format into an ASS event line before the engine sees it, so one override parser serves all of them: inline \b / \i / \u / \s / \c / \1c / \fn / \fs / \r become attributes on the runs (isBold, isItalic, isUnderlined, isStruckThrough, color, fontName, fontSize), and a cue that asks for nothing stays plain .text with the exact string it produced before. \N becomes a real newline either way, and cue.text flattens both forms. Cue-level \an and \pos lift out into SubtitleCue.placement (ASS numpad alignment plus an optional [0..1]-normalised anchor, y from the top) instead of splitting a run. WebVTT cue settings reach the same field from packet side data (AV_PKT_DATA_WEBVTT_SETTINGS): the WebVTT decoder drops line / position / align, but the demuxer keeps them, so line:10% align:start arrives as an alignment and an anchor. size and vertical have no equivalent in the placement model and are ignored, and a position without a line keeps only the alignment column, since an anchor point needs both axes. DVB teletext subtitles (live broadcast) decode through libzvbi (libzvbi_teletextdec, configured to emit ASS so the per-character colour broadcasters use to distinguish speakers survives rather than being flattened) with page-state semantics (#107). A page carrying colour publishes as SubtitleCue.body = .richText([SubtitleTextRun]) (each run an optional RGB SubtitleColor, nil meaning the host default); a page with no colour stays plain .text, and cue.text flattens either form for consumers that do not render colour. Teletext placement arrives two ways, because libzvbi writes it two ways: on a page flagged as a subtitle page it derives the vertical anchor from the grid row and emits {\anN}, which lands in SubtitleCue.placement like any other alignment; on a page it does not flag (broadcaster leaves NEWSFLASH / SUBTITLE / SUPPRESS_HEADER unset, or the row-0 header has not been seen yet) it writes the whole page instead, one row per line, and the ordinal of the first non-blank row is the only carrier of the position. The engine applies libzvbi’s own third-of-the-page formula to that ordinal, so a caption the broadcaster moved to the top of frame stays there either way, and never overrides an \an that did arrive. That same split governs whitespace. The unflagged page is the raw grid, every row at full column width with a hard space per cell, so the engine trims the padding off both ends of each row and folds the empty rows it also emits; a page the decoder curates itself arrives untouched, since there the surviving padding is the relative indentation carrying the alignment it chose and the blank lines are its vertical fine-positioning inside the block. The decoded page defaults to libzvbi’s auto-detected subtitle page, overridable per channel via LoadOptions.teletextPage (for example AU page 801, which libzvbi does not always flag as a subtitle page) and changeable while the channel plays via setTeletextPage(_:), which rebuilds the drain decoder of every channel currently showing a teletext track and leaves the others alone (#364). libzvbi emits page content open-ended (“valid until replaced”) and page erases as empty events, so every teletext event trims earlier open cues at its start; roll-up captions build and replace cleanly, an erase clears the line, and a 120 s cap bounds a ghost line if transmission stops without either. Validated end to end against Australian FTA broadcasts (1080i25 H.264, captions on page 801), where the interlaced video deinterlaces through the software path’s bwdif filter.
Bitmap codecs (PGS / HDMV PGS / DVB / DVD) → .image(SubtitleImage). The indexed pixel plane is walked through its palette, premultiplied against alpha, and wrapped as a CGImage. Position is normalised in [0..1] against the composition canvas, whose coded pixel size rides along as SubtitleImage.canvasSize (#112): a cropped-video rip can author a canvas taller than the coded video, so hosts map the canvas width-aligned and center-anchored onto the on-screen video rect to land cues where the disc authored them; .zero means treat canvas == video. A display set carrying multiple composition objects (a forced sign plus dialogue is a common real-disc shape) fans into one cue per object, all sharing the set’s start PTS, and every object is retained and rendered (#146); each object’s AV_SUBTITLE_FLAG_FORCED rides along as SubtitleImage.isForced, surfaced per cue via SubtitleCue.isForced (track-level forcedness stays on TrackInfo.isForced).
External files (a separate .srt / .ass / .vtt URL) → register as first-class tracks (see below) or one-shot via selectSidecarSubtitle(url:httpHeaders:), which opens its own short-lived AVFormatContext, decodes the whole file once, atomically swaps the result into subtitleCues. The fetch forwards the session’s LoadOptions.httpHeaders by default (WebDAV auth and friends); pass the call’s own httpHeaders to override per fetch.
In-band CEA-608 closed captions (eia_608 / QuickTime c608, a demuxable caption track) → .text(String). FFmpegBuild ships no ccaption decoder, so these never reach avcodec_decode_subtitle2. Instead a read-only tap on the segment producer’s existing source connection reads the caption track’s cc_data (its packets are kept in the demuxer’s keep-set, observed, then dropped, never muxed, so the loopback-HLS output stays byte-identical), an in-house line-21 decoder (validated against FFmpeg’s ccaption_dec.c) turns it into cues, and they publish on the same subtitleCues overlay path as every other codec. First cut: field-1 / channel CC1; CEA-708 (DTVCC) and field 2 are follow-ons. Captions carried only inside the video bitstream (ATSC A/53 cc_data, the US broadcast/cable case) are extracted too (#131): on the native remux path the segment producer scans H.264/HEVC video packets for user_data_registered_itu_t_t35 SEI (GA94), reorders the decode-order groups to presentation order by the packet DTS watermark, and feeds the same line-21 decoder; on the software-decode path (MPEG-2 and friends) the triplets come from AV_FRAME_DATA_A53_CC decoded-frame side data. Since no caption AVStream exists, a synthetic eia_608 track (id 99608) surfaces lazily on the first real (non-padding) caption pair, so uncaptioned channels never show a dead menu entry. Host-overlay only (no PiP / AirPlay), like the bitmap codecs. (#77)

External subtitle files as first-class tracks

External subtitle files register with the engine and appear in subtitleTracks next to the embedded streams, so a host keeps one track list and one selection call (#88):

Registration. LoadOptions.externalSubtitles: [ExternalSubtitleTrack] declares files at load; addExternalSubtitleTrack(_:) registers any time mid-session (returns the created TrackInfo). The descriptor carries url, optional name / language / disposition flags, per-track httpHeaders (nil forwards the session’s), and a formatHint for URLs whose path hides the extension.
Containers with several subtitle streams. An external URL can be a container rather than a sidecar (an MKV holding English, English SDH and Spanish). ExternalSubtitleTrack.sourceStreamIndex names which stream to decode as an ABSOLUTE AVStream index inside that container, so a host registers one track per stream against the same URL; nil decodes the container’s first subtitle stream. An index that is out of range or names a non-subtitle stream fails the decode rather than falling back, which would be indistinguishable from leaving it nil. Tracks sharing a URL and headers are filled from a SINGLE pass over the container, so N tracks cost one fetch, not N (#266). Note that TrackInfo.codec is still derived from the URL extension, so a container URL reports subrip; set formatHint: "ass" when the streams are ASS and the host drives a styled renderer.
Identity. External TrackInfo.ids are synthetic: AetherEngine.externalSubtitleTrackIDBase (100 000) + registration ordinal, monotonic per load; load-declared tracks get base + array index in order. TrackInfo.isExternal distinguishes them from AVStream-indexed embedded tracks.
Selection. selectSubtitleTrack(index:) and selectSecondarySubtitleTrack(index:) accept external ids and route onto the whole-file decode internally; activeSubtitleTrackIndex publishes the external id like any other selection. removeExternalSubtitleTrack(id:) unregisters (an active selection is cleared).
Renditions. Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load (one pass per container, covering every track pointing at it) and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). A store that could not be filled stays unfinished rather than serving a complete but blank .vtt. Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation.
Renditions on the nativeRemoteHLS bypass (#316). That path plays the origin playlist directly and has no loopback master to declare anything in, so a declared sidecar used to be dropped outright. For a VOD source the engine now fetches the origin master, absolutises every variant / audio / key URI against the origin, adds one EXT-X-MEDIA:TYPE=SUBTITLES per text sidecar (joining the origin’s own subtitles group when it has one, so a variant’s group choice cannot lose them) and serves that master from the loopback origin. The media never moves: AVPlayer fetches A/V from the origin exactly as before, which is the property the bypass exists for (E-AC-3 / Atmos passthrough). Renditions are declared DEFAULT=NO,AUTOSELECT=NO and without FORCED, the same discipline as the loopback master, so nothing self-engages; the track keeps its external id and selecting it drives AVMediaSelection rather than the overlay, so it survives PiP / AirPlay / an external display. Live playlists (no EXT-X-ENDLIST), bitmap sidecars (.sup), an unparseable or unrewritable playlist and an origin slower than the 5 s budget all keep the origin URL and overlay-only subtitles.
Preferences. preferredSubtitleLanguages ranks external tracks together with embedded ones. A track added mid-session re-runs the preference and auto-activates on a match, but only while the host has made no explicit subtitle call (select / sidecar / clear) in the session, so a deliberate subtitles-off stays off.

Track selection by language preference

LoadOptions can seed the initial audio and subtitle tracks from an ordered language preference, resolved from the engine’s single probe so a host honors a saved preference without a separate pre-probe or a post-load reload:

preferredAudioLanguages (ordered ISO 639-1 / 639-2 codes or English names, e.g. ["en", "de"]) picks the first-frame audio track: an explicit audioSourceStreamIndex wins, else the first track matching a preference in order, else the container default. The pick is muxed into the loopback HLS, so it is correct on the first frame with no selectAudioTrack reload.
preferredSubtitleLanguages activates a subtitle at the end of load. Within the first preference that has a match, it picks the best track by container disposition: full subtitles rank over SDH (HEARING_IMPAIRED), forced, and commentary (COMMENT), and text over bitmap. No match leaves subtitles off. It drives the host-overlay path, so unlike audio it needs no reload regardless; it only spares a host from language-matching subtitleTracks itself. The native menu (below) keeps its own host-driven default selection via setNativeSubtitleSelected(track:).

Matching is case-insensitive across ISO 639-1, 639-2/B, 639-2/T, and English names (en == eng == english); preference order dominates, so an earlier preference on a later track still wins. The resolved tracks are published on player.activeAudioTrackIndex / player.activeSubtitleTrackIndex (both match TrackInfo.id), and every TrackInfo carries isDefault / isForced / isHearingImpaired / isCommentary (from container dispositions) so a host can rank or filter the track lists the same way.

Second simultaneous subtitle track (bilingual)

A second subtitle channel can run alongside the primary for bilingual playback / language learning: selectSecondarySubtitleTrack(index:) for an embedded track and selectSecondarySidecarSubtitle(url:httpHeaders:) for a sidecar file, mirroring the primary API. Its cues land in a separate @Published secondarySubtitleCues list (so the host can render the two channels independently, e.g. top vs bottom), with isSecondarySubtitleActive and isLoadingSecondarySubtitles for UI state; clearSecondarySubtitle() tears it down. The secondary channel decodes through the same demux loop and PTS rules as the primary.

A single packet that carries multiple rects (PGS often emits signs/songs at the top alongside dialogue at the bottom) becomes multiple cues at the same time range, and the host renders all of them. Cues are inserted in sorted order; re-emitted events after a seek dedupe by time range plus content (so two simultaneous speaker lines with identical timing both survive) and the list doesn’t grow on rewind.

A PGS composition carries no end of its own: it is published with FFmpeg’s open-ended placeholder end (end_display_time = UINT32_MAX) and closed when its successor composition or clear arrives. A seek can outrun that successor, and the retention prune cannot help (it filters on endTime, which a placeholder end never ages out of), so a jump past an open cue used to leave it in the published window covering the new playhead until the next composition displaced it (#357). On a seek the drain now closes every still-open cue that began before its reconstruction window (playhead minus the 15 s backscan) at that window’s start: past that line nothing is re-decoded, so nothing there can be confirmed as still open. The cue stays in the retained list for a backward seek, only its unconfirmed end retires, and an authored duration is never touched. The #100 stale-arrival hold is dropped on the same tick for the same reason.

That close is the last resort, not the rule. The end a set is authored with is the PTS of the next packet on its stream, and the packet store holds that packet long before the drain window reaches it, so every tick closes each still-open cue at SubtitlePacketStore.firstPTS(streamIndex:after:) (#362). Without it the drain window’s forward edge decides the end whenever it falls between a set and its clear: the set publishes open, the cursor moves past it, and the next thing to touch it is a composition at the next landing, tens or hundreds of seconds later (report: 3.55 s authored, 76.7 s delivered, and 817 s in the same session). For the store to be able to answer, the harvest has to lead the decode: the forward prefetcher parks a 15 s margin BEYOND the drain window (subtitleForwardPrefetchLeadMarginSeconds), because with both lines at 60 s the set at the edge is systematically the one whose clear is stored nowhere. Where the store genuinely has nothing after a set (its harvest frontier, a stream cut short) the cue stays open and #357’s boundary close still owns it, since inventing an end there is the laundering this replaced.

The same report has a second face, and it is the store’s own bookkeeping rather than the cue’s. After a seek the pump restarts behind the landing and fills forward while the store still holds an island the previous run harvested further ahead, so the drain window reads as “packets, hole, packets”. Decoding across that hole carries the drain cursor to its far side, and the cursor only moves forward: the hole’s packets land a second later and are never read, so a stretch of the film carries no subtitles at all until some later seek resets the window behind it, and the set before the hole is closed at the island rather than at its own clear (report: eleven authored sets delivered as two). The size of the gap cannot tell that apart from a silence the author left, and a threshold on it is actively harmful, since a set is separated from its own clear by its display duration. Harvest ORDER can: a run reads a stream forwards, so within one run PTS and sequence rise together, and a PTS-ascending pair whose sequence DESCENDS is two runs meeting over a span neither of them has read. A tick stops at that boundary and waits (harvestGapCut, #362), which costs nothing visible because the drain runs its lead ahead of the playhead; the wait ends when the filling run re-harvests across the boundary, when the playhead catches up to it, or after a bounded tick budget, so an authored silence can never stall delivery. A tick that waited says so, as outcome=harvestHole gapAt=.

A hole behind the playhead has a third face, and it is the landing itself (#416). The gate that reconstructs the active line at a seek target seeds it from the newest set decoded behind the playhead, and that set is the active line only if nothing on its stream happened in between. The store’s silence is read as that proof, and over a stretch nobody read it proves nothing: a run re-aimed just after it harvested a set leaves that set’s own clear on the far side of the skipped ground, so it decodes at the landing looking unclosed and publishes over the new scene, ending at the next stored packet, which is the far side of the authored silence rather than its own successor (report: a two-second sound-effect caption standing ten seconds over the wrong scene, on both an Apple TV 4K and a Mac). #362 established that the packets alone cannot show this: a reader re-anchored FORWARD hangs its packets in ascending order behind the stretch it skipped, so the pair looks exactly like an authored silence, and only a reader restarted BEHIND leaves the descending sequence harvestGapCut reads. So the readers now say what they read. SubtitleHarvestCoverage in the packet store keeps one span per run, anchored where the reader positioned and extended as it goes: the forward prefetcher reports its own read position, the pump’s run begins where the producer opens or restarts and reaches at least the playhead, since playback is rendering there. A set whose ground up to the playhead is not covered cannot claim the landing (landingWithheld= on the delivery line), and the pass ends on the next authored set as it would have anyway. A store nobody reports to answers every span with yes, so a path without coverage notes behaves exactly as it did. The cost is the landing line in the case where a set really is still up and the proof is missing, which needs an authored dwell long enough to span the whole unread stretch; the alternative was paying it for every normally authored set that ends inside one.

Subtitle cues land in raw source PTS. On the native path, AVPlayer’s HLS clock sits at source_pts - producer.videoShiftPts (the producer applies a per-session shift to align the first segment’s tfdt with the playlist origin, and the shift can change on every restart). Render the overlay against player.sourceTime so cues match the spoken audio regardless of which producer session is active.

Native subtitle renditions (WebVTT for PiP, AirPlay, and external display)

Host-rendered subtitle overlays are invisible in Picture-in-Picture, AirPlay, and external-display sessions because those paths render the AVPlayerLayer content only; the SwiftUI / UIKit view tree is not composited. The engine therefore serves every text subtitle track as a real HLS SUBTITLES rendition over the loopback: the master playlist carries one language-tagged EXT-X-MEDIA:TYPE=SUBTITLES entry per track (DEFAULT=NO,AUTOSELECT=NO) plus SUBTITLES="subs" on the variant, backed by a per-track media playlist (subs_N.m3u8) whose WebVTT segments mirror the video segments 1:1. AVFoundation exposes the renditions as a standard legible AVMediaSelection group that travels with the stream everywhere AVPlayer goes, including PiP. (An earlier design muxed mov_text/tx3g traks into the fMP4 itself; in-band timed text is not HLS-conformant and AVPlayer rejected the stream, so the WebVTT rendition replaced it.)

Opt-in. Off by default (LoadOptions.prepareNativeSubtitles = false): no renditions in the master, no legible menu, output identical to before.

Cue source: the producer pump tap. The segment producer already reads the source’s full interleave, so the text subtitle streams stay in its keep-set and every packet is handed to a session-level tap that decodes into per-track cue stores (the same pattern as the CEA-608 tap). Zero side-channel bandwidth, and coverage is by construction the produced region, across seeks and producer restarts. The host overlay is fed separately, by the packet-store drainer (#112 rework, see Subtitles above); a lazy per-selection reader still covers AVKit’s ~240 s forward .vtt prefetch beyond the produced region. Load-declared external tracks (#88) have no demuxable stream to tap; their store is filled by one whole-file decode at load and marked finished, so their rendition serves complete .vtt files from the start.

Routing scope. A SUBTITLES rendition can only live in a master playlist, so native subtitles ride the master-routing rules: SDR sources on any panel, HDR / DV sources on HDR-ready panels. HDR-on-SDR-panel and DV Profile 5 on non-DV panels stay media-direct (no master, hence no native subtitles there); the host overlay still covers fullscreen. Bitmap subtitles (PGS / DVB / DVD) join as OCR-fed renditions: while a bitmap track is selected, a worker decodes its harvested packets ahead of the playhead (composition ends resolved at the next composition/clear event, the 5.14.1 sidecar semantics) and recognizes them on-device (Vision, track-language hinted) into plain-text cues for the track’s rendition. Recognition is lossy by design; a failed or empty read drops that line from the rendition while fullscreen keeps the pixel-accurate bitmap overlay. External .sup sidecars fill their store from the selection-time sidecar decode’s own image cues (no second fetch). Two shapes are out of scope for the loopback renditions entirely: live sources, whose own SUBTITLES renditions reach the host through the overlay instead (below), and the software decode path (AV1 without hardware decode, VP9), which has no AVPlayerItem for a legible group to live on, so prepareNativeSubtitles is inert there and the host overlay owns subtitles on every surface.

Wireless AirPlay (#86, #227). While an iOS session plays to a wireless AirPlay receiver, the engine reloads its loopback over the device’s LAN IP: the receiver fetches the stream for itself and cannot reach 127.0.0.1. It keeps the master there, so the SUBTITLES renditions travel and setNativeSubtitleSelected(track:) has a legible group to select against; the EXT-X-MEDIA URIs are relative, so they resolve against the LAN base with no further work.

Whether an HDR or Dolby Vision master is accepted is the receiver’s decision, and it turns on the receiver’s own output mode (measured 2026-07-27, iPhone 17 Pro to an Apple TV 4K, DV P8.1 4K source). With the Apple TV’s video format fixed to 4K Dolby Vision it takes the DV master and plays it with subtitles. With the format at 4K SDR it refuses every HDR master, and Match Dynamic Range does not help: that setting switches only when tvOS decides the content warrants it, which it evidently never does for AirPlay content. This is the same rule the engine already applies locally, where an HDR source on a panel that is not in HDR mode is served the media playlist, and it is what DrHurt described in #86. Dressing the manifest up does not move it either: against a parked receiver, dropping the DV SUPPLEMENTAL-CODECS, clamping the declared BANDWIDTH, omitting RESOLUTION and declaring HDCP-LEVEL=TYPE-1 each changed nothing, and declaring the range as SDR was already disproven in #98.

The refusal is silent, so it has to be caught by watching progress: no -11868, no failed item, the rate flickers to playing for a single tick so even hasEverPlayed latches, and the picture never starts while AVKit shows its “not playable on this display” sign. Five seconds without a segment fetched on a master handed to a receiver reloads the LAN media playlist, which every receiver takes, and that receiver is remembered by route UID for the rest of the process so it goes straight to media from then on. A second master attempt, to exploit the output switch the first one triggers, was tried on device and only doubled the wait. For subtitles on HDR content over AirPlay the answer is the receiver’s setting: fix its video format to HDR or Dolby Vision. $nativeSubtitleRenditionsServed reports which playlist is actually in use, so a host can tell the user their subtitles will not travel to this route rather than dropping them silently. A wired HDMI external display is a different route and keeps the loopback plus its master.

Master-rejection fallback (#98, #130). When AVPlayer rejects the served master (-11868 AVErrorNoCompatibleAlternatesForExternalDisplay, -11848 for an SDR-parked panel, or -1002 when every variant was filtered at master parse time), the engine reloads the bare media playlist in place; a live session rejoins at the edge instead of replaying its stale start position. HDR / DV on an SDR external display is therefore media-playlist-driven (an AVKit limitation: forcing VIDEO-RANGE=SDR does not fool the external-display compatibility gate, which checks the real colr / codec rather than the manifest string), so the SUBTITLES renditions do not travel there. The separate #35 cold-DV-start readiness gate, whose scenario is an HDR TV, first tries an HDR-preserving reduced master (SUPPLEMENTAL-CODECS dropped so it is plain HDR10, source range and SUBTITLES group kept) before the bare media playlist, so a cold DV start keeps HDR10 plus subtitles instead of dropping straight to subtitle-less media.

Rich ASS styling. With LoadOptions.preserveASSMarkup the tap keeps raw ASS event lines so the host overlay renders full styling (positions, colours); the WebVTT renditions strip the markup at serve time, so PiP shows plain text in the system caption style.

Timing. Served cues are on the AVPlayer clock axis, and producer restarts are timeline-exact (see architecture), so cues stay in sync with the picture across seeks and restarts.

Selection: deliberately not automatic. The renditions ship DEFAULT=NO,AUTOSELECT=NO so AVKit never engages one on its own and a host overlay never double-renders in fullscreen. A host that shows AVKit’s stock chrome can still let the user pick from the native legible menu; Sodalite-style hosts select programmatically per surface instead.

Selection: host-driven API. These members on AetherEngine drive the native renditions programmatically:

// true once cues from at least one text track are decoded into the native stores
engine.$nativeSubtitleRenditionAvailable   // @Published var Bool

// true while the served playlist carries the SUBTITLES group; goes false on a
// media-playlist fallback, on the wireless-AirPlay hop for an HDR / DV source, and
// when the receiver refuses the master it was handed (#227). Hosts use it to decide
// whether to draw their own subtitle window on a wired external display instead
// (#98), or to tell the user that subtitles will not travel to this route
engine.$nativeSubtitleRenditionsServed     // @Published var Bool

// ordered list of all native subtitle renditions (ordinal, language tag, display name)
engine.$nativeSubtitleTracks               // @Published var [NativeSubtitleTrack]

// select a rendition by ordinal (nil deselects); language-tag match, positional fallback.
// Re-asserts automatically if AVFoundation drops the selection during a stall recovery.
engine.setNativeSubtitleSelected(track ordinal: Int?)

// convenience for the enter/leave pattern below: true resolves the rendition matching the
// currently-active overlay track and selects it, false deselects
engine.setNativeSubtitleRendering(_ active: Bool)

NativeSubtitleTrack carries .ordinal (position in the rendition declaration), .language (ISO 639-2 tag), and .displayName (localized name suitable for a picker label).

The recommended host pattern for PiP / AirPlay:

Observe $nativeSubtitleRenditionAvailable (waits for the first cues to be ready before activating).
On entering PiP / AirPlay / external display: call setNativeSubtitleRendering(true) (or resolve the ordinal yourself via setNativeSubtitleSelected(track:)), and hide the host overlay.
On leaving: call setNativeSubtitleRendering(false) and re-enable the host overlay.

This avoids double subtitles during inline playback (where the host overlay is already painting them) and ensures the user sees subtitles the moment the stream is mirrored or sent to PiP.

Device-verification checklist (required before tagging a release):

Selecting a rendition displays it in the PiP window and survives seeks (including seeks that restart the producer).
Inline host ASS rendering unchanged: rich styling intact, tap-fed cues appear instantly on selection.
No double subtitles while inline; no rendition is auto-selected on session start.
Timing: no constant offset between audio and subtitle cues, before and after seeks.
SDR / HDR10 picture behavior unchanged with prepareNativeSubtitles = true (the renditions only add master tags + subtitle endpoints).
HDR-on-SDR-panel and DV Profile 5 on non-DV panels still play (media-direct, no renditions there by design).
Memory bounded by total cue count across all tracks.

Live HLS subtitle renditions (host overlay)

A live channel carries its subtitles the same way a VOD stream does, as an EXT-X-MEDIA:TYPE=SUBTITLES group in the upstream master, and the live ingest demuxes the picked video variant only, so that group never reaches the demuxer: before #359 a channel offering three subtitle languages produced no subtitle track at all. The engine now surfaces the master’s renditions as tracks (synthetic ids from liveSubtitleRenditionTrackIDBase, 300000) and, when the host selects one, fetches its WebVTT segments and publishes the cues on the same host-overlay surface the CEA-608 tap uses. This is the overlay path, not a legible AVMediaSelection: the live loopback serves a media playlist, which cannot declare a rendition (see the routing scope above).

It is deliberately lazy. The tracks come from the master’s declaration alone and nothing is fetched until a track is selected, so a channel watched without subtitles pays no second HTTP loop. Selection starts a poll of the rendition playlist at its EXT-X-TARGETDURATION, and only unseen segment URIs are fetched, so a poll that arrives before the window moved costs one small request.

Three properties are worth knowing when reading [LiveSubs] log lines:

Placement comes from the playlist geometry, not from X-TIMESTAMP-MAP. The spec’s own anchor is unusable in the field: measured against a public broadcaster, every segment carries the same constant map whose MPEGTS value sits two hours away from the video PTS. What renditions of one program do share, byte for byte, is EXT-X-MEDIA-SEQUENCE and EXT-X-PROGRAM-DATE-TIME, so a cue is placed at its segment’s broadcast time plus its offset inside that segment. An upstream without EXT-X-PROGRAM-DATE-TIME therefore has no placement at all: the loop says so in the log and stops rather than publishing something plausible and wrong.
The fetch is anchored at the playhead, not at the head of the playlist. A rendition playlist is not a handful of segments (a two hour DVR window is 3600 entries), so everything older than 120 s behind the playhead is marked seen without being fetched. Enabling subtitles and jumping back a little still lands in covered ground.
A producer seam republishes the source-axis shift, and cues already placed then refer to an axis that no longer exists. The loop watches its own pairing and re-anchors, which is what keeps a long session from drifting further out the longer it runs (6.21.1).

aetherctl play --live-ingest --subs <lang> exercises this against a real channel; it is the only CLI route that does, and its absence is why #359 stayed invisible for as long as it did.

Authored ASS styling

Hosts that render authored ASS styling themselves (positioning, speaker colours, karaoke) opt out of the stripping with LoadOptions(preserveASSMarkup: true): cues then carry the raw event line (override tags, style references, escapes intact), the script header ([Script Info] + [V4+ Styles]) is surfaced, and engine.fontAttachments carries the container’s embedded fonts (TTF / OTF) for the renderer’s font directory. ASSScriptBuilder reassembles raw event cues + header into a complete script for whole-file renderers such as swift-ass-renderer’s loadTrack(content:), hardened against real-world Matroska tracks (synthesized [Events] section, NUL stripping, content-keyed dedupe since real files hardcode ReadOrder: 0).

The header arrives differently per source: embedded tracks carry it on TrackInfo.assHeader, and external .ass / .ssa sidecars loaded through selectSidecarSubtitle(url:) under the same preserveASSMarkup flag carry it on engine.sidecarASSHeader (extracted from the file’s subtitle-stream extradata; nil for SRT / VTT and when preservation is off). Both pair with the raw event-line cues the same way (AetherEngine#48).

The host stays in charge of the actual paint: text styling, overlay layout, fade transitions, position scaling against the on-screen video rect.

Frame extraction

FrameExtractor produces still CGImages from a media URL through an FFmpeg decode context that is fully isolated from playback. It never touches the playback pipeline, the HLS loopback server, or the engine’s shared state, so a scrub-preview decode can’t perturb the frame on screen. Two modes share one decode core:

thumbnail(at:maxWidth:): seeks to the nearest keyframe, no forward decode, downscaled to maxWidth (default 320). Cheap and fast; built for scrub previews and Recents lists.
snapshot(at:maxSize:): decodes forward to the exact PTS, full or maxSize-clamped resolution. Built for user-triggered stills.

let frames = engine.makeFrameExtractor()           // nil if nothing is loaded
// or, for an arbitrary item (e.g. a Recents row):
let frames = FrameExtractor(url: url, httpHeaders: headers)

await frames.prewarm()                             // optional: hide cold-start at gesture begin
let preview = await frames.thumbnail(at: 612.0)    // CGImage?, nearest keyframe
let still   = await frames.snapshot(at: 612.0)     // CGImage?, frame-accurate
await frames.shutdown()                            // prompt teardown of the decode context

HDR sources come out looking right: PQ / HLG BT.2020 frames are tone-mapped to SDR BT.709 through a zscale + tonemap libavfilter graph before the CGImage is built, so HDR10 / HLG / DV P8.x stills match what the user sees instead of washed-out grey. Dolby Vision Profile 5 and AV1 Profile 10.0 (IPT-PQ base layers with no HDR10 fallback) route through DolbyVisionStillConverter, which applies the RPU colour transform (ycc_to_rgb + PQ EOTF + the IPT-PQ LMS matrices carried in AV_FRAME_DATA_DOVI_METADATA) before tone-mapping, so their stills come out with correct colour instead of the green / magenta cast a plain YCbCr read produces.

FrameExtractor is an actor. Blocking FFmpeg work runs on a dedicated serial queue, never on the cooperative thread pool. The decode context opens lazily on first use; a superseded request (the common case during an active scrub) cancels the in-flight decode so the latest position wins. Results land in a bounded LRU cache (snapshots and thumbnails kept in separate stores, thumbnails bucketed by second). After 10 s idle the context closes and the cache drops automatically; the next request reopens lazily. shutdown() is the explicit, permanent teardown. The engine does not retain the extractor returned by makeFrameExtractor(); the caller owns its lifecycle.

Disc (DVD / Blu-ray ISO)

Decrypted disc images play through the normal decode path via a synthetic seekable byte source. DiscReader detects and routes both local .iso URLs and MediaSource.custom ISO readers.

DVD-Video ISO: ISO9660Reader reads the ISO9660 bridge filesystem, DVDIFOParser reads the VMGI (VIDEO_TS.IFO) TT_SRPT to enumerate the disc’s titles and each title set’s VTS IFO (VTS_NN_0.IFO) program chain for the title duration and chapters, DVDTitleSelector groups each title set’s content VOBs (whole-VTS, largest first), and ConcatIOReader presents the selected title’s concatenated VOBs as one seekable source demuxed as MPEG-PS. On an unreadable VMGI it falls back to the VOB-size grouping; an unreadable VTS IFO leaves the title’s duration and chapters empty but still plays.
Blu-ray ISO: a read-only UDFReader (UDF 2.50, including the metadata partition and fragmented-file allocation descriptors) resolves BDMV, MPLSParser + BDTitleSelector enumerate every .mpls playlist as a selectable title (longest first so id 0 is the main feature; trivially short menu / FBI-warning playlists filtered), and the selected title’s .m2ts clips are concatenated and demuxed as MPEG-TS (H.264 / HEVC / VC-1, AC3 / EAC3 / DTS / TrueHD / LPCM, PGS subtitles).

Both: no decryption (CSS / AACS retail discs must be ripped decrypted first), no GPL nav libraries, no menus, BD-J, or multi-angle.

Title selection. engine.discTitles (@Published [TitleInfo]) lists the disc’s titles (id, name, duration, chapter count) and engine.selectedDiscTitle is the active one; engine.selectTitle(id:) switches title, rebuilding the pipeline from the new title’s head. The selection survives audio-track switches and background-resume reloads, and a fresh load defaults to the main title (an out-of-range id clamps to it). Blu-ray enumerates all playlists; a DVD enumerates its title sets (the VMGI TT_SRPT title list, resolved whole-VTS, with the duration read from each VTS’s main program chain; per-cell / episodic splitting is deferred).

Chapters. engine.discChapters (@Published [ChapterInfo]) carries the selected title’s chapters; engine.selectChapter(id:) seeks to one (a thin seek wrapper, no pipeline rebuild). For Blu-ray they come from the playlist’s PlayListMark entries (entry marks only; link points dropped), each mark’s timestamp on its clip’s STC offset by the clip’s in_time and the cumulative duration of preceding play items. For DVD they come from the main program chain’s program map plus the cumulative cell playback times. Chapter starts are title-relative (0-based); selectChapter adds the title’s content-start base (the native playlist shift, or the software path’s container start PTS) so the seek lands on the source-PTS playback axis.

Track languages. Neither disc format carries a track language in the stream, so a title demuxed on its own reports every audio and subtitle track as undetermined and preferredAudioLanguages / preferredSubtitleLanguages have nothing to match on. The languages are read out of the disc’s own navigation data instead and backfilled onto the tracks by stream id: on Blu-ray from every PlayItem’s STN table (the ISO 639-2 codes beside each stream’s PID, first declaration winning), on DVD from the VTS IFO audio and subpicture attribute tables, with the title’s main program chain naming the substream each attribute is actually carried as (a stream the chain marks absent is dropped, and without a readable chain the attribute’s position is used, which is how the great majority of discs are authored). Only an undetermined track is filled in: a language the container really declares stays authoritative. aetherctl disc-inspect prints what a disc declares per title, so a disc whose tracks stay undetermined can be told apart from a disc that declares nothing (#527).

Container chapters. engine.mediaChapters (@Published [ChapterInfo]) carries the chapters a Matroska or MP4 container declares, read off the probe demuxer at load. It is empty for disc sources, which publish discChapters instead, so exactly one of the two is populated. Unlike disc chapters these need no base: a non-disc source plays on the container’s own PTS axis on both backends, so startSeconds is a timestamp a host hands straight to seek(to:). selectChapter(id:) resolves against discChapters only and no-ops for a container chapter id. Ids are assigned sequentially in start order, so they stay usable as list indices, and untitled entries are numbered “Chapter N”. A chapter’s duration runs to the next chapter’s start rather than to its declared end, because muxers routinely write end == start; the last entry falls back to its declared end, then to the container duration.

Custom byte sources (`IOReader`)

MediaSource.custom(reader, formatHint:) hands the engine a host-owned byte source: a memory buffer, an encrypted container, a tuner spool, anything that is not a plain URL. read, seek and close are required; cancel(), makeIndependentReader() and discImageProbeEnabled have defaults. Every call arrives on the engine’s demux thread, never main, and inside an autorelease pool the engine opens, so a reader built on FileHandle or NSData does not strand one autoreleased object per read on a pump thread that runs for the length of a session (#445).

cancel() unblocks, it does not invalidate. Its whole job is to wake a read that is parked so teardown does not hang; a memory or file reader leaves it at the default no-op. A reader the engine may rebuild in place must be able to serve again immediately afterwards, because the rebuild reuses it. Reading cancel() as terminal (the way a socket’s in-flight-request cancel is terminal) is the difference between a rebuilt session and a dead one: the reopen fails on its first read and the session lands in .error carrying libavformat’s Operation not permitted. What that death LOOKS like is the reader’s own convention, so one host defect has two signatures: a latched reader whose read returns a negative value produces the Operation not permitted above, while one that returns 0 is mapped to AVERROR_EOF and the session ENDS instead of failing, which on a live source is a stream that was never supposed to have an end (cmcpherson274). The engine cancels a reader it intends to reuse exactly once, before the successor opens.

In-place rebuilds reuse the reader. reloadAtCurrentPosition() and reloadAtCurrentPosition(applying:), an audio-track switch, a disc-title switch and a background return all tear the pipeline down and reopen on the retained reader instead of reopening a URL, so a custom source gets the same session-preserving rebuild a URL source does. A reader that reports itself non-seekable is refused instead (SessionReloadRefusal.customSourceNotSeekable), since the rebuild cannot reposition it.

Where the reopen starts depends on whether the source is live. A fresh AVIOContext always starts its byte axis at 0, so the reader’s cursor and that axis have to agree, and there are two ways to make them:

VOD. The reader is rewound to 0 and the axis starts there. The reopen has to re-read the container header, and the backend seeks to the resume position afterwards.
Live. The reader is left exactly where the session left it and the axis is moved to the cursor instead ([Demuxer] live reopen aligned to the reader's cursor at N bytes). A live source has kept receiving during the rebuild, so rewinding it would replay the host’s whole delivered window: measured on aetherctl customio --live, the rewind dropped the playhead from 41.5 s to 1.9 s and asked the host to re-deliver 15 MB, a 61 s window, at I/O speed. Aligning instead makes the rebuild the edge rejoin LiveReloadPolicy already performs on the URL branch.

This costs one thing worth knowing: the reopen re-probes the container from a mid-stream byte offset, so it needs a source that can be joined there. MPEG-TS resyncs on its own sync bytes and recovers within a GOP. A live reader that will not answer seek(0, SEEK_CUR) cannot be aligned to, so it is rewound as a VOD source would be, and the engine says so rather than doing it silently.

The axis a reader reports on is the axis it is asked on. The alignment above reads the cursor with seek(0, SEEK_CUR) and hands that number straight back as a SEEK_SET; the seekability probe makes the same round trip at every open; libavformat’s own probe seeks then travel the same axis. Absolute file offsets and offsets counted from wherever this install’s stream joined are both fine, and a reader counting from its join is reported in that axis (aligned to the reader's cursor at N bytes carries bytes-since-join, not a position in a file). What breaks is reporting on one axis and taking SEEK_SET on the other: the round trip then MOVES the source by the join offset, on a probe whose whole purpose is that it moves nothing, and the session reads from a stretch the host has not delivered yet. Both halves are checked. The probe compares its own return against the position it was just told, which costs no extra callback and fires on every open including every in-place rebuild, and the alignment asks once more where the reader is, so a live reopen never reports an alignment that did not hold:

[Demuxer] custom source: the reader reported byte 2097152 and answered a seek back to it with
4194304, so its position report and its SEEK_SET argument are not on the same axis; the source
has been repositioned by the seekability probe

A host whose own position arithmetic is keyed to vending a stream has to re-key it before opting into in-place rebuilds (cmcpherson274’s corollary, and the reason his wrapper still refuses one). A rebuild consumes no new stream, so anything a host initialises at the point it hands one over, a live clock baseline, a DVR join anchor, the per-stream ownership that makes an unguarded cursor safe, is not re-initialised by a rebuild and will disagree with the session that comes back. The engine’s alignment fixes the bytes; it cannot fix a clock it cannot see.

Network sources (SMB)

The optional AetherEngineSMB product plays media off an SMB2/3 share through the normal decode path, no server-side mount. SMBConnection (backed by SMBClient, MIT, a pure-Swift SMB2 client over NWConnection) is a ByteRangeSource; SMBIOReader adapts it to the engine’s IOReader, bridging each synchronous demux-thread read to SMBClient’s async API across a happens-before semaphore edge. The reader is seekable, so audio-track switching, background reload, embedded subtitles, and scrub previews all work (makeIndependentReader() opens a second cursor over the same connection).

Read-only. NTLMv2 and guest auth (no Kerberos, which tvOS lacks). No writing, locking, or directory browsing. SMBClient negotiates only SMB 2.0.2 and 2.1, so there is no SMB3 transport encryption or AES-CMAC signing; a server configured SMB3-only or with smb encrypt = required won’t connect. The connection is persistent (SMBClient plus a FileReader) and reads are serialised per connection, which clears typical media bitrates comfortably. The dependency is linked only by consumers of the AetherEngineSMB product, so the core engine and its tvOS hosts never pull it. SMBClient replaced AMSMB2/libsmb2, which EPERMs on tvOS / iOS. On tvOS the host supplies the local-network entitlement (and NSLocalNetworkUsageDescription) to reach a LAN share.

Live ingest, AES-128, SSAI

A live HLS upstream can be ingested directly via HLSLiveIngestReader (a public forward-only IOReader), no media server in the data path. Segments are fetched through a bounded prefetch pipeline (up to 4 in flight, committed to the byte stream strictly in playlist order), so per-segment connection + TTFB latency overlaps and high-bitrate channels buffer ahead of real time (#177). Contract: MPEG-TS segments, including demuxed-audio variants (EXT-X-MEDIA audio groups, fetched by a companion reader and merged by DTS) and packed-audio renditions (raw ADTS framed by ID3 timestamps). AES-128 clear-key segments (EXT-X-KEY:METHOD=AES-128, the standard FAST-channel scheme) are decrypted in-line by HLSSegmentDecryptor: the key is fetched once per clip and memoised, each segment decrypted (AES-128-CBC / PKCS7) before demux. SAMPLE-AES / keyless AES-128 (no URI), fMP4 playlists (EXT-X-MAP), and a key-fetch / decrypt failure terminate with a typed HLSIngestError so the host can fall back to a server-mediated URL. This is standard HLS clear-key, not FairPlay / Widevine.

Server-side ad insertion (SSAI) plays through the direct path instead of bouncing to a server transcode at the ad break. FAST channels (Pluto and similar) splice ad creatives that restart the source clock and often carry a different video PID, resolution, and SPS than the program. The producer detects the program switch, parses the ad’s SPS/PPS by hand (H264SPS) to build a fresh codec config, rotates the fMP4 muxer, and emits a versioned #EXT-X-MAP per discontinuity so AVPlayer resyncs cleanly across the init and resolution change; audio is re-anchored to the video timeline at every creative boundary (including amux creatives that mux audio on a separate source clock) and an OutputTimestampSanitizer keeps the stream monotonic across the splice. A no-cut stall watchdog sits underneath as a safety net: it tells a genuinely wedged pod (reading at full rate but unable to cut) from a slow source (a trickle) by read rate, and a wedge-classified stall whose video PTS is still advancing (a source delivering just below real time, #177) is held with the watchdog re-armed (bounded consecutive holds) instead of retuned; only a genuine wedge or an exhausted hold budget escalates to a host retune.

The live path’s sliding-window eviction (which bounds resident memory) and DVR rewind are confirmed on Apple TV against a real broadcast feed: behindLiveSeconds holds at real-time pacing and the resident footprint stays bounded within the tvOS jetsam budget. The same behavior is exercised off-device through the aetherctl live / hlsfixture harnesses (sliding-window retention, real-time pacing, mid-stream reconnect, program-boundary discontinuities, DVR timeshift).

Raw live MPEG-TS over HTTP (a tuner or tuner proxy serving the transport stream directly, no HLS) plays on the software path; the forward-only reader routes it there automatically. Load it with isLive: true plus a dvrWindowSeconds for live semantics and the fastest open (an explicitly live load skips the open-time size probes entirely); SourceProbe.isLive flags no-duration network streams so hosts can detect and reload. Mid-stream-joined sources, which deliver their first samples at arbitrary PTS hours past zero, anchor the clock at the first decoded sample on every session shape (live, live+DVR, and a plain VOD open of the same URL or of a mid-broadcast capture file), publishing session-relative positions while sourceTime stays on the source axis for subtitles (#107).

Known limitations

Things that work today but have a documented edge case, or are deferred behind an upstream dependency:

TrueHD-MAT Atmos object metadata is not preserved. TrueHD / MLP sources route through the AudioBridge (FFmpeg’s EAC3 encoder doesn’t produce JOC). Bed channels and surround layout survive; object metadata is dropped. EAC3+JOC stream-copy from MKV / MP4 sources is intact.
.surroundCompat audio bridge caps 7.1 sources to 5.1. FFmpeg’s EAC3 encoder currently caps at 6 channels. Once FFmpeg PR 21668 lands the cap and the dynamic bitrate auto-scale to 1024 kbps engage without a code change here. Use .lossless (FLAC) today if 7.1 matters.
Manual MPNowPlayingInfoCenter writes race the HLS-loopback path on tvOS 26. The combination produces a libdispatch race. AVPlayerViewController with its standard transport bar surfaces Now Playing on its own, through AVKit’s private MediaRemote registration, and needs nothing from the host. A host with a custom transport instead sets engine.ownsVideoNowPlayingSession = true before load(): the native host then owns an MPNowPlayingSession bound to its player (published as videoNowPlayingSession), registers transport commands on that session’s remoteCommandCenter, and stages identity metadata through setVideoNowPlayingInfo(_:), which is replayed onto every fresh AVPlayerItem. The session auto-publishes elapsed time, rate and duration from the player, so nothing writes MPNowPlayingInfoCenter.default().nowPlayingInfo. The flag is off by default and must stay off under AVKit: two owners produce the half-working state where an empty identity card displaces AVKit’s and remote commands route into a session with no handlers.
Audio session is activated per playback, not at process launch. The engine declares the AVAudioSession category (.playback / .moviePlayback) and multichannel support at init, but does NOT activate the session there. The route-sharing policy is platform-split (#116): tvOS declares .longFormAudio, iOS declares .default, because .longFormAudio marks the process as a long-form audio client and pins AVPictureInPictureController.isPictureInPicturePossible to false for any host-built PiP controller around the engine’s player layer. Hosts do not need to re-declare the session for PiP. Activating once at launch used to pin the route to whatever the HDMI link reported at that instant: with tvOS Continuous Audio Connection off the link idles at stereo, so the launch-time activation negotiated the route to 2 channels and pinned it, downmixing non-Atmos multichannel for the whole session (AetherEngine#24). The native video path now lets the host’s AVPlayerViewController activate the session per playback; the renderer paths (software decode, audio-only) activate it themselves, off the main actor and before their host is built (AE#538). Hosts that mount the engine’s bare AVPlayerLayer instead of an AVPlayerViewController should ensure the session is active at playback. A genuine sink-side ch=2 (an AVR caching its HDMI EDID incorrectly after standby) can still force a downmix; power-cycling the sink restores it. Atmos passthrough is unaffected either way because EAC3+JOC ships as MAT 2.0 over a 2-channel carrier.
AV1 on Apple TV is software-decoded. No current Apple TV chip ships HW AV1. The SoftwarePlaybackHost + dav1d path handles it, but CPU use is meaningfully higher than HW HEVC. On iOS 17+ / macOS 14+ AV1 routes through Apple’s HW pipeline transparently. Future Apple TV chips with HW AV1 will be picked up automatically by VTCapabilityProbe.
AV1 Dolby Vision Profile 10.0 has wrong colours when software-decoded. dav1d / libavcodec cannot decode the proprietary DV colour space, so a Profile 10.0 source (DV-only, no fallback base layer) renders with incorrect colours on the SW path. Profiles 10.1 and 10.4 are unaffected because they carry an HDR10 / HLG base layer. Profile 10.0 only renders correctly through the native AVPlayer path on hosts with HW AV1 decode.
Dolby Vision Profile 5 / AV1 Profile 10.0 thumbnails skip the RPU reshaping curves. FrameExtractor now applies the DV colour transform (from AV_FRAME_DATA_DOVI_METADATA) so P5 / P10.0 stills come out with correct colour, validated against a libplacebo render. The per-frame reshaping polynomials are intentionally not applied: they are not what causes the visible green / magenta corruption, and skipping them keeps this a lightweight CPU pass rather than a full Dolby Vision compositor. Brightness / contrast can therefore differ marginally from a fully graded DV render. A frame that carries no DV metadata falls back to the standard path.