Table of Contents
NOTE: These methods have been tested on iOS Safari versions 14+ (up to 17.4.1).
Looking for an NPM package? Check out the generate-video-dumbnail!
Thumbnail generation in iOS did not work as expected
I recently worked on generating video thumbnails. It wasnโt too difficult. I quickly put together the code with a bit of inference, ChatGPT, and Googling. The thumbnail generation code worked well in macOS Safari and Chrome. I thought I could finish the task earlier than expected. However, when I checked it in iOS Safariโฆ NOTHING APPEARED! Something was wrong. ๐คฏ
Browser Specific Failures Graph (Source)
Have you ever faced a similar situation? iOS Safari is sleek and sophisticated, but it can be a pain for web frontend developers. In this article, Iโll explain how to generate video thumbnails in iOS Safari and Why itโs necessary. I also provide online demos for you to test.
I donโt know how long the thumbnail issue will last, but at least for now, this guide will help you create them as intended. I hope this content is helpful.
Before you begin
This article is in no particular order! Instead of reading from the top to the bottom, you can jump right in and read what you need. The titles without the magnifying glass (๐) stand for each situation. The content shows the solution (code) first, followed by a brief reason for it. The reasons are preceded by a magnifying glass (๐), which you can click to see a more detailed explanation.
Display Video Poster
Permalink to โDisplay Video Posterโw/ Static Video Element
Permalink to โw/ Static Video Elementโ<video
src="https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001"
preload="metadata"
></video>
- ๐ Append
#t=0.001
to the end of thesrc
attribute to show the first frame of the video. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - Video can be delivered via HTTP or HTTPS.
๐ Media Fragments URI
Permalink to โ๐ Media Fragments URIโiOS Safari does not display the first frame of a video element by default! Even setting the preload="metadata"
or preload="auto"
attribute does not work. Instead, you must use a Media Fragments URI. (Can I Use)
By using one of the Media Fragments, #t=<time>
, you can specify a particular time in the video (in seconds). In iOS Safari, you can use this approach to show video frame(poster). For example, if you specify 0.001 seconds(#t=0.001
), it will show the first frame. Note that setting it to #t=0
does not load the frame.
๐ Preload Attribute
Permalink to โ๐ Preload AttributeโWhile loading metadata, some video frame data can be retrieved. Therefore, the preload
attribute can be set to 'metadata'
instead of 'auto'
. Note that the default value of preload
in iOS Safari is 'auto'
, but it may vary depending on the device state(for example, in battery saving mode). For this reason, it is recommended to set it explicitly.
w/ Dynamic Video Element (HTTP URL)
Permalink to โw/ Dynamic Video Element (HTTP URL)โconst video = document.createElement('video');
document.body.appendChild(video);
video.preload = 'metadata';
video.src =
'https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001';
- ๐ Append
#t=0.001
to the end of thesrc
attribute to show the first frame of the video. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - Video data must be delivered via HTTPS. For security reasons, the
src
attribute of dynamically created video elements must always use HTTPS.
w/ Dynamic Video Element (Blob URL)
Permalink to โw/ Dynamic Video Element (Blob URL)โconst input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';
document.body.appendChild(input);
input.addEventListener('change', ({ target }) => {
const file = target.files[0];
const url = URL.createObjectURL(file);
const video = document.createElement('video');
document.body.appendChild(video);
video.src = url;
video.preload = 'metadata';
video.currentTime = 0.001;
});
- ๐ When using Blob URL, the
preload
must be set to'metadata'
.'auto'
does not work. - ๐ Media Fragments URI(
#t=<time>
) cannot be used with Blob URL. Use the'loadedmetadata'
event to setcurrentTime
to0.001
.
๐ Set preload='metadata'
for Blob URL
Permalink to โ๐ Set preload='metadata' for Blob URLโ Comparison of preload = 'metadata'
vs. preload = 'auto'
network requests
When creating a video element using Blob URL, setting preload
to 'auto'
does not retrieve frame data at the specified time. In fact, checking the network requests(image above), you can see that preload="metadata"
retrieves data (Bytes) in the range 48-1541 to display the first frame. However, preload="auto"
does not retrieve data.
Unfortunately, I could not find a technical reason for these differences. I used this code and tested it on iOS Safari 17.4.1.
๐ Media Fragments URI cannot be used with Blob URL
Permalink to โ๐ Media Fragments URI cannot be used with Blob URLโIn the current latest WebKit (Safari 17.4.1), Media Fragments URI (e.g., #t=<time>
) cannot be used with Blob URL. The relevant changes can be found in this commit[1]. Below is part of the commit:
BlobData* BlobRegistryImpl::getBlobDataFromURL(const URL& url, const std::optional<SecurityOriginData>& topOrigin) const
{
ASSERT(isMainThread());
- if (url.hasFragmentIdentifier())
- return m_blobs.get<StringViewHashTranslator>(url.viewWithoutFragmentIdentifier());
- return m_blobs.get(url.string());
+ auto urlKey = url.stringWithoutFragmentIdentifier();
+ auto* blobData = m_blobs.get(urlKey);
+ if (m_allowedBlobURLTopOrigins && topOrigin && topOrigin != m_allowedBlobURLTopOrigins->get(urlKey)) {
+ RELEASE_LOG_ERROR(Network, "BlobRegistryImpl::getBlobDataFromURL: (%p) Requested blob URL with incorrect top origin.", this);
+ return nullptr;
+ }
+ return blobData;
}
The URL::hasFragmentIdentifier
function, which checks for the use of Media Fragment URI, has been removed. Instead, the URL::stringWithoutFragmentIdentifier
function, which returns a string without the Media Fragment URI, is used. Therefore, you need to use currentTime
instead of the Media Fragment URI.
From the code history, it appears that Media Fragment URI was previously supported for Blob URL. This can be confirmed in the Safari Technology Preview 118 release notes and the related commit.
Generate Single Thumbnail
Permalink to โGenerate Single Thumbnailโw/ HTTP URL
Permalink to โw/ HTTP URLโconst THUMBNAIL_POSITION = 5.3;
const video = document.createElement('video');
video.crossOrigin = 'anonymous';
video.preload = 'metadata';
video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${THUMBNAIL_POSITION}`;
video.addEventListener('seeked', () => {
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => {
const img = document.createElement('img');
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
});
});
- ๐
crossOrigin
attribute must be set to'anonymous'
. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - ๐ Video frame data is available after the
'seeked'
or'timeupdate'
event.
๐ Set crossOrigin='anonymous'
to Avoid SecurityError with Canvas Element
Permalink to โ๐ Set crossOrigin='anonymous' to Avoid SecurityError with Canvas Elementโ Browsers, by default, only allow resources from the "same origin" (SOP, Same Origin Policy). If the origin is different, you need to request access through CORS (Cross-Origin Resource Sharing).
When requesting media resources, you can set the crossOrigin
attribute to determine how to handle CORS policy. Of course, you can request media resources that exist in different origins without setting the crossOrigin
attribute. However, the browser consider the integrity is not guaranteed. And when you draw this on a Canvas, it becomes a Tainted Canvas[2].
A tainted canvas means that the website is not allowed to share resources. In this state, pixel data cannot be accessed. Otherwise, unauthorized origins could arbitrarily leak pixel data within the image.
Canvas pixel data can be accessed with getImageData
, toDataURL
, toBlob
, etc. To prevent the security issues mentioned above, these methods throw a SecurityError
for a tainted canvas. Thatโs why you need to set crossOrigin
.
There are two options available. 'anonymous'
(or an empty string ''
), which validates CORS, but does not include any credentials (cookies, HTTP Authentication headers, SSL certificates, etc.) in the request. This helps to avoid situations where sensitive user information exposed. Therefore, I used 'anonymous'
here. However, you can include credentials via the use-credential
option if needed.
๐ Video frame data is available after the 'seeked'
or 'timeupdate'
event (HTTP URL)
Permalink to โ๐ Video frame data is available after the 'seeked' or 'timeupdate' event (HTTP URL)โ When requesting video data using an HTTP URL, frame data is available after the 'seeked'
or 'timeupdate'
event. (โ ๏ธ Note that Blob URL behave differently.)
w/ Blob URL
Permalink to โw/ Blob URLโconst THUMBNAIL_POSITION = 5.3;
const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';
document.body.appendChild(input);
input.addEventListener('change', async ({ target }) => {
const file = target.files[0];
const src = URL.createObjectURL(file);
const video = document.createElement('video');
video.src = src;
video.preload = 'metadata';
video.addEventListener('seeked', async () => {
await new Promise(resolve => setTimeout(resolve, 100));
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => {
const img = document.createElement('img');
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
});
});
video.addEventListener(
'loadedmetadata',
() => (video.currentTime = THUMBNAIL_POSITION),
);
});
- ๐ When using Blob URL, the
preload
must be set to'metadata'
.'auto'
does not work. - ๐ Media Fragments URI(
#t=<time>
) cannot be used with Blob URL. Use the'loadedmetadata'
event to setcurrentTime
to0.001
. - ๐ Video frame data is available a certain time after the
'seeked'
or'timeupdate'
event.
๐ Video frame data is available a certain time after the 'seeked'
or 'timeupdate'
event (Blob URL)
Permalink to โ๐ Video frame data is available a certain time after the 'seeked' or 'timeupdate' event (Blob URL)โ Blob URL also support the 'seeked'
and 'timeupdate'
events. However, if you are using the video frame data, you will need to wait a certain time. In my experience, Iโve been able to access it after 80-100ms.
Unlike HTTP URL, Blob URL may not have frame data immediately after the event is triggered (even when readyState
is 4
!). This is likely because Blob URL reference local files[3]. Delays can occur as video data is decoded and loaded into memory.
Generate Multiple Thumbnails
Permalink to โGenerate Multiple Thumbnailsโw/ HTTP URL
Permalink to โw/ HTTP URLโconst THUMBNAIL_POSITION_LIST = [
0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
];
const main = async () => {
for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
const correctedThumbnailPosition = thumbnailPosition || 0.001;
const video = document.createElement('video');
video.crossOrigin = 'anonymous';
video.preload = 'metadata';
video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${correctedThumbnailPosition}`;
generateThumbnail(video, thumbnailPosition);
}
};
const generateThumbnail = (video, thumbnailPosition) => {
const img = document.createElement('img');
document.body.appendChild(img);
video.addEventListener('seeked', () => {
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => (img.src = URL.createObjectURL(blob)));
});
video.currentTime = thumbnailPosition;
};
main();
- ๐
crossOrigin
attribute must be set to'anonymous'
. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - ๐ Use
#t=${time}
to specify the thumbnail position. - ๐ Video frame data is available after the
'seeked'
or'timeupdate'
event.
w/ Blob URL
Permalink to โw/ Blob URLโconst input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';
document.body.appendChild(input);
input.addEventListener('change', async ({ target }) => {
const file = target.files[0];
const src = URL.createObjectURL(file);
const THUMBNAIL_POSITION_LIST = [
0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
];
for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
const video = document.createElement('video');
video.preload = 'metadata';
video.src = src;
await generateThumbnail(video, thumbnailPosition);
}
});
const generateThumbnail = (video, thumbnailPosition) =>
new Promise(resolve => {
video.addEventListener('seeked', async () => {
await new Promise(resolve => setTimeout(resolve, 100));
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => {
const img = document.createElement('img');
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
resolve();
});
});
video.addEventListener('loadedmetadata', () =>
setTimeout(() => (video.currentTime = thumbnailPosition), 1),
);
});
- ๐ When using Blob URL, the
preload
must be set to'metadata'
.'auto'
does not work. <<<<<<< Updated upstream -
๐ To prevent thread blocking, use
Permalink to โ๐ To prevent thread blocking, use Promise to generate thumbnails sequentially.โPromise
to generate thumbnails sequentially. - ๐ To prevent thread blocking, use
Promise
to generate thumbnails sequentially.Stashed changes
- ๐ Video frame data is available a certain time after the
'seeked'
or'timeupdate'
event.
๐ Incorrect thumbnails may be generated due to thread blocking
Permalink to โ๐ Incorrect thumbnails may be generated due to thread blockingโThread blocking can occur when generating multiple thumbnails. If thread blocking occurs, thumbnails may not be generated correctly. In other words, transparent thumbnails are generated.
Attempting to generate thumbnails all at once (Timeline Raw Data)
The above image shows the result of attempting to generate all thumbnails at once. Frequent network requests are occurring, with a concentration in a specific range. Additionally, the main thread is being used continuously for an extended period. As a result, thread blocking occurs, and transparent thumbnails are generated.
To avoid this, I used Promise
to generate thumbnails sequentially. The generateThumbnail
function returns a Promise
, and I used await
to wait for each thumbnail to be generated.
Generating thumbnails sequentially (Timeline Raw Data)
Generating thumbnails sequentially ensures that network requests are evenly distributed, and the main thread is used only for a short period. This prevents thread blocking and ensures that thumbnails are generated correctly.
Miscellaneous
Permalink to โMiscellaneousโrequestVideoFrameCallback
Permalink to โrequestVideoFrameCallbackโ Instead of events, you can also use requestVideoFrameCallback
. However, this function is not supported by FireFox, so it was not considered.
To learn more about this function, please refer to web.dev.
Conclusion
Permalink to โConclusionโI use an iPhone and find it to be a truly fascinating device. However, the unique implementation of iOS Safari can sometimes present unexpected challenges. The non-standard behavior may feel unfamiliar and daunting, but understanding and overcoming the subtle differences of the platform is a crucial aspect of enhancing user experience. Moreover, the sense of accomplishment that comes with solving these issues is invaluable.
In this article, weโve covered "How to Correctly Generate Video Thumbnails in iOS Safari." We explored various approaches and their underlying reasons, sharing the process of solving the problem with practical, applicable code. I sincerely hope that this article is helpful to you and the services you create. May it assist you in resolving similar issues you may encounter. Thank you for reading!
This article was translated from Korean using ChatGPT and DeepL.
This commit appears to resolve issues related to Blob Partitioning. Blob Partitioning is a technology that divides Blobs into smaller chunks to efficiently process large data and provide fast responses to users. This feature was introduced in Safari 17.2(19671.1.17). โฉ๏ธ
If the origin is the same, the Canvas is not tainted by the SOP! โฉ๏ธ
Example code generates Blob URL using a
File
object. TheFile
object references a file from the local file system. ("TheFile
interface provides information about files and allows JavaScript in a web page to access their content." - MDN) โฉ๏ธ