Table of Contents
NOTE: These methods have been tested on iOS Safari versions 14+ (up to 17.4.1).
Looking for an NPM package? Check out the generate-video-dumbnail!
Thumbnail generation in iOS did not work as expected
I recently worked on generating video thumbnails. It wasnโt too difficult. I quickly put together the code with a bit of inference, ChatGPT, and Googling. The thumbnail generation code worked well in macOS Safari and Chrome. I thought I could finish the task earlier than expected. However, when I checked it in iOS Safariโฆ NOTHING APPEARED! Something was wrong.
Browser Specific Failures Graph (Source)
Have you ever faced a similar situation? iOS Safari is sleek and sophisticated, but it can be a pain for web frontend developers. In this article, Iโll explain how to generate video thumbnails in iOS Safari and Why itโs necessary. I also provide online demos for you to test. I donโt know how long this issue will persist, but at least for now, you can use this guide to generate thumbnails as intended.
Letโs get started!
Display Video Poster
Permalink to โDisplay Video Posterโw/ Static Video Element
Permalink to โw/ Static Video Elementโ<video
src="https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001"
preload="metadata"
></video>
- ๐ Append
#t=0.001
to the end of thesrc
attribute to show the first frame of the video. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - Video can be delivered via HTTP or HTTPS.
๐ Media Fragments URI
Permalink to โ๐ Media Fragments URIโiOS Safari does not display the first frame of a video element by default! Even setting the preload="metadata"
or preload="auto"
attribute does not work. Instead, you must use a Media Fragments URI. (Can I Use)
By using one of the Media Fragments, #t=<time>
, you can specify a particular time in the video (in seconds). For example, if you specify 0.001 seconds, it will show the first frame. Note that setting it to #t=0
does not load the frame.
๐ Preload Attribute
Permalink to โ๐ Preload AttributeโWhile loading metadata, some video frame data can be retrieved. Therefore, the preload
attribute can be set to 'metadata'
instead of 'auto'
. Note that the default value of preload
in iOS Safari is 'auto'
, but it may vary depending on the device state. For this reason, it is recommended to set it explicitly.
w/ Dynamic Video Element (HTTP URL)
Permalink to โw/ Dynamic Video Element (HTTP URL)โconst video = document.createElement('video');
document.body.appendChild(video);
video.preload = 'metadata';
video.src =
'https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001';
- ๐ Append
#t=0.001
to the end of thesrc
attribute to show the first frame of the video. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - Video data must be delivered via HTTPS. For security reasons, the src attribute of dynamically created video elements must always use HTTPS.
w/ Dynamic Video Element (Blob URL)
Permalink to โw/ Dynamic Video Element (Blob URL)โconst input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';
document.body.appendChild(input);
input.addEventListener('change', ({ target }) => {
const file = target.files[0];
const url = URL.createObjectURL(file);
const video = document.createElement('video');
document.body.appendChild(video);
video.src = url;
video.preload = 'metadata';
video.currentTime = 0.001;
});
- ๐ The
preload
must be set to'metadata'
.'auto'
does not work. - ๐ Media Fragments URI(
#t=<time>
) cannot be used with Blob URL. Use the'loadedmetadata'
event to setcurrentTime
to0.001
.
๐ Set preload='metadata'
for Blob URL
Permalink to โ๐ Set preload='metadata' for Blob URLโ Comparison of "metadata" and "auto" preload values
When creating a video element using Blob URL, setting preload
to 'auto'
does not retrieve frame data at the specified time. In the image above, byte data in the range 48-1541 was retrieved with preload="metadata"
. However, this was not retrieved with preload="auto"
. When the data was not retrieved, the poster for the video element was not displayed, indicating that the first video frame data was not retrieved.
Unfortunately, I could not find a technical reason for these differences. I used this code and tested it on iOS Safari 17.4.1.
๐ Media Fragments URI cannot be used with Blob URL
Permalink to โ๐ Media Fragments URI cannot be used with Blob URLโIn the current latest WebKit (Safari 17.4.1), Media Fragments URI (e.g., #t=<time>
) cannot be used with Blob URL. The relevant changes can be found in this commit[1]. Here are some of the changes:
BlobData* BlobRegistryImpl::getBlobDataFromURL(const URL& url, const std::optional<SecurityOriginData>& topOrigin) const
{
ASSERT(isMainThread());
- if (url.hasFragmentIdentifier())
- return m_blobs.get<StringViewHashTranslator>(url.viewWithoutFragmentIdentifier());
- return m_blobs.get(url.string());
+ auto urlKey = url.stringWithoutFragmentIdentifier();
+ auto* blobData = m_blobs.get(urlKey);
+ if (m_allowedBlobURLTopOrigins && topOrigin && topOrigin != m_allowedBlobURLTopOrigins->get(urlKey)) {
+ RELEASE_LOG_ERROR(Network, "BlobRegistryImpl::getBlobDataFromURL: (%p) Requested blob URL with incorrect top origin.", this);
+ return nullptr;
+ }
+ return blobData;
}
The URL::hasFragmentIdentifier
function, which checks for the use of Media Fragment URI, has been removed. Instead, the URL::stringWithoutFragmentIdentifier
function, which returns a string without the Media Fragment URI, is used. Therefore, you need to set the current time of the video element using currentTime
instead of Media Fragment URI.
From the code history, it appears that Media Fragment URI was previously supported for Blob URL. This can be confirmed in the Safari Technology Preview 118 release notes and the related commit.
Generate Single Thumbnail
Permalink to โGenerate Single Thumbnailโw/ HTTP URL
Permalink to โw/ HTTP URLโconst THUMBNAIL_POSITION = 5.3;
const video = document.createElement('video');
video.crossOrigin = 'anonymous';
video.preload = 'metadata';
video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${THUMBNAIL_POSITION}`;
video.addEventListener('seeked', () => {
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => {
const img = document.createElement('img');
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
});
});
- ๐
crossOrigin
attribute must be set to'anonymous'
. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - ๐ Video frame data is available after the
'seeked'
or'timeupdate'
event.
๐ Set crossOrigin='anonymous'
to Avoid SecurityError with Canvas Element
Permalink to โ๐ Set crossOrigin='anonymous' to Avoid SecurityError with Canvas Elementโ When requesting media resources, you can set the crossOrigin
attribute to determine how to handle CORS (Cross-Origin Resource Sharing) policies. Among the options, 'anonymous'
(or an empty string ''
) is used to exclude credentials (Cookies, HTTP Authentication headers, Client SSL Certificates, etc.) from the request. This option helps prevent the exposure of sensitive user information.
Why should you use this? Yes, you can request media resources from other origins even without setting the crossOrigin
attribute. However, the browser will consider that the integrity is not guaranteed. If you draw this on a Canvas, it becomes a Tainted Canvas.
Tainted Canvas can cause security issues. For example, consider drawing an image from another origin that contains user information on a Canvas. In this case, the pixel data of the image drawn on the Canvas can be sensitive information and should not be stolen. Therefore, access to pixel data for Tainted Canvas is restricted to prevent security issues.
Canvas pixel data access methods include getImageData
, toDataURL
, toBlob
, and more. To prevent the security issues mentioned above, these methods throw a SecurityError
for Tainted Canvas. Therefore, when using these methods, you must set the crossOrigin
attribute to 'anonymous'
.
Note that this is a security policy that applies to all browsers.
๐ Video frame data is available after the 'seeked'
or 'timeupdate'
event (HTTP URL)
Permalink to โ๐ Video frame data is available after the 'seeked' or 'timeupdate' event (HTTP URL)โ When requesting video data using an HTTP URL, frame data is available after the 'seeked'
or 'timeupdate'
event. Note that Blob URL behave differently.
w/ Blob URL
Permalink to โw/ Blob URLโconst THUMBNAIL_POSITION = 5.3;
const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';
document.body.appendChild(input);
input.addEventListener('change', async ({ target }) => {
const file = target.files[0];
const src = URL.createObjectURL(file);
const video = document.createElement('video');
video.src = src;
video.preload = 'metadata';
video.addEventListener('seeked', async () => {
await new Promise(resolve => setTimeout(resolve, 100));
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => {
const img = document.createElement('img');
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
});
});
video.addEventListener(
'loadedmetadata',
() => (video.currentTime = THUMBNAIL_POSITION),
);
});
- ๐ The
preload
must be set to'metadata'
.'auto'
does not work. - ๐ Media Fragments URI(
#t=<time>
) cannot be used with Blob URL. Use the'loadedmetadata'
event to setcurrentTime
to0.001
. - ๐ Video frame data is available a certain time after the
'seeked'
or'timeupdate'
event.
๐ Video frame data is available a certain time after the 'seeked'
or 'timeupdate'
event (Blob URL)
Permalink to โ๐ Video frame data is available a certain time after the 'seeked' or 'timeupdate' event (Blob URL)โ Blob URL also support the 'seeked'
and 'timeupdate'
events. However, you must wait a certain time. Empirically, I found that waiting about 80-100ms is necessary.
Unlike HTTP URL, Blob URL may not have frame data immediately after the event is triggered (even when readyState
is 4
!). This is likely because Blob URL reference local files[2]. Delays can occur as video data is decoded and loaded into memory.
Generate Multiple Thumbnails
Permalink to โGenerate Multiple Thumbnailsโw/ HTTP URL
Permalink to โw/ HTTP URLโconst THUMBNAIL_POSITION_LIST = [
0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
];
const main = async () => {
for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
const correctedThumbnailPosition = thumbnailPosition || 0.001;
const video = document.createElement('video');
video.crossOrigin = 'anonymous';
video.preload = 'metadata';
video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${correctedThumbnailPosition}`;
generateThumbnail(video, thumbnailPosition);
}
};
const generateThumbnail = (video, thumbnailPosition) => {
const img = document.createElement('img');
document.body.appendChild(img);
video.addEventListener('seeked', () => {
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => (img.src = URL.createObjectURL(blob)));
});
video.currentTime = thumbnailPosition;
};
main();
- ๐
crossOrigin
attribute must be set to'anonymous'
. - ๐
preload
attribute does not need to be set to'metadata'
, but this may delay thumbnail generation. - ๐ Use
#t=${time}
to specify the thumbnail position. - ๐ Video frame data is available after the
'seeked'
or'timeupdate'
event.
w/ Blob URL
Permalink to โw/ Blob URLโconst input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';
document.body.appendChild(input);
input.addEventListener('change', async ({ target }) => {
const file = target.files[0];
const src = URL.createObjectURL(file);
const THUMBNAIL_POSITION_LIST = [
0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
];
for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
const video = document.createElement('video');
video.preload = 'metadata';
video.src = src;
await generateThumbnail(video, thumbnailPosition);
}
});
const generateThumbnail = (video, thumbnailPosition) =>
new Promise(resolve => {
video.addEventListener('seeked', async () => {
await new Promise(resolve => setTimeout(resolve, 100));
const canvas = document.createElement('canvas');
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
canvas.toBlob(blob => {
const img = document.createElement('img');
img.src = URL.createObjectURL(blob);
document.body.appendChild(img);
resolve();
});
});
video.addEventListener('loadedmetadata', () =>
setTimeout(() => (video.currentTime = thumbnailPosition), 1),
);
});
- ๐ The
preload
must be set to'metadata'
.'auto'
does not work. - ๐ To prevent thread blocking, use
Promise
to generate thumbnails sequentially. - ๐ Video frame data is available a certain time after the
'seeked'
or'timeupdate'
event.
๐ Incorrect thumbnails may be generated due to thead blocking
Permalink to โ๐ Incorrect thumbnails may be generated due to thead blockingโThread blocking can occur when generating multiple thumbnails. If thread blocking occurs, thumbnails may not be generated correctly. In other words, transparent thumbnails are generated.
Attempting to generate thumbnails all at once (Timeline Data)
The above image shows the result of attempting to generate all thumbnails at once. Frequent network requests are occurring, with a concentration in a specific range. Additionally, the main thread is being used continuously for an extended period. As a result, thread blocking occurs, and transparent thumbnails are generated.
To avoid this, I used Promise
to generate thumbnails sequentially. The generateThumbnail
function returns a Promise
, and I used await
to wait for each thumbnail to be generated.
Generating thumbnails sequentially (Timeline Data)
Generating thumbnails sequentially ensures that network requests are evenly distributed, and the main thread is used only for a short period. This prevents thread blocking and ensures that thumbnails are generated correctly.
Miscellaneous
Permalink to โMiscellaneousโrequestVideoFrameCallback
Permalink to โrequestVideoFrameCallbackโ Instead of using events, you can also use requestVideoFrameCallback
. However, this function is not supported in FireFox, so it was not considered.
To learn more about this function, please refer to web.dev.
Conclusion
Permalink to โConclusionโEven though I am an iPhone user, iOS sometimes feels really strange! Especially to work well on iOS Safari, it takes quite a bit of effort. However, these efforts are not in vain. Better services and better user experiences will come back as rewards.
Here, we looked at "How to Generate Video Thumbnails Correctly". I sincerely hope this article helps you and your services. Thank you for reading! ๐ฅน
This commit appears to resolve issues related to Blob Partitioning. Blob Partitioning is a technology that divides Blobs into smaller chunks to efficiently process large data and provide fast responses to users. This feature was introduced in Safari 17.2(19671.1.17). โฉ๏ธ
Example code generates Blob URL using a
File
object. TheFile
object references a file from the local file system. ("TheFile
interface provides information about files and allows JavaScript in a web page to access their content." - MDN) โฉ๏ธ