NOTE: These methods have been tested on iOS Safari versions 14+ (up to 17.4.1).

Looking for an NPM package? Check out the generate-video-dumbnail!

Thumbnail generation in iOS did not work as expected

I recently worked on generating video thumbnails. It wasn’t too difficult. I quickly put together the code with a bit of inference, ChatGPT, and Googling. The thumbnail generation code worked well in macOS Safari and Chrome. I thought I could finish the task earlier than expected. However, when I checked it in iOS Safari… NOTHING APPEARED! Something was wrong.

Browser Specific Failures Graph (Source)

Have you ever faced a similar situation? iOS Safari is sleek and sophisticated, but it can be a pain for web frontend developers. In this article, I’ll explain how to generate video thumbnails in iOS Safari and Why it’s necessary. I also provide online demos for you to test. I don’t know how long this issue will persist, but at least for now, you can use this guide to generate thumbnails as intended.

Let’s get started!

<video
  src="https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001"
  preload="metadata"
></video>

CodePen

• 🔍 Append #t=0.001 to the end of the src attribute to show the first frame of the video.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• Video can be delivered via HTTP or HTTPS.

iOS Safari does not display the first frame of a video element by default! Even setting the preload="metadata" or preload="auto" attribute does not work. Instead, you must use a Media Fragments URI. (Can I Use)

By using one of the Media Fragments, #t=<time>, you can specify a particular time in the video (in seconds). For example, if you specify 0.001 seconds, it will show the first frame. Note that setting it to #t=0 does not load the frame.

While loading metadata, some video frame data can be retrieved. Therefore, the preload attribute can be set to 'metadata' instead of 'auto'. Note that the default value of preload in iOS Safari is 'auto', but it may vary depending on the device state. For this reason, it is recommended to set it explicitly.

const video = document.createElement('video');
document.body.appendChild(video);

video.preload = 'metadata';
video.src =
  'https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001';

CodePen

• 🔍 Append #t=0.001 to the end of the src attribute to show the first frame of the video.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• Video data must be delivered via HTTPS. For security reasons, the src attribute of dynamically created video elements must always use HTTPS.

const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';

document.body.appendChild(input);

input.addEventListener('change', ({ target }) => {
  const file = target.files[0];
  const url = URL.createObjectURL(file);

  const video = document.createElement('video');
  document.body.appendChild(video);

  video.src = url;
  video.preload = 'metadata';
  video.currentTime = 0.001;
});

CodePen

• 🔍 The preload must be set to 'metadata'. 'auto' does not work.
• 🔍 Media Fragments URI(#t=<time>) cannot be used with Blob URL. Use the 'loadedmetadata' event to set currentTime to 0.001.

Comparison of "metadata" and "auto" preload values

When creating a video element using Blob URL, setting preload to 'auto' does not retrieve frame data at the specified time. In the image above, byte data in the range 48-1541 was retrieved with preload="metadata". However, this was not retrieved with preload="auto". When the data was not retrieved, the poster for the video element was not displayed, indicating that the first video frame data was not retrieved.

Unfortunately, I could not find a technical reason for these differences. I used this code and tested it on iOS Safari 17.4.1.

In the current latest WebKit (Safari 17.4.1), Media Fragments URI (e.g., #t=<time>) cannot be used with Blob URL. The relevant changes can be found in this commit^[1]. Here are some of the changes:

BlobData* BlobRegistryImpl::getBlobDataFromURL(const URL& url, const std::optional<SecurityOriginData>& topOrigin) const
{
    ASSERT(isMainThread());
-    if (url.hasFragmentIdentifier())
-        return m_blobs.get<StringViewHashTranslator>(url.viewWithoutFragmentIdentifier());
-    return m_blobs.get(url.string());
+    auto urlKey = url.stringWithoutFragmentIdentifier();
+    auto* blobData = m_blobs.get(urlKey);
+    if (m_allowedBlobURLTopOrigins && topOrigin && topOrigin != m_allowedBlobURLTopOrigins->get(urlKey)) {
+        RELEASE_LOG_ERROR(Network, "BlobRegistryImpl::getBlobDataFromURL: (%p) Requested blob URL with incorrect top origin.", this);
+        return nullptr;
+    }
+    return blobData;
}

The URL::hasFragmentIdentifier function, which checks for the use of Media Fragment URI, has been removed. Instead, the URL::stringWithoutFragmentIdentifier function, which returns a string without the Media Fragment URI, is used. Therefore, you need to set the current time of the video element using currentTime instead of Media Fragment URI.

From the code history, it appears that Media Fragment URI was previously supported for Blob URL. This can be confirmed in the Safari Technology Preview 118 release notes and the related commit.

const THUMBNAIL_POSITION = 5.3;

const video = document.createElement('video');
video.crossOrigin = 'anonymous';
video.preload = 'metadata';
video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${THUMBNAIL_POSITION}`;

video.addEventListener('seeked', () => {
  const canvas = document.createElement('canvas');
  canvas.width = video.videoWidth;
  canvas.height = video.videoHeight;

  const ctx = canvas.getContext('2d');
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

  canvas.toBlob(blob => {
    const img = document.createElement('img');
    img.src = URL.createObjectURL(blob);
    document.body.appendChild(img);
  });
});

CodePen

• 🔍 crossOrigin attribute must be set to 'anonymous'.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• 🔍 Video frame data is available after the 'seeked' or 'timeupdate' event.

When requesting media resources, you can set the crossOrigin attribute to determine how to handle CORS (Cross-Origin Resource Sharing) policies. Among the options, 'anonymous' (or an empty string '') is used to exclude credentials (Cookies, HTTP Authentication headers, Client SSL Certificates, etc.) from the request. This option helps prevent the exposure of sensitive user information.

Why should you use this? Yes, you can request media resources from other origins even without setting the crossOrigin attribute. However, the browser will consider that the integrity is not guaranteed. If you draw this on a Canvas, it becomes a Tainted Canvas.

Tainted Canvas can cause security issues. For example, consider drawing an image from another origin that contains user information on a Canvas. In this case, the pixel data of the image drawn on the Canvas can be sensitive information and should not be stolen. Therefore, access to pixel data for Tainted Canvas is restricted to prevent security issues.

Canvas pixel data access methods include getImageData, toDataURL, toBlob, and more. To prevent the security issues mentioned above, these methods throw a SecurityError for Tainted Canvas. Therefore, when using these methods, you must set the crossOrigin attribute to 'anonymous'.

Note that this is a security policy that applies to all browsers.

When requesting video data using an HTTP URL, frame data is available after the 'seeked' or 'timeupdate' event. Note that Blob URL behave differently.

const THUMBNAIL_POSITION = 5.3;

const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';

document.body.appendChild(input);

input.addEventListener('change', async ({ target }) => {
  const file = target.files[0];
  const src = URL.createObjectURL(file);

  const video = document.createElement('video');
  video.src = src;
  video.preload = 'metadata';

  video.addEventListener('seeked', async () => {
    await new Promise(resolve => setTimeout(resolve, 100));

    const canvas = document.createElement('canvas');
    canvas.width = video.videoWidth;
    canvas.height = video.videoHeight;

    const ctx = canvas.getContext('2d');
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

    canvas.toBlob(blob => {
      const img = document.createElement('img');
      img.src = URL.createObjectURL(blob);
      document.body.appendChild(img);
    });
  });

  video.addEventListener(
    'loadedmetadata',
    () => (video.currentTime = THUMBNAIL_POSITION),
  );
});

CodePen

• 🔍 The preload must be set to 'metadata'. 'auto' does not work.
• 🔍 Media Fragments URI(#t=<time>) cannot be used with Blob URL. Use the 'loadedmetadata' event to set currentTime to 0.001.
• 🔍 Video frame data is available a certain time after the 'seeked' or 'timeupdate' event.

Blob URL also support the 'seeked' and 'timeupdate' events. However, you must wait a certain time. Empirically, I found that waiting about 80-100ms is necessary.

Unlike HTTP URL, Blob URL may not have frame data immediately after the event is triggered (even when readyState is 4!). This is likely because Blob URL reference local files^[2]. Delays can occur as video data is decoded and loaded into memory.

const THUMBNAIL_POSITION_LIST = [
  0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
];

const main = async () => {
  for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
    const correctedThumbnailPosition = thumbnailPosition || 0.001;

    const video = document.createElement('video');
    video.crossOrigin = 'anonymous';
    video.preload = 'metadata';
    video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${correctedThumbnailPosition}`;

    generateThumbnail(video, thumbnailPosition);
  }
};

const generateThumbnail = (video, thumbnailPosition) => {
  const img = document.createElement('img');
  document.body.appendChild(img);

  video.addEventListener('seeked', () => {
    const canvas = document.createElement('canvas');
    canvas.width = video.videoWidth;
    canvas.height = video.videoHeight;

    const ctx = canvas.getContext('2d');
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    canvas.toBlob(blob => (img.src = URL.createObjectURL(blob)));
  });

  video.currentTime = thumbnailPosition;
};

main();

CodePen

• 🔍 crossOrigin attribute must be set to 'anonymous'.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• 🔍 Use #t=${time} to specify the thumbnail position.
• 🔍 Video frame data is available after the 'seeked' or 'timeupdate' event.

const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';

document.body.appendChild(input);

input.addEventListener('change', async ({ target }) => {
  const file = target.files[0];
  const src = URL.createObjectURL(file);

  const THUMBNAIL_POSITION_LIST = [
    0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
  ];

  for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
    const video = document.createElement('video');
    video.preload = 'metadata';
    video.src = src;
    await generateThumbnail(video, thumbnailPosition);
  }
});

const generateThumbnail = (video, thumbnailPosition) =>
  new Promise(resolve => {
    video.addEventListener('seeked', async () => {
      await new Promise(resolve => setTimeout(resolve, 100));

      const canvas = document.createElement('canvas');
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;

      const ctx = canvas.getContext('2d');
      ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

      canvas.toBlob(blob => {
        const img = document.createElement('img');
        img.src = URL.createObjectURL(blob);
        document.body.appendChild(img);

        resolve();
      });
    });

    video.addEventListener('loadedmetadata', () =>
      setTimeout(() => (video.currentTime = thumbnailPosition), 1),
    );
  });

CodePen

• 🔍 The preload must be set to 'metadata'. 'auto' does not work.
• 🔍 To prevent thread blocking, use Promise to generate thumbnails sequentially.
• 🔍 Video frame data is available a certain time after the 'seeked' or 'timeupdate' event.

Thread blocking can occur when generating multiple thumbnails. If thread blocking occurs, thumbnails may not be generated correctly. In other words, transparent thumbnails are generated.

Attempting to generate thumbnails all at once (Timeline Data)

The above image shows the result of attempting to generate all thumbnails at once. Frequent network requests are occurring, with a concentration in a specific range. Additionally, the main thread is being used continuously for an extended period. As a result, thread blocking occurs, and transparent thumbnails are generated.

To avoid this, I used Promise to generate thumbnails sequentially. The generateThumbnail function returns a Promise, and I used await to wait for each thumbnail to be generated.

Generating thumbnails sequentially (Timeline Data)

Generating thumbnails sequentially ensures that network requests are evenly distributed, and the main thread is used only for a short period. This prevents thread blocking and ensures that thumbnails are generated correctly.

Instead of using events, you can also use requestVideoFrameCallback. However, this function is not supported in FireFox, so it was not considered.

To learn more about this function, please refer to web.dev.

Even though I am an iPhone user, iOS sometimes feels really strange! Especially to work well on iOS Safari, it takes quite a bit of effort. However, these efforts are not in vain. Better services and better user experiences will come back as rewards.

Here, we looked at "How to Generate Video Thumbnails Correctly". I sincerely hope this article helps you and your services. Thank you for reading! 🥹