Korean

NOTE: These methods have been tested on iOS Safari versions 14+ (up to 17.4.1).

Looking for an NPM package? Check out the generate-video-dumbnail!

Thumbnail generation in iOS did not work as expected

I recently worked on generating video thumbnails. It wasn’t too difficult. I quickly put together the code with a bit of inference, ChatGPT, and Googling. The thumbnail generation code worked well in macOS Safari and Chrome. I thought I could finish the task earlier than expected. However, when I checked it in iOS Safari… NOTHING APPEARED! Something was wrong. 🤯

Browser Specific Failures Graph (Source)

Have you ever faced a similar situation? iOS Safari is sleek and sophisticated, but it can be a pain for web frontend developers. In this article, I’ll explain how to generate video thumbnails in iOS Safari and Why it’s necessary. I also provide online demos for you to test.

I don’t know how long the thumbnail issue will last, but at least for now, this guide will help you create them as intended. I hope this content is helpful.

<video
  src="https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001"
  preload="metadata"
></video>

CodePen

• 🔍 Append #t=0.001 to the end of the src attribute to show the first frame of the video.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• Video can be delivered via HTTP or HTTPS.

iOS Safari does not display the first frame of a video element by default! Even setting the preload="metadata" or preload="auto" attribute does not work. Instead, you must use a Media Fragments URI. (Can I Use)

By using one of the Media Fragments, #t=<time>, you can specify a particular time in the video (in seconds). In iOS Safari, you can use this approach to show video frame(poster). For example, if you specify 0.001 seconds(#t=0.001), it will show the first frame. Note that setting it to #t=0 does not load the frame.

While loading metadata, some video frame data can be retrieved. Therefore, the preload attribute can be set to 'metadata' instead of 'auto'. Note that the default value of preload in iOS Safari is 'auto', but it may vary depending on the device state(for example, in battery saving mode). For this reason, it is recommended to set it explicitly.

const video = document.createElement('video');
document.body.appendChild(video);

video.preload = 'metadata';
video.src =
  'https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=0.001';

CodePen

• 🔍 Append #t=0.001 to the end of the src attribute to show the first frame of the video.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• Video data must be delivered via HTTPS. For security reasons, the src attribute of dynamically created video elements must always use HTTPS.

const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';

document.body.appendChild(input);

input.addEventListener('change', ({ target }) => {
  const file = target.files[0];
  const url = URL.createObjectURL(file);

  const video = document.createElement('video');
  document.body.appendChild(video);

  video.src = url;
  video.preload = 'metadata';
  video.currentTime = 0.001;
});

CodePen

• 🔍 When using Blob URL, the preload must be set to 'metadata'. 'auto' does not work.
• 🔍 Media Fragments URI(#t=<time>) cannot be used with Blob URL. Use the 'loadedmetadata' event to set currentTime to 0.001.

Comparison of preload = 'metadata' vs. preload = 'auto' network requests

When creating a video element using Blob URL, setting preload to 'auto' does not retrieve frame data at the specified time. In fact, checking the network requests(image above), you can see that preload="metadata" retrieves data (Bytes) in the range 48-1541 to display the first frame. However, preload="auto" does not retrieve data.

Unfortunately, I could not find a technical reason for these differences. I used this code and tested it on iOS Safari 17.4.1.

In the current latest WebKit (Safari 17.4.1), Media Fragments URI (e.g., #t=<time>) cannot be used with Blob URL. The relevant changes can be found in this commit^[1]. Below is part of the commit:

BlobData* BlobRegistryImpl::getBlobDataFromURL(const URL& url, const std::optional<SecurityOriginData>& topOrigin) const
{
    ASSERT(isMainThread());
-    if (url.hasFragmentIdentifier())
-        return m_blobs.get<StringViewHashTranslator>(url.viewWithoutFragmentIdentifier());
-    return m_blobs.get(url.string());
+    auto urlKey = url.stringWithoutFragmentIdentifier();
+    auto* blobData = m_blobs.get(urlKey);
+    if (m_allowedBlobURLTopOrigins && topOrigin && topOrigin != m_allowedBlobURLTopOrigins->get(urlKey)) {
+        RELEASE_LOG_ERROR(Network, "BlobRegistryImpl::getBlobDataFromURL: (%p) Requested blob URL with incorrect top origin.", this);
+        return nullptr;
+    }
+    return blobData;
}

The URL::hasFragmentIdentifier function, which checks for the use of Media Fragment URI, has been removed. Instead, the URL::stringWithoutFragmentIdentifier function, which returns a string without the Media Fragment URI, is used. Therefore, you need to use currentTime instead of the Media Fragment URI.

From the code history, it appears that Media Fragment URI was previously supported for Blob URL. This can be confirmed in the Safari Technology Preview 118 release notes and the related commit.

const THUMBNAIL_POSITION = 5.3;

const video = document.createElement('video');
video.crossOrigin = 'anonymous';
video.preload = 'metadata';
video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${THUMBNAIL_POSITION}`;

video.addEventListener('seeked', () => {
  const canvas = document.createElement('canvas');
  canvas.width = video.videoWidth;
  canvas.height = video.videoHeight;

  const ctx = canvas.getContext('2d');
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

  canvas.toBlob(blob => {
    const img = document.createElement('img');
    img.src = URL.createObjectURL(blob);
    document.body.appendChild(img);
  });
});

CodePen

• 🔍 crossOrigin attribute must be set to 'anonymous'.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• 🔍 Video frame data is available after the 'seeked' or 'timeupdate' event.

Browsers, by default, only allow resources from the "same origin" (SOP, Same Origin Policy). If the origin is different, you need to request access through CORS (Cross-Origin Resource Sharing).

When requesting media resources, you can set the crossOrigin attribute to determine how to handle CORS policy. Of course, you can request media resources that exist in different origins without setting the crossOrigin attribute. However, the browser consider the integrity is not guaranteed. And when you draw this on a Canvas, it becomes a Tainted Canvas^[2].

A tainted canvas means that the website is not allowed to share resources. In this state, pixel data cannot be accessed. Otherwise, unauthorized origins could arbitrarily leak pixel data within the image.

Canvas pixel data can be accessed with getImageData, toDataURL, toBlob, etc. To prevent the security issues mentioned above, these methods throw a SecurityError for a tainted canvas. That’s why you need to set crossOrigin.

There are two options available. 'anonymous' (or an empty string ''), which validates CORS, but does not include any credentials (cookies, HTTP Authentication headers, SSL certificates, etc.) in the request. This helps to avoid situations where sensitive user information exposed. Therefore, I used 'anonymous' here. However, you can include credentials via the use-credential option if needed.

When requesting video data using an HTTP URL, frame data is available after the 'seeked' or 'timeupdate' event. (⚠️ Note that Blob URL behave differently.)

const THUMBNAIL_POSITION = 5.3;

const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';

document.body.appendChild(input);

input.addEventListener('change', async ({ target }) => {
  const file = target.files[0];
  const src = URL.createObjectURL(file);

  const video = document.createElement('video');
  video.src = src;
  video.preload = 'metadata';

  video.addEventListener('seeked', async () => {
    await new Promise(resolve => setTimeout(resolve, 100));

    const canvas = document.createElement('canvas');
    canvas.width = video.videoWidth;
    canvas.height = video.videoHeight;

    const ctx = canvas.getContext('2d');
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

    canvas.toBlob(blob => {
      const img = document.createElement('img');
      img.src = URL.createObjectURL(blob);
      document.body.appendChild(img);
    });
  });

  video.addEventListener(
    'loadedmetadata',
    () => (video.currentTime = THUMBNAIL_POSITION),
  );
});

CodePen

• 🔍 When using Blob URL, the preload must be set to 'metadata'. 'auto' does not work.
• 🔍 Media Fragments URI(#t=<time>) cannot be used with Blob URL. Use the 'loadedmetadata' event to set currentTime to 0.001.
• 🔍 Video frame data is available a certain time after the 'seeked' or 'timeupdate' event.

Blob URL also support the 'seeked' and 'timeupdate' events. However, if you are using the video frame data, you will need to wait a certain time. In my experience, I’ve been able to access it after 80-100ms.

Unlike HTTP URL, Blob URL may not have frame data immediately after the event is triggered (even when readyState is 4!). This is likely because Blob URL reference local files^[3]. Delays can occur as video data is decoded and loaded into memory.

const THUMBNAIL_POSITION_LIST = [
  0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
];

const main = async () => {
  for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
    const correctedThumbnailPosition = thumbnailPosition || 0.001;

    const video = document.createElement('video');
    video.crossOrigin = 'anonymous';
    video.preload = 'metadata';
    video.src = `https://cdn.jsdelivr.net/gh/Gumball12/public-timer-mp4/timer.mp4#t=${correctedThumbnailPosition}`;

    generateThumbnail(video, thumbnailPosition);
  }
};

const generateThumbnail = (video, thumbnailPosition) => {
  const img = document.createElement('img');
  document.body.appendChild(img);

  video.addEventListener('seeked', () => {
    const canvas = document.createElement('canvas');
    canvas.width = video.videoWidth;
    canvas.height = video.videoHeight;

    const ctx = canvas.getContext('2d');
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    canvas.toBlob(blob => (img.src = URL.createObjectURL(blob)));
  });

  video.currentTime = thumbnailPosition;
};

main();

CodePen

• 🔍 crossOrigin attribute must be set to 'anonymous'.
• 🔍 preload attribute does not need to be set to 'metadata', but this may delay thumbnail generation.
• 🔍 Use #t=${time} to specify the thumbnail position.
• 🔍 Video frame data is available after the 'seeked' or 'timeupdate' event.

const input = document.createElement('input');
input.type = 'file';
input.accept = 'video/*';

document.body.appendChild(input);

input.addEventListener('change', async ({ target }) => {
  const file = target.files[0];
  const src = URL.createObjectURL(file);

  const THUMBNAIL_POSITION_LIST = [
    0, 1.1, 2.2, 3.3, 4.4, 5.5, 4.6, 3.7, 2.8, 1.9, 0,
  ];

  for (const thumbnailPosition of THUMBNAIL_POSITION_LIST) {
    const video = document.createElement('video');
    video.preload = 'metadata';
    video.src = src;
    await generateThumbnail(video, thumbnailPosition);
  }
});

const generateThumbnail = (video, thumbnailPosition) =>
  new Promise(resolve => {
    video.addEventListener('seeked', async () => {
      await new Promise(resolve => setTimeout(resolve, 100));

      const canvas = document.createElement('canvas');
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;

      const ctx = canvas.getContext('2d');
      ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

      canvas.toBlob(blob => {
        const img = document.createElement('img');
        img.src = URL.createObjectURL(blob);
        document.body.appendChild(img);

        resolve();
      });
    });

    video.addEventListener('loadedmetadata', () =>
      setTimeout(() => (video.currentTime = thumbnailPosition), 1),
    );
  });

CodePen

• 🔍 When using Blob URL, the preload must be set to 'metadata'. 'auto' does not work. <<<<<<< Updated upstream

• 🔍 To prevent thread blocking, use Promise to generate thumbnails sequentially.

  Permalink to “🔍 To prevent thread blocking, use Promise to generate thumbnails sequentially.” #
• 🔍 To prevent thread blocking, use Promise to generate thumbnails sequentially.

  Stashed changes

• 🔍 Video frame data is available a certain time after the 'seeked' or 'timeupdate' event.

Thread blocking can occur when generating multiple thumbnails. If thread blocking occurs, thumbnails may not be generated correctly. In other words, transparent thumbnails are generated.

Attempting to generate thumbnails all at once (Timeline Raw Data)

The above image shows the result of attempting to generate all thumbnails at once. Frequent network requests are occurring, with a concentration in a specific range. Additionally, the main thread is being used continuously for an extended period. As a result, thread blocking occurs, and transparent thumbnails are generated.

To avoid this, I used Promise to generate thumbnails sequentially. The generateThumbnail function returns a Promise, and I used await to wait for each thumbnail to be generated.

Generating thumbnails sequentially (Timeline Raw Data)

Generating thumbnails sequentially ensures that network requests are evenly distributed, and the main thread is used only for a short period. This prevents thread blocking and ensures that thumbnails are generated correctly.

Instead of events, you can also use requestVideoFrameCallback. However, this function is not supported by FireFox, so it was not considered.

To learn more about this function, please refer to web.dev.

I use an iPhone and find it to be a truly fascinating device. However, the unique implementation of iOS Safari can sometimes present unexpected challenges. The non-standard behavior may feel unfamiliar and daunting, but understanding and overcoming the subtle differences of the platform is a crucial aspect of enhancing user experience. Moreover, the sense of accomplishment that comes with solving these issues is invaluable.

In this article, we’ve covered "How to Correctly Generate Video Thumbnails in iOS Safari." We explored various approaches and their underlying reasons, sharing the process of solving the problem with practical, applicable code. I sincerely hope that this article is helpful to you and the services you create. May it assist you in resolving similar issues you may encounter. Thank you for reading!

This article was translated from Korean using ChatGPT and DeepL.