Extracting selected images from downloaded video files?

Elixir Framework Forums Membrane Forum
, , , , ,
1

I want to extract selected images from downloaded video files. I’d like to know whether Membrane is a reasonable tool for this job, whether anyone else has done similar tasks with it, etc. I’m also unsure about how its “pipeline” approach would fit a “random access” use case.

For purposes of discussion, assume that I’ll be using VLC media player to download videos from web sites such as YouTube, storing them as MP4 files. Because I’ll be processing the images, I’d like to get them into Nx tensor format.

Can someone give me some advice, clues, and/or suggestions?

1 Like
2

I would use ffmpeg to extract images from the video files. Could probably skip VLC and use ffmpeg directly as well.

Then loading the extracted images from disk and do whatever I have to do in nx :slight_smile:

Probably not gonna be the most efficient way of doing it, but might be good enough :slight_smile:

No idea on how membrane works.

6 Likes
3

I am also using ffmpeg to extract images from video, there is a ffmpex wrapper, but I wrote a simple one.

Membrane is good if You want to receive or stream rtmp, or webrtc…

But for YT, I would check this python package, and maybe use erlport to interface python script.

There are tools to transform images from/to nx…

by @kip

and

by @cocoa

4 Likes
4

It seems like FFmpeg only extracts images in PNG format. This might work well with ExPNG, but I think I’ll want to get them into Nx format at some point. Suggestions?

5

It can use more format… like jpg. But if You want them in nx, the 2 packages have a to_nx() function.

2 Likes
6

The wonderful image library can load images and convert them to/from tensors

https://hexdocs.pm/image/Image.html#to_nx/2

Edit: didn’t notice you already got this suggestion;)

1 Like
7

As of image version 0.21.0 which I just published you can leverage the Evision.VideoCapture module to extract images from a video.

You can therefore use either the more idiomatic API in Image.Video or simply leverage the code that is there to use Evision directly.

Depending on the codec support of the OpenCV installation on your system I suspect you may not need to use the FFMEG CLI approach any more.

Example using Image

# Extract using a frame offset
iex> {:ok, video} = Image.Video.open("./test/support/video/video_sample.mp4")
iex> {:ok, _image} = Image.Video.image_from_video(video, frame: 0)

# Extract using a millisecond offset
iex> {:ok, _image} = Image.Video.image_from_video(video, millisecond: 0)

# Capturing images from a camera (note that seeking is not supported
# on image streams. On MacOS you will receive a prompt to permit the
# app to access the camera. You may need to restart the BEAM and/or
# your system if capturing fails immediately after granting permission.
iex> {:ok, video} = Image.Video.open(:default_camera)
iex> {:ok, _image} = Image.Video.image_from_video(video)

Example using Evision directly

# Extract by frame offset
video = Evision.VideoCapture.videoCapture(video_filename)
true = Evision.VideoCapture.set(video, Evision.cv_CAP_PROP_POS_FRAMES(), frame_offset),
cv_image = Evision.VideoCapture.read(video)

Using images in Nx

Both Image and Evision support zero-copy “export” of images to Nx. Just call Image.to_nx/1 or Evision.Mat.to_nx/1 as appropriate.

9 Likes