live callbacks for live-streams. #151

mhashim6 · 2020-02-03T13:37:14Z

With this, we can use continuous camera feed (eg, cv2::VideoCapture) and have live detection of scenes/events. Basically, issue #5

mhashim6 · 2020-02-03T13:45:22Z

EDIT FROM @Breakthrough : This example is outdated as of v0.6. See #273 for an updated example until one gets added to the official documentation.

A usage example:

cam = cv2.VideoCapture(0)
scene_manager.add_detector(ContentDetector())
scene_manager.detect_scenes(frame_source=cam, callback=lambda frame: print("a new scene: ", len(frame)))

Breakthrough · 2020-02-29T14:08:25Z

scenedetect/scene_manager.py

        # type(int, numpy.ndarray) -> None
        """ Adds any cuts detected with the current frame to the cutting list. """
        for detector in self._detector_list:
-            self._cutting_list += detector.process_frame(frame_num, frame_im)
+            cuts = detector.process_frame(frame_num, frame_im)
+            if len(cuts) and callback:


Can this just be if cuts and callback:?

Breakthrough · 2020-02-29T14:09:07Z

scenedetect/scene_manager.py

        for detector in self._sparse_detector_list:
-            self._event_list += detector.process_frame(frame_num, frame_im)
+            events = detector.process_frame(frame_num, frame_im)
+            if len(events) and callback:


Same as above. I think PyLint will flag this line is why.

Breakthrough

Looks good overall, suggest using PyLint or something similar though.

Very minor changes requested for code style/quality, but overall great PR!

Breakthrough · 2020-02-29T14:11:35Z

scenedetect/scene_manager.py

        # type: (VideoManager, Union[int, FrameTimecode],
-        #        Optional[Union[int, FrameTimecode]], Optional[bool]) -> int
+        #        Optional[Union[int, FrameTimecode]], Optional[bool], optional[callable]) -> int


optional[callable] should be Optional[Callable[numpy.ndarray]]

Breakthrough · 2020-02-29T14:11:59Z

scenedetect/scene_manager.py

-            self._cutting_list += detector.process_frame(frame_num, frame_im)
+            cuts = detector.process_frame(frame_num, frame_im)
+            if len(cuts) and callback:
+                callback(frame_im)


Should the frame number or timecode be passed to the callback as well?

I can do the frame number, but where do I obtain the timecode?

Good point, I forgot the scene manager only deals with frame numbers. Just the frame number should be sufficient. Sorry for the delay in getting back to you on this one, will definitely be trying to get this in for the next release though!

No problem at all. I pushed a commit with frame_num.

mhashim6 · 2020-02-29T16:15:25Z

Looks good overall, suggest using PyLint or something similar though.

Very minor changes requested for code style/quality, but overall great PR!

Thanks! I installed PyLint in PyCharm and pushed some changes.

Breakthrough · 2020-08-08T00:12:52Z

I may retarget this for the v0.6.x branch before merging it, and have to update the documentation with some examples, but this will definitely make it in for the next release. Thank you so much for this!

mhashim6 · 2020-08-08T13:51:21Z

I may retarget this for the v0.6.x branch before merging it, and have to update the documentation with some examples, but this will definitely make it in for the next release. Thank you so much for this!

This is great! You are welcome!
In fact, thank you in the first place, we used this lib + live feed feature in our graduation project : )

Breakthrough · 2020-08-08T17:10:53Z

@mhashim6 you are most welcome! Would it be okay if I referenced your paper on the following page:
https://pyscenedetect.readthedocs.io/en/latest/other/resources/

(If not, no worries!) Thank you so much!

mhashim6 · 2020-08-08T19:12:16Z

@mhashim6 you are most welcome! Would it be okay if I referenced your paper on the following page:

https://pyscenedetect.readthedocs.io/en/latest/other/resources/

(If not, no worries!) Thank you so much!

Oh not at all, I'd actually be glad! Thank you!

live callbacks for live-streams. Resolves #5.

research-boy · 2022-07-07T12:43:27Z

A usage example:

cam = cv2.VideoCapture(0)
scene_manager.add_detector(ContentDetector())
scene_manager.detect_scenes(frame_source=cam, callback=lambda frame: print("a new scene: ", len(frame)))

when i tried this on a live video input, it throws error,

    scene_manager.detect_scenes(frame_source=cam, callback=lambda frame: print("a new scene: ", len(frame)))
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/scene_manager.py", line 752, in detect_scenes
    self._base_timecode = video.base_timecode
AttributeError: 'cv2.VideoCapture' object has no attribute 'base_timecode'
```
any idea why is this happening?

Breakthrough · 2022-07-07T22:06:22Z

@research-boy The API has changed slightly in v0.6, you need to use a VideoStreamCv2 which is a wrapper over cv2.VideoCapture, e.g.:

from scenedetect import VideoStreamCv2, SceneManager, ContentDetector
scene_manager = SceneManager()
cam = VideoStreamCv2(0)
scene_manager.add_detector(ContentDetector())
scene_manager.detect_scenes(video=cam, callback = ...)

Hope this helps!

Breakthrough · 2022-07-07T22:15:50Z

@research-boy I also filed #273 to track adding an official example for this use case to the API documentation. Hopefully that will clear up any inconsistencies, since this example is outdated.

research-boy · 2022-07-08T09:32:54Z

@Breakthrough Thanks for the example.
With the use case i am trying to get this run, the input is from a gstreamer pipe.
Like:

# in opencv how i read 
cap = cv2.VideoCapture("videotestsrc ! appsink")
# As per the example you shared
cam = VideoStreamCv2("videotestsrc ! appsink")

when i tried the example with this it throws error

    cam = VideoStreamCv2(source)
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/backends/opencv.py", line 92, in __init__
    self._open_capture(framerate)
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/backends/opencv.py", line 297, in _open_capture
    raise OSError('Video file not found.')

Breakthrough · 2022-07-11T02:28:16Z

Ah thanks for sharing, that definitely needs to be fixed. This check needs to also see if there is a ! in the path:
https://github.com/Breakthrough/PySceneDetect/blob/master/scenedetect/backends/opencv.py#L296=
As a workaround, you can fall back to using v0.5, or comment the above check out temporarily.

In the meantime, I've filed #276 and will make sure to include a fix for this in the next release (v0.6.1.). Thanks for bringing this to my attention, and sorry for the inconvenience.

Breakthrough · 2022-07-11T03:12:20Z

@research-boy do you also need to set cv2.CAP_GSTREAMER when you are opening the cv2.VideoCapture? In v0.6.1 I may instead add the ability for you to create a VideoStreamCv2 from an existing cv2.VideoCapture.

Thanks!

research-boy · 2022-07-11T06:15:23Z

@Breakthrough

Ah thanks for sharing, that definitely needs to be fixed. This check needs to also see if there is a ! in the path: https://github.com/Breakthrough/PySceneDetect/blob/master/scenedetect/backends/opencv.py#L296= As a workaround, you can fall back to using v0.5, or comment the above check out temporarily.

In the meantime, I've filed #276 and will make sure to include a fix for this in the next release (v0.6.1.). Thanks for bringing this to my attention, and sorry for the inconvenience.

Ya , the video feed is getting initialized right now after commenting out those lines.

@research-boy do you also need to set cv2.CAP_GSTREAMER when you are opening the cv2.VideoCapture? In v0.6.1 I may instead add the ability for you to create a VideoStreamCv2 from an existing cv2.VideoCapture.

Thanks!

cv2.CAP_GSTREAMER is not necessary , opencv by default recognizes the argument accordingly

After the video is initialized i am getting another error:

[ WARN:0] global /opencv-build/opencv/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
[ WARN:0] global /opencv-build/opencv/modules/videoio/src/cap_gstreamer.cpp (1217) getProperty OpenCV | GStreamer warning: unhandled property: 6
Video codec detection failed. If output is incorrect:
  - Re-encode the input video with ffmpeg
  - Update OpenCV (pip install --upgrade opencv-python)
  - Use the PyAV backend (--backend pyav)
For details, see https://github.com/Breakthrough/PySceneDetect/issues/86
Traceback (most recent call last):
  File "tm_sd.py", line 11, in <module>
    scene_manager.detect_scenes(video=cam, callback=lambda frame: print("a new scene: ", len(frame)))
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/scene_manager.py", line 801, in detect_scenes
    self._process_frame(position.frame_num, frame_im, callback)
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/scene_manager.py", line 676, in _process_frame
    cuts = detector.process_frame(frame_num, frame_im)
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/detectors/content_detector.py", line 147, in process_frame
    frame_score = self._calculate_frame_score(frame_num, curr_hsv, last_hsv)
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/detectors/content_detector.py", line 98, in _calculate_frame_score
    delta_h, delta_s, delta_v, delta_content = calculate_frame_score(curr_hsv, last_hsv)
  File "/usr/local/lib/python3.6/dist-packages/scenedetect/detectors/content_detector.py", line 48, in calculate_frame_score
    numpy.abs(current_frame_hsv[i] - last_frame_hsv[i])) / float(num_pixels)
ValueError: operands could not be broadcast together with shapes (1080,1920) (243,360) 
terminate called without an active exception
Aborted (core dumped)

Breakthrough · 2022-07-11T22:58:24Z

Would you be able to file a new bug report for this in the issue tracker? I would like to get some more information from you about how to reproduce this, but would rather start a new issue specifically for this case (and keep the discussion there). Thanks!

If possible, in the bug report, could you include some steps for how I can reproduce the issue? I am not familiar with gstreamer or pipes (do they work with the opencv-python package from pip or did you compile OpenCV yourself?), so if you could point me in the right direction for how I can reproduce the issue, that would be very helpful.

It would also be useful to know if this issue exists on v0.5 as well, or if this is something that fails only in v0.6. I can certainly investigate further into that if you can point me in the right direction, or provide some steps for how to set this up.

Edit: Also, if I am reading the error correctly, it seems like the stream somehow returned two frames of different sizes. Are you using any kind of filters or anything which might change or crop the frame?

Breakthrough · 2022-07-12T02:39:24Z

I've updated #276 as well for the next v0.6.1 release so this should work more smoothly by letting people pass their own VideoCapture objects using an adapter:
https://github.com/Breakthrough/PySceneDetect/blob/7913228/tests/test_api.py#L133-L153=
And also included various examples of this in the manual.

Breakthrough · 2022-07-15T20:59:48Z

Do you know why those frames are smaller than the rest of the video, and is it always the same number of frames? Is there an easy way I can reproduce this on my end?

PySceneDetect requires each frame to be the same size, so you may want to skip over the first few frames until you start getting frames with the correct size. In v0.6.1 there will be a new adapter that lets you use a cv2.VideoCapture object directly:
http://scenedetect.com/projects/Manual/en/v0.6.1/api/backends.html#devices-cameras-pipes

In the meantime, if you are still using a VideoStreamCv2 object, you can call read() several times until it starts returning frames with the correct resolution, and then call detect_scenes. This will effectively skip that part before doing the scene detection.

Hope this helps!

research-boy · 2022-07-16T05:45:14Z

Do you know why those frames are smaller than the rest of the video, and is it always the same number of frames? Is there an easy way I can reproduce this on my end?

PySceneDetect requires each frame to be the same size, so you may want to skip over the first few frames until you start getting frames with the correct size. In v0.6.1 there will be a new adapter that lets you use a cv2.VideoCapture object directly: http://scenedetect.com/projects/Manual/en/v0.6.1/api/backends.html#devices-cameras-pipes

In the meantime, if you are still using a VideoStreamCv2 object, you can call read() several times until it starts returning frames with the correct resolution, and then call detect_scenes. This will effectively skip that part before doing the scene detection.

Hope this helps!

@Breakthrough as of now i am not sure why this is happening, i tried getting the frame directly in gstreamer itself , still it's same, it's look like that how the plugin is initializing the frame

live callbacks for live-streams.

727ce65

mhashim6 requested a review from Breakthrough February 3, 2020 13:37

guard for callback.

d347d14

Breakthrough reviewed Feb 29, 2020

View reviewed changes

Breakthrough requested changes Feb 29, 2020

View reviewed changes

Breakthrough added the status: accepted label Feb 29, 2020

mhashim6 added 2 commits February 29, 2020 18:12

remove unnecessary call for len.

4646867

proper callback type.

ad5353c

This was linked to issues Jun 27, 2020

Support for Live Video Stream (Non-Terminating) Processing Mode #5

Closed

Extra frames show up when using split-video with ffmpeg #159

Closed

Breakthrough removed a link to an issue Jun 27, 2020

Extra frames show up when using split-video with ffmpeg #159

Closed

pass frame_num in callback.

23b8dc2

Breakthrough added in progress feature and removed status: accepted labels Aug 8, 2020

Breakthrough added this to the v0.6 milestone Aug 8, 2020

Breakthrough changed the base branch from master to v0.6.x August 8, 2020 17:11

Breakthrough approved these changes Aug 8, 2020

View reviewed changes

Breakthrough merged commit ab7222d into Breakthrough:v0.6.x Aug 8, 2020

Breakthrough added status: completed and removed in progress labels Aug 9, 2020

Breakthrough removed this from the v0.6 milestone Jan 10, 2021

Breakthrough added this to the v0.5.5 milestone Jan 10, 2021

Breakthrough added a commit that referenced this pull request Jan 10, 2021

Merge pull request #151 from mhashim6/live_callback

8b79bc8

live callbacks for live-streams. Resolves #5.

Breakthrough mentioned this pull request Jul 7, 2022

Missing usage example for live stream/camera #273

Closed

Breakthrough mentioned this pull request Jul 11, 2022

VideoStreamCv2 does not work with gstreamer pipelines #276

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

live callbacks for live-streams. #151

live callbacks for live-streams. #151

mhashim6 commented Feb 3, 2020

mhashim6 commented Feb 3, 2020 •

edited by Breakthrough

Loading

Breakthrough Feb 29, 2020

Breakthrough Feb 29, 2020

Breakthrough left a comment

Breakthrough Feb 29, 2020

Breakthrough Feb 29, 2020

mhashim6 Feb 29, 2020

Breakthrough Aug 6, 2020 •

edited

Loading

mhashim6 Aug 6, 2020

mhashim6 commented Feb 29, 2020

Breakthrough commented Aug 8, 2020

mhashim6 commented Aug 8, 2020

Breakthrough commented Aug 8, 2020

mhashim6 commented Aug 8, 2020

research-boy commented Jul 7, 2022 •

edited

Loading

Breakthrough commented Jul 7, 2022 •

edited

Loading

Breakthrough commented Jul 7, 2022

research-boy commented Jul 8, 2022

Breakthrough commented Jul 11, 2022

Breakthrough commented Jul 11, 2022

research-boy commented Jul 11, 2022 •

edited

Loading

Breakthrough commented Jul 11, 2022 •

edited

Loading

Breakthrough commented Jul 12, 2022

Breakthrough commented Jul 15, 2022

research-boy commented Jul 16, 2022

live callbacks for live-streams. #151

live callbacks for live-streams. #151

Conversation

mhashim6 commented Feb 3, 2020

mhashim6 commented Feb 3, 2020 • edited by Breakthrough Loading

Breakthrough Feb 29, 2020

Choose a reason for hiding this comment

Breakthrough Feb 29, 2020

Choose a reason for hiding this comment

Breakthrough left a comment

Choose a reason for hiding this comment

Breakthrough Feb 29, 2020

Choose a reason for hiding this comment

Breakthrough Feb 29, 2020

Choose a reason for hiding this comment

mhashim6 Feb 29, 2020

Choose a reason for hiding this comment

Breakthrough Aug 6, 2020 • edited Loading

Choose a reason for hiding this comment

mhashim6 Aug 6, 2020

Choose a reason for hiding this comment

mhashim6 commented Feb 29, 2020

Breakthrough commented Aug 8, 2020

mhashim6 commented Aug 8, 2020

Breakthrough commented Aug 8, 2020

mhashim6 commented Aug 8, 2020

research-boy commented Jul 7, 2022 • edited Loading

Breakthrough commented Jul 7, 2022 • edited Loading

Breakthrough commented Jul 7, 2022

research-boy commented Jul 8, 2022

Breakthrough commented Jul 11, 2022

Breakthrough commented Jul 11, 2022

research-boy commented Jul 11, 2022 • edited Loading

Breakthrough commented Jul 11, 2022 • edited Loading

Breakthrough commented Jul 12, 2022

Breakthrough commented Jul 15, 2022

research-boy commented Jul 16, 2022

mhashim6 commented Feb 3, 2020 •

edited by Breakthrough

Loading

Breakthrough Aug 6, 2020 •

edited

Loading

research-boy commented Jul 7, 2022 •

edited

Loading

Breakthrough commented Jul 7, 2022 •

edited

Loading

research-boy commented Jul 11, 2022 •

edited

Loading

Breakthrough commented Jul 11, 2022 •

edited

Loading