Implement multithreading

[MythicManiac]

Implement multithreading

Implement multithreaded clip rendering to file
Allow for more CPU utilization especially on heavily edited clips

Refs #584, #139

[MythicManiac]

Python 3.6 Travis seems to fail on something unrelevant to the changes

[MythicManiac]

Utilizing all 12 of my cores with this implementation allowed me to go from 3 it/s to 24 it/s, effectively resulting in 800% rendering speed and cutting my render time from 3.5 hours to 26 minutes.

The current implementation is rather gimmicky, but the performance boost is very much worth exploring

[Zulko]

I agree, great research. Multithreading is not always a great advantage for various reasons (sometimes the ffmpeg encoding is the bottleneck, and sometimes the make_frame function already uses all the core) but in your case it seems to make a big difference.

Some remarks: the user should have the choice of multithreading in e.g. write_videoclip. If the user decides to use only one thrad it should be the current moviepy code running, and if more, fall back to your parralel workers. Also in the final commit please make sure that the code is clean, pep8, and that there is no additional files like test2.py. Can you also add some formal tests in the test suite ?

Thanks for this contribution !

[MythicManiac]

Yeah I agree, threading should definitely be an opt-in. By my research the best ratio of python threads to ffmpeg threads is 1:3, so in my case I gave 9 threads to ffmpeg and 3 to python.

EDIT: Ratio is heavily dependant on what kind of edits you have applied

Ideally I think the write_videoclip should be able to take in ffmpeg thread count and python thread count individually. Maybe to avoid breaking backwards compatibility just adding a new parameter for moviepy_threads could make sense.

Right now I'm trying to tackle the issue of subprocesses not exiting if the main process is killed (CTRL+C for example). For some reason calling sys.exit() or os._exit() doesn't actually exit within a subprocess.

[coveralls]

Coverage decreased (-0.7%) to 56.49% when pulling 0196ac4 on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[coveralls]

Coverage decreased (-0.7%) to 56.49% when pulling 0196ac4 on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[coveralls]

Coverage decreased (-0.7%) to 56.475% when pulling f0a2c22 on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[MythicManiac]

Unfortunately I don't think I can get this to work in a very user friendly manner. The issue is serializing/deserializing the clips so that they can be re-created on the parallel processes. Naturally we want to open new file handles on the parallel processes, and this is the part that becomes problematic.

[MythicManiac]

This is the best and most user friendly solution I could come up with. Can be improved if someone figures out a good way to serialize/deserialize clips.

Usage example:

# Single threaded
def concat_clips():
    files = [
        "myclip1.mp4",
        "myclip2.mp4",
        "myclip3.mp4",
        "myclip4.mp4",
    ]
    clips = [VideoFileClip(file) for file in files]
    final = concatenate_videoclips(clips, method="compose")
    final.write_videofile("output.mp4")

# Multi threaded
from moviepy.multithreading import multithread_write_videofile

def concat_clips():
    files = [
        "myclip1.mp4",
        "myclip2.mp4",
        "myclip3.mp4",
        "myclip4.mp4",
    ]
    multithread_write_videofile("output.mp4", get_final_clip, {"files": files})


def get_final_clip(files):
    clips = [VideoFileClip(file) for file in files]
    final = concatenate_videoclips(clips, method="compose")
    return final

[MythicManiac]

Added a simple test now, let's see if it works

@Zulko thoughts on the current state?

[coveralls]

Coverage increased (+0.5%) to 57.681% when pulling 767b6ee on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[coveralls]

Coverage increased (+0.8%) to 57.956% when pulling c89117d on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[MythicManiac]

Everything is cleaned up and seems to be working correctly now, let me know what you think

[coveralls]

Coverage increased (+0.5%) to 57.671% when pulling 18e1bd8 on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[coveralls]

Coverage increased (+0.8%) to 57.97% when pulling 18e1bd8 on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[coveralls]

Coverage increased (+0.8%) to 57.97% when pulling 18e1bd8 on MythicManiac:multithread-wip into a90d4ab on Zulko:master.

[MythicManiac]

@Zulko sorry for bugging you, but could I get any comments on this? 😄

[mgaitan]

this looks like a code smell to me. It's very ugly and hard to read.

[MythicManiac]

I do more or less agree, what would you suggest as an alternative?

Additionally, I never did figure out the actual cause of this broken pipe error, but I assume it has to do with how Python handles multiple processes and passing data between them.

[mgaitan]

I'd use dedent with a unique multiline text

[alexebbs]

With the usage example you posted I'm getting the error: AttributeError: 'VideoFileClip' object has no attribute 'endswith'
Any ideas what could be causing this?

[mbeacom]

Based on your reported error, it's trying to perform a str operation against the VideoFileClip object.
Can you provide the trace? @alexebbs

[MythicManiac]

@alexebbs I can't remember any longer if I ever tried actually running my usage example, however the test does work. You can take example off of it here: https://github.com/MythicManiac/moviepy/blob/18e1bd817b6e554319567b82f921fa6f09ab0530/tests/test_multithreading.py

[zamirkhan]

Great work @MythicManiac - any potential performance improvement would be welcomed in this camp. I'm using moviepy for tasks that can take > 1 hour depending on the inputs. Now I don't know if multithreading would help, but would love to try. Is this PR on the roadmap?

[MythicManiac]

I would say this PR is only suitabe as a proof of concept, I ran into several issues while running it myself using my own fork. There definitely is performance to be gained, especially when there are lots of effects involved, but I have doubts of this implementation being the proper way to achieve that.

@zamirkhan I'd suggest you to try out the fork and see how it works for you if you have lots of edits in your clips. Multithreading moviepy sadly doesn't offer much advantage if there's not much edits applied to the clips you're processing, in that case more threads should be allocated to ffmpeg.

[jingxuanlim]

Hey @MythicManiac, could I ask what's the status of this implementation?

I also found multi-threading to not work with VideoClip.write_videofile(). My understanding from Zulko in another thread (#584 (comment)) is that if making the frame, which runs on a single thread, is slower than the encoding, then multi-threading, which is used for the latter, won't improve efficiency.

Does you fix help implement multi-threading on the frame-making bit?

[MythicManiac]

@jingxlim multithreading does make rendering faster, but what the optimal ratio between ffmpeg and moviepy threads is depends on how many effects you have and what kind of effects are they.

The implementation in this PR has a few issues, but it did work for me when I needed it. It however has not been updated since I opened the PR, so results may vary.

[shenhavmor10]

hey @MythicManiac can you help me with the multithreading on moviepy with your library ?
this is my email : [email protected]
i just cant figure out how you exactly used it and it does not work for me.. it says 'CompositeVideoClip' object is not callable.
id love to get some help from you please..
thanks !

[MythicManiac]

@shenhavmor10 I'm not able to provide support on this given that it's already over 2 years old and was never merged, not to mention it was more of a proof of concept anyway.

https://github.com/MythicManiac/moviepy/blob/18e1bd817b6e554319567b82f921fa6f09ab0530/tests/test_multithreading.py the test here is a working example, refer to that and the code itself.

[mondeja]

Since this was a proof of concept and is not clear that could be suitable for moviepy, I'm closing this for now. Anyone, feel free to use it to implement some multithreaded rendering system. Anyway, thank you for the attempt @MythicManiac!