Our standard procedure to process livestreams
As the team that is in charge of publishing content through our social media channels, we usually have to take time (or ask the Design Team for help) to process live streamed events (e.g.: Flock, Release parties, Fedora Week of Diversity), which usually takes a lot of time to do manually. First to download all of the videos, then to manually cut them all up and then to upload it all back, so we decided on a standard procedure for future events, in order to ease the work for our contributors, specially for beginners.
-
yt-dlp (if you prefer to use the command line) / Parabolic (if you would rather use an interface);
-
Sponsorblock (optional, but will spare you a lot of work later on);
-
Inkscape and its Next Generator extension for working with the video thumbnails.
1. Create the video chapters and separate the breaks
First, you will need to tag the timecodes for all of the talks in the video. This is by far the most time consuming part of the whole process.
We recommend Sponsorblock to do so, first because it offers a nice interface, and second because you do not need admin access to the event or project’s YouTube channel to tag everything. The extension maintains a public database that people can use (usually to remove ads inside of videos, but in our case, we use it to remove breaks between talks).
1.1 Using Sponsorblock
First, press the shortcut ; or click the icon shaped like a play button inside a shield, which the extension creates right in the lower right corner of your video player, along the other video controls, to create a segment.
If you hover over the button instead of clicking it, another button with an ''i'' icon should appear. If you click it, a sidebar with all of the video’s segments and a couple of options should appear, which we will use later:
From there you can click one of the three icons that takes its place:
The first one (called Open Submission Menu), with the shape of an arrow, opens the submission menu, where you can see a summary of all the segments you created, edit them and upload them to the database.
The second one (called Cancel Created Segment), with the X shape, cancels the segment you started creating.
The third one (called End Segment Now), with the stop symbol, marks the end of the segment you created.
Now that you started a segment, find the point of the video you would like to end your segment (be it a break or a talk) and either press ; again or click the End Segment Now button.
After that, click the Open Submission Menu button, so you can choose a category. The ones we will use are "Chapter" and "Intermission/Intro Animation", the former for the Chapters we will later use to export to YouTube and the latter to have the breaks auto-magically cut up.
You can upload segments one by one or en masse by clicking the "Submit" button in the lower right corner, the latter of which we recommend, so that you can more easily double check and edit everything.
The end result should make your video’s timeline look like this:
Now, open the extension sidebar, and select the option to export segments.
1.2 Creating the YouTube Chapter Timestamps
Sponsorblock helps a lot to ease the burden when it comes to chapter creation (as more of the process can be completed by people who dont have permissions to access the YouTube channel), however, YouTube uses a different format than Sponsorblock to define timestamps, so you will need to reformat the output before adding them to the video.
1.2.1 Python Script
To save some effort compared to manually editing timestamps, a python script has been made to automatically create the chapters in the correct format. This script is designed to use the timestamps from Sponsorblock, as well as information from an extended version of the thumbnail CSV. This extended csv format will be referred to as the "stream CSV". Each row of the stream CSV is intended to represent one of the non-intermission segments of the event as-streamed, so if the livestream suffered a power outage that caused the opening remarks to get split up, the opening remarks may have multiple entries in this file
To get started with this script, first clone its repository and cd into the scripts/timestamps directory.
There you will find a README file with instructions for how to set up and run the script.
The only thing the script needs in order to run is the Youtube link to the livestream being processed. This allows the script to rely on the intermissions already tagged in Sponsorblock from the previous steps.
If no stream CSV file is provided, the script will generate sequential names automatically, for example:
0:00:00 Talk #0 0:31:59 Intermission 0:32:14 Talk #1 0:47:47 Intermission 0:47:55 Talk #2 [...]
Then the names of the talks can then be manually edited into this output prior to uploading to YouTube.
If you have access to the stream CSV containing the names of the talks in the livestream, you can pass that into the script to have it automatically fill in the names of the talks for you.
1.2.2 Manually editing timestamps
If using a script is not something you are comfortable with, you can also manually reformat the timestamps in a text editor. VSCode/VSCodium is great for this due to its multicursor editing features, but any text editor will work.
For example, here is the output of one of Flock to Fedora 2023’s streams:
0:00.000 - 12:22.000 Intermission 12:22.436 - 27:25.232 Keynote: Fedora Mentor Summit kickoff and reflection on mentoring in Fedora community 27:25.232 - 2:00:26.540 Panel: Mentoring and mentorship best practices in Fedora 2:00:26.540 - 4:58:40.000 Intermission 4:58:40.742 - 5:31:21.672 Why You Should Join a Community 5:31:21.672 - 5:55:26.775 Intermission 5:55:26.775 - 6:40:51.611 State of Fedora (without stream issues this time lol) 6:40:51.611 - 7:33:59.086 Intermission 7:33:59.086 - 8:15:25.041 Conference closing
…you only need the starting timecode formatted in hours : minutes : seconds, so it should look like this:
0:00 Event Starting 12:22 Keynote: Fedora Mentor Summit kickoff and reflection on mentoring in Fedora community 27:25 Panel: Mentoring and mentorship best practices in Fedora 2:00:26 Break 4:58:40 Why You Should Join a Community 5:31:21 Break 5:55:26 State of Fedora (re-recorded due to stream issues the first time) 6:40:51 Break 7:33:59 Conference closing
1.3 Adding the chapters to YouTube
After formatting the timestamps, they need to be added to the video’s description. This causes them to appear in the video for everyone (even people that do not use Sponsorblock).
Adding the timestamps to the video can be done by anyone with at least the permission to edit the description (i.e. anyone with at least Editor level permissions on the YouTube channel). If you are processing a fedora event, there is a list of people who have access to the Fedora YouTube channel.
You will need it for…
2. Downloading the talks and splitting them up
|
Due to the way all modern video codecs work, its quite difficult to quickly make cuts in a video at precise timestamps without going through a lengthy re-encoding process. The methods below may yield trimmed chapters that can be split at slightly different points than intended. If this happens, you may be able to try running the video through LosslessCut to see if that works. |
2.1 Using yt-dlp
cd into the folder you want to save the video in and run the command:
yt-dlp --sponsorblock-mark all --sponsorblock-remove all --split-chapters $VIDEO_LINK
…where $VIDEO_LINK is the link of the video you want to download.
It should download and automatically remove all of the breaks and split the video based on its chapters.
2.2 Using Parabolic
First, open the app and either use the keyboard shortcut Ctrl + , or click the hamburger menu on the top right and select "Preferences" to open the preferences menu.
After that, turn the "Use SponsorBlock for YouTube" option on:
Now, close the preferences, click the "Add Download" button, paste the link to the video you want to download and click the "Validate" button:
It will present you with a page containing download options. In there, click the "Advanced Options" option:
In there, turn the "Split Chapters" option on:
After all of that, select your preferred download options, your destination folder and click "Download". You should have all of your talks properly split up by their chapters in your desired folder, ready to upload by the time your download ends. If you used Parabolic you’ll likely have videos for the breaks as well, so watch out for that.
3. Creating the thumbnails
Thanks to the work of the design team, we now have a way to automate the process of thumbnail creation as much as possible. This part will be a quick summary of the tutorial linked above, for the full thing in detail, check it out!
First, create a spreadsheet in your preferred app for that (we recommend LibreOffice), you just need to have something that outputs a proper .CSV file.
In it, create two columns, one with the name of the talk and another with the name of the presenter(s). The first line of each column should have a variable name to use later on.
It should end up looking something like this:
| TalkName | PresenterNames |
|---|---|
Why Fedora is the Best Linux |
Colúr and Badger |
The Best Talk You Ever Heard |
Dr. Ver E. Awesome |
… |
… |
Save that file with the .csv termination.
Now, open Inkscape and start creating the design of your thumbnail template, taking into consideration some of the important rules of design, like visual hierarchy, color theory (specially taking into consideration your project’s color scheme).
After you have a basic layout of the template, it is time to prepare your template for the information.
First, change the talk name to %VAR_TalkName% (change TalkName by whatever variable name you decided to use in your .CSV file);
Now change the presenter name to %VAR_PresenterNames%.
You will probably have to edit the text box sizes in order to account to longer talk names and multiple presenters, trial and error will probably be your friend here.
Once you have a result you are comfortable with go to Extensions > Export > NextGenerator, add the path to your spreadsheet in "CSV file", define your preferred file format to export and path to export your files to, set "File name Pattern" to %VAR_TalkName and, finally, click "Export".
Your files should be good to go in an instant!
4. Editing thumbnail intros into the video (optional)
Having an intro before your talk isn’t the best for viewer retention but if it is short enough it can be a good way to show the branding of your project and your event.
If you choose to include a few seconds of some kind of branding image, such as a thumbnail, at the start of each of the video chapters, there is another python script that has been created in order to save some effort compared to manually editing videos. This is not needed but can be a nice touch
This script takes in a .ini` config file and a CSV file that is based on the one from the thumbnail creation process.
In addition to the talk name, speaker, and thumbnail filename from the thumbnail CSV, this extended version, referred to here as the "talk CSV", also contains the filename of the downloaded video chunk so that the script knows where to find it when processing.
This inclusion of the downloaded chunk filename is currently a manual process, but will hopefully be automated in the future.
Each row of this talk CSV is intended to represent one of the non-intermission segments of the event.
While order of events does not matter for this CSV file, any recombining of segments that needs to happen (for example if the opening remarks were interrupted by a power outage), should be done first before being included in this file.
Unlike the previous script for processing timestamps, this script requires that a CSV file be present. The README file of the script contains more details and examples for setting up the CSV and INI configuration files, as well as running the script.
Having an intro before your talk isn’t the best for viewer retention but if it is short enough it can be a good way to show the branding of your project and your event.
If you choose to include a few seconds of some kind of branding image, such as a thumbnail, at the start of each of the video chapters, there is another python script that has been created in order to save some effort compared to manually editing videos. This is not needed but can be a nice touch
This script takes in a .ini` config file and a CSV file that is based on the one from the thumbnail creation process.
In addition to the talk name, speaker, and thumbnail filename from the thumbnail CSV, this extended version, referred to here as the "talk CSV", also contains the filename of the downloaded video chunk so that the script knows where to find it when processing.
This inclusion of the downloaded chunk filename is currently a manual process, but will hopefully be automated in the future.
Each row of this talk CSV is intended to represent one of the non-intermission segments of the event.
While order of events does not matter for this CSV file, any recombining of segments that needs to happen (for example if the opening remarks were interrupted by a power outage), should be done first before being included in this file.
Unlike the previous script for processing timestamps, this script requires that a CSV file be present. The README file of the script contains more details and examples for setting up the CSV and INI configuration files, as well as running the script.
What we still want to automate
Thumbnail CSV creation
Since Pretalx is beginning to get more use as a CFP and talk scheduling platform, there is a strong likelihood that the data it provides (in various formats such as JSON and XML) can be used to create an initial version of the thumbnail CSV much earlier in the process. This would help take some work off of the design team when automatically creating thumbnails and would also allow speakers a chance to enter their preferred name into pretalx and verify that it is correct to help reduce the need for re-dos later
Chapter creation
Sponsorblock helped a lot to ease the burden when it comes to chapter creation (as it used to be limited to people with direct admin access to the channel), but it is still a lot of manual work that we would like to ease.
Talk CSV creation
The talk CSV currently needs to be manually edited with the paths of the downloaded video chunks before it can be fed into the script to edit the thumbnails into the video. This is another area where it would be nice to provide some automation in the future to make the process less labor-intensive.
Captions
YouTube automatic captions have gotten better with the years, but are still not perfect, and the situation doesn’t get better when you select the option to automatic translate them. This doesn’t paint a great picture for us, as a global project, to alienate potential contributors that do not speak English, but we understand that manually transcribing entire events and then translating those captions is a massive undertaking, and we current do not have ideas on how to automate that process.
End Cards
One thing you can do for your events is having different artwork for the event opening and the breaks in between talks, and use the latter as a place to add end cards to your video (which helps a lot with continuous viewership of your channel’s content).
We would like to have an alternative for cases where this is not planned ahead, where it would work similarly to the intro case, but with an image file to the end of the video (and be able to define a number of seconds that image would be shown).
Want to help? Learn how to contribute to Fedora Docs ›