VOD Transcoding and Packaging
About this guide
This tutorial walks you through how to use Eyevinn Open Source Cloud (OSC) to create a solution for creating VOD files for streaming and distributing through a MinIO storage service.
This solution is based on the following open source projects made available as a service in OSC:
About SVT Encore
SVT Encore is a scalable video transcoding tool, built on open-source giants like FFmpeg and Redisson.
Encore was created to scale and abstract the transcoding power of FFmpeg, offering a simple solution for Transcoding - Transcoding-as-a-Service.
Encore is aimed at the advanced technical user who needs a scalable video transcoding tool, for example, as a part of their VOD (Video On Demand) transcoding pipeline.
The version of SVT Encore run in Eyevinn OSC is compiled from sources available at https://github.com/Eyevinn/encore. This version may contain features that have not yet been incorporated in the upstream repository at https://github.com/svt/encore.
Getting started
Prerequisites
- If you have not already done so, sign up for an OSC account.
- You need to either have 4 services left on your subscription plan, or you will need to purchase the services individually for this solution.
- MinIO storage service in OSC or S3-compatible object storage solution
Step 0: Set up storage
Set up your storage where the Video On-Demand files will be placed. To set up a storage bucket in Eyevinn Open Source Cloud, follow the steps in the MinIO getting started guide. Enable public read-access to the storage bucket.
Step 1: Create Encore Queue
Go to the web user interface and navigate to the service called SVT Encore. Click on the button “Create queue” and give the queue a name, and fill in the desired parameters as specified by Table 1.
| Field Name | Description | Example Value |
|---|---|---|
| Name | Name for the SVT Encore instance | blog |
| Profiles URL | URL pointing to the list of transcoding profiles | |
| S3 Access Key Id | In this example, the RootUser you set when creating your MinIO instance |
{{secrets.objstorekeyid}} |
| S3 Secret Access Key | In this example, the RootPassword you set when creating your MinIO instance |
{{secrets.objstoresecretkey}} |
| S3 Session Token | If your S3 set up requires a session token, enter it here | {{secrets.token}} |
| S3 Region | Region of the S3 bucket | se-sto-1 |
| S3 Endpoint | Endpoint URL of the S3-compatible object storage | https://eyevinn-demo.minio-minio.auto.prod.osaas.io |
Table 1. Instance parameters for an SVT Encore instance
You can leave the Profiles URL empty for now and then press "Create".
Now you have an instance of SVT Encore running with one single queue, ready to receive transcoding jobs for processing. You can try this out by clicking on the menu item "Open API Docs" to access the online REST API documentation and submit a job.
|
|
|
The URL to the instance can be found on the instance card you just created. Press the copy button and save it for later. The instance URL for this example is https://demo-blog.encore.prod.osaas.io.
Step 2: Create a Valkey queue
Valkey provides a Redis-compatible key/value store, and we will use it to create a queue for the packaging jobs. Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and optionally a password, and press "Create". For this example, we will not give our instance a password.
Note down the IP and port of the Valkey instance card, and this is what will be the Redis URL that we will refer to later in this blog. The Redis URL is constructed by adding redis://in front of the IP and port.
In this example, it would be redis://172.232.131.169:10511. Save this for later.
Step 3: Launch an Encore Callback Listener
We will now create a separate service that can be used to monitor a transcoding job in SVT Encore, so we know when the file is ready to be packaged. Navigate to the Encore Callback Listener in the web user interface. Click on the button “Create callback” and enter the instance parameters according to Table 2. In this example:
- Redis URL:
redis://172.232.131.169:10511 - SVT Encore URL.
https://demo-blog.encore.prod.osaas.io
| Field Name | Description | Example Value |
|---|---|---|
| Name | Name for this Encore Callback Listener instance | vod |
| RedisUrl | Redis URL created from your Valkey instance | redis://172.232.131.169:10511 |
| EncoreUrl | URL to the SVT Encore service instance - without trailing slash | https://demo-blog.encore.prod.osaas.io |
| RedisQueue | Name of the Redis queue | package |
Table 2. Parameters for configuring an Encore Callback Listener instance
Important: the URL to the SVT Encore instance is without the trailing slash.
The URL to the callback listener in this case is https://eyevinn-vod.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback. Save the URL to your Encore Callback Listener instance for later.
Step 4: Create Encore Packager service
We can now move on with creating the Encore Packager service. Enter the instance parameters according to Table 3. In this example, we will have the following values:
| Field Name | Description | Example Value |
|---|---|---|
| Name | Name for this Encore Packager instance | vod |
| RedisUrl | Redis URL created from your Valkey instance | redis://172.232.131.169:10511 |
| RedisQueue | Name of the Redis queue, from the Encore Callback Listener instance | package |
| OutputFolder | Destination S3 path | s3://osc-tutorial/ |
| Concurrency | Number of parallel packaging tasks | |
| PersonalAccessToken | Your OSC personal access token. Located in OSC under 'Settings' --> 'API'-tab | {{secrets.osctoken}} |
| AwsAccessKeyId | The RootUser that you specified when creating the MinIO service instance | {{secrets.objstorekeyid}} |
| AwsSecretAccessKey | The RootPassword that you specified when creating the MinIO service instance | {{secrets.objstoresecretkey}} |
| AwsRegion | Region for the object store service | se-sto-1 |
| AwsSessionToken | Temporary session token for AWS credentials | |
| S3EndpointUrl | Endpoint URL of the S3-compatible object storage | https://eyevinn-demo.minio-minio.auto.prod.osaas.io |
| OutputSubfolderTemplate | Template for naming output subfolders | |
| CallbackUrl | URL to your Encore Callback Listener instance | https://eyevinn-vod.eyevinn-encore-callback-listener.auto.prod.osaas.io |
Table 3. Parameters for configuring an Encore Packager instance
Then press "Create" and wait for the instance to be ready.
Step 5: Submit a job
We will use an example video available at https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4.
Start the process by running this command in your terminal.
Navigate back to the SVT Encore service and press the menu item to open the API docs again. Click on the POST /encoreJobs bar and button “Try it out”.
Then, enter the following JSON in the Request body and press "Execute". The progressCallbackUri must point to our Encore Callback Listener for VOD packaging that we created.
{
"externalId": "tutorial1",
"profile": "program",
"outputFolder": "/usercontent/",
"baseName": "tutorial1",
"progressCallbackUri": "https://eyevinn-vod.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
"inputs": [
{
"uri": "https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4",
"seekTo": 0,
"copyTs": true,
"type": "AudioVideo"
}
]
}
And then press the button "Execute". Now a job is submitted, and if you want to see the progress, you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.
When the transcoding process is completed, it will place a job on the packaging queue that will be picked up by the Encore Packager service. When the packaging job is completed, you will, in this example, find a VOD package ready for streaming on the bucket specified for the Encore Packager instance.
Access to source file on S3 bucket
AWS S3
As described above, the transcode command points out the source file that Encore will use as the input for the transcode process. This file needs to be accessible via HTTP(S) by Encore, and in some cases, there may be a challenge to provide public HTTP(S) access to the source files. One reasonably easy way to do this is to share an S3 stored file via a pre-signed URL.
- Select the file you wish to share
- Click Actions -> Share with a pre-signed URL
- Enter a suitably short expiration time (make sure to allow enough access for the transcode process to complete)
Note: The presigned URL will be a very long string (including some "special" characters), so to make sure the transcode command works properly, please enclose the URL with quotes:
'https://.....very....long....url....'
Akamai Object Storage
To be added, how to create a signed URL to an object on Akamai Object Storage.
Transcoding files on your computer
Minio
To process files that are on your local computer, we will need an upload and input storage for the pipeline. For this, we will be using MinIO, an open-source project that offers an S3-compatible storage service and is available as a service in Open Source Cloud.
Step 1: Set up secrets
Create two service secrets in the MinIO service in the OSC web user interface.
user=guidepassword=guidepassword123
The user and password here are just to serve as an example.
Step 2: Create storage service
Create a storage service with this command in your terminal.
% osc create minio-minio transcoding \
-o RootUser="{{secrets.user}}" \
-o RootPassword="{{secrets.password}}"
We will now create a storage bucket in the storage service we created. First, we will download and install the MinIO client using Homebrew.
% brew install minio/stable/mc
Create an alias for the storage service you created, where <tenant> is the ID of your tenant.
% mc alias set transcoding\
https://<tenant>-transcoding.minio-minio.auto.prod.osaas.io \
guide \
guidepassword123
Added `transcoding` successfully.
Now we can create a bucket that we call input
% mc mb transcoding/input
Bucket created successfully `transcoding/input`.
Step 3: Upload and Transcode
Now we have everything in place to be able to upload the file and start the video processing (transcode) in the pipeline.
We will be using the AWS command line S3 client to upload the file. You can use any other S3 compatible client (e.g., Cyberduck) if you prefer. Set the username and password for the storage service as AWS S3 credentials in your terminal environment.
% export AWS_ACCESS_KEY_ID=guide
% export AWS_SECRET_ACCESS_KEY=guidepassword123
Upload video file and generate a signed HTTPS URL pointing to it.
% aws --endpoint-url=https://demo-transcoding.minio-minio.auto.prod.osaas.io \
s3 cp ./stswe-tvplus-promo.mp4 s3://input/
upload: ./stswe-tvplus-promo.mp4 to s3://input/stswe-tvplus-promo.mp4
% aws --endpoint-url=https://demo-transcoding.minio-minio.auto.prod.osaas.io \
s3 presign s3://input/stswe-tvplus-promo.mp4 \
--expires-in 86400
<signed URL>
Pipeline without packaging
If you don't want to create streaming packages and are only interested in transcoding to an ABR bundle you can create an Encore pipeline. In this pipeline the service Encore Transfer is used instead of Encore Packager, and for the actual transfer of the files the service Retransfer is used.
Prerequisites
This solution is based on the following open source projects made available as a service:
- SVT Encore
- Valkey
- Encore Callback Listener
- Encore Transfer
- Retransfer
After completed this tutorial you will be able to transcode a video file on an S3 compatible storage and the output is placed on another S3 compatible storage when the processing is completed.
Step 1: Create Encore Queue
Create an Encore Queue as described before.
Step 2: Create Valkey queue
Valkey provides a Redis compatible key / value store and this i what we will use to manage the queue for transferring files out from Encore and to out output bucket.
Navigate to the Valkey service in Open Source Cloud and press “Create valkey”. Give the instance a name and press Create.
Note down the IP and port to the Valkey instance card and this is what will be the Redis URL that we will refer to later in this blog. In this example it would be redis://172.232.131.169:10507.
Step 3: Launch Encore Callback Listener
Now we need something that monitors a transcoding job in SVT Encore so we know when the file is ready to be transferred. For that you navigate to the Encore Callback Listener in the web user interface. Click on button “Create callback” and enter the name of the instance, Redis URL (above), URL to the SVT Encore instance and the name of the transfer queue. We call it “transfer” in this example.
Press Create and you are done with this step for now.
Step 4: Setup secrets
Now we have the Callback Listener service running that will monitor transcoding job and place completed jobs in the transfer queue. Now we need a service that picks up a job from the transfer queue and actually transfers the file out from SVT Encore and to our destination bucket.
First we need to configure the transfer job service with API secrets needed for the access to the S3 bucket. Navigate to the Retransfer service in Open Source Cloud and click on the tab Secrets.
Create the secrets containing the Access Key Id and Secret Access Key for the destination storage access. Note down the name of these secrets as you will be using it later.
Now navigate to the Encore Transfer service in the web user interface and click on the tab Secrets. Add a secret with your personal access token (OSC token) that you find under Settings and the tab API.
Step 5: Create Encore Transfer service
When the service is created and saved we can now move on with creating the Encore Transfer service. Enter the name of the instance, Redis URL, name of queue in Redis (Valkey), output URL, OSC token and the name of the access key secrets in the Retransfer service. In this example we will have the following values:
Then press Create and wait for the instance to be ready.
Navigate back to the SVT Encore service and press the menu item to open API docs again.
Click on the POST /encoreJobs bar and button Try it out and enter the following JSON
{
"externalId": "tutorial1",
"profile": "program",
"outputFolder": "/usercontent/",
"baseName": "tutorial1",
"progressCallbackUri": "https://eyevinn-blog.eyevinn-encore-callback-listener.auto.prod.osaas.io/encoreCallback",
"inputs": [
{
"uri": "https://testcontent.eyevinn.technology/mp4/stswe-tvplus-promo.mp4",
"seekTo": 0,
"copyTs": true,
"type": "AudioVideo"
}
]
}
And then press button Execute. Now a job is submitted and if you want to see the progress you can go to the Encore Callback Listener service and open the instance logs to check that it is receiving the callbacks.
When the transcoding process is completed it will place a job on the transfer queue that will be picked up by the Encore Transfer service. And when all the transfer jobs are completed you will in this example find a set of files in your output bucket where you have set of different variants with different resolutions and bitrates.
Using the CLI
You can also use the OSC Command Line Interface to create a VOD file for streaming from using a pipeline named demo. Given that you have node and npx installed you can then run below command.
% export OSC_ACCESS_TOKEN=<personal-access-token>
% npx @osaas/cli vod create demo https://testcontent.eyevinn.technology/mp4/VINN.mp4