Introduction
The AWS Export App in VIDIZMO enables the migration of content files to a designated S3 Bucket. You can create a criteria in the application that determine which type of Portal content is targeted and what content files are exported. In addition, the AWS Export App gives you options to modify the export activities carried out on your Portal. For instance, you can automate the export process or delete it after it undergoes an export activity.
Concept
The AWS Export App functions as a means of exporting specified file types of your Portal content to an AWS S3 Bucket, without the content being affected. It acts as a one-way bridge that you can modify or control via its configuration.
The AWS Export App can be activated and configured from your Portal's application settings. During configuration, you enter your AWS credentials, assign an S3 Bucket to contain the exported files, define the export criteria and preferences if you are opting for automatic export, and lastly select the content files you want to export.
Automatic Export
A crucial setting to consider during configuration is the Automatic Export option. If this option is on and the AWS Export App is enabled, then the content on your Portal that meets the export criteria will undergo the export activity immediately. The automatic export activity takes place on the eligible content once, the content does not undergo the activity again unless the criteria is changed, or the application is re-enabled. The AWS S3 Export app does this by initializing a continuous workflow.
The workflow continuously monitors all content on your Portal, including content added or created through processing, ingestion, upload, clipping, and more. If the content meets the export criteria, then its content files are automatically exported to the destination S3 Bucket. However, you have the option to make the automatic export activity exclusive to new Portal content, see Export Preference below for more.
To understand this, let's assume that the Export App is configured to export the thumbnails and supporting files of transcoded content on a Portal with automatic export turned on. A user uploads a video titled "ABC" to the Portal. The Export App does not consider it for export as it does not meet the transcoded condition defined in the export criteria in the application's settings.
However, when the user processes ABC for transcoding, it results in a state where it now meets the export requirements. The Export App then exports its thumbnails and supporting files to the destination S3 Bucket, according to its configuration.
1. Export Preference
Inside the application configuration, you will also find the option to set an Export Preference for the content present on your Portal when automatic export is turned on. This setting determines which content will be eligible for automatic export depending on when the Export App was enabled or disabled.
Two options are available to control this setting. You can enable the application with the Export Preference set to "Already Present." By doing so, the application will automatically export the content files of all content present on the Portal according to the defined export criteria. This also applies to past Portal content, and export-eligible content added to the Portal after the Export App has been enabled.
Alternatively, you can choose "Export After Enable" to restrict the automatic export activity only to content that is added to your Portal after the Export app is enabled from the Integration App page.
2. Post Export Options
Users can determine what happens to their Portal content after it goes through an automatic export activity. The application can be configured to either delete the content and move it to the Portal's Recycle Bin or keep it as it is by performing no action.
Manual or On-Demand Export
Aside from automatic export, users can export the files of their Portal content manually, which is also referred to as "on-demand" Export. The options to perform an on-demand export are available regardless of whether automatic export is enabled or disabled. You can also manually export content regardless of the Export Preference selected in the application settings.
The user can perform a manual export on Content present in their Portal's Evidence Library or Media Manager. However, the export app only exports the content files selected in its configuration.
Manual Export also allows you to "update" the content files on your S3 Bucket. For instance, if new files were generated when an already exported content was processed, then performing an on-demand export on its content will export those newly generated files to the content's named folder in the S3 Bucket.
Export Flow
When an export activity begins (either manually or automatically), the Export App generates requests according to the Portal content selected by the user or the defined export criteria. For each request, the Export App adds a record to your Portal's Activity Logs.
A log or record provides details, such as the status of the export activity. Initially, the status of the activity is "Pending," which means that the content is in queue and being checked to see if it meets the export criteria. If the criteria is met, then the export activity proceeds with the content files being exported to the destination S3 Bucket and the tag being updated to "Processing." However, if the criteria is not met, the export request is rejected, and the log is marked as "Rejected."
After the export activity is finished, the status tag is updated to "Complete," "Failed," or "Content Not Found," depending on whether any issues were encountered.
You can view the progress of your export activity from your Portal's Activity Log. The log also consists of all instances of export activities that took place on your Portal. It provides detailed information for each instance of export activities, including the number of exported files, errors encountered, who initiated the export activity, and more. You can learn more here: Understanding App Activity Logs.
Creating Batches for Load Management
If there are numerous requests for exporting content, then the Export App has a system to handle them efficiently. During an export cycle:
- It creates batches of all the export requests it receives for a specified duration. The size of a single batch can be up to 100 requests.
- Scans a batch and checks if the selected Portal content meets the export criteria.
- It approves the eligible ones for processing and rejects the rest.
- The application then divides the approved processing requests into chunks, which are then processed in parallel.
- When all of the requests in the batch are processed, the Export App then moves on to process subsequent batches.
- After processing all the batches, the application waits for a specific duration (5 minutes) before starting the next export cycle.
Export Status
You can find out which Portal content is undergoing export activity (both manual and automatic) by looking at the dynamic tag on its thumbnail. The tag also updates according to the status of the export activity on the content, here are the following status it can have:
- Exporting: Indicates that the export activity is in process.
- Exported: Indicates that the export activity for the content has been completed successfully.
- Not Exported: Indicates that the export activity could not be completed due to an error.
These tags will only be visible on your Portal's library if the AWS Export App has been enabled. You can also use these tags as search filters in your Portal's library to search for content affected by an export activity.
Note: Once the content is updated with the "Exported" status after an automatic export activity, the application does not run the automatic export activity on that specific content again.
Webhook Notification
Additionally, you can set up Webhook notifications for the export activity on your Portal. To do this, add “Exported” as an Event on the Webhook App you've created from the Integration Apps page. Visit How To Create VIDIZMO Webhook App for more information.
Analyzing Folder Structure For Export
Another intuitive functionality of the AWS Export App is to export files of your Portal content while they are present in a Case or Folder.
Exporting a Folder Hierarchy Ingested from an S3 Bucket
If you're exporting content that was ingested from an S3 bucket with its folder hierarchy preserved, then the application will recreate the same folder hierarchy on the destination S3 bucket.
If the folder hierarchy contains content that's eligible for export, then the application will export its files to the destination S3 bucket. The exported files will be present on the same path (or location) as the original content in the exported folder hierarchy.
If the content's files (namely supporting files) are present in another folder within the hierarchy on the source S3 bucket, then the application preserves its location during the export activity.
With the entire folder hierarchy maintained on the destination S3 bucket, the supporting files are present in the same folder, path, or directory as the source. However, if any new files are created when the export-eligible content is processed, they will be saved in the same folder that contains the original content.
Figure 1 illustrates how content present in a folder hierarchy on a source S3 bucket was ingested to a VIDIZMO Portal and then exported to a Destination S3 Bucket.
.
For the example in Figure 1, the Export App has Original Files, Supporting Files, and other file types selected in its Export Criteria. This is the reason why you see the original files and the supporting files present on the destination S3 bucket. The folder hierarchy on the Destination S3 Bucket also has the Original Files and additional Content Files that were generated during a processing activity.
Exporting Content Present in a VIDIZMO Folder
When a user exports a VIDIZMO Folder containing Portal content, the Export App creates a folder on the destination S3 Bucket for each eligible content it finds inside the folder. The exported content will be organized into folders with the same name as the corresponding Portal content, making it easy for users to identify which folder contains which files. It then exports the specified file types inside of those folders according to the Portal Content.
If the same content is processed for an activity that creates new files and then exports them, then the named folder on the destination S3 bucket will be updated with the latest files.
Figure 2 shows how the content files are exported to the Destination S3 bucket when the export-eligible content is present inside a folder hierarchy in VIDIZMO.
For the example in Figure 2, the user has chosen to export the Original File and other file types, like Transcoded files or Thumbnails. The folders on the Destination S3 Bucket are named after the Portal content that underwent export activity. All the relevant content files belonging to a specific piece of content are stored in the respective named folder.
Exporting Content Present in Case-Folder Hierarchies
Another important consideration for the export process is the content contained within a Case-folder hierarchy. If the user has configured the app for automatic export with the Formats field containing only “Folder”. Then, the Export App will run the export activity on the content meeting the export criteria if they are present in the folder across the portal. This also applies to:
- Folder-Folder Hierarchies where a child Folder containing export-eligible content is present inside a parent Folder.
- Case-Folder Hierarchy where a Folder containing export-eligible content is present inside a Case.
If the user decides to re-configure the application and selects “Case” as a value in the Formats field instead of “Folder”, then the application runs the export activity on all eligible content present in Cases across the Portal, even if they are present inside a Folder or Folder hierarchy.
To run the export activity on eligible content present in both Folders and Cases, the user needs to add both “Case” and “Folder” as values for the Formats field.
For a complete guide on configuring and utilizing the Export App's functionality, visit: How to Export Content Files using the AWS Export App.