Building a scalable MAM system

Resources and quotas

Most media processing jobs require access to other systems and software to carry out meaningful work. For example, moving a file requires access to a storage resource, and transcoding a file requires access to transcode hardware and software. So the key to enabling scale is to understand the availability of resource.

The resource object, like an action, has a type such as processing, storage or I/O. Each type has a subtype, such as transcode. For example, processing resources could be transcoding and QC, and I/O resource could be network services for file transfer. This subtype can then be implemented for various specific resources. With a transcode, resource implementation would be a specialized software adaptor for delegating transcode requests to a transcode engine.

The benefit of the resource adaptor model is that it provides an elegant interface to the job scheduling layer, where all available resources and resource types are made available for consumption. This is where multitenancy comes in, as one needs to control access to resources to ensure that files, folders, transcodes, etc., can only be utilized by named users or roles.
 
So with the use of actions, jobs and resources, you have a layered and extensible approach to managing media assets. You can create new action types and use a whole range of networked resources for supporting all kinds of media management tasks.

However, with a resource-hungry job scheduler, executing potentially thousands of jobs, how do we ensure we make the most of our expensive resources? How do we ensure that jobs don’t get starved of resource or that resources don’t go idle? The compute power of a resource is limited, and over time, demand will inevitably exceed supply. Intelligent scheduling decisions can significantly improve the effectiveness of media processing with more jobs being run with quicker turnaround. Subject to the constraints of the traffic control and mission policies, it is the job of the scheduler to use whatever freedom is available to schedule jobs in such a manner so as to maximize system performance.

The missing piece in controlling job access to resources and, therefore, making the most of the resources available, is the quota. One might assume that the only parameters that are useful for orchestrating access to a resource are named user, priority, start time and end time. All are useful, but they are far too coarse-grained for advanced scheduling.

A quota encapsulates a user’s access to a resource. The rules governing a user’s access are retained within the quota, thus making the architecture pluggable and configurable. Remember the importance of multi-tenanting?

A quota is abstract in the sense that it encapsulates different rules for different resource types.

For example, in the case of storage or folder resources, a quota represents total storage allowance (gigabytes) allowed to a certain user. It may contain a threshold for raising alerts when a user is close to reaching their limit.

In the case of network resources, quotas can be allocated in terms of bandwidth. This might be a consistent rate, contended per session or throttled at times of the day.

In the case of processing jobs such as transcoding, this becomes much harder to implement as the time taken for a transcode job can depend on the input format, the output format and a huge range of transcode parameters.

In the case of processing a transcode resource request, the quota may throttle total jobs per user or total concurrent jobs per user. Again, the quota may be configured or implemented to allow access at certain times of day.

The beauty of a pluggable model is that access to precious processing resources can be throttled in an unlimited number of ways. A quota could look up access to another system to see if a user has paid his or her bill or whether he or she is a bronze, gold or platinum user. Bringing cloud technologies into the quota and resource layer enable the concept of elastic computing. This makes resource scheduling much easier as the job scheduler can instantiate new resources and tear them down as and when required.

In summary, the key to scalability, extensibility and powerful media management is the interaction of actions, jobs, resources and quotas. With the work being carried out by the Advanced Media Workflow Association (AMWA), this is a great opportunity for more standardized interfaces in the area of media processing, which can only be good in a large and rich media technology ecosystem.

---

—Jon Folland is CEO of Nativ.