Backing up critical data
Jun 1, 2009 12:00 PM, By Brad Gilmer
Follow these steps to keep your facility on-air.
When it comes to protecting data, broadcasters have it rough. Not only do they face all of the normal IT issues with protecting important desktop business data, but increasingly they are faced with protecting vital data having to do with on-air operations.
As broadcasters think about protection, it may be useful to divide the areas of concern into protecting on-air programming, meaning video, audio and closed captioning, and protecting the data having to do with the operating systems that get video to air, meaning automation and other systems.
Protecting programming
Figure 1. Many different RAID schemes exist. The two most commonly encountered by broadcasters are RAID 1 and RAID 5. RAID 1 mirrors all data onto two disks. RAID 5 stripes the data across multiple disks. You may hear about RAID 0 as well. RAID 0 stripes data across multiple disks to provide increased performance, but provides no redundancy in case of disk failure.
Select image to enlarge.
For many broadcasters, especially those just moving to server-based playout technology, it's important to decide how to protect the programs and commercials stored on the server. Most broadcasters keep tape as a backup for a while, and in some cases, videotape is kept as the backup method indefinitely. However, for many broadcasters, the move to server-based playout was driven by a requirement to do things that could not be done with a tape-only facility. Some of these drivers are a request from management to reduce staff, or to provide multiple feeds from a consolidated master control facility.
Whatever the motivation, broadcasters frequently find that after being on servers for a while, videotape is unsuitable for backup. Or they discover that videotape serves as a backup of last resort, meaning that the facility functionality will ultimately suffer if the operators must revert back to videotape. Finally, there is a cost to maintaining videotape as the backup of choice. This cost shows up in maintaining the videotape library, tape machines and sufficient tape-based playback facilities to actually be able to execute a tape-based backup plan. If retaining videotape as a backup is not feasible for many broadcasters, then what is a good way to protect this vital data once it moves into the server environment?
The obvious answer is a one-for-one backup plan; all content on one server is duplicated on a second server. This backup plan is simple and intuitive, and the only real issues are cost and ensuring that the two servers really contain identical content.
If you can't deploy redundant playback servers, all is not lost. When thinking about protecting data, it pays to decide what the most likely failure modes are and protect them. The two components most likely to fail in servers are the power supply and the disk drives. Buying a server with redundant power supplies is a good idea, especially if you do not have a backup server. When it comes to disk drives, professional quality broadcast servers employ a scheme that provides protection from disk drive failures, most typically through RAID technology. There are different levels of RAID, but generally, RAID 5 is used. In this configuration, data is striped across several disks. As the data is written, parity data is calculated and recorded as well. If a drive fails, the RAID array can recreate the missing data by using the data from the remaining disks plus the parity data. Figure 1 on page 24 illustrates RAID 5 along with two other common RAID configurations.
An important point when considering RAID is that if a drive fails, the technology is so good that the operator will never notice unless he is looking at error logs or is notified by monitoring software. If a second drive fails before the first one is replaced, the failure will be catastrophic: All data on the drives will be lost. So it is extremely important to take advantage of all notifications provided by the server manufacturer regarding disk failures.
It is also key to understand what it takes to replace a drive and to rebuild the data so that the drive is back online. Many servers will rebuild the data on the new drive in the background, but some will not. Also, note that it may take a long time for the process to complete. This means that the system is vulnerable during the rebuild period. Another question to ask is whether the system performance is restricted during the rebuild process. In other words, if you lose a drive and replace it with a new one, are there things you cannot do because the server is busy rebuilding the drive?
To further protect the data, some RAID systems allow you to purchase extra drives and designate them as hot standbys. This means that a drive is instantly available if another drive fails — well, not quite. Actually, the drive is instantly available, but it does not contain any data. The system must still rebuild the data on the new drive. But in this case, you will reduce the exposure to a second drive failure by beginning the rebuild process immediately rather than waiting until someone notices the error and physically replaces the failed drive. Understanding RAID and its limitations can be an excellent way to protect yourself from drive failures in video servers.
Protecting the most important data
As I said earlier, when broadcasters think about protecting data, they almost invariably think about protecting the content. But there is other data that may be even more important than protecting the content itself. Most facilities contain some form of automation. In some cases, an automation failure means that normal operations cannot continue, and the on-air look suffers.
Again, a one-for-one backup is a simple solution, but in this case, not only is cost an issue, but controlling one server from two automation systems presents nontrivial issues. Also because most automation systems operate from a centralized database, it's critical to ensure that this centralized database remains available and that its data is protected.
One way to protect a centralized database is to store the data on a RAID drive, just as you would your content. While this may be sufficient, consider what it would mean if you lost all of the automation data. How long would it take you to reingest all of the material? How long would it take you to enter all of the metadata required to get the content on the air? This database is so important and the ramifications of losing it can be so serious that a belt-and-suspenders approach is appropriate.
Many different database technologies exist. One technology is SQL. It supports something called database replication. Replication allows the SQL server to duplicate the database onto a database running on another server in near real time. In fact, it is possible to design server/database systems such that the loss of one database or server is transparent to the application accessing the database.
Although database systems attempt to protect you from data corruption, errors can occur. And, of course, few systems will protect from human error. For these reasons, it is important to maintain a rolling backup of the database system. How frequently you back up this database depends upon how quickly it changes, but I would recommend making backups at least once every 24 hours and keeping each backup for one week.
| Want to use this article? Click here for options! |  |
