Everything you need to know about non-linear... but were too afraid to ask!
Non-linear Production Technologies and Techniques
The broadcast and film industry is no different to any other: as new technologies are introduced, its operators have to adapt quickly. Inevitably mistakes are made. And like other sectors (and for understandable reasons), everyone keeps quiet about their slip-ups. This is a great shame: with little sharing of negative experiences, best practise can be slow to evolve.
Tapeless production is a case in point. It is in its infancy and sound advice is still hard to come by. Because of this there are some significant knowledge gaps when it comes to non-linear acquisition, data transfer and archiving. This article's objective is to fill in some of these gaps.
Today, most tapeless production is carried out on solid state memory cards, magnetic drives and to a lesser extent optical disks. The main formats are:
- Panasonic P2 cards
- XDCAM EX using SxS cards (ExpressCard), a standard adopted by Sony and JVC
- SD cards (HC - high capacity, XC - extended capacity and UHS - ultra high speed)
- Compact flash
- Solid state hard drives
- XDCAM disks
There certainly is a bewildering diversity of formats which makes investing heavily in a particular format a risk. However, despite a confusing picture of conflicting codecs and formats, tapeless acquisition nevertheless delivers many significant benefits over tape and film production.
The rapid pace of tapeless development has had another advantage: length of shooting. Housing up-to 5 64Gb P2 cards, a camera such as the Panasonic AJ-HPX3700G can record continuously for over 6 hours. Moreover, its card slots are hot swappable so the camera could potentially run for days on end - perhaps even indefinitely. And because the systems have no tape mechanisms, there are no moving parts, no fear of ageing heads and less potential for dirt fouling the cameras. Tapeless camcorders go longer between services - meaning greater availability of equipment.
The plus points don't stop there: when shooting with a 16GB E series card in AVC-Intra 100 720/24p, the P2 HD transfer speed into post is 20:1, which allows the import of 40 minutes of recorded video in under 2 minutes. The SxS format runs at the same high speed of 800 Mbs. As a result, crews routinely save hours on ingesting and digitizing compared to tape.
The miniaturisation of the cards themselves and their near commodity pricing have everyday practical implications. Safeguarding these small and relatively inexpensive cards from loss or incidental damage is a real consideration, as they are by their very nature, mission-critical. Optical disks, such as XDCAM HD, are less likely to be lost in a kit bag and they are potentially more robust. However, their larger dimensions and slower speed have meant that manufacturers have concentrated their R&D on pocket-sized solid state memory.
On remote location shoots, tape-based camera crews have typically securely stored their tapes before shipping them home for ingest at the studio. This is a simple, proven workflow, even if it is expensive, wasteful and slow.
For tapeless shooters, things are not so straight-forward, although the rewards are many. There are numerous reasons to move to tapeless systems - greater speed, lower media costs, higher capacities, increased functionality and reusable media. Still, the increased amount of data stored, the reusing of cards several times during a shoot and the creation of multiple non-linear files rather than a single linear track means that there are hazards that must be avoided. Working towards a best practise and preparing for likely pitfalls are key. Mistakes can and do happen.
Cards at the Ready
Low-cost, reusable media needs a new mindset that ignores the very fact that they are inexpensive! Cards should be labelled for easy identification. They are to be reused so they won’t therefore have a tape roll number, but keeping a track of all the cards with a logical labelling system is no less essential. The media should be kept safe and securely boxed.
P2 and SxS cards have the security measure of a write protect switch on them; it is very important to make sure this feature is activated before placing a card into any reading device. Not only is this a handy tag to indicate that the card is ready for ingest but it also prevents computers writing back to the card, thereby potentially corrupting it.
Pro media cards are highly durable and able to withstand harsh environments but while damaging them is difficult, losing them or trying to use a full card is all too easy. Entrusting data management and transfer services to a reliable crew member is advisable.
Beyond basic, practical housekeeping, there are a number of technical reasons why card management is essential. In the case that files span multiple cards, they must be transferred to archive at the same time. Failure to do this could mean that clips are not complete; data can be lost. Secondly, where different frame rates and slow-motion are used, this footage should not be captured onto the same cards as media holding normal frame rate content. Separating these types of files will give the editors the opportunity to set up their workspace correctly and deal with different frame-rate material logically.
An alternative to using the recording system built in is a third-party device mounted on the camera. Instantly, virtually any legacy tape-based camera can be brought into the non-linear world. These recording modules still use memory cards - typically low cost CF cards (such as the 32Gb or 64Gb SanDisk Extreme Pro) which can be imported using a high speed Sonnet Qio or standard card reader. Therein lies their first advantage: cost of the physical media and of the content transfer process.
While potential cost-savings are important, technologies such as Convergent Design's Nano Flash and AJA's Ki Pro Mini deliver considerable advantages over the camera manufacturer's system. Firstly, production teams have the opportunity to acquire in different formats, side stepping a manufacturer's codec constraints. Data can be recorded to accommodate a particular editing system, or with long-term archiving in mind. Indeed, it is possible for these units to achieve both: attach a Ki Pro Mini to a Panasonic HPX-3700 and the camera operator can record natively in-camera in AVC-Intra 100 while the external recorder captures content in Final Cut Pro-ready Pro-Res. A secondary benefit of dual format acquisition is therefore that the production will meet the requirement to create a secure data backup.
Using the fastest UDMA 6 compact flash cards, these systems have the speed to capture Pro-Res 422. The latest CF cards can reliably sustain write these high speeds and with rugged and waterproof products on the market, this is a formidably versatile media. This level of performance enables camcorders, such as the Panasonic AG-AF101, to record at 50Mbs using 4:2:2 colour space and thereby qualify as a BBC HD broadcast camera. Users need to pay particular attention, however, to the CF card they select: the Ki Pro Mini, for example specifies only a handful of compliant types. Even with less discerning DSLRs and the Nano Flash, care should be taken that a given card can handle the speed required the chosen format requires. Most manufacturers include recommended models on their websites, and it is advisable to check these out before embarking on a shoot.
Nevertheless, the drawbacks of these recorders are few. Although they add weight to the set-up, on smaller camcorders such as the Sony EX3, many find that the unit balances better when configured with a recorder and battery. The issue of synchronising the camera with the external recorder is avoided by configuring the capture data to record when the timecode is changing, so provided the camera is set to record run (not free-run) timecode, then, triggering the camera to record onto its own card will activate the external unit automatically.
A Note on Metadata
Planning ahead in this manner is equally critical when considering metadata - extra information added to the recorded material. Thanks to the IT protocols on today's systems are based, metadata is now standard throughout the workflow has arrived in the camera world.
Clips captured as files can now be appended with additional data that can speed up post production and make archiving material far more logical. There are a number of tags that are automatically created in-camera such as time and date so there is no need to record the roll number as with tapes. Standard tags include the camera name, camera operator, programme module, scene number and much more. Data technicians can also modify metadata parameters to capture the specific information they need.
Editors and media managers can then quickly search for clips taken at certain times on particular days, in specific locations; scenes can be searched by keyword, scene number or camera name. This kind of added functionality can shave off hours in the editing workflow. For media managers and content library owners, it is a godsend.
If time is spent at the outset considering what data tags will needed in post, the metadata can be built into the workflow at the start. In this way editors can readily find the clip or the camera angle they need to build their story. For future programming, the metadata process is even more vital: automated metadata will help avoid the painful process of reviewing every file looking for a pertinent clip in the future.
Quirks and Errors
New versions of edit systems are often still in need of fine tuning at the point they are released by manufacturers. New camcorders will not have been tested with every version of a computer operating system. Software programme updates may make corrupt ingested material or throw up new and unexpected error messages. Examples are many, and painful: when Panasonic introduced the AVC Intra codec, it was found to be incompatible with AVID systems... crash! In the early days RED famously caused technicians to undertake time-consuming data wrangling to make the system work smoothly. With the arrival of Final Cut Pro X, the tests begin anew.
Similarly, to rely blindly on cameras to transfer data faultlessly to a hardware back-up device is a leap of faith. Many have fallen foul of unusual errors or glitches because they omitted to test their particular set-up before going out on a shoot.
The basic lesson is crucial: do not leave post and media transfer to chance. Test your set-up in advance and plan how your content will be ingested and edited in post, before you begin shooting.
It is usual practise for the data recorded on the camera to be regularly transferred to a separate device to free up space for filming, to allow for instant editing of the rushes and to create secured backup copies.
The most common approach is to use an existing laptop for this process. Armed with a card reader which connects via a dedicated slot, a USB, firewire or eSATA port, data can be transferred painlessley, with copies sent to external hard drives. New on the scene and built into the latest PCs and Macs are high-speed Thunderbolt i/o and USB 3 ports - innovations that are quickly being supported by third party vendors and which are dramatically increasing transfer speeds.
The plug and play nature of modern computers makes it tempting to drag and drop files from the camera onto the machine. This is not advisable as crews can lose the delicate file structure of the card which can make the rebuilding of the media very difficult, if not impossible! Also, this approach does not support importing spanned clips, where media is spread over 2 cards. This is an issue with certain editing programmes: although they feature the facility to import data from cards, there are a number of quirks and hazards that have caused productions to lose data or unnecessarily complicate the ingest process. The reliable, trusted approach is to use the manufacturer's dedicated tools where available, such as Panasonic's P2 contents manager software or the XDCAM clip browser software. These apps typically allow technicians to view footage and to have immediate access to the metadata.
Dedicated transfer and storage devices are popular tools for archiving data. Tools, such as Nexto DI 2525, Panasonic HP-G10/G20, Panasonic Panasonic Rapid Writer or a dedicated XDCAM EX device (the Sony PXU-MS240 which comes with a removable 240GB hard drive cartridge) are designed specifically for data transfer. They are efficient, reliable and relatively affordable.
High-end specialist tools, like the MARVIN, offer very sophisticated archiving features: the MARVIN 800 can handle 800 GB per day with 12 TB of onboard RAID storage. It automates the creation of backups, LTO (linear tape-open) masters, QuickTime proxies for offline editing and DVD dailies as well as shot logging. As you can imagine, this is not a cheap solution!
Finally, while some Panasonic cameras (such as high-end P2 models, such as HPX-3700) can be used as P2 playback devices to record direct to HDD. Unfortunately they are quite slow. Far better are the new achive devices, such as LT04 archive system from Cache-A or the new AJ-SF100 software from Panasonic that archives P2 content to Blu-Ray, linear tape or hard drives.
On the issue of hard drives, it's important to understand files structures. The FAT32 structure is compatible with PC and Mac and is used by most solid state cameras, such as all P2 and XDCAM cameras, as well as DSLR cams. However, it is hindered by a 4GB file size limit so larger clips may have to be spanned over different files. While the UDF file structure is free from this limitation, it is incompatible with FAT32. Therefore if ARRI's Alexa or the Sony PMW-500 are set to record in UDF configuration, its cards can only be copied to a UDF-compatible device, such as the Nexto DI 2525 UDF Special. In short, the file format of the camera's capture card must match the file format of the archive device.
For security, many broadcasters demand that at least 2 copies of all media are made as soon as possible. Any devices reliant on hard drives should be tested before the shoot. These can have a curtailed life span if left on the shelf. If unpowered for lengthy periods they are more susceptible to areas of data loss due to thermal decay. One technology that is increasingly popular for backing up tapelss productions is... tape! Cheap, dependable and high capacity, a Linear Tape-Open (LTO4) cassette can hold 800Gb very cost-effectively (under £40). Perfect for large jobs, the newly introduced LTO5 tapes can store 1.5TB and operate at up-to 140Mbs.
The Data Transfer Process
As long as the individual steps have been thought through and tested, the process itself is now remarkably straight-forward. With a labelled memory card full of clips (complete with rich metadata information), a device ready to receive the data and the format tested to ensure that it is edit-ready, the transfer can now take place. To facilitate easy matching of clips and fast transfer to post, all the media should be stored together. The final stage is quickly to test and verify the media before it is used again.
The Real World
Zeb Chadfield, Head of Online at Clearcut Pictures, managed a series called Escape from Scorpion Island from pre-production right through to final delivery. Two series were shot back to back, filming for 6 hours per day on 9 Panasonic P2 cameras, 12+ mini cams and 6 Go-Pros.
To organise the high quantity of P2 rushes, Zeb set up a laptop to programme all metadata for the camera set-ups on a daily basis. He used individual SD cards to load these settings into all cameras at the start of each day. Consequently, each camera shot media and captured unique metadata – camera number, location, day and episode number.
The cameras were all checked when they came back to be ingested to be sure that the correct metadata had been loaded. Then they were transferred via Panasonic's P2 Rapid Writer, copying all the media to fast, mirrored storage. At the end of each day's ingesting, one drive went to a secure location and the other was put aside to be sent back to Clear Cut Pictures at the end of the week - an archive of the whole week's rushes.
Once at Clear Cut the media was consolidated to their ISIS shared storage, managed into projects and grouped for multicam cutting. It was then edited at full resolution in Avid Media Composer. As episodes were finished in offline, they were transferred tapelessly to its film master grading system then back to Avid Symphony for online and layed off to tape for delivery.
The complexity of the workflow and amount of data stored required a combination of logistical expertise and technical ability. The metadata captured enabled the crew, from end-to-end, to access and manage each file quickly and accurately and meet their deadline.