Blog / Resources / Media, DIT and Streaming Best Practice for avoiding Data Loss and strategies to fix if you do Media, DIT and Streaming This article gives some useful tips to prevent data loss from happening and to know what to do if you are unlucky enough to experience it. We all know the importance of the data and footage we shoot, either for yourself or for that important client. Should the unthinkable happen and you lose your data, whether it be by deleting the wrong file or formatting the wrong card, seeing the dreaded ‘card error’ message, or worse still your card becomes corrupted and unreadable, it can be a stressful time. This article gives some useful tips to prevent this from happening and to know what to do if you are unlucky enough to experience it. Section 1 – Common causes and preventing Data Loss Preventing the loss of data in the first place is vital. Memory cards can be fragile and deserve respect, since data loss is at best inconvenient and worst a disaster, so here are some useful tips to avoid this from happening to you. Use only reputable Brands Ensure that you only use brands from reputable manufacturers like Sony, Sandisk etc. Don’t buy cheap cards online that are priced lower than they should be, as these are probably counterfeit cards, perhaps not of the true capacity or integrity. Use a compatible card reader to transfer your files. These devices are easy to set up and use and make the transfer of photo and video from your camera faster, and also prevent problems that can occur with camera battery issues. Most card readers are inexpensive, but it is worth checking it is compatible with your exact card and paying a little extra if you shoot important data and on high end cards. Some SD readers for example do not support SDHC or SDXC cards, so check. Don’t pull the card out while it is in use Whether it is in the camera or in a reader attached to the computer, do NOT pull the card before it is finished being written to as this will cause trouble. Any missing data will cause an error that the camera can’t overcome. Sometimes it will cause a corruption and tell you that you need to reformat the card. Even when you think the camera or computer is done writing to the card, particularly with HD video due to buffering, allow an extra minute or so to be sure and turn the camera off before removing the card. Environmental concerns Temperature extremes can ruin memory cards, so beware! Even specialty cards rated for extreme temperatures are only good to about 50˚C (120˚F) so try not to expose your devices to extreme heat. Be wary of water too. Fresh water may not be a disaster as long as the card is dried before it is plugged into the device so it won’t short out but with salt water, it becomes more complex due to corrosion. Any flash card that gets wet needs to be dried thoroughly. Moisture and dirt are also causes of failure. The contact points on memory cards can become dirty and cause a poor connection or short. If you can see the connectors, you can keep them clean them with a couple of drops of isopropyl alcohol on a cotton-tipped swab. The bad news is if you can see the contacts, you can also touch them and this can cause a static jolt that can destroy the card too, so avoid touching the contacts and always keep the device in a protective case when it is not in the camera or card reader. Taking the memory card out of one camera to use in another model of camera This may sound harmless but it changes the format and change the file structure. Every device has its own format and numbering sequence. Have a card specifically for each device you use, so if you take your camera card and use it in your MP3 player, phone or another device, it will have to be formatted in the camera you then want to use it in before use. If you must use the same card in different devices, save all of the data to your computer first and then format the card in the new device. Use the write-protect function on cards Many professional cards have write/protect tabs which prevent data being written. As soon as you remove a card from a camera, activating the write/protect tab will prevent your PC/Mac or camera from being able to write to the card. This may minimise overwriting existing data or even causing a corruption, good practise if you work in a busy environment and someone could pick your card up. If this tab exists, then do use it. Low batteries in the camera Keep an eye on battery levels. If you do transfer data by camera and the battery goes too low during the downloading process you risk losing your data and possibly corrupting the card. Low batteries during shooting can also prevent all of the necessary information from getting to the card. Do not use a camera with low batteries or you risk losing the last file as it may not finalise, or worse still, the whole card due to a corruption. Static Watch out for carpets and static, because static electricity can easily corrupt a card. Physical damage Dropping or bending your card will cause problems, breaking circuits, loosening components or worse still damaging the internal memory chip itself, so eject and handle with care. Do not place your card in the back pocket or let the dog near it! Again, always keep the device in a protective case when it is not in the camera or card reader. Backup Strategy and Making a Disaster Recovery Plan Although memory cards can fail and become corrupted before or during backup, where possible your data can also be protected by automating backups. Individuals can setup automated external hard drive or cloud services to back up on a schedule. By establishing routine backups, you can ensure they will lose the bare minimum of data in the event of a failure. Individuals should follow the lead of businesses by establishing a formal written disaster recovery plan. This can be as simple as setting a routine for data backup, or instituting multiple levels of redundancy such as cloud backups combined with external HDDs kept at a separate address. This is known as the 3-2-1 theory of backup. Establishing a plan will make data recovery simple and efficient. Using cloud services is a great way to get off-premises backup, especially for large video files. The costs of cloud storage have plummeted, so think about using more than one service for redundancy. Consider storing financial or sensitive information on multiple password protected HDDs instead, as cloud services can possibly be exploited by hackers, so combine and use multiple backup options. Section 2 – Media Card Types and Best Practice Whatever device you use there are good practises to help avoid issues and here are some things to consider depending what type of card you are using. Micro SD cards: Handle with care. A cracked or physically broken MicroSD card means you will almost certainly lose your data. Does your card reader support your type of SD card? SD card =2GBSDHC = 4GB – 32GBSDXC = 64GB+ Some card readers do not support some SD cards even though the slot size is the same, so it can cause corruption. Always keep the card in a protective case when not in the camera or card reader. Always source from a reputable seller to avoid counterfeits. SD Cards Handle with care. Some SD cards can have a flimsy casing. Inside is one or more memory chips, damaging these chips may mean you lose your data or worse still make data recovery impossible. Does your card reader support your type of SD card? SD card = 2GBSDHC = 4GB – 32GBSDXC = 64GB + Some card readers do not support some SD cards even though the slot size is the same, so it can cause corruption. Always keep the card in a protective case when not in the camera or card reader. Always source from a reputable seller to avoid counterfeits. CF cards Check periodically that the pins on your card reader, and camera, are lined up and how they should be. If you see a bent pin in your reader it is time to replace it. If you see one in your camera it is time to see an approved repair centre or contact the manufacturer. Also, handle with care, although the casing is generally metallic and sturdier than SD cards the device still contains electrical components and one or more memory chips, which can be damaged causing data loss. Broadcast cards – CFast/Cfast2 cards, P2, SxS, and XQD Cards As professional and broadcast camcorders have evolved, so too has the media technology that they use. The development of technologies like XQD™, P2 Express, XDCAM/XDCAM Pro + & CFast™ address the increasing capabilities of today’s cameras, as well as the ever-growing demands placed on professional photographers and camera operators shooting bigger and bigger files. Make sure your camera can utilise all the speed your card can deliver. Consult your instruction manual or search the manufacturer’s website for the fastest card speeds supported, many will also recommend specific cards and warn of incompatible cards. Also, check the card reader can support your type and speed of card to lower the chances of a corruption. Ensure the card is not removed from any device while it is still writing. Always know what file system your device is. Many broadcast cameras use the exFAT or UDF file systems, always check you are following the guidelines and if using a new card it is good practise to format it in the camera you are using to prepare it for that environment. SSD Drives As cards become larger, the distinction between memory cards and SSDs becomes blurred. Many of today’s high end cameras record to SSD drives, as do many off-board HD video recorders as well. Unlike hard disk drives, solid state drives store data in flash memory chips. Data loss can occur with SSD storage devices due to physical damage to the flash chips and how data is logically stored on them. Here are some common failure types with SSD drives Electronic component failureController chip failureFlash cell degradation from natural usePower surges or failuresDamage to printed circuit boardsDamage to connectorsData corruption after firmware updates Additionally, SSD storage devices are not immune from traditional data loss events such as human error, computer viruses, natural disasters, and software/program corruption. Handle with care. Keep an eye on your battery levels and follow the guidelines from the camera manufacturer. Section 3: What to do if you lose data Even the best-laid data protection plans can be thwarted by bad luck, corruptions or failures. If you are confronted with a data loss there are only limited options available to and it is possible that you can do more harm then good from trying to recover data where a loss has occurred. I myself remember a situation where upon mounting a P2 card that I knew already had data on it into a Mac, was presented with the asking me to ‘Initialise a Drive’. That seemed like a good idea at the time… At this point, if this happens to you, then you should stop what you are doing and contact the manufacturer or a reputable data recovery company for advice. If the data recovery company is any good, then they will offer you free support and can guide you to pursue various strategies which may help you to recover your media without cost to you. Losing data due to human error such as deletion or formatting may sometimes be recoverable using specific software tools but this is by no means a caveat for cavaliar data management. Whilst there are many free file and data recovery utilities, that is often a “get what you pay for” proposition, as these tools can be riddled with malware and do not offer any level of support, and they are often useless with today’s more complex files and file systems. Finally, If your issue is more complex or serious than a simple delete, such as a seemingly dead or badly corrupted device, and the data is of importance, then you may require sending the device off for a lab recovery. Sometimes, this is the only option to you. Over the years, we have had many clients who have believed their data to have been irretrievable but still managed to recover their data in full. Our advice is simple – if your data is important, then seek an expert! VMI recommend LC Technology who are experts in data recovery of files stored on flash media and hard drives and and they do offer free support. At the very least, call them to use their free support and see if this helps you to rectify your situation with a DIY solution. The following guide to remedial data recovery was written by LC Technology and we are grateful for their input into this article and important area. Section 4 – The LC Technology Guide to Remedial Data Recovery LC Tech provide both software tools and lab services, to over 100 countries and are recommended by many of the major manufacturers of both memory cards and cameras, including SanDisk, Canon and Sony, as well as suppliers to the BBC and ITV, due to their experience and expertise with reading memory chips. LC Technology has written tools for the likes of SanDisk and Toshiba for over a decade, and will guide you through the process.Their tools can be tested as an evaluation and unlike many freeware solutions they have tools suitable to recover complex HD video recoveries, which typically need to be recovered in their original folder structure.Their FILERECOVERY software can be downloaded here on a try-before-you-buy evaluation basis for PC or MAC.LC Technology can perform chip (NAND) reads, whereby the memory chip is removed from the device to attempt to access the raw data for reassembling.A complex forensic data recovery if the data is of value is a real lifesaver. Click here for full details of LC Technology services. Barry Bassett, VMI and Grant Woods, LC Technology, October 2016 Related articles WORKING RESOLUTION: CONSIDERATIONS & BEST PRACTICES Cameras, Media, DIT and Streaming Netflix have published a very useful article, which takes an in depth look at the concept of a working resolution and outlines the best practices for image scaling throughout the production lifecycle… Brand New VMEDIA on-line database V2 Media, DIT and Streaming, Technology With so many different recording formats, memory card types, media sizes, compatibility issues and upgrade requirements, it can be a challenge trying to find compatible memory cards or SSDs for your camera or recorder. However, Lewis has been extremely hard at work to make this easier for you with the VMEDIA Online Database V2 which is organised into the most popular brands and sortable by camera model, which gives details on compatibility, data rates and approximate recording times too Types of Apple Prores Codecs Media, DIT and Streaming A brief introduction to the various types of Apple ProRes formats with a link to download the full Apple White Paper.