Tip:
Highlight text to annotate it
X
Jamie Lean - Weíll Need a Bigger Hard Drive!.wma
Jamie: Thank you very much Clifford. Kia Ora, Tena Katou Katoa. First, a word from my Sponsor.
The Film Archive, I'll just mention the Film Archive briefly because itís a slightly unusual
organisation, itís a charitable trust set up over 30 years ago to fill a gap in the
market of housing and restoring New Zealandís film heritage. We've since expanded our scope
to look after current film productions, television programming, advertising, home movies and
pretty much all forms of moving image.
We don't own the collection, itís all owned by depositors; they still retain ownership
and copyright, so we can't actually leverage the collection to help pay for its restoration.
We need to fundraise all the time. We have a really good cafe with excellent coffee and
we have a media library that you can go and sit in and watch New Zealand films but you
can't borrow them at the time. That's enough of the advertising.
Our operations are divided into three main areas pretty familiar to all the Archivists.
So the main focus of my talk is digital storage. If everything is digital in the near future
then there's a couple of questions we really need to answer now.
How we store our films. We've been making digital videos since 1997 on the brilliant
DV format and weíre still liking that a lot. We started off using it to record TV programming
for New Zealand on Air and we've been using it as a mezzanine format up until about a
year or so ago. Itís fairly robust for a very small tape, better than DAT, and we've
been storing digital films since 1999 when we received our first digital feature; New
Zealandís first digital feature in ëComfortable, Comfortableí, also deposited on DV tape.
Weíre not quite at the stage of receiving films by email but we do receive lots of digital
files on tape and on hard drives, of various configurations and with various issues around,
even accessing them. The Film Archive is a totally Macintosh based environment. So we
have some issues around that. The boss doesnít like to see anything but an Apple in the building,
so that's an ongoing battle for us.
The great bulk of our collection is still film and video. The process is to preserve
them now though, are almost all digital. The small gauge film, 8 ml, super 8 9.5 ml. We
now have a scanner in-house, so this is supplied by Damn Smart, our Conference Sponsors up
there; they do a really good job. So we can scan all those films individually frame by
frame. There's a laser in this that means that we don't have to worry about film that
is severely shrunk or damaged; it doesnít rely on sprockets. Basically we scan them
at roughly a 1k resolution, store them as individual .dpx files and then pump them out
to a Quick Time or JPEG 2000 movies.
These files live briefly on a MacPro that captures them and then we have to shove them
to another MacPro because there's not enough room to keep capturing and processing them,
then we process them and then we pump them out to LTO tapes. With feature films we outsource
the final preservation work to Park Road Post and create new film elements, new negatives
into positive sound negatives. These are close to the archival ideal of retaining the original
object, and weíre confident in the longevity of Polyester based film elements. In the past
it caused nitro film and acetate film of major issues which are what weíre grappling with
today. Hopefully Polyester won't develop any new diseases in the meantime.
Weíre hopeful that, stored correctly, it should be 300 years to 400 years. Again itís
a bit like the CD manufacturers; itís the manufacturerís concept of how long theyíll
last. I won't be around to worry about it. Well I do worry about it otherwise I wouldnít
be here. Even with this seemingly traditional preservation path, the features; almost everything
is changed. Much of the preservation work involves scanning the original film and then
doing restoration work on digital files and then outputting it back to film. So there's
no escaping the digital world.
So talking about ëWeíll need a Bigger Hard Driveí, itís just one of those moments ëcause
ëJawsí was one of those movies - when I was a kid, that it made me not want to go
swimming. It was also that moment where you have this sudden adrenalin rush when you realize
that you actually don't have the right equipment to do the job and oops, try not to panic.
Park Road Post rang to say they had a scan (well I can't mention the feature); they needed
to clear it off their servers and the owner of the film said that we could have it. The
problem was they asked, ìCould we stitch it back together at our place?î ëcause it
was over 7 terabytes and they couldnít fit it all on a single hard drive. The largest
hard drive at the moment, apparently, is about 4 terabytes. There's a few old pictures just
for entertainment sake, that 30 years ago 1 gigabyte was state of the art.
Typically our files returned to us from Park Road Post and where the digitals .dpx 10 bit
uncompressed Quick Time and JPEG. With videotape we transfer them in-house as well, again to
10 bit uncompressed files. At the moment weíre using Quick Time, theyíre not in a wrapper,
weíre going to have to move them on to something else eventually, but we very quickly have
to move them onto LTO tape.
The use of LTO tape occurred somewhat organically. It followed a traditional model of binary
tape deck and cartons of tapes to store some digital files, very much like our previous
model of copying master videotape. Thatís how that worked and as you can see weíve
already managed to get up to 1.2 petabytes, almost without really trying. This is of some
concern ëcause weíre certainly not quality checking them very effectively as we go.
Is this the way we should continue? I think weíve heard a few things, certainly Kris
from the internet archive answered my question yesterday about how much herÖ I was rather
envious of her petabyte of storage all in one rack. Today the Church of the Latter Day
Saints also had a cost factor about that, which showed that data tapes are just so much
cheaper than servers. The problem being of course, the accessibility, the checking and
the migration, you need to have those tapes in some sort of system.
It is a worry, data tape is essentially the same type of tape that weíve been using for
videotape and in fact, it is the same, it just is recording ones and zeros instead of
images, and weíre really not too sure how long that will last.
Might be skipping ahead here. With hard drives and things, theyíre not a realistic long-term
storage medium for us. They will of course be part of the solution in terms of access
and things. They do have quite a high failure rate, a bit higher than the manufacturers
advertise. Probably because itís a failure rate in the real world, rather than a clean
room. Thereís also solid state devices, theyíre way more expensive and not very large at the
moment, but you never know, something else might happen.
Before you even start putting stuff into digital files though, you need to be confident about
what youíre putting in. You have to check that it is what it says it is and itís the
quality it promises. Once upon a time, this is the archivist thing, is the fear of the
black box, is once upon a time if you found a piece of film you could hold it up to the
light and take a reasonable guess at what it was. This is the classic publicity photo
for film archivists, no such luck with video tape and a lot of film archivists really didnít
even wanna deal with video tape, let alone get into digital files.
When you stumble across a hard drive or data tape in the rubble in the future, or in a
suitcase under the bed, youíll need not only a machine, but the operating system to match
and probably a key to unlock it.
Ideally a transfer to a new format would take place as soon as possible and the whole thing
would be quality checked. At the moment quality checking means watching the whole thing, listening
to it and not missing any bits in case there are any bits missing. This almost never happens
at the archive. Those files returned to us by Park Road Post and Weta Digital are viewed
in their entirety at the superb facilities at Park Road and thereís usually a queue
of film archivists for the car to go out and have really great coffee and sit in the big
squishy couches out at Park Road Post. Thatís possible and desirable because thereís only
a few of these titles being preserved at a time and theyíre costing a lot of money to
do so.
Meanwhile, back at the ranch, weíre receiving more than 10,000 titles every year and our
acquisition staff, if they did nothing else all day, they couldnít possibly watch them.
We just have to take a guess.
As our analogue material is being digitised, itís being scrutinised for technical issues
and we essentially checking the head and tail of items as they go. There may be a quick
look at the file once itís recorded. If thereís a major stuff-up then usually we catch it,
but I'm sure that somewhere down the road, thereís dirty data already in our system.
Back to the question of handling incoming material. When I started at the archive we
had a brand new professional super V8 chess machine, that was the best thing we had. An
old Betamax and a pneumatic player that made a Massey Ferguson Tractor look like a Porsche.
We now have the ability to handle about half of the titles on this slide, half of the formats
there, but essentially itís the same thing with digital files. In this case we wonít
just need a lot of machines, but a lot of applications, licences, codex, etc. Especially
if we wanna retain the original in its original format, as per our archival promise, if you
like.
Thereís never been an international standardised format for archiving film and television.
Theyíre still arguing about it. Thereís a lot of manufacturers investing a lot of
money in creating new interesting formats. There is one for audio archiving. I donít
know quite how official it is, but basically the WAV format is pretty much used internationally
now for audio. So Michael, a bit easier on that part for ya.
A recent survey in the US showed non-profit archives using 12 different tape and file
formats to preserve the movie image. Thatís not even dealing with whatís coming in, thatís
what theyíre creating as well, so something else for the future.
Thereís also the extra complication for hard drives and files lockable. When we collect
physical films it sometimes looks like this, this is a collection; we were alerted to this
by a conscientious contractor as he was demolishing this house, he found some film. This contained
masters for a New Zealand features, plus numerous other very interesting bits and pieces. Now
imagine that room as a steel vault with a combination padlock on the outside. Obviously
we could ask for the key from the depositor, but surprisingly frequently orphan films do
turn up and I would anticipate being handed hard drives in the future by relatives and
strangers alike, who had no idea what was on it or what the password was.
In addition thereís also the Digital Cinema Package. DCP is used for approximately two-thirds
of all screenings in the US now and probably more here in New Zealand. It arrives at cinemas
as an encrypted package and the key is an XML file that can be read by the destination
cinema. It also includes dates for the run that the film is gonna be running for. Itís
a bit like ìMission Impossible,î I donít know that it actually dissolves in a cloud
of steam, but you probably wonít have very much luck even if you do have the key afterwards
at actually accessing that. This pretty much precludes the kind of film rescue that we
did in the past, where weíd find lost features and resurrect them.
A big plus for DCP though is that it has helped to solidify a digital format and JPEG 2000,
which weíve heard a bit about in the last two days, is what DCP uses. Itís pretty much
the only one they use and itís immensely scalable, it has a whole bunch of advantages
as well as a couple of disadvantages, but thatís a whole other talk really. Somebody
more technically minded could address that one day.
Meanwhile weíre hoping that the studios will look after the masters of those films so we
wonít have to rescue them from the rubble. Hopefully theyíll do a better job than they
did back at the transition from silent to sound. At that point they lost over 80% of
their product.
Part of the good news story, I guess, about digital, is that it gives us a chance at something
to do with holding onto the originals. This is from our collection and with digital technology
we now have the opportunity to go back and scan directly off the original nitrate. This
film was actually preserved in the 80s chemically, a new InterNeg and then a new print made,
thatís a couple of generations right there. Weíve managed to keep it in a pretty much
stable condition and hopefully now, with the help of Weta Digital, weíre going to take
those films and have another go at scanning them directly to create a digital master from
the original.
In Germany I believe the German archive is busy burning their nitrate film because they
donít wanna store it anymore. Thatís if you believe everything you read on the internet.
So we do wanna hold onto it. Also the other thing about holding onto it is things that
we donít know that we had. We had a film called ìUp Stream,î in our foreign nitrate
collection because weíre a member of the International Federation of Film Archives,
we maintained looking after it, we thought it was some sort of fishing film or something
you know, we never looked at it ëcause weíre not funded to do anything more with foreign
films. Fortunately the NFPF from America sent an archivist out to have a look for us and
we found John Fordís first feature. In addition to that also Alfred Hitchcockís first directorial
debut or at least three out of about five reels of it. Weíre still looking for the
other two, we havenít found them.
Thereís a lot of pitfalls in acquiring material and holding onto it, but once we do have them
as digital files, how long will they last. If our culture is ones and zeros, that requires
machinery to access it, how long can we sustain that. Weíve been storing data for 50 years,
thatís great isnít it, I just love that. Weíve only had electricity for 150 years
and yet how long are we gonna try and maintain this moving image culture. Weíre aiming for
more than 150 years at least.
That means we require a sustained power and mechanical maintenance. It also means a sustained
environment and a sustained culture of care and thatís one of the major risks I think.
Thatís a flood map of Wellington by the way, that was in the Dominion Post on the weekend.
I think it pretty much covers Archive New Zealand, National Library, the Film Archive.
Thatís only 100 years out according to one prediction so weíd better get all painting
all the roads white to reflect sunlight back.
Some of these risks are just forces of nature and we just have to do our best to mitigate
that. Some of them are actual ongoing technical risks that are complicated by low budgets
and misunderstanding of the demands of the digital age and others are a constant factor
in our archival practice.
Those are some of the technical risks, I think weíre all familiar if weíre dealing in the
digital environment, I think we can read that and thereís a number of issues there.
A few years ago, at a film conference the question was posed, ìThe buildingís on fire,
what do you grab to save your film, if you just take one thing?î There were people saying,
ìOh weíll grab the negative,î ìWeíll do this.î The answer was a film print because
it has the picture and the sound on it. Of course these days youíd think the answer
is now grab the hard drive off your desk, youíre probably gonna save several films
at once. The real answer is leave everything and get out of the building because itís
on fire and also because youíve back it up somewhere else, so it doesnít matter, thatís
the answer.
Itís this sort of reduction in storage size and the ease with which it can be transferred
that ensures the survival of digital film. Itís the distributed archive, where many
copies in many places. Thereís challenges to that, youíve got HD and 3D and recorded
a much higher shooting rate by digital cameramen, ìOh it doesnít matter weíll just churn
up the tapes.î Thatís fine, except when it comes to documentaries, the film archive
as our policy, keeps all wild tapes and footage for documentaries. For obvious reasons, has
great interviews. But weíre just getting enormous quantities coming in, all wonderful
stuff I'm sure, but it does create more stress on the system and you seem to wonder whether
the film making community is just going to expand to take up whatever capacity we can
create.
Thereís also a whole bunch of stuff about acquisitions; accessioning and stuff; automated
systems that collect metadata; face recognition software and cataloguing programmes; data
seeking programmes on the internet that might be able to catalogue some of this stuff without
too much human intervention. Thatís the dream of a number of my staff thatís been used
before and I have a big laugh at. I think it obviously says something to us about our
vision of future archives, theyíre a big grandiose and look a bit like what they used
to look like, theyíre kind of like books except theyíre glowing, so theyíre obviously
modern, thatís pretty good.
Having so much material on hand does risk getting swamped. You do wonder whether archivists
will continue to mediate the stories or whether weíll just become truly non-hierarchical
and people will do their own searching out and hunting down on the internet. Is the real
future an invisible archive? That was hinted at this morning at the session weíre in.
One where we retreat into the background.
For the Film Archives Charitable Trust, which relies on funding, becoming invisible would
be the death of us because we need to be out there to keep showing people that weíre doing
the job. We canít actually afford to become invisible.
Thereís a whole bunch of strategies and Iíve assumed in most of them that migration is
the key but weíve also heard about emulation. I'm not too sure about emulation, itís seems
like another high risk strategy. Sorry Ewan. The idea that weíll just keep something and
then in 50-60 years weíll be able to access it using some clever software, I canít afford
to pay clever enough people at the archive to do that at the moment.
There are other less - and this is the other great vision of the archive, the opposite
image, which is youíre just gonna store it away and forget about it and maybe never get
it back.
We do need to keep it visible, we canít just do that with it. If nobody looks at them in
100 years is it worth even keeping them, so you really do need to keep it out there.
I might have just enough time for my favourite, this little run down at the end, which is
thereís nothing new under the sun and this is actually from a talk I gave a little while
back which is digital technology is all sparkly and new, but is there anything we can learn
from the past? Which culture retained itís records for the longest time, the Romans,
Greeks, Egyptians? We know their stories, 2,500 years, 3,000 years, 5,000 years, what
are we aiming for? This lasted in the nice dry desert pretty well. Interestingly, I think
the great excess story also comes from the desert and maybe the Aboriginal cultures of
Australia.
For 40,000 years the indigenous people of the biggest island kept their cultural life
through a highly structured oral tradition. Verbal record, maybe not as accurate as a
document and it cannot guarantee word for word accuracy, but that was less important
than maintaining the consistency of the content. Indeed long term flexibility with words changing
to reflect changes in language and customs may enable the basic story to retain relevance
to the people it was designed to serve, rather than becoming seen as quaint, old or irrelevant.
So how did the Aboriginal people, in what has been a vast and harsh environment, keep
these stories alive and essentially unchanging? These stories were important to their survival,
as well as having an economic and cultural value. An elaborate system was devised that
included multiple redundancies and disaster planning. The stories were linked to parts
of the land and to the animals that lived there. These acted as mnemonics, anchoring
the tale. Landmarks were essentially unchanging; it could also be accompanied by carved trees
and rock art.
Each story, dance or ceremony was entrusted to a set of custodians, with each custodian
maintaining responsibility for a specific part of the story. The custodians were also
the educators and were charged with teaching the next generation their particular part
of the story. The custodians would be aware of the whole story, but could only teach the
part of it that they were responsible for. In this way, when the younger generation recited
their new knowledge, it had to all fit together and the custodians would realise if any part
of the story was changing. So thatís an ancient form of error correction.
Other family groupings would be responsible for certain stories or ceremonies and these
would be spread to other family groupings in the neighbouring areas. Thereís geographic
mitigation there. In this way, if a set of custodians died, or one tribe was wiped out,
then a back-up copy could be obtained from a related tribe.
Of course, unfortunately this system which seems to have sustained Aboriginal culture
for thousands of decades collapsed almost completely under the onslaught of European
visitation. Disease spread from the very first contact and it was so devastating that by
the time the first settlers arrived in New South Wales, only a couple of decades later,
they described the native people as primitive and without sophistication. What they were
seeing was people suffering from starvation, whoíd suffered a population collapse of 60%
or more in some areas.
So a system that had survived for 500 centuries was all but destroyed in the space of just
20 years. They couldnít anticipate what was essentially an alien invasion just around
the corner.
I guess thatís a cautionary tale, we donít know what we donít know, thank you [24.23].
Thatís pretty much the conclusions, despite maybe slightly pessimistic thing, I think
if we do 40,000 years weíll be doing pretty well, keeping ìGoodbye Pork Pie.î For those
foreigners, thatís a classic New Zealand comedy, if you like. I think the digital age
actually promises everything, I donít think it was sustainable to keep our old archives
the way they were and I think the digital age offers a hell of a lot.
[End of recording 24.57]
1
Transcribed by Audio Transcription & Secretarial Services Ltd
www.audiotranscriptionservices.co.nz