Tip:
Highlight text to annotate it
X
Church: From the new College of Florida, he has worked as a professional bronze sculptor
and has been involved in 3D imaging for over twenty years. He’s also the co-inventor
with Tom Malzbender of the Computational Computer Technique High Reflectance Transformation
Imaging. Mark serves on several international committees including the International Council
Museums Documentation Committee. I would now like to now turn it over to Carla.
Schroer: I’d like to start by just taking a moment and telling you a little bit about
Cultural Heritage Imaging, and we are based in San Francisco and we are a non-profit and
we have a mission to drive both the development and adoption of digital imaging solutions.
We have this really big broad vision, but I think given the conversation over the last
couple of days, I want to point out some things that may make us a little different from a
lot of the things we’ve seen, which is that our core philosophy is about really getting
tools in the hands of people so they can integrate them in their day to day work and use them
so that lots of material can get imaged as opposed to a service provider model where
you have to pay specialists to come in and do things. So our approach is to look for
things that people can do inexpensively with a minimum amount of training and with off
the shelf equipment. We also want to look at making sure the results that are produced
are high quality and we want to be thinking about issues like archiving and data reuse
over time. We’re a small organization and we function
through collaboration. We have a number of technical groups and research labs that we
work with. Here are some of the folks that we’re working with currently, and we also
work with a lot of people on the cultural heritage side, including a number of museums
and most recently fine art conservation. So, I’m going to talk a little bit about
two primary techniques here but I just want to set the stage for what we mean when we
say computational photography and really what we’re talking about is taking a sequence
of images with a camera and then in the computer, extracting information from different images
in that sequence to create a new type of representation that has information that’s not available
in any one image from that sequence. And there are a number of examples of this and there’s
a lot of work happening in this field. The one’s we’re going to be talking about
here are the reflectance transformation imaging and algorithmic rendering.
So, something we have as a big idea is that we know today, right now, how to with a digital
camera, take image sequences that give us an awful lot of information about the objects
that we’re trying to document or research, including RTI and AR but also as we’ve seen
full 3D models, and we’re also very interested in this idea of keeping track of the process
history and what’s happened to make the data more valuable and more reusable.
So, just quickly, we’ve seen a couple of things about RTI recently in the last couple
of talks, but let me show you a quick example. This is a 15th century manuscript page from
the Bancroft Library and when we have an RTI we can also apply some mathematical enhancements
to the surface. In this case, what I’ve done is taken all the color out so we’re
just looking at surface shape information, and we can see on this very dark projector,
some of the surface for this piece of velum is warped, and I want to zoom in here and
look at some details. Right here we can see where a letter was scraped off of the velum.
We can also see information about ink that’s spalling off the individual characters here
and some of the really fine line details. Another really quick example, this is a Japanese
wood block print from the Fine Arts Museums of San Francisco and here is just a screen
shot. On the bottom we’re showing the color information from the RTI and at the top we’re
showing the surface shape information without any color and you can see the really fine
details around the hair and the brow line and so forth.
Okay, so what’s going on with an RTI is that we have a fixed camera position and a
fixed object position and we take a sequence of images with light in different positions
around the object and then that set of images is synthesized into a new file format, an
RTI and this is based on work from Tom Malzbender of HP Labs that was presented at Siggraph
in 2001, so we’re over ten years old with this approach at this point. What we have
is 2D information or a 2D image that carries also some 3D information. So to explain this
a little more, and Roy touched on this and so did Rick a little while ago, but in computer
graphics we have a notion of a surface normal. In this graphic here which is a cross section
of a surface, the red arrows depict the surface normal and the surface normal is the vector
that’s perpendicular at any point along the surface. When we throw light into the
equation, light has a physical property of bouncing off of the surface where the incident
angle and the exident are equal angles to the surface normal. So if we know where the
light was coming from, we know where we are collecting the data, which is the camera,
and we have images from a number of light positions, then we can calculate that surface
normal per pixel. So what we have in an RTI is for each pixel, we have the color data
like a regular photograph, the RGB data, and we have a mathematical description of the
surface normal per pixel. One thing that is cool about this technique is that we can work
with shiny material, so in the picture on the left of this gold coin, we have some blowouts
on the surface from one of the input images but because we’ve sampled all the way around,
when we put it together we have complete surface information.
There are two main ways that we collect RTI data. Down here on the right was the original
dome developed by Tom Malzbender. This is an early dome system. We developed, we built
a dome for the Wooster Art Museum Conservation Lab and also for the Museum of Modern Art
Conservation Lab, but what has really allowed this technique to take off and get more widely
adopted was the development in 2006 of highlight RTI and this is an approach that we co-invented
with Tom Malzbender. In this case what we do is replacing reflective spheres in the
image and then we get a highlight on the sphere and that allows us to figure out where the
light was after the fact. So now we can do RTI with a very small amount of equipment.
It’s portable. Its equipment most people would already have. Here’s a basic set-up.
I’ve got a camera pointing down. We use a string to keep the radius, so we’re basically
recreating a virtual dome with a flash. Here’s a similar set-up for a vertical object we
can use coffee stands, camera stands, a variety of set-ups.
This is just a screen shot from the software showing the highlights on the sphere. There
is software that can detect that and that determines the light positions. This is a
little map showing all of it and I’ll put in a plug for our demo tomorrow. If you come
by we’ll talk a little more, and we’re going to demo how to capture this and we’ll
talk a little bit about the software pipeline in more detail. I want to note that all the
software is open source. It’s available from our website along with user guides and
videos and things. Another quick example, something that happens
in rock art a lot is people want to figure out if one line is on top of another and you
can see from this Paleolithic petroglyph in the little call out that it’s really quite
easy to tell which line is on top of which other line. We can do this under the microscope
as the last paper was showing. This is an example from the Metropolitan Museum and they’re
very interested in looking at tool marks on these saddle fragments. We’re seeing a lot
of adoption with RTI a number of major museums, primarily driven by conservation, are starting
to use this now and this has been really aided by a grant we received from the Institute
of Museum & Library Services. That has allowed us to deliver a four-day training in RTI and
we’re delivering it ten times and that’s occurring over a broad range of museums and
graduate programs. So at this point I’m going to turn it over
to Mark.
Mudge: Thanks, Carla and I’d like to thank the NCPTT and the National Park Service for
putting together this really terrific forum. I’m going to talk about algorithmic rendering,
which is another form of computational photography and in this case, we’re using the same data
that you collect in an RTI but taking the shape information and the color information
and applying single processing filters to it to generate scientific illustrations. Here
we have an illustration of a pine cone taken from a normal and color thing and that’s
just not going to laser scan very well. Here are examples of different types of signal
processing filters. There’s all sorts that we could show you but we don’t have a lot
of time so I’m going to move through here quickly. This would be, I think, of interest
to the National Parks people. Here we have a petroglyph at the Legend Rock State Park
in Wyoming and you can see down in the bottom area there is a very heavily patinated section
that’s very difficult to see and it’s actually not just the projector but it’s
difficult to see in the real world. However, if we do a signal processing run on this,
you’ll see that you can see all sorts of little zoomorphs that were hidden under the
patina, and the result was that the Wyoming State Archeologist was able to uncover two
new zoomorphs from this information and this panel had been studied heavily for over thirty
years. So, there are actually hundreds and hundreds
of signal processing routines that can be brought to bear for new types of illustrations.
Now we’ve received a grant from the National Science Foundation and we are partnering with
Princeton to develop what we call the collaborative algorithmic rendering engine and it’s a
three-year collaboration and we hope to have some results in about eighteen months. Of
course, like RTI, all of the software we’re producing is going to be open source. Now
the thing about the care engine is that it creates an ostensible framework for anyone
who wants to design signal processing routines to plug them into this system, and it will
allow the user to bring in RTI data and select the algorithms that most effectively represent
their material. You can mix and match, change parameters and we’ll keep a complete process
history of everything they do such that at the end of the day, they’ll not only have
a scientific illustration with complete provenance information of how it was done but we’ll
also have an expanding recipe book of how to represent different types of subject matter
and that can grow as people add more and more variations to the operations.
I’d like to say that we also frequently shoot RTI with photogrametry, which gives
us the ability to distortion correct the RTI input images or to rectify them and provide
that kind of input for the algorithmic rendering process but when we’re talking about close
range photogrametry, I just want to show us a little example that we shot a few years
ago. It’s a piece of a cuneiform cone that’s about this big and we got a really good camera
calibration and from two overlapping stereo images, we developed this you see on the lower
left that we’ve got texture. In the upper right we have the surface mesh and as we get
closer and closer, we can finally start seeing some of the vertices and the mesh. So you
can take your photogrametry and get it as precise and sub-millimeter as your optics.
Let’s take a brief second right now and jump around because I’m going to jump into
the philosophy of science for just a second. We think that it’s useful to think of digital
representations in the three ways. The first is fine art and entertainment, which we all
understand. The second are visualizations and we’ve all seen visualizations like the
comet that came down and hit the Yucatan and wiped out the dinosaurs. Well there’s some
scientific data in this visualization but a lot of speculation. We have no idea what
the shape and color and craters on the asteroid or the comet looked like. But we’re just
putting in speculative stuff and visualizations are part scientific content and part speculation.
But finally we have something called digital surrogates and digital surrogates are digital
representations that enable the scientific study of the subject without the physical
presence of that subject and digital surrogates are built scientifically. They’re built
along scientific principles and you have data that can presumably be confirmed by somebody
else and everything you do to the data after that, all the processes run on it and the
final result are all captured in a scientific lab notebook. This is a way that permits other
people to evaluate the quality of your work and to permit replication. The quality and
the evaluation of quality of one person’s work by another is the key to scientific imaging
and it’s critical that you have this account or either have a nice visualization or entertainment.
So at CHI, we have a concept of a digital lab notebook and that’s to both collect
information about the capture and then everything that happened afterwards and we want to both
collect this metadata and manage it in an organized way throughout its lifecycle. It’s
really important that we’re collecting this lab notebook metadata, and the process for
collecting this can be built into the imaging tools if you structure the imaging tools properly.
It can, in fact, with computational photography tools, be done automatically so that you yourself
as the user need worry very little about the metadata, only at the beginning when you’re
saying what you’re shooting, whose there and so forth. So the goal is management of
the digital lab notebook metadata in a way that includes the relationships among that
metadata and hopefully, in a form that can be linked semantically and allow for excellent
query and other forms of search. So if you have the capacity to evaluate the
quality of an image, the digital representation, this enables distributed scholarship because
you can have people pooling digital surrogates from around the world. You can have other
people using other people’s work and this is exactly the idea behind how the human genome
project worked. If they all had processed accounts of how they’re data was collected
and that enabled them to determine how to trust that data that was there and it also
allows for future reuse of the data we collect today because in the future if someone wants
to be sure that they can trust a piece of data, they’ve got to be able to see how
it was done for them to be able to do this. That allows for growth and maturation of investments
in digitization. So we have the digital knowledge lifecycle
for capture, use, regeneration, and we want to track the metadata all the way through
and both RTI and algorithmic rendering are designed to use this digital lab notebook
process and it’s not just us at CHI talking. Exactly the process history in metadata management
I just discussed had been accepted by the top computer scientists working in cultural
heritage in Europe in the 3D collection formation framework 7 project of the European Union
and they bought into all of this stuff including laser scanning, so reuse and repurposing are
fundamental for digital imaging and the test is passed if the information permits the evaluation
of its quality by contemporary and future scholars and digital information that does
not permit qualitative evaluation is of little value to science and scholarship. So we know
now how to capture many sequences of images that give us rich raw data and metadata notebooks
and this can democratize the capture of information around the world. It breaks the reliance on
a client service bureau method. It allows quality evaluation and almost anyone can create
scientically reliable documentation and thank you for listening.