Motion Amplification in video – seeing the invisible


When we think of enhancing vision, we tend to think in terms we can really get a grasp on:

  • Zoom in (enhance!)
  • night vision
  • microscopy (really zoom in)
  • long exposure (astronomy)
  • timelaspse (watch that glacier hussle)
  • slow motion (mythbusters finales)

My favourite development in computer vision during SIGGRAPH 2012 was called Eulerian Motion Enhancement of Video.

Essentially, it amplifies motion in video. You have to see it to really get a grasp on what it means, so here is the video. Check out the throbbing arm artery!

The secret here is to look at changes in the video from moment to moment, just like when you flash between two photos you took at a party and can see the differences easily. The algorithm tracks the differences between many frames over time noting the differences only.

But then the clever bit – what if you only pay attention to certain rates of change and ignore the rest? For example what if you had video of two pendulums swinging next to each other – one short and fast, the other long and slow – using Eulerian methods, you could ignore the fast motions or ignore the slow motions, much like a graphic equalizer in audio can isolate bass and treble frequencies in your music. You could effectively filter out either the slow or the fast pendulum depending on your ‘equalizer’ setting.

Once you’ve isolated the motions you want to enhance, you add the resulting difference image frame back into the video. If you really want to enhance it  you do this several times in a row using a feedback loop. The more times you feed back the difference, the more that specific motion in the video gets exaggerated compared to other motions in the video.

This works best for periodic motions, like a pulse or guitar string or heartbeat, but enhancing specific motions is pretty awesome. In fact, you do it all the time without really being aware.

How? Well, try waving your hand at the very edge of your vision holding while up 2 fingers. Can you notice the motion – YEP. But can you count the fingers? NOPE. The edges of your vision are tuned to detect motion changes, not detail, while the centre portion of your vision is tuned for detail. You’re not going to read a book out of the corner of your eye, but you’ll definitely notice the sabre toothed tiger coming at you.

Of course, you have to watch the video, still images don’t really catch the effect!

Chris Hadfield – Science outreach wrap-up

800px-ISS_Expedition_35_PatchPretend this was a long article summarizing the incredible outreach and education opportunities that were maximized by ISS crew member Chris Hadfield during his time as Commander. From Reddit AMA’s to videos about everyday life in space, what he has demonstrated is that science outreach is about curiosity, it’s about asking questions, and it’s about trying to find out the answers for yourself.

Sure, we’ve figured out things like gravity intuitively for ourselves, but in a new environment, where ringing out a facecloth leads to crazy results, all the answers can be different, and curiosity is something that can get rewarded even in the small details.  Sometimes we forget, in the era where all the basic questions have refined and polished answers, that science starts with ignorance, and the more ignorance, the better the questions. Being in orbit, all the rules change, and the answers are no longer polished or quite what was expected. Forcing a reset on intuition and common sense reawakens curiosity.

And really, the ONLY reason we do science and  ask questions is ‘Why?’ and the ‘What if?’ Commander Hadfield wraps up this amazing outreach success with a “Why Not?”

Frank Borman, Astronaut and Leader in tough times

220px-Frank_BormanIn creating new content for the web – educational, corporate, entertainment – we have a bit of a pressure-cooker atmosphere. Teams are quickly assembled. Ideas are tried and tested and implemented in hours and days, and deadlines are often set with everything but technical feasibility in mind. 

Of course, in web, the stakes are rarely as high as those of, say NASA. We can update, beta test, and the consequences are not typically life altering. There is always “CTRL-Z”

Nonetheless, we need good teams – exceptional teams, really – to get the job done.

Looking back over 50 years, one of the most amazing teams, goals, and deadlines was the Apollo program to land a man on the moon by 1969. 0.4% of the Gross Domestic Product of the United States at that time was diverted to NASA, and when the project had started it was science fiction – it would be like us today building nuclear starship in 15 years, just enough feasibility to not be pure science fiction.

But things did go wrong, because of the rush to meet the deadline, and because of the unique nature of risk: nothing goes wrong until things go wrong – the only symptom of high risk is catastrophe. Continue reading

Visualizing Feedback control

I wrote a jsfiddle sample as a demo for my brother’s control systems class at NAIT to show the differences between closed-loop feedback controllers – the tiny algorithms behind cruise-control autopilots, and autofocus in your camera.

I have to generalize it to a PID object and so on, but it is fun to play with the coefficients – and I learned a thing or two I’d forgotten about digital controller behaviour, like sampling error and sampling bandwidth. And as all Control Systems engineers are warned: Turn up your Gains slowly!

Ideas for feedback loop tuning games? The unicyclist? better cruise control, Autopilot lander? Fill the acid vat?

Book: The Art of Explanation

(ISBN 978-1-118-34758)

Amazon Look Inside preview

I work with clients who have complicated things to say to the public. Topics like physics, policy, engineering and ethics.

It is frustrating and wonderful. Here’s why:

Even though we all speak English and/or French, there are also languages we each speak that can be totally foreign to others, the Language of 9-5. We spend 8 hours a day talking to our coworkers in the language of our business: baking, nuclear physics, legal frameworks, aviation, military, medicine. The words are English or French, but the meaning, the acronyms  even the assumptions are all mysterious language to people outside our 9-5 tribe.

Just try and answer “How was your day?”  in any sort of technical detail to someone unfamiliar with what you do.

This is a bigger language gap than just about anyone can appreciate. In fact, you are wired to not appreciate it, because the mind is so good at turning “new and crazy” into “normal” in just a few days or minutes.

In The Art of Explanation, Lee Lefever sums this language gap up as the “Curse of Knowledge”. And lordy, does that hit home for me.

I see it when a client attempts to do public outreach. This is the frustrating part of “Frustrating and wonderful”.

Typically, they start in the middle of and concept and then dive way, way too deep. They know the fundamentals so thoroughly it’s like asking a fish about water. They have it all summed up beautifully in a PowerPoint slide that looks precisely like the Mayan Calendar Wheel of the Apocalypse. Their presentations are well received at industry conferences, so hey, they have no problem with outreach, they just need it cleaned up a bit and put in front of the public. No problem, right?

What they really need to do is step back and start from square one. The Curse of Knowledge, however, is a curse because when you know a topic well enough, you most likely can’t even find square one again. Think of it this way – could you ever unlearn to skate? Can you even remember when it was hard, or how to skate like an amateur?

Lee often shows a particular species of graph in his book: at  one end of this graph is the audience and at the other is you – the person that wants to get a message across and have the audience understand it. Really understand it. The space between –  that’s the gap in language, and your audience never agreed to meet you in the middle. You are making a promise in Public Outreach, to fully bridge the gap, and bring the public along for the trip towards your end of the spectrum – to better understanding.

So, you are going to need some tools to bridge this gap, to bring the audience to your side, to the “I get It!” side. His book is a toolbox, carefully organized and complete.

He begins with two chapters that diagnose the problem in clear and brutal terms: What is an Explanation and Why Explanations Fail. They essentially remove any sort of illusions you might have about the traditional approach of getting an idea to an audience AND having them understand it. It’s like finding out your team is all quarterbacks and no receivers. It kind of hurts to read, actually.

However part Two is where the toolbox is opened up and the steps needed to build a bridge of explanation and understanding are laid out one at a time: Context, Story, Connections, Description, Simplification, Constraints, and finally the end where most people start: Writing.

The book itself is an example of the craft of explanation – you would expect it to be. There are analogies, examples, clear writing, storytelling,  and yes, it works. And it feels scientific – you will understand the method behind the Magic that makes a good teacher or good documentary, or great BBQ assembly manual. You will also find that many of the things that are sacred, like Accuracy, can hinder an explanation. In fact, this is where we find it the most challenging when working with clients – there is often no place for high-resolution HD accuracy when an impressionist painting of a concept is what’s needed. The broad strokes, not the bristles. I’ll get into that next time: what is the Right Kind of Accuracy?

In the meantime, have a look at the book if you have Explanation as part of your day. It’s not just for Outreach people, it’s for anyone giving a presentation or sharing an idea – who want other people to be interested, to be curious; and really, that’s the soul of communication.

Suspension of Disbelief

Look at this amazing demo of silliness. I love Peceptive Pixel, and the idea behind the Windows 8 touch experience. But one does not use a vertical touch wall in real life. It is a matter of pixel density versus arm length, and while a retina-class smartphone held at waist level is too far, a 40+ inch vertical touchpanel at arms length is fatiguing and too close at it’s nominal 60dpi.

Another factor is performance anxiety – in a public space, a large interactive display effectively requires a public performance (with no privacy) and people tend to shy away from using it at all for this reason (unless they are kids!)

The numbers behind “quality of education”

Gathering up numbers to shock, awe, and entertain is a core part of making interactive media engage people. It’s one of our favorite tricks:

150 Billion Stars, 50 million transistors, 50,000 blood cells on the head of a pin.

This works well enough, but even numbers can get political, in fact it is especially numbers that get political:

“The new fighter replacement program will cost over 30 Billion dollars”

“One in five kids in Canada go to school hungry”

“Every day, 500 Canadians are diagnosed with Cancer”

These numbers affect us quite personally – it’s our money and our lives and our culture. It matters a lot more than say, the mass of the moon in our perception.

And unfortunately we consider both the mass of the moon and the number of kids that go to school hungry as equivalent in terms of factual accuracy – because they are both expressed as numbers.

People – and headlines – don’t seem to have time to factor in the nuance of turning a complicated system into a single statistic; a stat should really have a big asterisk after it, just like the price of a new car, and small print should say what the accuracy of the stat is and how it was calculated.

Continue reading