Let’s say you have an idea for an app, a website, or some hardware which uses computer vision. The good news is that there is an incredible amount of information to collect from a video stream. The bad news is that despite decades of research and really cool presentations at SIGGRAPH you’re likely going to need to develop and implement your algorithms from scratch. To understand our perspective on the matter, let me tell a brief story.
I remember sitting with the rest of the team watching the keynote live blog when Jobs announce the iPad 2. The bit about the front camera made us all excited.
Up until this point we had been using iPads to build interactive cardio experiences for people on cardio machines. As people would use their bikes, treadmills or ellipticals they would create vibrations in the machine which were transmitted to the iPad and to it’s accelerometer. Our app then used those readings to guess at your exercise frequency. It was alright, but not perfect, and it was missing other information we wanted like exercise phase and body lean.
We suspected if the device had a front camera we had a shot at discerning this information via computer vision. Now, at this point none of us had any special training in computer vision, but in hindsight this was to our advantage.
This is because the computer vision solution space is wide and shallow*. There are many, many possible computer vision tasks and the computer-aided solutions for them are generally only moderately sophisticated. Whats more, computer vision research tends to cluster around a few peaks in this solution space, iteratively adding on to research which came before.
It turns out our best solution for exercise tracking on cardio machines was not nearby the existing solution clusters. To get there by iterating on existing research would have been far less direct than successive creative guesses. We probably would have started with feature detection and tracking research and customized it to the application constraints. It would have been likely to violate existing patents and it worst of all it would have had unnecessary complexity and computational overhead.
The solution we ended up with takes advantage of our unique constraints while elegantly resisting the domain specific challenges. It’s efficient, robust to background noise and lighting, capable of detecting very minor exercise motions and best of all it’s only about a thousand lines of code. It did take quite a while to figure out the right thousand lines of code, and this video shows a bit of how we got there:
I’ll end on some tips for people looking at implementing computer vision in the real world:
- Check the research literature for exact solutions to your problem. It’s worth a shot, if only to understand what doesn’t work.
- Even if it’s covered, plan on understanding and implementing it yourself. There is very little chance it’s been written well enough for practical use, for your platform, and using your unique constraints.
- If it’s not covered, as most likely it will not be, try successive creative guesses in a rapid prototyping environment like Processing.
- As you port promising solutions onto your platform and start measuring performance, optimize the algorithm in C/Java before diving into ASM, GPU or NEON programming. Those options will really lock you down into your solution(s), and one thing we found was that when you’re using successive creative guesses to find increasingly direct solutions, you end up changing your algorithms frequently.
Feel free to comment on the article submission on hacker news.
* If you were working on AI, you could view the computer vision problem space as deep and narrow.
Recently someone asked me how I thought about game design, and the holiday downtime is a good time synthesize my two year long crash course on the subject. With no further ado:
Game Design Experts
Experts like Raph Koster and Jessie Schell are generally right about everything they say because they take a pretty conservative position: Here’s a bunch of principals and ideas which have worked before, and will probably work for you in some combination. This combination is hard to theorize about and you’ll need a design process with enough time for trial and error to find the fun.
Go read their books. There are no easy answers for you there, but it’s probably as helpful as anything can be, because every single game design pursuit asks us to solve a wicked problem. A wicked problem is one where the solution is essentially novel and unique, and so every solution to a wicked problem is a one shot operation. More info on wicked problems on wikipedia.
Lately gamification “gurus” have emerged who come in and try to synthesize these raw materials into formulas you can drop right in to your business. I feel they are well intentioned but naive, and their formulas can’t be as universally applied as their conference slides imply. By the way, this behavior is why gamification has become a bad word to traditional game designers.
Gamification isn’t impossible though. Jessie Schell gave a talk on the matter here. Successful addition of game mechanics to non-game aspects of our lives will require careful application of fundamentals, not haphazard layering of formulas.
To recap: while the idea of giving a non-game experience some game qualities isn’t impossible, it is harder than many would have you think.
Application to Fitness (or wider behavior)
We spend a bit of time thinking about fun, and we have Koster’s and Schell’s books in the office. We spend a lot more time thinking about how game design can drive behavior change. One of the leading thinkers here is Amy Jo Kim, whose long history making games and PhD in Behavioral Neuroscience give her a very helpful perspective. But there are only a few people thinking hard about this, and they haven’t been thinking about it very long, so there still isn’t a lot of wisdom written down. Any would-be students need to seek out experts in more general motivation and willpower. Daniel Pink with Drive, Kelly McGonigal with The Willpower Instinct, Charles Duhigg with The Power Of Habit, or Nir Eyal who is knocking it out of the park synthesizing thoughts on how technology can drive habit formation on his blog.
To recap: the intersection of game design and behavior is nascent and no one has proven much yet, but there are smart people thinking about it, and you can be too. Take some time and dive into the fundamentals presented in the links above.
I hope this was helpful, I plan to write more posts like this diving into the specifics of how fitness and game mechanics can interact. Feedback much appreciated, so please make use of the comments section!
It’s been a crazy few days as we scrambled to get everything together for this release. I slept about an hour last night because of a terrible issue with how unity cross-compiles DLLs. Advanced vision processing can seem easy compared to packing, documenting and testing up a multiplatform developer tool.
But now we’re relieved to announce that our SDK is available to start prototyping with right this very second. Learn more at www.bitgym.com/developer
If you haven’t see it yet, check out the video outlining the technology:
We’ve just released version 2.0 of Virtual Active: BitGym Edition! We’ve listened to everyone’s comments and added a bunch of goodies:
- Our new camera-based speed detection (much more accurate! )
- Brand new biking and hiking content
- City and trail names (where available) will be shown whenever you enter a new area
- Factoids for the place you’re in will pop up throughout your workout
- New look & feel
- Want to slow down and smell the roses? You can now tweak overall playback speed
- Randomly shuffled workouts
- Preferences for subtitles, speed, and autoplay are saved across sessions
- Choose your workout by time and starting point
- No matter how fast you move, the app will continue playing video until your full workout time has elapsed.
New Hiking Footage
The new hiking footage is meant for a walking pace. We’ve added three hiking packs and will add more soon!
New Biking Content
We’ve also added footage suitable for people who like road bikes. These videos are shot on streets and highways which are popular with cyclists.
Head over to the app store check out the new version of Virtual Active: BitGym Edition!
This is a huge update of the underlying tracking technology!
- The old version only used vibration to track your speed, but now it uses vibration AND camera input which is much more accurate at detecting your speed.
- The steering controls have been revamped to work with subtler motions and to center more intelligently.
- The tracking system works in all lighting conditions.
- You can play by just jogging in place in front of an upright device on a desk or table.
This image hit the front page of reddit yesterday. A pretty cool vision of the future courtesy of http://storyboardcomics.blogspot.com/
Unfortunately it won’t be possible any time soon. Michael Abrash wrote a great explanation on the valve blog:
Leave aside the issues associated with tracking objects in the real world in order to know how to virtually modify and interact with them. Leave aside, too, the issues associated with tracking, processing, and rendering fast enough so that virtual objects stay glued in place relative to the real world. Forget about the fact that you can’t light and shadow virtual objects correctly unless you know the location and orientation of every real light source and object that affects the scene, which can’t be fully derived from head-mounted sensors. Pay no attention to the challenges of having a wide enough AR field of view so that it doesn’t seem like you’re looking through a porthole, of having a wide enough brightness range so that virtual images look right both at the beach and in a coal mine, of antialiasing virtual edges into the real world, and of doing all of the above with a hardware package that’s stylish enough to wear in public, ergonomic enough to wear all the time, and capable of running all day without a recharge. No, ignore all that, because it’s at least possible to imagine how they’d be solved, however challenging the engineering might be.
Fix all that, and the problem remains: how do you draw black?
All that said, here at BitGym we’re still very excited about the future of AR and exercise. It just won’t quite look like this in the next 5 years. For a while, it will just be a lot closer to Zombies Run - a location based running game for your smart phone. The coming first wave of practical AR glasses will simply give you a better HUD you can occasionally use and some interaction methods which are better than holding your phone in your sweaty hands. Don’t get me wrong, that will unlock a lot of potential. It just won’t look quite as futuristic.
The following is a guest post from Emily Marsh.
A recent cancer research study conducted in the United States adds to other findings made over the years that moderate exercise lowers breast cancer risk. The results of the study were reported by GMA News, who stated that women who are physically active during their childbearing years experience a lower percentage of post-menopausal cancer development. This study arrives on the heels of many other recent similar studies linking active participation in moderate physical fitness to reduced risk of cancer. Even those who don’t start exercising until they’re menopausal can still reduce the risk of cancer. This and other cancer research studies find that exercise provides many benefits that help fight cancer.
- Weight Reduction. According to Boston.com, losing weight or maintaining a healthy weight during menopause can greatly reduce the risk of cancer in women. This is because the accumulation of extra abdominal fat during the menopausal years puts women at a greater risk for cancer. But doctors and researchers involved in the National Cancer Institute’s Long Island Breast Cancer Study Project state that it’s not too late for a woman to begin participating in moderate physical fitness to lose weight and reduce the presence of hormones that contribute toward a higher risk of cancer.
- Boost Immune System. Moderate exercise is an effective immune system booster that helps the body to fight many different illnesses and diseases. For years, doctors have extoled the virtues of exercise to prevent heart disease, high cholesterol and Type 2 Diabetes. Some people even avoid catching colds, coughs and the flu due to an immune system that’s been strengthened through exercise. It also now known that the immune system benefits gained through moderate exercise help in the fight against cancer, too.
- Reduce Stress. Many people lead stressful, busy lifestyles without any idea of the potential harm they’re doing to their bodies. Consistent stress has been linked to many illnesses and diseases, and reducing stress is one of the most effective ways to be proactive with your health. The Huffington Post finds that eating healthy and engaging in regular exercise is very effective in preventing the development of cancer, and stress reduction is a natural by-product of both. When you engage in exercise, the level of cortisol– the stress hormone– in your body is reduced. At the same time, endorphins– the feel good hormone– are released to help you feel calm and happy.
Moderate Exercise Options
The best way to incorporate exercise into your life on a regular basis is to engage in exercise you enjoy. Walking, biking, jogging and swimming are a few examples of moderate exercise that provide cancer-fighting benefits. A recumbent bike is an effective way to burn calories, while enjoying time outdoors. If you are using a treadmill, elliptical, stationary or recumbent bike try BitGym during your workout to keep it fresh and engaging.
Although the National Cancer Institute’s study recommends 10 to 19 hours of exercise per week, you don’t have to engage in sports or fitness activities to accumulate this amount of time. Gardening, housecleaning and walking the dog also count toward moderate exercise. If you spread activities throughout the day, it’s not difficult to engage in 1 1/2 to 2 hours of physical activity per day.
Be Proactive in the Fight Against Cancer
No matter what your age is now, it’s not too late to get active and lower your risk of cancer. Women as old as 98 participated in the breast cancer study and it was found that post-menopausal as well as pre-menopausal women all benefitted from being physically active. In fact, women in this group experienced a 30 percent decrease in the risk of cancer.
Games For Health is a yearly convention where people discuss the potential of games to influence physical and mental health. I had the chance to visit this year and also to speak about BitGym’s perspective on exergames on smart devices. Here are my high level notes from the conference.
1. My talk was on the future of mobile devices and tablets in exergaming. My thesis is that mobile devices will usher in a golden age of exergame design. I say this because collectively the game design community has only scratched the surface of what’s possible in exergames. It’s similar to where core gaming was in the early 80s. Except, in the 80s you could make a commercial game in a few man months. In the console gaming world, where most exergames live now, that’s laughable. The expectation of quality is too high. This is where mobile devices come in – they have much lower barriers to entry and expectations of polish, allowing games to be built on a similar timescale as in the 80s. If we can get core technologies that allow effective exercise games on mobile devices, it will foster a back and forth cycle of innovation in the exercise game design sphere similar to the game design explosion of the 80s and 90s.
2. The latest research in exergames is starting to get interesting. I was happy to see a lot more than just yet another measurement of exertion while playing wii sports. I saw researchers trying to determine how much coaching was necessary, if people would exergame without a forced regimen, what made some exergames fun, and how exergames effected different population demographics.
3. I enjoyed all the talks on behavior change and adherence. The rest of the tech industry is just starting to wake up to how powerful behavior change and habit formation can be, but this crowd has been there for years. Of note Jane McGonigal at SuperBetter gave an excellent presentation about their formation and growth.
The new Audi e-bike Wörthersee features a 2.3kW electric motor and an onboard computer that can connect to a smartphone, offering challenges and tips.
More at Springwise
- See our new Kickstarter!