Tuesday, 10 March 2015

Developing the "Footsies" App at PennApps

Author's Note: I have decided to do a blog post for all the hackathons/projects I have done thus far. I feel like I can explain a lot about what I did and how each of the hackathons went a lot better in a blog post than just boringly talking about it on my resume.

Author's Note 2: Thanks for Global Hackathon Seoul for giving me a Raspberry Pi as the winner of the "describe your hack" contest! I submitted this blog post as my submission!

Github Repo of the App: https://github.com/phantomkirby/pennapps

My Experience at the Hackathon


Poster of PennApps Photographed at PennApps 2015

As you can tell from the above picture, I fortunately got the opportunity to go to PennApps for the first time! (I've never went from Canada to the US just for a hackathon before!)

We figured we wanted to do a hardware and health app, considering that PennApps was promoting those two types of hacks that year. So, we got our hands on a Sensoria Sock, which is basically a sock that has pressure sensors and an accelerometer, and we could read those raw values into our Android application by bluetooth, and interpret the data from there.

Project

Conception

We figured we wanted to do a hardware and health app, considering that PennApps were promoting those two types of hacks that year. So, we got our hands on a Sensoria Sock, which is basically a sock that has pressure sensors and an accelerometer, and we could read those raw values into our Android application by bluetooth, and interpret the data from there.

We got this idea to create a gait (walking) and posture analysis/rehabilitation app. Until now, most smart fitness devices only measured heart rate, blood pressure, and other things in the body. There has never been a consumer device which can measure the pressure on the bottom of your foot (gait). It is also novel because patients can send data about their gait/walking data to doctors and it can diagnose walking problems and train/rehabilitate walking problems all through a mobile app.

Idea

Posture and the way you walk is important. Very slight variations in walking posture times thousands of steps a day can mean extra stress on various part of your foot, leg, and even the rest of your body. Thus, our target user is anyone interested in analyzing their own walking behavior. Some people may be walking wrong their whole lives without even knowing it. This app is good for people who might not even know they have common foot problems to "self-diagnose" themselves (though obviously, consulting a real doctor is better).

Product






Our app, Footsies, is currently on the Android platform. Even though Sensoria has developed a suite of frameworks for the smart sock (on iOS and Windows Phone), there has not been any major framework for Android. Thus, the technical difficulty was difficult as we needed to store/interpret all of the raw data/numbers from the sensors and manually look at all of the accelerometer data/pressure data, and calibrate the app so that we can accurately determine what is a step, and also manually make a visual map of the pressures on the feet in real time. It connects to the Sensoria smart sock via Bluetooth, and allows the user to get a variety of information from the device.

It calibrates to each users' steps' pressure values through a short and easy process of having them pose in the four basic gait phases (pictured below) for a few seconds each, allowing the app to get more a more meaningful/fine-tuned measurement for accurate diagnosis, monitoring, etc.

We had the user calibrate the sensor with the four main stance phases of gait.


This kind of screen appeared for each calibration pose the user had to do. They would press the calibrate button, and then after all the calibrations are done, main menu (the figure below this one) would pop up.

The app has three main modes: diagnosis, monitor, and training


In diagnosis mode, the user is asked to take 10 normal steps, and based on those steps the app attempts to make a diagnosis based on the gait pressures measured. In particular, we focused on being able to diagnose two types of "simple-to-diagnose" foot conditions (because we only had limited time as this was a hackathon, and could only research about these two): pronation (inward feet) and supination (outward feet).

Our app could easily diagnose this, but if we had more time at the hackathon, we could've researched more common feet problems and then we could've diagnosed those conditions with our app too.

Even if no meaningful conclusion is reached, the app graphs the data (gait phase type as a function of time) to which may help to find correlations. This graph data is also sent to Google Fit API and our own server. This server would help "connect" doctors to patients using this app by easily allowing the doctor to see the data remotely, without the patient being beside the doctor in real life. 

Screen that had an interactive circle that filled up as you took the 10 steps to diagnose your feet problem.

Note: I didn't take a screenshot of the "diagnostics results" page unfortunately... but this screen looked cool, trust me. We had a good designer (a.k.a. one of my teammates!) Below is a figure demonstrating what the graph kind of looked like (we just used a simple interactive-graph-library for Android, that's open source on Github).

Link to the library here: https://github.com/PhilJay/MPAndroidChart

In monitor mode, the user can watch the details of their every step. A "heat" map of pressure is shown, which can show if a user has a habit of leaning in- or out-wards, for example. Data collected could be sent to the doctor, so in essence, our app could help patients with foot problems by not having to force them to stay in the hospital. This app can easily be used in the patient's own home and can be used for therapy in their own home as well.

Screenshot of monitor screen. The foot at the bottom was an interactive heat map I made, that highlighted with color the varying intensities of pressure on the foot. The official frameworks for Sensoria Sock on iOS and Windows Phone had this heat-map, but Android didn't, so I made it myself by heavily calibrating/modifying/hacking.

In training mode, the user can decide on a number of steps as a goal, and the app will count them to that goal while alerting them of any bad posture steps they made along the way. Footsies even has Pebble integration; the smartwatch will alert you via vibrations if you make a "bad step".


Screenshot of training screen. The graph updates in real time each time the user took a step.