mHealth isn't just about SMS & GPRS

Earlier this week I read a rather enlightening article that was shared via Twitter by my friend Lucky - Evaluating the Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice. I was particularly interested in this piece because I have been thinking a lot about evaluating mHealth initiatives (see my recent More, Better, Faster, Cheaper: An Evaluation Framework for ICT4D, eHealth, mHealth, etc. for more on this topic) and, honestly, struggling a bit with finding good evidence to support these kinds of interventions.

As evidenced by the title, the (admittedly small) study focuses mainly on evaluating error rates for different methods of mobile-facilitated data collection. The main results extracted from the abstract:

Our results indicate error rates (per datum entered) of 4.2% for electronic forms, 4.5% for SMS, and 0.45% for voice. These results caused us to migrate our own initiative (a tuberculosis treatment program in rural India) from electronic forms to voice, in order to avoid errors on critical health data. While our study has some limitations, including varied backgrounds and training of participants, it suggests that some care is needed in deploying electronic interfaces in resource-poor settings. Further, it raises the possibility of using voice as a low-tech, high-accuracy, and cost-effective interface for mobile data collection.

That's pretty impressive: an order of magnitude difference in error rates. Having this sort of data is great, but the team actually went beyond this and provided additional numbers on cost and time spent per datum. It was like I had hit the jackpot!

As luck would have it, the day after reading this I had a meeting with a colleague, Bright, from our Nigeria office who wanted to discuss opportunities to apply mobile technology to data collection. His initial thought was to use SMS and structured data. After some discussion about the monitoring needs of his program (which include the now familiar approach of a community health worker [CHW] using a notebook that is brought to the health facility on a regular basis) we quickly realized that the amount of data that needs to be collected - while not all that much - is too much for a text message. Plus, I had come to the realization a few weeks ago that SMS & structured data just don't mix:



So armed with our new data, we proceeded to talk through the results and brainstorm a bit on how they might influence a potential restructuring of the current paper-based collection process. And as we progressed I realized that the results from the study could be plugged directly into the More, Better, Faster, Cheaper evaluation framework! Well, minus the More. I grabbed a marker & ran to the whiteboard:



It was trivial to create a matrix of approaches and evaluation criteria. And writing this out made it easy to visualize the best choice for each measurement and have a more focused discussion on their relative importance. As you can see, voice leads to higher quality (Better) but SMS is the Faster & Cheaper option. Obviously it would have been a "slam dunk" if all the highest values were in the same row. But we had to decide: how will we weight each criteria in the final analysis (essentially performing a weighted sum)? After talking a bit we decided that data quality was just too important to be outweighed by the others.

Just to be on the safe side we did some quick calculations to get a ballpark figure on the cost for implementing a voice-based mobile data collection intervention for one year. With 734 CHWs reporting once a month, spending 3 minutes on each report, at a cost of 5¢ per minute, over 12 months, we came up with a cost of $1,321.20. While we don't have an exact number for how much we're currently spending to gather this data, Bright seemed pretty pleased with the figure :)

One thing we do know is that following the current processes & procedures it sometimes takes months to get good data. Part of that is due to data not being reported on a regular basis by CHWs, but it's also a symptom of having to conduct in-person site visits to investigate data quality issues (which, of course, costs money - transportation, per diem, etc). So if this voice-based approach can result in more timely data collection/reporting and can also reduce the amount of follow-up required to ensure data accuracy then it seems it would be worth the investment.

I'm keeping my fingers crossed that Bright & I will be able to successfully make the case & convince the rest of the team to try this approach. It would be great to at least pilot it in one district. I'm keeping my fingers crossed...

Apples to Apples: Conducting a Cost Analysis of mHealth Interventions

Note: I am not a clinician, a health systems expert or an economist! What follows is my naive attempt at beginning to understand what sort of cost-analysis is needed for mHealth. I make a number of assumptions along the way but try to be very transparent about them. As a good example of my ignorance, I refer to community health workers (CHWs) and show a breakdown for Preventing Mother-to-Child Transmission of HIV (PMTCT) - I don't even know if CHWs are providing PMTCT services or if it's reasonable to think that they would. Regardless, I think the basic principles of my approach and argument are relatively sound if not well implemented. Of course, you can help me work through this by leaving a comment below. And perhaps most importantly, I really like Neal Lesh & think that tools like CommCare are pretty cool :) I've only used it as an example.

Lately, I've been thinking a lot about methods for evaluating mHealth (and more broadly, eHealth) projects. And I was excited to learn about the upcoming Education Technology Debate on assessing ICT4E initiatives. As I wrote recently, I think the simple framework of "More, Better, Faster, Cheaper" provides a promising foundation for developing and conducting these sorts of evaluations. Within that framework, though, some elements are easier to tease out than others. In particular, I've spent a significant amount of time wondering how one might effectively assess if an mHealth approach is less expensive than one that is not technology supported.

I've been inspired to think more about this lately as it seems every week I learn about another mHealth project that involves putting high-end devices (e.g, Android phones) in the hands of CHWs, requires data network access, uses "the cloud" as a key element of its foundation, or depends on other advanced - and more importantly for this discussion, expensive - technologies. My knee-jerk reaction to these sorts of applications is to wonder: Is this appropriate? Do these solutions, which are available but not necessarily accessible, provide benefits commensurate with their cost?


At the recent FNIH mHealth Summit I had a conversation with Neal Lesh (I always enjoy our talks) and was giving him a hard time about the financial feasibility of rolling out something like CommCare to the 30,000 CHWs that the Ethiopian government is planning to deploy to address challenges in providing maternal and child (MCH) services. If I understand correctly, the phones (which have to be Java-enabled) that are being used in the CommCare pilot project in Tanzania cost around $100. So just providing the hardware to implement CommCare in Ethiopia would cost $3,000,000. We all know that there are plenty of other costs associated with an mHealth initiative - training, support, maintenance, replacement, etc. - but let's just work with that figure.

Neal's counter to my point was to look at that cost on a per capita basis. I think that's a great idea because it will allow us to form a picture that can then be compared to widely available data on per capita healthcare costs. According to Community Health Workers, The Ethiopian Experience there are 77.4 million people in Ethiopia (BTW, we know the calculation of that figure is fraught with issues) and women and children make up 72% of the population. So we get:

$3,000,000 / (77,400,000 * 0.72) = $0.06/person

That looks pretty good, right? 6¢ per capita? And when compared to the $13/person that was spent in 2006 (see the UN's Human Development Report for Ethiopia - 2009) that's downright cheap!

But as the title of this post implies, we need to "compare apples to apples." We need to keep in mind that CommCare most likely will not be involved in the delivery of the entire suite of MCH services, probably only a subset of them. And remember, the application does not actually deliver services directly but rather is used to support services being delivered by a CHW. So we cannot logically compare the 6¢ to the $13. We need to break that $13 down into the individual services that are being delivered and then pull out the costs for the specific ones that are being supported through the use of CommCare. And further, we need to pull out the cost of those components of the individual services that are effectively being replaced by the application. Only then will we have a more accurate picture of the relative costs.

Hierarchy of health services, their constituent components and where mHealth is being applied [click on the image to see a larger version]

In the diagram above I have attempted to give an example of how we need to break down the components of the healthcare delivery system in order to arrive at a state where accurate one-to-one cost comparisons can be made between current methods of delivery and those that are technology-supported. As you can see, that original $13 gets spread over a number of elements that span multiple levels in the hierarchy. And it's not until we get a significant number of levels deep that we begin to see where the mHealth intervention is being applied.

Hierarchy of health services and the cost per element at each level [click on the image to see a larger version]

If we make some rather sweeping assumptions (as we did above with the $3,000,000 cost for implementing CommCare) we can get a sense for how much each individual element of the healthcare service delivery system costs. The main assumptions that I've made (that are most likely rather spurious) are:

1. Costs are spread equally across elements at the same level
2. Where "..." denotes other elements at that level, only one additional element has been considered in calculating costs

Working with these assumptions, the $13 for all health services gets reduced rather quickly and is almost immeasurable at the level where the technology is being introduced. Once we traverse the hierarchy to the level where performance support and data collection are introduced the per capita cost per element is significantly less than 1¢. Given that there are two elements the total per capita cost of these components where mHealth is being introduced is just over 1¢. And that is 1/5th the cost of implementing CommCare (which you'll remember is 6¢ per capita).

So from a cost perspective it seems it might be difficult to make the case that this sort of mHealth intervention is warranted: total expenditures for these elements of healthcare service delivery would be about 5 times higher than they currently are without the technology. But remember that cost is only one component of the "More, Better, Faster, Cheaper" framework and it could very easily be the case that impact in the others - More, Better & Faster - would outweigh the negatives in the Cheaper component.

I think a similar analysis conducted by someone who actually knows what they're doing would be really helpful! Any takers? Or can anyone point me to existing work in this area?

More, Better, Faster, Cheaper: An Evaluation Framework for ICT4D, eHealth, mHealth, etc.

Earlier this year I participated as a technical expert in a consultative meeting at USAID to discuss eHealth & mHealth viz-a-viz the Agency's Global Health initiatives. Those in attendance came from many of the "usual suspects" of large, international NGOs, a few technology firms and also other departments at AID. The conversation was really no different than the many others that have focused on the same topic and that have included similar types of participants, but there was one element that was particularly unique: the proposal of a potential evaluation framework - "More, Better, Faster, Cheaper." And this same framework was the basis, a few months later, of the session, "Bigger, Better, Faster, Cheaper: How Mobile Technology is Transforming Health Programs," at the MAQ Mini-University.

Better, Faster, Cheaper: A traditional system of constraints in software development

This framework is particularly interesting to me given my background in application systems development. By removing the "More" component one is left with a constraint system that typically gets mentioned during the initial stages of an application development effort: "Better, Faster, Cheaper." The general idea is that one can achieve only two out of the three options simultaneously. So for example, if one wants to develop something good in a short period of time (Better and Faster) it will require significant investment (not Cheaper). Or if one develops something fast and without spending much money (Faster and Cheaper) the quality will suffer (not Better).

But the deceptively simple "More, Better, Faster, Cheaper" framework transforms constraints into opportunities. The idea is that through the application of ICT one can potentially:

• Replicate and scale initiatives more easily;
• Improve the quality of interventions;
• Expedite processes and procedures; and
  
• Save money over existing approaches.

And the framework readily lends itself to the development of indicators to test each potential outcome. One only needs to monitor if after the ICT intervention they're doing more, doing it better, doing it more quickly than before, or doing it for less money. (The last element - Cheaper - is arguably the most tricky to measure and will be the topic of a future post.)

As the eHealth, mHealth and other ICT4D communities grapple with the need to demonstrate the outcomes and impact of their technology-supported initiatives - a need reiterated at last week's FNIH mHealth Summit - I think the "More, Better, Faster, Cheaper" evaluation framework provides a simple yet powerful and effective means for doing so. Do you agree?

Who needs a listserv when there's Twitter?

My colleagues over at NPOKI recently sent out an email to crowdsource information about "the best listserve ever." What follows is the short back-and-forth of messages where the question is posed and I, half-jokingly at first but then completely seriously, explain how I use Twitter (and other tools) to learn and seek answers to questions.


NPOKI: When you have pressing question about your work, where do you turn for answers? While NPOKI is your trusted resource for technology solutions for results and performance management questions, we know that we don’t answer all of your critical questions. We want to know which listservs or forums you use frequently and provide useful replies to your business questions.

Me: Forget listservs. Who needs them when there's Twitter? :)

NPOKI: Thanks - Im taking you seriously... Any specific practice you use? hash tags? Do you get quality replies?

ME: It's not so much about hastags but rather about following the right people. Tools like WeFollow & other lists (e.g. the ICTWorks ICT4D Twitter list) make finding these people easier. Couple that with a tool like TweetDeck that allows you to group those people into separate feeds & you've got an amazingly effective learning tool.

As for replies, I've worked on cultivating relationships w/in the groups of people I follow (& who follow me) & so have had good response rates when looking for input. See my blog post enquiring about the meaning of mHealth for an example of how I was able to crowdsource data from Twitter. And that led to an in-person engagement @ a Technology Salon conversation [about mHealth] in DC :)

Twitter has rapidly become one of my top learning tools. It takes some work - you get out what you put in - and some getting used to, but once you get comfortable & settled it's really quite useful.


So what about you? Do you agree? Has Twitter become more useful than listservs?

uHealth, Ubiquitous Health, You Health!

In a recent series of posts on mHealth - What is mHealth all about? and What is mHealth all about? Part 2 - I grappled a bit with what the term itself actually means. Is mHealth about mobile phones and other mobile devices: the technology? Or, is it somehow more about mobility? While we may not all be in agreement as to what is actually moving - people, services, data, etc - I think most agree that the latter, mobility, gets more at the core concept of mHealth.


But, inspired by David Isaak's comment to the post, I found myself thinking that mHealth is really just one element of a larger picture. That mHealth is one "tool in the toolbox" that can be used to ensure that health services - broadly defined to include service delivery, behavior change & communication, learning, monitoring & evaluation, etc - are available whenever & wherever they're needed. That mHealth along with all the other traditional approaches, and the ones not yet even known, should be combined - or to borrow a term from the learning technology world, "blended" - to ensure ubiquitous health. uHealth. And in the end, this really is (or at least should be) the goal of any health systems strengthening initiative.

Now, I want to be clear. As with mHealth, uHealth is not about technology. I understand that the term "uHealth" has been used by the IEEE in Asia to describe the use of ubiquitous computing technology in the delivery of healthcare. But I want to advocate for a more broad usage of the term. uHealth should be about developing the right combination - engineering (& re-engineering when necessary) the right ecosystem - of tools, technologies & approaches to meet the goal of providing quality health services to all. uHealth is about you!