Last week’s mHealth working group meeting laid out the opportunities, challenges, and potential of monitoring and evaluation (M&E) within the mHealth sector.

 

Several experts in the field presented their experiences of monitoring and evaluating mHealth projects, emphasizing the considerable potential that mobile projects offer in generating robust and accurate data. Kelly L’Engle, a behavior scientist at FHI 360, discussed the need for M&E in order to gauge the impact of mHealth. She claimed that mHealth technology is not being fully leveraged and that the current mHealth research “doesn’t provide evidence on actual impacts…or answers to critical research questions…”

James BonTempo from Jhpiego presented on evaluating behavior change evidence from a text message project that promotes male circumcision (MC) for HIV prevention in Tanzania. He referred to this evidence extraction as “mining the data exhaust” – that is, the data generated as a byproduct of ICTs, the “trail of clicks” that ICT users leave behind.

The MC program offered a toll-free text-messaging/SMS service. With the service, individuals could request to receive information on the benefits of MC (using key word: TOHARA), where to find MC services (WAPI), as well as receive follow-up care information after undergoing the circumcision (BAADA). While the SMS platform was intended to capture requests to the three keywords and generate access logs for system performance monitoring, Jhpiego has analyzed this data in order to see if there was a relationship between requests for the MC data and actual utilization of MC services.

The data set included 12,056 keyword requests sent by 4,954 users. After performing a Chi Square analysis on the data, the project team found that requests for information on the benefits of MC (TOHARA) was not strongly associated with going to receive the circumcision. However, it was found that those who requested where to find the data (WAPI) did have a statistically significant association with undergoing the circumcision. These results are consistent with pathway models of behavior change, indicating that those who simply wanted to know more about MC were not quite prepared to undergo the procedure, but those looking for service availability were ready to use the services.

The associations found in this particular project imply that providing text or voice messages that tell people where to get a particular service could be more effective in encouraging clients to utilize that service. M&E that finds results such as these could help project leaders design SMS services that generate more useful data.

Like most forms of M&E, there are limitations on findings that use mobile data. There is a lot of information that mobile devices do not provide, such as which phones belong to whom and who actually sends the text messages; this makes it difficult to link messages to specific individuals. In the MC case, data analysts found an association, but did not necessarily know the nature of the association. It can also be challenging to find the time or manpower to rake through the massive amount of data that is produced by mobile devices.

While M&E in mHealth has its limitations, it is difficult to find data that can be collected and analyzed as quickly, cheaply, or easily using other means. Paper, radio, and television simply do not offer the same kind of easily-mined data exhaust that mobile does.