Data for Decision-Making
All stakeholders and partners of The Challenge Initiative (TCI) need to understand the key performance indicators used to track program performance and have access to high-quality data to regularly monitor those indicators. Only with these data in hand can TCI stakeholders ensure the project is on track to rapidly scale up high-impact family planning approaches, including adolescent and youth sexual and reproductive health (AYSRH) services. Likewise, they need to use that data at regular intervals to iteratively adjust the program strategy and implementation and management of that strategy. Good data can also make a compelling case for advocacy for needed resources on the one hand and help decide when certain locations can be “graduated” from donor support on the other hand.
TCI's MEL Framework
TCI’s Monitoring, Evaluation and Learning (MEL) Framework highlights the critical role of data use for decision-making (D4D) to track and assess the TCI model across all implementation sites. MEL activities serve three key objectives:
- To continuously monitor, assess and advocate scale up and sustainability progress and generate lessons for improving the implementation and design of future strategies, approaches and materials
- To report results of TCI interventions to provide accountability to cities, states, counties, TCI hubs and the Bill & Melinda Gates Foundation
- To feed new learning back into the project in a timely way to inform adaptations and improvements to program design, implementation, supervision and management
Learning and adaptation can occur only if relevant data are (1) available; (2) accurate and of good quality; and (3) used to inform decision-making and enhance program performance.
Thus, D4D is a key component of the way TCI does business and a foundational competency for all TCI staff and partners at the global, hub and implementer levels. It is also one of the key indicators TCI uses to measure sustainability of family planning programming at the geography level. TCI believes investing in strengthening capacity in D4D can bring high returns, not only in family planning but also in other technical health areas as most of the data used for decision-making comes from health management information systems (HMIS) that encompass all health sectors. It is important to remember that nurturing an environment that fosters data use for decision-making requires an assessment of capacity building needs during the initial steps of program design.
TCI has developed the Data Reporting and Utilization Checkup to help TCI hubs conduct a participatory assessment of such capacity in three key D4D domains:
- Data reporting
- Data review and interpretation
- Data-informed decisions
The tool measures capacity in a number of D4D indicators in these domains on a simple three-point scale (nonexistent/nascent, emerging, robust) to align with TCI’s existing three-stage coaching approach (Lead, Assist, Observe). Together, the hubs and geographies can use this as an initial assessment tool to determine geographies’ strengths and weaknesses, as a planning tool to decide which D4D areas to focus coaching efforts on and as an ongoing monitoring tool to assess capacity-building progress of coaching efforts.
A number of other resources are also available to help develop capacity in the broader but related areas of monitoring and evaluation, data collection and data analysis and interpretation. See the Box for links to key training toolkits, guides and online courses from MEASURE Evaluation and the Global Health eLearning Center.
Resources to Develop Health Information Systems Capacity
While staff and stakeholders at the national level usually have the necessary capacity to analyze, interpret and use data, that capacity often decreases as you move down the service delivery level (Harrison & Nutley, 2010). Capacity often needs to be strengthened in:
- Monitoring and evaluation (M&E) basics (See the Trainer’s Guide to the Fundamentals of M&E by MEASURE Evaluation)
- DHIS2 Fundamentals that provide an introduction to DHIS2 principles and terminology and how to use it to collect and analyze aggregate and individual-level data (DHIS2 Academies)
- Data compilation and quality assurance (See the online course on Data Quality by the Global Health eLearning Center)
- Data analysis, interpretation and communication (See the training toolkit on an Introduction to Basic Data Analysis and Interpretation by MEASURE Evaluation)
- Data use (See the Facilitator’s Guide to Building Leadership for Data Demand and Use in English and French by MEASURE Evaluation and an online course provided by the Global Health eLearning Center on Data Use for Program Managers)
How To Do It: TCI’s Data Use for Decision-Making Cycle
TCI has outlined a four-step cycle for using data to make program decisions.
Step 1: Use Existing Data to Inform Program Design and Build Data-to-Action Plans
In TCI’s D4D Cycle, local stakeholders use data at the outset to design evidence-based family planning and AYSRH TCI’s Stage 1 (Expression of Interest) and Stage 2 (Program Design) lay the groundwork for understanding the landscape and existing gaps that the city hopes to address. Assessing the state of family planning and AYSRH in a specific city requires not only baseline data but also knowledge of what data are available and how often. This information will also shed light on the quality of data for a particular city, which provides necessary context for developing a data-to-action plan not only for monitoring the Initiative’s progress but also for fostering a culture of data use for decision-making at the implementation level. Using a template adapted from MEASURE Evaluation, TCI hub staff coach city officials and health facility providers to develop data-to-action plans for regular review of their program activities to ensure the program stays on track to achieve the defined outcomes. These data-to-action plans provide a crucial bridge between the data that have been collected and analyzed and the use of that data to improve program design and implementation. To create the data-to-action plans, key stakeholders should first meet during program planning or before program implementation to decide on the specific programmatic questions they will be exploring during their routine feedback meetings, and the data and the data sources that they will use to answer those questions. The data-to-action template will prompt them to identify the following:
- The programmatic or policy questions that need to be answered to ensure the program is on track to achieve defined outcomes. For example: Are we reaching clients in need of family planning services? Are methods in stock at facilities? Are resources adequate to maintain quality of care?
- The data necessary to answer the programmatic or policy questions. These can include quantitative and/or qualitative data. Examples include the number of modern contraceptive adopters by background characteristics and the percentage of facilities with long-acting reversible contraceptives in stock. Country HMIS typically collect this type of information, but TCI hubs and geographies may need to supplement these data with complementary data collection efforts. For example, after the TCI hub in India found critical data were missing from the HMIS, including the number of women counseled on family planning and mCPR disaggregated by age and parity, the TCI hub developed and rolled out a plan across their geographies to collect and present these additional indicators on an online visual dashboard (see image below). The color scheme helps to communicate to local stakeholders whether or not TCI proven approaches are meeting the estimated levels of achievement that have been established in order to ensure rapid scale-up of the approaches and ultimately increase in mCPR.
- The data sources containing the data. Key data sources are project records, the HMIS, service delivery point assessments, client exit interviews, household surveys and Most Significant Change stories (see box below).
[DiviShortcode id="24098"]
Step 2: Collect Data on Ongoing Activities
As TCI activities are implemented, a range of stakeholders, including service providers, state data managers and TCI data managers, collect relevant data to track project implementation, service statistics, commodity supplies and broader health outcomes. Ensuring these data are of high quality is critical to making good decisions about resource allocation, planning and programming. Low confidence in the quality of health data has been shown to negatively impact use of data by program managers and other decision-makers (O’Hagan et al., 2017). While data quality has improved in a number of places, important challenges remain in many other locations. A number of factors contribute to poor data quality, including (Harrison & Nutley, 2010):
- Complex reporting procedures and multiple reporting forms
- Lack of digitization, which often translates into more human error
- High client loads at the facility level, leaving little time for providers to spend on activities outside of service delivery
To ensure data are complete and accurate, TCI staff and partners implement routine data quality checks. For example, surveys conducted to monitor family planning activities and outcomes will also serve as data quality checks of routine service statistics: if the trends in the two data sources are similar, then this validates the accuracy of the routine data. If the trends are not similar, then it can point to quality issues with the routine data, which can prompt further to identify and address the problems.
Step 3: Analyze Data Routinely and Employ Data-to-Action Plans
TCI Hubs, city officials and service providers attend monthly and quarterly meetings with the local steering committees (also known as TCI managers and implementers), where they analyze all relevant data from the four key data sources – project records, the HMIS, surveys and Most Significant Change stories – to review recent performance. Using (and adapting as necessary) the initial data-to-action plan that was made during program design (see Step 1 above), TCI stakeholders review the identified programmatic or policy questions at their routine meetings and analyze the defined data. They then complete the data-to-action plan by:
- Identifying results from the data collected thus far
- Developing specific action points to address any performance issues or identified gaps. During the semiannual program outcome review meetings, the team may identify broader actions, possibly entailing course correction and/or the abandonment of one approach or intervention in favor of another TCI evidence-based approach, than actions identified during the more routine monthly or quarterly feedback meetings.
- Determining the stakeholders who will take the necessary actions.
- Clarifying the timeline for completing the actions. An additional template is also available to document detailed steps and a schedule for completing the necessary actions.
Routine feedback meetings provide a useful opportunity to share findings in a variety of ways for a variety of objectives. One objective that is important to scaling up TCI evidence-based approaches is advocacy for family planning at national and sub-national levels. A helpful method for sharing data to a wide variety of audiences is through data visualizations since people can process and understand information faster and better when presented visually than through text-based reports.
TCI has developed an online visual dashboard that collates and presents key performance indicators across TCI geographies as well as related to TCI’s three-stage model. This includes everything from data on expressions of interest and program designs – submitted and approved – as well as financial data related to donor contributions and local geography commitments, TCI coaching support and data from TCI University, and ultimately geography-level HMIS data.
Digital tools and applications can also be used to share data with key stakeholders. For example, in East Africa, the hub shares key data from the HMIS and project MIS with the program implementation team implementing TCI proven approaches in the East Africa TCI geographies through a WhatsApp group.
In addition to data analysis and use at the regional hub and geography levels, data are sent to TCI headquarters on a monthly basis where they undergo further cleaning, consolidation and analysis. These collated data are shared back with the hubs and other stakeholders through the monthly TCI dashboard, quarterly briefs, annual reports and other formats as needed (see TCI MEL Data Flow and Utilization).
Step 4: Iteratively Adjust Activities and Identify Capacity Building Needs
Steps 3 and 4 go hand-in-hand: Once data are analyzed and actions are identified to address performance issues or gaps – or conversely, to scale up successful activities – TCI stakeholders then make iterative adjustments to activities to ensure greater success. For example, the TCI Nigeria hub used data from service delivery point assessments to advocate for facility upgrades. Kano State used the findings to install sinks and faucets at selected facilities that lacked running water. Similarly, Bauchi State just completed a number of renovations based on its performance improvement assessment.
At the next feedback meeting, existing data-to-action plans should be reviewed to assess progress on the defined actions and to identify any required follow-up actions. New programmatic or policy questions may also arise that need to be added to the data-to-action plan template. The template should be viewed as an iterative tool that is continually updated, allowing TCI coaches, managers and implementers to:
- Identify potential programmatic shifts
- Assess and address data quality concerns
- Identify new capacity building opportunities to support or enhance data comprehension and analysis skills as well as foster a D4D culture among all TCI stakeholders
Related Posts
When Data Drives Decisions: How Francophone West Africa Is Building Family Planning Programs That Last
How Data and Specialized Training Helped One Counselor Revolutionize Family Planning Services in Dhanbad
TCI’s Tools for Monitoring HIPs & HIIs Measure Coverage, Institutionalization, and Quality for Lasting Impact
