Mixed Methods Evaluation of Home Visiting Program

March 8, 2016

Download article in PDF

This month, we interview Nane Zadouri, a researcher at WestEd Center for Prevention and Early Intervention, a California-based non-for-profit organization.

Could you tell us something about the organization you work for and your professional background and research interests?

WestEd is a nonpartisan, nonprofit research, development, and service agency that works with education and other communities throughout the United States and abroad to promote excellence, achieve equity, and improve learning for children, youth, and adults. Success for every learner is WestEd’s prime focus and has been for close to five decades. WestEd staff meet the needs of our clients and customers through consulting & technical assistance, evaluation, policy analysis, professional development, and research.

IMG_2131I am currently the research manager for the California Home Visiting Program External Evaluation under the supervision and direction of the Principal Investigator, Dr. Karen Moran Finello. This external evaluation is being conducted by a team at WestEd to evaluate home visiting programs in California. It is federally-funded through HRSA’s Maternal, Infant, and Early Childhood Home Visiting Program (MIECHV). and California was selected as a recipient. We are sub-contracted through the California Department of Public Health.  My research interests are in the field of early intervention, infant and early mental health, and family health services and programs. I strongly believe in the importance and need for early childhood care programs and family serviced for all children and family. I see early childhood and family mental health and school-based services as my niche.

In what study are you using ATLAS.ti?  Tell us something about it.

We are using ATLAS.ti in the project The WestEd External Evaluation of the California MIECHV Program focused on Home Visiting Services for High-­Risk and Hard to Engage Populations in California Study. The study’s principal investigator is Dr. Karen Moran Finello.

The purpose of the California Home Visiting Program (CHVP) competitive project, funded by HRSA’s Maternal Infant and Early Childhood Home Visiting Program (MIECHV), is to evaluate mechanisms for the engagement and retention of a diverse set of high-risk populations in ten unique communities located in rural areas, the central valley, large urban areas, and moderate sized cities throughout the state. The ten communities have reported significant family and community risk factors, including cultural and language barriers; limited access to prenatal care; teen parents; domestic violence; substance abuse; high rates of mental health issues; high rates of child mal-treatment; and high rates of infant mortality and low birth weight as well as transportation and housing issues.  Families are enrolled in home visiting services from the prenatal period until the child turns age three.

The external evaluation conducted by WestEd is utilizing quantitative and qualitative evaluation methods to rigorously monitor and document factors contributing to the successes and challenges seen in the delivery of home visiting services, with the ultimate goal of building knowledge to support the improvement of the service delivery system for home visiting in California. The new phase of the competitive project is focused on supporting home visiting program staff in their efforts to coordinate services within high-risk and hard-to-engage communities in order to reduce the impact of toxic stress upon child and family development.  The WestEd evaluation team is continuing to build upon important information already gathered through site visits, on-line surveys, phone interviews, focus groups, and written mailed family surveys in order to examine, in-depth, the critical factors that impact engagement and retention of high risk families of very young children.

How are you using ATLAS.ti in this project?

I have been using ATLAS.ti extensively in this project, both during Phase I and currently in Phase II. As a mixed-methods design, ATLAS.ti was used for analyzing site visit interviews conducted with home visiting staff and leadership at each site and for the family focus group data during both baseline and follow-up periods. There are over 150 Primary Documents that were assigned to families, coded, and analyzed into memos and well over 20 different family focus group data files (see figures 1, 2 and 3 below).  Also for interim-period evaluation tasks, family focus groups were conducted at select sites, online surveys were used to get updated data, and we conducted mailed family surveys. The open-ended data questions from these evaluation activities was also assigned into the HU and coded as needed to create an overall project file where all the documents are stored, organized by Document Family (by interview type, date, site location), and code lists exist that were a priori and deductively created. As the data was analyzed by coding (myself-individual user), preliminary memos were written about quotations, themes, key issues that emerged. This was always reviewed and discussed with the PI to confirm reliability.

Figure 1. The document family structure

Figure 1. The document family structure.

Figure 2-The project code system

Figure 2-The project code system.

Figure 3. The system of memos.

Figure 3. The system of memos.

Most importantly, because I did not conduct the site visit interviews nor the family focus groups, I was neutral in coding the raw data based on what was recorded by the interviewers themselves onto Word documents using the interview protocols. I developed codes based on question types and themes for most of the primary documents. Some codes overlapped and were used throughout the primary documents. Some codes were unique to only a certain interview family or time period (i.e. baseline versus follow-up). Based on this, preliminary findings were written and reviewed by the PI always before preparing into summary reports disseminated to the California Department of Public Health and at times select reports were sent to the sites.

The real highlights and key functions of ATLAS.ti for this project have been plentiful. Some of these are the following:

  • I have been able to store and manage in an organized manner a LOT of raw data files overtime and add to the HU additional data files based on the iterative process.
  • I have been able to organize our raw data by timeframe, site location, interview type, informant role, and evaluation task (i.e. focus group, interview, online survey, mailed survey).
  • I have used the tools and features that are built-in to print code books that I share with the team and store in project files and review periodically.
  • I have learned how to use the Query Tool for code co-occurrence such as when a quotation was “memorable” and “family story.” Along with this the Word Cloud was used occasionally to get a glimpse of the raw data files before delving into the coding.
  • I also became familiar with the network views initially to link codes and explain relationships and some quotations but have focused on other techniques mainly.
  • And as always, the copy bundle feature and tool has been the best way to back-up this large HU file.

Would you like to add a few final words?

The ATLAS official training by Dr. Contreras was very helpful in the beginning. I highly recommend it for all beginners. Also we used the recommended trainers as project consultants (Dr. Nick Woolf & Dr. Kristin Kostick) to help us out as needed during the early years of the project and they were great! I continue to enjoy receiving and reading the newsletters, checking out the website periodically for updates and resources and overall this is a good software program for qualitative analyses.

Thank you!      

Contact information for Nane Zadouri

6 North 1st Ave
Suire 105
Arcadia, CA 91006
Phone: 626-574-6904
Email: [email protected]

Share this Article