Teaching Qualitative Research Methods with ATLAS.ti: Beyond Data Analysis

October 31, 2014

Authors: Trena M. Paulus (University of Georgia) and Ann Bennet (University of Tennessee).

Abstract

During the spring, summer and fall 2013 semesters, graduate students in qualitative methods courses were required to use ATLAS.ti as a project management tool for their semester’s work in order to develop the skills they would need to continue its use during the thesis phase of their programs. In these courses students are typically engaged in independent field work projects, in which they are reviewing the literature, collecting data, transcribing, and/or engaging in data analysis. Each of these phases were conducted within ATLAS.ti and shared with the instructor at regular intervals throughout the semester for feedback. By introducing ATLAS.ti during coursework, positioning it as a project management tool in addition to a data analysis tool, and supporting students’ early experiences with its use, we anticipate that these novice researchers will be more likely to continue using the tool to support their work. Best practices recommendations include: providing adequate access and technical support, balancing methodological and technical instruction, creating meaningful student assignments, and providing effective feedback.

Introduction

Davidson and di Gregorio (2011) noted in the most recent fourth edition of The Sage Handbook of Qualitative Research that “most senior researchers in the field of qualitative research, and many rising researchers, still lack exposure to QDAS use in their graduate training” (p. 635). At the same time, members of the “digital native” millennial generation are more comfortable than ever with pervasive computing environments and, in our experience, are actively seeking ways to use the technologies for their research. These factors and other considerations have led to our integration of one QDAS tool, ATLAS.ti, into our advanced qualitative research courses in an effort to frame the tool as not just a data analysis tool but a project management tool. This reframing follows the lead of Muhr (1997) who referred to ATLAS.ti as “the knowledge workbench”, Konopsek’s (2008) description of ATLAS.ti as a “textual laboratory” and diGregorio and Davidson’s (2008) conceptualization of QDAS tools as supporting “e-projects” as a research design.

While misunderstandings, skepticism and distrust of QDAS persist for a variety of reasons (Davidson & di Gregorio, 2011), any new technology does indeed challenge the way things have traditionally been done. Rogers’ (2003) diffusion of innovations theory argued that how the new innovation is communicated to people is important and suggested that both early adopters and resisters can be highly influential to those who remain undecided. In the case of qualitative research, it is possible that the way that the established scholars in the field, including methods instructors, introduce novice scholars to new technologies may shape how they adopt the tools in their own practice. For this reason we decided to make the use of QDAS “the norm” in our qualitative research courses by providing support and “demystification” of their features.

QDAS tools, we argue, have the potential to increase reflexive and ethical practices, transparency of choices and collaboration during the research process (Paulus, Lester & Dempster, forthcoming). Our goal for requiring the use of ATLAS.ti as part of coursework, and as a project management tool, was to increase student comfort level with ATLAS.ti early in their research careers, to build a network of users on our campus, and demonstrate its utility beyond data analysis for their dissertation and future research work. In this paper we report on the introduction of ATLAS.ti at our university and its subsequent integration as a required component of three advanced qualitative research courses. Through a reflective practice approach, we describe our course design decisions, outcomes, and “best practices” for integrating QDAS tools into qualitative methods courses.

Context

Since 2008 our university has offered a 15 semester hour graduate certificate in qualitative research methods (coordinated by first author Paulus), most of which are taught by faculty in the Department of Educational Psychology and Counseling as service courses for our College. In January 2013 our university acquired a site license for ATLAS.ti along with 20 hours of dedicated graduate assistant support for qualitative research (second author Bennett) funded by the university’s Office of Information Technology Research Computing division. Bennett provides workshops, class visits and individual consultations for issues related to qualitative research design, including the use of ATLAS.ti. These circumstances made it feasible for the first time to begin to integrate ATLAS.ti into our advanced qualitative methods courses. (While we provide an overview of ATLAS.ti in the introductory qualitative methods course, students are not required to use it for their work.)

Table 1 describes the courses in which ATLAS.ti was integrated in 2013.

Table 1. Description of courses.

Table 1. Description of courses.

In the spring 2013 Advanced Qualitative Research, students created their own individual project proposal for the course, a project in which they were required to demonstrate both methodological competence (generally by completing a methodological literature review) and data analysis competence (generally by analyzing pilot study data.) During this first semester of integrating ATLAS.ti, students were required to submit one HU several times throughout the semester. The HU contained their individual project proposal, two interim progress reports and the final project report, as illustrated in Figure 1.

Figure 1. Spring 2013 Assignments Submitted as an HU.

Figure 1. Spring 2013 Assignments Submitted as an HU.

The instructor provided feedback as memos and comments on each iteration of the HU as illustrated in Figure 2.

Figure 2. Instructor feedback and student response as a memo.

Figure 2. Instructor feedback and student response as a memo.

The goal of this requirement was for students to become comfortable with and proficient in creating an HU, adding primary documents, reading comments and memos, and bundling and unbundling the HU for feedback. ATLAS.ti as a project management tool can be used to document decisions, adding transparency to research work, work in teams for a more collaborative approach, and engage in reflexivity through writing regular reflective memos. Students were encouraged, but not required, to create a second HU in which to conduct their methodological literature reviews and/data analysis projects, and most did so, further developing their proficiency with the tool as they neared the dissertation phase of their work. See Figure 3.

Figure 3. Student project entering field notes as new text document, coding and memoing data.

Figure 3. Student project entering field notes as new text document, coding and memoing data.

In summer 2013 the students in Digital Tools for Qualitative Research were also required to use ATLAS.ti (some for the second time) as a project management tool to submit their required assignments for instructor feedback. These assignments included two “skill builder” activities in which students chose two tools (e.g.citation management software, transcription software, data analysis software) to master. Nearly all of the students chose to focus on a particular use of ATLAS.ti (for literature reviews, transcription, or data analysis) as part of these skill builders. See Figure 4.

Figure 4.Summer 2013 Assignments submitted as an HU.

Figure 4.Summer 2013 Assignments submitted as an HU.

This semester, fall 2013, students in Discourse Analysis in Educational Environments, some of which have also taken the previous two courses, are being required to use ATLAS.t. Rather than as a project management tool to submit course assignments, however, all students will conduct a mini-literature review (of 5 sources), transcribe an audio or video file and analyze their data within ATLAS.ti. Because, unlike the previous two courses, this course is focused on one particular type of analysis, the students can conduct their analysis in the tool and receive instructor feedback on it.

In addition to what has been described thus far, students in all three courses were/are required to engage in reflective journaling. As reflexivity is a key part of qualitative research (Watt, 2007), students were asked to reflect regularly on course readings and their learning experiences on individual blogs, and these reflections were valuable as formative evaluation data points throughout the course. These blog posts, all course materials, instructor communications and instructor reflections are currently being systematically analyzed in order to compile best practices for the use of CAQDAS tools for teaching qualitative methods. This initial paper reports on our reflections, as instructors, on the course design, outcomes and tentative best practices that emerged from these first iterations of ATLAS.ti integration.

Best practices

Provide adequate access and technical support

We found it important to let students know about the pre-requisite skills and tools they would need to be successful well in advance of the course. For example, students needed access to a PC machine with administrator privileges. Since a good number of our students are Apple/Mac users, this issue required individual consultations well in advance of the start of the course about how best to access the program. (A set of netbooks had been purchased by our department for students to check out specifically for this purpose.)

Students also needed to be proficient in the use of the cloud-based shared folder system Dropbox (or Google Drive) for sharing HUs with the instructor. The instructor set up a shared folder for the class with subfolders for each student (see Figure 5). Students were to bundle their HU and put it in the folder, after which the instructor would move it onto her computer for feedback, then rebundle and put it back in the folder. This helped keep student work organized and accessible throughout the course.

Figure 5. Shared Google Drive for assignment submission.

Figure 5. Shared Google Drive for assignment submission.

Students were also instructed to have the site license downloaded and successfully launched prior to the class. All of this preparatory work was important so as not to spend valuable class time on trouble-shooting technical difficulties. Without Ann’s support as the graduate assistant dedicated to qualitative research design it is unlikely our attempts to integrate ATLAS.ti into our courses would have been successful. She held introductory training sessions prior to the start of each semester, and for students who could not attend we directed them to webinars, tutorials and/or an individual consultations to get everyone up to speed before the course even began.

Balance methodological and technical instruction.

We felt it was extremely important for the ATLAS.ti component of the course to not overshadow the primary focus of the class on developing methodological competence, though some may argue that the two are becoming increasingly intertwined. While the Digital Tools course was obviously focused more on technology, even that course (and the accompanying textbook) is situated primarily in the phases of the qualitative research process, before moving on to discussion of new tools that could support it.

While at least one class session each semester was dedicated to an overview of ATLAS.ti in the context of what would be expected in that particular course, student support for using the tool typically took place as individual consultations outside of class with Ann. In this way course time could be spent on issues of methodology and not just technology. Individual consultations were mostly on topics related to setting up the HU (organization) and basic functions of ATLAS (coding). Most of the students wanted to be sure everything they needed for the class was placed in the HU correctly. Many inquired about families and how they worked, the difference between supercodes and families, and how to create and modify families. Once the students received HUs back from the instructor, a few students needed assistance accessing comments and memos or how to reply to them.

One of the biggest challenges was helping students understand how to bundle, share and unbundle their work. This was particularly challenging because the instructor stores her own work on the shared university server, which makes the files more difficult for students to recognize (see Figure 6).

Figure 6. Unpack copy bundle error messages create student confusion.

Figure 6. Unpack copy bundle error messages create student confusion.

Create meaningful student assignments.

We have found that course assignments in advanced qualitative research courses ideally provide the flexibility for students to pursue their own research agendas and make progress towards their dissertation. This is important at our institution because students come to these courses from a variety of program areas (sport studies, teacher education, nutrition, communications, educational leadership, English, business) and with a variety of preferred research approaches (ethnography, phenomenology, case study, etc.) Historically, Advanced Qualitative Research Methods has been geared toward students working on their dissertation proposals or engaging in pilot studies. Thus we limited the requirements for using ATLAS.ti in the spring 2013 course to submitting project proposals and progress reports and learning to become comfortable with ATLAS.ti as a project management tool. Thus we were pleasantly surprised that a good number of students did take the opportunity to learn ATLAS.ti beyond what was required.

In the spring 2013 course a practice HU was created and shared with the students to use as a “playground” in which to demonstrate basic functionality (adding primary documents, memos, codes, network views, user accounts, bundling) as well as allow students to become comfortable manipulating an HU (see Figure 7). Soon, however, students who had their own data preferred to work with their own HU rather than the practice HU. Others learned to use the transcribing features and still others experimented with conducting their literature review in an HU.

Figure 7. Practice HU created for Spring 2013 class.

Figure 7. Practice HU created for Spring 2013 class.

Peer workshops are a major component of the Advanced Qualitative Research course during which students engage in peer feedback, collaborative analysis and discussion. In the spring, one student group tried to engage in collaborative analysis using ATLAS.ti, but we did not realize this soon enough to help them create their HU in a way that would allow successful merging. In the future we will build in collaborative analysis as part of the course in order to demonstrate how to set up an HU that is ready for teamwork. We also feel that incorporating more peer feedback and workshop opportunities across courses will be a good next step.

In the summer 2013 Digital Tools course more ATLAS.ti features were introduced since the purpose of the course was to learn new tools. More extensive demonstrations and workshops took place around reviewing the literature, collecting data (field notes), coding/memoing/creating networks, transcribing, direct image/audio/video analysis, and even importing survey data. Students had the choice, however, to focus on two tools for the purpose of their skill builder assignments. Nearly every student chose to master some aspect of ATLAS.ti for at least one of their assignments. These included using ATLAS.ti for conducting a literature review, transcribing an audio file, and analyzing a data set. These were submitted as bundled files and reviewed by the instructor, who could then make comments and suggestions as to their efficiency and effectiveness with the tool.

Discussing ATLAS.ti and other new tools in the context of affordances and constraints is particularly important to keep the focus on methodology rather than only technology. Clearly there are tradeoffs whenever new tools are adopted, and the ability to discuss these in an informed manner can go a long way towards encouraging the adoption of the best tools at the appropriate time. The methods proposal paper in the Digital Tools class required students to make a case for which tools they would use to conduct their proposed research. By learning how to provide a justification for the use of a tool, including its affordances and constraints, students were preparing themselves to talk in an informed manner with their committee members who may not be as familiar with the tools or understand how QDAS could contribute to the transparent, reflexive and collaborative nature of their study (see Figure 8).

Figure 8. Rationale for ATLAS.ti use as part of final methodology proposal in Summer 2013 course

Figure 8. Rationale for ATLAS.ti use as part of final methodology proposal in Summer 2013 course

This is particularly important as many senior faculty members at our university remain skeptical of and even resistant to the use of QDAS tools.

Students in this summer’s Digital Tools course recommended that ATLAS.ti be incorporated even more thoroughly throughout the course. They suggested that we require all students to complete an entire project from start to finish within ATLAS.ti. This suggestion is being implemented as part of this fall’s Discourse Analysis class, as it is easier to do this when all students are using the same research methodology and doing the same kind of analysis assignments.

Provide effective feedback.

In all classes, students regularly submitted HUs as bundled files uploaded to Dropbox for instructor review. This modeled the use of ATLAS.ti as a collaborative tool – both in terms of project management and, as will be the case for the fall 2013 Discourse Analysis courses, for data analysis. By creating user accounts, it becomes clear who has contributed what to the HU (see Figure 9). By using comments and memos to provide feedback and engage in conversation with the students and their work, ATLAS.ti is positioned as a tool that affords the ability for researchers to transparently document their work and share it with others. This kind of collaboration was not really possible prior to the development of QDAS tools, and is one of their great strengths.

Figure 9. Author names reflect which team member created memos.

Figure 9. Author names reflect which team member created memos.

Using ATLAS.ti in this way, however, requires extensive use of the families and comments features to keep the HU organized. More attention is needed in future classes to convey to students the importance of organizing the HU in a way that someone else could understand. For example, deciding when to use primary document families and when to start a new HU was often a difficult one for students to make without having extensive experience with how the software works.

Throughout, however, our feedback was intended to convey that students should view the ATLAS.ti requirement as an opportunity to learn a QDAS tool in a safe space. Coursework is a lower-stakes environment than the dissertation itself, and with instructor and dedicated graduate assistant support, this is an ideal environment in which to invest the time in learning a tool that will have a great payoff down the road. We believe that early research experiences are crucial, with analytic methods and processes being put into place that become more difficult to change later. Thus, learning how to use a QDAS tool early on, we hope, will result not only in continued use of the tool but the ability to pick up other tools later and collaborate with researchers in a more transparent and reflexive way.

Conclusions

Integrating ATLAS.ti into the coursework has resulted in some challenges. Not all senior faculty whose students take our courses are comfortable with QDAS tools and some careful navigation of questions and concerns has been necessary to avoid alienating our colleagues. By introducing ATLAS.ti during coursework rather than at the dissertation phase, we are attempting to position it as a project management tool rather than solely a data analysis tool and to demystify its functionality and uses. This has allowed features other than data analysis, such as the report writing through memos and transcription tools, to be highlighted and explored in a safe and supported space. By supporting students’ early experiences with its use, we anticipate that they may be more likely to continue using the tool to support their work.

Based on feedback from the summer course, in this fall’s discourse analysis class we are also moving beyond submitting project reports to requiring a mini-literature review and discourse analysis of data to take place within ATLAS.ti. This will be possible in large part because approximately one third of the students were in previous classes and are familiar with the tool and will be able to support their peers. To prepare students for the challenges of teamwork, this fall we will not only provide instructor feedback, but have students engage in peer feedback as well.

In order to effectively support students learning ATLAS.ti, we cannot emphasize enough the importance of having robust technical support and training, as instructors will likely not have the time to troubleshoot 15 or more students learning the tool for the first time. Having support available also reflects the institution’s commitment to the value of the tool, which encourages more people to attempt to learn it, and allows instructors to focus on issues of methodology rather than technology.

References

Davidson, J., & di Gregorio, S. (2011). Qualitative research and technology: In the midst of a Revolution (4th ed.). In Denzin & Lincoln The Sage Handbook of Qualitative Research. London, England: Sage Publications.

di Gregorio, S., & Davidson, J. (2008). Qualitative research design for software users. NewYork, NY: McGraw Hill/Open University Press.

Konopasek, Z. (2008). Making things visible with ATLAS.ti: Computer-assisted qualitative analysis as textual practices. FORUM: Qualitative Social Research, 9(2), Art. 12.

Muhr, T. (1997). ATLAS.ti 5: The Knowledge Workbench. Berlin: Scientific Software. Development.

Paulus, T., Lester, J. & Dempster, P. (In press), Digital Tools for Qualitative Research. London, Sage.

Paulus, T., Lester, J., Britt, V. (In press). Constructing hopes and fears: A discourse analysis of introductory qualitative research texts. Qualitative Inquiry 19(9).

Rogers, E. M. (2003). Diffusion of Innovations (3rd ed.). New York: The Free Press.

Watt, D. (2007). On becoming a qualitative researcher: The value of reflexivity. The Qualitative Report, 12(1), 82-101.

About the Authors

paulusTrena M. Paulus, Ph.D., is a Professor of Qualitative Research at the University of Georgia in the College of Education. She holds her M.S. and Ph.D. in Education from Indiana University and an M.A. in Applied Linguistics from Ohio University. Her research interests include digital tools for qualitative research and the application of conversation and discourse analysis methodologies to online talk.

bennett-1

Ann M. Bennett is a doctoral candidate in Theory and Practice in Teacher Education at the University of Tennessee. She holds a B.A. in History and Political Science, a B.A. in Classics, and an M.S. in Education from the University of Tennessee. Her research focuses on the use of anti-deficit frameworks to develop counternarratives surrounding literacy achievement in impoverished and urban areas.

Corresponding author: Trena M. Paulus. University of Georgia, Qualitative Research Program, 850 College Station Road, 410 River’s Crossing, Athens, GA 30602, tpaulus@uga.edu

Share this Article