How to analyze survey data like a pro

Analyzing survery data - an Overview

A survey asks multiple questions about respondents' perspectives or experiences regarding a particular topic. While other programs help collect survey data, ATLAS.ti can help you analyze survey data, draw conclusions, and share numerical data as well as qualitative data to share your findings with others.

ATLAS.ti can help you identify trends and gain a deeper understanding of the perspectives of your respondents. Whether you are conducting qualitative or mixed-method analyses, ATLAS.ti can help you draw meaningful conclusions from your survey data.

• Uses for survey analysis
• Designing effective surveys
• Survey data analysis
• Analyzing survey data with ATLAS.ti
• Data organization tools
• Tools to analyze survey data
• How to present survey analysis
• After survey analysis

Uses for survey analysis

When researchers collect survey data, they often do so to analyze respondents' perspectives and experiences. Surveys can be used to study just about any topic, such as:

• Customer surveys to gain feedback and customer insights
• Census survey to collect demographic data
• Needs analysis survey for institutions to plan future strategies
• NPS survey (net promoter score) to assess customer experience and loyalty
• Open-ended survey to collect data for qualitative research

Designing surveys for analysis

The research questions and objectives for the study will inform the survey design. Survey analysis methods aim to take the data collected from surveys and generate valuable insights about a particular issue or group of people.

Collecting good survey data for analysis

The types of questions asked will inform the survey data analysis methods and the conclusions that can be drawn from survey respondents. Multiple-choice questions, for example, are fast and easy to analyze, while questions with open-ended responses can collect much richer data.

There are other concerns affecting survey design:

• Reaching a sufficient number of people and motivating them to answer the survey
• Ensuring that survey questions are understood correctly and collect useful data from respondents
• Crafting a survey that respondents will complete without providing incomplete responses
• Deciding between in-person surveys or online surveys when collecting data

Let's look at the different types of survey questions to understand what kinds of questions are best suited for different kinds of objectives that a researcher or data scientist may have.

Types of questions

In surveys, you usually ask a set of questions to a target group (e.g., a company's customers, a certain demographic group, etc.). Surveys ask the same questions to different people to provide researchers with comparative data between respondents and aggregate data representing a population's perspectives.

Close-ended questions

A common question type in survey data research is the close-ended question. These questions can have yes/no options, true/false options, or an extended list of multiple-choice options that respondents can choose from.

Advantages of close-ended questions

Naturally, close-ended questions are the easiest to answer and analyze because of their simplicity. With only a fixed number of answers, researchers can turn survey responses into quantitative data with numerical values for data analysis.

Cross-tabulation analysis on numerical data can also identify patterns. For example, when analyzing data from a survey about lifestyle choices, researchers may be interested in the percentage of people who say they sleep more than six hours a night and exercise twice a week (i.e., sleep and exercise may be two independent variables being assessed).

To allow for cross-tabulation, a survey needs close-ended questions about respondents' sleep and exercise habits with a list of choices for each (e.g., "less than six hours of sleep," "more than six hours of sleep"). Survey data analysis can check for the number of respondents who answered with a specific combination of choices (e.g., "more than six hours of sleep" and "exercise twice a week").

Challenges for close-ended questions

Close-ended questions restrict the array of possible answers. If a survey has to ask what kinds of exercise a respondent does, for example, it is likely impossible to list all of them at once.

Likert scale questions

A special kind of close-ended question asks respondents to select from a range of numbers. Some examples of Likert scale questions include:

• Rate your level of satisfaction with this product
• Rate the likelihood that you will use our service again
• Rate your level of familiarity with this company

• Survey items like these almost always come with a numerical scale (e.g., one to five). Taking the last survey item as an example, a one on a five-point scale may represent little to no familiarity, while a five may represent total familiarity.

Advantages of Likert scale questions

It's easy to perform a survey results analysis on Likert scale questions because the answers exist in an ordered range, unlike other close-ended questions where some items may not be related (e.g., badminton and soccer in a list of favorite sports). As a result, researchers may employ Likert scale questions when they want to perform quantitative analysis of an ordered phenomenon (e.g., level of satisfaction, willingness to buy something, etc.).

Challenges of Likert scale questions

The numbers in a Likert scale usually represent different values and are open to interpretation. For example, if someone selects three on a five-point scale to indicate their level of satisfaction with a product, do they mean that they are undecided about their satisfaction or have balanced that product's good and bad points?

Without further inquiry or more probing survey questions, knowing more about their answer may be impossible. Researchers may benefit from combining survey results with other forms of data, such as interviews and observations. In addition, open-ended questions may be included in the survey.

Open-ended questions

People are very different and have various attitudes and insights. Researchers may then want to consider asking questions in an open-ended manner to effectively analyze an issue or phenomenon.

Advantages of open-ended questions

Respondents can write free text in an open-ended question rather than choose from a list of answers. This is helpful when the list of possible answers to a question (e.g., "Who's your favorite professional athlete?") may be too long.

Another benefit is that respondents can provide detailed answers. For example, a survey on customer experience may ask respondents how they have experienced a product or service. In such a case, a respondent might need at least a paragraph to provide a sufficient answer, and these questions can generate rich qualitative data.

This approach is especially useful for uncovering and effectively analyzing details that the researcher might not know about beforehand. If the research question is descriptive, the researcher may need to allow respondents to express themselves in detail and in ways that may not be predicted.

Challenges for open-ended questions

Open-ended questions pose difficulties for survey analysis. Qualitative data from free responses require the researcher to read carefully for important data points that can serve as valuable research findings.

Moreover, it may be difficult to establish meaningful understanding of a particular theme that arises from free responses that vary widely. In a survey with a sample size of hundreds of respondents, the number of people that share the same free-response answer might be small, and researchers seeking to establish statistical significance may have to carefully consider this.

As a result, analysis of free responses is best supported by digging into respondents' answers. For surveys with many open-ended items, supplementary inquiry with respondents through interviews might also be useful.

Survey data analysis

Analysis aims to turn raw data into a compact summary that yields valuable insights. A large-scale survey may include thousands of respondents, so the researcher's goal is to describe patterns and trends among survey responses succinctly.

A survey example

When you read an article, you might come across statements like “60% of all customers” bought a T-shirt with a particular brand because of its unique design.

This is a straightforward claim about a particular product. However, it can only be made after time-consuming data collection and thorough data analysis.

Companies regularly conduct surveys to be able to make such claims. They might examine answers to survey questions and perform a statistical analysis to determine the level of demand for and satisfaction with their products among the public.

Potential survey data analysis

Answers to free-response questions can provide further insights. Respondents might elaborate on why they like the design, on the reactions they have received when wearing the T-shirt, and so on.

When combining the responses with qualitative and quantitative data, you may find that customers writing about their friends' reactions are younger than 35. When adding survey data to ATLAS.ti, the primary data will consist of the answers to the open-ended questions.

You can also incorporate and analyze quantitative data to compare and contrast responses of the various groups in your target sample.

Analyzing survey data with ATLAS.ti

ATLAS.ti is a useful software tool for helping researchers organize and analyze qualitative survey data. The next few sections detail how you can analyze survey results and build actionable insights with ATLAS.ti.

Preparing survey data

Dedicated survey platforms like Qualtrics and SurveyMonkey often use a spreadsheet to compile individual survey results into an easily searchable database. ATLAS.ti can read most forms of data, including spreadsheets from Microsoft Excel.

Importing your survey results

Survey platforms can export your survey results as an Excel spreadsheet. You can use the Import Survey tool to walk you through adding the data to your project.

You decide which section should make up the document name (e.g., the respondent number, the IP address, an email, a name), which variables should be turned into document groups for later data comparisons, and which columns in the Excel table contain answers to open-ended questions.

The software creates a document for each participant, and the answers to the open-ended questions make up the body of each document. The documents are added to their respective groups as indicated by the answers to multiple-choice questions.

Each response is even automatically coded to help you track data. A short name can be used as a code label, and the complete question can be saved in the comment space of the code. The survey data is now ready for you to continue digging deeper.

The next step could be to run a concept search. This tool suggests suitable concepts that you can use to code the data automatically. Other machine learning (ML) tools that will help you quickly generate insights are Sentiment Analysis and Named Entity Recognition (NER). Sentiment analysis codes the data by positive, neutral, or negative sentiment. NER finds persons, organizations, locations, and miscellaneous items like famous art pieces, landmarks, etc. This allows you to relate text found through the concept and NER searches to their sentiments. Which aspects were reported on positively, and which ones have been perceived as unfavorable? Further, you can run a comparative analysis exploring differences among target groups. Do men and women express the same opinions? If not, where do they differ and why? Are there differences between age groups or locations?

For further information on how ATLAS.ti works, please read this article.

Survey data analysis

In your project, you can manually read and code each document. However, ATLAS.ti also offers an array of automatic coding tools that are powered by artificial intelligence, machine learning, and natural language processing. These AI-driven tools can save invaluable time and effort.

Word Clouds

Word clouds let you see which words are used in your data and how often. This is helpful because frequently used words might reveal patterns about respondents' perspectives. Moreover, ATLAS.ti can identify and filter words by their specific part of speech (e.g., adjectives, nouns, etc.).

You can also narrow your analysis to a set of survey questions or respondents. You can use the document groups created by the Import Survey tool to analyze respondents' answers to demographic questions (e.g., age group, gender, occupation, etc.).

For example, you may be interested in how respondents aged 18-29 answer free-response items. If your survey has an age question, the Import Survey tool can create a document group for each answer to that question, which you can use to narrow your search in the Word Cloud tool.

Concepts

The Concepts tool identifies concepts and sub-concepts frequently mentioned in the data. You can select a particular concept or sub-concept and load the data where that concept is mentioned. Then, you can easily apply the code suggested by ATLAS.ti to save the data that is most relevant to you.

Text Search

Additionally, you can use the Text Search tool to look for any words or phrases that interest you. This tool brings together all excerpts across documents for quick and easy coding.

The Text Search tool also looks for a word's inflected forms and synonyms. Suppose you are looking for data where respondents are satisfied with their customer experience. You can search for the word "satisfied" along with synonyms like "content" and "delighted" and inflected forms like "satisfaction" and "satisfactory."

The search results return sentences or paragraphs across the data, which can then be coded simultaneously. Alternatively, you can look at the results of the Text Search tool and select the excerpts you want to code, either one or several at a time.

Sentiment Analysis

If you want to understand how participants feel about something, Sentiment Analysis can automatically code data for positive, neutral, or negative feelings that are expressed by respondents.

For example, this tool can be useful for looking at free responses to a question that elicits customer feedback and distinguishing between positive and negative comments. Sentiment Analysis returns a compilation of data excerpts from a set of selected documents and suggests sentiment codes you can review and add.

Named Entity Recognition

This tool finds persons, organizations, locations, and miscellaneous entities like famous art pieces or landmarks. Descriptive codes generated through tools like Named Entity Recognition are helpful for later analysis.

Named Entity Recognition analyzes selected documents and returns search results with proposed codes. You can easily review which entities appear as well as what the specific entities are (e.g., names of people, geographic locations, organizations, etc.).

Opinion Mining

This tool analyzes sentiments of identified concepts. Free-response answers to survey items can be automatically coded for expressing positive or negative opinions about something, which can allow you to quickly determine how people feel about a particular topic or phenomenon.

Tools to analyze survey data

With the data coded, ATLAS.ti has several tools that can facilitate the survey analysis process to provide helpful insights about the population you are surveying.

Query Tool

The Query Tool is a powerful search tool that lets you identify a data point based on multiple criteria. This tool can search for free responses with certain codes or include one code but not another.

This allows you to relate different parts of your codings. For example, which aspects were reported positively, and which were perceived as unfavorable?

Moreover, you can run a comparative analysis exploring differences among target groups. Do men and women express the same opinions? If not, where do they differ and why?

The Query Tool lets you construct a search using codes as search terms in conjunction with set operators and proximity operators to find relevant data segments. This allows you to find quotations with multiple conditions while excluding other conditions.

For example, you can create a query that seeks out quotations from high-income respondents with positive sentiments about your product but do not use it more than once a week. The more specific the query, the greater the insights you can extract from the data.

Code Co-Occurrence Table

ATLAS.ti's Code Co-Occurrence Table can help perform cross-tabulation analysis of survey data. You can create a Code Co-Occurrence Table to find relationships between descriptive and interpretive codes to identify trends among particular demographics.

For example, you can create a table with age groups as columns and levels of satisfaction as rows. The Code Co-Occurrence Table tool will populate the cells with the number of quotations that contain each respective pair of codes (e.g., "18-29 years old" and "very satisfied").

These frequencies will tell you the extent of the relationship between different codes, which can help to generate useful insights from your study. The tables can also be exported into Microsoft Excel, and you can export the Sankey diagrams and bar charts generated by ATLAS.ti to visually display your findings to others.

Code-Document Table

This tool resembles the Code Co-Occurrence Tool, except that it determines the frequency of selected codes in specified documents. This tool can also compare participants by their answers if you used the Import Survey tool to create document groups for different responses.

How to present survey analysis

Survey results should be concise. With the help of quantitative analysis and visualizations, a researcher should be able to briefly explain the main insights arising from their analysis of large data sets.

Quantitative analysis of survey data

Particularly in scholarly research, a researcher needs to demonstrate the trustworthiness of their survey by explaining their methodology and data collection methods. Audiences may also expect to see statistically significant research. Researchers can also rely on quantitative analysis to determine adequate sample size and statistical significance to persuade other scholars.

ATLAS.ti can facilitate further quantitative analyses by allowing researchers to export key elements of their research as Excel tables or PDF reports. Moreover, the entire ATLAS.ti project can be exported to a format that can be imported into other statistical analysis software, such as SPSS or R. Combining qualitative and quantitative analysis methods can be a great way to build deeper insights, assess how statistically significant findings are, and explore dependent and independent variables with regressions and other statistical analyses.

Survey data in visual presentation

A survey report should provide easy-to-understand research findings to make sure your survey produces valuable, actionable insights. Visuals can give a more concise way to express information better than a lengthy written explanation.

Tables produced in the Code Co-Occurrence and Table Code-Document Table tools can be exported to Microsoft Excel. These tables can summarize large sets of data in a compact space.

Bar charts and Sankey diagrams can also be exported as PDFs. You can then include these visualizations in your survey report or presentation.

After survey analysis

Researchers may use survey data to illuminate further inquiries to collect other data. When analyzing survey results, it might make sense to consider what other research questions the data might pose.

For example, in our imaginary survey about lifestyle choices and health, how many respondents answered that they don't get enough sleep but still feel they are in excellent health? If it is a common assumption that sufficient sleep is good for health, it might be worth following up with the respondents about their choices. Interviews and focus groups can elicit in-depth data about respondents' answers.

Market researchers often use these methods with surveys to further understand customer behavior. The important point is to supplement your data with further data collection methods to continue expanding your research.