Attention Check Questions: Use Carefully in Research Surveys

A perfectly designed survey is useless if the people taking it are not reading the questions.

To filter out bots, speeders, and distracted respondents, researchers rely on attention checks.

But dropping a blatant trap into the middle of a thoughtful questionnaire carries a hidden cost.

It signals to the participant that you do not trust them.

Getting this balance right determines whether you end up with high-quality data or an alienated panel.

What is an attention check in a survey?

An attention check is a specialized survey item designed to measure a respondent's diligence rather than gather substantive data about their opinions or behaviors.

When people take long surveys, cognitive load builds up.

To cope with this mental fatigue, respondents often engage in satisficing - taking mental shortcuts, skimming for keywords, and choosing the first acceptable answer rather than the optimal one.

Attention checks disrupt this behavior by requiring the reader to process the actual text of the question to answer correctly.

If a participant fails the check, it indicates they are likely satisficing or rushing through the rest of the survey.

There are several ways to format these checks.

The most effective ones blend naturally into the surrounding questions so they only catch genuine inattention.

Here are three distinct examples of attention checks used in research:

• The directed response This type gives a clear, simple instruction hidden within a standard question format. To show that you are reading carefully, please select 'Strongly disagree' for this row.

• The implausible behavior This check asks about a habit or event that is practically impossible, filtering out respondents who are blindly clicking "Yes" or "Frequently" to every option. In the past twelve months, how often have you driven a submarine to work?

• The forced text entry This requires the user to type a specific word into a box, which prevents automated bots from simply selecting random radio buttons. Please ignore the question below and type the word 'blue' into the text box.

Designing these questions requires care.

If your check is too confusing, you risk failing honest participants who simply misunderstood your phrasing.

• ❌ Weak: Which of the following is not a non-living thing? Please select 'tree' to prove you are reading.

• ✅ Strong: To help us verify data quality, please select 'tree' from the list below.

Why it works: The weak version uses a double negative that increases cognitive load unnecessarily, while the strong version is plain and direct.

Why do researchers use instructional manipulation checks?

Instructional manipulation checks (IMCs) are a specific, more complex type of attention check.

Instead of a simple trap question, an IMC typically presents a large block of text that looks like standard survey instructions or a reading comprehension passage.

Buried inside that text is a specific instruction telling the participant to ignore the standard question format and do something unexpected - like clicking a small icon or leaving a question blank.

Researchers use IMCs because they are highly effective at catching professional survey takers.

Many online panel participants have learned to spot basic trap questions instantly.

An IMC forces the respondent to actually read the preamble, not just the question stem.

However, using IMCs introduces a significant trade-off between the purity of your data and the goodwill of your respondents.

When academic researchers use IMCs, they must carefully weigh these factors.

A panelist who feels tricked by a hidden instruction often shifts their mindset.

Instead of acting as a cooperative partner providing honest opinions, they begin treating the survey as an adversarial test.

This shift can damage the quality of the data in the sections following the IMC, as the participant becomes hyper-focused on finding the next trap rather than reflecting on their actual feelings or behaviors.

What are the most common types of trap questions?

Beyond the complex IMC, survey designers rely on a variety of simpler trap questions to monitor data quality.

Choosing the right type depends on whether you are trying to catch automated bots, distracted humans, or people who are simply trying to please the researcher.

Each of these traps serves a different diagnostic purpose.

Directed response questions are the safest and most common.

They are transparent and do not require the participant to lie or think deeply.

Because they are simple, they do not accidentally filter out respondents with lower reading comprehension levels.

Red herring questions are particularly useful in market research and product surveys.

When you ask participants if they have used a list of tools, some will check every box to appear knowledgeable or to qualify for a higher incentive.

By including a completely fabricated product in the list, you can easily identify respondents who are exaggerating their experience.

This relies on understanding acquiescence bias - the psychological tendency for respondents to agree with the researcher or claim knowledge they do not have.

Reverse-worded pairs are less of a trap and more of an analytical tool.

If a participant strongly agrees that they love a product, but also strongly agrees that the product is completely useless, they are likely straight-lining.

Straight-lining happens when a respondent selects the exact same column down a massive grid of radio buttons just to get past the page.

Implausible fact questions are blunt instruments.

They are highly effective at catching careless clickers, but they must be absurd enough that no genuine participant could accidentally agree.

If you ask, "Have you ever been to Antarctica?", you might actually have a respondent who has been there.

Asking if they commute via submarine removes any ambiguity.

How do attention checks affect participant behavior?

When you insert a trap into a survey, you alter the environment.

Survey taking is inherently a social exchange, even when conducted through a computer screen.

The researcher asks for information, and the participant provides it under the assumption that their time is valued.

Attention checks can quickly violate this unwritten contract.

If a participant is answering thoughtfully and suddenly encounters a question that assumes they are lying or not paying attention, it breaks their flow.

This is especially true if the attention check is written in a condescending or aggressive tone.

Expert tip: Keep your attention checks polite and transparent. Explain briefly why the check is there, such as To help us ensure our data is accurate, please select... This maintains the participant's dignity and keeps the tone collaborative rather than adversarial.

When participants encounter adversarial design, their behavior changes in predictable ways.

• Increased drop-off rates The most immediate effect of a frustrating attention check is form abandonment. When faced with a confusing trick question, many respondents simply close the browser window.

• The Hawthorne effect When people know they are being actively tested on their attention, they alter their natural reading habits. They may spend an unnatural amount of time scrutinizing simple questions, inflating your completion time metrics.

• Retaliatory data entry A frustrated participant who feels disrespected may decide to finish the survey just to get their incentive, but they will intentionally provide garbage data for the remaining questions. They might write keyboard smashes in open text fields or select contradictory answers out of spite.

To mitigate these effects, the placement of the check is just as important as the wording.

Placing a harsh attention check on the very first page sets a tone of suspicion.

Placing one on the very last page is often useless, as the respondent has already provided all their data.

The ideal placement is roughly one-third of the way through the survey, just as natural fatigue begins to set in, serving as a gentle reset for their focus.

How do you set up data quality validation in Google Forms?

Google Forms does not have a native "attention check" button, but you can build highly effective traps using its built-in logic and validation rules.

The goal is to design a system that either prevents bad data from being submitted or silently flags the respondent for later removal.

If you want to stop a respondent immediately when they fail a check, you use response validation.

If you want to let them finish but separate their data, you use section branching.

Here is how to set up silent section branching to screen out failed checks:

1. Create your survey sections

   Divide your survey into at least three parts. Section 1 contains your introduction and early questions. Section 2 contains the core of your survey. Section 3 is a polite closing page (Thank you for your time. Your responses have been recorded.).

2. Add the attention check question

   At the bottom of Section 1, add a multiple-choice question. Make it a directed response, such as Please select 'Green' from the options below. Provide three color options.

3. Enable section branching

   Click the three-dot menu in the bottom right corner of the question box and select Go to section based on answer. Drop-down menus will appear next to each multiple-choice option.

4. Route the correct answer

   For the correct option (Green), set the drop-down to Continue to next section (which leads to your core questions in Section 2).

5. Route the incorrect answers

   For the incorrect options, set the drop-down to Go to section 3. If a participant selects the wrong answer, they will completely bypass your core questions and be sent straight to the end of the form.

This method is ideal for data integrity because it does not alert the participant that they failed a trap.

If you use response validation to display an error message (like Please read the question carefully), the respondent will simply change their answer until the error goes away.

That defeats the purpose of the check entirely, as you have now trained an inattentive person to pass the test without actually improving their focus.

If you must use a text-based attention check, you can enforce it using regular expressions.

1. Add a short answer question

   Write your prompt: Please ignore the question above and type the word 'apple' here.

2. Open response validation

   Click the three-dot menu and select Response validation.

3. Set the rule

   Change the first drop-down to Regular expression and the second to Matches.

4. Enter the pattern

   In the pattern box, type (?i)^apple$. This specific regular expression ensures the form only accepts the exact word "apple", but the (?i) makes it case-insensitive, so "Apple" or "APPLE" will also work.

5. Add a custom error text

   Leave the Custom error text blank if possible, or provide a generic message. Giving away the answer in the error text ruins the trap.

By using these native Google Forms features, you can automate your data cleaning process before the results even hit your spreadsheet.

When is it ethical to exclude participants who fail a check?

Once your survey is complete, you will inevitably have a spreadsheet containing failed attention checks.

The immediate instinct is to delete those rows to clean the dataset.

However, aggressively throwing out data introduces ethical and methodological risks.

Before you delete a response, you must consider whether the participant failed because they were truly inattentive, or because your check was poorly designed.

Complex IMCs, in particular, tend to penalize specific demographics.

If your hidden instruction is buried in a dense paragraph of academic English, non-native speakers are far more likely to fail it.

Similarly, neurodivergent respondents or older adults may process large blocks of text differently.

If you automatically exclude everyone who fails a complex reading test, you are not just filtering out bad data - you are systematically erasing specific voices from your research.

This creates a heavily biased sample that does not accurately represent the population.

To manage this fairly, researchers should establish strict, pre-registered exclusion criteria before the survey launches.

If you notice a massive failure rate on a single attention check (for example, 40% of your panel gets it wrong), the problem is almost certainly the question design, not the respondents.

In these cases, the most ethical approach is to throw out the attention check itself, rather than discarding half of your participants' hard work.

Always look at the totality of the respondent's behavior.

If someone fails a single trap but writes a thoughtful, detailed paragraph in an open-ended text box, they are clearly engaged with the survey.

Human error happens.

Throwing out rich qualitative data because of a minor technical failure on a trick question undermines the ultimate goal of your research.

FAQ

Can attention checks introduce bias into survey results?

Yes, poorly designed attention checks frequently introduce demographic bias. Complex reading traps disproportionately screen out non-native speakers, older adults, and neurodivergent individuals. If your filtering removes these groups systematically, your final dataset will skew heavily toward younger, highly educated native speakers, compromising the external validity of your research.

How many attention checks should be in a 10-minute survey?

A standard 10-minute survey should contain no more than one or two simple attention checks. Adding more than this increases cognitive load and makes the survey feel like an interrogation. If you place one near the middle to catch waning attention, that is usually sufficient to gauge the overall quality of the response.

Do attention checks work on mobile devices?

Attention checks often fail on mobile devices because of how screens format text. A large paragraph that looks fine on a desktop monitor becomes an endless wall of scrolling text on a phone, causing even diligent participants to skim and miss hidden instructions. If your panel includes mobile users, you must keep your checks extremely brief and visually simple.

What is the difference between a speed trap and a trap question?

A trap question relies on the content of the answer, forcing the user to select a specific word or avoid a fake concept. A speed trap relies entirely on timestamps, measuring how long a respondent spent on a specific page or the entire survey. If a participant completes a five-minute reading block in twelve seconds, they are caught by the speed trap, regardless of which buttons they clicked.

Balancing data integrity with a respectful respondent experience is the hardest part of survey design. If you are regularly building complex questionnaires, managing the formatting and logic can take hours away from actual analysis. Tools like Doc2Form can help by instantly turning your drafted Word documents or plain-text briefs into fully formatted Google Forms. Ultimately, the best way to get good data is to write clear, engaging questions that participants actually want to answer, relying on traps only as a quiet safety net in the background.