Survey Sampling Bias Audit Builder

Audit survey results for sampling bias, nonresponse bias, weighting needs, and reporting caveats before making decisions from the data.

Prompt Template

You are a survey research analyst. Audit the following survey for sampling bias and decision risk before results are presented.

Survey context:
- Research question: [what the survey is trying to answer]
- Target population: [who the findings should represent]
- Sampling method: [email list, panel, in-product popup, social media, customer database, etc.]
- Number invited and number responded: [counts]
- Response rate: [percentage if known]
- Respondent profile: [demographics, customer segments, plan tiers, regions]
- Known missing groups: [segments likely underrepresented]
- Key results: [paste headline results or table]
- Decision this survey will inform: [product, marketing, policy, pricing, etc.]
- Available benchmark data: [CRM, census, customer base, prior surveys]

Provide:
1. Sampling frame assessment
2. Nonresponse and self-selection bias risks
3. Representativeness check against known population benchmarks
4. Recommended weighting or segmentation approach, if appropriate
5. Questions/results that are safe vs risky to generalize
6. Reporting caveats written in plain English
7. Follow-up data collection plan to fill gaps
8. Executive summary explaining confidence level and decision risk

Be precise about what the data can and cannot support.

Example Output

Survey Bias Audit โ€” Pricing Research

Sampling Frame Assessment

The survey was sent only to active newsletter subscribers, so it represents engaged prospects better than the full buyer market. It likely underrepresents evaluators who visited pricing pages but never subscribed.

Representativeness Check

| Segment | Customer Base | Survey Responses | Risk |

|---|---:|---:|---|

| SMB | 48% | 68% | Overrepresented |

| Mid-market | 37% | 24% | Underrepresented |

| Enterprise | 15% | 8% | Underrepresented |

Safe to Say

"Among engaged SMB-heavy respondents, annual billing was preferred by 54%."

Risky to Say

"Most customers prefer annual billing." The sample does not represent the full customer base.

Follow-Up Plan

Run a targeted follow-up with at least 80 mid-market and 40 enterprise responses, then compare willingness-to-pay by segment before changing pricing.

Tips for Best Results

  • ๐Ÿ’กProvide both invited and responded counts; response rate changes how much confidence to place in results.
  • ๐Ÿ’กInclude the population you want to generalize to, not just who answered the survey.
  • ๐Ÿ’กAsk for plain-English caveats you can paste into a slide deck.
  • ๐Ÿ’กUse benchmark data from your CRM or user base whenever possible to spot overrepresented segments.