Blog

· By Thesisroom

Pilot Study in Research: What It Tests and How to Run One for a Thesis

What a pilot study tests in a thesis, how it differs from a feasibility study, whether pilot data can be reused, eight steps, an invented example and a printable log.

This guide has a printable version: open the printable pilot study plan and log

A hand holding a silver pen fills in a paper form clipped to a black clipboard
Photo by Zulfugar Karimov on Unsplash

A pilot study is a small trial run of a planned study, or a pre-test of one instrument such as a questionnaire, done before the main study (van Teijlingen and Hundley). Its job is to show whether your methods work (instruments, recruitment, procedures and timing), not whether your hypothesis is true (Leon, Davis and Kraemer).

For a Master’s or PhD thesis, that purpose shapes what you write in the proposal, what you ask the ethics committee for, and what you may do with the pilot data afterward. Below are the definitions, eight steps, an invented worked example and a pilot plan and log you can print.

What is a pilot study in research?

A pilot study is a smaller version of your study, run first to find out whether the plan works. Van Teijlingen and Hundley note that the term is used in two ways: a mini version of the full study (a feasibility study), and the pre-testing of one particular instrument (Social Research Update 35). Both count in a thesis.

The same authors say a pilot can warn you where the main project could fail, where the protocol may not be followed, and whether a method or instrument is unsuitable or too complicated. They are also clear that a pilot does not guarantee success; it raises the odds.

Several of the papers cited here were written about clinical trials. Leon, Davis and Kraemer illustrate with a trial, but state that the feasibility purpose “applies to all types of research studies” (Journal of Psychiatric Research). The logic carries over to a survey, an interview study or a lab experiment.

Is a pilot study the same as a feasibility study?

Not quite. In the framework behind the CONSORT extension for pilot and feasibility trials, every pilot study is a feasibility study, but not every feasibility study is a pilot (Eldridge et al., BMJ).

That framework says a feasibility study asks whether the future study can be done, should be done and, if so, how. A pilot study asks the same questions, with one design feature: the future study, or part of it, is run on a smaller scale. The authors set this out in a separate paper on defining the two terms (Eldridge et al., PLOS One).

Other sources draw the line differently. Thabane and colleagues treat a pilot study as synonymous with a feasibility study (BMC Medical Research Methodology).

Eldridge et al. also report (PLOS One) that the UK National Institute for Health Research developed definitions that kept the two separate. Its later wording, as they quote it, defines a pilot as “a smaller version of the main study used to test whether the components of the main study can all work together”.

Feasibility studyPilot study
What it asksCan the main study be done, should it be done, and howThe same questions
DesignAny work done to judge feasibility before a main studyThe main study, or part of it, run on a smaller scale
Thesis example (ours)Asking schools whether they keep the records you plan to analyzeRunning your questionnaire and recruitment route with a few people from your target group

Because the terms vary by author and funder, define the one you use in your proposal. One sentence and a citation is enough.

What does a pilot study test?

A pilot study tests any part of your method that could fail in the main study: the instrument, recruitment, procedures, time, data handling and the analysis plan. Thabane and colleagues group the reasons for piloting into four kinds: process, resources, management and scientific (A tutorial on pilot studies).

What you testWhat to look forSource
Questionnaire or interview wordingAmbiguous or hard questions, items left blank, too narrow a range of answers, question ordervan Teijlingen and Hundley, Tables 1 and 2
Recruitment and retentionHow many people you can reach, how many agree and how many stayThabane et al. (process)
ProceduresWhether people follow the protocol as writtenvan Teijlingen and Hundley
Time and resourcesHow long each form, interview or session takesThabane et al. (resources)
Data handlingPersonnel and data management problemsThabane et al. (management)
Analysis planWhether the planned analysis techniques run into problemsvan Teijlingen and Hundley, Table 1

The procedures row is easy to underestimate. In a Scottish maternity care study that van Teijlingen and Hundley describe, the pilot showed the planned way of handing out questionnaires would not be followed. The person responsible chose another route without asking the research team.

Thabane and colleagues also report what goes wrong in pilot designs. Reviewing submissions to a research ethics board, they found most pilot projects had no clear feasibility objectives, no clear analysis plan and no criteria for success. Each of those gaps is a step below.

Can a pilot study test your hypothesis or give you an effect size?

No to the first, and only with caution for anything that feeds a sample size. Leon, Davis and Kraemer state it plainly: “A pilot study is not a hypothesis testing study.” They add that no inferential statistical tests should be proposed in a pilot protocol (Journal of Psychiatric Research).

The CONSORT extension agrees for trials: formal hypothesis testing for effectiveness is not recommended, because a pilot will usually be underpowered (Eldridge et al., BMJ). Your research hypothesis waits for the main study.

On effect sizes the sources differ in tone, so be precise about what you take from a pilot:

  • Between-group effect size: Leon, Davis and Kraemer say pilot results should not be used for sample size determination, because effect sizes from small samples are too imprecise. They suggest basing the main sample on a clinically meaningful difference instead.
  • Effect estimates in general: Thabane and colleagues warn that pilot estimates of treatment effects can mislead power calculations.
  • Variability and rates: van Teijlingen and Hundley list estimating variability in outcomes as a reason to pilot. Thabane and colleagues suggest a sample size table across a range of values, to show the uncertainty.

In thesis terms: use the pilot to check rates and spread, and do not power your main study on its effect size. Van Teijlingen and Hundley add that a pilot’s response rate is a hint, not a guarantee. If an examiner asks how you chose your main sample, the answer lies in how you defend your sample size.

Does your thesis need a pilot study?

A pilot earns its place when part of your method is new, adapted or untried with your population. It is worth proposing when any of these is true:

  • You wrote your own questionnaire or interview schedule, or changed an existing one.
  • You will recruit through a route you have not used before, such as a gatekeeper or a mailing list.
  • Your procedure depends on other people doing their part as written.
  • You are new to interviewing. Van Teijlingen and Hundley cite Holloway’s view that qualitative piloting helps when the researcher “lacks confidence or is a novice”.

A supervisor may also say no, and the sources give reasons. Holloway argued that separate pilots are not necessary in qualitative approaches, because early interviews already improve the later ones. Van Teijlingen and Hundley also note that leaving pilot participants out can make the main sample too small, as with clusters such as schools or hospitals.

The weakest case is calling a small study a pilot because a larger one is out of reach. Thabane and colleagues reject lack of resources as a sound reason to label a study a pilot. They add that anyone given a project as a pilot should discuss how its results will inform the planning of the main study.

A middle option is the second sense of the term. Pre-test the instrument alone with a few people, without rehearsing the whole study.

How do you run a pilot study? Eight steps

Run a pilot as a small study with its own objectives, criteria and record. These eight steps follow the sources above.

  1. Fix the main study’s questions and plan first. A pilot tests a plan, so the plan must exist, and every pilot objective should trace back to a research question in your proposal. Scope Check reads a research proposal and asks where what it asks for and what it plans do not obviously meet; it does not say whether a project can be done.
  2. Write one feasibility objective per thing you test. For example: “Can I recruit through head teachers?” Thabane and colleagues name unclear feasibility objectives as a common flaw.
  3. Set your criteria for success before you start. Thabane and colleagues give four outcomes: stop; continue but modify the protocol; continue unchanged but monitor closely; continue unchanged. The CONSORT extension says the decision should be built into the design, for example as progression criteria. Write a threshold for each objective before you see any data.
  4. Check your university’s ethics rule for pilots. St Mary’s University, Twickenham requires approval before any pilot with human participants, while Boise State and Caltech treat some pilots as outside research review. See the ethics section below.
  5. Choose participants like your main sample. Thabane and colleagues say the pilot sample should be representative and use the same inclusion and exclusion criteria. The CONSORT extension says the number should follow from the feasibility objectives, with a rationale. Decide now whether pilot participants may join the main study.
  6. Run it exactly as the main study will run. Van Teijlingen and Hundley, citing Peat et al., say to give the pilot questionnaire exactly as it will be given in the main study.
  7. Collect feasibility data, not findings. Time each session, ask participants which questions were unclear, note unanswered items and narrow answer ranges, and check that replies give the information you need. Run your analysis plan on the pilot data to see that it works, not to report results.
  8. Decide, change and record. Apply your criteria, make the changes, and note each change with its reason. Van Teijlingen and Hundley’s last questionnaire step is to shorten, revise and, if possible, pilot again.

A worked example: piloting a survey and interviews (invented)

This example is invented to show the steps. A Master’s student in education plans to study how secondary school teachers use written feedback. The main study is an online questionnaire for teachers at four schools, then interviews with eight volunteers.

The pilot. Five teachers at a fifth school, outside the main sample, complete the questionnaire. One of them also does a practice interview. The ethics application states that pilot data will be used only to refine the methods.

Feasibility objectives and criteria, set in advance:

  • Can teachers finish the questionnaire in the fifteen minutes the invitation promises? Criterion: most of the five finish within that time.
  • Is every item clear? Criterion: no item is read two different ways.
  • Does recruitment through the head teacher work? Criterion: replies arrive within a week.
  • Does the interview fit in the hour booked? Criterion: all questions are covered.

What the pilot found and what changed:

Pilot findingChange madeReason
Three of the five teachers took over twenty minutesTwo overlapping items cut; invitation now says twenty minutesThe promised time was wrong
Two teachers read “formative feedback” differentlyOne-line definition added above the itemAmbiguous term
All five chose “agree” on one itemItem reworded and flagged for the supervisorPossible narrow range; five people is a weak signal, not proof
Head teacher route: one reply after a weekInvitations also sent through heads of department; ethics amendment filedRecruitment too slow
Practice interview ran long, with repeated answersTwo questions merged, order changedTiming and flow

The decision. Using Thabane and colleagues’ outcomes, the student records “continue, but modify the protocol”. None of the pilot answers enter the main analysis, so the contamination question never arises.

How do you describe a pilot study in your proposal?

Give it a short subsection inside your methods: what it tests, with whom, how you will judge it and what happens to the data. In a standard PhD research proposal format, that means the methodology section. Place it next to the sampling and data collection plans in the methodology section of your proposal.

Thabane and colleagues suggest reporting items for pilot studies that work well as a proposal checklist:

  • Rationale: why this part of the method needs piloting.
  • Feasibility objectives: one per thing you test, kept separate from the main study’s research questions.
  • Participants: the same eligibility criteria as the main study, or the differences stated.
  • Feasibility outcomes: for example recruitment, consent and completion rates.
  • Sample size: a rationale tied to the feasibility objectives.
  • Criteria for success: what result leads to which decision.
  • Ethics and consent: whether approval is needed, and how consent covers a pilot.

Here is model wording you can adapt (ours, not from a source):

Before the main study, I will pilot [the instrument and procedures] with [number] [participants] who meet the main study’s inclusion criteria but will not take part in it. The pilot will test [feasibility objectives]. I will proceed unchanged if [criteria], revise [the instrument or procedure] if [criteria], and return to the design if [criteria]. Pilot data will be used only to refine the methods.

On consent, Thabane and colleagues suggest that the consent form state that the study is a pilot, name its feasibility objectives and give the criteria for success.

How do you report a pilot study in the methodology chapter?

Report what the pilot found and what you changed because of it, not only that a pilot took place. Van Teijlingen and Hundley criticize papers that say only that the authors “had learned from the pilot study” without saying what.

Glasgow Caledonian University’s guide for dissertations and final year projects, from its School of Engineering and Built Environment, gives one model. Its example structure has a subsection, “Pre-test meeting or Pilot study”, for pilot work that made you review or change the procedure or instruments (Methodology for Dissertations and Final Year Projects). One of its examples: questions were removed or redrafted because they were ambiguous or imprecise.

A change table like the one in the worked example does this in a few lines. Then use the advice on writing the methodology chapter so each change reads as a justified decision.

If the pilot showed that the study could not run as planned, say so. Thabane and colleagues argue that a pilot showing the main study is not feasible is not a failed pilot; it is a success, because it avoided wasted resources.

Can you use pilot data in your main study?

Sometimes, but the sources disagree, so decide before the pilot and write the decision into your proposal and ethics application.

  • Generally no: Leon, Davis and Kraemer say pilot data generally should not be combined with the main study, because methods are likely to change after a pilot. They allow a rare exception for a pre-specified adaptive design.
  • No in much quantitative work: van Teijlingen and Hundley quote Peat et al. that pilot data are not used to test a hypothesis or included with the main results. They add that data may still be of value where a validated tool is used and the pilot tests other things, such as recruitment rates.
  • Yes, with conditions: Thabane and colleagues say pilot and main data can be combined if the sampling frame and methods are the same. Any pooling should be planned beforehand and described in the protocol.
  • Often yes in qualitative work: van Teijlingen and Hundley note that qualitative researchers often use some or all of their pilot data, because data collection improves as it goes.
  • Institution rules: Boise State University’s research compliance office says data collected from a pilot study cannot be used as research data (Definitions of Human Subjects and Research). Rules differ between universities.

A separate question is whether pilot participants can join the main study. Van Teijlingen and Hundley explain the worry: people who have already been through the procedure may respond differently from those who have not.

A safe rule for a thesis: if there is any chance you will want pilot data in your results, treat the pilot as research from the start. Get approval first, keep the methods identical and make sure consent covers that use.

Does a pilot study need ethics approval?

It depends on your university, and published rules point in different directions, so check yours before you contact a participant. Three examples show the range; rules differ between universities.

  • St Mary’s University, Twickenham: its ethics FAQ (version 3) says you must secure approval before you begin any part of your research, including a pilot study, that involves human participants. It adds that approval cannot be given retrospectively (St Mary’s University ethics FAQ).
  • Boise State University: a pilot that helps refine data collection procedures and instruments is not considered to contribute to generalizable knowledge. So it is not defined as research and does not need IRB review, and its data cannot be used as research data (Definitions of Human Subjects and Research).
  • Caltech: its IRB procedure does not classify a pilot as human subjects research, but requires an Initial Query in some cases. These include invasive procedures or device testing, physical or psychological distress, vulnerable groups such as minors, and sensitive information from Caltech students or lab members (Caltech IRB pilot studies procedure).

Read your own university’s ethics pages and ask your supervisor which category your pilot falls into. Ask before step 5. At St Mary’s, for example, approval cannot be given retrospectively.

Printable pilot study plan and log

Fill in one row per feasibility objective before the pilot, then complete the log as you go.

Pilot plan

Feasibility objectiveLinked research questionWhat I will measureCriterion for successDecision if not met

Before the pilot

  • Main study questions and plan agreed with my supervisor
  • Ethics rule for pilots checked at my university
  • Approval in place, or written confirmation that none is needed
  • Pilot participants meet the main study’s inclusion criteria
  • Decided whether pilot participants and pilot data may enter the main study
  • Consent form says this is a pilot and names its objectives

Pilot log

DateWhat I ranWho took partTime takenWhat went wrong or was unclearChange madeReason

Decision (tick one)

  • Stop: main study not feasible as planned
  • Continue, but modify the protocol
  • Continue unchanged, but monitor closely
  • Continue unchanged

Frequently asked questions

What is the difference between a pilot study and a pre-test?A pre-test tries out one instrument, such as a questionnaire, on a few people. A pilot study can be a pre-test, but it can also be a small version of the whole study, testing recruitment, procedures and analysis together. Van Teijlingen and Hundley describe both uses of the term.
How many participants does a pilot study need?No single number applies. The CONSORT extension for pilot trials says the number should follow from the feasibility objectives, with a rationale. Thabane and colleagues say a pilot should be large enough to give useful information on what it tests, and may need no formal sample size calculation.
Do qualitative studies need a pilot study?Not always. Van Teijlingen and Hundley report the argument that separate pilots are unnecessary in qualitative work, since early interviews improve later ones. They also note that piloting helps a researcher who is new to interviewing and can sharpen the study’s focus.
What if my pilot study shows the main study will not work?Report it and revise the plan with your supervisor. Thabane and colleagues argue that a pilot showing the main study is not feasible is a success, not a failure, because it prevented wasted effort. Their outcomes range from stop to continue unchanged.
Can I publish a pilot study?Thabane and colleagues say every attempt should be made to publish pilot results, with the focus on feasibility rather than statistical significance. Rules on participant data differ: Caltech’s IRB procedure says data from or about pilot participants may not be published. Check what your own approval covers.

Sources

How this guide was made: Thesisroom drafted this guide with the help of AI writing tools. Every fact, figure and source in this post was checked against its original source before publishing.