Therefore, the goals of an investigator conducting a valid trial must be to:

• limit the impact of random variation, in particular by ensuring that the trial has sufficient sample size
• eliminate (to the greatest extent possible), all other sources of uncertainty.

If any systematic sources of uncertainty remain, the validity of the trial is diminished, and whether the trial is salvageable will depend on a subjective assessment of the potential impact of the systematic error.

Given that rescuing an invalid trial requires expert judgment, we propose an alternative statement of the goal of the trial:

to establish a treatment’s effectiveness to the satisfaction of a skeptic who is not prepared to believe the treatment is effective without convincing evidence.

We note that this goal explicitly puts the burden on the investigator/sponsor of the trial to conduct a trial that will be convincing, not just to themselves, but to the broader scientific community.

In our experience, all too often, study sponsors choose to design, conduct and analyze trials in a way that they believe will maximize the chance that they will “win.”

Rather, study sponsors have an ethical obligation to

• patients in the trial
• future patients
• the scientific community

to conduct a trial according to the highest possible scientific standards.

In my class I show students a list of mythical creatures:

• Unicorns
• Bigfoot
• Loch Ness Monster
• Yeti (Abominable Snowman)
• $\beta$ (Treatment Effect)

Why is $\beta$ a mythical creature?

• It is unclear that there is a coherent definition of “treatment effect.”
• Many of the misconceptions people have regarding trial conduct and analysis stem from an overemphasis on estimation.
• Trials are designed around hypothesis tests with specified power. There is usually no mention of estimation in the sample size calculation.
• It’s difficult to conceive of a universe in which the effect of treatment can be fully captured by a single number.

… and the final step:

If, to the desired level of statistical certainty, on average the outcomes in the experimental treatment group are superior to those in the control group, then we conclude that the experimental treatment is superior to the control treatment.

This step requires a profound leap of logic!

This is because …

Statistical comparisons are simply computations involving observed data from which we draw conclusions about the distributions of the underlying data.

However, claims regarding the effectiveness of treatment are inherently about causal relationships in the physical universe.

Statistical analysis and causation are independent, unrelated concepts.

It is by the magic of randomization that we are able to make this leap.

Now we add arrows.

The arrows represent causal pathways by which treatment with a statin might provide benefit.

Statistics can’t tell us anything about the direction of the causal pathways, or even that they’re causal at all!

Maybe:

• Someone with high LDL has an MI.
• Post MI they’re taken off statins, and subsequently die from the MI.

Unless we know, for example, the temporal relationship (non-statistical!) between these features, we may not be able to rule this out.

Maybe there is some unknown fifth factor that simultaneously changes all four of these and induces the observed association.

Maybe:

• Death reaches backwards in time, causing an MI and elevated LDL.

• The MI and elevated LDL reach backwards in time and cause the subject to not take a statin.

We likely reject this because we believe that causes can only work forward in time, not because of statistics!

Fun fact: There is no statistical evidence showing causes only work forward in time.

In every one of these scenarios (including the “reverse time” scenario), the statistical associations might be exactly the same! We need other, non-statistical, criteria to distinguish between them.

To the extent that we believe this version of the figure, we do so by incorporating non-statistical evidence, both direct and indirect.

For example, we believe, based on RCTs, that Statins lower LDL and risk of MI and Death.

We have statistical associations between LDL and MI/Death, and coupled with biological evidence, conclude that the association is causal.

Every computation that we perform occurs in “Statistics Land” and therefore statistical computation cannot bridge the gap between these worlds. We require another way across.

Our skeptic is acting as the troll preventing anyone from crossing unless their study design and conduct, combined with proper statistical analysis, allow for meaningful inference about how the world works.

• Primary outcome observed on less than half of randomized subjects.
• Primary analysis uses last-observation-carried-forward (LOCF) for subjects not completing study.
• The plotted trajectories are based on least squares means from an ANCOVA model, a hard-to-understand “black box”.
• Because we don’t observe the 6-week outcome on all randomized subjects, our “skeptic” has no way to know whether the active treatments are really better than placebo or whether the observed differences are due to lack of adherence and follow-up.

With a modest degree of missingness, the extent of misalignment may not be too large.

In this example, the extent of missingness compromises the result of the trial.

The first step uses statistical models to deduce correlations between risk factors and outcomes of interest.

If we can assume that the data at hand are a random sample from the overall population of interest, we can safely cross from Data Land into Correlation Land.

On the other hand, randomization coupled with proper study conduct and analyses is a bridge that gets us directly from Statistics Land to Causation Land.

Anything short of this requires our skeptic to trust in some unverifiable assumption made by the investigator/analyst.

We collect information at baseline:

• age
• smoking status (current, former, never)
• height
• weight
• (baseline) FEV₁%
• disease duration

The y-axis represents the expected response.

Note that the x-axis represents subject status at study start, and is (at least in part) unobservable. The y-axis represents the measured outcome at the study conclusion.

Because populations are heterogeneous, an arbitrarily chosen individual will have different baseline characteristics, different prognosis, and different outcome.

To try to characterize the outcomes across the population, we might take a sample and calculate its mean. This will vary from sample to sample.

The variability of the mean is dependent on the sample size, of course.

However, because the population is heterogeneous, both the magnitude and variability of the sample mean depend on the kinds of subjects we sample.

We can imagine two possible outcome curves, one representing group “A”, and the other representing group “B”.

As before, we can take samples from each group and calculate the sample means.

On what basis should we compare them to see if they are different?

Before collecting any data, the null hypothesis is assumed to be true and the statistical goal is to show that the null hypothesis is false.

A common approach is to reject the null hypothesis if the difference between the group means is large compared to their standard errors.

We note that were we to sample different kinds of subjects the observed responses differ in several ways, which can affect our hypothesis test.

What if we sample from a more homogeneous, “healthier” subpopulation? The responses in the two groups are larger on average and less dispersed.

Also, because we’ve sampled from a subpopulation that has a smaller average group difference, the observed difference is smaller.

Instead, what if we sample from a more homogeneous, “sicker” subpopulation. The responses in the two groups are now smaller on average but, as above, are less dispersed.

Also, because we’ve sampled from a subpopulation that has a larger average group difference, the observed difference is larger.

In particular, suppose our samples are chosen in a group-dependent way (healthy “A” subjects and sick “B” subjects).

This is confounding — a baseline characteristic that is associated with both treatment and outcome.

The result of the (extreme) confounding in this example is a reversal of the effect, commonly known as “Simpson’s paradox”.

In our context, we say that “X causes Y” if Y would have been different if X had been different. Causation is, at it’s heart, about the question:

What would have happened if …

In our heterogeneous population, if we were omniscient, we would see both outcomes of every subject and could identify the causal effect of treatment for every individual.

Perfect twins are more than identical. They are perfectly identical. Not only are all their baseline characteristics, observed and unobserved, exactly the same …

Even their hypothetical outcomes (“what would have happened if …”) match precisely.

With such a population of twins, we could assign one twin to group A (on the left) and the other twin to group B (on the right).

Note that the observed outcome difference between twins (93 − 79 = 14) is the causal effect (in either twin).

And we could do this for everyone in our trial, creating a sample made up of pairs of perfect twins.

By definition, the average causal effect is the average difference in outcome between twins.

Mathematically, this is equal to the difference between average outcomes of the two groups (because the difference of averages is the average of differences).

In a population of twins, this makes a comparison of group means just as good as being omniscient.

This is largely a true statement (although it’s not entirely clear what “comparable groups” means).

However, it makes no explicit reference to causation.

Our skeptic would argue that the three features above are side effects of randomization and are neither necessary nor sufficient to ensure that we can make valid causal claims.

… or (again, only at a population level) this is the same as studying a population of perfect twins.

This works because randomization takes random subsamples of the “A” and “B” twins, and random subsamples have the same distribution as the originals.

The outcomes may depend on the covariates:

• sicker subjects might have worse outcomes on either treatment
• subjects with certain covariates might have larger causal treatment effect

But if we are omniscient, the baseline covariates are irrelevant to ascertaining the causal effect of treatment in each participant.

Baseline covariates may help improve the efficiency of our inference.

If so, we may need fewer subjects to draw causual conclusions about the underlying population.

But they do not affect the validity of the inference.

If we aren’t omniscient, we can’t open both doors at once, but we can randomize.

This allows us to (randomly) open Door A for half the subjects and Door B for the other half.

Statistical inference we perform on the random doors we open is equivalent, at the population level, to statistical inference performed on all the doors.

It is valid statistical inference on the causal effect of treatment.

Baseline characteristics come along for the ride. They are not relevant to the validity of the inference.

A random imbalance in one or more baseline covariates can’t change this.

That’s fortunate! Otherwise, the validity of clinical trials would depend on the absence of imbalances in covariates we forgot to measure or couldn’t afford to measure or that hadn’t yet been discovered.

“The difficulty [with randomization] arises from the limited degree to which traditional methods can assure comparability of treatment and control groups.” Taves, 1972

“… imbalance between groups in baseline variables that may influence outcome … can bias statistical tests, a property sometimes referred to as chance bias. Observed differences in outcome between groups in a particular trial could by chance be due to characteristics of the patients, not treatments …” Roberts and Torgerson, 1999

There’s no doubt that:

In comparison to balancing strategies, randomization causes chance imbalances in baseline covariates.

But this isn’t the question of interest.

The question of interest is:

In comparison to balancing strategies, does randomization cause biased or invalid statistical analysis?

Specifically, does randomization cause detection of spurious treatment differences that would not have been observed if only we had employed a balancing strategy?

Note that this is a causal question, and we can evaluate it in our causal framework.

Consider a simple example based on sex (M/F) and smoking status (S/NS). The first patient is randomized. Subsequent patients are assigned to the treatment that minimizes imbalance or randomized in case of tie.

A spurious result is a trial where we detect a statistically significant treatment difference that is not due to treatment.

To us, this only makes sense in a case where the treatment is ineffective.

Some people might argue that, even if the treatment is effective, a trial result might still be spurious if a finding of effectiveness is because of a baseline covariate imbalance instead of because of the treatment.

This strikes us as pretty stupid meaningless, so we’ll stick with trials where the treatment has no effect.

Let’s imagine a clinical trial of this ineffective treatment.

Each subject has the same outcome behind Door A and Door B.

Because we’re omniscient, we can figure out what would have happened if we randomized subjects …

Here, the treatment group difference was 6.4 (in favor of B), p = 0.30.

… and what would have happened if we used minimization.

Here, the treatment group difference was 4.4 (in favor of A), p = 0.48.

This allows us to directly ascertain the causal effect of minimization versus randomization on trial results.

For this particular trial, the treatment allocation method had no causal effect on the spuriousness of the results.

We don’t have data on a large COPD trial.

But we do have data on a randomized clinical trial of 15,000 patients (7500 per treatment arm) of a CETP inhibitor, a drug that raises HDL-C (the “good” cholesterol).

We can also see the causal effect of minimization compared to randomization on the spuriousness of the trial results.

Assuming α = 0.05, there was no causal effect on spuriousness for this particular trial, though it does look as if minimization is improving the “accuracy” of the estimate.

Is this typical?

When balance advocates claim that minimization improves validity of trials, it pays to remember that their “treatment” for invalid trials is completely ineffective 91.5% of the time.

For this trial, here is the causal effect of randomization on trial results:

It looks like randomization caused the spurious result by slightly increasing the mean in group A and decreasing the mean in group B.

Here’s is the dramatic baseline covariate imbalance that randomization caused and minimization fixed.

For this trial, here is the causal effect of minimization on trial results:

Minimization caused the difference to be vastly overstated. How did we go so wrong?

Obviously, minimization caused this invalid result by eliminating crucial imbalances in covariates.

A more serious way of looking at it: even in trials of completely ineffective treatments, balance in covariates does not ensure balance in outcomes.

Just because minimization causes covariate balance, that does not mean it will cause outcome balance.

In fact, in the course of balancing covariates, it may — literally — cause outcome imbalance!

Of course, if the trial lacks power to detect a clinically significant difference, or if the actual treatment effect is too small, the trial may fail.

We will consider adequately powered trials (at least 90% power) for a highly effective treatment.

Once again, we want to simulate a large number of trials as if we were omniscient, but we’d like to do this with real data.

How do we proceed?

Here’s an example where minimization did a great job fixing a big age imbalance (mean 55 vs. 63 years) in this 40-person trial, one that could have caused a missed opportunity.

… except it didn’t cause anything of the kind. This trial was going to be successful, regardless of allocation strategy.

Again, there was no causal effect of minimization versus randomization on missed opportunities for this trial.

As before, in the vast majority of cases (99.2%), there is no causal effect of minimization versus randomization on missed opportunities in detecting a successful treatment.

But maybe the trial that inspired Taves’ came from that pesky 0.8%. (To be fair, this fraction would be slightly larger if the trial had been less well powered.)

For this trial, here is the causal effect of randomization on trial results:

Here’s the baseline imbalance that minimization fixed to rescue this trial.

For this trial, here is the causal effect of minimization on trial results:

Minimization did another great job balancing covariates. Too bad it wrecked this trial.

… and prevented only 0.1% more missed opportunities than it caused …

… while having absolutely no effect (useful or otherwise) in 91.5% of trials of ineffective treatments and 99.2% of trials of highly effective treatments.

Still, the fact remains that, in our simulation, the causal effect of minimization was to:

• reduce Type 1 error from about 0.05 to 0.04
• reduce Type 2 error from about 0.005 to 0.004

So it is clearly superior to randomization, right?

Normally, when we use statistical techniques to improve efficiency:

• we increase our power (i.e., decrease our Type 2 error)
• while maintaining the same level of Type 1 error (equal to our choice of nominal α=0.05)

Because minimization is a relatively inefficient way of incorporating baseline covariate information to improve efficiency:

• we increase our power (i.e., decrease our Type 2 error)
• while also decreasing our Type 1 error below our desired level of α=0.05, allowing our tests to become too conservative

To realize the full efficiency gain of using baseline covariate information, we would want to maintain our preselected level of Type 1 error and increase our power even more!

But, I have a confession to make.

In this particular simulation, minimization does a pretty good job of making use of available baseline covariate information.

Because we are considering …

• small trials (n=40)
• where a relatively small subgroup (females, 22%)
• exhibits a big systematic difference in outcome (17% higher than males)

… modelling can’t take full advantage of baseline covariate information in those randomized trials where there are too few females in one group or the other (e.g., where all the females are in group A).

So, minimization leads to a slight efficiency gain. (The gain is about half the gain realized by increasing the sample size from n=40 to n=42.)

However, please note:

• This is not, as balance advocates claim, because the resulting baseline imbalance invalidates the trial (even when all the females end up in group A).
• This is because there are too few females in one group to serve as a proper comparison group for the females in the other group.
• In these cases, the modelling analysis can’t do any better than an analysis that ignores baseline covariates, so opportunities get missed.

The assumption is frequently made that observations are missing at random (we’re using this term informally and won’t define it).

If this assumption is correct, the only consequence of missing data is that we have a smaller sample size and, therefore, lower power and precision. Otherwise, the analysis of the available data will give the same results as if there were no missing data.

Sometimes, it’s acknowledged that observations are not missing at random, but the assumption is made that the missingness is not influenced by treatment. (Again, we won’t define exactly what we mean here.)

If this assumption is correct, it can change the results of the analysis, but at least the analysis still has a causal interpretation.

The problem is that, by their very nature, missing data are unobserved and it is impossible to know whether they are actually missing at random.

Non-random, treatment-dependent missingness can easily subvert the benefits of randomization.

Straightforward analysis of such data no longer has a causal interpretation.

Now suppose there are some pairs of twins with a missing outcome.

If we include only complete pairs, we will still have a valid causal analysis.

Having missing pairs in a population of twins is analogous to “missingness is not influenced by treatment” in a randomized, twin-less population.

This is because, even if missingness depends on the attributes of an individual, if twins always go missing in pairs, then treatment (the only difference between twins) is not influencing missingness.

Again, if we include only complete pairs in our analysis, we will still have a valid causal analysis

However, this requires that we are able to identify “twins”, specifically the twin with complete data that corresponds to the one with missing data, so the former can be removed from the analysis.

If we really were studying a population of perfect twins, this would be easy.

But we are using randomization to approximate a population of twins. If an individual in group A has missing data, there is no perfect twin in group B to remove!

We have no reason to believe that the unmatched group A twins are comparable to the unmatched group B twins. We might hope that their differences will “cancel out” to give a causal effect, but this requires untestable assumptions.

If the data are not missing at random but missingness isn’t influenced by treatment, the analysis changes, but we are still making a causal comparison.

If neither assumption holds, as we’d expect in most real-world trials, then all bets are off.

Even under the scentific null hypothesis (where there is no causal effect of treatment by definition), missingness can create a statistically significant difference.

Note that the statistics is working fine: there is a difference between these groups.

But the randomization has been subverted. The difference can’t be a causal effect of treatment.

Overall, vital status is missing for about 10% of subjects.

Using the complete data, the Pearson chi-square Z statistic is 2.80, p = 0.0051, favoring group B.

What is the impact of the missing data? For this particular table, the answer requires sensitivity analysis, which we’ll discuss later.

For now, let’s just consider the impact of missing data more generally.

Imputation Methods

• Last observation carried forward (LOCF)
• Single imputation
• Multiple imputation

Assumes that either:

• The missing outcome is EQUAL to the last observed value (which is unverifiable and unrealistic)
• The outcome of interest is actually “your final value before you drop out” (which is scientifically absurd)

• Contrary to some claims, LOCF is not guaranteed to be conservative.
• For example, even when the “treatment effect” is conservatively estimated, the variability can be underestimated.
• It is also very easy to construct examples where the LOCF estimate is anti-conservative.
• LOCF has no causal basis.

Single imputation using a simple mean involves:

• replacing the missing value by the average of all values in the treatment group
• assuming random missingness

The resulting estimate is very similar to complete-case analysis, but based on an artificial, falsely increased sample size that underestimates variability. We consider this approach worse than complete-case.

Single imputation using regression or another modelling approach involves:

• replacing the missing value by a prediction (plus random error) from a model based on complete subjects
• assuming we have accounted for all of the “important” variables
• assuming our model is correct

The resulting estimate might be better than complete-case analysis, but is still based on an artificial, falsely increased sample size that underestimates variability.

The fact remains:

No form of single imputation has any direct causal basis.

Multiple imputation is:

• much better than single imputation with respect to assessing variability in the estimates
• still requires the assumptions that we have all of the “important” variables and that the model is correct (though it can be fairly robust against deviations from the model)

The fact remains:

No form of multiple imputation has any direct causal basis.

In other words, subjects with complete outcome data who were more likely to be missing get counted more in the analysis to compensate for similar subjects with missing outcome.

If we consider groups of “similar” subjects by treatment, inverse probability methods assign bigger weight to subjects with complete data in those groups where more subjects have missing data.

But if we go back to the twin population, the method still includes unmatched twins, and even includes complete pairs with different weights.

• Requires model assumptions
• We cannot claim the results provide a valid causal analysis.

Overall, vital status is missing for about 10% of subjects.

Using the complete data, the Pearson chi-square Z statistic is 2.80, p = 0.0051, favoring group B.

Under the “best” case (most favorable for treatment B), we assume:

• all 10 A subjects are dead
• all 9 B subjects are alive

In this case, the Z-score is 4.07, p=0.00005.

Under the “worst” case (least favorable for treatment B), we assume the opposite:

• all 10 A subjects are alive
• all 9 B subjects are dead

In this case, the Z-score is 1.25, p=0.21.

• Complete case analysis is correct if data are missing at random, which is probably unrealistic.
• Under “worst” case, the conclusion changes.
• “Best” and “worst” cases are probably too extreme.
• A better approach allows deviations from missing at random to be quantified

Different values for $R_A$ and $R_B$ cover all the scenarios we have considered so far:

• $R_{A}=R_{B}=1$, data are missing at random.
• $R_{A} \rightarrow \infty$, $R_{B} \rightarrow 0$, is “best case”
• $R_{A} \rightarrow 0$, $R_{B} \rightarrow \infty$, is “worst case”

If complete case analysis holds up for plausible values of $R_{A}$ and $R_{B}$, then the result is robust.

Of course, “plausible” is highly subjective.

With 1000 per group, and 10% missingness, the table looks like this:

With 10,000 per group, and 10% missingness …

With 15% missingness, and 100 per group …

With 1000 per group …

With 10,000 per group …

The Panel on Handling Missing Data in Clinical Trials …

“… encourages increased use of [modern statistical analysis tools]. However, all of these methods ultimately rely on untestable assumptions concerning the factors leading to the missing values and how they relate to the study outcomes …

… There is no ‘foolproof’ way to analyze data subject to substantial amounts of missing data.” The Prevention and Treatment of Missing Data in Clinical Trials

Based on the examples presented earlier, “substantial” amounts of missing data might be 10 or 15%.

In summary:

• Without complete follow-up on all types of data (not just the “primary endpoint”), the results cannot be interpreted causally without unverifiable assumptions.
• Many of these assumptions are unrealistic, and there is usually insufficient testing of deviations from these assumptions through sensitivity analysis.
• All this applies to the importance of complete follow-up after withdrawal from treatment.^†

^†More on this next!

Subjects may be non-adherent for lots of reasons:

• They took one dose, didn’t like it or experienced a side effect, and stopped their study medication.
• They took their assigned dose for part of the intended follow-up period, but their disease worsened and they stopped their study medication.
• Study procedures became too onerous and they stopped their study medication.

Most of these reasons will probably result in adherence that depends on the assigned treatment.

Many investigators view non-adherence as subverting the purpose of the trial:

• The goal of the trial is to “estimate the treatment effect” on the outcome in question.
• Subjects who do not receive the treatment as intended do not benefit from the treatment
• Non-adherence prevents us from correctly “estimating the treatment effect”

This objection is certainly valid.

The “correct” experiment is the one in which we strictly enforce full adherence.

At least in human trials, this isn’t possible.

If we want to recover the “full-adherence treatment effect” from a trial with incomplete adherence …

… we somehow need to recover the full-adherence outcomes in subjects with partial or no adherence.

Unless we’re omniscient, we can’t know what the full-adherence outcomes for these people would have been.

Without somehow correctly guessing them, it is impossible to estimate the “full-adherence treatment effect.”

One common proposed solution to this problem is to exclude non-adherent subjects and include in the analysis only those subjects who are fully adherent (or nearly so).

This is the so-called “per-protocol” analysis set.

What’s the problem with the “per-protocol” analysis?

By excluding non-adherers, we have (deliberately!) induced missing data! In terms of our twin model, we have excluded some non-adhering twins, leaving some unmatched twins.

We have already seen that when data are missing like this, we can’t know if the differences between groups are causally related to treatment, or induced by the missing data.

The “per-protocol” analysis can’t possibly recover the full-adherence treatment effect!

• We know that the best basis for assessing the causal effect of treatment is inference based on randomization.
• We’ve done a randomized experiment.
• If we have complete data, we can draw valid causal conclusions from our trial.

• Unless we have full adherence, our causal conclusions can’t be conclusions about the effect of full adherence to treatment.
• We can, however, reach valid conclusions about the effect of assignment to treatment.
• After all, in randomized trials in humans, we can randomly assign people to treatments, but we can’t force them to receive treatments.

Note that we don’t know that full-adherence is even possible.

In some cases, it may be impossible for subjects to receive treatment under any circumstances. (For example, imagine trying to administer an exercise intervention to a subject in a coma.)

Hence, in some cases, “full-adherence” hypotheses may not have any meaning.

Even if hypothesis 3 makes sense, because we cannot observe full-adherence responses for some subjects, there is no objective way to ensure alignment of hypotheses 2 and 3.

These hypotheses may be “misaligned” …

… in either direction.

… which is precisely what our skeptic wanted to see.

“When unbiased, intelligent people consider ITT, they cannot understand how it can be used by scientists trying to make sense out of their data, but, unfortunately, it is in almost every experiment.”

“Why, you may ask, could seemingly intelligent people do something so stupid as use ITT to evaluate data?”

While this may be true (although it is unclear how close the conditions in a trial mimic those of the real world), the real reason we use ITT is this:

ITT is the only analytical approach that relies solely on randomization to infer the effectiveness of treatment.

To reiterate, we do not use ITT because the ITT estimate is more realistic or more clinically relevant or more useful to calculate.

We use ITT because it is the only assumption-free, causal analysis we can do with the experiments we can perform. It is the only analysis that can convince our skeptic.

Some might argue that, since adherence was about the same in both groups, we can use per-protocol or as-treated analysis.

However, even if rates of non-adherence are equal, we cannot assume (hope) that non-adherence is independent of treatment.

Here is a summary of survival in the clofibrate group by adherence to treatment.

There is a large, highly statistically significant difference in mortality between people taking at least 80% of their assigned medication and those taking less than 80%.

This might suggest that if you take your medication, you will receive a mortality benefit, relative to those who only take part of their medication.

Here we see a difference at least as large between those taking at least 80% of their placebo and those taking less than 80%.

• placebo is highly effective at improving survival, or
• subjects adhering to placebo have better prognoses than those who fail to adhere.

Of course, it’s most likely that subjects discontinue their placebo because they’re doing poorly. That is, subjects who adhere are different than subjects who don’t.

It is likely that the difference within the clofibrate group is driven by a similar phenomenon and is unrelated to any benefit of treatment.

We can see how this works in our hypothetical COPD trial.

However, we can adhere to the spirit of ITT by reviewing every subject for a particular set of eligibility criteria in a systematic way:

• This should be pre-specified, including (a) which criteria and (b) the review process
• This must be done in a blinded way (which is often harder than it sounds)

A proper definition of MITT is:

An MITT analysis includes every subject who received at least one dose of study drug, or who at least “started” treatment.

Consider a subject in a double-blind trial who is assigned to A but does not receive any treatment. If the blind is very secure then had that same subject been assigned to B, she also would not have received any treatment.

In either case, she would not have received treatment, so in either case, she would have had the same outcome (i.e., whatever outcome would result from receiving no treatment).

In a sense, initiating treatment becomes part of the randomization process (or even the eligibility process), and you can think of the time of first treatment as the randomization time.

If this were a trial of perfect twins, either both twins would initiate treatment, or both twins would never start treatment.

The MITT analysis excludes twins who never start treatment, leaving only complete pairs who took at least one dose, a valid causal analysis.

If we make the assumption that the mistake was “completely random”, or at least independent of the actual treatment assignment, then switching the group to what they actually received gives a valid analysis.

In many cases, this assumption is almost certainly true. However, it is still an additional assumption.

Also, if these errors are rare, there will be little impact on the analysis.

If the errors are common, you have conducted your trial poorly and will have difficulty convincing the skeptic, regardless of how you analyze.

Recommendation: safest route is to analyze based on the treatment that was assigned by randomization. Replace the “kit” that was mistakenly used so it will be there when that randomization number comes up.

The best approach is to prevent the errors in the first place.

If they do occur, always make note of the errors so the skeptic has all the info he/she wants.

Do NOT just throw out those subjects in the intervention arm.

You have two options:

• Leave everybody in the analysis.
• Throw out all subjects who could have been randomized to the bad batch of study drug.

The second option is OK, as long as blinding (especially of results) is secure.

But why stratify randomization?

Conclusion: there are no compelling reasons to stratify randomization and nothing that could affect the causal bridge.

In order to determine whether a treatment has a causal effect on an outcome, we must either:

Analytical methods for dealing with missing data (including complete-case analysis) do not adequately restore the pure causal relationship.

No analysis can give a valid answer to the question

“If you take your medicine, does it work?”

• If, by design, our data collection ends when treatment ends (e.g., stop routine visits, stop laboratory tests, track adverse events for only 30 days following discontinuation), we intentionally make key data unavailable for determining the causal effect of assignment to treatment.
• This design decision will sabotage our trial!

If our trial results are to be interpretable, we must understand:

There must be a process in place for withdrawing from treatment without withdrawing from data collection.

Withdrawal from data collection should be extremely rare.

We must follow and collect data on all subjects, regardless of adherence to treatment.

3. Therefore, if we have partial adherence, the estimated treatment effect would be somwhere between the “true” treatment effect and no treatment effect.

Obvious, right?

Unfortunately, while #1 and #2 are plausible, there is no reason to believe #3.

The final manuscript presented several analyses. The “as randomized” analysis (top) excluded 6 untreated subjects, though it gave similar results to an ITT analysis.

Here are the results in tabular form:

The p-value for PP versus Violators within the Polyheme group is 0.012. The “violators” in the Polyheme group were likely higher risk subjects than the adherers.

No other comparisons reach “statistical significance.”

Remember …

• The premise for using PP is that violators look more like controls, shrinking the ITT difference.
• There is no evidence of that here!