The Baseline Is the Experiment

I spent months building a system whose most useful result was an arithmetic mean.

The system was Argus Vision: two image classifiers looking at the same dermoscopic image, an uncertainty trigger deciding whether the case deserved deeper inspection, spatial explanations from both models, a compact feature contract, and a learned model whose job was to arbitrate when the two classifiers disagreed.

It had all the ingredients that make an ML architecture diagram feel satisfying. There were parallel model paths. There was conditional computation. There was a notion of disagreement. There was even a historical version where the models generated textual arguments before a second model decided what to believe.

Then I evaluated the thing against this:

No debate. No attention features. No learned arbitration. Add the two probability vectors, divide by two, take the argmax.

That baseline achieved a balanced accuracy of 0.6353.

The full Argus consensus achieved 0.6190.

Worse, the system I had built partly to reason about uncertainty had an Expected Calibration Error of 0.2060, compared with 0.1048 for the arithmetic ensemble.

And the supposedly selective disagreement trigger? It fired on 3,538 of 3,740 test images: 94.6% of the held-out split.

The sophisticated system had lost the classifier comparison, become worse calibrated, and routed almost everything through its expensive path.

This was not the result I wanted.

It was a much better experiment because of it.

A baseline is an epistemic instrument

I used to think of a baseline as the boring row in a results table.

You build something simple because research convention demands a comparison. Then you move on to the architecture you actually care about: the new loss, the new agent, the learned router, the clever feature set. The interesting work is supposed to begin after the baseline.

I now think this gets the relationship backwards.

A baseline is not there merely to tell you whether your final number is high. It gives you a reference point against which every extra assumption has to justify itself.

If I replace

$f_0(x)$

with a larger system

then I have not just added capability. I have added claims.

I am claiming that the new signals $z_i$ contain useful information. I am claiming that the learner $g$ can extract that information from the amount of data I have. I am claiming that the extra training procedure will generalize. I am claiming that the features will have the same semantics at training and serving time. I am claiming that the extra latency, configuration, failure modes, and calibration behavior are worth carrying.

A complex model can be better. But complexity does not arrive as free expressive power. It arrives bundled with hypotheses.

The baseline is the instrument that tests those hypotheses.

This is why the advice to start simple appears again and again in good ML engineering practice. Karpathy’s training recipe argues for introducing complexity incrementally and validating concrete hypotheses at each step. Google’s Rules of Machine Learning makes the same point from a production perspective: get a simple end-to-end system working, establish observable metrics, and make more complicated machinery earn its way in. Sculley and collaborators describe the systems version of the failure mode as technical debt: ML systems accumulate hidden coupling, configuration debt, data dependencies, and maintenance costs much faster than an architecture diagram suggests.

I understood all of this intellectually.

Argus made me understand it experimentally.

The seductive architecture

The original intuition behind Argus still seems reasonable to me.

Suppose two classifiers produce:

If both models are structurally similar, their errors may be strongly correlated. But EfficientNet-B4 and ViT-B/16 do not process images in the same way. One builds hierarchical convolutional features; the other reasons over patch tokens with self-attention.

So there is at least a plausible hypothesis:

if the models fail differently, the shape of their disagreement may tell us something that the final class labels do not.

That last part matters. Argmax destroys information.

These two outputs:

and

produce the same top class. But they are very different predictive states.

With two models, the space becomes richer still. They can agree confidently, agree weakly, both be uncertain, or confidently contradict each other.

Argus tried to preserve that structure instead of immediately averaging it away.

The system measured each model’s entropy, Jensen-Shannon divergence between their distributions, a maximum probability difference, and—on triggered cases—spatial statistics derived from Grad-CAM++ and attention rollout.

The question was not simply, “Which model is right?”

It was:

Can the pattern of disagreement itself help decide what to trust?

That is a good question.

The mistake was letting a good question justify too much machinery before each piece had earned its place.

Version one: make the disagreement talk

The first consensus architecture was the clearest example.

When the visual models disagreed, I passed their predictions into an LLM-based debate stage. The generated arguments were embedded with a sentence transformer, concatenated with model outputs, and fed into another neural network.

The resulting representation was roughly 788 dimensions.

At the time, this felt like the system was becoming more intelligent. The models were no longer just emitting numbers. They could produce arguments. A downstream model could supposedly use those arguments as another source of evidence.

But follow the information path carefully:

image
  ↓
vision-model probabilities
  ↓
prompt construction
  ↓
generated text
  ↓
sentence embedding
  ↓
788-dimensional feature vector
  ↓
neural consensus
  ↓
final probability vector

Every arrow looks reasonable in isolation.

Together, they form a long chain of opportunities for information to be distorted, duplicated, reordered, spuriously correlated, or simply overfit.

And one of those opportunities became real: the feature contract between training and serving became fragile enough that a mismatch in feature ordering could make the downstream model consume the wrong semantics without producing a convenient exception.

This is a nasty property of ML code. Ordinary software often fails loudly when a contract breaks. A model can accept a perfectly valid tensor of the perfectly correct shape and still be semantically wrong.

float[788] tells you almost nothing about what dimension 417 means.

The system still runs.

The output still looks like a probability vector.

The bug has become statistical instead of syntactic.

That version of Argus was removed.

Not patched.

Removed.

788 → 23

The replacement was deliberately less magical.

Instead of converting model disagreement into generated language and then back into numbers, I kept the numerical evidence explicit:

8 Agent A probabilities
8 Agent B probabilities
4 distribution statistics
3 spatial statistics
-------------------------
23 features

The feature vector was small enough that I could write down what every dimension meant.

That sounds like an implementation detail. It changed the way I could reason about the system.

With 23 explicit values, I could ask useful questions:

• Are the raw probabilities doing most of the work?
• Does JSD add anything once both distributions are already present?
• Do the spatial features help prediction, or only interpretation?
• Does a feature exist because the hypothesis requires it, or because it was easy to add?
• Can training, evaluation, and serving agree on the exact ordering and semantics?

This is an underrated property of simple representations: they are easier to interrogate.

The consensus problem had also changed character. With only a few thousand second-stage examples and 23 tabular features, another neural network was no longer an obviously natural default. I eventually tried LightGBM.

This version was much cleaner.

Then the ablation made it cleaner still.

23 → 16

The full model used all 23 features.

Balanced accuracy:

I removed the three spatial features.

Balanced accuracy:

Then I removed the disagreement statistics too and left only the two probability vectors: 16 numbers.

Balanced accuracy:

The arithmetic ensemble was:

There is an important temptation here: bold 0.6360, bold 0.6353, declare a winner by seven ten-thousandths, and move on.

I do not think the evidence earns that.

These are point estimates on one held-out split. I did not retain the paired uncertainty analysis needed to make a strong claim about such a small difference. The useful conclusion is not that a 16-feature learned fusion “beat” averaging.

The useful conclusion is that the additional feature machinery failed to demonstrate incremental predictive value.

That is a much stronger engineering result than a microscopic leaderboard win.

The probability vectors were already carrying most of the information the second-stage model could exploit.

This also separated two ideas I had been mixing together.

The attention machinery was useful for inspection. A Grad-CAM++ map from the CNN and an attention-rollout map from the transformer gave me a way to look at how the two models’ spatial emphasis differed on an image.

But:

useful for inspection ≠ useful as a predictive feature.

A visualization can be scientifically or ergonomically valuable without increasing classification performance.

Once stated, this is obvious.

The ablation was what forced me to state it.

A second failure was hiding in the control flow

Argus had a fast path and a triggered path.

Spatial analysis costs more than two ordinary forward passes, so the design was:

If the models agreed confidently, skip the expensive spatial machinery. If uncertainty or disagreement crossed a threshold, compute the deeper analysis.

This is a nice systems idea because it turns uncertainty into a computational routing signal.

It only works if the router actually routes.

On the held-out set:

which means:

Only 202 images took the fast path.

The conditional-compute architecture was therefore conditional mostly in source code.

At the configured thresholds, the runtime behavior was much closer to “run the expensive path almost always.”

This taught me to distinguish two different levels of evaluation:

component correctness

Does the trigger compute JSD and entropy correctly?

and

system usefulness

Does the trigger produce a routing policy that meaningfully trades compute for risk?

The first can be completely true while the second is false.

A threshold is part of the model.

A branch in the architecture diagram is not evidence that meaningful branching occurs in practice.

The worst number was not accuracy

The headline comparison was already unfavorable:

Configuration	Balanced Accuracy	Macro AUC	ECE ↓
EfficientNet-B4	0.5469	0.9034	0.0745
ViT-B/16	0.6214	0.9502	0.0758
Standard Ensemble	0.6353	0.9433	0.1048
Argus Full	0.6190	0.9023	0.2060

But the number that changed my view of the project was 0.2060.

Expected Calibration Error is imperfect, but it is trying to ask an important question: do the model’s confidence values correspond to empirical correctness frequencies?

If a classifier emits probabilities, there is a semantic claim hiding inside them.

A score of $0.9$ should not merely mean “larger than 0.8.” If we want to use it as confidence, it should behave approximately like a probability.

Modern neural networks are often miscalibrated; Guo et al.’s classic calibration study made this point very clearly. That matters even more in a system whose architecture uses uncertainty to control downstream behavior.

Argus was supposed to reason about uncertainty.

Yet the learned consensus was considerably less calibrated than the simple ensemble.

This creates a nasty feedback loop in the design:

poor confidence
    ↓
routing / abstention decisions
    ↓
the system decides when to trust itself
using a quantity that is itself unreliable

The right response is not to add another agent that judges the confidence of the confidence.

It is to stop and characterize the quantity properly.

This is where my interest shifted from arbitration to selective prediction.

Maybe disagreement should not choose the label

Selective classification asks a different question from ordinary classification.

Ordinary classification asks:

What label should I output?

Selective classification adds:

Should I output a label at all?

The system trades coverage—the fraction of inputs it answers—against risk on the examples it keeps. This is a mature research problem in its own right, not an error-handling trick.

I ran a small post-hoc confidence-threshold sweep on Argus. Across the tested range, coverage fell from 100% to 89%, while selective balanced accuracy showed an overall upward trend from 0.6190 to 0.6550, with small non-monotonic fluctuations.

This experiment was based on output confidence, not JSD, so it does not establish that model disagreement is the right abstention score.

But it suggests a much cleaner next question:

Instead of asking disagreement to manufacture a better label, can uncertainty help rank which labels deserve the least trust?

That is a different research program.

It would compare confidence functions—maximum probability, entropy, JSD, ensemble disagreement, calibrated risk estimates—using proper risk-coverage analysis.

The most interesting output of the disagreement machinery might not be another prediction.

It might be a reason to withhold one.

Averages hide failure structure

Aggregate metrics can also make a system look calmer than it is.

Argus’s recall for Squamous Cell Carcinoma was 0.28 on the held-out split. Of 100 SCC examples, only 28 were classified correctly; 43 were sent to BCC.

Actinic Keratosis recall was 0.3609.

Those numbers need context—the class supports are small, so the estimates themselves are noisy—but that is precisely why a single global accuracy number is not enough.

The individual cases were revealing too.

In one example, the ground truth was melanocytic nevus. Agent A strongly preferred dermatofibroma. Agent B preferred nevus. The arithmetic ensemble leaned toward dermatofibroma.

Then the learned consensus intervened.

It predicted melanoma.

Wrongly.

Argus also has the opposite kind of example: cases where the learned consensus changes an incorrect ensemble output into the correct class.

It is very easy to build a demo from those.

Pick the successful example. Put four heatmaps beside it. Add an arrow. Call the section “When agents reason together.”

The picture is true.

The conclusion would be false.

A successful anecdote demonstrates that a behavior can happen. Evaluation tells you how often the behavior helps, how often it hurts, and whether the trade is worth making.

This is one of the reasons I now distrust case-study screenshots unless they are paired with aggregate evaluation and failure cases.

The most impressive example in a dataset is almost never the dataset.

Three ways complexity has to pay rent

After Argus, I have a more explicit test for added machinery.

When I add a component to an ML system, I want it to pay rent in at least one of three currencies.

1. Predictive value

Does it improve the metric that matters?

Not “can I construct an example where it helps?” Not “does the feature importance plot look interesting?” Does it improve performance on held-out data, and is the gain larger than the uncertainty in the comparison?

2. Reliability value

Does it improve calibration, robustness, interpretability, uncertainty ranking, or some other property we genuinely care about?

A component can be worthwhile even if top-1 accuracy is unchanged. But the claimed benefit has to be measured in the dimension where the component is supposed to help.

3. Systems value

Does it reduce latency, compute, operational complexity, failure probability, or debugging cost?

A router that sends 94.6% of examples to the expensive branch is not paying rent as a compute-saving router, even if the routing function is implemented perfectly.

This is not a theorem. It is just a useful engineering accounting system.

A component that pays none of these rents is decoration.

And in ML, decoration is expensive because it tends to be executable.

Complexity debt compounds

There is a deeper reason simple baselines are so hard to beat in deployed systems.

Suppose the complicated system improves a metric by some small $\Delta$.

The actual engineering question is not:

It is closer to:

The “everything” includes things the benchmark does not price:

• another artifact to version,
• another feature contract to preserve,
• another threshold to tune,
• another calibration behavior to monitor,
• another path through the serving code,
• another distribution shift interaction,
• another place where train and inference semantics can diverge.

This is why ML technical debt is unusually slippery. The dependency graph includes data, distributions, learned behavior and configuration, not just function calls.

A baseline has value beyond its score because it gives you a low-complexity control system.

If the complex method wins by a mile, great.

If it wins by a rounding error, the correct response should be skepticism, not confetti.

And if it loses, the baseline has done its job perfectly.

If I restarted Argus

I would invert the order of construction.

The arithmetic ensemble would be the product until something earned the right to replace it.

Start with:

Freeze the evaluation protocol.

Then add exactly one idea.

Learn a fusion over the 16 probabilities.

Measure.

Add entropy.

Measure.

Add JSD.

Measure.

Add spatial features.

Measure.

Change one hypothesis at a time.

I would also separate three questions that were too entangled in the original project:

1. Does disagreement identify harder examples?
2. Does disagreement improve classification when used as a feature?
3. Can uncertainty rank examples for abstention?

These are not different wordings of the same question.

They require different experiments.

For the classifier comparisons, I would retain per-sample predictions and run paired, class-aware resampling so that small metric deltas come with uncertainty intervals instead of four decimal places pretending to be certainty.

For the routing problem, I would evaluate the trigger as a curve rather than a magic threshold: error detection versus coverage versus computational cost.

For calibration, I would compare reliability before and after every learned fusion stage instead of treating calibration as a final cosmetic operation.

And I would make deletion a planned experimental outcome.

If a feature group fails an ablation, remove it.

If a router routes everything, redesign it.

If a second-stage model cannot beat averaging, ship the average.

The experiment is a deletion machine

There is a style of engineering where progress is measured by accumulation.

More components. More abstractions. More agents. More lines in the architecture diagram.

ML encourages this because almost every additional idea sounds plausible. The model could use another feature. The agent could call another agent. A confidence score could gate another branch. An explanation could become another input.

The hard part is not inventing reasons to add things.

The hard part is constructing experiments capable of taking them away.

That is what the baseline is for.

It is a deliberately stubborn object in the experiment. It refuses to become more impressive. It refuses to care about the elegance of the new architecture. It keeps asking the same question:

What did the extra complexity buy?

In Argus, the answer was uncomfortable.

The learned consensus did not beat simple averaging on the main classification comparison. The spatial features did not demonstrate incremental predictive value. Calibration got worse. The selective-compute trigger selected almost everything.

So the project became smaller.

First 788 dimensions became 23.

Then 23 became 16.

And eventually the most annoying competitor in the room was still:

I used to think a failed experiment was one where the sophisticated idea lost.

Now I think a failed experiment is one that cannot tell you what to delete.

The baseline did.

That is why the baseline was never the boring part.

The baseline was the experiment.

---

Further reading

A few pieces that shaped how I now think about this problem:

• A Recipe for Training Neural Networks — Andrej Karpathy — especially the discipline of moving from simple to complex while validating one hypothesis at a time.
• Rules of Machine Learning — Martin Zinkevich / Google — practical arguments for simple first systems, observable metrics, and avoiding unnecessary complexity.
• Hidden Technical Debt in Machine Learning Systems — Sculley et al. — why the cost of ML complexity extends far beyond model parameters.
• On Calibration of Modern Neural Networks — Guo et al. — the classic modern treatment of confidence calibration.
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles — Lakshminarayanan et al. — a useful reminder that simple ensemble methods can be remarkably strong uncertainty baselines.
• SelectiveNet — Geifman & El-Yaniv — selective prediction framed explicitly as a risk/coverage problem.
• Research Debt — Chris Olah & Shan Carter — a broader argument for making research ideas easier to reason about rather than merely accumulating them.