Argus Vision: The Anatomy of Neural Network Disagreement

The Argmax Was Throwing Something Away

Every standard classification model outputs a continuous probability distribution across target classes. Before that distribution reaches an interface or a downstream decision module, a single mathematical operation almost always occurs: the argmax. The highest probability scalar wins, and the remaining values are discarded. This reduction is convenient for user interfaces, but it destroys structural information about predictive uncertainty.

Consider two contrasting scenarios in an eight-class dermoscopy classifier. In the first case, Melanoma receives a probability of 0.51 while Melanocytic Nevus receives 0.47. In the second case, Melanoma receives a probability of 0.99 while Melanocytic Nevus receives 0.002. In both scenarios, the argmax operation outputs the exact same label: Melanoma. Yet the underlying clinical and mathematical reality of these two cases is fundamentally different. The first represents a high-uncertainty boundary case where two major diagnostic hypotheses are nearly tied. The second represents an overwhelming single-class consensus.

When a single model is uncertain, its internal probability distribution flattens. However, when two structurally different models evaluate the same visual input, a far richer set of behaviors becomes possible. The models might both predict the same class with high confidence. They might both exhibit high internal entropy. Alternatively, each model might output a highly confident prediction for a completely different class.

A conventional ensemble immediately averages these conflicting probability vectors together and takes the mean. Argus Vision was built to explore a different question: what if we refuse to average that disagreement away immediately? What if distributional disagreement, predictive entropy, and spatial representation conflict are preserved as first-class inputs to an arbitration layer?

This case study documents the engineering, architecture, and empirical findings behind Argus Vision. It covers the visual backbones, the mathematical disagreement trigger, the failure of an early LLM debate prototype, the shift to a canonical 23-dimensional numerical contract, and the trade-offs observed while evaluating multi-agent consensus on imbalanced medical imaging data.

---

Experiment at a Glance

To evaluate whether preserving model disagreement improves classification performance over traditional single-model and ensembling baselines, Argus Vision was benchmarked on the ISIC 2019 dermoscopy dataset across eight diagnostic categories. Because dermoscopic datasets exhibit class imbalance, raw classification accuracy is a secondary metric. Macro AUC, balanced accuracy (unweighted mean per-class recall), and Expected Calibration Error (ECE) serve as the primary evaluation criteria.

The table below summarizes the core experimental metrics evaluated across 3,740 held-out test images (15% lesion-grouped split).

Model / Configuration	Accuracy	Balanced Accuracy	Macro AUC	ECE	Evaluation Notes & Status
Agent A (EfficientNet-B4)	0.6369	0.5469	0.9034	0.0745	Local feature detection
Agent B (ViT-B/16)	0.7644	0.6214	0.9502	0.0758	Global patch self-attention
Standard Ensemble (Arithmetic Mean)	0.7687	0.6353	0.9433	0.1048	Unweighted mean of $p_A$ and $p_B$; primary baseline
Argus Full (23-dim Consensus)	0.7580	0.6190	0.9023	0.2060	Full trigger + 23-dim feature vector + LightGBM
Argus (No Attention Features)	0.7636	0.6300	0.9016	0.2001	Ablation: dims 20–22 set to zero
Argus (Probabilities Only)	0.7639	0.6360	0.9088	0.2024	Ablation: dims 16–22 set to zero

Source: Evaluation benchmark results across 3,740 held-out test split images. (Note: A separate Deep Ensemble variant was planned but skipped due to unattached second-seed checkpoints.)

Raw accuracy can be deceptive in skin lesion classification. A naive model that predicts the majority class (Melanocytic Nevus) for every ambiguous lesion can attain overall accuracy while failing on minority classes like Melanoma or Squamous Cell Carcinoma. Balanced accuracy and Macro AUC force the evaluation framework to weigh all eight diagnostic categories equally regardless of their frequency in the training set.

×

Figure 1: Empirical comparison across model configurations on the 3,740 held-out test images. The simple Standard Ensemble outperforms full Argus consensus on balanced accuracy and calibration.

---

Two Models, Two Ways to Be Wrong

The decision to pair a Convolutional Neural Network with a Vision Transformer was driven by a fundamental premise: models with different inductive biases fail on different visual patterns. If two models possess identical architectural assumptions, their errors will be correlated. When one model makes a mistake, the second model will likely make the exact same mistake.

EfficientNet-B4 represents the traditional CNN paradigm. Its inductive biases are spatially localized. Through translation equivariance and hierarchical convolutional receptive fields, EfficientNet excels at detecting micro-morphological structures. In dermoscopy, these features correspond to fine pigment networks, streaks, dots, and localized border irregularities. However, because its effective receptive field expands gradually layer by layer, a CNN can struggle to capture global spatial symmetry across the entire lesion.

ViT-B/16 operates under different architectural assumptions. By dividing the input image into a grid of 16x16 patch tokens and processing them through multi-head self-attention, the Vision Transformer has weak initial spatial inductive biases. It does not assume that adjacent pixels are more related than distant pixels. Initialized with ImageNet-21k pre-trained weights, ViT-B/16 captures global spatial relationships, overall lesion symmetry, and macro color contrasts immediately at the first attention layer.

×

Figure 2: Architectural divergence between CNN local receptive fields and ViT global patch self-attention. The design hypothesis relies on their errors being imperfectly correlated.

The core hypothesis behind Argus Vision does not claim that Vision Transformers are inherently superior to CNNs or vice versa. Rather, it posits that their error distributions are imperfectly correlated. When an EfficientNet misclassifies a lesion because of an atypical local border artifact, the Vision Transformer’s global self-attention may remain unfooled. Conversely, when a ViT misinterprets a lesion due to global color variations, the CNN’s local feature hierarchy can identify characteristic pigment patterns.

---

The Dataset Refuses to Make This Easy

Building a classifier on the ISIC 2019 dataset immediately highlights two data-level challenges: class imbalance and data leakage across splits.

The ISIC 2019 dataset contains 25,331 images divided into eight categories. The majority class, Melanocytic Nevus (NV), contains 12,875 images—accounting for over 50% of the entire dataset. In contrast, Dermatofibroma (DF) contains 239 images, and Vascular Lesions (VASC) contain 253. This represents a class imbalance ratio exceeding 50:1 between the most frequent and least frequent diagnostic categories.

×

Figure 3: ISIC 2019 dataset class distribution (25,331 total images). Majority class Melanocytic Nevus (12,875 samples) dwarfs minority classes like Dermatofibroma (239 samples) and Vascular Lesions (253 samples).

Standard cross-entropy loss applied to this distribution causes the model to optimize heavily for Nevus recall while failing on minority classes. To address this without synthesizing fake pixels or discarding training samples, loss and sampling strategies were integrated into the training pipeline.

The second major data challenge is data leakage across cross-validation splits. In clinical datasets, multiple dermoscopic photographs are frequently captured from the same patient or the same lesion over time. If images sharing a lesion_id are randomly assigned across training and validation folds, the model can memorize patient-specific skin characteristics, producing validation scores that collapse when tested on unseen lesions.

The dataset splitting pipeline enforces isolation using scikit-learn’s StratifiedGroupKFold. Samples are grouped strictly by lesion_id. For metadata records lacking an explicit lesion identifier, the image filename is treated as an isolated single-item group. The split function enforces an explicit runtime check:

assert_no_lesion_leakage(train_df, val_df)

If a single lesion_id is detected on both sides of the partition, dataset initialization halts execution. This lesion-level grouping prevents samples sharing a lesion_id from appearing on both sides of the train/validation partition across the 15% held-out test set ($N=3,740$). (Note: Because patient-level metadata identifiers are incomplete in the source dataset, lesion grouping provides lesion-level isolation rather than proven patient-level separation.)

---

Training Disagreement

Training visual backbones on imbalanced data requires separating feature representation learning from decision boundary re-balancing. Argus Vision implements the decoupled training framework established by Kang et al. (2020), configured under decoupled training mode.

Jointly training a network with heavy class-balanced sampling from epoch zero often distorts the backbone’s feature space. Because minority samples are oversampled continuously, early layers can overfit to the repeated visual artifacts of those images. Decoupled training resolves this by breaking optimization into two distinct phases.

×

Figure 4: Decoupled training strategy. Stage A learns representation features with instance-balanced sampling. Stage B freezes the backbone and retrains only the classifier head using effective-number sample weighting.

In Stage A, representation learning is conducted for up to 30 epochs using standard instance-balanced data loading. Every image in the dataset is sampled once per epoch. The loss function is an unweighted Focal Loss ($\gamma = 2.0$) with label smoothing ($\epsilon = 0.1$). This phase forces the backbone to learn visual representations without warping feature space toward artificial class frequencies.

In Stage B, classifier head re-balancing takes place for up to 10 epochs. The backbone feature extractor is frozen, and only the final linear classification layer is retrained. Data loading shifts to a WeightedRandomSampler, where per-class sampling weights are computed using the Cui et al. (2019) effective-number-of-samples formulation:

The hyperparameter $\beta$ is set to $0.999$. Setting $\beta = 0.99$ produces near-uniform weights for datasets of this scale because $1/(1-\beta) = 100$, causing weight saturation for all classes with over 100 samples. Conversely, $\beta = 0.9999$ creates extreme oversampling of rare classes, forcing the sampler to repeatedly pass the same 239 DF images while ignoring dataset diversity. $\beta = 0.999$ scales weights across the 50:1 sample count gradient.

To check that Stage B does not modify backbone weights, the training pipeline implements a parameter verification check. Before Stage B begins, snapshot_frozen_params() clones parameter tensors where requires_grad == False. After Stage B completes, assert_frozen_unchanged() compares the current model parameters against the snapshotted tensors using exact bit-level matching:

if not torch.equal(param.detach(), snapshot[name]):
    changed.append(name)
assert not changed, f"Frozen backbone parameters CHANGED during Stage B: {changed[:5]}"

Furthermore, standard PyTorch parameter freezing does not stop BatchNorm modules from updating their running statistics when model.train() is invoked. To prevent BatchNorm statistics from drifting during Stage B, freeze_backbone_bn() sets BatchNorm modules to eval() mode and replaces their .train() method with a dynamic no-op lambda function. This enforces parameter and buffer preservation throughout classifier head retraining.

Optimization utilizes AdamW with differential learning rates: $1 \times 10^{-3}$ for the linear classifier head, $1 \times 10^{-5}$ for convolutional backbones, and $1 \times 10^{-6}$ for transformer self-attention layers. Cosine annealing schedules the learning rate decay, paired with early stopping monitoring macro AUC over an 8-epoch patience window.

---

Agreement Is the Fast Case

Once both agents are trained, running inference on an input image produces two 8-dimensional probability distributions: $p_A$ from EfficientNet-B4 and $p_B$ from ViT-B/16. To determine whether an input image requires spatial analysis or can proceed directly to classification, the pipeline evaluates a mathematical disagreement trigger.

The trigger combines two complementary information-theoretic metrics: Jensen-Shannon Divergence (JSD) and Shannon Entropy.

×

Figure 5: Dual-metric disagreement trigger logic. Shannon entropy evaluates individual model uncertainty, while Jensen-Shannon divergence evaluates inter-model contradiction.

Jensen-Shannon Divergence measures the symmetric distance between two probability distributions. Unlike Kullback-Leibler divergence, JSD is bounded between 0 and 1 bit (when using base-2 logarithms) and remains stable when probability vectors contain near-zero values:

Shannon Entropy quantifies the internal predictive uncertainty of each individual agent. For an 8-class probability distribution, entropy is calculated as:

A uniform distribution across all 8 classes yields an entropy of $3.0$ bits, whereas a completely certain prediction yields $0.0$ bits.

These two metrics identify three operational regimes:

1. Confident Agreement (Low JSD, Low Entropy): Both agents predict the same class with high probability. Here, $\text{JSD} \le 0.15$ and $\max(H_A, H_B) \le 0.6$ bits. The pipeline executes the Fast Path, bypassing spatial attention maps to minimize latency.
2. Individual Uncertainty (High Entropy): One or both agents produce flat probability distributions (e.g., $H(p_A) = 1.4$ bits). Even if JSD is moderate, high entropy signals that the visual features are ambiguous. The trigger fires to initiate spatial analysis.
3. Confident Contradiction (High JSD, Low Entropy): Both agents are confident, but they predict different classes (e.g., Agent A predicts Melanoma at $0.88$, while Agent B predicts Benign Keratosis at $0.85$). Individual entropies are low, but $\text{JSD} > 0.15$. This represents an inter-model contradiction and triggers spatial attention analysis.

Note a documentation discrepancy uncovered during forensics: early notes listed trigger thresholds of $\text{JSD} > 0.25$ and $H > 0.8$. However, the authoritative application configuration defines DEBATE_JS_THRESHOLD = 0.15 and DEBATE_ENTROPY_THRESHOLD = 0.6. The runtime serving code enforces these tighter thresholds.

---

Looking Inside the Disagreement

When the debate trigger fires, the pipeline executes spatial attention extraction to evaluate where each backbone focused its visual capacity.

Because EfficientNet-B4 and ViT-B/16 belong to different architectural families, extracting spatial saliency maps requires model-specific interpretability methods.

For EfficientNet-B4, the pipeline applies Grad-CAM++. Grad-CAM++ calculates gradients of the target class logit with respect to the final convolutional feature maps (model.blocks[-1]). This produces a 224x224 heatmap normalized to [0, 1], highlighting the localized pixel regions that contributed to Agent A’s top predicted class.

For ViT-B/16, Grad-CAM++ cannot be applied directly because Vision Transformers lack spatial convolution channels. Instead, the pipeline uses Attention Rollout based on Abnar & Zuidema (2020). Forward hooks intercept the pre-softmax activations in each of the 12 transformer blocks, re-deriving the 12-head attention tensors of shape (1, 12, 197, 197)—representing 196 image patch tokens plus 1 [CLS] token across 12 attention heads.

Attention weights are averaged across heads to produce a (197, 197) matrix per layer, combined with an identity matrix to account for residual skip connections ($0.5 \mathbf{A} + 0.5 \mathbf{I}$), row-normalized, and recursively multiplied across layers:

The row corresponding to the [CLS] token in the final rollout matrix $\mathbf{R}^{(12)}$ represents the cumulative attention flow from the classification token to all 196 image patch tokens (rollout[0, 1:]). This 196-element vector is reshaped to a $14 \times 14$ grid and upsampled via cubic interpolation (cv2.resize) to $224 \times 224$, normalized to [0, 1].

An implementation detail is documented in the attention rollout module. When PyTorch executes timm Vision Transformers with fused_attn=True, attention operations dispatch to C++/CUDA FlashAttention kernels. These fast kernels do not expose intermediate attention matrices to PyTorch forward hooks. To compute Attention Rollout, the rollout function temporarily disables fused_attn across transformer blocks, executes the forward pass, captures the attention matrices, and restores fused_attn inside a finally block:

try:
    for block in model.blocks:
        if hasattr(block.attn, "fused_attn"):
            block.attn.fused_attn = False
    # Execute forward pass and extract attention hooks...
finally:
    for idx, block in enumerate(model.blocks):
        if hasattr(block.attn, "fused_attn"):
            block.attn.fused_attn = original_fused[idx]

Once both heatmaps $M_A$ and $M_B$ are generated, the pipeline computes the spatial disagreement map $M_{\Delta}$:

A spatial bounding box is extracted around the region containing the top 20% activation intensity of $M_{\Delta}$, identifying the visual region where the CNN and Vision Transformer exhibit spatial evidence divergence.

---

Refactoring the Early Prototype

The evolution of Argus Vision’s consensus mechanism provides an example of architectural simplification in machine learning systems.

In an early prototype (Generation 1), when two agents disagreed, their probability predictions were passed to a Large Language Model (Groq API using Llama-3). The LLM was prompted with diagnostic descriptions of the 8 ISIC classes to generate text arguments representing each agent’s viewpoint.

These natural language debate transcripts were then passed through a sentence transformer (all-MiniLM-L6-v2) to generate 768-dimensional text embeddings. These text embeddings were concatenated with the agents’ probability vectors to produce a 788-dimensional feature representation that fed into an MLP consensus head.

×

Figure 6: Architectural evolution from Generation 1 to Generation 2. Replacing 788-dimensional text embeddings with a 23-dimensional numerical contract eliminated train/serve feature drift and reduced dimensionality by 97%.

This design introduced severe engineering issues.

First, passing natural language embeddings into a downstream classifier introduced variance. Small changes in LLM prompt phrasing altered sentence embeddings, shifting input features independently of the underlying visual evidence.

Second, the system suffered from a feature contract ordering misalignment. As documented during architectural review, a discrepancy between the text embedding concatenation code in training and the live backend serving pipeline caused feature columns to misalign, creating substantial overfitting risk and performance degradation.

Third, fitting an MLP on 788 continuous features using a disagreement dataset of ~3,800 samples created overfitting risk.

Generation 1 was deprecated. Commit d64001a (“feat: implement numeric consensus pipeline and remove LLM debate logic”) stripped out the Groq API integration, sentence transformer dependencies, and the 788-dimensional feature vector. The LLM debate pipeline was replaced with a deterministic 23-dimensional numerical contract.

---

Twenty-Three Numbers

The consensus layer relies on a fixed-width, 23-dimensional numerical feature contract. This vector distills predictive, uncertainty, and spatial disagreement information into an array.

×

Figure 7: Anatomy of the 23-dimensional consensus feature contract. The vector combines raw agent probabilities (16 dims), distribution statistics (4 dims), and spatial attention metrics (3 dims).

The 23 features are organized into four semantic groups:

• Indices [0:8] — Agent A Probabilities ($p_A$): Softmax probabilities across the 8 ISIC classes (MEL, NV, BCC, AK, BKL, DF, VASC, SCC) from EfficientNet-B4.
• Indices [8:16] — Agent B Probabilities ($p_B$): Softmax probabilities across the 8 ISIC classes from ViT-B/16.
• Indices [16:20] — Uncertainty & Disagreement Statistics (4 Dims):
  • [16] js_div: Jensen-Shannon Divergence between $p_A$ and $p_B$.
  • [17] entropy_a: Shannon Entropy of $p_A$ in bits.
  • [18] entropy_b: Shannon Entropy of $p_B$ in bits.
  • [19] max_prob_delta: Maximum absolute difference $\max_c |p_{A,c} - p_{B,c}|$ across all 8 classes.
• Indices [20:23] — Spatial Attention Statistics (3 Dims):
  • [20] attn_iou: Intersection-over-Union (IoU) of the two attention heatmaps thresholded at $0.5$.
  • [21] attn_entropy_a: Spatial entropy of Agent A’s normalized heatmap.
  • [22] attn_entropy_b: Spatial entropy of Agent B’s normalized heatmap.

When an image executes via the Fast Path (disagreement below threshold), spatial attention maps are not computed. On this path, indices [20:22] are explicitly populated with 0.0, allowing fast-path and triggered-path samples to pass through a unified contract.

To maintain contract consistency, this 23-dimensional feature extractor is defined identically across the training and live backend serving pipelines.

---

Replacing the MLP with Gradient Boosting

With the consensus input reduced from 788 text dimensions to 23 numerical features, the arbitration task shifted to a tabular classification problem over ~3,800 samples.

Initially, a PyTorch MLP (ConsensusMLP) was trained on this vector (Linear(23->128) -> BatchNorm -> ReLU -> Dropout(0.3) -> Linear(128->64) -> BatchNorm -> ReLU -> Dropout(0.3) -> Linear(64->8)). In the final consensus pipeline, the consensus classifier was replaced with LightGBM (consensus_lgbm.pkl).

LightGBM offered two practical characteristics for this feature contract:

1. Scaling Invariance: Decision trees split on feature ordinal ranks, making them invariant to monotonic feature scaling. While input features are still passed through StandardScaler for backward compatibility with the PyTorch MLP fallback, LightGBM decision boundaries are unperturbed by feature scaling.
2. Sample Weighting Integration: LightGBM supports sample-level training weights. To penalize clinically dangerous misclassifications, Melanoma (MEL) samples in the consensus training set were assigned a $4.0 \times$ weight multiplier. Squamous Cell Carcinoma (SCC) samples received a $2.0 \times$ multiplier.

The backend serving implementation maintains dual loading. At startup, ConsensusClassifier attempts to load the LightGBM model (consensus_lgbm.pkl). If LightGBM or its scikit-learn dependency is missing, it logs a warning and falls back to loading the PyTorch consensus_best.pth checkpoint.

---

Production Serving Architecture

Serving this multi-agent architecture required building an asynchronous backend infrastructure.

×

Figure 8: Argus Vision serving topology. FastAPI offloads PyTorch model inference to worker threads via asyncio.to_thread, streaming pipeline stage events to Next.js via Redis pub/sub WebSockets.

The production application runs via Docker Compose across four containerized services:

• Nginx Proxy (:80): Routes static UI traffic to Next.js (:3000) and API/WebSocket connections to FastAPI (:8000).
• Next.js 14 Frontend (:3000): Provides a workstation-inspired interface featuring a 4-quadrant viewer, probability progress bars, and spatial visual overlays.
• FastAPI Backend (:8000): Orchestrates the ML pipeline, exposes REST endpoints (/api/classify, /api/jobs), and manages WebSocket subscriptions (/ws/debate/{job_id}).
• Redis Store (:6379): Functions as a key-value job store and pub/sub message broker for real-time WebSocket event streaming.

To prevent PyTorch model executions from blocking FastAPI’s main asyncio event loop, inference steps—Agent A prediction, Agent B prediction, Grad-CAM++ rendering, Attention Rollout matrix multiplication, and LightGBM evaluation—are offloaded to background thread pools using asyncio.to_thread.

As each pipeline stage completes, the orchestrator updates job state in Redis and publishes a JSON event payload. Connected WebSocket clients receive stage updates (agents_running $\rightarrow$ agents_done $\rightarrow$ trigger_evaluated $\rightarrow$ attention_computed $\rightarrow$ consensus_done), driving the step-by-step progress UI.

Before image tensors enter the visual agents, a Dermoscopy Input Gate evaluates the upload. The gate runs heuristic checks verifying aspect ratios ($\le 2.5$), minimum dimensions ($\ge 50\text{px}$), and non-zero RGB channel standard deviations ($\ge 8.0$). It then passes the image through a MobileNetV3 classifier to check whether the upload resembles a skin lesion photograph. Non-dermoscopic images are rejected immediately, protecting processing capacity.

Note regarding the visual debate transcript: the dialogue displayed in the UI sidebar is rendered client-side using a template state machine. No external LLM API calls are executed during live inference.

---

Empirical Evaluation Results

The complete evaluation suite was executed against the 3,740 held-out test split images.

The empirical measurements reveal the following comparison across configurations:

1. Agent B (ViT-B/16) achieved 0.7644 Accuracy, 0.6214 Balanced Accuracy, and 0.9502 Macro AUC with an Expected Calibration Error of 0.0758.
2. Standard Ensemble (Arithmetic Mean) achieved 0.7687 Accuracy, 0.6353 Balanced Accuracy, and 0.9433 Macro AUC with an ECE of 0.1048.
3. Argus Full Consensus (23-dim LightGBM) achieved 0.7580 Accuracy, 0.6190 Balanced Accuracy, 0.9023 Macro AUC, and 0.2060 ECE.

The simple unweighted arithmetic mean (Standard Ensemble) achieved higher balanced accuracy and lower calibration error than the full Argus consensus model on the test split.

Standard Ensemble attained higher accuracy (0.7687 vs 0.7580), higher balanced accuracy (0.6353 vs 0.6190), higher macro AUC (0.9433 vs 0.9023), and better calibration (ECE 0.1048 vs 0.2060).

---

Feature Group Ablation

To evaluate the contribution of individual feature groups in the 23-dimensional contract, an ablation study was conducted.

×

Figure 9: Consensus feature ablation study. Removing spatial attention features changed balanced accuracy from 0.6190 to 0.6300. Using probabilities only yielded a balanced accuracy of 0.6360.

The ablation systematically set feature groups to zero:

• Full 23-Dim Contract: Balanced Accuracy = 0.6190 | Macro AUC = 0.9023 | ECE = 0.2060
• No Spatial Attention Features (dims 20–22 = 0): Balanced Accuracy = 0.6300 | Macro AUC = 0.9016 | ECE = 0.2001
• Probabilities Only (dims 16–22 = 0): Balanced Accuracy = 0.6360 | Macro AUC = 0.9088 | ECE = 0.2024

The additional spatial features did not demonstrate incremental predictive value in this evaluation.

Removing spatial attention features changed balanced accuracy from 0.6190 to 0.6300. When the consensus model was evaluated on raw probability features alone (dims 0–15), its balanced accuracy reached 0.6360—similar to the Standard Ensemble baseline (0.6353).

Without paired statistical uncertainty bounds, small differences between these point estimates should not be overinterpreted. However, the measurements indicate that the primary predictive information was already captured by the agents’ probability vectors.

---

Disagreement Subset Metrics ($D_{\text{hard}}$) & Trigger Selectivity

Evaluating the system on the full test split ($N=3,740$) vs. the disagreement-triggered subset $D_{\text{hard}}$ ($N=3,538$) exposes an important operational insight regarding trigger selectivity.

Out of 3,740 held-out test images, 3,538 images (94.60%) triggered the disagreement path, while only 202 images (5.40%) proceeded down the fast path.

On $D_{\text{hard}}$, observed performance across configurations dropped relative to the full split:

• Agent A: Accuracy = 0.6165 | Balanced Accuracy = 0.5317 | Macro AUC = 0.8955 | ECE = 0.0775
• Agent B: Accuracy = 0.7512 | Balanced Accuracy = 0.6094 | Macro AUC = 0.9469 | ECE = 0.0808
• Standard Ensemble: Accuracy = 0.7557 | Balanced Accuracy = 0.6236 | Macro AUC = 0.9388 | ECE = 0.1086
• Argus Full: Accuracy = 0.7444 | Balanced Accuracy = 0.6076 | Macro AUC = 0.8979 | ECE = 0.2176

The configured thresholds ($\text{JSD} > 0.15$ or $\max(H_A, H_B) > 0.6$ bits) successfully identify a population $D_{\text{hard}}$ exhibiting somewhat lower observed metrics across all models compared to the fast-path subset. However, because 94.60% of the test set triggers the spatial analysis path, the current threshold configuration does not yet demonstrate an efficient selective-computation mechanism. Rather than isolating a small, highly selective set of ambiguous cases, the trigger dispatches almost the entire dataset to the expensive pipeline. Re-calibrating these thresholds to optimize the trade-off between error detection risk and compute overhead remains an essential topic for future experimentation.

Examining the binary Malignant vs. Benign regrouping shows the following point estimates:

• Agent A: Malignant Recall = 0.7059 | Precision = 0.7589 | F1 = 0.7314
• Agent B: Malignant Recall = 0.7730 | Precision = 0.8189 | F1 = 0.7953
• Standard Ensemble: Malignant Recall = 0.7689 | Precision = 0.8279 | F1 = 0.7973
• Argus Full: Malignant Recall = 0.7931 | Precision = 0.7828 | F1 = 0.7879

The cost-sensitive Argus configuration exhibits higher malignant recall (79.31%), lower precision (78.28%), and higher ECE (0.2060). Class-weighted sample boosting during LightGBM training is a plausible contributor to this shift in decision threshold toward malignant sensitivity.

---

Per-Class Metric Disparities

Examining the normalized confusion matrix for Argus Full consensus reveals variation across diagnostic categories.

×

Figure 10: Normalized confusion matrix across 8 classes. High performance on Nevus (0.86) and Basal Cell Carcinoma (0.80) contrasts with confusion on Squamous Cell Carcinoma (0.28 recall, 43% misclassified as BCC).

The per-class breakdown shows the following recall distribution:

• Melanocytic Nevus (NV, Support = 1,876): Recall = 0.8598 | Precision = 0.8771 | F1 = 0.8684
• Basal Cell Carcinoma (BCC, Support = 453): Recall = 0.7991 | Precision = 0.6882 | F1 = 0.7395
• Melanoma (MEL, Support = 759): Recall = 0.6891 | Precision = 0.6714 | F1 = 0.6801
• Benign Keratosis (BKL, Support = 350): Recall = 0.6143 | Precision = 0.5688 | F1 = 0.5907
• Vascular Lesions (VASC, Support = 33): Recall = 0.8485 | Precision = 0.7179 | F1 = 0.7778
• Dermatofibroma (DF, Support = 36): Recall = 0.5000 | Precision = 0.9000 | F1 = 0.6429
• Actinic Keratosis (AK, Support = 133): Recall = 0.3609 | Precision = 0.5714 | F1 = 0.4424
• Squamous Cell Carcinoma (SCC, Support = 100): Recall = 0.2800 | Precision = 0.3733 | F1 = 0.3200

The lowest recall occurred on Squamous Cell Carcinoma (SCC). Out of 100 true SCC cases in the test set, Argus correctly classified 28. In this test split, 43% of SCC lesions were misclassified as Basal Cell Carcinoma (BCC). For Actinic Keratosis (AK), 24% of cases were misclassified as BCC and 23% as BKL.

---

Analysis of Sample Predictions

Inspecting individual test predictions demonstrates how consensus arbitration changes predictions relative to individual agents and simple ensembles.

×

Figure 11: Test Case ISIC_0000163 (Melanoma). Agent A predicted NV (0.648) and Agent B predicted NV (0.545). The Standard Ensemble predicted NV (0.60), while the consensus output changed the prediction to Melanoma (0.556).

In test case ISIC_0000163 (Ground Truth: Melanoma), Agent A predicted Melanocytic Nevus ($p_A(\text{NV}) = 0.648$), and Agent B also top-predicted Melanocytic Nevus ($p_B(\text{NV}) = 0.545$). The Standard Ensemble predicted Nevus at 0.60. However, the LightGBM consensus head changed the output prediction to Melanoma (0.556).

Similarly, in test case ISIC_0000145 (Ground Truth: Melanoma), Agent A top-predicted NV (0.602) and Agent B top-predicted NV (0.704). The simple ensemble predicted NV (0.65), whereas the consensus model changed the output prediction to Melanoma (0.768).

×

Figure 12: Test Case ISIC_0000058 (Ground Truth: Melanocytic Nevus). Agent A predicted DF (0.954) and Agent B predicted NV (0.597). Ensemble predicted DF (0.48), while Argus consensus misclassified the lesion as Melanoma (0.701).

Conversely, test case ISIC_0000058 (Ground Truth: Melanocytic Nevus) illustrates a failure mode. Agent A predicted Dermatofibroma ($p_A(\text{DF}) = 0.954$) while Agent B predicted Nevus ($p_B(\text{NV}) = 0.597$). The Standard Ensemble predicted DF (0.48). The LightGBM consensus head misclassified the image as Melanoma (0.701).

Inspecting the confident error analysis shows 101 instances where Argus output a confidence score exceeding $0.95$ on an incorrect prediction. Many of these confident errors involved Melanocytic Nevi misclassified as Melanoma ($0.9999$ confidence).

---

Selective Prediction (Abstention) Analysis

To evaluate whether confidence scores can gate predictions, a selective prediction sweep was executed across confidence thresholds.

×

Figure 13: Selective prediction sweep across confidence thresholds tau=0.30 to tau=0.90.

Evaluating confidence threshold $\tau$ from 0.30 to 0.90 yields the following coverage and selective accuracy profile:

• At $\tau = 0.30$ (100% coverage), selective balanced accuracy is 0.6190.
• At $\tau = 0.60$ (97.5% coverage), selective balanced accuracy increases to 0.6354.
• At $\tau = 0.75$ (94.1% coverage), selective balanced accuracy reaches 0.6510.
• At $\tau = 0.90$ (89.0% coverage), selective balanced accuracy reaches 0.6550.

Across the evaluated range ($\tau = 0.30 \rightarrow 0.90$), confidence-threshold selective prediction showed an overall upward trend in selective balanced accuracy as coverage decreased (from 0.6190 to 0.6550), accompanied by small non-monotonic fluctuations (such as $\tau = 0.40 \rightarrow 0.45$, where selective balanced accuracy moved from 0.6192 to 0.6183).

---

Empirical Findings Summary

The empirical evaluation of Argus Vision establishes the following observational findings:

1. Standard Ensemble Outperformed Learned Consensus: On this test split, an unweighted arithmetic mean (Standard Ensemble) achieved higher balanced accuracy (0.6353 vs 0.6190) and lower calibration error (ECE 0.1048 vs 0.2060) than the 23-dimensional LightGBM consensus head.
2. Spatial Attention Features Provided No Measured Advantage: Removing spatial attention metrics from the feature vector changed consensus balanced accuracy from 0.6190 to 0.6300, while evaluating raw probability vectors alone yielded 0.6360.
3. Cost-Sensitive Weighting Shifted Thresholds: The cost-sensitive Argus configuration exhibited higher malignant recall (79.31% vs 76.89%) at the expense of lower precision (78.28% vs 82.79%) and higher ECE.
4. Disagreement Trigger Dispatched 94.60% of Inputs: The configured trigger thresholds sent 94.60% of test samples to the spatial analysis path, demonstrating that threshold re-calibration is required to achieve selective computational efficiency.
5. Selective Abstention Showed Upward Accuracy Trend: Sweeping confidence thresholds from $\tau = 0.30$ to $\tau = 0.90$ increased selective balanced accuracy from 0.6190 to 0.6550 while retaining 89.0% coverage.

---

Future Directions

Based on these empirical findings, potential modifications to this architecture include:

• Evaluating Simple Calibrated Ensembles: Utilizing calibrated arithmetic probability averaging as the primary classification output rather than training secondary tabular arbitration models.
• Re-calibrating Disagreement Thresholds: Re-evaluating JSD and entropy trigger thresholds to isolate a truly selective, high-uncertainty subset rather than dispatching 94.60% of inputs to the spatial path.
• Harmonizing Saliency Maps: Replacing model-specific interpretability methods (Grad-CAM++ and Attention Rollout) with uniform attribution frameworks such as Integrated Gradients across both backbones.
• Addressing Visual Boundary Confusions: Exploring targeted loss formulations to resolve fine-grained morphological confusion between Squamous Cell Carcinoma and Basal Cell Carcinoma.

---

Conclusion

Argus Vision evaluated whether explicitly preserving model disagreement across structurally divergent vision backbones improves classification performance over standard ensembling.

Empirical evaluation on the ISIC 2019 test split demonstrated that simple unweighted probability ensembling remains a strong baseline, outperforming learned tabular consensus heads on balanced accuracy and calibration. While confidence/uncertainty-based selective prediction provides a signal for selective abstention, constructing complex secondary arbitration models over spatial and disagreement features did not yield incremental performance over simpler baselines.