Two X-Ray Silhouettes Produced a 3D Femur, but the Hard Part Was Correspondence

A Horizon MachineLearning discussion has surfaced a 3D reconstruction pipeline built around only two X-ray silhouettes and 50 reference femur meshes. Its author reports millimeter-scale errors on selected held-out shapes, despite using no neural network or patient CT during reconstruction.

The more important result is less flattering. Several common correspondence methods produced surfaces far rougher than the original CT-derived geometry. ShapeWorks was the only tested method that passed the author’s predefined acceptance threshold.

That shifts the argument away from neural networks versus classical modeling. The central contest is between a compact anatomical prior and the geometric diversity found in real patients. A model can optimize its coefficients perfectly and still fail when the target anatomy lies outside its learned shape space.

The work was shared publicly, not released as a peer-reviewed study. Its numerical results have not been independently reproduced, and validation on real radiographs remains unfinished. Still, the experiment exposes where this kind of reconstruction succeeds, where it breaks, and which evidence would make it clinically meaningful.

The Project Fits a 3D Shape to Two 2D Outlines

The reported pipeline turns a constrained inverse problem into a small optimization problem, but only after substantial geometric preparation.

According to the two-view reconstruction post, the target is a patient-specific distal femur. That is the lower end of the thigh bone, including the rounded surfaces that form the knee joint.

The proposed inputs are two orthogonal silhouettes. One represents a posterior-anterior view, while the other represents a lateral view. Each image records only whether a projected pixel belongs inside or outside the bone outline.

These silhouettes discard internal density, cortical thickness, and trabecular structure. They constrain the external contour from two directions, but they do not uniquely determine every point on the hidden surface.

To manage that ambiguity, the author constructed a statistical shape model from 50 CT-derived femur meshes. A statistical shape model represents anatomical variation with a mean surface and a limited set of coordinated deformation patterns.

The model uses principal component analysis, or PCA, to compress variation across corresponding surface points. Instead of optimizing every vertex independently, the pipeline adjusts 10 shape coefficients.

Those coefficients move the mean femur along the dominant variation patterns found in the reference collection. This restriction makes the optimization manageable and prevents arbitrary, anatomically implausible deformations.

The pipeline also applies a Mahalanobis prior. This penalty measures how unusual a coefficient combination is relative to the model’s observed distribution. Large or unlikely deformations therefore become more expensive during fitting.

For every optimization step, the current 3D candidate is projected into both camera views. The projected silhouettes are compared with the target silhouettes, and the resulting loss updates the shape coefficients.

The author says Adam performs roughly 1,000 iterations. Adam is a gradient-based optimizer that adapts the update size for each parameter during repeated loss minimization.

That gradient comes from differentiable rendering. PyTorch3D’s renderer connects image-space differences back to the vertices and parameters that produced them.

Conventional rasterization makes hard decisions about whether a triangle covers a pixel. Those decisions create discontinuities that are difficult to optimize through.

Soft rasterization replaces sharp boundaries with continuous probabilities around triangle edges. A projected contour can then move gradually toward its target instead of receiving an abrupt, uninformative signal.

The author anneals the renderer’s sigma parameter during optimization. Sigma controls boundary softness, so annealing starts with forgiving gradients and ends with a sharper contour.

This approach does not train a neural network to predict anatomy. It performs patient-specific optimization against a precomputed statistical model. The reference meshes still supply prior knowledge, but there is no learned image-to-shape predictor.

That distinction matters. The project does not eliminate training data from the problem. It replaces a large supervised imaging dataset with a smaller, carefully processed collection of aligned anatomical surfaces.

The underlying idea also predates this experiment. Researchers have previously reconstructed femurs from calibrated biplanar images using statistical models and contour matching. The new element is an accessible implementation built from current open-source components.

Correspondence Became the Real Horizon MachineLearning Story

The decisive engineering problem was not rendering the femur, but deciding which surface points represented the same anatomy across 50 different bones.

PCA assumes that every input contains variables with consistent meanings. For images, the pixel grid supplies that consistency. Anatomical meshes usually do not.

Vertex 4,000 on one femur might lie near a condyle. Vertex 4,000 on another might occupy a different feature or have no meaningful relationship at all.

Mesh resolutions can differ. Triangle layouts can change. Surface sampling may cluster around some regions and become sparse around others.

A statistical model built from mismatched vertices does not learn clean anatomy. It blends unrelated locations, producing unstable deformation modes and visibly rough reconstructed surfaces.

The author first tried nearest-neighbor matching with a KD-tree. That method assigns points according to spatial proximity, but proximity does not guarantee anatomical equivalence.

Nearest-neighbor matching can fold correspondences across narrow gaps or slide them along smooth surfaces. Small local mistakes then accumulate across the cohort.

The post reports that this approach generated surface roughness 50.7 times the reference CT surface. That number is an author-reported diagnostic, not an independently established benchmark.

Coherent Point Drift, or CPD, was also tested. CPD treats one point set as centroids in a probabilistic mixture and moves them coherently toward another set.

The reported roughness remained 28.2 times the CT-derived reference. Bayesian CPD reportedly reached 47.5 times, while FilterReg did not complete successfully in the author’s setup.

These results should not become universal rankings of registration software. Implementations, initialization, preprocessing, hyperparameters, topology, and surface quality can strongly affect each method.

They do show that pairwise registration was insufficient in this particular workflow. Matching every femur independently to one template did not create a population-wide coordinate system suitable for PCA.

ShapeWorks changed that outcome. Its particle correspondence workflow optimizes landmarks across the complete shape collection rather than accepting local proximity as correspondence.

The software balances two goals. Particles should represent every individual surface accurately, and their distribution across the population should create a compact statistical model.

The post reports surface roughness 3.3 times the CT reference after using ShapeWorks. This was the only tested result below the author’s five-times acceptance gate.

That acceptance gate also deserves scrutiny. The post says it was selected before testing, which reduces one form of retrospective threshold selection. However, no clinical meaning is attached to a five-times roughness ratio.

Surface roughness is not the same as reconstruction accuracy. A smooth femur can have the wrong dimensions, and an accurate surface can retain small segmentation artifacts.

A complete evaluation needs both model-quality and patient-level measurements. Compactness, generalization, and specificity describe the statistical model. Surface distance and anatomical landmarks describe the fitted result.

Downstream measures matter too. Knee geometry affects implant planning, alignment analysis, and biomechanical simulation. Errors around a clinically important landmark may matter more than equal errors along the shaft.

The correspondence result nevertheless carries the strongest general lesson. A sophisticated optimizer cannot rescue a malformed shape space.

Better loss functions might improve silhouette agreement. More iterations might reduce the objective. Neither change can repair anatomical variables that never meant the same thing across the reference meshes.

A Small Shape Prior Challenges Data-Hungry Reconstruction

The project pressures neural reconstruction pipelines by showing that strong anatomical constraints can reduce the amount of data and supervision required.

Medical imaging teams often reach for a neural model when mapping images to geometry. That route can learn complex relationships, but it needs suitable training pairs and careful external validation.

Paired data are particularly difficult here. A useful dataset would include calibrated radiographs, high-quality segmentation, and corresponding CT-derived geometry from the same patients.

Access, consent, protocol differences, and annotation effort complicate collection. Clinical populations also vary across age, sex, disease, implants, and acquisition hardware.

A statistical model takes a different position. Human femurs vary, but they do not occupy the full universe of possible meshes. A compact prior can encode recurring anatomical relationships directly.

The project’s reference shapes reportedly came from MedShapeNet. The broader medical shape dataset was introduced with more than 100,000 annotated shapes across 23 component datasets.

That total should not be confused with the project’s femur cohort. The Reddit author says only 50 CT-derived femur meshes entered this model.

With 10 coefficients, the fitting process searches a narrow subspace rather than predicting millions of unconstrained coordinates. That makes the system more interpretable than many end-to-end models.

A coefficient can be connected to a model mode, visualized, bounded, and compared with its training distribution. The Mahalanobis term also exposes how far a candidate departs from the reference population.

This transparency does not make the result inherently safer. It makes some failure mechanisms easier to inspect.

The model’s central limitation is explicit: it can only generate anatomy represented by its mean and retained modes. It cannot invent a deformation that the shape basis excluded.

A neural model also struggles outside its training distribution. However, its representation may hide that failure behind a plausible-looking output and a confident prediction.

The statistical approach makes the boundary more visible. A coefficient reaching its allowed range is a direct warning that the model lacks coverage.

That property could support a useful rejection system. Instead of returning a reassuring mesh for every case, the software could identify patients who require CT or manual review.

Yet the small-data advantage depends on expensive upstream work. The 50 surfaces needed segmentation, cleaning, alignment, correspondence optimization, and quality control.

The project therefore exchanges one data burden for another. It avoids a massive labeled image collection, but it demands a geometrically consistent anatomical cohort.

This route also differs from reconstructing a generic object from ordinary photographs. X-ray silhouettes are projections of anatomy under calibrated imaging geometry, not color images with texture and lighting cues.

Two views constrain scale and outline better than one view, especially when the cameras are orthogonal. They still leave unseen depressions and depth variations ambiguous.

The model fills those gaps with population statistics. That is the mechanism behind both its efficiency and its risk.

A larger neural system might absorb more diverse anatomy, imaging noise, and segmentation behavior. The compact model offers clearer constraints, lower data requirements, and a narrower operating envelope.

Neither route wins in the abstract. The relevant question is whether a system recognizes when a patient falls beyond that envelope.

The Millimeter Results Hide a Coverage Failure

The promising errors apply to in-range synthetic targets, while extreme anatomies exposed failures in the model and its initial alignment.

The author reports leave-one-out testing on five held-out femurs. In each test, one target is excluded while a model is constructed from the remaining 49 meshes.

For targets within the model’s represented range, the reported errors span 0.86 to 1.43 millimeters. The public post does not provide a full protocol, per-region errors, confidence intervals, or downloadable evaluation artifacts.

Those values are technically plausible within the field. A previous distal-femur study reported submillimeter rigid-registration accuracy before evaluating shape reconstruction from calibrated biplane images.

Other research has reported roughly millimeter-scale average errors for lower-limb reconstruction under controlled biplanar conditions. Comparisons remain imperfect because datasets, metrics, calibration, and target anatomy differ.

The current project’s two extreme cases are more informative than its best numbers. The post says both targets lay outside the 49-shape model’s coverage along its first PCA mode.

Once the target passed beyond that supported range, the optimizer had no valid coefficient capable of reproducing it. Additional iterations could not create a missing anatomical direction.

This is not merely an optimizer failure. It is a representation failure.

The author also reports poor bridge ICP alignment for those cases, with an inlier fraction of 0.6. ICP, or iterative closest point, estimates alignment by repeatedly matching nearby geometry and updating a transformation.

The post says this initialization error contributed more than shape fitting to the failed reconstructions. That creates two interacting boundaries.

First, the target must be placed close enough to the model for silhouette optimization to converge correctly. Second, the model must contain enough anatomical variation to represent the target.

A pipeline can fail either test. It can also pass the image loss while returning the wrong hidden geometry.

Two silhouettes create many equivalent or near-equivalent projections. A femur can match both outlines yet differ in regions that neither view constrains strongly.

The Mahalanobis prior selects a plausible solution among those possibilities. However, plausibility reflects the 49 training shapes, not a universal patient population.

This distinction becomes important for deformity, trauma, pediatric anatomy, prior surgery, tumors, and unusual morphology. These cases may be the ones where patient-specific geometry matters most.

A model trained on relatively typical adult anatomy might perform best where reconstruction is least clinically necessary. It could reject or smooth away precisely the cases that need three-dimensional assessment.

The small validation set cannot establish subgroup performance. Five held-out surfaces are useful for debugging, but they cannot support claims about clinical reliability.

Leave-one-out testing also reuses nearly the entire cohort for every evaluation. Results across folds are therefore correlated, and the test population remains close to the development dataset.

Independent external validation would provide a harder test. It should use femurs from another institution, scanner, segmentation process, and patient population.

The published literature already treats this area cautiously. A hip shape review found that statistical models can connect radiographs with 3D anatomy, while also identifying validation and population gaps.

The correct reading of 0.86 to 1.43 millimeters is therefore narrow. It is a reported proof-of-concept result for selected, model-supported meshes under simulated conditions.

It is not evidence that two ordinary clinical X-rays can replace CT. It is not evidence for diagnostic use, surgical planning, or implant selection.

Renderer Calibration Can Overwhelm the Shape Model

A single scale-sensitive rendering parameter reportedly changed accuracy by 87 times, revealing how fragile image-space optimization can become.

Soft rasterization needs a transition width around projected triangle boundaries. If that width is too large, the silhouette becomes blurry and its loss cannot localize the true contour precisely.

If the width is too small at the start, gradients vanish away from the current boundary. The optimizer then receives little guidance when the initial projection misses the target.

Sigma annealing addresses this conflict. A broad early boundary attracts the candidate shape, while a narrow final boundary supports accurate fitting.

The author reports that the annealing endpoint had to match the sigma used to generate the reference render. A constant tuned for one shape model reportedly caused an 87-fold accuracy degradation on another.

Again, this is a claim from the public project description. The post does not provide enough artifacts to verify the ratio or isolate every contributing variable.

The proposed fix tied the endpoint to the camera extent multiplied by 0.0001. Camera extent describes the scale of the rendered scene under the chosen coordinate system.

This normalization makes conceptual sense. A softness value expressed in absolute model units will behave differently when another dataset uses a different scale.

The lesson extends beyond sigma. Camera calibration, mesh units, image resolution, projection type, clipping planes, and coordinate conventions can all influence the reconstructed geometry.

A method that works after dataset-specific tuning might fail when exported to another hospital. Different systems can encode distances in millimeters, meters, normalized coordinates, or pixel-relative units.

Synthetic experiments can conceal these mismatches because the same renderer creates the target and evaluates the reconstruction. The forward model is perfectly matched to itself.

Real radiographs break that symmetry. They include magnification, scatter, overlapping anatomy, noise, distortion, incomplete fields of view, and acquisition metadata errors.

Their contours are also less clean than binary synthetic silhouettes. The distal femur may overlap the tibia, patella, soft tissue, implants, or positioning devices.

The project currently assumes that the bone outline is already available. The author identifies automatic segmentation as unfinished work.

That missing stage is substantial. A contour error becomes a geometric instruction to the optimizer, which will deform the femur to explain the wrong pixels.

Segmentation uncertainty should therefore enter the reconstruction model. The system could weight ambiguous boundaries less heavily or propagate contour uncertainty into its final surface confidence.

Testing only perfect silhouettes cannot reveal that behavior. Controlled perturbations would help measure sensitivity to missing edges, shifted contours, and calibration errors.

A third view could add constraints because the loss can sum errors across additional projections. It would also increase acquisition complexity and depart from the project’s two-view premise.

Density information could offer more evidence than a binary outline, but that would require a more realistic image-formation model. It would also introduce exposure, material, and attenuation assumptions.

The simple silhouette formulation is attractive because it avoids those variables. Its weakness is that it throws away information before optimization begins.

The right next step is not merely lowering the synthetic surface error. It is testing whether the entire pipeline remains stable after realistic acquisition errors enter the loop.

Three Tests Will Decide Whether the Approach Travels

The project now needs paired clinical validation, broader anatomical coverage, and explicit failure detection before its reported accuracy has practical meaning.

The first signal is performance on real paired radiograph and CT data. Each patient would provide two calibrated X-rays for reconstruction and a CT-derived surface for independent comparison.

That test should keep the CT geometry outside the fitting process. It should also separate segmentation error, camera-calibration error, registration error, and shape-model error.

Per-region surface distances would be more informative than one global average. Errors near the condyles, intercondylar notch, and surgical landmarks deserve separate reporting.

The evaluation should include anatomical measurements relevant to the proposed use. A small average distance does not guarantee accurate axes, widths, angles, or implant contact regions.

Success on paired cases would strengthen the claim that synthetic silhouette fitting transfers to clinical images. A large drop would show that the current renderer solves a cleaner problem than radiography presents.

The second signal is external validation across a broader cohort. Fifty reference meshes can establish feasibility, but they provide limited coverage of demographic and pathological variation.

A larger study should disclose cohort composition, exclusion criteria, pathology, laterality, and segmentation procedures. It should also reserve a completely independent test set.

The most valuable cases will sit near or beyond the model boundary. Testing only typical anatomy would hide the exact limitation already observed in the two extreme examples.

Researchers should report how accuracy changes as more reference shapes and PCA modes enter the model. More modes can improve coverage, but they can also weaken regularization and introduce unstable deformations.

This experiment would reveal whether the current 10-coefficient model captures a durable anatomical basis or merely fits one small collection efficiently.

The third signal is an automatic rejection mechanism. The system should identify unreliable inputs before anyone uses the resulting mesh.

Possible warnings include coefficients near model limits, a high Mahalanobis distance, poor silhouette agreement, low ICP inlier fraction, or disagreement across repeated initializations.

A useful confidence score must correlate with actual 3D error on unseen patients. It cannot rely only on the optimization loss because hidden surfaces can remain wrong despite matching contours.

Prospective testing should specify rejection thresholds before examining final outcomes. That prevents favorable thresholds from being selected after failures become visible.

If these three signals arrive together, the project would become more than an interesting geometry demonstration. It would offer a testable route toward selective, patient-specific reconstruction without routine CT.

If real-image accuracy collapses, broader cohorts remain unsupported, or failures cannot be detected, the narrower conclusion will still matter. Differentiable rendering can fit silhouettes, but it cannot supply anatomical evidence that the shape model never contained.

For developers following the Horizon MachineLearning thread, the immediate challenge is reproducibility. The field needs code, fixed data splits, camera parameters, correspondence settings, and complete error distributions.

For medical teams, the question is stricter: can the method recognize the patient it should not reconstruct? That answer matters more than its best millimeter result.