a new benchmark for supervised particle pose estimation in Cryo-EM. (2024)

1.1 Cryo-EM Single Particle Analysis

Determining the structure of macromolecules is crucial to deciphering the intricacies of biological processes and the underlying mechanisms of diseases. With the advent of the resolution revolution, Cryogenic Electron Microscopy (Cryo-EM) has emerged as a leading technique for elucidating structures [36, 6]. This revolution, driven by significant advances in direct electron detectors and image processing algorithms, has made Cryo-EM a routine, often unrivaled, method for many complex samples [12]. Its advantages include, among others, the relative ease of sample preparation compared to other techniques (e.g., x-ray crystallography), the capability to analyze protein complexes previously considered out of reach, and the ability to recover different conformations, offering a dynamic view of molecules in action [37]. The pivotal role of Cryo-EM in structural biology was globally recognised in 2017 when the technique was awarded the Nobel Prize in Chemistry.

The primary aim of Cryo-EM Single-Particle Analysis (SPA) is to reconstruct the three-dimensional (3D) structure of a given macromolecule at near-atomic resolution, ideally better than 3 Å. This process uses electron beams to capture thousands of two-dimensional (2D) images of the macromolecules, which are flash frozen in vitreous ice to preserve their native state without the distortions typical of crystalline ice or other fixation methods [38]. Each image, called a micrograph, can display several hundred snapshots of the macromolecule (referred to as particle images or just particles) in unknown random orientations. If the orientations of these images were known, the reconstruction task would closely resemble the algorithms used in tomography, which reconstruct 3D volumes from 2D projections taken at predetermined angles [18]. However, the unknown orientations of the particles in SPA present a unique challenge not encountered in tomography [3]. Compounding this challenge is the inherently low contrast and extremely poor signal-to-noise ratio (SNR) of the images, a consequence of the delicate biological nature of the samples. Given these challenges, a highly sophisticated image processing pipeline is essential to accurately resolve the 3D structure of the macromolecule [32, 50].

The fundamental principle of image processing in SPA is grounded in the intuitive strategy of employing averaging to mitigate noise. Since images are characterised by a low SNR, averaging multiple images of the same particle, assumed to be identical, can significantly enhance the underlying signal [37]. However, before averaging can be effectively carried out, each particle projection must be aligned to a common orientation. This ensures that the differences observed across the images are solely due to noise, allowing its effective cancellation during the averaging process.

The standard Cryo-EM image processing pipeline encompasses several key steps, beginning with various preprocessing operations to correct errors such as beam-induced movement blur, followed by particle picking, which extracts the individual particle images from the micrographs [32, 50]. Subsequent stages include, among others, clustering (commonly referred to as 2D classification in the context of Cryo-EM) and particle alignment against references, leading to a cleaner subset of the data and an initial low-resolution 3D volume of the protein. This preparatory work sets the stage for the refinement step, a critical phase where the poses of the particles are precisely estimated, a requirement to achieve the high-resolution volumes needed to reveal atomic level details.

Traditional refinement algorithms perform pose estimation by exhaustive comparison of experimental particle images and simulated projections of 3D volumes that are iteratively improved [7, 16, 43, 46, 49]. When sample hom*ogeneity can be assumed, the simplest approach to the pose estimation problem is the projection matching algorithm [41], which consists of $T$ iterations of two steps: alignment and reconstruction. First, in the alignment phase, the pose $(R,s)_{i}\in\mathrm{SO(3)}\times\mathbb{R}^{2}$ of each experimental particle image $x_{i}$ is set to be the same as the one of the most similar 2D projection of the reference volume $V^{t}$ at iteration $t$,

where $P_{(R,s)}$ is the projector operator, $f_{i}$ is the point spread function of the microscope for the $i$-th particle and $*$ the convolution operator. Then, in the reconstruction phase, a new volume (in reciprocal space) is computed from the estimated poses as

with $\hat{V}$ being the Fourier transform of the volume $V$, $C_{i}$ a constant depending on the SNR, $\hat{f_{i}}$ the Fourier transform of the point spread function (CTF, contrast transfer function) and $N$ the number of particles. This iterative process continues until convergence.State-of-the-art methods build on this approach, for example, Relion [46] employs a Bayesian probabilistic model with a prior for the map, making it much more robust. CryoSPARC [43] accelerates Bayesian methods through branch-and-bound search and gradient descent optimisation. See [3] for a review.

Despite the innovations aimed at enhancing efficiency, the refinement process still poses significant computational challenges. The primary factor contributing to these challenges is the large number of image comparisons required for each experimental image. Furthermore, the iterative refinement of the volumes, beginning with an initial low-resolution model and progressively improving it, further increases the computational cost, making the refinement stage the most computationally intensive step in the Cryo-EM workflow.

1.2 Deep Learning for Pose Estimation in Real-World Objects

Similar to refinement algorithms in Cryo-EM, traditional pose estimation techniques for real-world images primarily focus on matching 2D images with 3D objects. The significantly higher SNRs characteristic of real-world images enable the use of more sophisticated and efficient methods beyond simple template matching. Among these, landmark-based registration methods are particularly prevalent. Such methods involve extracting distinctive landmarks through various feature extraction techniques [30, 2], followed by a registration process to identify the relative orientation of the landmarks in the image with respect to the reference landmarks [5] .

PoseNet [23] was a groundbreaking development in this field, leveraging a Convolutional Neural Network (CNN) to directly regress the absolute pose of an object using quaternions and $xyz$ shifts. This direct approach contrasts with earlier techniques that relied heavily on feature extraction and landmark identification, allowing for end-to-end pose estimation. Subsequent innovations have built on the foundation laid by PoseNet. Improvements in network architectures [33], the introduction of more sophisticated loss functions [22], and the incorporation of multitask learning [52] have contributed to significant improvements in pose estimation performance.

Addressing the inherent challenges of symmetry and occlusion in pose estimation has also seen considerable progress through deep learning. Strategies have evolved from breaking symmetry during the data labelling process [51] to implementing loss functions specifically designed to accommodate known symmetries [52]. Probabilistic models offer alternative approaches that either classify poses within a discretised space or explicitly learn the parameters of probability distributions [8, 31, 34, 35, 42]. Due to their probabilistic nature, these models are better suited for challenging datasets with high levels of ambiguity or noise.

1.3 Deep learning methods for Cryo-EM structure determination or pose estimation

While traditional Cryo-EM refinement algorithms tend to be relatively robust and accurate, they are computationally intensive and slow. In an attempt to overcome this, deep learning (DL) alternatives have begun to emerge.

Unsupervised DL methods aim to determine the 3D structure of macromolecules from experimental images alone. Some of them tackle the problem using a distance learning approach in which the angular distance between pairs of images is estimated as a preprocessing step to retrieve their relative poses [1]. Other unsupervised DL methods mirror traditional techniques by maintaining a 3D volume representation to compute 2D projections in a differentiable manner [11]. Unlike traditional refinement methods, which compare each experimental particle against all images in an SO(3) projection gallery with up to millions of members, these methods try to limit the number of comparisons between experimental images and projections. For instance, in CryoGAN a 3D volume, randomly initialised, serves as the generator in the Generative Adversarial Network (GAN) framework [14]. This generator produces a set of projections from random orientations that are then fed to a discriminator network along with real experimental images. The objective of the training process is to refine the generator until the discriminator can no longer distinguish between the generated projections and the actual experimental images, effectively capturing the underlying 3D structure present in the experimental data. In some other approaches [27, 26] , particle images are first processed by an encoder designed to predict particle orientations. Following this prediction, a projection of the representation of the volume corresponding to the inferred orientation is rendered. This projection is then directly compared to the original experimental particle image. A loss function is utilized to concurrently refine both the encoder’s parameters and the representation of the volume, improving the accuracy of orientation predictions and the fidelity of the reconstructed volume.

Supervised DL models, on the other hand, are trained using experimental images and some form of (possibly noisy) labels, such as the poses of prealigned sets of particles. The simplest alternative consists of only an encoder module that predicts the orientation of the particle directly from its image [21, 28]. Although supervised approaches offer remarkable efficiency and speed, they require labelled data for training, thus limiting their applicability in de novo situations. However, there are use cases where supervised DL methods could offer and advantange. For instance, it should be possible to apply them to on-the-fly pipelines in which a first batch of particles is pre-aligned before the end of the data stream. This initial alignment could be used to train a supervised model to be applied to subsequent batches of data, inferring their poses in real time. Even more interestingly, a pre-trained supervised model could be used to infer poses in different projects, providing the new samples are similar to the training data. This second use case relies on the fact that pose estimation in classical methods is mainly driven by low- to mid-resolution frequencies [47]. As similar proteins have similar low- to mid-resolution frequencies, trained models are expected to generalise to these new samples. In addition, because ligand binding does not generally modify the overall shape of proteins, supervised approaches can be especially valuable in drug discovery, where pre-aligned data for target proteins is often available.

In the context of Cryo-EM, only two supervised methods have been proposed to perform direct pose estimation given prealigned particles. DeepAlign [21], a set of CNNs that perform binary classification over a discretisation of $S^{2}$, and the approach of Lian etal. [28] who implemented a CNN to perform direct regression of quaternions. However, due to its limitations, especially for symmetric data, Lian et al. finally adopted a hybrid model with a projector as in some Cryo-EM Unsupervised estimators. Neither of the two methods has been used in practical scenarios.

While much slower, classical refinement methods still outperform DL pose estimation models in terms of performance and reliability. This gap can be partly attributed to the unique characteristics of Cryo-EM data, which differ from the natural images DL artchitectures were designed for and, importantly, to the lack of a standardized benchmark that would allow for a direct comparison of methods in order to stimulate progress, much as ImageNet [9] did for image classification. In this paper, we introduce CESPED (Cryo-EM Supervised Pose Estimation Dataset), a benchmark specifically designed to evaluate supervised pose estimation methods. As the first benchmark dedicated to pose estimation in Cryo-EM, CESPED addresses a crucial gap in the array of available datasets, which have, until now, primarily focused on other Cryo-EM challenges, such as model building [13] and particle picking [17, 10]. CESPED aims to foster advancements in DL methods for particle processing by promoting improvements in supervised pose estimation models, which, due to shared architectural building blocks and data challenges, are likely to benefit methods for related tasks as well.

1.4 Main contributions.

2.1 Benchmark Compilation and preprocessing

In our effort to build a comprehensive benchmark, our primary goal was to identify a diverse set of EMPIAR entries containing at least 200,000 particles, a number deemed sufficient for effective model training. Due to the limitations of EMPIAR’s search functionality and the inconsistencies in dataset annotations, we conducted a manual search for entries exceeding this particle count and containing standard Relion files ( .star and .mrcs). Subsequently, we verified the consistency and accuracy of the metadata by running relion_reconstruct [46] and visually assessing the resulting volumes. This step was crucial for eliminating a significant number of entries due to metadata issues that either crashed the reconstruction process or led to incorrect volumes. To ensure consistent estimation of particle poses, the data was re-processed using the Relion version 4 auto-refine program [46, 24] (see Appendix Appendix A for details). Only entries for which the reconstructed volume exhibited resolution values close to those reported in the literature were selected for inclusion in the benchmark. Finally, for consistency, all images were downsampled to $1.5$ Å/pixel, with different image dimensions in each entry as macromolecules vary in size. See Appendix A for a list of the entries and their properties.

The images fed to the deep learning model were preprocessed on-the-fly. We performed per-image normalisation following the standard Cryo-EM procedure, which involves rescaling the intensity so that the background (noise) has a mean of 0 and a standard deviation of 1. We also corrected the contrast inversion caused by the defocus via phase flipping [40]. Finally, since the macromolecule typically represents only between 25% to 50% of the whole particle image, the images were cropped so that neighbouring particles are not included. It is important to note that our benchmark package allows users the flexibility to choose whether or not to apply any of these normalisation steps.

The data labels are represented as rotation matrices and then converted into grid indices by finding the closest rotation matrix in the $\mathrm{SO(3)_{grid}}$. For the cases in which the macromolecule exhibits point symmetry, the labels are expanded as $L_{i}={\{g_{j}R_{i}|g_{j}\in G}\}$ where $G$ is the set of rotation matrices given a point symmetry group (e.g., $C1$), and $R_{i}$ the ground truth rotation matrix. As a result, the labels consist of vectors with $|G|$ non-zero values and $|\mathrm{SO(3)_{grid}}|-|G|$ zeros.

2.2 Baseline model

We adapted the state-of-the-art Image2Sphere model [25]. Image2Sphere is a hybrid architecture that uses a ResNet to produce a 2D feature map of the input image, which is then orthographically projected onto a 3D hemisphere and expanded in spherical harmonics. Then, equivariant group convolutions are applied, first with global support on the $S^{2}$ sphere, and finally as a refinement step, on SO(3). The output of the model is a probability distribution over a discretised grid of rotation matrices. Other supervised Cryo-EM methods for pose estimation were not considered for this work due to the lack of publicly available code [28] or their GUI requirements [21].

2.3 Evaluation metrics

The most widely used metric in pose estimation is the mean angular error ($\mathrm{MAnE}$), averaged across all poses,

The angular error ($\mathrm{angError}$) measures the geodesic distance between the predicted and ground truth poses, typically expressed in degrees or radians. This distance can be directly calculated from the rotation matrix of the ground truth pose $\mathrm{trueR}_{i}$ and the predicted rotation matrix $\mathrm{predR}_{i}$ as

When evaluating predicted orientations of macromolecules exhibiting point symmetry, it is necessary to adjust the angular error, as several rotation matrices become equivalent. In this context, the angular error is defined as the minimum geodesic distance between the predicted orientation and any orientation equivalent to the ground truth under the molecule’s symmetry group

with $G$ being the set of rotation matrices given a point symmetry group.

which weights the $\mathrm{angError_{i}}$ by $\mathrm{conf_{i}}$, the confidence in the ground truth pose, measured as Relion’s rlnMaxValueProbDistribution. This confidence estimation is a number between 0 and 1 that measures the probability of the particle having the reported ground truth orientation according to the Relion model. While $\mathrm{wMAnE}$ is still sensitive to ground truth and confidence estimation errors, due to its simplicity, we used it as the criterion for hyperparameter tuning.

The quality of volumes reconstructed from the predicted poses is assessed by comparing them with the ground truth volumes generated from the original poses (see Appendix H). For this comparison, we employ the real space Pearson’s Correlation Coefficient (PCC) and the Fourier Shell Correlation (FSC) Resolution as metrics.

The Pearson’s correlation coefficient is a value between -1 and 1, where values closer to 1 indicate a higher similarity. It measures the linear correlation between the pixels of the two volumes as follows:

with $X_{i}$ and $Y_{i}$ being the pixel $i$ of the two volumes, $n$ the number of voxels, and $\bar{X}$ and $\bar{Y}$ the average value of the volumes.

The FSC quantifies the correlation between two signals at different spatial frequencies. For each frequency $k$, a value between -1 and 1 (with higher values indicating greater similarity) is computed by comparing the concentric shells in the Fourier transforms of the two volumes corresponding to $k$:

where $\hat{X}(\mathbf{r})$ and $\hat{Y}(\mathbf{r})$ represent the Fourier transforms of the two volumes at frequency $\mathbf{r}$, $\text{shell}(k)$ is the shell of frequency $k$ and $\hat{Y}^{*}(\mathbf{r})$ is the complex conjugate of $\hat{Y}(\mathbf{r})$.

To summarize the FSC curves into a single number, the FSC Resolution is computed by selecting a threshold $t$ and identifying the highest frequency $k$ such that $\text{FSC}_{k}(X,Y)<t$. As thresholds we employ the commonly used values of 0.5 ($\mathrm{FSCR_{0.5}}$) and 0.143 ($\mathrm{FSCR_{0.143}}$), which corresponds to the highest frequency at which the two maps agree with an SNR of 1 and 0.5 respectively [44].

In order to decouple the different quality levels of the different benchmark entries, we report the differences of the metrics with respect to the ground truth levels, estimated from the half-maps of the ground truth,

2.4 Training

Each benchmark entry was trained independently with the same hyperparameters (see Appendix B). Due to the uncertainty in the estimated orientations, we employed a weighted cross-entropy loss using the pose reliability estimate of each particle as the per-image weight

where $Q(R_{c,i})$ is the predicted probability for the rotation matrix with grid index $c$ and $P(R_{c,i})$ is $1/|G|$ when any of the ground truth matrices is $c$ and zero otherwise.

2.5 Evaluation protocol

Due to the uncertainty in the ground truth labels and the fact that what matters to Cryo-EM practitioners is the quality of the reconstructed volume, we devised an evaluation protocol inspired by the Cryo-EM gold standard [20], which is a per-entry 2-fold cross-validation strategy in which the poses of each half of the data are independently estimated and used to reconstruct two volumes (half-maps). For benchmarking supervised methods, it involves training an independent model for each half of the dataset to infer the poses of the other half of the dataset. After that, the final 3D volume is computed by reconstructing the two half-maps and averaging them. The final averaged map can then be compared with the ground truth map obtained from the original poses (see Figure 1). It is important to note that the FSC resolution values derived from this comparison are analogous to map-to-model FSC resolution estimations and not equivalent to the gold standard half-to-half resolution.

Since Image2Sphere predicts only rotation matrices but not image shifts, when reconstructing the volumes, we employed the ground truth translations. This could result in an overoptimistic estimation of performance, however, since the effect of the translations is tightly coupled with the accuracy of the angular estimation, this overestimation should be small. We leave for future work the full inference of both the rotations and translations. Finally, to avoid overfitting to the validation set, we performed hyperparameter tuning only one half of the data using $\mathrm{wMAnE}$ as a metric.

3.1 Benchmark, ParticlesDataset class and evaluation tool

Our benchmark consists of a diverse set of eight macromolecules, with an average number of 300K particles including soluble and membrane macromolecules, symmetric and asymmetric complexes, and resolutions ranging from $5\text{\,}\mathrm{\SIUnitSymbolAngstrom}$ to $3.2\text{\,}\mathrm{\SIUnitSymbolAngstrom}$ (see Appendix A). For each particle in the dataset, we provide its image and estimated pose together with an estimate of the reliability of the poses. The benchmark can be automatically downloaded from Zenodo [53] using our $\mathrm{cesped}$ Python package.

The package includes a ParticlesDataset class, which implements the PyTorch Dataset API for seamless integration. It also offers optional yet recommended preprocessing steps commonly adopted in Cryo-EM (e.g., image normalisation, phase flipping), and specialised data augmentation techniques, like affine transformations that adjust both the image and its corresponding pose (see Appendix B). While the $\mathrm{cesped}$ package was designed with PyTorch in mind, the benchmark is accessible to a broader audience, as the data is stored in standard formats and accompanied by utility programs to assist users of other frameworks in adopting the CESPED benchmark.

Additionally, the package offers an automatic evaluation pipeline that only requires as inputs the predicted poses (grey box in Figure 1). For ease of use, a Singularity¹¹1https://zenodo.org/records/4667718 image definition file is included, eliminating the need to install Cryo-EM-specific software like Relion. This design enables those without Cryo-EM experience to utilize the cesped benchmark and package as effortlessly as they would with standard datasets such as MNIST. Usage examples can be found in Appendix C.

3.2 Performance of the baseline model on the benchmark

Table 1 summarises the results of the Image2Sphere [25] model on our benchmark, with per-entry results in Appendix D. While the $\mathrm{wMAnE}$ is $\sim$ 24°, for the best cases, the error is as small as 9°. The $\Delta\mathrm{PCC}$ for the worse cases is > 0.1, highlighting that, for some entries, the reconstructed volumes are far from the ground truth solution. For a few cases, the results are much better, with $\Delta\mathrm{PCC}$ < 0.03. In terms of prediction vs ground truth $\mathrm{FSCR_{0.5}}$, most maps are in the 8-6 Å range, with $\Delta\mathrm{FSCR_{0.5}}$ of 3.6 Å. However, the $\mathrm{FSCR_{0.143}}$ values between 4-5 Å, indicate better correlation at lower signal levels. This visually translates into a relatively well-resolved central part of the map that becomes blurrier away from the centre (see Figure 2 andAppendix E). For the top-performing cases, a simple and fast local refinement of the predicted poses is sufficient to obtain high-resolution reconstructions comparable to ground truth volumes, at a computational cost threefold less than global refinement (Appendix F). Since the Image2Sphere model inference takes only minutes, far less than the hours needed for traditional refinement, further improvements could reduce computing times by at least one order of magnitude if local refinement is no longer needed (see Appendix G for running times).

Given the inherent difficulties of Cryo-EM data, the fact that a generic pose estimation model can produce meaningful results in some examples without major modifications suggests that equivariant architectures can be useful for the Cryo-EM data domain.

3.3 Example of model generalisability across samples

One of the main potential applications of Supervised Pose Estimation models is to infer poses on similar, yet different projects. In this section, we illustrate this use case by using an Image2Sphere model trained on the EMPIAR-10280 dataset, to predict poses of the same protein under different experimental conditions (EMPIAR-10278 dataset).

Figure 3 showcases three reconstructed volumes: (1) EMPIAR-10278 using ground truth poses (grey); (2) EMPIAR-10278 with poses predicted by the model trained on EMPIAR-10280 (yellow), illustrating the model’s generalisability; and (3) EMPIAR-10280 using poses inferred by the model trained on its own dataset, serving as a control for model performance. As expected, the EMPIAR-10278 map reconstructed with original poses shows superior quality compared to the others. Similarly, the EMPIAR-10280 map generated from the model trained on EMPIAR-10280 exhibits better quality than the EMPIAR-10278 map inferred using the EMPIAR-10280 model, reflecting the differences between the two datasets despite containing the same protein. Independently of these quality differences, the model’s capacity for generalization across datasets is evident through visual inspection of the EMPIAR-10278 inferred map (yellow), as the overall shape of the protein and several key secondary structure elements are clearly recognizable. This suggests that further improvements in the model could lead to the desired goal of training the model once and then inferring the poses of similar datasets at much faster speeds.

3.4 Challenges and future directions

Cryo-EM particle images are fundamentally different from the kinds of images encountered in other fields. One of the most critical challenges is their poor SNR, which can be as low as $0.01$ [3]. While some methods have tried to mitigate this issue by applying filtering techniques [26] or using CNNs with larger kernel sizes [4, 21, 45], these solutions are not entirely effective.

Symmetry presents another complex facet of Cryo-EM data. Exploiting symmetry can drastically reduce the computational requirements for pose estimation, but it can also prevent simple models from learning. The unique combination of rotationally equivariant convolutions with the probabilistic estimation of poses makes the Image2Sphere model an ideal candidate to exploit this feature. However, the hybrid $S^{2}$ / SO(3) formalism means that the separation of rotational degrees of freedom from translational in-plane shifts is not easily achieved within this framework. A significant area for future work lies in leveraging rotational equivariance and translational equivariance for the joint estimation of the rotational and translational components of the poses (e.g. SE(3) equivariance).

In this work we have considered only the case of hom*ogeneous refinement, which assumes that all particles are projections from a single macromolecule in a unique conformation. However, this is not always the case and our benchmark could potentially be extended to deal with such examples. Models would then need to perform conformation classification alongside pose estimation.