13 Image analysis and models of cells
This topic is not part of the 2025 exam description. It was added in version 0.4 of this book because image-based methods, and the large projects that aim to classify cell types and build models of whole cells, have become a central part of bioinformatics. Ask your examiner whether it is included in your exam.
13.1 Overview
Images are now one of the largest sources of biological data. Automated microscopes image millions of cells in a single screen, histology slides are scanned at high resolution in hospitals, and electron tomography shows the molecules inside cells. Deep learning has made it possible to analyse such images at scale, and the results are increasingly combined with sequence, expression and structure data. This chapter builds on the machine-learning methods of Chapter 12 and covers:
- images as data: kinds of images, classical analysis, software, file formats and repositories (Section 13.2);
- deep learning for images: classification and segmentation, U-Net, Cellpose and StarDist, vision transformers, self-supervised learning and foundation models, and the pitfalls (Section 13.3);
- image-based profiling: Cell Painting, morphological profiles and the JUMP and RxRx data sets (Section 13.4);
- protein localisation from images: the subcellular atlas of the Human Protein Atlas, image-classification challenges, OpenCell and self-supervised models (Section 13.5);
- tissues: multiplexed imaging, spatial omics and computational pathology (Section 13.6);
- cell atlases and AI virtual cells: the Human Cell Atlas, HuBMAP, the virtual-cell efforts of CZI, the Arc Institute and SciLifeLab’s Alpha Cell, and how well current models work (Section 13.7);
- mechanistic and structural models of whole cells, and cryo-electron tomography (Section 13.8).
The chapter connects to many others: spatial proteomics and the Human Protein Atlas in Chapter 8, sequence-based localisation prediction in Chapter 10, single-cell and spatial transcriptomics in Chapter 7, and cryo-EM and molecular dynamics in Chapter 9.
13.2 Images as data
Microscopy is the oldest way of studying cells, and for most of its history the results were judged by eye. Modern microscopes produce far more images than anyone can inspect: an automated screen can image millions of cells, and a single light-sheet experiment can produce terabytes. Bioimage analysis turns these images into measurements that can be analysed with the same statistical and machine-learning methods as other omics data.
13.2.1 What an image is
A digital image is an array of numbers. A fluorescence image has one channel for each fluorescent label, and may in addition have several focal planes (z) and time points (t), so that a single experiment is a five- dimensional array (x, y, z, channel, time). Each value is an intensity, not a count of molecules: it depends on the label, the microscope, the exposure and the detector, and images are only comparable when they were acquired in the same way. The main kinds of images in biology are:
- Transmitted-light (brightfield, phase contrast) images, which need no labels but show little molecular detail;
- fluorescence images (widefield, confocal, light-sheet and super-resolution microscopy), where proteins or structures are made visible with antibodies, dyes or fluorescent proteins;
- electron microscopy, which reaches molecular resolution but only for fixed or frozen samples (Section 13.8);
- histology, thin tissue sections stained, for example with haematoxylin and eosin (H&E), and scanned as whole-slide images of up to 100,000 × 100,000 pixels (Section 13.6).
13.2.2 Classical image analysis
The classical workflow, still used in many pipelines, has four steps: preprocessing (background and illumination correction, denoising), segmentation (finding the objects, usually nuclei and cells, for example by thresholding followed by a watershed transform to split touching objects), feature extraction (measuring size, shape, intensity and texture for each object) and analysis of the resulting table. Each step involves choices, and the results depend on them; this is why image analysis is often less reproducible than it appears.
13.2.3 Tools, formats and repositories
Most bioimage analysis uses open-source software. ImageJ and its distribution Fiji are the general-purpose tools of light microscopy, extended by thousands of plugins (Schindelin et al. 2012; Rueden et al. 2017). CellProfiler builds analysis pipelines from modules without programming and is the standard tool for high-throughput screens (Carpenter et al. 2006; Stirling et al. 2021). ilastik trains pixel and object classifiers interactively from a few user annotations (Berg et al. 2019). QuPath is designed for whole-slide pathology images (Bankhead et al. 2017), and napari is a multidimensional image viewer for Python, to which many deep-learning tools are added as plugins.
Images need metadata (organism, sample preparation, microscope and acquisition settings) to be reusable. The REMBI recommendations define the minimum metadata for publishing biological images (Sarkans et al. 2021), and the OME-Zarr format (OME-NGFF) stores large multidimensional images in chunks that can be read directly from the cloud (Moore et al. 2021, 2023). Published images are deposited in the BioImage Archive at EMBL-EBI (Hartley et al. 2022), and curated reference data sets, linked to genes, compounds and phenotypes, in the Image Data Resource (IDR) (Williams et al. 2017). The Broad Bioimage Benchmark Collection (BBBC) provides annotated image sets for testing methods (Ljosa et al. 2012), and the Cell Tracking Challenge has benchmarked segmentation and tracking methods for more than ten years (Maška et al. 2023). The volume of data is part of the challenge: unlike sequences, images cannot be compressed to a few bytes per measurement, and moving them is often harder than analysing them.
See also: further explanation in Section D.13.1; related Wikipedia articles in Section E.13.1.
Background from the Wikipedia articles Microscopy, Fluorescence microscope, Image segmentation and Fiji (software) (CC BY-SA 4.0, retrieved 5 October 2026).
13.3 Deep learning for images
Image analysis is where modern deep learning started, and microscopy was one of the first scientific fields to adopt it (Moen et al. 2019). The general methods are described in Section 12.4; this section describes how they are used for biological images.
13.3.1 Tasks
Four tasks cover most applications, and it is worth keeping them apart:
- classification assigns a label to a whole image or to a cell (for example a cell type, a cell-cycle phase or the location of a protein);
- segmentation assigns each pixel to a class (semantic segmentation, e.g. nucleus vs. background) or to an individual object (instance segmentation, which separates touching cells);
- detection and tracking find objects, such as cells or particles, and follow them over time;
- image-to-image methods restore noisy images or predict one kind of image from another, for example fluorescence from brightfield.
13.3.2 Convolutional networks and U-Net
Convolutional neural networks (CNNs) learn filters that detect local patterns and combine them into increasingly abstract features. The breakthrough came in 2012, when AlexNet won the ImageNet challenge by a large margin (Krizhevsky et al. 2017), and residual connections later made networks with more than a hundred layers trainable (He et al. 2016). For biomedical images, the most influential architecture is U-Net (Ronneberger et al. 2015). An encoder reduces the image to coarse features, a decoder expands them back to full resolution, and skip connections pass fine detail from the encoder to the decoder, so that the network produces precise pixel-wise output from relatively few training images. A U-Net plugin for ImageJ made the method usable without programming (Falk et al. 2018).
13.3.3 Instance segmentation of cells
Separating touching cells is the central problem of cell segmentation. StarDist describes each nucleus as a star-convex polygon: for every pixel, a CNN predicts whether it belongs to an object and the distances to the object boundary along a fixed set of rays, and overlapping candidates are removed by non-maximum suppression (Schmidt et al. 2018). Cellpose predicts, for every pixel, the direction towards the centre of its cell (a flow field), and groups pixels whose flows end at the same point. Trained on a diverse collection of images, it segments many kinds of cells without retraining (Stringer et al. 2020), and Cellpose 2.0 adds human-in-the-loop training, so that a model can be adapted to new data from a few corrected images (Pachitariu and Stringer 2022). For tissue images, Mesmer was trained on TissueNet, more than a million manually annotated cells, and reached the accuracy of human annotators (Greenwald et al. 2021).
13.3.4 Transformers, self-supervision and foundation models
Vision transformers (ViTs) split an image into patches and treat them as a sequence of tokens, as language models do with words (Dosovitskiy et al. 2021). They need more training data than CNNs, which has made self-supervised learning important. In DINO, a student network learns to match the output of a teacher network on different views of the same image, without labels (Caron et al. 2021), and DINOv2 scaled this to general-purpose visual features (Oquab et al. 2023). Masked autoencoders (MAE) hide most of the patches of an image and learn to reconstruct them (He et al. 2022), the image counterpart of masked language modelling (Section 12.4). For microscopy, Cell-DINO showed that self-supervised features of fluorescence images classify protein localisation well with few labels (Moutakanni et al. 2025).
The Segment Anything model (SAM), trained on more than a billion masks, segments objects from a prompt such as a point or a box (Kirillov et al. 2023). It was not trained on microscopy, but versions fine-tuned for light and electron microscopy, such as μSAM (Archit et al. 2025) and CellSAM (Marks et al. 2025), are now general segmentation tools. These are examples of foundation models (Section 12.6) for images.
13.3.5 Label-free prediction
Networks can also learn to predict one kind of image from another. In silico labelling predicted fluorescent labels of nuclei, cell types and dead cells from transmitted-light images of unlabelled cells (Christiansen et al. 2018), and similar models from the Allen Institute predicted the three-dimensional fluorescence of several cellular structures from brightfield images (Ounkomol et al. 2018). Such predictions are useful, but they can only reproduce what is predictable from the input image, and they should not be mistaken for measurements.
13.3.6 Pitfalls
The general pitfalls of machine learning (Section 12.2) all apply, and some are specific to images. Images from different plates, microscopes, days or hospitals differ systematically (batch effects), and a network may learn these differences rather than biology; training and test sets should therefore be split by experiment, not by image. Ground-truth annotations are produced by people, who disagree with each other. Restored or predicted images can contain plausible structures that are not in the sample. Guidelines for validating and reporting deep-learning image analysis address these problems (Laine et al. 2021), and good introductions for biologists are available (Chamier et al. 2019; Hallou et al. 2021). Notebooks such as ZeroCostDL4Mic let users train common models without installing software (Chamier et al. 2021).
See also: further explanation in Section D.13.2; related Wikipedia articles in Section E.13.2.
Background from the Wikipedia articles Convolutional neural network, U-Net and Vision transformer (CC BY-SA 4.0, retrieved 5 October 2026).
13.4 Image-based profiling
A cell’s appearance reflects its state. Image-based profiling (also called morphological profiling or phenomics) uses this systematically: cells are treated with chemical or genetic perturbations, imaged, and described by thousands of measurements, so that perturbations can be compared in the same way as expression profiles (Chapter 7).
13.4.1 Cell Painting
The most widely used assay is Cell Painting (Gustafsdottir et al. 2013; Bray et al. 2016). Six fluorescent dyes, imaged in five channels, label eight cellular components: the nucleus (DNA), nucleoli and cytoplasmic RNA, the endoplasmic reticulum, actin, the Golgi apparatus, the plasma membrane and mitochondria. The assay is cheap, needs no antibodies, and works in many cell types. Software such as CellProfiler segments each cell and measures its size, shape, intensity and texture in every channel, giving more than a thousand features per cell (Section 13.2). A third version of the protocol was optimised quantitatively by the JUMP consortium (Cimini et al. 2023).
13.4.2 From images to profiles
The analysis follows a standard workflow (Caicedo et al. 2017):
- Single-cell features are extracted, either hand-designed features from CellProfiler or embeddings from deep networks (Section 13.3).
- Features are aggregated per well, usually as the median over cells, normalised against negative controls on the same plate, and reduced to a smaller set of informative, uncorrelated features.
- Batch effects between plates and laboratories are corrected, the same problem as in expression data (Section 7.4).
- The profiles are compared: perturbations are tested for whether they change the cells at all, and similar profiles are matched, for example by cosine similarity, to group compounds by mechanism or to connect a gene to a compound that mimics its loss.
Profiles are used to find the mechanism of action of compounds, to identify toxicity, to group genes into pathways, and to find disease phenotypes that a drug reverses.
13.4.3 Large public data sets
The JUMP Cell Painting Consortium, a collaboration of the Broad Institute and pharmaceutical companies, has released Cell Painting images of more than 100,000 chemical and genetic perturbations. A pilot data set, CPJUMP1, contains about three million images of cells treated with matched chemical and genetic perturbations of the same genes, designed as a benchmark for methods (Chandrasekaran et al. 2024). A morphological map of the over- and underexpression of more than 15,000 genes in human cells followed (Chandrasekaran et al. 2025). The company Recursion has released RxRx3, images and embeddings of more than 17,000 gene knockouts and 1,674 compounds, with some identities hidden (Fay et al. 2023). These data sets are large enough to train foundation models of cell morphology, and to test whether such models can predict the effects of perturbations that were not measured (Section 13.7).
See also: further explanation in Section D.13.3; related Wikipedia articles in Section E.13.3.
Background from the Wikipedia article High-content screening (CC BY-SA 4.0, retrieved 5 October 2026).
13.5 Protein localisation from images
Where a protein is in the cell is part of its function. Section 10.5 describes how localisation is predicted from sequence; this section describes how it is measured and classified from images, which also provides the training data for the sequence-based predictors.
13.5.1 The subcellular atlas of the Human Protein Atlas
The subcellular section of the Human Protein Atlas (Section 8.6) is based on immunofluorescence: each protein is stained with antibodies in cell lines, together with markers for the nucleus, microtubules and the endoplasmic reticulum, and imaged by confocal microscopy. The first version mapped 12,003 proteins to 30 subcellular structures and showed that about half of all proteins are found in more than one location (Thul et al. 2017). Many proteins also vary between individual cells of the same cell line; about a fifth of the proteome shows such cell-to-cell variability, and only part of it is explained by the cell cycle (Mahdessian et al. 2021).
13.5.2 Image classification as a community challenge
Annotating localisation by eye does not scale, so the HPA organised two open Kaggle competitions. In the first, more than 2,000 teams classified images into 28 location classes, with up to six labels per image. The best models clearly outperformed the earlier automated method, but did not reach the accuracy of human experts (Ouyang et al. 2019). The second competition was weakly supervised: labels were given for whole images, and the models had to predict the localisation of each individual cell (Le et al. 2022). The two competitions illustrate several general issues of image classification: multi-label output, rare classes, the choice of evaluation metric (macro-F1 gives each class equal weight) and labels that are themselves uncertain.
13.5.3 Endogenous tagging and self-supervised models
Antibodies can bind other proteins than their target. An alternative is to tag the endogenous gene with a fluorescent label. OpenCell, at the Chan Zuckerberg Biohub, tagged more than a thousand human proteins with CRISPR, imaged them in living cells and identified their interaction partners by mass spectrometry; proteins with similar localisation patterns often interact (Cho et al. 2022). Using these images, cytoself learned a representation of localisation without any labels, by self-supervised learning, and grouped proteins down to the level of individual complexes (Kobayashi et al. 2022). SubCell applies self-supervised vision transformers to the whole HPA image collection and produces a hierarchical map of the proteome learned from images alone (Gupta et al. 2024). Such image embeddings can be combined with sequence embeddings from protein language models (Section 1.10).
13.5.4 Organisation of the whole cell
The Allen Institute for Cell Science has taken a complementary approach: instead of many proteins in one cell line, it tagged 25 key structures in human induced pluripotent stem cells and imaged more than 200,000 living cells in three dimensions. The data show that the organisation of the cell’s interior is robust to large variation in cell shape, and provide a quantitative description of what a “normal” cell looks like (Viana et al. 2023). The images are available through the Allen Cell Explorer.
See also: further explanation in Section D.13.4; related Wikipedia articles in Section E.13.4.
Background from the Wikipedia articles Human Protein Atlas and Immunofluorescence (CC BY-SA 4.0, retrieved 5 October 2026).
13.6 Tissues: multiplexed imaging and computational pathology
Cells in a tissue are defined not only by what they express but also by where they are and which cells they touch. Images of tissues are therefore analysed both to classify cell types and to describe how they are arranged.
13.6.1 Multiplexed imaging and spatial omics
Highly multiplexed imaging measures tens of proteins in the same tissue section (Section 8.6). In CODEX, antibodies carry DNA barcodes that are read in repeated cycles (Goltsev et al. 2018). In imaging mass cytometry (Giesen et al. 2014) and multiplexed ion beam imaging (MIBI) (Angelo et al. 2014), antibodies are labelled with metal isotopes that are read by mass spectrometry. Imaging-based spatial transcriptomics, such as MERFISH (Chen et al. 2015) and Xenium (Janesick et al. 2023), measures hundreds to thousands of RNAs in the same way (Section 7.6); an overview of the technologies is given by Park et al. (2022).
The analysis is the same for all of these methods: segment the cells (Section 13.3), quantify each marker in each cell, assign cell types, and analyse spatial neighbourhoods, for example which cell types are found next to each other more often than expected by chance. Cell types are usually assigned by clustering, as for single-cell RNA-seq, but they can also be classified directly from the images; CellSighter, a CNN trained on a few thousand labelled cells, reaches about the agreement between human annotators (Amitay et al. 2023).
13.6.2 Computational pathology
Pathologists diagnose cancer and other diseases from H&E-stained tissue sections, and digital scanners have made these slides available for machine learning. A whole-slide image is too large to process at once, so it is split into thousands of small tiles, each tile is embedded with a network, and the embeddings are aggregated, often with attention, to make a prediction for the slide (multiple-instance learning). Networks trained in this way grade tumours, predict mutations, and predict gene expression from morphology, which links histology to spatial transcriptomics (He et al. 2020).
Since 2024, several pathology foundation models have been trained by self-supervised learning on very large collections of slides: UNI on more than 100,000 slides (Chen et al. 2024), Virchow on about 1.5 million (Vorontsov et al. 2024) and Prov-GigaPath on more than 170,000 slides, with a model of the whole slide rather than single tiles (Xu et al. 2024). CONCH was trained on pairs of images and captions, so that slides can be searched and classified with text (Lu et al. 2024). Their embeddings are used as general features for many clinical tasks.
A warning example shows why careful evaluation is needed. In The Cancer Genome Atlas, the hospital that submitted a slide can be recognised from the image itself, from differences in staining and scanning, even after colour normalisation. Models can use these site signatures to predict outcomes, which inflates accuracy when training and test sets contain slides from the same sites (Howard et al. 2021). This is the image version of the data leakage described in Section 12.2.
See also: further explanation in Section D.13.5; related Wikipedia articles in Section E.13.5.
Background from the Wikipedia articles Digital pathology and Spatial transcriptomics (CC BY-SA 4.0, retrieved 5 October 2026).
13.7 Cell atlases and virtual cells
The methods of this chapter and of Section 7.6 are now being combined in very large projects. Their first aim is to catalogue all human cell types; their more ambitious aim is a model that can predict how a cell behaves.
13.7.1 Cell atlases
The Human Cell Atlas (HCA), launched in 2016, aims to describe every cell type in the human body by its molecular profile, location and morphology, and to make all data openly available (Regev et al. 2017). A collection of papers published in 2024 described progress from a census of cells towards three-dimensional maps of organs and foundation models trained on the atlas (Rood et al. 2024). The NIH Human BioMolecular Atlas Program (HuBMAP) builds three-dimensional maps of human tissues at single-cell resolution, combining imaging with omics (HuBMAP Consortium 2019; Jain et al. 2023). The Human Protein Atlas (Section 8.6) and the Allen Institute for Cell Science (Section 13.5) contribute the protein and imaging dimensions.
13.7.2 The idea of an AI virtual cell
A widely cited perspective from 2024 proposed the AI virtual cell: a model, learned from data, that represents molecules, cells and tissues in different states, and can simulate the effect of perturbations that have never been measured (Bunne et al. 2024). Unlike the mechanistic models of Section 13.8, such a model would not be built from known reactions but learned from atlases, perturbation screens and images. Its proposed uses include predicting the effects of drugs and mutations and designing experiments in silico.
Several large initiatives are working towards this goal:
- The Chan Zuckerberg Initiative (CZI) and its Biohubs have made virtual cells a central aim. They run an open Virtual Cells Platform with models, data sets and benchmarks, and in 2025 started the Billion Cells Project to generate single-cell data for training models. Models include TranscriptFormer, trained on single-cell data from twelve species (Pearce et al. 2025), and rBio, a language model trained with virtual cell models as a source of feedback (Istrate et al. 2025).
- The Arc Institute developed State, a model of the response of cells to perturbations trained on more than 100 million perturbed cells (Adduri et al. 2026), and runs the annual Virtual Cell Challenge, a blind test of perturbation prediction in the spirit of CASP (Roohani et al. 2025) (Section 9.2).
- Alpha Cell, a strategic programme of SciLifeLab funded by the Knut and Alice Wallenberg Foundation and led by Jan Ellenberg and Mathias Uhlén, aims to build predictive AI models of human cells. It starts from the data of the Human Protein Atlas, and adds spatial models from light and electron microscopy and temporal models of how cells change. In 2026 it was funded until 2033, with KTH, Stockholm University, Karolinska Institutet and Uppsala University as partners (scilifelab.se/alpha-cell).
- Google DeepMind has said publicly that a virtual cell is a long-term goal after AlphaFold, but has not described a specific project.
13.7.3 How good are the models?
Predicting the effect of a perturbation on gene expression is the most common test of a virtual cell. GEARS used a knowledge graph of gene–gene relations to predict the effect of combinations of perturbations (Roohani et al. 2023), and single-cell foundation models (Section 7.6) have been fine-tuned for the same task. Independent evaluations, however, found that deep-learning models did not outperform deliberately simple linear baselines (Ahlmann-Eltze et al. 2025), and that the embeddings of foundation models used without fine-tuning were often no better than simpler methods (Kedzierska et al. 2025). The organisers of the first Virtual Cell Challenge in 2025 reached a similar conclusion: most models did not consistently beat simple baselines across all metrics, and no single metric captured what makes a model good. These results do not show that the goal is wrong, but they show that the claims should be tested, as for protein structure prediction, by blind benchmarks against simple baselines (Section 12.2).
See also: further explanation in Section D.13.6; related Wikipedia articles in Section E.13.6.
Background from the Wikipedia article Human Cell Atlas (CC BY-SA 4.0, retrieved 5 October 2026), and from the web pages of the CZI Virtual Cells Platform, the Arc Institute Virtual Cell Challenge and SciLifeLab Alpha Cell (retrieved 5 October 2026).
13.8 Whole-cell models and cryo-electron tomography
The AI virtual cell (Section 13.7) is learned from data. An older and complementary approach builds a model of a cell from its known parts: genes, reactions, molecules and their structures. Such models are explicit about mechanism, but need a complete parts list, which exists only for the simplest cells.
13.8.1 Mechanistic whole-cell models
The first whole-cell model simulated the complete life cycle of Mycoplasma genitalium, a bacterium with 525 genes. It combined 28 sub-models of different cellular processes, such as metabolism, transcription, translation and DNA replication, each described with the mathematics best suited to it, and predicted phenotypes, such as the growth of single-gene deletion strains, from the genotype (Karr et al. 2012).
The synthetic minimal cell JCVI-syn3A, with 493 genes, has been modelled in more detail. A kinetic model in which the geometry of the cell and the positions of its ribosomes were taken from cryo-electron tomograms reproduced the doubling time and the distribution of mRNA half-lives (Thornburg et al. 2022), and a later model simulated the complete cell cycle, including chromosome segregation and division, in space and time (Thornburg et al. 2026). Such models show which behaviours follow from the known parts, and where knowledge is missing.
13.8.2 Structural models of a cell
Going further, the physical structure of an entire cell can be modelled. Structural models of a whole Mycoplasma cell have been built by packing experimental and predicted protein structures, the genome and membranes into a cell volume (Maritan et al. 2022), and work towards simulating an entire minimal cell with molecular dynamics (Section 9.11), at a coarse-grained level, is under way (Stevens et al. 2023). All-atom simulations of a small part of a bacterial cytoplasm showed that crowding and weak interactions between proteins affect their stability and diffusion (Yu et al. 2016). Predicted structures and complexes (Chapter 9 and Chapter 11) provide many of the parts for such models.
13.8.3 Cryo-electron tomography and visual proteomics
Cryo-electron tomography (cryo-ET) images frozen cells, or thin slices of them cut with a focused ion beam, from many angles, and reconstructs a three-dimensional image of the cell interior at close to molecular resolution (Young and Villa 2023; Beck and Baumeister 2016). Unlike single-particle cryo-EM (Section 9.4), it shows complexes in their native context. The idea of visual proteomics is to identify and locate every large complex in such tomograms, so that the composition and organisation of a cell can be read from an image (Nickell et al. 2006). In the small bacterium Mycoplasma pneumoniae, cryo-ET combined with cross-linking mass spectrometry revealed how RNA polymerase and the ribosome are coupled (O’Reilly et al. 2020), and averaging many copies of the ribosome in cells reached 3.5 Å resolution (Tegunov et al. 2021).
Finding particles in noisy tomograms is an image-analysis problem of the kind described in Section 13.3. Deep-learning methods segment membranes and locate complexes, either trained for specific targets (Teresa-Trueba et al. 2023) or with embeddings that generalise to new ones (Rice et al. 2023), and dedicated tools analyse membranes and membrane proteins (Lamm et al. 2026). The CZ Imaging Institute’s CryoET Data Portal provides standardised, annotated tomograms for training (Ermel et al. 2024), and a Kaggle competition on particle picking showed how open challenges can improve the methods (Peck et al. 2026). Cryo-ET thus connects the two directions of this chapter: it provides images of whole cells at molecular detail, and the geometry needed by mechanistic and structural models.
See also: further explanation in Section D.13.7; related Wikipedia articles in Section E.13.7.
Background from the Wikipedia articles Cryogenic electron tomography, Cellular model and Mycoplasma genitalium (CC BY-SA 4.0, retrieved 5 October 2026).
13.9 Key concepts
A checklist for revision.
- Images as data: pixels, channels, z-stacks and time series; intensity is not a molecule count; transmitted light, fluorescence, electron microscopy and histology; whole-slide images
- Classical analysis: preprocessing, thresholding and watershed segmentation, feature extraction; Fiji, CellProfiler, ilastik, QuPath, napari
- Data and standards: metadata (REMBI), OME-Zarr; BioImage Archive, Image Data Resource, BBBC; the Cell Tracking Challenge
- Tasks: image and cell classification; semantic vs. instance segmentation; detection and tracking; restoration and label-free prediction
- Architectures: CNNs (AlexNet, ResNet); U-Net and skip connections; StarDist (star-convex polygons) and Cellpose (flow fields); vision transformers
- Self-supervision and foundation models: DINO and DINOv2, masked autoencoders; Segment Anything and its microscopy versions; why self-supervision matters when labels are scarce
- Pitfalls: batch effects between plates, microscopes and hospitals; splitting by experiment; disagreement between annotators; hallucinated structures in predicted images; site signatures in pathology
- Image-based profiling: Cell Painting (six dyes, five channels, eight components); single-cell features, aggregation and normalisation to controls; batch correction; profile similarity and mechanism of action; JUMP and RxRx3
- Protein localisation: the HPA subcellular atlas; multi-localising proteins and cell-to-cell variation; the HPA Kaggle competitions, multi-label classification and macro-F1; weak supervision; OpenCell, endogenous tagging and cytoself; SubCell; combining image and sequence embeddings
- Tissues: CODEX, imaging mass cytometry and MIBI; imaging-based spatial transcriptomics; segmentation, cell-type assignment and neighbourhood analysis; multiple-instance learning on whole slides; pathology foundation models (UNI, CONCH, Virchow, Prov-GigaPath); predicting expression from histology
- Atlases: the Human Cell Atlas, HuBMAP, the Allen Institute for Cell Science; what is a cell type?
- AI virtual cells: definition; perturbation prediction (GEARS, State); the Virtual Cell Challenge; CZI (Virtual Cells Platform, Billion Cells Project, TranscriptFormer) and SciLifeLab’s Alpha Cell; why simple baselines are often as good as deep models
- Mechanistic models: the M. genitalium whole-cell model; the JCVI-syn3A minimal cell in 4D; structural and molecular-dynamics models of whole cells; learned vs. mechanistic models
- Cryo-electron tomography: in situ structural biology; focused-ion-beam milling; subtomogram averaging; visual proteomics; deep learning for particle picking and membrane segmentation; the CryoET Data Portal
13.10 Recommended reading
These papers are recommended for reading in full:
- Chamier et al. (2019) — what a biologist should know about deep learning for microscopy, including its pitfalls.
- Stringer et al. (2020) — Cellpose; a generalist deep-learning method for cell segmentation.
- Ouyang et al. (2019) — image classification of protein localisation as a community challenge, with lessons for evaluation.
- Chandrasekaran et al. (2024) — Cell Painting and image-based profiling of matched chemical and genetic perturbations (JUMP).
- Bunne et al. (2024) — the AI virtual cell; what it is meant to be and how it could be built.
- Ahlmann-Eltze et al. (2025) — a critical test; deep-learning models of perturbation effects against simple baselines.
- Thornburg et al. (2026) — a mechanistic, spatial model of the complete cell cycle of a minimal cell.
Titles, abstracts, links and the full texts are in Section A.13; a longer list of further papers is in Section B.13.
13.11 Background reading (open access)
- Microscopy (Wikipedia) — Microscopy is the technical field of using microscopes to view subjects too small to be seen with the naked eye. There are three well-known branches of microscopy: optical, electron, and scanning probe microscopy, along with the emerging field of X-ray microscopy.
https://en.wikipedia.org/wiki/Microscopy - Image segmentation (Wikipedia) — In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple image segments, also known as image regions or image objects. The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries in images.
https://en.wikipedia.org/wiki/Image_segmentation - Convolutional neural network (Wikipedia) — A convolutional neural network (CNN) is a type of feedforward neural network that learns features via filter optimization. This type of deep learning network has been applied to process and make predictions from many different types of data including text, images and audio.
https://en.wikipedia.org/wiki/Convolutional_neural_network - U-Net (Wikipedia) — U-Net is a convolutional neural network that was developed for image segmentation. The network is based on a fully convolutional neural network whose architecture was modified and extended to work with fewer training images and to yield more precise segmentation. Segmentation of a 512 × 512 image takes less than a second on a modern (2015) GPU using the U-Net architecture.
https://en.wikipedia.org/wiki/U-Net - Vision transformer (Wikipedia) — A vision transformer (ViT) is a transformer designed for computer vision. A ViT decomposes an input image into a series of patches, serializes each patch into a vector, and maps it to a smaller dimension with a single matrix multiplication. These vector embeddings are then processed by a transformer encoder as if they were token embeddings.
https://en.wikipedia.org/wiki/Vision_transformer - High-content screening (Wikipedia) — High-content screening (HCS), also known as high-content analysis (HCA) or cellomics, is a method that is used in biological research and drug discovery to identify substances such as small molecules, peptides, or RNAi that alter the phenotype of a cell in a desired manner.
https://en.wikipedia.org/wiki/High-content_screening - Human Protein Atlas (Wikipedia) — The Human Protein Atlas (HPA) is a Swedish-based program started in 2003 with the aim to map all the human proteins in cells, tissues and organs using integration of various omics technologies, including antibody-based imaging, mass spectrometry-based proteomics, transcriptomics and systems biology.
https://en.wikipedia.org/wiki/Human_Protein_Atlas - Digital pathology (Wikipedia) — Digital pathology is a sub-field of pathology that focuses on managing and analyzing information generated from digitized specimen slides. It utilizes computer-based technology and virtual microscopy to view, manage, share, and analyze digital slides on computer monitors.
https://en.wikipedia.org/wiki/Digital_pathology - Spatial transcriptomics (Wikipedia) — Spatial transcriptomics, or spatially resolved transcriptomics, is a method that captures positional context of transcriptional activity within intact tissue. The historical precursor to spatial transcriptomics is in situ hybridization, where the modernized omics terminology refers to the measurement of all the mRNA in a cell rather than select RNA targets. It comprises an important part of spatial biology.
https://en.wikipedia.org/wiki/Spatial_transcriptomics - Human Cell Atlas (Wikipedia) — The Human Cell Atlas is a global project to describe all cell types in the human body, to understand human health and for diagnosing, monitoring, and treating disease.
https://en.wikipedia.org/wiki/Human_Cell_Atlas - Cellular model (Wikipedia) — A cellular model or virtual cell is a computational model of aspects of a biological cell, for the purposes of in silico research.
https://en.wikipedia.org/wiki/Virtual_cell - Cryogenic electron tomography (Wikipedia) — Cryogenic electron tomography (cryoET) is an imaging technique used to reconstruct high-resolution (~1–4 nm) three-dimensional volumes of samples, often biological macromolecules and cells.
https://en.wikipedia.org/wiki/Cryogenic_electron_tomography
Descriptions of Wikipedia articles are taken from the introductions of the articles (CC BY-SA 4.0).
13.12 Student recommendations
Recommended by students preparing for the exam: papers, reviews, online lectures, tutorials and short explanations. To suggest something, use the suggestion form (SU login).
No recommendations yet.
13.13 Review problems
The format of these problems follows Matteo Tadiello, Modern Bioinformatics: From Foundations to Frontiers (CC BY 4.0); the problems were written for this book, since that book has no chapter on image analysis. In the HTML version, click a box to show the extensions, solution outline and common errors.
13.13.1 Intensity is not abundance
A fluorescence screen reports that the mean nuclear intensity of a tagged protein is twice as high in treated as in untreated cells. What does this show about the amount of the protein in the nucleus, and what would you check before concluding that the treatment increases it?
Extensions:
- Which steps of acquisition and analysis affect the measured intensity?
- How do segmentation errors change a per-nucleus mean?
- What metadata are needed for the result to be reusable?
Solution outline:
- Each pixel value depends on the label, microscope, exposure, detector and background, not only on the number of molecules; images are comparable only if acquired in the same way.
- Check that treated and untreated wells were imaged with the same settings, on the same plates or with plate controls, and corrected for background and uneven illumination.
- Segmentation determines which pixels are counted: if the treatment changes nuclear size or shape, the segmentation may include cytoplasm or miss parts of the nucleus, and a mean can change while the total does not (or the reverse).
- A change in localisation (from cytoplasm to nucleus) is not the same as a change in total amount; measure both compartments.
- Use replicate wells and plates as the units, not individual cells, and confirm with an independent method such as western blotting or mass spectrometry.
- Report acquisition and analysis metadata following REMBI, and deposit the images, for example in the BioImage Archive.
Common errors:
- Treating intensity as a molecule count.
- Counting thousands of cells as independent replicates.
- Ignoring that the treatment can change cell shape and so the segmentation.
Connections: Section 13.2 (images as data, classical analysis, metadata); Section 12.2 (non-independent examples).
13.13.2 Choosing a segmentation method
A laboratory needs to segment densely packed, touching nuclei and irregularly shaped cells in a new kind of tissue image. Compare thresholding with watershed, StarDist and Cellpose, and describe how you would evaluate the result.
Extensions:
- What is the difference between semantic and instance segmentation?
- Why do U-Net skip connections help?
- How can a general model be adapted to new data with few annotations?
Solution outline:
- Separating touching objects is an instance-segmentation problem; semantic segmentation (nucleus vs. background) is not enough.
- Thresholding with a watershed transform is fast and transparent but sensitive to uneven staining and tends to over- or under-split.
- StarDist predicts star-convex polygons, which suits roughly round nuclei but not irregular cell shapes.
- Cellpose predicts flow fields towards cell centres and works for many shapes without retraining; Cellpose 2.0 can be adapted with a few corrected images (human in the loop).
- Evaluate on images annotated independently, ideally by more than one person, from experiments not used for training, with object-level measures (for example matched objects at an overlap threshold) rather than pixel accuracy alone.
- Check that downstream measurements, such as cell counts or marker intensities, are robust to the choice of method.
Common errors:
- Evaluating on tiles from the same images used for training.
- Reporting pixel accuracy, which is dominated by background.
- Assuming that a general model works on a new tissue without checking.
Connections: Section 13.3 (U-Net, StarDist, Cellpose); Section 12.2 (evaluation and test sets).
13.13.3 Self-supervised models and foundation models for images
Explain how self-supervised learning can produce useful features for microscopy images without labels, and how you would test whether a pre-trained image model is better than CellProfiler features for classifying protein localisation.
Extensions:
- Compare DINO and masked autoencoders.
- Why have self-supervised methods become important for vision transformers?
- What is a fair comparison?
Solution outline:
- Vision transformers treat image patches as tokens and need large training sets; labels for microscopy are scarce, which makes self-supervision attractive.
- In DINO, a student network learns to match a teacher network on different views of the same image; masked autoencoders hide most patches and reconstruct them, like masked language modelling.
- Features are evaluated by probing: train a simple classifier, for example logistic regression, on frozen embeddings, and compare with the same classifier on CellProfiler features.
- Split by protein, cell line or experiment rather than by image or cell, so that the same protein does not occur in both training and test sets.
- Use measures suited to multi-label, imbalanced classes (for example macro-F1 or per-class precision–recall).
- Check that the embeddings do not mainly encode plate, batch or microscope.
Common errors:
- Splitting by cell, so that images of the same protein are on both sides.
- Comparing a tuned deep model with an untuned baseline.
- Assuming that a model pre-trained on natural images, such as SAM, works on microscopy without fine-tuning or testing.
Connections: Section 13.3 (vision transformers and self-supervision); Section 13.5 (HPA and image-based localisation); Section 12.6 (evaluating foundation models).
13.13.4 Image-based profiling
Describe how a Cell Painting experiment is turned into profiles, and how such profiles can be used to suggest the mechanism of action of an uncharacterised compound.
Extensions:
- What do the six dyes label?
- Why normalise to negative controls on the same plate?
- How would you test whether a compound has any effect at all?
Solution outline:
- Six dyes, imaged in five channels, label DNA, RNA and nucleoli, the endoplasmic reticulum, actin, Golgi and plasma membrane, and mitochondria.
- Cells are segmented and hundreds to thousands of size, shape, intensity and texture features (or deep embeddings) are measured per cell.
- Features are aggregated per well, normalised against negative controls on the same plate, reduced to informative, uncorrelated features, and corrected for batch effects between plates and sites.
- Test whether a compound’s profile differs from controls more than replicate controls differ from each other.
- Compare its profile with profiles of compounds of known mechanism, or of genetic perturbations, for example by cosine similarity; similar profiles suggest a shared mechanism, a hypothesis to be tested.
- Evaluate such matching with held-out plates and replicate consistency, as in the CPJUMP1 benchmark.
Common errors:
- Interpreting profile similarity as proof of a shared target.
- Ignoring plate and batch effects, so that compounds on the same plate look similar.
- Using cells rather than wells or plates as replicates.
Connections: Section 13.4 (Cell Painting, JUMP, RxRx3); Section 7.4 (normalisation, batch effects, clustering).
13.13.5 Site signatures in computational pathology
A model trained on whole-slide images from The Cancer Genome Atlas predicts patient survival with high accuracy. Explain how site signatures could inflate this result and how you would design a fair evaluation.
Extensions:
- How is a whole-slide image turned into a single prediction?
- Why does colour normalisation not solve the problem?
- What role can pathology foundation models play?
Solution outline:
- Slides are split into tiles, each tile is embedded, and the embeddings are aggregated, often with attention, in multiple-instance learning.
- Staining, scanning and preparation differ between hospitals, and the submitting site can be recognised from the image even after colour normalisation.
- If survival or patient mix differs between sites, a model can predict outcome by recognising the site; with slides from the same sites in training and test sets, accuracy is inflated.
- Split by site, keep all slides from a patient together, and test on an external cohort.
- Check whether the site can be predicted from the embeddings and whether performance holds within sites.
- Foundation models such as UNI or Virchow give general features, but their embeddings can also encode site, and their pre-training data may overlap with the test slides.
Common errors:
- Splitting by tile or slide rather than by patient and site.
- Assuming colour normalisation removes all site effects.
- Interpreting attention maps as the cause of the prediction.
Connections: Section 13.6 (computational pathology); Section 12.2 (data leakage and external validation); Section 12.4 (interpretability).
13.13.6 Spatial analysis of multiplexed images
A study of multiplexed tissue images (for example imaging mass cytometry) reports that T cells are found next to tumour cells more often than expected in patients who respond to therapy. Describe the analysis, and what is needed before the result is trusted.
Extensions:
- What are the steps from image to cell types?
- What is a suitable null model for neighbourhood enrichment?
- What is the independent unit?
Solution outline:
- Segment the cells, quantify each marker per cell, and assign cell types by clustering or by a classifier such as CellSighter.
- Segmentation errors cause spill-over of markers between neighbouring cells, which can create apparent mixed cell types and spurious neighbours.
- Neighbourhood enrichment is tested against permutations of cell-type labels within each image, which keep the positions and density of cells.
- Tissue architecture (for example tumour borders) and the area sampled can cause enrichment by themselves.
- The patient is the unit: many cells or regions from one patient are not independent replicates, and the comparison between responders and non-responders must be made between patients, with an independent cohort.
- Spatial association is not causation; a role of T-cell contact would need perturbation or longitudinal data.
Common errors:
- Treating each cell or image region as an independent sample.
- Using a null model that ignores tissue structure.
- Ignoring marker spill-over from segmentation errors.
Connections: Section 13.6 (multiplexed imaging and spatial omics); Section 7.6 (spatial transcriptomics and cell types); Section 12.8 (prediction and causation).
13.13.7 How good is a virtual cell?
A company claims that its AI virtual cell model predicts the transcriptional response of human cells to genetic perturbations that were never measured. What evidence would you require?
Extensions:
- What baselines should be used?
- How should the test perturbations be chosen?
- Why did the first Virtual Cell Challenge find no single good measure?
Solution outline:
- Define what is predicted (expression changes, cell-type shifts, combinations) and in which cell types.
- Hold out perturbations, and ideally cell types or experiments, that were not in training; combinations should not have their single perturbations in the test set if that is the claim.
- Compare with simple baselines, such as predicting the mean response or a linear model, which in independent evaluations have matched deep models.
- Use several measures (correlation of changes, error on differentially expressed genes, ranking of perturbations), since a model can look good on one and poor on another.
- Prefer blind benchmarks, such as the Virtual Cell Challenge, in which predictions are made before the data are released.
- Confirm selected predictions experimentally.
Common errors:
- Accepting performance on perturbations similar to the training data as generalisation.
- Omitting simple baselines.
- Reporting only the most favourable measure.
Connections: Section 13.7 (virtual cells and their evaluation); Section 12.6 (foundation models and baselines); Section 9.2 (blind benchmarks).
13.13.8 Mechanistic and learned models of cells
Compare a mechanistic whole-cell model, such as that of Mycoplasma genitalium or JCVI-syn3A, with a learned AI virtual cell. What can each do, and how can cryo-electron tomography contribute to both?
Extensions:
- Why have mechanistic models been built only for minimal cells?
- What is visual proteomics?
- What are the image-analysis problems of cryo-ET?
Solution outline:
- Mechanistic models are built from known genes, reactions and structures, combine sub-models of processes such as metabolism and replication, and predict phenotypes, such as the growth of deletion strains, from genotype; they need a nearly complete parts list and many parameters.
- Learned models are trained on atlases, perturbation screens and images; they can cover human cells but are only as good as their training data and must be tested against simple baselines.
- Mechanistic models show which behaviours follow from known parts and where knowledge is missing; learned models may predict without explaining.
- Cryo-ET gives three-dimensional images of the cell interior near molecular resolution; visual proteomics aims to identify and locate every large complex.
- Finding particles in noisy tomograms is a deep-learning problem; annotated data such as the CryoET Data Portal and open challenges help.
- The geometry from cryo-ET (cell shape, ribosome positions) was used in the JCVI-syn3A model, and structural maps can provide training data and parts for both kinds of model.
Common errors:
- Assuming either kind of model is complete or correct because it reproduces some observations.
- Confusing cryo-ET with single-particle cryo-EM.
Connections: Section 13.8 (whole-cell models and cryo-ET); Section 13.7 (AI virtual cells); Section 9.4 (cryo-EM); Section 11.3 (systems biology and metabolic models).