The authors confirm that all data underlying the findings are fully available without restriction. All relevant data are included within the Supporting Information files.
Introduction {#s1}
============
In recent years, deep learning has become a popular tool for the analysis of complex datasets. Deep learning based methods for network reconstruction, inference and analysis are generally more accurate and more computationally efficient than previous approaches [@pone.0113273-Bengio1]--[@pone.0113273-LeCun1], and have led to remarkable results in various fields [@pone.0113273-Goodfellow1], [@pone.0113273-Lecun1]. For instance, Deep Convolutional Neural Networks (DCNNs) have recently been successfully employed for learning from retinal [@pone.0113273-Jia1]--[@pone.0113273-Benson1] and blood vessel [@pone.0113273-Long1], [@pone.0113273-Tschirren1] images, or more generally complex natural images [@pone.0113273-Krizhevsky1], [@pone.0113273-Sermanet1], [@pone.0113273-Zeiler1].
While there are many successful applications of deep learning to image reconstruction, they often lack any spatial information about the location of features or cells. A typical example is a single image produced by a fluorescent microscope, in which individual cells are unrecognizable, despite the clear image.
To gain spatial information about individual cells, we propose a new deep learning based method, named Spatial Encoding (SE). SE uses an adaptive approach to learn which pixels of the input image are associated with individual cells and which ones should remain unassigned. It uses a neural network and trains it to learn a probability distribution over the pixels of the input image by means of backpropagation through the layers of a convolutional neural network. Specifically, we employ a DCNN containing multiple convolutional and pooling layers. The last convolutional layer is followed by a final pooling layer which reduces the output dimensions to a set of probabilities over all the pixels of the input image. Then, we train a softmax classifier to map the image pixels from their probabilities of representing cells, to a discrete assignment of which pixel should be assigned to individual cells. We call this layer "Spatial Encoding" (SE) layer, and its output a "Spatial Encoding layer" (SEL). The classification approach is applied sequentially to the outputs of the final pooling layer, and the resulting probabilities are aggregated at each step. As a result, a new probability distribution over the whole input image is obtained.
In order to quantify the performance of SE for the task of cell segmentation in fluorescence microscopy images, we measure the quality of cell segmentation and counting by comparing the output of SE with ground-truth maps. Using a synthetic dataset, we show that the SEL layer can accurately identify single cells, while it only misses a very small fraction of real cells. In a set of real microscopy images, the accuracy of SE improves upon other methods for cell segmentation.
As a final step, we apply SE to analyze single-cell variability in order to assess the heterogeneity of cell subpopulations. This approach is based on a recently proposed deep learning method [@pone.0113273-Truyen1] which estimates from a set of fluorescence images the probability of observing a cell as a function of its time of development. The method was shown to produce accurate cell segmentation and counting, as well as cell trajectory estimates [@pone.0113273-Truyen1], and allows for the identification of cell subpopulations in single images by combining the information of segmentation maps for all cells within the image, and by classifying the resulting cell trajectories with a cell classification method [@pone.0113273-Truyen1]. In this work, we demonstrate that it is also possible to classify cell trajectories with a machine learning method by relying on the SEL outputs alone, without explicitly building cell segmentation and counting.
We also apply SE to quantify the heterogeneity in the response of different cell populations to an external stimulus in a series of images acquired during different phases of cell development. To our knowledge, this is the first attempt to estimate cell heterogeneity in complex images of a developing cell population.
Results {#s2}
=======
Spatial encoding: Deep learning for cell segmentation {#s2a}
-----------------------------------------------------
We develop a deep learning method, named Spatial Encoding (SE), which can distinguish single cells from each other and from the background (see [Fig. 1](#pone-0113273-g001){ref-type="fig"}). As a cell segmentation method it provides the location of individual cells and determines their boundaries without using pre-defined markers or assumptions. To obtain the location of individual cells, it builds on a DCNN, which we refer to as a *Spatial Encoding layer* (SEL). This model consists of a *Convolutional layer*, followed by a *Max Pooling layer*, followed by a *Sigmoid layer* and an optional *Dense layer* (see [Fig. 1](#pone-0113273-g001){ref-type="fig"} and methods). The spatial encoding layer is the last convolutional layer of the DCNN and has a low number of filters, compared to the input image. Each filter extracts information about a small patch of the image, e.g. about a single cell or a small region of the cell. Each filter learns to generate an activation pattern which represents the probability of finding each pixel belonging to the corresponding patch. The spatial pooling then selects the most probable pixel, i.e., the central pixel of the most activated filter. The probability distribution over all pixels is given by a multi-dimensional probability distribution, i.e., a sparse probability matrix over all image pixels. This spatial probability distribution represents the input image at its full resolution.
{#pone-0113273-g001}
Thus, SE allows for an intuitive interpretation, as each filter can be associated with a pixel. For example, if the DCNN is trained on images containing cells which have a circular shape, then the central pixel of each filter should correspond to the center of the cell. This is illustrated in [Fig. 2](#pone-0113273-g002){ref-type="fig"}, in which a DCNN is trained on images of yeast cells and an example of the spatial probability matrix is presented. In this example, the filters are assigned to different cells in an image and are in general assigned to different pixels.
{#pone-0113273-g002}
This interpretation is confirmed by experiments which show that the SEL is a good approximation of the output of a *softmax* classifier (see [Fig. 2](#pone-0113273-g002){ref-type="fig"} and [Text S1](#pone.0113273.s005){ref-type="supplementary-material"} for a detailed description). We note that the first layer of the DCNN consists of a convolutional layer without pooling, so that the SEL has the same dimensionality as the input. This first layer computes a sparse activation of the input pixels and it produces a high probability in some pixels, whereas the rest of the pixels are assigned low probability values. In this case, the pixels which represent cells are assigned the highest probability value in the matrix. A typical example is shown in [Fig. 3A](#pone-0113273-g003){ref-type="fig"}.
