Posted on Dec 17, 2019

FastTrack AI describes MetaVi Labs automated analysis system based on our Artificial Intelligence Engine.

Fast Track Artificial Intelligence (AI) Logo

Contents

The movie below is an example of how AI can be used to find specific features within a cell. On the top are the input images (phase contrast). On the bottom are the outputs of the FastTrack AI engine. The engine was trained to find cell nuclei and also cell bodies excluding nuclei. Different colors are painted by the AI engine to indicate its prediction results (light blue indicates a prediction of nuclei, dark blue indicates a prediction of cell body).

What Is AI?

Artificial Intelligence (AI) is a broad term that refers to mathematical and software models that allow computers to learn patterns and then find the patterns they have learned. Machine Learning or Deep Learning are other terms which are roughly synonymous. AI applies to many different types of problems and machine visioni s just one sub-set.

MetaVi Labs specializes in vision and microscopy analysis so we focus on the machine vision category of AI.Fundamentally, we teach computers what cells, or other structures, look like, then the computer finds them for us.

The key concerns in AI system for vision are:

  • the neural network architecture
  • the training process
  • the pre and post processing required to capture useful and usable data
  • computational cost

Benefits of FastTrack AI

While several automated tracking software packages are on the market, most require careful adjustments to image processing parameters such as thresholding and contrast. These software solutions often rely on techniques developed in the early days of image analysis for object classification.

These crude methods are error prone and require user intervention. Modern methods use complex filters which recognize shapes much more like the human vision system.

Some important advantages of the FastTrack AI

  • Highest accuracy of cell identification on a wide array of cell morphologies
  • Tracks very high numbers of cells (up to 1000 per image) for much higher relevance of results
  • Saves hundreds of hours of labor over manual analysis
  • Improved accuracy over human tracking
  • Works on Phase contrast with no labels or dyes required
  • Complete end to end solution for full automation for high content screening applications
  • Very simple drag-and-drop interface for maximum ease-of-use
  • Very comprehensive reports with easy to read condition comparison charts and also raw-data

Neural Network Architecture

MetaVi Labs uses a form of deep residual convolutional neural networks (ResNet). The name of this machine learning model comes from the fact that it is modeled after our limited understanding of vision systems in nature.Object recognition in the brain is a complex multi-stage, multi-step process. Signals flow through layers where each layer appears to have a specialized task in the process of converting signals from light sensitive neurons into labeled objects.

By labeled objects we mean to say names or concepts of what an object is. For example, I see a "dog" chasing a "cat".Dog and cat in this example are labels. Likewise, MetaVi Labs teach our ResNet what objects look like using labels.For example, we use labels for objects such as "cells" or "apoptotic bodies", or "cell nuclei", etc. There will be more on the training process below, but essentially it is associating pixel patterns with labels.

While the animal vision system is complex and our understanding is vague, a recent article in Nature (Cichy, 2016) shows a remarkable similarity in deep neural networks and human vision: Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence.

The image below, from the Nature (Cichy, 2016) article, depicts the typical deep learning network (but the residual return paths are not shown).The term 'deep' comes from the multiple layers. While only 5 layers are depicted, it is typical to use50 to 150 connected layers in a deep network.

FastTrack AI 3

FastTrack AI 2

2015: A Pivotal Year for Machine Vision

There are two major international annual competitions for visual recognition:

  • Imagenet - Large Scale Visual Recognition Challenge (ILSVRC)
  • Microsoft COCO: Common Objects in Context

In the year 2015, a network design by a team from Microsoft beat all challengers in ILSVRC2015 and COCO-2015.This design is captured in a paper that now has over 34,000 citations (He, 2015) and became known as ResNet.Their design had 152 layers. This approach was so powerful that it revolutionized AI for vision applications.

The MetaVi Labs FastTrack AI implementation is based on this same design. The image below from (He, 2015) is a graphical representation of the 152 residual network design.

FastTrack AI 4

The Training Process

Training means teaching the ResNet what objects are by associating pixels with labels. For example, we can produce a set of images of many cells and label all of them as "cell".And we can provide images of apoptotic bodies and label them as "apoptotic bodies".But in reality the process is not so simple. We must also teach the computer model what pixels are not cells and not other objects we are training on.

One of the key challenges in images of living and moving cells is that the cells are constantly morphing in shape as they locomote, divide, and die. Additional challenges are the wide variety of imaging modes, exposures, fields-of-view, and overall image quality in the microscopy images.

MetaVi Labs trainer's are individuals who have viewed thousands of hours of cell images and movies such that they have become experts in what the target objects look like. Then the experts are able to create specialized training images (with the aid of software)that cab be used to train the network. So human experts train computers; human knowledge is transferred to the computer.

In the sample image below (also see the sample movie at the top of this page), the trainers created a few hundred examples of what cell nuclei are in the phase contrast image, and also labeled cell bodies excluding nuclei. They also indicated what are not cell nuclei and what are not cell bodies. Once trained, the FastTrack AI engine was able to predict what pixels are nuclei and what pixels are cell bodies (indicated with the difference shades of blue). The sample movie at the top of this article is an example of a fully predicted time sequence based on a training set of a few frames.

FastTrack AI_1

The pre and post processing required to capture useful and usable data

To make AI useful a complete system must be in place to capture user's images from microscopes, prepare them for AI object classification, and then convert AI classification predictions into objects that can be counted and tracked. For example in Chemotaxis additional processing is requires to find x, y, z cell centers, connect new centers to previous cell tracks, and then to calculate useful indicators of directed cell movement.

Additionally, a database is required to organize and track the massive amount of data that can be generated in a 96 well plate, for example.A presentation layer is required to present the information in a manageable fashion to the investigators.

The MetaVi Labs FastTrack AI system is a complete end-to-end automation system that supports high content screening applications.The system interfaces with popular living cell high content screening systems (such as the Incucyte) and automates the entire processing pipeline.

The Computational Cost

AI has enormous computational complexity. Only recently with the advent of 12nm silicon processes has it been possible to build AI machines that can keep pace with the human vision system.

As an example, consider the Lamba MAX Machine Learning workstation. It boasts:

  • 4x Quadro RTX 8000 Nvida GPUS providing 490 Tera-Flops of TensorFlow performance
  • Intel W-2195 CPU with 18 Cores
  • 256 GB Memory
  • 2 TeraBytes SSD with 3,500 MB/sec read rate
  • List Price of $29,000 USD

FastTrack AI_5

To allocate one of these machines for each experimenter would be prohibitive in costs. So an alternative is to rent space on a cloud hosted virtual network, such as Amazon Web Services.

FastTrack AI_6

Cloud service providers, such as Amazon, Microsoft, Google, et al., provide elastic computation networks.These are vast networks of virtual machine fabrics. A virtual machine fabric is an array of specialized computational blocks where each block is design to load and run complete operating systems on an as-needed basis.

For example, as the FastTrack AI platform detects well recordings have been uploaded for analysis, it asks Amazon Web ServicesElastic Compute Cloud 2 (EC2) to launch an instance of our analysis platform. At this time, EC2 finds on its network an available virtual computer slice and copies the entire C:\ drive that has been stored on its S3 storage platform into this computer slice and starts it up. Once running the analyzer processes all the well images and generates reports. When completed it shuts down.EC2 then wipes the disk drive of the computer slice and shuts it down. This all happens within a few minutes by using solid state disk drives and memory based drive technology.

The MetaVi Labs FastTrack AI platform can launch hundreds of these virtual computer instances as needed and then shut them down when not in use. This allows FastTrack AI to scale up to thousands of wells of analysis per day. Sharing resources in this way also dramatically reduces costs, cost savings that can be passed on to the end user.

Feel free to contact us if you would like to find out how our FastTrack AI platform can help your lab.

External References

Posted on Feb 14, 2019

MetaVi Labs has developed algorithms that remove motion jitter induced by stage re-position errors.

For example, in this un-stabilized time-lapse movie, we see cancer cells and T cells. The T cells move at a much higher speed than the cancer cells. But with all the cell movement, you may not notice that the entire image is moving from frame to frame. If you focus your attention on the cells indicated in the center you will notice an Y-axis jitter. This is very pronounced in the first three seconds of the movie.

If this were uncorrected, it would introduce additional motion into the cell tracks. When measuring cell speeds this jitter in the tracks would add additional speed which is not actually present. So its important to correct this motion jitter introduced by the X Y stage. When time-lapse scanning a multi-well plate, at each time point, the X Y stage must return to the original position in each well. But all stages introduce some re-position error, some more than others, because of the precision limits of the stage motors.

So we developed a specialized technique to fix this problem. The first step is to find the cells which move the least and also move in unison. While each cell is doing its own thing, stage positioning errors affect all cells equally. So the trick is to remove the change in position due to individual cells but find the X Y shift that is common to all the cells. This takes a massive amount of calculations on a large images (the original images from the microscope were 2048 x 2048). The goal is to correct the error down to sub-pixel resolution. Sub-pixel correction is achieved by estimating pixel values between pixels.

As you can see in the corrected movie below, there is no longer and stage error shift. The only motion is due to the cells moving individually.

Posted on Nov 26, 2018

Researchers at the Institut für Experimentelle Immunologie und Bildgebung, University of Essen/Duisberg, have discovered a major finding that may in the future provide a simple way for oncologists to monitor the progression of Myelodysplastic Syndromes (MDS) disease. Today, patients suspected of having MDS undergo a series of bone marrow analysis assays. Due to the invasiveness of this procedure, it is not used to monitor the patient's progress on a regular basis. But Prof Matthias Gunzer has shown that with a simple assay based on peripheral blood it may be possible to provide accurate (sometimes more accurate than existing bone marrow assays) assessments of the patients' risk. MetaVi Labs provided the first ever system that can accurately and reliably measure the extracted lymphocyte cell motility of these patients.

The MetaVi Labs Automated Cellular Analysis System (ACAS) was used to test thousands of blood tissue samples in Prof. Gunzer's lab over the past four years. The ACAS system is also capable of scaling to handle the thousands of samples per day that would be required for a clinically viable assay. Purified human neutrophils were stimulated with fMLP [10 nM], CXCL1 [100 ng/ml] and CXCL8 [100 ng/ml]. PBS was used as control.

In the left column, raw videos are shown, the right column contains the same videos tracked via MetaVi's ACAS® software. Yellow circles and blue crosses point to the positions of the cells. Red lines highlight the cell tracks. The results are representative for the HeinzNixdorf-recall control individuals. The video is presented at a frame rate of 30 pics/sec. The video was acquired at 1 pic/8 sec for a total duration of 1 hour.

Researchers at the Institut für Experimentelle Immunologie und Bildgebung, University of Essen/Duisberg, have discovered a major finding that may in the future provide a simple way for oncologists to monitor the progression of Myelodysplastic Syndromes (MDS) disease. Today, patients suspected of having MDS undergo a series of bone marrow analysis assays. Due to the invasiveness of this procedure, it is not used to monitor the patient's progress on a regular basis. But Prof Matthias Gunzer has shown that with a simple assay based on peripheral blood it may be possible to provide accurate (sometimes more accurate than existing bone marrow assays) assessments of the patients' risk. MetaVi Labs provided the first ever system that can accurately and reliably measure the extracted lymphocyte cell motility of these patients.

The MetaVi Labs Automated Cellular Analysis System (ACAS) was used to test thousands of blood tissue samples in Prof. Gunzer's lab over the past four years. The ACAS system is also capable of scaling to handle the thousands of samples per day that would be required for a clinically viable assay. Purified human neutrophils were stimulated with fMLP [10 nM], CXCL1 [100 ng/ml] and CXCL8 [100 ng/ml]. PBS was used as control.

In the left column, raw videos are shown, the right column contains the same videos tracked via MetaVi's ACAS® software. Yellow circles and blue crosses point to the positions of the cells. Red lines highlight the cell tracks. The results are representative for the HeinzNixdorf-recall control individuals. The video is presented at a frame rate of 30 pics/sec. The video was acquired at 1 pic/8 sec for a total duration of 1 hour.

Posted on Oct 19, 2018

assay report center of mass displacements

Sample Report

MetaVi Labs Chemotaxis Assay Reports provide numerous characteristics of moving cell behavior. This page will describe each of the analysis results.

User Input

In the Experiment-set edit page, choose 'Advanced Settings' to display the chemotaxis settings(Click on the image below to see full size).

Experiment-set edit page chemotaxis settings

Critical Settings for Chemotaxis Analysis

Microscopic Resolution Pixels/µm. This key factor is used by the image analysis algorithms to find and track the cells and all downstream calculations of distances and speeds.  For help calculating this important factor, see: Tech Note: Calculating Microscopic Resolution
Time Between Images The number of seconds between each image (the inverse of the frame rate). This factor is used in calculations of speed and results graphed with respect to time.
Minimum Track Duration Number of seconds a track should be exist (based on time-between-images) to be considered in the analysis. Tracks shorter than this time are excluded.
Evaluation Interval The time frame for calculating speed. For example, if the interval is 120 seconds, then the distance moved from in each 120 second interval  determines the speed in that interval.
Movement Threshold Minimum number of µm a cell must move to be considered as having moved. Cells that oscillate in position less than this threshold are excluded from the analysis.

Re-Doing Reports

redo report icon Look for the re-do-report icons on the experiment-set page next to each experiment-trial.  Make adjustments in the above settings then click the re-do-report.  Chemotaxis reports typically take 2 minutes to complete. Create as many reports as necessary to determine your preferred settings (there is no cost associated with generating reports). Then use these same settings for future experiments.

Interpreting the Results

Number Tracks in Interval.

In each evaluation interval, the number of tracks (that are longer than the minimum track duration) are plotted for each well in the experiment.

number tracks in interval chart

Number Tracks Starting in Interval

In each evaluation interval, the number of new tracks (that are longer than the minimum track duration) are plotted for each well in the experiment.

numbe tracks starting in interval chart

Number Tracks Ending in Interval

In each evaluation interval, the number of tracks which termited (that are longer than the minimum track duration) are plotted for each well in the experiment.

number tracks ending in interval chart

Forward Migration Index

The Forward Migration Index is a measure the cell's track directionality forward where forward is defined as toward the bottom of the view.  The calculation is a measure of Chemotactic Bias as Defined by Foxman et al. 1999 (please consult this paper for the calculation and a detailed explanation).   The first and final positions of cell cell track (Euclidean Distance) are used in the calculation.

forward migration index chart

Center of Mass Displacements

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vectors of the cell tracks.  All the vector X values are averaged and all the vector Y values are averaged to form average x,y vector for the well. This final average vector is the Center of Mass Displacement for the well.  This is an indicator of the average Chemotaxis Bias with distance and directionality.

center of mass displacements chart

Center of Mass Displacement Magnitudes

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track.  All the vector X values are averaged and all the vector Y values are averaged. The Euclidean distance of the average-X and the average-Y is the Center of Mass Displacement Magnitude for the well.  This is an indicator of the average Chemotaxis Bias as a distance only.

center of mass magnitude chart

Mean Cell Speed (both including and excluding non-moving cells)

In each evaluation interval, the cell speed is calculated by taking the distance it made in the interval divided by the interval duration (time).  The speeds over all intervals for each cell are averaged. Then the average speed for each cell track is averaged for the entire well.

This calculation is done for two cases a) including all cells and b) excluding non-moving cells. Non-moving cells are cell tracks shorted than the Minimum Track Duration.

The Movement Threshold applies to the calculation of all tracks - that is, all cells must move greater than the Movement Threshold in any given interval to be considered has having moved at all.

Mean Cell Speed in Interval includes both moving and non-moving cells.

Note: cells that move in circles and end up where they started can have cell speed but zero velocity. Refer to the Mean Cell Velocity explanation below.

mean cell speed in excluding non-moving chart

mean cell speed in interval chart

Mean Cell Velocity (both including and excluding non-moving cells)

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track.  The Velocity is calculated as the Euclidean Distance divided by the time duration of the track. Then the velocities for all cell tracks in the well are averaged for the entire well; this is the Mean Cell Velocity.

This calculation is done for two cases a) including all cells and b) excluding non-moving cells. Non-moving cells are cell tracks shorted than the Minimum Track Duration.

The Movement Threshold applies to the calculation of all tracks - that is, all cells must move greater than the Movement Threshold in any given interval to be considered has having moved at all.

Note: cells that move in circles and end up where they started can have cell speed but zero velocity.  The greater the final distance from the starting point, the greater the potential velocity (depending of course on the time required to make that distance).

mean cell velocity excluding non-moving chart

Average Directness

The directness represents a measure of the cell's tendency to travel in a straight line. It is the Euclidean distance divided by the accumulated distance for each cell. A value greater than or equal to 1 indicates straight-line migration.

average directness of cell paths chart

Mean Euclidean Distance

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track.  The the Euclidean Distance for all cell tracks in the well are averaged.

mean Euclidean distance chart

Accumulated Distance

In each evaluation interval, the cell distance is calculated. This distance is accumulated for all cell tracks in the well.

accumulated distance chart

Accumulated Euclidean Distance

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track. The magnitude of this distance is accumulated for all cell tracks in the well.

accumulated euclidean distance chart

assay report center of mass displacements

Sample Report

MetaVi Labs Chemotaxis Assay Reports provide numerous characteristics of moving cell behavior. This page will describe each of the analysis results.

User Input

In the Experiment-set edit page, choose 'Advanced Settings' to display the chemotaxis settings(Click on the image below to see full size).

Experiment-set edit page chemotaxis settings

Critical Settings for Chemotaxis Analysis

Microscopic Resolution Pixels/µm. This key factor is used by the image analysis algorithms to find and track the cells and all downstream calculations of distances and speeds.  For help calculating this important factor, see: Tech Note: Calculating Microscopic Resolution
Time Between Images The number of seconds between each image (the inverse of the frame rate). This factor is used in calculations of speed and results graphed with respect to time.
Minimum Track Duration Number of seconds a track should be exist (based on time-between-images) to be considered in the analysis. Tracks shorter than this time are excluded.
Evaluation Interval The time frame for calculating speed. For example, if the interval is 120 seconds, then the distance moved from in each 120 second interval  determines the speed in that interval.
Movement Threshold Minimum number of µm a cell must move to be considered as having moved. Cells that oscillate in position less than this threshold are excluded from the analysis.

Re-Doing Reports

redo report icon Look for the re-do-report icons on the experiment-set page next to each experiment-trial.  Make adjustments in the above settings then click the re-do-report.  Chemotaxis reports typically take 2 minutes to complete. Create as many reports as necessary to determine your preferred settings (there is no cost associated with generating reports). Then use these same settings for future experiments.

Interpreting the Results

Number Tracks in Interval.

In each evaluation interval, the number of tracks (that are longer than the minimum track duration) are plotted for each well in the experiment.

number tracks in interval chart

Number Tracks Starting in Interval

In each evaluation interval, the number of new tracks (that are longer than the minimum track duration) are plotted for each well in the experiment.

numbe tracks starting in interval chart

Number Tracks Ending in Interval

In each evaluation interval, the number of tracks which termited (that are longer than the minimum track duration) are plotted for each well in the experiment.

number tracks ending in interval chart

Forward Migration Index

The Forward Migration Index is a measure the cell's track directionality forward where forward is defined as toward the bottom of the view.  The calculation is a measure of Chemotactic Bias as Defined by Foxman et al. 1999 (please consult this paper for the calculation and a detailed explanation).   The first and final positions of cell cell track (Euclidean Distance) are used in the calculation.

forward migration index chart

Center of Mass Displacements

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vectors of the cell tracks.  All the vector X values are averaged and all the vector Y values are averaged to form average x,y vector for the well. This final average vector is the Center of Mass Displacement for the well.  This is an indicator of the average Chemotaxis Bias with distance and directionality.

center of mass displacements chart

Center of Mass Displacement Magnitudes

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track.  All the vector X values are averaged and all the vector Y values are averaged. The Euclidean distance of the average-X and the average-Y is the Center of Mass Displacement Magnitude for the well.  This is an indicator of the average Chemotaxis Bias as a distance only.

center of mass magnitude chart

Mean Cell Speed (both including and excluding non-moving cells)

In each evaluation interval, the cell speed is calculated by taking the distance it made in the interval divided by the interval duration (time).  The speeds over all intervals for each cell are averaged. Then the average speed for each cell track is averaged for the entire well.

This calculation is done for two cases a) including all cells and b) excluding non-moving cells. Non-moving cells are cell tracks shorted than the Minimum Track Duration.

The Movement Threshold applies to the calculation of all tracks - that is, all cells must move greater than the Movement Threshold in any given interval to be considered has having moved at all.

Mean Cell Speed in Interval includes both moving and non-moving cells.

Note: cells that move in circles and end up where they started can have cell speed but zero velocity. Refer to the Mean Cell Velocity explanation below.

mean cell speed in excluding non-moving chart

mean cell speed in interval chart

Mean Cell Velocity (both including and excluding non-moving cells)

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track.  The Velocity is calculated as the Euclidean Distance divided by the time duration of the track. Then the velocities for all cell tracks in the well are averaged for the entire well; this is the Mean Cell Velocity.

This calculation is done for two cases a) including all cells and b) excluding non-moving cells. Non-moving cells are cell tracks shorted than the Minimum Track Duration.

The Movement Threshold applies to the calculation of all tracks - that is, all cells must move greater than the Movement Threshold in any given interval to be considered has having moved at all.

Note: cells that move in circles and end up where they started can have cell speed but zero velocity.  The greater the final distance from the starting point, the greater the potential velocity (depending of course on the time required to make that distance).

mean cell velocity excluding non-moving chart

Average Directness

The directness represents a measure of the cell's tendency to travel in a straight line. It is the Euclidean distance divided by the accumulated distance for each cell. A value greater than or equal to 1 indicates straight-line migration.

average directness of cell paths chart

Mean Euclidean Distance

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track.  The the Euclidean Distance for all cell tracks in the well are averaged.

mean Euclidean distance chart

Accumulated Distance

In each evaluation interval, the cell distance is calculated. This distance is accumulated for all cell tracks in the well.

accumulated distance chart

Accumulated Euclidean Distance

The first and final positions of each cell cell track (Euclidean Distance) are used to determine the magnitude of x,y vector of the cell track. The magnitude of this distance is accumulated for all cell tracks in the well.

accumulated euclidean distance chart