Visible Embryo Project

In the mid-1990s, Michael Doyle collaborated with Harvard's Betsey Williams to create an internet atlas of mouse development, in a project named "Muritech." A prototype two- and three-dimensional color atlas of mouse development was developed, using two embryos, a 13.5 d normal mouse embryo and a PATCH mutant embryo of the same age. Serial sections of the embryos, with an external registration marker system, introduced into the paraffin embedding process, were prepared by standard histological methods. For the 2D atlas, color images were digitized from 100 consecutive sections of the normal embryo. For the 3D atlas, 300 gray-scale images digitized from the mutant embryo were conformally warped and reconstructed into a 3D volume dataset. The external fiducial system facilitated the three-dimensional reconstruction by providing accurate registration of consecutive images and also allowed for precise spatial calibration and the correction of warping artifacts. The atlases, with their associated anatomical knowledge base, were then integrated into a multimedia online information resource via the VEP's Web technology to provide research biologists with a set of advanced tools to analyze normal and abnormal murine embryonic development.[17]

The Human Embryology Digital Library and Collaboratory Support Tools project was begun in 1999 as a demonstration of the biomedical application potential of the Next Generation Internet (NGI). The collaborators included eight organizations at sites around the continental USA, a mix of medical and information technology organizations, including George Mason University, Eolas, the Armed Forces Institute of Pathology, Johns Hopkins University, Lawrence Livermore National Laboratory, the Oregon Health & Science University, the San Diego Supercomputer Center, and the University of Illinois at Chicago. The project undertook three major applications, based on the Carnegie Collection of Embryos at the AFIP's National Museum of Health and Medicine Human Development Anatomy Center (HDAC), a collection of cellular-level tissue slides that is one of the world's largest repositories of human embryos.

These applications included:

1. Digitization, curation, and annotation of embryo data: The VEP team created a production digitization capability, using automated digital microscopy, with data automatically registered for tiling and transmitted to the repository at the San Diego Supercomputer Center, and annotated by teams of biomedical volunteers with expert-level quality control. 2. Distributed embryology education using materials derived from the Carnegie Collection to create animations of embryo development and recorded master classes that can be streamed over the Internet or downloaded to create a portable electronic classroom. 3. Clinical management planning where medical professionals and expectant parent patients can review normal and abnormal development patterns with collaborative consultation from distant experts.[9]

To enable new ways to interactively explore the VEP's massive volume datasets, Michael Doyle created the zMap system, using the Visible Human Project image data for the first prototype. In 2011, Doyle collaborated with Steven Landers, Maurice Pescitelli, and others to use zMap to create an interactive tool that allows the user to select desired sets of anatomical structures for the automated generation of 3D Quicktime VR visualizations. The system used the resources available in the Eolas AnatLab knowledgebase, which has over 2200 structures identified involving a total of over 4600 sections and 700,000 annotations overall, to access the anatomical structure surface information for individual structures. This surface information was then used to automatically extract the contained volumetric image data and convert the data into a format compatible with the Osirix volume imaging system. Automated scripts then controlled Osirix in the creation of a 3D visualization of the group of selected anatomical structures. Photorealistic results were obtained by using the original color voxel information from the original Visible Human cryosection images to color the surface of the 3D reconstruction. The system then automatically progressed through a pre-defined set of rotations to generate the set of image frames required to create a Quicktime VR (QTVR) interactive movie. This system thereby allowed an anatomy instructor to quickly and easily generate customized interactive 3D reconstructions for use in the classroom.[18]

Even though spatial mapping of Omics data had been described as an initial goal of the VEP, it wasn't until 1999 that four VEP collaborators, Michael Doyle, George Michaels, Maurice Pescitelli, and Betsey Williams worked together to create a system for what they called "spatial genomics."[6] Today, this technology is known as Spatial transcriptomics. As their 2001 U.S. patent application states,[6] their system solved the need "to gather gene expression data in a manner that supports the type of exploratory research that can take advantage of the broad-spectrum types of biologic activity analysis enabled by today's microarray tools," as well as the need for "technology to allow the collection of large volumes of these types of data, to enable exploratory investigations into patterns of biologic activity ... to correlate gene expression data with morphological structure in a useful and easy to understand manner, such as in a volume visualization environment ... to allow the collection of larger volumes of gene expression data across a wider spectrum of gene types than ever before."

They named their system SAGA, short for Spatial Analysis of Genomic Activity. As described in the related U.S. patents,[6][19][20] the SAGA system enabled the multidimensional morphological reconstruction of tissue biologic activity and "makes it possible for biological tissue specimens to be imaged in multiple dimensions to allow morphological reconstruction. The same tissue specimen is physically sampled in a regular raster array, so that tissue samples are taken in a regular multidimensional matrix pattern across each of the dimensions of the tissue specimen. Each sample is isolated and coded so it can be later correlated to the specific multidimensional raster array coordinates, thereby providing a correlation with the sample's original pre-sampling morphological location in the tissue specimen. Each tissue sample is then analyzed with broad-spectrum biological activity methods, providing information on a multitude of biologic functional characteristics [mRNA, etc.] for that sample. The resultant raster-based biological characteristic data may then be spatially mapped into the original multidimensional morphological matrix of image data. ... various types of analysis may then be performed on the resultant correlated multidimensional spatial datasets."

Spatial transcriptomics was named the "Method of the Year for 2020" by Nature, in January 2021.[21]

In 1993, Dr. Doyle became the Director of the UCSF Center for Knowledge Management (CKM). To create the underlying software and hardware that would provide the needed computational power for the VEP, Doyle's CKM group designed a new paradigm for performing remote client-server volume visualization over the Internet.[10] This involved creating a system for remotely computing visualizations through a networked cluster, or cloud, of distributed heterogeneous computational engines, and coordinating the computations to pass user interface control messages to those engines, causing the cloud computers to generate new rendered visualizations and stream the resulting views to the users' desktops, while delta-encoding and compressing the streamed data to optimize performance over low-bandwidth connections.[22]

To hide the complexity of the system from the user, they modified one of the earliest versions of the NCSA Mosaic Web browser[23] to allow their interactive cloud-computing applications to be automatically launched and run embedded within Web pages, so any user would need only to load a Web document from the VEP and would be able to immediately interactively explore the project's multidimensional datasets, rather than static representations of those datasets.[24]

In November 1993, the CKM's VEP research group demonstrated this system, the first Web-based Cloud application platform, on-stage to a meeting of approximately 300 Bay Area SIGWEB members at Xerox PARC.[25]

Today, this capability is called "the Cloud." The VEP team's work opened the door to the potential of the Web to provide rich information resources to users, regardless of where they were located and spawned a multi-trillion-dollar industry as a result.[26]

Dr. Doyle then began to focus more directly on the problem of how to navigate within these complex biomedical volume datasets and developed a system for mapping the semantic identity of morphological structures within the datasets and integrating those mappings with the hypermedia linking mechanism of the Web. This led to the creation of the first three-dimensional Web image map system and was used to create a variety of online interactive reference systems for biomedical education and research throughout the 90s and beyond.[27][28]

One of the challenges for large collaborative knowledge bases is how to assure the integrity of data over a long period of time. Standard cryptographic methods that depend upon trusting a central validation authority are vulnerable to a variety of factors that can lead to data corruption, including hacking, data breaches, insider fraud, and the possible disappearance of the validating authority itself. To solve this problem for the VEP data collections, Dr. Doyle created a novel type of cryptographic system, called Transient-key cryptography. This system allows the validation of data integrity without requiring users of the system to trust any central authority, and also represented the first decentralized blockchain system, enabling the later creation of the Bitcoin system. In the mid-2000s, this technology was adopted as a national standard in the ANSI ASC X9.95 Standard for trusted timestamps.[29][30]

Since the mid-2000s, the VEP team has made great use of digital voice and text communications systems, to facilitate communications among geographically-distributed team members. To increase the efficiency of these communications, Michael Doyle and Steve Landers collaborated to create the Skybot system, the first AI-based mobile virtual assistant system. Skybot used the power and flexibility of AI to dramatically expand the use of messaging systems. Using Skybot, one could create a variety of programmable responses to incoming calls and chat messages. The system incorporated a state machine that could be configured to automatically trigger automated responses to various communication and user-context events. This provided the user with a surprisingly broad and powerful set of capabilities for automating mobile communication operations and pioneered the mobile intelligent-assistant product category that is now ubiquitous worldwide. [31][32]