Michael J. Black

Black's thesis[1] reformulated optical flow estimation as a robust M-estimation problem. The main observation was that spatial discontinuities in image motion and violations of the standard brightness constancy assumption could be treated as outliers. Reformulating the classical optimization problem as a robust estimation problem produced more accurate results.

This "Black and Anandan" optical flow algorithm has been widely used, for example, in special effects.[2] The method was used to compute optical flow for the painterly effects in What Dreams May Come and for registering 3D face scans in The Matrix Reloaded.

A version of this work received the IEEE Outstanding Paper Award at CVPR 1991[3] and the Helmholtz Prize at ICCV 2013 for work that has "stood the test of time".[4]

His early focus on statistical modeling of motion, particularly at motion discontinuities, led to two other prize papers. His work with David Fleet on the "Probabilistic Detection and Tracking of Motion Boundaries" won honorable mention for the Marr Prize at ICCV'99.[5] Black's work with Stefan Roth "On the spatial statistics of optical flow" received honorable mention for the Marr Prize at ICCV 2005.[6]

His work with Deqing Sun and Stefan Roth on the "Secrets of Optical Flow" was awarded the 2020 Longuet- Higgins Prize. The prize is given annually by the IEEE Pattern Analysis and Machine Intelligence (PAMI) Technical Committee for "Contributions in Computer Vision that Have Withstood the Test of Time." The "secrets" paper helped establish the state of the art in the field and led to the widely used Classic+NL flow algorithm.

The "Black and Anandan" method helped popularize robust statistics in computer vision. This was facilitated by several papers that connected robust penalty functions to classical "line processes" used in Markov Random Fields (MRFs) at the time. Black and Rangarajan[7] characterized the formal properties of robust functions that have an equivalent line-process form and provided a process to convert between these formulations (known now as "Black-Rangarajan Duality"[8]). Black and colleagues applied these ideas to image denoising,[9] anisotropic diffusion,[10] and principal-component analysis (PCA).[11][12]

The robust formulation was hand crafted and used small spatial neighborhoods. The work on Fields of Experts with Stefan Roth removed these restrictions.[13] They learned the potential functions of an MRF with large spatial cliques by modeling the field potentials as a product of experts. Their formulation can be viewed as a shallow convolutional neural network.

In 1993, Black and Jepson used mixture models to represent optical flow fields with multiple motions[14] (also called "layered" optical flow). This introduced the use of Expectation Maximization (EM) to the field of computer vision.

In the 2000s, Black worked with John Donoghue and others at Brown University to create the technology behind the BrainGate neural prosthetics technology. Black and colleagues developed Bayesian methods to decode neural signals from motor cortex. The team was the first to use Kalman filtering[15][16][17] and particle filtering[18] to decode motor cortical ensemble activity.  With these Bayesian decoding methods, the team demonstrated the successful point-and-click control of a computer cursor by a human with paralysis[19][20] and the decoding of full arm and hand movement in non-human primates.[21]

Black is best known for his work on human motion and shape estimation. With Hedvig Sidenbladh and David Fleet, he introduced the use of particle filtering for tracking 3D articulated human motion.[22] This work was awarded the Koenderink Koenderink Prize for Fundamental Contributions in Computer Vision at ECCV 2000.

His current work focuses on modeling and estimating human shape and pose from images and video. His team was the first to fit a learned 3D human body model to multi-camera image data at CVPR 2007,[23] under clothing at ECCV 2008,[24] from a single image at ICCV 2009,[25] and from RGB-D data at ICCV 2011.[26]

His group produced the popular SMPL 3D body model[27] (and various extensions like FLAME[28] for 3D human faces, MANO[29] for 3D hands, and SMPL-X,[30] an expressive 3D body model with hands and faces) and popularized methods for estimating 3D body shape from images.[31][32] SMPL is widely used in both academia and industry and was one of the core technologies licensed by Body Labs Inc.

Loper and Black popularized "differentiable rendering",[33] which has become an important component of self-supervised training of neural networks for problems like facial analysis. Classical methods for analysis by synthesis formulate an objective function and then differentiate it. The OpenDR[34] method was more generic in that it (approximately) differentiated a graphics rendering engine using Automatic differentiation. This provided a framework for posing a forward synthesis problem and automatically obtaining an optimization method to solve the inverse problem.

Black has contributed to several significant datasets. The Middlebury Flow dataset provided the first comprehensive benchmark for the field.[35] The MPI-Sintel Flow dataset demonstrated that synthetic data was sufficiently rich and similar to real data to provide a rigorous benchmark and to be useful for learning optical flow.[36]

The HumanEva dataset[37] was the first dataset with ground truth 3D human poses in correspondence with RGB video of people in motion. The approach used a combination of optical motion capture and multi-camera video capture. This dataset enabled the field to evaluate accuracy and compare performance for the first time.

Related to human pose, shape, and activity, Black has contributed to the SURREAL dataset of human motions,[38] the JHMDB dataset of human actions,[39] and the FAUST dataset of 3D body shapes.[40] FAUST received the Dataset Award from the Eurographics Symposium on Geometry Processing (SGP), 2016.[41]