The explosion of breakthroughs, investments, and entrepreneurial activity around artificial intelligence over the last decade has been driven exclusively by deep learning, a sophisticated statistical analysis technique for finding hidden patterns in large quantities of data. A term coined in 1955—artificial intelligence—was applied (or mis-applied) to deep learning, a more advanced version of an approach to training computers to perform certain tasks—machine learning—a term coined in 1959.
The recent success of deep learning is the result of the increased availability of lots of data (big data) and the advent of Graphics Processing Units (GPUs), significantly increasing the breadth and depth of the data used for training computers and reducing the time required for training deep learning algorithms.
The technology that animated movies like "Toy Story" and enabled a variety of special effects is the ... [+]
The term “big data” first appeared in computer science literature in an October 1997 article by Michael Cox and David Ellsworth, “Application-controlled demand paging for out-of-core visualization,” published in the Proceedings of the IEEE 8th conference on Visualization. They wrote that “Visualization provides an interesting challenge for computer systems: data sets are generally quite large, taxing the capacities of main memory, local disk, and even remote disk. We call this the problem of big data. When data sets do not fit in main memory (in core), or when they do not fit even on local disk, the most common solution is to acquire more resources.”
The term was in use at the time outside of academia, as well. For example, John R. Mashey, Chief Scientist at SGI, gave a presentation titled “Big Data… and the Next Wave of Infrastress” at an April 1998 USENIX meeting. SGI, founded in 1981 as Silicon Graphics, Inc., focused on developing hardware and software for processing 3D images.
SGI’s founder Jim Clark completed his PhD dissertation in 1974 at the University of Utah under the supervision of Ivan Sutherland, the “father of computer graphics.” Clark later founded Netscape Communications whose successful Web browser and 1995 IPO launched the “Internet boom.” The invention of the Web in 1989 by Tim Berners-Lee and its success in making billions of people around the world consumers and creators of digital data, facilitated the annotation of billions of widely shared digital images (e.g., identifying a photo of a cat as a “cat”).
In 2007, computer scientist Fei-Fei Li and her colleagues at Princeton University started to assemble ImageNet, a large database of annotated images designed to aid in visual object recognition software research. Five years later, in October 2012, a deep learning artificial neural network designed by researchers at the University of Toronto achieved an error rate of only 16% in the ImageNet Large Scale Visual Recognition Challenge, a significant improvement over the 25% error rate achieved by the best entry the year before, heralding the resurgence of “artificial intelligence.”
Big data was indeed big. In 1996, digital storage became more cost-effective for storing data than paper, according to R.J.T. Morris and B.J. Truskowski in “The Evolution of Storage Systems.” And in 2002, digital information storage surpassed non-digital storage for the first time. According to “The World’s Technological Capacity to Store, Communicate, and Compute Information” by Martin Hilbert and Priscila Lopez, the world’s information storage capacity grew at a compound annual growth rate of 25% per year between 1986 and 2007. They also estimated that in 1986, 99.2% of all storage capacity was analog, but in 2007, 94% of storage capacity was digital, a complete reversal of roles.
In October 2000, Peter Lyman and Hal Varian at UC Berkeley published “How Much Information?”—the first comprehensive study to quantify, in computer storage terms, the total amount of new and original information (not counting copies) created in the world annually—in 1999, the world produced 1.5 exabytes of original data. In March 2007, John Gantz, David Reinsel and other researchers at IDC published the first study to estimate and forecast the amount of digital data created and replicated each year—161 exabytes in 2006, estimated to increase more than six-fold to 988 exabytes in 2010, or doubling every 18 months.
The information explosion (a term first used in 1941, according to the Oxford English Dictionary) has turned into the big digital data explosion. But the quantity of available data was only one of the two catalysts that made deep learning successful. The other one was GPUs.
While the development of deep learning algorithms and their practical application have progressed steadily during the 1980s and 1990s, they were limited by inadequate computer power. In October 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published ”Learning representations by back-propagating errors,” in which they describe “a new learning procedure, back-propagation, for networks of neurone-like units,” a conceptual breakthrough in the evolution of deep learning. Three years later, Yann LeCun and other researchers at AT&T Bell Labs successfully applied a backpropagation algorithm to a multi-layer neural network, recognizing handwritten ZIP codes. But given the hardware limitations at the time, it took about 3 days (still a significant improvement over earlier efforts) to train the network.
Computer graphics, where big data was born, came to the rescue. By the 1990s, real-time 3D graphics were becoming increasingly common in arcade, computer and console games, leading to an increased demand for hardware-accelerated 3D graphics. Sony first used the term GPU for Geometry Processing Unit when it launched the home video game console PS1 in 1994.
Video game rendering requires performing many operations in parallel quickly. Graphics cards are designed to have a high degree of parallelism and high memory bandwidth, at the cost of having a lower clock speed and less branching capability relative to traditional CPUs. It so happened that deep learning algorithms running on artificial neural networks require similar characteristics—parallelism, high memory bandwidth, no branching.
By the end of the 2000s, a number of researchers have demonstrated the usefulness of GPUs to deep learning, specifically for artificial neural network training. General-purpose GPUs, enabled by new programming languages such as NVIDIA’s CUDA, were applied to a variety of deep learning tasks. The most visible such application was the winning entry in the 2012 ImageNet Challenge, mentioned above.
On March 18, 2020, the Association for Computing Machinery (ACM) named Patrick M. (Pat) Hanrahan and Edwin E. (Ed) Catmull as the recipients of the 2019 ACM A.M. Turing Award for fundamental contributions to 3D computer graphics, and the revolutionary impact of these techniques on computer-generated imagery (CGI) in filmmaking and other applications.
Today, according to the ACM press release “3-D computer animated films represent a wildly popular genre in the $138 billion global film industry. 3-D computer imagery is also central to the booming video gaming industry, as well as the emerging virtual reality and augmented reality fields. Catmull and Hanrahan made pioneering technical contributions which remain integral to how today’s CGI imagery is developed. Additionally, their insights into programming graphics processing units (GPUs) have had implications beyond computer graphics, impacting diverse areas including data center management and artificial intelligence.”
Like Jim Clark, Catmull was Ivan Sutherland’s student and received his PhD from the University of Utah in 1974. As Robert Rivlin wrote in his 1986 book The Algorithmic Image: Graphic Visions of the Computer Age, "almost every influential person in the modern computer-graphics community either passed through the University of Utah or came into contact with it in some way.”
In a 2010 interview with Pat Hanrahan, Catmull described the U of U working environment:
“Dave Evans was the chairman of the department and Ivan was teaching, but their company, Evans and Sutherland, took all their excess time. The students were pretty much independent, which I took as a real positive in that the students had to do something on their own. We were expected to create original work. We were at the frontier, and our job was to expand it. They basically said, ‘You can consult with us every once in a while, and we'll check in with you, but we're off running this company.’
I thought that worked great! It set up this environment of supportive, collegial work with each other.”
Later in the same discussion, Hanrahan said:
“When I first got interested in graphics in grad school, I heard about this quest to make a full-length computer-generated picture. At the time I was very interested in artificial intelligence, which has this idea of a Turing test and emulating the mind. I thought the idea of making a computer-generated picture was a prelim to, or at least as complicated as, modeling the human mind, because you would have to model this whole virtual world, and you would have to have people in that world—and if the virtual world and the people in it didn't seem intelligent, then that world would not pass the Turing test and therefore wouldn't seem plausible.
I guess I was savvy enough to think we weren't actually going to be able to model human intelligence in my lifetime. So, one of the reasons I was interested in graphics is I thought it had a good long-term career potential.”
See also
12 Artificial Intelligence (AI) Milestones: 2. Ramon Llull And His ‘Thinking Machine’
