[[
wikihub
]]
Search
⌘K
Explore
People
For Agents
Sign in
Explore
People
For Agents
Sign in
@jeremynixon / Thinking / mind-of-demis-hassabis.md
Suggest edit
Cancel
Submit suggestion
Title
Name
Note
--- title: "Mind of Demis Hassabis" visibility: public --- # Mind of Demis Hassabis Category: [[technical|Technical]] [Read the original document](https://docs.google.com/document/d/1n-nXvSt9xpYDGTu6KGyYRTxLUMZa-YUaAlD3kG0q9iU/edit?usp=sharing&sa=D&ust=1596495076379000&usg=AOvVaw24Npd-6BpjErhIR3ruDo15) <!-- gdoc-inlined --> --- The Mind of Demis Hassabis By Sonia Joseph [@soniajoseph_] and Jeremy Nixon [@jvnixon] DeepMind's Dreamer AI learns from the past to predict the future ... Left: DeepMind logo, source DeepMind. Right: Demis Hassabis, source Bloomberg Businessweek. Resources 2 Video 2 Papers 3 Writing 3 Articles 3 Background 4 Narrative 4 Game-player 6 Life Views 9 Approach to AGI 10 Hallmarks of AGI 10 Grounded Cognition 10 Good Old Fashioned AI 11 Interim Goals / Other Approaches 12 Whole Brain Emulation 12 The Systems Neuroscience approach to AGI 13 Motivation 13 Recent Advances in Neuroscience 13 Memory 16 Neuroscience’s Relationship to AI 18 Imagination 19 Transfer Learning and Learning Abstract Concepts 20 Creativity 23 Meta Problems - “Solve Everything Else” 23 Relevance of Reinforcement Learnin g to AGI 24 Game-playing 25 Future Directions 26 AI Applications 27 AI Ethics 28 AI Safety 29 Demis’s Neuro Research 29 Deepmind 29 Founding & Early Days 30 Research at Smaller Companies/Labs 31 Dealing with AI Hype 32 Open Access 32 DeepMind’s Future Directions 33 Google Acquisition 33 Resources Video Systems Neuroscience and AGI [Winter Intelligence Conference] [2011] [1] [First 50%] Singularity Summit Talk [2011] [2] FDOT14 Atari Talk Breakout Demo [2014] [3] Is AI the Problem or The Solution [2014] [4] BBC Pentamind Interview [2014] [5] Google Zeitgeist Talk [2015] [6] Breaking the Wall to Mind Machines [2015] [7] Studio Giochi Interview [2015] (Conversation that blew my mind (esp ~11:00 into the video)) [8] Royal Society Public Lecture [2015] [9] Future Capabilities of AI [2015] [10] <- Replica of [9] which duplicates talk. ICML DL Workshop Panel [2015] [11] NIPS 2015 Symposium [2015] [12] Towards General Artificial Intelligence (MIT CBMM) [2016] [13] Games are a microcosm of real-world problems [2016] [14] AI and the Future: RSA Replay Interview [2016] [15] Seminar at KAIST [2016] [16] Symposium: Brains, Minds and Machines at Microsoft Research [2016] [17] Royal Academy of Engineering Lecture [2016] [18] Campus Chats: Google for Startups [2017] [19] CSAR Talk: Explorations at the Edge of Knowledge [2017] [20] BBC Interview Desert Island Discs [2017] [21] Superintelligence: Science or Fiction? [2017] [22] Explorations in Optimality (Beneficial AGI) [2017] [23] Academy Class of 2017 Interview [2017] [24] Learning from First Principles (NIPS) [2017] [25] AlphaGo Documentary [2017] [26] Creativity and AI Lecture [2018] [27] Cheltenham Science Festival [2018] [28] The Disrupters Podcast [2018] AI and the Future [2019] [29] Stanford HAI Panel Discussion [2019] [30] The Power of Self-Learning Systems [2019] [31] Deepmind Podcast Interview [2020] [32] Other Video Playlists: Demis Hassabis, HUmAI Club Demis Hassabis, Mingway Huang Demis Hassabis, Kim Jong-lol Papers Cited First Author Publications Only. Patients with hippocampal amnesia cannot imagine new experiences [2007] Using imagination to understand the Neural Basis of Episodic Memory [2007] Decoding Neuronal Ensembles in the Hippocampus [2009] Writing http://www.gatsby.ucl.ac.uk/~demis (link got Redirected to Deepmind, using Wayback Machine) https://goo.gl/YTgrNu (linked in) https://twitter.com/demishassabis Articles http://www.ft.com/cms/s/2/47aa9aa4-a7a5-11e4-be63-00144feab7de.html https://web.archive.org/web/20140222212701/http://www.gatsby.ucl.ac.uk/~demis/TuringSpecialIssue%28Nature2012%29.pdf Quality Article from The Information Interview with an Evil Genius Deepmind - Google: Who’s in Charge? https://www.bbc.com/news/business-46099587 https://www.standard.co.uk/lifestyle/london-life/exclusive-interview-meet-demis-hassabis-londons-megamind-who-just-sold-his-company-to-google-for-9098707.html https://www.economist.com/1843/2019/03/01/deepmind-and-google-the-battle-to-control-artificial-intelligence Background Narrative * So from a young age at school, I kind of came to this realisation that in some fundamental sense there are only two subjects really worth studying: physics and neuroscience. For physics, of course, it’s all about explaining the external world, so the external world out there, including, of course, the entire universe, and neuroscience and psychology is really about conversely explaining about what’s inside here, our internal world. Then when I thought about this more, I actually came to the conclusion that the mind was more important because obviously that’s the way we actually interpret the external world out there, and ‘the mind interprets the world’ is something that was an idea that was first proposed by the great philosopher Immanuel Kant, and really it’s the mind that creates our reality around us. So this is where AI comes in because the ultimate expression of understanding something is being able to recreate it, and as Richard Feynman said, one of my scientific heroes, ‘What I cannot build, I do not truly understand.’ And that’s one of the things that I’m excited about with artificial intelligence-- I think ultimately it will help us understand our own minds better. [6] * I was taught how to play chess when I was four, ended up playing various England chess times, and captaining various England junior chess teams and by the age of twelve I was a chess master. The thing is, when you teach a kid from a very young age how to play chess and if they have quite a reflective personality like I did when I was young, you can’t help but thinking and introspecting about what it is about your mind that is actually coming up with these moves. What are the mechanisms that allow you to make these plans in such a complex game as chess? [6] * So then when I was around eight years old, I actually took some winnings that I won from an international chess tournament, and I bought my first computer, a ZX Spectrum 48k, and taught myself how to programme. One of the first, sort of, big programmes I can remember creating was actually a programme to play chess, and it didn’t play very well but it was able to beat my little brother which I was really pleased about when I was small. You know, it worked, and this was the beginning of the path to me towards AI. [6] * I realized on an intuitive level that computers were a special sort of machine. Most machines like cars and planes extend our physical capabilities. Cars allow us to move faster than we can run. Planes allow us to fly. But I think computers do that but in the realm of our minds. They really extend the capabilities of the brain. [9] * This really became clear to me when I used to write my first programs and did basic math calculations and other things, which really struck me. You could set something running overnight and go to sleep and you’d wake up the next morning and your computer would have solved something for you whilst you were asleep. * Now my love for programming, and chess ad games, sort of, came together naturally in the form of video games, and my first career was in creating and designing video games. I did this for ten years, and I wrote several bestselling games, but probably my most famous game I wrote was called Theme Park which I wrote when I was seventeen. [6] * This is one of the first games to use artificial intelligence as the main gameplay component. So this came out in the mid-90s, in ‘94, and the idea of the game was that you designed your own Disney World and thousands of little people with their own desires and characteristics came into that Disney World and judged and decided how much fun they had in your theme park. They would go and tell their friends, and they would come the next day. So this game spawned a whole genre of games called management simulation games, and really, sort of, started this whole genre of creative games, where instead of shooting and killing things in games, actually you create and built stuff yourself, and the game would react to how you played the game, so no two people ended up with the same game, because the AI adapted to how the player played it. [6] Above: Theme Park, Demis Hassabis’s most popular video game. Source. * And then the final piece of the jigsaw for me was, after doing this for ten years, I sold my games company and I went back to university to do a PhD in neuroscience to study how the brain itself solved some of these hard problems. I chose, as my topics, imagination and memory and an area of the brain called the hippocampus which is responsible for imagination and memory, because these are two of the capabilities that we don’t know how to do very well in AI. I wanted to see and get some inspiration for how the brain actually solves these problems. [6] * So after a couple of post-docs at MIT and Harvard, I then decided that I had all the ingredients and the components to start DeepMind, and actually attack the AI problem head on. So all these experiences then culminated, in 2010 with me co-founding DeepMind, and the idea behind DeepMind was really to create a kind of Apollo Programme mission for AI. [6] Game-player * Q: Would you tell us which game you like best and why you like this game? A: I love pretty much all games, but I have some favorites, and for different reasons. I can list my favorite games-- obviously chess was my first game, a core game. And I think chess is the foundation of all my other game skills, because it teaches you all the core thing sof planning and strategy and so on-- and discipline. Now I would probably say my more favorite games in the middle of my career, when you first met me, I think games like Diplomacy, which is I think an amazingly designed game. The problem is you can’t play it with friends, because you lose those friends [laughs]. It’s kind of limited because of that, but I think it’s actually the most fascinating games and one of the most beautifully designed games. The rules are so simple, right? And very unique, I think, in the style of game it is, with the secret moves and alliances and so on. Then of course more recently I got a lot into poker, and I think poker is a very interesting game-- even from the psychology aspect. It’s only interesting actually playing live poker, because when you play online, then you lose most of the psychology aspect, which I think is the most interesting aspect. Then in invented games, like more modern invented games, I loved Entropy-- I won the world championship many times in that. And every game is different, because obviously how you draw it out is different, and it has some luck in it, but I think it’s about the same as backgammon where it’s probabilities you’re trying to minimize. And finally I would say Go. I would say that’s probably the king of all games in terms of beauty and the aesthetic… And it’s so old, I mean it’s 3000 years old or something, like it’s always existed… I can’t talk about it yet, but in a few months I think there will be quite a big surprise… you know, we’re working on AI and Go is one of those very hard things for computers to play. Because it’s so aesthetic, right? It’s not about brute-force calculation like chess, it’s actually about patterns and beauty and shapes-- and usually those things computers are not good at. …. Settlers of Catan, actually, is also a genius game. Everyone was playing monopoly and things like that before at home, and then they made this game. The genius of it is that it’s so simple and yet you can reconfigure the boards. Every game is different, and also this German idea of making sure all the players are involved no matter whose turn it is. [8] * Q: Which achievement are you prouder of? The Pentamind or something else? A: I think the first couple of Pentaminds I am the proudest, because I played chess at a very high level when I was young, but I always felt more of a general games player, like a decathlete rather than a sprinter. So I always enjoyed many games rather than one. And the first Pentamind, when the Mind Sports Olympiad was very big, big prize money and all the best players came-- it was fantastic, right? One day hopefully maybe we can get it back to that. But I think it was quite an amazingly strong field and we had all the best games players there, and winning the first two or three times was, you know. * Q: But after a few years, and after a lot of achievement, you quit as a serious competitor in mindsports… could you explain why? A: Yeah, so about ten years ago I guess that was, and I felt partly… more than ten years ago now maybe, this was 2003 or 2004. I’d just one the Pentamind five times, so there wasn’t much more I could do with the Pentamind. It was several influencing factors together-- so I’d won it five times, I’d achieved what I wanted to achieve. Things changing in my work. I’d finished my games company, my kids were born, I was doing a PhD, I was moving into my new phase of life with my neuroscience and in my AI, so it didn’t fit quite as much to be able to go away for weekends and play lots of games all night like I used to do. Maybe I was getting older as well. And also the other thing was… one of the things I really enjoyed was learning new games, right? But by that point I knew how to play, I don’t know, 50, 100 games, I can’t remember. Some large number. So that means now… I still like learning new games, but there’s less joy because mostly I can see a motif in the game from something else-- so I already know. There’re not whole new kinds of game genres I haven’t seen before. By the time I got to my late twenties, I felt that I had explored a lot of what games really meant. And I still love them, but I have extracted… and maybe the final reason was it was around that time the Mindsports Olympiad started going downhill, and it turned off sponsorship. So it wasn’t as big an event anymore. I love anybody who is the maximum at their chosen thing. So today in modern day poker players I like Phil Ivey… and Johnny Chan and Phil Hellmuth, those guy. But I quite like the classic chess players and Go players as well, so the famous Go players from 200 years ago, Shusaku, then these guys called the Invincible, and then from the chess players I do really like Kasparov and Bobby Fischer and Mikhail Tal. Q: But these players only play one game. A: I actually don’t think many of them would have been that good at many games. I’ve talked to quite a few chess players, shown chess players other games, and it’s a different type of skill being a general games player. I think it’s more about how fast can you transfer your knowledge between games. And that’s what I love doing… and that’s what makes us all round games players, while I think a lot of those chess players, for example, they like only chess, and they only see the specifics of chess. I may comment they’re really not very good at the other games. It’s not true of everyone, but you see that in quite a lot of places. I think one of the most impressive competitors we played against-- there were a couple of really strong ones back in the day when we were playing. I always thought Larry Kaufman was very impressive because he was master of four games: Go, Shogi, chess, and Chinese chess. But still not fully all-around, right? But still quite impressive. But it’s hard to know who in the past would have been good until the Mind Sports, that’s why the Mind Sports Olympiad is so great, because really for the first time we could see who was good at all these different games. [8] * [When asked if he would be involved in making Mindsports big again] Mindsports absolutely deserves to be really big. And I think also it would be great for children to be encouraged to come and play, because in some ways-- and this is what I’m teaching my children-- it’s actually more important to be a general games player, because then you’re training life skills that you can use in other areas: business, science-- whereas if you go very narrow to try and become like Magnus Carlsen at chess, I think then you’re learning skills that are only very specific to chess after a certain level. [8] * Chess is probably the fundamental way I approach all the planning that I do, whether that’s business or anything else. I think about things with that kind of logical planning that you train up when you play chess for very long. [19] * I think chess is an amazing thing for a child to train in, because it teaches you so many meta-skills. Planning, problem-solving, imagination, how to cope with pressure, competition, what it means to excel at something, the work that that takes, all of these are meta-lessons. [19] * If I were actually to design an MBA, one of the things you could do is you could design it around games, and different games teaching you different skills that I think are really important in business but also in science and many other things. [19] * There’s not very many cases in real life where you can practice scenarios again and again and test how you would cope under that pressure, where there aren’t any real big consequences of that. So in a way, obviously that’s connected with simulations, where the whole point of using simulations for the AI system to learn in is that they can try out things in the simulation. It’s totally safe, it has no consequences, but they can still learn from it. So in a way what we’re doing with our AI algorithms is kind of what I did to myself when I was training as a kid. [19] Life Views * [On money] I’m not interested in money ever. I want to do the achievements and do brilliant things and achieve big things. And that’s what motivates me. The only good thing about money is that it means you are freer to do things that you want to do and support the things you want to do. I’ve given lots of money to my old university, I supported a little bit in Mindsports… so I think that’s what money is for, helping people you love and causes you believe in and gives you personal freedom. I can really concentrate on what I think matters in life, like my science and family and other things without worrying. So I think that’s what money should be for, not material things. [8] * You’ve been a world class chess player, you’ve been an academic, you’ve been a games designer. All those fields-- how have they brought to DeepMind and the culture you’ve created? Well I think all of my experiences, if you look back at them, people sort of think it’s quite an eclectic path I’ve taken. But my plan has always been, since I can remember, to work on AI ultimately, to do DeepMind. And I always felt from a very young age that that would be one of the most profound things that could be worked on, and it would also be extremely interesting and fun to work on. And the impact of it could be enormous. All the things I chose to do, from around the age of 14 or 15 were with that end goal in mind. So I started off in games and learned programming-- games themselves ended up being a huge part of what we do at DeepMind, in terms of using simulations, including games, to develop our AI algorithms. I’m really fascinated by the idea of simulations in general, as a way of generating data, so you don’t have to use data that’s already out there. And then the neuroscience component was finding out how the brain works, obviously the human brain is the only example we have of such a general learning system, the type of system we’d like to build and mimic artificially. So the existence proof seems to me like it’s worth spending a lot of effort trying to understand how the brain works, at least on a functional level. And that would inspire new ideas about algorithms and representations and so on. And simultaneously, because I never like doings for only one reason, ideally you’re experiencing things for multiple reasons: one was learning the neuroscience, another was understanding how academia worked, how cutting-edge research worked in the best labs in the world, what was good about those environments, what was bad, comparing that against startup environments. And actually DeepMind is this sort of hybrid between what I learned from the best practices of the best startups, and the best practices from the best academic labs. [19] Approach to AGI Hallmarks of AGI * The hallmarks of artificial general intelligence, the type of AI we work on, is that it’s built to be flexible and general and adaptive from the ground up, so there’s no special casing or pre-programming of the task involved. It has to learn everything from first principles. [6] * And we’re also interested in the notion of generality, so the idea that the set of same single system or single set of algorithms can operate out of the box across a wide range of tasks, and in fact this relates to our operational definition of intelligence that we use at DeepMind-- we define it as the ability to perform well across a wide range of environments. So while I agree with Tommy, and I think some of the other speakers today about intelligence as quite an amorphous term, we found that operationalizing it like this and putting generality at the center of it works very well for us. And you can argue about all these other properties that it doesn’t cover, you know, creativity, etc, but then actually it just depends on the tasks that you include in the set of tasks that you’re going to try and attempt. So the wider and the more diverse, the better. [12] * So we call this type of AI artificial general intelligence or AGI, and the hallmark of these kinds of algorithms is that they’re flexible and adaptive and perhaps even one day inventive, and they’re built to deal with the unexpected from the beginning. [12] Grounded Cognition * So one of the other sort of philosophies we committed to at the start of DeepMind was the idea of grounded cognition. So this is the idea, the notion, that a true thinking machine has to be grounded in a sensory-motor datastream. Now this kind of gives rise to notions of end-to-end learning agents that perhaps start with raw vision, so it may be pixels on a screen, and go all the way to making a decision about what action to take. We’re interested in that entire stack of things that’s involved, all the way from perception to decision-making. Now usually when people talk about grounded cognition or embodied cognition, people start working on or thinking about robots, and of course that’s the ultimate grounding, robots in the real world-- but actually, we decided to start off with simulations, and specifically games. [12] * Good Old Fashioned AI * The top level distinction is between non-biological approaches and biological approaches. And in fact, the non-biological approaches were taken first, and is being referred to by other speakers as ‘good-old fashioned AI’. which include things like Formal Logic Systems, Logic Networks, Lambda Calculus, Expert Systems. So a whole range of techniques were tried, and they’re all mathematically grounded. The general problem with these systems was that they tended to be brittle, didn’t deal well with uncertainty, they were time consuming to trained, needed lots of data, were poor at generalizing, struggled to generate symbols that they don’t already know, and have the symbol grounding problem, where if you have a system that is just describing entities with other symbols, how do you actually refer to things outside of the agent? [1] * GOFAI was taken to its extreme in the CYC project which was led by Doug Lenat. Lenat proposed that the real problem was not with the approach but that there wasn’t enough information in these databases. He attempted to put every piece of commonsense knowledge into one massive database - the CYC database. It’s still ongoing 25 years later and still no sign of any intelligence. [1] * On the other side are the biologically inspired approaches, which makes sense because there are 6 billion human examples of AGI systems out there, which is the brain. It makes intuitive sense that maybe we should use the brain as a blueprint. But even on this side of the [non-biological / biological] divide, this actually convers a very large range of quite different approaches. [1] * We can argue about which of these two approaches makes more sense, but thinking about this what I think this boils down to is your personal intuition about what the search space of possible intelligences looks like. [1] * We could be in a regime where the search space is quite small, and there are a large number of possible solutions. In this kind of scenario the human brain is not that special, and perhaps it’s not worth taking the time and effort and resources to understand how the brain solves these problems. * On the other hand, we could be in a regime which is very large but with sparse solutions. In this space we should pay attention to the human brain rather than taking some other approach to searching in this search space. * Which space are we in? * Two pieces of evidence: * Evolution has only produced human level intelligence once. * Large ‘non-bio’ projects in the 80s/90s failed to make progress. * We can subdivide biological approaches from the most abstract (least tied to the biology) to the most tied to the biology. [1] * The most abstract, loosely based on the brain is the cognitive architectures. SOAR, ACT-R, OpenCog, and more. What these mostly have in common is that they can be displayed and thought of as box and wire diagrams. The boxes represent modules loosely based on brain regions that are wired up together to create what is hopefully some kind of intelligence. The reason I think there are so many approaches on this front is that these are based on introspective processes on the part of designers of these systems. That’s what makes them unsatisfactory. In general there aren’t deep underlying principles as to why an architecture is being designed in a specific way. You can see that cropping up now and again when psychology discovers a new module, say episodic memory. We realized 20 years ago that that’s an important property in the brain, and then all of these architectures need to retrofit that into their existing wire diagram. Feels like an unsatisfactory, unprincipled approach to AGI. [1] Interim Goals / Other Approaches * So some intermediate goals that I’ve seen proposed by some people with a lot of efforts going into them at the moment are on the robotics side. In order to have AGI we need to have robots in the physical world. Honda research has a dedicated research team on this which has a budget in the 10s of millions a year. They produce impressive robots in terms of their physicality. But really what we’re talking about is this hugely complex engineering problems of movement, server motors, and mechanical engineering problems that I think are very difficult to solve in their own right and slightly distracting if your main interest is actually intelligence. [1] * Another quite common interim goal that I’ve seen a lot and some researchers are pursuing is toddler AGI. Some AI controlled robot that displays qualitatively similar behavior to a 3 year old. What I would say about this is that advocates of this are really talking about a cut down human. With perception, actuation, linguistics, communication, social interaction, problem solving, imagination, all the things 3 year olds do effortlessly. But that’s a massive breadth of capabilities required. That equals an extremely hard problem. [1] Whole Brain Emulation * On the other hand we have the very biological approaches. I put things like Blue Brain and Modha’s SyNAPSE project. Things akin to the Whole Brain Emulation camp. What this camp proposes to do is to look at copying or mimicking the brain to very fine grained detail. Ex., the wiring pathways in a macaque monkey brain. How much is this wiring diagram really telling us about processes or the underlying functions the brain is supporting? The other issue I have with whole brain emulation approaches is that they rely on very intricate imaging techniques. And I think that to get down to the level of detail we’re going to need to do something like whole brain emulation - and what level we go down to is an open question - is synapses enough, do we need to go to calcium channels, the atomic level, what level do we stop at - is actually that our imaging techniques are still decades away from being sophisticated enough. Especially if you want to keep the original brain or the original copy intact. [1] The Systems Neuroscience approach to AGI Motivation * Often I get asked why bother with the brain or neuroscience at all, there are probably many other ways of building intelligence. And that may well be true, but I think there’s a couple of arguments to take neuroscience seriously. First of all, I think it’s likely that the number of possible solutions is actually very small compared to the size of the search space. So if that’s actually true, then it’s probably worth honing in on trying to reverse engineer… at least to some extent, the solution that we know exists. After all, the brain is the only existence proof we have that general intelligence is possible at all. [12] * So what I’m suggesting here is there is a sweet spot in the middle, the systems neuroscience approach, where we’re really interested in is the algorithms the brain is implementing. Not the specific implementation details, but the algorithms and representations. [1] * This maps onto Marr’s three levels analysis. What he suggested is that to fully understand any complex biological system, we need to understand it at three levels - the computational level, the algorithmic level and the implementation level. [1] * The computational level is the what - the goals of the system. What is the system trying to do? * The algorithmic level is the how. The representations and algorithms. How it achieves those goals. * And then finally the implementation details. The medium. The substrate. The physical realization of the system. * If we compare these levels of analysis to Brain Emulation and Cognitive Architectures, Brain Emulation is focused on the implementation level, Cognitive Architectures at the computational level. What we are focused on is the how. [1] Recent Advances in Neuroscience * Rapid advances in Neuroscience: [1] * In the last 10-15 years there has been a revolution in cognitive neuroscience. * There’s been an explosion to hundreds of thousands of citations in cognitive neuroscience. * The proportion of fMRI neurotechnology is growing at the same time as the total. * There are new experiments coming online every year. Recently we’ve had optogenetics where you can have cells firing when you shine laser on them * TMS where you can non-invasively disable pathways in the human brain * There are new analysis tools like multivariate pattern analysis and support vector machines. Machine learning technology. * This has resulted in exponential growth in our understanding of what the brain does. Our understanding is rapidly growing. * We should not just passively observe neuroscience research, but actively conduct neuroscience research which may be useful for AGI questions we may have. * It would be no point talking about [the brain] if we didn’t have amazing new techniques and data streams to actually analyze and actually give us information about what’s going on inside the brain. And pretty much yearly now amazing new techniques get developed, so all the way from optogenetics, connectomics, two-photon microscopy, the list goes on and on now. You know there’s a huge proliferation now of amazing techniques to get over closer to what’s going on in the brain .[12] * This is all resulting in a kind of exponential increase in our understanding of the brain, of course it’s got a long way to go-- I’m not saying we have a full or even close to full understanding of what’s happening in the brain, but there are many clues now, many interesting nuggets of information that if used in the right way I think can be helpful for AI development. [12] * Role of Neuroscience [1] * It can provide direction. Inspiration for new algorithms and architectures. * Ex., hierarchical neural network object recognition systems such as Tomaso Poggio’s HMAX inspired by the primate visual system. * Ex., Navigation systems inspired by hippocampal place cells and entorinal grid cell findings in rodents. * If you go back as far as hebbian learning in neural networks, that was loosely inspired by biology. * It can provide validation. Does this algorithm constitute a viable component of an AGI systems? If we find that the algorithm is implemented in the brain, we can make the case that is is viable. * Ex., the finding in the late 90s that Reinforcement / TD Learning were ubiquitously implemented in the brain through the dopamine reward circuit. * How can it not be of benefit to add systems neuroscience to the mix of approaches to AGI? * The ‘hybrid’ approach: Combine the best of machine learning with the best of neuroscience. [1] * Use the state of the art algorithm in RL, MC and HNNs when we know how to build a component. * When we don’t know how to build a component, continue to push pure machine learning approaches hard but also look to systems neuroscience for solutions. * What does the Systems Neuroscience procedure as related to AGI look like? It means extracting the principles behind an algorithm the brain uses, and creatively re-implementing the algorithm in a computational model, not slavishly copying the way the brain does it. The result should be, if we’re lucky, a state of the art technique and new component to AGI. [1] * One other passion of mine other than artificial intelligence is actually the brain itself and trying to understand our own minds. And I believe that trying to build artificial intelligence and trying to distill intelligence into an artificial construct will be one of the best ways of trying to understand what’s special about the human mind. One of my scientific heros Richard Feynman said: “What I can’t build, I cannot truly understand.” And I really believe that. [3] * So I think there’s sort of two buckets, if you like, of the purpose of neuroscience in terms of how it can help AI development. [12] * So firstly, maybe more obviously, it can provide direction, so research direction and inspiration for new types of algorithms, architectures, and even analysis techniques for analyzing these machine learning systems. And I think we should be looking to neuroscience where we have the least idea or the most uncertainty, if you like, about what to do in machine learning to solve a particular problem or to have a particular capability. And also I think something we’re pushing very hard on at DeepMind is building and taking inspiration from the analysis and visualization tools that are now quite mature in neuroscience, say, for analyzing fMRI images and then applying that in some analogous way to analyzing machine learning systems. Another interesting thing is actually the experimental techniques and design techniques that are kind of standard in things like fMRI, which in terms of controlling for what you’re looking for in an experiment, and that’s something again that in machine learning hasn’t come across yet, and I think it could be very useful, this sort of idea of designing these kinds of experiments that we do in neuroscience. [12] * The second way I think neuroscience can end up being useful is what I call validation testing. So if you have some idea or notion of your favorite type of algorithm, maybe it’s reinforcement learning, and you’re sort of arguing with another machine learner that actually this should constitute or could constitute a viable component of an overall AGI system. How do you decide if this is a reasonable conjecture? And so maybe you go away and you start trying to build a system and probably it doesn’t work straight away. So then you’ve got to decide how much more effort should be put into that. Is it just a question of another few years, and then something viable will happen and something interesting will happen. So there are only very difficult decisions to be made especially if you’re running a large group or you have a large team of where you should put your effort. And I think if you can point to a system in the brain that analogously does that, sort of mimics that algorithm or indeed the algorithm mimics that part of the brain, then I think that can give us confidence that we should put more effort into that area of research, and that ultimately it will all yield some fruit. And in fact that’s what we thought about reinforcement learning and why we committed to that so heavily, because the brain indeed in most biological systems use forms of reinforcement learning like TD learning in order to learn about their environment. [12] * The next question, then, is that if you’re going to take neuroscience seriously… there’s so much neuroscience, so what parts of neuroscience should we be paying attention to? And so here I like quoting David Marr’s three-level analysis, or it should be called Tommy’s three-level analysis as well because I think Tommy was involved heavily in this. And David Marr used to say in the seventies, probably one of the founding fathers of computational neuroscience, that to fully understand a complex biological system, you need to understand on three levels: [12] * the computational level, “what” - the goals of the system * the algorithmic level, “how” - the representations and algorithms the system uses * the implementation level, “”medium” - the physical realization of the system * At DeepMind, we focus on the computational and algorithmic levels when we come to analyze neuroscience in the brain. So collectively I refer to that as systems neuroscience, and really what we’re interested in then is the algorithms and representations and the architectures that the brain uses, and we’re working on all sorts of cutting-edge areas where we’re using systems neuroscience ideas as well as machine learning ideas to try and make progress. [12] * So here’s a summary of some of the areas we’re looking at at the moment: [12] * Representations * Memory * Attention * Concepts * Planning Navigation * Imagination * And in terms of part of the brain, Christoph was talking earlier about prefrontal cortex and high-level cortex, but actually one of the old parts of the brain, the hippocampus, which is a bit of the brain in pink here in the middle-- it turns out to be quite critical for a lot of these capabilities. So let’s talk about those in order. Let’s talk about memory first. * So we’re doubling down on systems neuroscience at DeepMind, and I think there’s an incredible wealth of information and ideas if you know where to look. I think we’re actually just at the beginning of the influence of neuroscience and AI on each other. And one thing we’re especially excited about is developing new tools and techniques borrowed from neuroscience to help with the analysis of ML systems. [12] Memory * So we’ve been experimenting a lot with memory and adding memory to neural networks. So one way you can think about the work we’re doing is you take a classical computer… we implement a sophisticated recurrent neural network, perhaps with some LSTMs, and then we give it a huge memory store that it can learn to control, and then that whole system is differentiable end-to-end and then what we end up with is what we’re dubbing this Neural Turing Machine, in the sense that it has all the components now that a Turing machine would need but its neural and that it’s learnt from input and output examples. [12] * * [above figure] So here’s a cartoon diagram of the architecture. So in the center there, the controller is the recurrent neural network, and then there’s the input-output tape, and it learns by example. It learns to control this very large memory store on the right, and the learns to read and write from that. * “Shrdlu” - a classic problem in AI * We’re not ready to talk about full Shrldlu yet, but we created a kind of mini version where now we’re looking side-on onto this blocks world. The system has to convert the start configuration into the goal configuration by moving one block at a time. So the only moves that are allowed are moving the top block to any of the adjacent columns. So it’s a little bit like a complex Tower of Hanoi problem and it’s actually quite difficult for humans to do in some sort of reasonably optimal way. * * * DQN, and one of the reasons it works so well was partially inspired by neuroscience, and specifically hippocampal replay-- the idea of replaying your experiences so that you can maximize the use of that information-- and that was critical. The use of replay was critical in making DQN tractable in a reasonable amount of time when it was training on these games. [12] Neuroscience’s Relationship to AI * You know, of course, I'm a neuroscientist as well as an AI researcher and I think that by trying to distill artificial intelligence into an algorithmic construct and comparing it to the human mind, that might help us to unlock some of the deepest mysteries of the mind, like consciousness, creativity, and even dreams. [6] * I'd decided to go back to academia and study neuroscience for my PhD, and really this was the final part of the puzzle for me in terms of launching an effort to solve AI. I specifically looked at areas in neuroscience like imagination and memory where we are very bad at understanding how that worked in machine learning in AI so I wanted to gain inspiration from how the brain sold some of these hard problems of intelligence to come up with new inspirations for new types of algorithms that we could implement. [7] * Back in the late 90s, IBM’s Deep Blue beat Garry Kasparov very famously at chess now the thing was of course this was a very impressive technological feat, but actually I came away from that match much more impressed by Garry Kasparov’s mind than the machine itself, because of course Deep Blue, whilst it was amazing at chess, can't do anything else whereas gallery of course can speak multiple languages, drive cars, and do all these other things with his mind, as well as play chess. [7] * As Feynmann said, one of my all time scientific heroes, “What I cannot build I do not understand”. [9] Imagination * Now, it turns out that imagination is actually quite dependent on a region of the brain called the Hippocampus. The Hippocampus is quite a small part of the brain, but it’s a very critical brain region. It’s been known for more than 50 years now that if you damage the Hippocampus you become amnesic. It’s well known that the Hippocampus is vial for episodic memory. But what wasn’t know was, was it involved with imagination? I suspected that it might be. I came across this literature talking about memory as being a reconstructive process rather than something like a video tape. And that’s actually the way memory works. When you remember this lecture tomorrow, it’s not stored like a video tape somewhere in your mind. Actually, you’ll be combining it from all sorts of components or things or other experiences that you’ve had before. What your brain does is pull all those parts together into a coherent whole which is recognized by the rest of your brain as an episodic memory. So I was thinking, well if memory works as a reconstructive process, then if we think about imagination as being a similar process but in this case it’s a constructive process, if we think of memory as trying to put the components that you have together in a way that your brain judges as familiar, then perhaps creativity is the converse of that, where now you’re trying to put together something novel, which your brain judges as unfamiliar. If memory is dependent on the Hippocampus, then maybe imagination is heavily dependent on the same brain structure and the same processes. [9] * Imagination experiments in hippocampus damaged humans [9] min 28 * Rats, Dreaming and Imagination [9] min 31 * Imagination based planning (neural dreaming) [9] min 33 * Imagination Network. This is also something I did in my PhD-- looking at how people imagine and plan for the future. And one thing we decided to test on was hippocampal patients-- patients without a hippocampus, who had damaged the hippocampus: could they imagine? We know they’re amnesiac, we know they don’t have episodic memory, but could they imagine things about the future? So we’d say “Imagine you are lying on a white sandy beach in a beautiful tropical bay. Describe what you see.” And when we looked at the patient descriptions, they were hugely impoverished. Specifically, in their spatial coherence. So what that means it that they couldn’t bind the disparate components of a scene into a coherent whole. We found that the hippocampus is critical, but there’s a network in the brian, a very reliable one, that mediates imagination. It’s also involved in episodic memory. [12] * Hassabis, Kumaran, Vann, Maguire, PNAS 2007 * Can rats imagine? What interesting thing we can think of is also animal intelligence. So I was fascinated to see whether rats can actually imagine. We designed this study that I think was quite elegant and quite simple to show categorically that rats can imagine. We looked at place cells-- for those of you who don’t know, place cells are neurons in the hippocampus of the rat that signal where a rat is in a location. We also know from these rat studies that sequences of these cells play when a rat runs along a trajectory, like a linear track. We also know that when a rat goes to sleep, these trajectories are replayed at very speeded rates, probably to aid learning. [12] * So what we did is design a T-maze-- there’s a barrier on the T-maze that stops the rat from reaching the arms, and the barrier is see-through so the rat can see the arms, just can’t move them. So initially in the first phase, the rat runs up and down the stem of the T-maze. And we show it a reward on the right-hand side. It can see the food pellet but it can’t reach it. Then the rat goes to sleep, and we’re recording from the rat’s brain. This is the critical data that we come back to to analyze. Then in the second session, we remove the barrier, so now the rat can freely move up and down the T-maze. Now on the arms of the T-maze, we can find its place cells, and what we find is that when we go back to analyze the sleep phase, that in fact these place cells were being replayed-- or pre-played if you like-- during that sleep, even though the rat had never experienced it yet. They’d only seen it, maybe it was thinking about moving toward the food pellet. So this is the first time anyone’s ever shown “imagined” place cells. And what was really cool was that there were significantly more preplays, if you like, on the right-hand arm, where the food pellet was, than to the left-hand arm in every one of the four rats. * Olafsdottir, Barry, Saleem, Hassabis, Spiers - eLife 2015 * [12] * Integrated Agents. So if you now combine all the things I’ve been talking about into a single agent, then perhaps we have something that could be deemed “rat-level” AI. So it would be able to do unsupervised 3D vision, attention, have memory, navigate, and perhaps even do imagination-based planning. So the rat is pretty smart, and I think if we can get to this level that would be pretty spectacular, and maybe you can think of the Atari agent as lizard-level. [12] Transfer Learning and Learning Abstract Concepts * The key to flexible general intelligence: “Apply previously learnt knowledge to a new situation.” * 1) Identify the salient features in an environment. * 2) Re-represent those features as an abstract concept. * 3) Select and appropriately apply prior knowledge. [12] * All of these things are very challenging, but step 2 is probably the most c hallenging. And really at the beginning of DeepMind, we’ve identified learning abstract concepts as one of the key breakthroughs that are needed toward getting us toward AGI. * Jennifer Aniston and Halle Berry neurons may be one clue toward this. * Jennifer Anniston Neuron, Halle Berry Neuron - Quiroga et al, Nature 2015 * We did some experiments, I did some during my PhD with my colleague Kumaran, where we looked at conceptual learning in the brain with fMRI studies. In this study here, we had people learning about fractal patterns. They saw two fractal patterns on a TV screen, and they had to predict the weather outcome of the next day, if it would be sunny or rainy. And they learnt this by trial-and-error, and eventually what you learn is that there’s some underlying pattern. So certain fractals, you can ignore where they’re positioned, and other fractals you can ignore their identity, and what’s important is where their position is on the screen. So there are these underlying rules that are independent of the fractals themselves. And we scan people learning this task while they were learning the first task, and once they mastered that, there was the second task, where now we change the fractals to new fractals, but the underlying rules were the same. And we scanned them learning the second task, and they were much faster at learning the second task. And what was really interesting is that the part of the brain that correlated with this increased learning in the second task when we were scanning in the first task was actually the hippocampus. So the amount of activity in the hippocampus in the first task predicted later transfer to the second task. It seems quite strong evidence that the hippocampus is very critical in learning concepts. [12] * Kumaran et al, Neuron (2009) * Here’s another famous study, a sleep study, also related to hippocampal replay and the learning of concepts. So in this one, the concepts you’re trying to learn is this linear hierarchy: A is better than B is better than C is better than D, E, and F. But you don’t see the whole hierarchy at once. You only see the individual pairs. You get shown B and A together, B and C together, and so on. You have to learn through trial-and-error which one is better, but you never see the whole hierarchy. Your brain has to infer this from the individual premise pairs. And what’s interesting about this study is that when they did this, and they tested people twenty minutes later, on the separated pairs, like is B better than D, then people are at chance level. But after 12 hours or 24 hours, and a night of sleep, they’re now up to 80% or 90% successful on separated pairs. But even after 20 minutes, there’s no difference in remembering the premise pairs. So it’s really this extraction of conceptual information that’s happening during sleep. [12] * Walker et al, PNAS (2007) * So what about representations? Here’s another thing that I saw that made me think about… this is a huge clue as to how representations are structured in the brain. So there’s this very famous effect in psychology called the “DRM” effect. And what happens here is you get shown, as a subject, lists of words. You get shown all these study words and are told to memorize them. And then later, you get tested on these words. You get tested on, “did you see the word cat?” And you say “yes” or “no.” But there’s also, critically, some lure words, like the word black or river or cold, which are related to the word lists, but were not shown during the study phase. And actually people get fooled very reliably to say that they’ve seen the word black, or they’ve seen the word river, when they didn’t actually see it in the study phase. And this is an incredibly reliable effect. It was known since the sixties, it was rediscovered in the nineties, it’s been repeated in thousands of psychology experiments, it’s one of the most cited studies. But interestingly, no one had ever thought about doing it under fMRI. * Dark, Cat, White, Coal - Black * Water, Boat, Stream, Lake - River * Snow, Warm, Winter, Ice - Cold * Chadwick et al, under review * So what we decided to do is look at this under fMRI, where you can think about what’s going on in the brain, and why is this happening, and one of the reasons we thought was a partial priming effect. So the idea is that the neural representations underlying these words are maybe partially overlapping. We thought the degree of overlap may predict how much the brain of that individual would be fooled into thinking that they’ve seen that word. Perhaps all these partial primings, these partial overlaps, would end up creating a full priming effect on the word, which is maybe the way the brain judges if you’ve seen something recently or not. And that’s what we’ve found: once we’ve started scanning people and looking at these lists, we found that afterward there was one part of the brain that predicted reliably whether individuals were going to get confused about whether they saw a specific word. And the part of the brain that was predicting that was the anterior temporal lobe. And that’s actually known to be the area involved in semantic dementia. [12] * Chadwick et al, under review Creativity * What about creativity? This is something I get asked about all the time. We’re a long way away from machines being truly creative. I don’t think it’s impossible. And as we understand what this process is, this mysterious process of creativity is, I think it will become more obvious how to implement that in an algorithm. So I just want to show you a couple of things that might surprise you. Let’s just take a picture of the british museum - the front of this building. If you then say to the machine.. and this is a new kind of algorithm that was first implemented at the Max Planck institute in Germany, and then we’ve implemented our own version of this internally. You can give it a artistic picture picture like this Van Gogh picture, and you can say you want that picture in the style of Van Gough. You end up with outputs like this - it’s not ready yet to be hung at the Louvre - but you can sort of start thinking it’s pretty surprising when I saw these things, how coherent the output can be. [9] * There isn’t much creativity here [in style transfer], so it may be that creativity isn’t as mysterious as it seems to us when you ultimately find out what it is. [9] * I talked a lot about Go being a very intuitive and creative game. So what do I mean by that? One operational definition of intuition is that it’s implicit knowledge acquired through experience, but that’s not consciously accessible or expressable. So if you can’t access it consciously, how do you know the knowledge is really there? Well, you can test for this implicit knowledge by behaviorally testing for it. For a Go player, you can give them a Go position, and you can see the quality of the move and the output that they decide to play, and that tells you how good their implicit knowledge, their intuition, is. [14] * Creativity you can define operationally as the ability to synthesize knowledge to produce a novel or original idea. Again, I feel AlphaGo clearly demonstrates these abilities, albeit, of course, within the limited domain of Go. [14] Meta Problems - “Solve Everything Else” * Well I think there are two big issues facing our society today. One is information overload-- we’re just deluged as individuals but also as scientists and business people by the amount of data that’s coming into us every day, and you can see that everywhere from genomics to entertainment. Now personalization technologies are one way to try to deal with that, because generally speaking they’re based on technologies like collaborative filtering, which is really about averaging the wisdom of the crowds. It’s not about tailoring it to the individual. [7] * There’s the problem of system complexity. You know, many of the systems we would like to master as a society like climate, disease, energy economics, even physics are getting so complex now, it's difficult for even the best and the smartest humans to master it in their lifetimes and still leave enough time for them to innovate. So one of the reasons I work on AI and why I think it's going to be one of the most important technologies out there, is that solving AI is potentially a, kind of, meta-solution to all these other problems. We can use it to help us solve all of these other problems. The dream for me, the thing I get most excited about working on AI, is in the future being able to make and create AI scientists, or AI assisted science, making that possible, working in tandem with human experts and human scientists. [6] * Some of the big problems facing us as a society are information overload and system complexity. Everywhere we go now we’re deluged by information. In genomics, big data in general, but in the world of TV, you know entertainment, there are so many TV channels now and modes of watching things, how can you really find what it is that you’re interested in? Personalization is one technology that might help but it doesn’t really work. It’s really based at the moment on wisdom of the crowd collaborative filtering technology. And that doesn’t give you unique recommendations that are unique to what I would call your long tail of interests. [9] * In terms of system complexity, the types of system we’d like to master - climate, disease, energy, macroeconomics, even particle physics - are becoming so complex now that even teams of the brightest human experts are having difficulty comprehending the implications of these systems and actually making predictions about them. * I think solving AI is potentially a meta-solution to all of these problems. If we can solve intelligence, then maybe we can use it to help us as human experts to get a better handle on all these systems. [9] * We could use AI to build better scientists or make AI assisted science possible. [9] Relevance of Reinforcement Learning to AGI * So we think about the intelligence problem within this framework called “reinforcement learning” and it’s a very simple framework to describe, and I’m just going to describe it with this simple diagram. So on the left-hand side here you have the system itself, the AI system, and the AI system finds itself in some kind of environment that it’s trying to achieve a goal in, and that environment could be real world or virtual. [6] * Now the system only interacts with the environment in two ways. So firstly it gets its observation about the environment through its sensory apparatus. We normally use vision at DeepMind, but you could use other modalities, and these observations are always noisy and incomplete. So unlike the game of chess, the real world is actually very noisy and messy, and you never have full information about what’s going on. The job of the system is to build the best model of the world out there, a statistical model of the world out there based on these noisy observations, and once it has that model of the world, the second job of the system is to pick the best action that will get it closest towards its goal from the set of actions that are available to it at that moment in time. Once the system has decided which action it is, it outputs that action, that action gets executed, it may or may not make some change to the environment, and that drives an new observation. [6] Above: Basic framework of reinforcement learning, DeepMind’s main strategy toward AGI. Image source. [6] * This whole system although it’s very simply described in this diagram, it has lots and lots of hidden complexities. If we could solve everything behind this diagram, that would be enough for intelligence. We know that’s enough for intelligence because this is the way all mammals, including humans, learn. In humans it’s the dopamine system in our brains that implements reinforcement learning. [6] Game-playing [embed video] https://www.youtube.com/watch?v=V1eYniJ0Rnk Above: DeepMind’s Deep Q-learning playing ATARI breakout. Source. * [On “Playing Atari with Deep Reinforcement Learning” demo] So when our researchers, the really amazing guys working on this, saw that, it actually shocked them because we hadn’t added in the sort of capabilities like long-term memory which at that point we thought was going to be needed to solve that kind of game in that way. [3] * [On “Playing Atari with Deep Reinforcement Learning” demo] You can see the huge diversity in type of games that the same algorithm can play… So all these games, look how visually different they look. Each game looks completely different in terms of its inputs, but of course it just learns the structure behind the game… One of my favorite ones, boxing. So here, the AI is controlling the white boxer, and it does a bit of sparring, and eventually it gets the other boxer into the corner and then just carries on pummeling it just racking up points-- and it’ll do this forever. It sort of ruthless exploits the weakness in the system that it’s found. [3] * In the nineties when IBM built Deep Blue and they used it to… the machine was able to beat Garry Kasparov the human world chess champion. And I think that was a kind of watershed moment in terms of, you know, showing that machines, if they’re programmed in a specific way, they could be better than a human at an intellectual pursuit-- but only in one narrow area. So the thing I always found quite amusing about, say, Deep Blue, was that, of course, it can play chess to human world champion standard, but it can’t for example play noughts and crosses. [4] * We use computer games as a testing platform for new intelligence because games obviously were designed to be challenging for humans. They involve quite complex perceptual information. So in this case we’re using classic ATARI 8-bit games back from the seventies and eighties. [4] * The second thing we believe in other than reinforcement learning is that a true thinking machine has to be embedded in a sensory motor stream, has to be embedded in a sensory motor reality, but we use actually games as a testing platform for developing our AI algorithms. And we think it’s kind of a perfect platform for that, for many reasons. One there’s unlimited training data-- of course you can run these games for as long as you like. There’s no testing bias because they’ve been designed by other people, not the AI creators. You can test millions of agents in parallel, and progress is very easy to measure because most games have game scores. [7] Dimensions of AI There are at least four dimensions worth thinking about AI in. [15] 1. Learning vs handcrafted 1. Systems that learn directly from raw data, versus handcrafted heuristic systems, which have been specifically preprogrammed with a particular solution to a problem. 2. General vs specific 1. Many systems are handcrafted and built for a particular purpose in mind. What we’re interested in is the idea of generality: one system that out of the box can do a wide range of tasks. 3. Grounded vs logic-based 1. For a true thinking machine to be able to think about and achieve higher-order tasks, they need to be grounded in a sensorimotor reality. They have to experience the world around them through their senses, and ground the knowledge they acquire in their sensorimotor experience. 2. Opposed to that, logic-based systems, or symbolic systems, are hardcoded. The problem with these systems is when they interact with the outside world, with real events, they find it very difficult to map the logic, or knowledge, they have to these real-world messy situations that they find themselves in. A lot of the research in 80s and 90s at MIT were based on these logic systems and expert systems. They only got so far, and then they couldn’t encode things like commonsense, because they weren’t ultimately grounded in perceptual experience. 4. Active vs passive 1. A lot of AI systems are passive observation systems. They get some data, then they try to classify that data in some sense. What we’re interested in instead is active agents, agents that have a goal in mind, that actually direct the actions they undertake, and direct their own learning, so actually decide what they should explore. * At DeepMind, we’re committed to the left=hand side of the above. Future Directions * So where are we going with our research next? We’ve done all this perceptual processing... You can think of the information in the brain having three levels: there’s the perceptual information, sort of the sensory information; there’s the conceptual information which is sort of abstract information; and then there’s symbolic, like language. And if we look at the AI side, or the machine learning side, a lot of work has been done on the symbolic natural language processing side, that’s all this logic networks stuff, and we’re making amazing progress now on the perceptual processing side with deep learning. Computer vision is improving all the time. But what so far is missing is this conceptual layer, learning abstract concepts. One of these inspirations for how to do this is from the brain. [3] * * The hippocampus is critical for memory, but it turns out that another thing the hippocampus is also for is for learning new concepts. So if you have a damaged hippocampus, you can’t learn new concepts. You can still access the concepts you already know, but you can’t learn new concepts. So it seems like the hippocampus might be the gateway to learning new concepts. [3] * [2016] In terms of… what we can’t do at the moment, I mean obviously I agree with unsupervised learning, also the idea of having something like a hippocampal system or function to bring in episodic memory and working memory into deep learning. That’s something we’re working very hard on with Neural Turing Machines, and Facebook are working on as well. I think that’s going to be huge. The other things that we think are really important that haven’t been cracked yet are things like continual learning so that’s the idea of doing one task after another task but then not forgetting and still being able to perform well on the original tasks. So even within the domain of Atari games that’s pretty tricky. And you know that’s a hard problem, and then moving on beyond that, things like transfer learning where not only don’t you overwrite what you already know when you learn a new task, you actually bring to bear your previous knowledge in some useful, appropriate way to the new task. [11] AI Applications * So I think the kinds of things we’ll start seeing in the next five, ten, fifteen years are things like household robots that help sort of clean up around the house or do household chores, maybe care for the elderly. I think we’ll have things like… we’re already starting to see things like self-driving cars, so I think the entire transport system may be revolutionized by automated cars and some kind of automated system. [4] * I think the thing I’m most excited about is how science might change. You talk about things like macroeconomics, climate change, disease, even things like energy. The science of all these things comes down to masses of information, crunching masses of information. It’s too much for any group of humans, even very smart human scientists, to fully understand. We’re probably missing things. So I think we need some aides like artificial intelligence technology to help us make sense and make better use of all this data for the good of society. [4] * Healthcare is actually one of the main application areas we’re working on first. We could probably revoutionize the quality of the care and the efficiency of it. As you say, we’re still using 19th century methods. Having information available to GPs and Surgeons and so on must really help the whole health care space. It’s something we’re looking to get heavily involved in the next few days. [9] * [in 2016] I think there are just so many areas and markets that are going to be revolutionized by deep learning in the next few years and other machine learning technology. So I know there are lots of people in the room working on things like drug discovery. We’re really interested in healthcare applications, with medical imaging being one of those. And I know friends of mine who are using these techniques in finance and hedge funds, if you’re so inclined, I think it’s going to cause a revolution there. There’s been a lot of QA dialogue work, both with us and Facebook and other places. Robotics and continuous control-- I think Ian mentioned earlier in his talk cybersecurity and fraud detection, and really anywhere where if you combine it with RL, kind of sequential decision-making. [11] AI Ethics * I think a lot Hollywood science-fiction has covered the obvious sort of dystopian endpoints. And, you know, I think it’s within our power and our ability to make sure that those things-- that this kind of technology in say twenty, thirty years time doesn’t get used for things like military purposes. We made an agreement with Google when we came in and Google doesn’t do military things and actually this technology will never be used for military or intelligence purposes. [4] * I think like any technology, the technology itself is neutral… it has the capability for great good and also for harm if we use it incorrectly. So I think at some point, certainly when we come to thinking about any type of military use of this… I can categorically say we’re not going to do that. But of course, as you say, that doesn’t guarantee anyone else won’t. I think Google are very…are generally speaking a very responsible company, and that’s one of the reasons that we decided to team up with them with this kind of technology. And actually, an obvious example of how much thought they’ve put into this is, you know, that they agreed with our suggestion that we needed to set up some kind of advisory council to think about the ethics of this kind of technology as it grows. [4] * And of course if we have something this powerful, obviously we need to think about the ethics of the use of it. As with all powerful technologies, and AI is no different from technologies in the past in this regard, I think we have to be very coginisant about using these technologies ethically and responsibly. [9] * And although human level AI is many decades away, I think we should start the debate now. Indeed, that’s what we’re doing, both as our own internal ethics committees, but also by supporting academic work and academic conferences on these topics. [9] * Q: Corporate responsibility vs. your own personal ethics. That’s a big one in terms of where power lies in society. A: Obviously we spent a lot of time doing due diligence ourselves on Google. We had a lot of other options, including stay independent. We decided to join forces with them partly because people high up in Google agreed with things like the ethics committee and thought it was a good idea… Obviously we’ve had our inaugural meeting of the ethics community, there are very big luminaries on that, many of them people who you mention who are worried about that, not just people who are positive. A big part of that is educating everyone on what the real issues are and sort of separating fact from science fiction. And I think that’s the first starting point. And then we can actually get to the hub of the core technical difficult problems there are. And there are some. But I’ve very confident if we apply enough brainpower to it with enough time we’ll solve those problems. [9] * Why don’t we publicize the ethics board? First, we’re very early days, and there’s a lot of scrutiny on this. Once you start making things public, that can change the debate, and I wanted to have a period of behind the scenes calm collected debate before we had public scrutiny. [9] AI Safety * [on Elon’s AI safety views] I mean I know Elon Musk quite well because he was an investor in DeepMind… actually I was the first person who sort of introduced him to the idea of AI, so maybe it’s kind of partially my fault. The thing with Elon and also Stephen Hawking who I recently had the pleasure of meeting-- he invited me up to Cambridge to talk about AI. He had like three hours of questions for me, and it was a very interesting chat, and I think by the end of it he was more reassured. Because the thing about these guys that are mostly talking in the press like Elon, and Stephen Hawking, Steve Wozniak, you know they are amazing obviously in their fields and incredible people but they don’t work on AI and they don’t work close to the engineering side of it, so they don’t see the everyday struggles. I think they tend to read books like Nick Bostrom’s book on superintelligence and kind of get carried away with some sort of almost sci-fi scenarios in there that are many, many decades away. [11] * Having said that, though, I do think there are some legitimate concerns about optimization systems in general that when they get hugely powerful, you know, how do we specify things-- for example, their value systems and their goals robustly, and correctly specify them so they do the things that we want them to do. And I think that those kinds of questions, while the impact of that won’t be for many decades, they’re very tough questions. It’s tough or may be tougher than getting it and then actually building AI, so I think that it is worth thinking about and possibly doing some academic work, I think that’s what Elon is doing. I think there’s no harm in doing that, even though it’s taking some CPU cycles, human cycles away from other research. I think there are plenty of people working directly on AI research, machine learning, so I think having a few people, philosophers, mathematicians, and other things, other people of disciplines thinking about these issues is worthwhile so that we’re ready in the decades to come for any challenges that come up. [11] * Demis’s Neuro Research Deepmind Founding & Early Days * So after a couple of post-docs at MIT and Harvard, I then decided that I had all the ingredients and the components to start DeepMind, and actually attack the AI problem head on. So all these experiences then culminated, in 2010 with me co-founding DeepMind, and the idea behind DeepMind was really to create a kind of Apollo Programme mission for AI. [6] Above: Demis intends DeepMind to be the Apollo Programme mission for AI. Image source. * Now, at DeepMind, we have over 100 research scientists, 100 PhDs, top people in their machine learning fields and neuroscience fields working on solving AI. The type of AI we work on is this neuroscience-inspired AI, so inspired by how the brain works at a very high level, a systems level. [6] * We’re going to… build the world’s first general purpose learning machine, and the two key words here are the words “general” and “learning.” So all the AI that we do at DeepMind involves learning algorithms. So these are algorithms that learn automatically how to master tasks from raw data. They’re not pre-programmed or handcrafted in any way, so that’s unlike most AI out there that you’ve heard of. There’s also the second part. We enforce this idea of wanting the system to be general, so i.e. the same system or same set of algorithms can actually operate across a wide range of tasks. [6] * Now the mission, the way we articulate is, to try and solve intelligence and use that technology to make the world a better place. [7] * So firstly, I think in science, I think the biggest advances in the future in the next few decades are going to come from combining two different fields together at a very deep level. Obviously, for us at DeepMind, we’re doing that by synthesizing machine learning with systems neuroscience. The other thing we do at DeepMind is try to blend the best from two sorts of cultural organizations. So one is take the best from startups, the energy and buzz that they have, and focus, and blend it with what’s best from academia: the long-term thinking and collaborative nature of academia. And try and bring this together to create a new environment that makes scientific research more efficient and more productive. It enhances collaboration and cooperation. [7] * Just like Facebook, at Deepmind and at Google in general we publish pretty much everything we do. We published like 40 papers in the last 18 months, so it’s a lot of work out there. Also we try where we can release code and platforms to open source like the Atari stuff and DQN code, obviously we work on Torch as well like the Facebook guys, so we share that neural network library. I think in general that’s all the reasons Yann said, it’s important that we do that as well and engage with the wider machine learning community. In fact we have tons of collaborations with various academics that are in neuroscience and in machine learning. [11] * So we’ve tried to combine, and take the best from academic institutes and combine that with what’s best from the greatest sort of Silicon Valley-style startups. We’ve tried to create a hybrid environment that maximizes and is sort of optimized for research progress, to try to make it as efficient and productive and collaborative as possible. [12] * We’re interested in algorithms, we call them general-purpose learning algorithms. And we’re only really interested in algorithms that can learn automatically from raw inputs or directly from raw experience so are not pre-programmed or handcrafted in any way. And we’re also interested in the notion of generality, so the idea that the set of same single system or single set of algorithms can operate out of the box across a wide range of tasks, and in fact this relates to our operational definition of intelligence that we use at DeepMind. [12] Research at Smaller Companies/Labs * Maybe I can give some insight into those of you who work for smaller companies or in smaller labs. When we were independent as DeepMind, one of the reasons we chose to work on games was-- apart from a lot of us have background in games, so we knew what games to choose and how to convert them into useful platforms-- also we chose them because we can create as much synthetic data as we’d like, so that’s a very important point. Obviously if you’re not at a big company you can’t access tons of customer data and things like that, which actually is harder than you might think even in a big company, due to privacy and other things that are taken very seriously I’m sure at Facebook, and they are at Google. So if you create sort of synthetic data generators, then you can get around the sort of problem of not having enough data. [11] * I believe a lot of big algorithmic breakthroughs are going to be needed before we solve AI. I don’t think it’s going to be a matter of just purely scaling in terms of computation and data. So I think if you are in academia or in a smaller company, it’s important to pick the right problem. There are many, many open problems… and picking the right ones that aren’t necessarily just a question of getting large amounts of proprietary data or computer power. Of course, having said that compute power does help and that’s one of the reasons we decided to join forces with Google so that we could access their obviously incredible cloud compute that has no doubt accelerated our overall research program. But I think there are many complimentary things that academia can do that industry isn’t doing. [11] Dealing with AI Hype * I think that we have to be careful about overhyping the field, especially since it’s such a hot topic right now obviously, not just in academia but also in venture capitalist and industry sort of circles. So it’s very important that we keep grounded with the actual rigorous results that have been produced and don’t try and overclaim certain results and things. Obviously openly publishing things in rigorous peer review helps with that. There are some companies that don’t do that, some startups that don’t do that, and that makes it harder to assess their claims. I would say, though, that the converse of it is that two AI winters that have happened… I think we understand the causes of those better now, like why those algorithms didn’t work or why they didn’t scale. Obviously a lot of it was not having a lot of compute power. I think there’s nothing wrong with aiming high, but making sure you’re not claiming success that isn’t there. [11] Open Access * Q: DeepMind is working heavily on reinforcement learning. Let’s say one day, in several years, your agent suddenly started talking to you with a certain level of artificial intelligence, or human-level intelligence. Would you make it public or would you keep it to yourself? A: Well it would partly depend on how it’s doing that. So we need to understand why that happened-- but it really wouldn’t be a surprise to us. So I think there is a question of certainly until… for the next foreseeable future we’ll be publishing and releasing everything we see, we build, as the systems become more integrated, there’s sort of a large, big system that has many components, maybe it’s being used commercially, and the capabilities get more and more advanced. Then there are other considerations that have to come in, like potentially people using those technologies in a constrained way, in an ethical way, but what happens if someone uses it in a different way. So I think some of those issues will come in. [11] * We try to be as open as possible about what we’re doing, so we generally publish almost everything that we do. When we can, we open source things as well. Torch is open source. There was a demand from some of the nature reviewers and editors that it would be nice if we could release our code. So we thought about it and we decided in that case that it was fine. But that may not always be the case for the stuff that we do. We’ll obviously have to consider that on a case-by-case basis. But in general where we can we like to engage and support the general academic community. And we think it’s important that knowledge is shared, and I think that’s the way that humanity can advance as quickly as possible. [9] DeepMind’s Future Directions * [In 2015] Now we're moving on to adding things, capabilities, and like concepts, and learning abstract concepts, and long-term memory These are things that are inspired by my work and other people's work in neuroscience and around mimicking the workings of this part of the brain called the hippocampus. Of course, we're not just building these algorithms just to play Atari games We're moving now towards 3D games, Go, simulations and then ultimately real robots at some point, and more near-term, in terms of applications we're using it to improve recommendation systems like on YouTube and also moving into predictive healthcare applications. [6] Google Acquisition * Almost nothing has changed [due to the google acquisition]. That’s the whole point of it - that’s one of the main agreements. Our headquarters is still in the UK, around King’s Cross The whole of Deepmind is still UK side. . We’ve invested in the research team there. We work as a semi-autonomous type of unit. The plus sides are the amount of compute power we have access to has really accelerated our progress, and obviously the other resources at Google. * Q: how has Deepmind affected Google? A: That’s hard to say, as Google is very big. There are thousands of people in Google Research working on machine learning, But we have a more coherent, specific vision than the more applied machine learning that happens elsewhere in Google. We bring together a longer term research focus, and that requires different organizational structures and management structures, some of which is being adopted in Mountain View and Silicon Valley. [9] Deep Reinforcement Learning * We wanted to go beyond narrow AI and work on general purpose learning systems. We came up and pioneered a technique called deep reinforcement learning, which has now become all the rage in AI research. We’ve been working on this since the beginning of DeepMind. It’s combining two techniques together: deep learning, which is hierarchical neural networks, used to perceive the world around them. We combine that with another technique, called reinforcement learning, which is trying to select the right action from the set of available ones to you at the moment, that best get it towards its goal. So we combine these two techniques together and scale them up. [15] * Learning from DQN * AlphaGo * For those of you who don’t know, a Go board is a 19x19 grid. The board starts empty. It’s played between two players, and they take turns to put down these stones, black and white stones. And they put the stones down on the vertices of the board. The idea of the game is to surround your opponent’s pieces, or to surround empty territory. The player that has the most territory wins the game. The rules of Go are very simple, but lead to incredible complexity. In fact, Go has a very long and storied history. It’s what they play in Asia instead of chess. It was originated in China 3000 years ago, and more than 40 million players play it today. There are 2000 professionals. So there are professional Go schools that you go to as a child, if you live in Korea or Japan or China and show promise in the game of Go. You play Go 7 days a week, 12 hours a day. [14] * One idea of the complexity of Go, why it’s so much more complex than chess, one way to illustrate that, is that there are more possible board positions in Go than atoms in the universe: 10^170. Even if you were the use all the compute power in the planet, and run it for a million years, you wouldn’t have enough computer power to calculate in brute force everything about the game. This has why trying to crack the game of Go has taken twenty years beyond the game of chess, beyond DeepBlue. [14] * The way humans get around Go’s complexity is they use their intuition and their instinct. In fact, Go is a much more intuitive game than chess. Chess is much more about calculation, and enumerating the possibilities. Go is much more about feel and instinct and intuition. In fact, if you ask a top Go player why it is they made a particular move in a complex position, often they’ll tell you it just felt right, whereas a great chess player will never say that. They’ll tell you, ‘I’m planning A because I thought B was going to happen and then I’ll do C.’ Now that plan may not pan out, but at least you have an explicit plan in mind, whereas with Go players, it’s much more intuitive and implicit. So even the idea of intuition, trying to codify that, as you all know-- intuition is not normally associated with computers, whereas calculation is. That’s one of the reasons that makes Go so much harder. [15] * So the two really big reasons why Go is hard for a computer to play is that the complexity is actually enormous. Go is one of the most profound games mankind has ever devised. So from these very simple rules, comes this enormous and beautiful complexity. This complexity makes brute force search, a type of search that was used in DeepBlue, intractable. You can’t use those kinds of approaches that were used for chess computers-- they simply won’t scale to Go. So there are two main challenges if you break it down: one is that the search space is enormous, and the second thing that’s even harder is that it was thought to be impossible to write an evaluation function to tell the system which player, black or white, was winning in a particular position. This is incredibly hard for Go. [14] * Let’s dive deeper into the two problems: * in Go, for any particular position, there are an average of 200 possibilities-- 200 possible moves. In chess, by contrast, in an average position there are only 20 moves. Go has an order of magnitude bigger branching factor. That’s what leads to this huge number of possibilities. So it’s much harder to figure out what you should do next, if you’re going to consider all the different possibilities. * And the second thing is the evaluation function: telling the system, how does the system know whether a position is desirable for black or for white. Well for chess, the way you can do that is you handcraft a specific set of rules. You can talk to chess grandmasters, and you can figure out how they make decisions, and you can codify that in a sort of database of rules. And you can use that database to tell you who is winning the game. Now that’s easy in chess because chess has a concept of materiality. So for example, if you want to as a first approximation, you can just add up the value of the pieces on the board. So the queen is worth 9 points, the rook is 5 points, the knight is 3 points, and so on. Whichever side has the most value on the board to a first approximation is winning the game. So that’s already a very useful heuristic. But in Go, there’s no concept of materiality: every piece is worth the same. * Another issue with Go is that Go is a constructive game. What I mean by that is the board starts completely empty, and the position gets built up piece by piece. So the game gets more complex as time goes on. If you compare that to chess, chess is a destructive game. So all the pieces start on the board, and as the game goes on, it simplifies, pieces are taken off the board. So what this means that in Go, if you want to evaluate a current position in the middle game, you have to involve predicting the future, how the game might go-- whereas in chess, you can assess the current position in isolation. You don’t have to worry about the future so much. * And the final issue with Go is that very small changes, one piece moving by one position, can actually change the whole evaluation of the position. Very small changes can make a big global difference. * How do humans deal with this kind of complexity? Because of course, humans play this game to a level of very high complexity and have been playing it for 3000 years. And Go is thought of as a much more intuitive game than a calculation game like chess. Top players rely on their instincts and intuition to figure out all these complexities, and feel their way to the right move, rather than calculate out the right move like a chess player would. In this 3000 year very rich history of Go, there’s this concept of what’s called a divine move, a move so profound that it’s not understandable by humans, and it comes from a divine inspiration. Professional Go players spend their whole careers hoping to play one move that’s worthy of the name. That idea of the divine move is unique to Go. * So how do we overcome these issues and build AlphaGo? We turn to a number of new technologies that we’ve been using at DeepMind, especially deep neural networks, which are all the rage right now, called deep learning. What we did first is we trained two neural networks, and we started off by downloading 100k games off the internet from the Go servers. These are games played by human amateurs on Go servers in Japan and Korea. We trained our first neural network, which we called the policy network to mimic the behavior of the human players. So we gave it a position, and we trained this network to figure out and copy what a human player would do in that position. Now the thing is we don’t want to just mimic strong amateurs. We want our program to be stronger than the best players in the world, the best professionals in the world. So the next stage is we took this first network and we had it playing against itself millions of times on its servers. And each time, learning from its own mistakes, and learning incrementally to make fewer and fewer mistakes with each game. And we used a technology called reinforcement learning, where we each game, it updated its neural network depending on whether it won or lost. After we have this new neural network, we freeze that network, and we play that final network 30 million more times against itself, and that gives us what we call our goal dataset, our final dataset that we need. We use that dataset to train a final neural network, which we call the value network here in pink. This is the second neural network. And what this network does is it takes a board position from Go, and train the new value network to predict, from a particular position, who will win the game, and by how much. So this network in pink here, the value network, is the fabled evaluation function that people thought would be impossible to program for Go. What we did instead is that we sidestepped that problem by having a neural network learn how to evaluate position, rather than us programming it with the definitions. We took these two neural networks, and we applied them together in a search tree. This first neural network, the green one, takes a board position, and outputs probability distributions around which moves are most likely to be played in the current position. The second neural network, the pink network, takes the board position and outputs a single real number between zero and one, which gives a prediction as to which side AlphaGo thinks is winning, and by how much. So if it’s 0.5, that means AlphaGo thinks the game is even. This cuts down the enormous search space of possibilities, more than the atoms in the universe. Alphago calls the policy network first to narrow down the breadth of the search. Instead of searching 200 possibilities from every position, it only searches the top four or five possibilities. Secondly, it uses the value network to cut down the depth of the search. Instead of having to search all the way till the end of the game to find out who won, it can call the evaluation function at any time to give it an estimate of who is winning. * After we built this AlphaGo program, it’s time to test it against the current Go programs, which are built using traditional methods. We found AlphaGo won 99% of its games compared to all the other programs, some of which people have been building for more than 10 years. So then it was time to challenge some of the best humans in the world. We then took on the European champion Fan Hui, who is 3 times European champion. * AlphaGo won 5 million, which is the first time a Go program has ever beaten a professional. This is around a decade before the AI experts and Go experts were expecting this to happen. This was published as the cover article in nature, and it caused a huge stir in the machine learning community. In fact, it’s one of the top 20 most downloaded papers ever in Nature. * Then we took our final step which was to take on the World Champion, South Korean grandmaster Lee Sedol, he’s a legend of the game, he won 18 world titles, and is considered to be the greatest player of the past decade. We took him on in a 1 million dollar challenge match in Seoul. It was a huge event, an incredible press conference with hundreds of journalists, the whole country came to a standstill. All of Asia were watching the game. Incredibly, we managed to win 4-1, and AlphaGo won the match and it was an amazing experience. We were able to beat the top player in the world by four games to one, which is incredibly unexpected. * Not only did AlphaGo win, it played some incredibly creative moves. I just want to show you one example of those moves. Go, as I said, has been going for 3000 years. There are two important lines on a Go board: the third line, which is marked. If you play a piece on a third line, what you’re trying to do is capture territory on the side of the edge of the board. Instead of that, if you play on the fourth line, what you basically are saying to your opponent is you’re trying to get power and influence into the center of the board. And for three thousand years, the received wisdom has been that playing on the third and fourth lines is equal for both players. But what AlphaGo decided to do is play on the fifth line, and take power in the center. This is completely unheard of and against all the principles of playing Go. For a professional, this would be unthinkable. In fact, about 50 moves later, it turned out this move ended up perfecting the fight in the bottom left-hand corner of the board. * He was so surprised by the move that he thought our computer operator had misclicked on the computer screen. That’s how astounded he was. He didn’t even know where to put the piece. * Did we think this move 37 was going to go down in history? Famous moves in Go end up getting names and books written about them and all sorts of things down the years. This move is being analyzed now by the whole Go world for the last three months. * Lee Sedol himself did his own incredible move in game four, this move 78 which will also go down in history, that surprised the computer, AlphaGo. He won game four because of it. Again, another incredibly intuitive and creative move. * What’s really interesting is the cultural impact of this match. 280 million people watched the five games across Asia and the rest of the world. 60 million in China just for game one. There were 35k press articles written about the match. The thing I’m most proud of is that worldwide, online, they were sold out of Go board and Go pieces because so many people around the world wanted to learn and try this great game. * * Above: AlphaGo press conference (source [15]) * Another interesting postscript is that Lee Sedol himself felt that he’d improve because of the match and come up with new ideas. This is the really interesting thing about machine learning and AI helping to improve human performance as well. Since this match, Lee Sedol has gone on an incredible winning streak himself against all his professional opponents, and he said many interesting quotes about how AlphaGo has allowed him to think about new ideas about how to play Go and rejuvenated his passion for the game. * What is the difference between AlphaGo and DeepBlue? Both of these programs beat their world champions and their respective board games. But although AlphaGo is a big achievement, the key point here is how we did it. DeepBlue is preprogrammed with specialized, hand-crafted knowledge that was distilled from chess grandmasters. Whereas AlphaGo used general purpose algorithms, these neural networks and reinforcement learning, to learn how to play Go, and improve itself. It’s a modular system that combined several different aspects together: deep learning, reinforcement learning, Monte Carlo tree search. So it combined pattern recognition with planning. I think that’s what we’re going to see in the future with AI algorithms: combining the strengths of different types of algorithms. Because of this, AlphaGo thinks and plays in a much more human-like way. DeepMind Applications Data Centers * Google has huge data centers, some of the biggest ones in the world, and they consume a lot of power. The latest assessment that I read somewhere is that about 2% of the world’s power is currently used by data centers around the world-- cloud computing. And of course that’s only going to get more as more and more of our compute power is going into the cloud. What we did is use a similar system to AlphaGo, but instead of playing Go, we applied it the cooling system in our data centers to try to increase the energy efficiency of these data centers. What we managed to do, which was quite surprising to the data center engineers, is save 40% of the energy used by the cooling systems, which ends up meaning that the whole data center has 15% less power usage. And obviously that’s worth tens of millions of dollars a year, but it’s also very good for the environment. What AlphaGo does is that it controls all the cooling systems: the fans, the windows, the cooling water, even where the compute power is being routed to within the data center-- all of those things are optimized, and its inputs are all the sensory datas: the thermometers, the temperature gauges, and the fan speeds and so on. Here’s the graph of what happened: this is the power that’s being used in the data center, and you can see the sharp spike down where we turn on this AI system, and now that’s controlling the data center. When we turn it off, it spikes back up, so it makes a huge difference. [15] * What we are thinking now is why don’t we optimize the energy grid at national scale. There’s no need to think about just a data center: there must be huge inefficiencies also at grid scale. If that’s true-- and we’re investigating this now-- maybe we could save 10% of the energy consumption of a country-- which obviously has huge implications for climate change and other things. [15] Speech Synthesis * We now have one of our models called WaveNet-- is one of the best text-to-speech systems in the world. Text-to-speech systems are used for synthesis-- so if you speak to your phone, and it tells you back the result, the voice that it uses is speech synthesis. Most of the time, the current state-of-the-art models are called concatative models, and what they are is they get an actor to speak for 30 hours, lots of dialogue, and then they chop up that dialogue into syllables, and then whey you ask it to speak something new, it stitches back those syllables together. And that’s why they sound a bit warbly, and also why they sound a little bit robotic. Instead of that, WaveNet actually learns to model the raw audio waveform directly. [15] * What our system was able to do was 50% better than current systems. [15] --- *Source: [Original Google Doc](https://docs.google.com/document/d/1n-nXvSt9xpYDGTu6KGyYRTxLUMZa-YUaAlD3kG0q9iU/edit?usp=sharing&sa=D&ust=1596495076379000&usg=AOvVaw24Npd-6BpjErhIR3ruDo15)*