What ? Artificial General Intelligence Is Not as Imminent as You Might Think

Don’t be fooled. Machines may someday be as smart as people, and perhaps even smarter, but the game is far from over. There is still an immense amount of work to be done in making machines that truly can comprehend and reason about the world around them. What we really need right now is less posturing and more basic research.

 

To be sure, there are indeed some ways in which AI truly is making progress—synthetic images look more and more realistic, and speech recognition can often work in noisy environments—but we are still light-years away from general purpose, human-level AI that can understand the true meanings of articles and videos, or deal with unexpected obstacles and interruptions. We are still stuck on precisely the same challenges that academic scientists (including myself) having been pointing out for years: getting AI to be reliable and getting it to cope with unusual circumstances.

Take the recently celebrated Gato, an alleged jack of all trades, and how it captioned an image of a pitcher hurling a baseball. The system returned three different answers: “A baseball player pitching a ball on top of a baseball field,” “A man throwing a baseball at a pitcher on a baseball field” and “A baseball player at bat and a catcher in the dirt during a baseball game.” The first response is correct, but the other two answers include hallucinations of other players that aren’t seen in the image. The system has no idea what is actually in the picture as opposed to what is typical of roughly similar images. Any baseball fan would recognize that this was the pitcher who has just thrown the ball, and not the other way around—and although we expect that a catcher and a batter are nearby, they obviously do not appear in the image.

When systems like DALL-E make mistakes, the result is amusing, but other AI errors create serious problems. To take another example, a Tesla on autopilot recently drove directly towards a human worker carrying a stop sign in the middle of the road, only slowing down when the human driver intervened. The system could recognize humans on their own (as they appeared in the training data) and stop signs in their usual locations (again as they appeared in the trained images), but failed to slow down when confronted by the unusual combination of the two, which put the stop sign in a new and unusual position.

Unfortunately, the fact that these systems still fail to be reliable and struggle with novel circumstances is usually buried in the fine print. Gato worked well on all the tasks DeepMind reported, but rarely as well as other contemporary systems. GPT-3 often creates fluent prose but still struggles with basic arithmetic, and it has so little grip on reality it is prone to creating sentences like “Some experts believe that the act of eating a sock helps the brain to come out of its altered state as a result of meditation,” when no expert ever said any such thing. A cursory look at recent headlines wouldn’t tell you about any of these problems.

The subplot here is that the biggest teams of researchers in AI are no longer to be found in the academy, where peer review used to be coin of the realm, but in corporations. And corporations, unlike universities, have no incentive to play fair. Rather than submitting their splashy new papers to academic scrutiny, they have taken to publication by press release, seducing journalists and sidestepping the peer review process. We know only what the companies want us to know.

In the software industry, there’s a word for this kind of strategy: demoware, software designed to look good for a demo, but not necessarily good enough for the real world. Often, demoware becomes vaporware, announced for shock and awe in order to discourage competitors, but never released at all