The Turing test has always been an approximate benchmark for good AI. In the test, a human is supposed to converse with a machine over text for five minutes; if the human doesn’t realize that they are talking to a machine, then the computer passes as AI “indistinguishable” from human intelligence.
How about a computer that tricked hundreds students for an entire semester?
DON’T MISS: To make the iPhone exciting again, Apple has to launch… an Android phone?
Earlier this year, Georgia Tech professor Ashok Goel noticed he was spread thin for teaching assistants for his computer science course. So Goel programmed IBM’s Watson system to work as an online chatbot, answering some of the 10,000 online questions submitted by students during the course.
As The Washington Post reported, Goel failed to tell any of the students that “Jill Watson” was a computer until after final exams had been turned in. There was some suspicion from students (just look at the forum screenshot above), but overall, Watson passed the test without being called out.
It helped that Jill was trained by the professor and his TAs for months beforehand, carefully programmed to answer questions like a human. More importantly, Watson only answered questions if it was 97% certain it had the correct answer — otherwise, it passed the question off to a human TA.
That limitation means computers won’t be replacing TAs fully any time soon. But it is an interesting, and potentially powerful tool for education. Anyone who has sat through mandatory lectures in college has witnessed the students incapable of reading the course pack, who always ask the most maddening questions. A chatbot could answer that, and also provide a level of personality to massive online courses with tens of thousands of students.
