Updated | Like the rest of us, data scientists can no longer claim their careers are safe from being replaced by robots. These days, computers can sift through immense amounts of data, find patterns and make predictions just like human beings.
Case in point: Massachusetts Institute of Technology researchers have developed an algorithm that can find predictive patterns in unfamiliar data sets. To test its efficacy, they enrolled a computer in three data science competitions against human beings.
The computer, Data Science Machine, finished better than 615 of the 905 total competitors while spending two to 12 hours for what a human could only do over the course of several months.
In two of the competitions, Data Science Machine's results were 94 percent and 96 percent as accurate as the winning submissions.
The brain behind the algorithm, known as Deep Feature Synthesis, is Max Kanter, a graduate student at MIT. His program uses the structure of the database, like tables on a spreadsheet and creates new tables to calculate for possible patterns.
Deep Feature Synthesis calculates the usual mathematical identifiers—like the minima and maxima and the averages—but also "categorical data"—like the days of the week or brand names. Then, the algorithm narrows its field down to what may have a correlation. Finally, it combines the correlations in ways to optimize the accuracy of their predictions.
Kanter and his co-founder, Kalyan Veeramachaneni, plan to present a paper on Data Science Machine at the Institute of Electrical and Electronics Engineers' international conference in Paris next week.
"We view the Data Science Machine as a natural complement to human intelligence," Kanter told MIT News. "There's so much data out there to be analyzed. And right now it's just sitting there not doing anything."
A previous version said Veeramachaneni was a thesis advisor to Kanter. In fact, he was a co-founder of the Deep Feature Synthesis.
Uncommon Knowledge
Newsweek is committed to challenging conventional wisdom and finding connections in the search for common ground.
Newsweek is committed to challenging conventional wisdom and finding connections in the search for common ground.
About the writer
Seung Lee is a San Francisco-based staff writer at Newsweek, who focuses on consumer technology. He has previously worked at the ... Read more
To read how Newsweek uses AI as a newsroom tool, Click here.
