The Age of Big Data

February 11, 2012

By Steve Lohr
via nytimes.com

GOOD with numbers? Fascinated by data? The sound you hear is opportunity knocking.

Mo Zhou was snapped up by I.B.M. last summer, as a freshly minted Yale M.B.A., to join the technology company’s fast-growing ranks of data consultants. They help businesses make sense of an explosion of data — Web traffic and social network comments, as well as software and sensors that monitor shipments, suppliers and customers — to guide decisions, trim costs and lift sales. “I’ve always had a love of numbers,” says Ms. Zhou, whose job as a data analyst suits her skills.

To exploit the data flood, America will need many more like her. A report last year by the McKinsey Global Institute, the research arm of the consulting firm, projected that the United States needs 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired.

The impact of data abundance extends well beyond business. Justin Grimmer, for example, is one of the new breed of political scientists. A 28-year-old assistant professor at Stanford, he combined math with political science in his undergraduate and graduate studies, seeing “an opportunity because the discipline is becoming increasingly data-intensive.” His research involves the computer-automated analysis of blog postings, Congressional speeches and press releases, and news articles, looking for insights into how political ideas spread.

The story is similar in fields as varied as science and sports, advertising and public health — a drift toward data-driven discovery and decision-making. “It’s a revolution,” says Gary King, director of Harvard’s Institute for Quantitative Social Science. “We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.”

Welcome to the Age of Big Data. The new megarich of Silicon Valley, first at Google and now Facebook, are masters at harnessing the data of the Web — online searches, posts and messages — with Internet advertising. At the World Economic Forum last month in Davos, Switzerland, Big Data was a marquee topic. A report by the forum, “Big Data, Big Impact,” declared data a new class of economic asset, like currency or gold.

Rick Smolan, creator of the “Day in the Life” photography series, is planning a project later this year, “The Human Face of Big Data,” documenting the collection and uses of data. Mr. Smolan is an enthusiast, saying that Big Data has the potential to be “humanity’s dashboard,” an intelligent tool that can help combat poverty, crime and pollution. Privacy advocates take a dim view, warning that Big Data is Big Brother, in corporate clothing.

What is Big Data? A meme and a marketing term, for sure, but also shorthand for advancing trends in technology that open the door to a new approach to understanding the world and making decisions. There is a lot more data, all the time, growing at 50 percent a year, or more than doubling every two years, estimates IDC, a technology research firm. It’s not just more streams of data, but entirely new ones. For example, there are now countless digital sensors worldwide in industrial equipment, automobiles, electrical meters and shipping crates. They can measure and communicate location, movement, vibration, temperature, humidity, even chemical changes in the air.

Link these communicating sensors to computing intelligence and you see the rise of what is called the Internet of Things or the Industrial Internet. Improved access to information is also fueling the Big Data trend. For example, government data — employment figures and other information — has been steadily migrating onto the Web. In 2009, Washington opened the data doors further by starting Data.gov, a Web site that makes all kinds of government data accessible to the public.

Data is not only becoming more available but also more understandable to computers. Most of the Big Data surge is data in the wild — unruly stuff like words, images and video on the Web and those streams of sensor data. It is called unstructured data and is not typically grist for traditional databases.

But the computer tools for gleaning knowledge and insights from the Internet era’s vast trove of unstructured data are fast gaining ground. At the forefront are the rapidly advancing techniques of artificial intelligence like natural-language processing, pattern recognition and machine learning.

Those artificial-intelligence technologies can be applied in many fields. For example, Google’s search and ad business and its experimental robot cars, which have navigated thousands of miles of California roads, both use a bundle of artificial-intelligence tricks. Both are daunting Big Data challenges, parsing vast quantities of data and making decisions instantaneously.

The wealth of new data, in turn, accelerates advances in computing — a virtuous circle of Big Data. Machine-learning algorithms, for example, learn on data, and the more data, the more the machines learn. Take Siri, the talking, question-answering application in iPhones, which Apple introduced last fall. Its origins go back to a Pentagon research project that was then spun off as a Silicon Valley start-up. Apple bought Siri in 2010, and kept feeding it more data. Now, with people supplying millions of questions, Siri is becoming an increasingly adept personal assistant, offering reminders, weather reports, restaurant suggestions and answers to an expanding universe of questions.

To grasp the potential impact of Big Data, look to the microscope, says Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management. The microscope, invented four centuries ago, allowed people to see and measure things as never before — at the cellular level. It was a revolution in measurement.

Data measurement, Professor Brynjolfsson explains, is the modern equivalent of the microscope. Google searches, Facebook posts and Twitter messages, for example, make it possible to measure behavior and sentiment in fine detail and as it happens.

In business, economics and other fields, Professor Brynjolfsson says, decisions will increasingly be based on data and analysis rather than on experience and intuition. “We can start being a lot more scientific,” he observes.

There is plenty of anecdotal evidence of the payoff from data-first thinking. The best-known is still “Moneyball,” the 2003 book by Michael Lewis, chronicling how the low-budget Oakland A’s massaged data and arcane baseball statistics to spot undervalued players. Heavy data analysis had become standard not only in baseball but also in other sports, including English soccer, well before last year’s movie version of “Moneyball,” starring Brad Pitt.

Retailers, like Walmart and Kohl’s, analyze sales, pricing and economic, demographic and weather data to tailor product selections at particular stores and determine the timing of price markdowns. Shipping companies, like U.P.S., mine data on truck delivery times and traffic patterns to fine-tune routing.

Online dating services, like Match.com, constantly sift through their Web listings of personal characteristics, reactions and communications to improve the algorithms for matching men and women on dates. Police departments across the country, led by New York’s, use computerized mapping and analysis of variables like historical arrest patterns, paydays, sporting events, rainfall and holidays to try to predict likely crime “hot spots” and deploy officers there in advance.

Research by Professor Brynjolfsson and two other colleagues, published last year, suggests that data-guided management is spreading across corporate America and starting to pay off. They studied 179 large companies and found that those adopting “data-driven decision making” achieved productivity gains that were 5 percent to 6 percent higher than other factors could explain.

The predictive power of Big Data is being explored — and shows promise — in fields like public health, economic development and economic forecasting. Researchers have found a spike in Google search requests for terms like “flu symptoms” and “flu treatments” a couple of weeks before there is an increase in flu patients coming to hospital emergency rooms in a region (and emergency room reports usually lag behind visits by two weeks or so).

Global Pulse, a new initiative by the United Nations, wants to leverage Big Data for global development. The group will conduct so-called sentiment analysis of messages in social networks and text messages — using natural-language deciphering software — to help predict job losses, spending reductions or disease outbreaks in a given region. The goal is to use digital early-warning signals to guide assistance programs in advance to, for example, prevent a region from slipping back into poverty.

In economic forecasting, research has shown that trends in increasing or decreasing volumes of housing-related search queries in Google are a more accurate predictor of house sales in the next quarter than the forecasts of real estate economists. The Federal Reserve, among others, has taken notice. In July, the National Bureau of Economic Research is holding a workshop on “Opportunities in Big Data” and its implications for the economics profession.

Big Data is already transforming the study of how social networks function. In the 1960s, Stanley Milgram of Harvard used packages as his research medium in a famous experiment in social connections. He sent packages to volunteers in the Midwest, instructing them to get the packages to strangers in Boston, but not directly; participants could mail a package only to someone they knew. The average number of times a package changed hands was remarkably few, about six. It was a classic demonstration of the “small-world phenomenon,” captured in the popular phrase “six degrees of separation.”

Today, social-network research involves mining huge digital data sets of collective behavior online. Among the findings: people whom you know but don’t communicate with often — “weak ties,” in sociology — are the best sources of tips about job openings. They travel in slightly different social worlds than close friends, so they see opportunities you and your best friends do not.

Researchers can see patterns of influence and peaks in communication on a subject — by following trending hashtags on Twitter, for example. The online fishbowl is a window into the real-time behavior of huge numbers of people. “I look for hot spots in the data, an outbreak of activity that I need to understand,” says Jon Kleinberg, a professor at Cornell. “It’s something you can only do with Big Data.”

Big Data has its perils, to be sure. With huge data sets and fine-grained measurement, statisticians and computer scientists note, there is increased risk of “false discoveries.” The trouble with seeking a meaningful needle in massive haystacks of data, says Trevor Hastie, a statistics professor at Stanford, is that “many bits of straw look like needles.”

Big Data also supplies more raw material for statistical shenanigans and biased fact-finding excursions. It offers a high-tech twist on an old trick: I know the facts, now let’s find ’em. That is, says Rebecca Goldin, a mathematician at George Mason University, “one of the most pernicious uses of data.”

Data is tamed and understood using computer and mathematical models. These models, like metaphors in literature, are explanatory simplifications. They are useful for understanding, but they have their limits. A model might spot a correlation and draw a statistical inference that is unfair or discriminatory, based on online searches, affecting the products, bank loans and health insurance a person is offered, privacy advocates warn.

Despite the caveats, there seems to be no turning back. Data is in the driver’s seat. It’s there, it’s useful and it’s valuable, even hip.

Veteran data analysts tell of friends who were long bored by discussions of their work but now are suddenly curious. “Moneyball” helped, they say, but things have gone way beyond that. “The culture has changed,” says Andrew Gelman, a statistician and political scientist at Columbia University. “There is this idea that numbers and statistics are interesting and fun. It’s cool now.”

A Reason to Major in the Humanities

February 9, 2012

By Catherine Rampell
via economix.blogs.nytimes.com

We’ve written several blog posts about the small share of American college students majoring in the so-called STEM fields — science, technology, engineering and mathematics — with the implicit premise that more bachelor’s degrees in technical fields is both desirable on a macroeconomic level and is professionally advantageous at the individual worker level. A former dean of an engineering school suggests that thinking may be wrong.

Daniel Jelski, a chemistry professor at the State University of New York at New Paltz and previous dean of New Paltz’s School of Science & Engineering, argues in a New Geography essay that humanities degrees may be more important for tomorrow’s job market than is generally believed.

He bases this on three “Laws of Future Employment”:

Law #1: People will get jobs doing things that computers can’t do. Law #2: A global market place will result in lower pay and fewer opportunities for many careers. (But also in cheaper and better products and a higher standard of living for American consumers.) Law #3: Professional people will more likely be freelancers and less likely to have a steady job.

The implications of Laws #1 and #2, he says, are that STEM jobs will not be particularly safe in the future, since he believes they are “easily computerized and tradable.”

The more valuable skill sets, he argues, will be those that computers can’t offer, like empathy and sociability — skills that you might be more likely to learn in an English course than one in linear algebra.

How One Hospital Entices Doctors To Work In Rural America

February 2, 2012

By Peggy Lowe
via npr.org

Recruiting doctors to live and work in rural America is a chronic problem. Most health centers try to attract workers with big salaries and expensive homes.

Shots previously reported that one center in Maine was trying to lure medical students to the countryside for their final two years with the hope that they stick around.

The Ashland Health Clinic, a tiny hospital in southwest Kansas, is trying a different tack — a reverse-recruitment model. It’s called mission-focused medicine, and it’s based on serving problems most commonly found in third-world countries.

Ashland, population 855, sits about a five-hour drive south of Kansas City. It’s one of the last outposts on the Kansas open range, where buffalo still really roam along the rolling, dusty plains. There’s no gas station, unless you count the pump at the farmer’s co-op that uses dial-up for credit card approval. The nearest Starbucks is 160 miles away.

The Ashland clinic has 24 beds. The next closest center is 50 miles north, in Dodge City. (Yeah, the same one from those old cowboy movies.)

So when Benjamin Anderson interviewed for the clinic’s CEO job in 2009, he says the board chairman was exceedingly blunt.

Anderson says the chairman told him, “Ben, our facilities are 55 years old. Our finances are challenged. Our morale is low. Turnover is up. We’ve been without an administrator for six months. We’ve been without a doctor for seven or eight months. We really need this facility in this community. And if we don’t have this facility, we’ll lose our school. And if we don’t have our hospital and our school, this will become a ghost town very quickly.”

That pitch clicked for Anderson. He gave up his physician recruiter job in Dallas and moved with his wife out here to become Ashland’s new CEO.

“I’ve always had to have a job that matters,” he says. “I have to have a position that I know it’s not just a paycheck.”

Anderson is now well known on Ashland’s Main Street. But he knew he needed doctors for the hospital to succeed, and he knew he had to offer something different than the thousands of small towns he was competing with.

So he came up with a novel plan. He offers potential candidates eight weeks off to do missionary work overseas. Because he’s found that a doctor who is willing to sleep on a cot in the Amazon or treat earthquake victims in Haiti is ready to serve in rural Kansas. He calls it mission-focused medicine.

“When you recruit a mission-focused provider, they want to see the ghettos,” he says. “They want to know that there’s no Spanish-speaking provider in more than a one-hour drive. They want to see houses that are falling down, widows that are uncared for. They want to know that there’s need and that by them coming there, they would fill a disparity that would otherwise not be filled. So we reversed it.”

It worked. Last July, Dr. Dan Shuman and his family moved here from the Austin, Texas, area. The difference between here and all the other needy areas was his ability to continue his missionary work in Haiti and Mexico during his eight weeks off. But Shuman says Ashland’s own challenges were equally attractive.

“When you’re primary focus is sort of a mission-based focus, when you get into things in order to try to relieve suffering or work toward eliminating disparities,” Shuman says, “then you want to know about those things. It’s appealing to see opportunities.”

Studies suggest that finding primary care providers in rural areas is at crisis levels. More medical students are specializing, so general practitioners are very hard to find. And those who can deal with the lower pay and isolation in rural areas? Even harder. So Anderson raised the bar by shooting for a higher cause. At every staff meeting, he’s part cheerleader, part chaplain.

Indeed, things are looking up for Ashland. In addition to Shuman, Anderson has recruited a nursing director. And that doc who slept on the cot in the Amazon was recently in Ashland for interviews.
Peggy Lowe is a reporter for Harvest Public Media.

Unemployed College Grads Are Turning to the Army

February 1, 2012

By Liz Dwyer, Education Editor
via www.good.is

Army Recruiting

With the unemployment rate hovering near the 10 percent mark—and hiring freezes and layoffs still the norm—an increasing number of college grads are turning to the one employer who’s always hiring, even in tough economic times: Uncle Sam. The number of bachelor’s degree holders enlisting in a branch of the United States Armed Services is on the rise.

Over the past two years, the Army’s seen the biggest spike in diploma-holding enlistees. In 2010, almost 6,000 college graduates signed up for duty, 2,000 more than in 2008 when the economy still seemed healthy. The Navy saw 1,425 college graduates enlist, up from 1,000 in 2008 and the Air Force bumped its college graduate enlistment up to 900 from 2008′s total of 553.

We may be at war, but Ben Harris, a political science and communications double major from Ohio State University, isn’t dwelling on the possibility of dying in combat. He graduated two years ago, and given the realities of the recession, he had to settle for a job at a “chicken-finger place” and shack up with his parents. Instead of heading off to graduate school to wait out the tough economy—and rack up more student loans—Harris told the Columbus Dispatch that he’s considering joining the Army or the Air Force because, “I’ll get more skills and more education.”

The perks—VA benefits, access to military base stores, preference for government jobs, and a reduction in student loans (the Army will repay up to $65,000 of a soldier’s qualifying student loans, the most of the armed services branches)—are undeniable. Plus, when you enlist with a bachelor’s degree, you enter as an officer, which means you receive higher pay.

The only branch to not see a significant enrollment bump is the Marines. According to Maj. John Caldwell, a spokesman for the Marine Corps Recruiting Command, “Young men and women join our ranks to become a United States Marine. They do not see the Marine Corps as a path to something else but rather as a destination unto itself.”

photo (cc) via Flickr user Boston Public Library

Practitioners’ Forum: What Does the Future of Jobs Mean for Your Community?

January 30, 2012

Over the past two months, we have been engaged in a conversation about the future of jobs with economic development practitioners at the TEDC and IEDC conferences.

Now, we’d like to create an open forum to continue this dialogue beyond the conference setting. In the comments section of this post, you’re invited to respond to the following questions, or pose additional questions for your peers.

How will the “future of jobs” change how you approach economic development?

What mechanisms have you created to support corporations and freelance workers in your community?


Below you’ll find a video of Jon’s recent IGNITE presentation from IEDC’s Leadership Summit in San Antonio. The IGNITE structure allows speakers 5 minutes total to present in the form of 20 slides, with 15 seconds per slide. A brief overview of the presentation follows the slide show.

The Future of Jobs from GIS Planning on Vimeo.

The Future of Jobs from GIS Planning on Vimeo.

This is a discussion about the future of jobs. The idea of what a job is has changed throughout history (and continues to change). Farmers and craftsmen have always had trades, or livelihoods. Since the industrial revolution, a fundamental shift in the nature of jobs has occurred; individuals are employed by entities (corporations) and in return for their labor (9-5), they are compensated (wages) and receive benefits (healthcare, etc.). When unemployment is high, as it has been in the aftermath of the recent recession, we must ask ourselves who should create jobs: the public sector? the private sector? Can the economy continue to grow, even if jobs are not being created? (answer: yes).

The economy grows when value is created. Corporations can create value by increasing productivity (but not necessarily increasing employment), and independent contractors can create value outside of a traditional employee-employer relationship. If we take this thought experiment to its logical extreme, could there be corporations without people on the horizon? Will trade guilds become an organizing structure for independent contractors in a variety of professions?

If jobs are no longer the most useful or accurate measure of economic development success, how can practitioners best promote economic vitality in their communities? Are there mechanisms by which cities, regions, and states can offer resources to corporations and freelancers that will support their ability to create value, regardless of hiring trends or employment status?

We invite you to participate in this conversation in the comments section below.

Unemployment Scars Likely to Last for Years

January 9, 2012

By Ben Casselman
via online.wsj.com

The U.S. job market is showing signs of a sustained recovery. But the country’s prolonged struggle with unemployment will leave scars that are likely to remain for years, if not generations.

The latest labor-market snapshot, out Friday, gave cause for continued, if tepid, optimism. U.S. employers added 200,000 jobs in December, and the unemployment rate ticked down to 8.5%, its lowest level since early 2009.

But economists gathered here for the American Economic Association’s annual convention took a longer and generally dimmer view. Even if recent progress continues, the recession already has had a lasting effect on a generation of workers. Worse, the crisis has laid bare problems in the U.S. labor market that won’t quickly recover when the economy eventually rebounds. And the longer that unemployment remains high, the greater the risk that it will create structural problems that will endure.

The economists here, mostly academics, are studying the causes and effects of the jobs crisis from different angles, and they frequently disagree. Nonetheless, a few common themes emerge.

Long-term ProblemLong-term unemployment may be a bigger problem than high unemployment. Americans have been understandably frustrated by the slow pace of job growth. But economists say much of that slowness is explained by the weak economic growth since the recession ended more than two years ago. In that sense, the problem isn’t the “jobless recovery” but rather that the recovery itself has been so weak. If the recovery gains steam—as some economists believe has been happening in recent months—the growth in jobs should pick up as well.

Unprecedented rates of long-term unemployment could threaten the economy’s recent gains. Some 5.6 million Americans have been out of work at least six months, 3.9 million of them for a year or more. Research shows that the longer people are unemployed, the less likely they are to find jobs. Economists aren’t sure why—to what degree it’s because workers’ skills deteriorate, or because they find ways to cope and give up looking for work, or whether the stigma of being unemployed for so long makes companies unlikely to hire them—but the effect is the same: Many of the people out of work the longest likely will never work again.

The risk, economists say, is that the U.S. will develop an underclass of semipermanently unemployed workers, with severe consequences for productivity, public finances and even social stability. Europe, which faced a similar problem in the 1980s, is still dealing with the consequences.

“We know that once you get into that type of situation it’s very hard to get out,” said Steven Davis, a professor at the University of Chicago’s Booth School of Business.

Many problems predated the recession. The recession caused a dramatic rise in unemployment, but it also revealed deeper challenges that had been brewing for decades. By a wide range of measures, the U.S. labor market has over the past two decades lost much of the edge it enjoyed over other developed countries. The big gains in education in the early 20th century have slowed. Americans are moving less frequently and changing jobs less often, making the job market less flexible. And most critically, a smaller share of Americans are working. The labor force participation rate—the percentage of adults who are working or looking for work—peaked in 2000 and has been falling for more than a decade.

Job applicants
Economists don’t agree on what is behind the shift, although the aging of the baby-boom generation is almost certainly part of the answer. But whatever the reason, the effect is a labor market that recovers less quickly when it runs into trouble, a trend that was evident in the slow pace of hiring after the recessions of the early 1990s and 2000s. The recent recession exacerbated the trends, but it didn’t create them, said Harvard economist Lawrence Katz.

“Even if we had never had the Great Recession/lesser depression and the world had stayed the way it was in early 2007, we were already in a pretty sluggish labor market,” Mr. Katz said.

The recovery is only the beginning. Increased hiring and a falling unemployment rate are good signs, economists say. But even if the unemployment rate were to return to a healthy level by the end of the year—an outcome almost no experts predict—the labor market would still be far from healed. The length and depth of the crisis, unprecedented since the Great Depression, will have a lasting effect on both individuals and the economy as a whole.

For individuals, a growing body of research points to long-term effects of unemployment, especially during recessions. A recent paper by Mr. Davis and Till von Wachter, a Columbia University economist, found that workers who lose their job when unemployment is low—below 6%—lose on average 1.4 years’ worth of earnings, a substantial blow. But those who lose their jobs when unemployment is above 8% lose twice as much, 2.8 years’ worth of their pre-job-loss wages, which the authors call “staggering.”

Mr. von Wachter and colleagues also found that unemployed workers’ earnings fall 1% for each additional month they are out of work, and that those losses can last for years even after they find another job. Other researchers have found that unemployment can take a toll on family relationships, mental and physical health, and even the economic prospects of jobless workers’ children.

“There are people who are going to bear scars for the rest of their careers,” said Henry Farber, an economist at Princeton University.

For the broader economy, the stakes may be even higher. The long-term decline in manufacturing has eroded a major source of stable jobs that pay well. The construction boom of the mid-2000s helped offset those losses, until the housing market collapsed and took home builders with it. And the slow recovery has put a premium on productivity, giving companies incentives to invest in technology that lets them produce more with fewer workers—a trend that has spread from manufacturing into the service sector.

Much of the recent job growth, meanwhile, has come in the health-care and hospitality sectors, which generally employ many low-skilled workers at low wages. Those jobs help shrink the unemployment rolls, but they don’t replace the middle-class jobs that have been lost in the manufacturing and construction sectors.

“A huge issue is going to be the quality of jobs and whether we’ll have a type that generates a shared prosperity,” Mr. Katz said.

The challenge facing the country, then, is not just putting people back to work, but helping to retrain and rehabilitate the long-term unemployed, reversing a multidecade stagnation in the labor market, and finding a new source of jobs to rebuild the middle class. No one in Chicago had any easy solutions.