Nate vs. Leo (and Another Nate): On FiveThirtyEight and The Upshot

Statistics guru Nate Silver has long been considered a master political prognosticator, and for some time he held a virtual monopoly over what has since come to be known as “data journalism.” But around the time that Silver ended his four-year tenure at the New York Times to build an expanded version of his popular FiveThirtyEight blog under the patronage of ESPN, several other players moved into the market: Ezra Klein, formerly of the Washington Post’s Wonkblog, hired “literally everybody” to help him launch Vox.com, and the Times’ David Leonhardt, creator of the now-defunct Economix blog, succeeded Silver as the paper’s nerd-in-chief with the debut of The Upshot in March 2014.

Much has been written about the strengths and weaknesses of each of the sites, including by our own Chris Fegan a few months back. But there’s one story about the data journalism food fight that has largely slipped under the radar: when it came time to start making forecasts for this fall’s elections for the U.S. Senate, The Upshot and its main politics writer, former New Republic contributor Nate Cohn, somehow managed to completely steal Nate Silver’s thunder.

Part of Silver’s unique appeal during the past few campaign seasons stemmed from the fact that he used statistical models to make quantitative forecasts of the outcomes of presidential and senatorial contests, as opposed to simply offering up the sort of qualitative assessments that are a dime a dozen elsewhere in the world of punditry. For all their weaknesses, these models had some important advantages: they allowed for new data to be quickly incorporated into FiveThirtyEight’s view of a race, and they made it possible to systematically attend to a wider range of variables than a mere human could on his own.

When The Upshot debuted “Leo,” its own model for forecasting the results of the 2014 Senate elections, I initially assumed that it would be a cheap knockoff of Silver’s more refined approach, and that I should really just wait for the new incarnation of the FiveThirtyEight model if I wanted to hear from professionals about what we ought to expect come November.

Leo’s methodology page features a Vox-style Q&A that walks readers through the mechanics of the model. Here’s the response to the first question, which asks about how Leo interprets polls:

We focus on the margin between two major candidates, taking steps to make different polls directly comparable. We tweak polls that count registered voters instead of likely ones. We make further adjustments depending on who conducted the poll.

“Well yes,” I thought, when I read that for the first time, “but they probably don’t make as many adjustments as Nate Silver would, like weighting polls based on their sample size or how recent they are.” Then I scrolled down to the next paragraph:

After adjusting the polls, we take a weighted average for each race, giving more weight to polls with a larger sample size and more recent polls (with a poll’s date being especially important the closer we get to Election Day). We also give more weight to a poll when we are more certain about its pollster’s house effect.

“That’s nice,” I chuckled condescendingly as I kept scrolling, “but I bet Leo doesn’t include any of the other sort of data that Nate Silver would, like candidates’ approval ratings or fundraising totals!” False:

For incumbents running for re-election, we consider their approval ratings. We also consider each candidate’s political experience; money raised; the state’s most recent presidential result; national polls on the public’s mood; and whether the election happens in a midterm or presidential year.

“Alright, this is a little better than I expected,” I said to myself, beginning to furrow my brow, “but Leo probably doesn’t account for the fact that the outcomes of races in different states tend to be correlated, which was something Nate Silver always thought was very important to model.” Also false:

We don’t think the races are independent. If the economy starts booming, it will probably help Democrats everywhere. If President Obama bungles an international crisis, Republicans everywhere could benefit. Even on Election Day, our model assumes the races will be correlated to some extent: The pollsters will tend to miss consistently in one direction or the other across the different races.

I finally realized that Leo was not only quite sophisticated, but that it was virtually identical to the old FiveThirtyEight model. In fact, the methodology page basically admits as much:

Leo owes an intellectual debt to earlier models, including those created by political scientists and especially the FiveThirtyEight model, which popularized ideas about adjusting polls, combining polls with other information and national swings.

FiveThirtyEight has been releasing informal reads on the most competitive Senate races at regular intervals for the past several months. Silver has noted that it is the site’s “tradition” to begin transitioning to algorithmic predictions sometime during the summer. This is indeed what FiveThirtyEight did in 2010, when it began publishing results from its model at the end of August. Yet does one data point make a “tradition”? In 2012, Silver’s model was launched at the beginning of June – right around the same time of year that he made this comment.

One obvious response to those (like myself) who would criticize Silver and his team for letting The Upshot beat them to the punch is that unveiling a quantitative model too early might give a false impression about the precision with which the results of an election can be forecasted many months out. Silver may have been worried that readers would fail to realize just how much uncertainty is associated with early predictions, and would put too much stock in seemingly precise numbers that aren’t really all that informative.

But this is always a danger, and Silver dealt with it in 2012 by posting confidence intervals alongside his forecasts of the popular and electoral votes. Moreover, FiveThirtyEight has argued on multiple occasions that early Senate polls have plenty to tell us about November. Here’s Harry Enten, in a piece from April entitled “Early Senate Polls Have Plenty to Tell Us About November”:

More than six months from the midterm elections, current polling and past precedent are competing for our trust. I analyzed which measure is more indicative come November, and it turns out that polls are a more robust metric even though their numbers are still sparse and there’s still so much time remaining before the election.

It’s not clear what Silver can do at this point to reassert his dominance. Maybe he’ll just try to rely on FiveThirtyEight’s superior name recognition. The site has about three times as many Twitter followers as The Upshot, so it’s possible that the efforts of Leo and Nate Cohn will simply be forgotten in the buzz surrounding the eventual rollout of FiveThirtyEight’s own model. But among hardcore political junkies, I can only assume that Silver’s brand has lost some of its luster. Barring a new model that features some truly innovative bells and whistles, it looks like he allowed himself to be totally outflanked by another guy named Nate.

In Silver’s first post at the new FiveThirtyEight, he explained that “we’ve elected to sacrifice something else as opposed to accuracy or accessibility. The sacrifice is speed – we’re rarely going to be the first organization to break news or to comment on a story.” Fair enough! RM prizes depth over quick turnaround too. (This may or may not be an attempt to offer a noble-sounding excuse for our frequent dry spells.) But it’s not clear that FiveThirtyEight is gaining much of anything by taking its time in rolling out its Senate model. Silver and his colleagues have certainly sacrificed speed, but the upshot is that they seem likely to get nothing in return.

 

Advertisements

Initial Impressions of Vox

Earlier this year, I wrote about the coming influx of new journalism sites and opined on their prospects against current media stalwarts.  All of the sites that I referenced at the time are now live and have been operating at full capacity for at least a couple of months.  Here’s what Vox, my pick for the most promising of the new ventures, has accomplished and what it should do differently going forward.

Vox’s Mission Statement

Ezra Klein, Matt Yglesias, and Melissa Bell launched Vox (originally codenamed Project X) in late March.  Their goal: to explain the news and to make complex stories both accessible and engaging.  From their soft-launch site explainer:

In journalism, you’ll sometimes hear articles about hard topics referred to as “vegetables” or “the spinach” — the idea being that readers don’t like those subjects but they should be reading about them anyway. Our view is that there’s no important topic that can’t be made interesting to the audience. If we’re writing about something important — something that matters in people’s lives — and we’ve made it boring then that failure is on us, not on our readers.

They also promised a unique, technology-rich feature set to accomplish this goal:

It’s our job to experiment with all kinds of preparations: Feature articles, traditional news articles, Q&As, FAQs, graphics, videos, visualizations, and even faux-conversations like this one. It means being willing to adopt a tone that isn’t intimidating and being honest that we’re also trying to figure this stuff out. It means developing some innovative new editorial products that let us deliver contextual information more cleanly, clearly, and regularly. Our only promise is that our goal in all cases will be to move people from curiosity to understanding.

Five-Card Stud: Vox’s Card Stack Model

The centerpiece of Vox’s media model has been what it calls “card stacks.”  Card stacks are pop-up interactive features that break down a given issue into topic-based cards, with each card representing a sub-issue or important piece of information related to the topic at hand.  Check out Timothy Lee’s card stack about net neutrality as an example.  The stack walks the reader through the basics of what net neutrality is, the history net neutrality, and the peripheral questions and debates in the news that involve it.

The card stack idea isn’t necessarily original; Bloomberg has been publishing similar “QuickTakes” of given issues for a while now.  But Vox has given primacy to card stacks within its editorial framework and sees them as the bedrock for future stories that pertain to the issue in question.  Most news pieces are relegated to the trash heap of history almost immediately after publication, but Vox’s cards are a promising case study of how certain stories can have continued value in the long run.

Successes

1. The writing is fantastic.  Vox’s journalists have been producing high-quality work from the get-go.  Sarah Kliff, Libby Nelson, Max Fisher, and Tim Lee, among others, have produced really excellent coverage of healthcare, education, foreign policy, and technology, respective.  Matt Yglesias’ review of Capital in the 21st Century is more or less my platonic ideal of the book review: it explains the book’s content and provides a succinct overview of why that content matters.  (I really hope additional book and film reviews like this will be published in the future.)  Dylan Matthews is still Dylan Matthews.  With a team this good, quality is basically a guarantee, which suggests readers will return to the site.

2. The features are innovative and varied.  I love the execution of the card stacks thus far, especially those that dig into the background of huge issues or topics like the Affordable Care Act.  The card stacks are easy to read and are arguably a better, more concise go-to source than Wikipedia or any other information portal, at least for general topic overviews.  Matt (Mazewski, not Yglesias) mentioned to me a little while back that he enjoys the video pieces for their succinctness and quality.  The Q&A breakdowns are a good change of pace and function as mini-card stacks for quick reads.  Vox’s format mix is the best example yet of the new course that innovative internet journalism can plot.

3. Inclusive, unobstructed design.  Credit Melissa Bell’s team for crafting a site that has avoided the design and technology pitfalls of so many other news organizations.  The design is clean yet bold.  Card stacks are simple and easy to navigate.  The mobile site functions flawlessly compared to the desktop site.  Ads are tastefully juxtaposed to pieces and don’t clutter the page.  There are no sponsored content links adorning the end of articles.  Comments are not currently accepted, which is a surprising (and not unwelcome) break from the model nearly every other news site employs.  All of these factors add up to a reading experience that puts the focus on the news and respects the reader by avoiding content gimmicks.  That could be a distinguishing factor as Vox’s reputation solidifies as a haven from the inescapable garbage of Buzzfeed, Forbes, etc.

Room for Improvement

Though Vox’s site design is quite good, the site’s organizational layout needs to be improved as its team begins to aggregate content.   Its current biggest weakness is the lack of a framework for browsing previous stories and posts.  For a site that’s designed to use older stories as a running basis for new events, that’s more than a small problem.  There isn’t even a search bar to locate old articles by topic.

Creating an archive would go a long way towards solving that problem and making Vox’s content more accessible.  But it wouldn’t solve the underlying issue of Vox’s content organization: it’s often unclear why a given story format was paired with a given story.   As Vox continues to grow, it would be helpful to establish consistency in what kinds of stories are posted in each format to better inform readers how each piece fits into the Vox ecosystem.

For example: it makes a lot of sense for Sarah Kliff’s primer on the Affordable Care Act to be published in a card stack, since the ACA is a huge topic and will likely be referenced in a substantial number of future posts.  Based on Vox’s mission statement, it seems like card stacks are the engine for repurposing basic story information for new events and updates.  So why was coverage of Jill Abramson’s firing also published as a card stack?  It was certainly an important event, but it’s highly unlikely this specific incident will be used a as a reference for more than one or two future posts.  If card stacks are most closely linked with recurring utility, why not publish this post as an FAQ or a faux conversation?

I’d take two broad measures to add clarity to Vox’s organizational framework:

1. Use card stacks for stories about institutions, histories, ongoing movements, and key historical events.  Use other features for everything else.

Or, basically, divide content into “things we will use again” and “posts with issue-specific longevity or scope.”  Designate card stacks as the medium for overarching issues that will have a probable chance for future re-use.  My crude measuring stick: imagine it’s December 2015 and we’re recounting key events from 2014 that will probably be relevant going into 2016.  Anything that’s deemed as relevant should be in a card stack; anything that’s deemed less important should be published via a different feature.

So the ACA, the violence in Ukraine, and the coup in Thailand all make sense as card stacks since they’re important issues that will likely have recurring relevance in the medium-term.  The Jill Abramson case and the Bryan Singer allegations would probably have more of a short-term lifespan and should be published as something else, though overarching topics like pay discrimination by gender certainly warrant card stacks.  Of course, any events that magnify in importance could be upgraded to a card stack accordingly.

This might not seem like a significant issue, but as Vox’s published material continues to increase in quantity, it’s important for readers to get a sense of what each story format represents.  I’m also a firm believer that a repository of quality card stacks would be a distinctly monetizable product, an asset collection that could be billed as a friendlier Wikipedia.  It’s important to ensure this product would be of consistently high-level quality rather than a mix of lasting issues and ephemera.  A card stack about the new X-Men film, for example, would have only detracted from the product’s holistic value as it pertains to schools, libraries, and other possible third-party vendors.

2. Create an archive and divide it into four sections: card stacks, non-video features, videos, and opinion pieces.

This would be simple.  Create four product pages that aggregate content for easy navigability.  Card stacks would have their own separate page for the reasons described above.  All other features would be collected elsewhere.  Vox’s video series has been very, very good so far and it would make sense to distinguish video material from all other content, both for easier viewing and the unique ad revenue opportunities associated with the medium.  An opinion page wouldn’t be necessary, but some analysts have bemoaned the loss of Matt and Ezra’s non-“Voxsplaining” musings, and creating an opinion page would ensure Vox has great and clearly delineated takes on both the news and commentary.

Why This Matters

Yglesias recently responded to a critique of Vox from Facebook Content Director Mike Hudack, who argues that Vox’s content decisions betray the team’s original goal of publishing serious journalism:

Personally I hoped that we would find a new home for serious journalism in a format that felt Internet-native and natural to people who grew up interacting with screens instead of interacting with screens from couches with bags of popcorn and a beer to keep their hands busy.

And instead they write stupid stories about how you should wash your jeans instead of freezing them. To be fair their top headline right now is “How a bill made it through the worst Congress ever.” Which is better than “you can’t clean your jeans by freezing them.”

The jeans story is their most read story today. Followed by “What microsoft doesn’t get about tablets” and “Is ’17 People’ really the best West Wing episode?”

It’s hard to tell who’s to blame. But someone should fix this shit.

In response, Yglesias argued that social media sites like Facebook are the reason that this kind of content receives more attention than serious journalism:

As of writing, the jeans story has been shared 1,062 times on Facebook while the DATA Act story has been shared just 242 times. That’s why the jeans story has been read by more people. We featured the DATA Act story much more prominently on our home page, but these days the bulk of web traffic is driven by social media and the bulk of social traffic is driven by Facebook.

Yglesias is correct insofar as social media sites are quickly becoming the go-to source for news and peer-recommended stories.  But even though Vox might give greater homepage weight to stories like the DATA Act article, there is no such primacy given to these lengthier pieces in Vox’s own story history OR on its Facebook and Twitter streams.  Content is publicized with equal weight and filed away with equal weight.

If Vox wants to change this perception, preserving content like card stacks and videos becomes a critical tool in showing which assets and stories have real centrality in Vox’s content bank.  No one should fault Vox for pushing stories on frozen jeans and what “basic” means, since these kinds of traffic-boosters are easy to read and attract eyeballs and revenue.  But Vox should be faulted for not doing enough to distinguish the content it wants to prioritize after publication, especially when so much of it is really damn good.

I mentioned at the beginning of this post that Vox was my pick for the most promising of the new internet sites.  To my mind, Vox’s talent and broad story variety, coupled with its focus on long-term storytelling and backing of the powerful Vox Media brand, will ensure its continued expansion and innovation going forward.  Taking these steps to streamline its content will go a long way in buttressing its mission statement and most effectively leveraging its assets.

The Benefits and Dangers of Personal Brand Journalism

Earlier this month, John Allen, Jr. announced that he would be leaving the National Catholic Reporter to take a new position at The Boston Globe. In its press release about the hiring, the Globe included this interesting nugget: “(Allen) will also help us explore the very real possibility of launching a free-standing publication devoted to Catholicism, drawing in other correspondents and leading voices from near and far.”

Allen’s move comes days after Wonkblog founder Ezra Klein’s pitch to the Washington Post for “an ‘eight-figure’ investment to launch a new site focused on explanatory journalism.” It also comes as Nate Silver, erstwhile New York Times employee and current Editor-in-Chief of FiveThirtyEight, prepares to re-launch his site as a Disney venture.

These and other recent high-profile journalist departures from newspapers (Walt Mossberg from The Wall Street Journal, Glenn Greenwald from The Guardian) are the latest indicators of what Michael Wolff calls the rise of “personal brand journalism.” Wolff argues:

There is a new vision of journalism – call it the auteur school – in which the business shifts from being organized by institutions to being organized around individual journalists with discrete followings… That’s a new notion, this solipsistic brandedness. The old organizational notion in journalism was exactly the opposite. There were never enough readers interested in one subject or one writer so you created a package of many subjects and writers, sharing the attention and the rewards.

Wolff focuses on the profitability of personal brand journalism in his column, but another question arises from his astute observation. Is personal brand journalism beneficial or harmful to the integrity of news reporting?

Newspapers, of course, are not doing well. They are losing both money and readers and have been for almost a decade. It would seem that high-profile reporters like Klein, Silver, and Greenwald would thus benefit the entire industry by leveraging their reputations to deliver high-quality reporting and content. Klein’s demand for $10+ million to start his new site seems outrageous at face value, but it suggests he is serious about providing an excellent and informative product with broad public appeal. This is certainly a good thing.

The problem with “personal brand journalism” is that while its lead figures may intend to provide substantive and objective reporting, their central roles might preclude the balanced, wide-ranging coverage that traditional news organizations provide. This doesn’t refer so much to an implicit ideological slant as to the fact that coverage will be largely aligned with each founder’s primary interests, or a more narrow news spectrum than “old media” organizations. Wonkblog tackles a pretty wide range of policy issues but does so from a specific “wonkish,” graph-based perspective. FiveThirtyEight will utilize data analysis to report on politics, business, sports, and more. Mossberg’s site focuses primarily on technology and culture.

The risk is that each of these individual ventures will lead an increasing number of viewers into more limited sub-strata of news coverage that (perhaps unintentionally) masquerades as a holistic overview of the news. Some people might argue that Wonkblog can be categorized as an objective news source that replaces the need for the Washington Post, but it’s a tough argument to win. That Klein’s name is so integrated with the site’s genesis and production reinforces the idea that it is not a true equivalent to the Post and cannot serve as a 1:1 supplement in its reporting and scope.

This issue isn’t particularly urgent at the moment. Major news sources, which provide broader coverage and demarcate news and opinion, are sufficiently vital to act as complements to personal brand journalism. But the trends are increasingly ominous going forward. The rise of “sponsored content” at sites like Time and The New York Times suggests readers will have to be more vigilant about what information is news and what constitutes well-integrated advertising. If this trend continues, more people will flow to personal brand journalism portals for their news than today, and fewer stories will be sourced from independent news organizations.

This is, in other words, a double edged sword that threatens to strike at the profitability and sustainability of objective news. The incorporation of too much sponsored content risks diluting the reputations major news organizations have developed as impartial and comprehensive, while personal brand journalism features an inherent risk of bias based on limited coverage and the influence of each key figurehead. Personal brand journalism might result in devoted readership bases that help prop up the news industry, but it does not necessarily follow that personal brand journalism is helping save the industry. The actual result might be precisely the opposite.

Of course, it’s not fair to group all of the aforementioned personal brand journalism purveyors in one uniform pool. Different blogs or sites have different aims and some might be more inclined to produce objective reporting than others. We might distinguish the problematic potential of personal brand journalism by breaking its purveyors down into different categories. Wolff distinguishes ventures like The Huffington Post, Buzzfeed, and Business Insider from Klein and Greenwald’s sites by noting that the former are “working at a self-sustaining level.” I’d say it’s more a question of scale. The Huffington Post and Buzzfeed have expanded to a point where the editorial spectrum is no longer dictated by a single person. In contrast, people like Klein and Greenwald will have more significant influence on what their sites publish.

In addition to these two groups, I would add a third category to personal brand journalism in order to recognize a separate type of new media entity: the professional blogger-curator whose only original content is his/her personal analysis of the news. The commonality between the Huffpost / Greenwald groups is that they provide readers with either original reporting or sophisticated data analysis. Larger media organizations like Huffpost might feature editorials as a driving source of traffic, but they do publish independent reporting as a key part of their content stream.

In contrast, the blogger-curator publishes no independent reporting and primarily serves as an aggregator of both original journalism and other blog content. My main point of reference here is Andrew Sullivan and his site, the Dish. Sullivan left The Daily Beast just over a year ago to launch his popular blog as a “freemium,” paywalled independent venture. The Dish is a curated collection of articles supplemented with Sullivan’s commentary; as of now, aside from interviews with Mikey Piro and Dan Savage, the Dish has published no independent journalism or sophisticated data analysis. All of the Dish’s exclusive content has been based off of Sullivan’s own reflections or ruminations.

Sullivan’s work is perhaps the most dangerous case study of how of personal brand journalism could siphon resources and traffic away from news organizations providing original content and reporting. One of Sullivan’s goals since going independent was to provide groundbreaking long-form journalism, but it remains uncertain whether Dish revenue will be able to support this initiative. In the interim, Sullivan is essentially charging readers for personal commentary and article aggregation. That readers are being asked to pay for full access to the Dish means money is flowing away from alternate news organizations that do produce independent reporting.

I do not mean to unfairly malign or downplay the quality of the work that Sullivan produces, nor do I mean to suggest that the Dish’s published material is of negligible value. The Dish is a wonderful site with an unparalleled range of content and commentary and Sullivan has every right to ask people to pay for his analysis of contemporary culture. But it seems hypocritical to hear Sullivan argue how the Dish is “building a future for a whole range of new media on the ashes of the old.” In no way has the Dish, as constructed, been able to replace traditional old media without losing the core principles of journalistic integrity that Sullivan lionizes.

Again, this is not a pressing issue now, and the popularity of personal brand journalism does not mean traditional reporting is soon to be extinct. But the question still remains: is there a way to combine the best of personal brand journalism with the broader objectivity of old media?

Perhaps this is where John Allen’s new deal with the Globe could prove instructive. Allen’s case is unlike any of the other personal brand journalists listed here. He is a reporter first instead of a commentator, and he’s partnering with a major old media institution to create something new. He’s clearly going to be the most important figure in this “free standing” Catholic publication but it’s not implied that he will be a dominant figurehead a la Klein or Silver. In many ways, it seems like Allen is splitting the difference: leveraging his credentials to do something new and specialized within the context of a broader reporting service.

This seems like a good path for old media organizations to follow: hire high-profile, respected reporters (and commentators) to run specialized affiliates that combine intensive reporting and high-quality analysis. A well-run niche site will always have viewers, and tying that site to a major news brand allows readers to access both a broad spectrum of news and a specific range of coverage with personal flair. Wonkblog is probably the most successful example of this now, though Ezra Klein’s initial influence on the site (and his affiliation with things like JournoList) skews it slightly more towards the “personal brand” side of the spectrum. That said, if I were Jeff Bezos, I would pay Klein to start that new site and try something even grander. In this climate, there will be no great rewards without risks.

It is good to see so many “new media” organizations arising amidst the ascendance of personal brand journalism. Readers are better off when there is more choice and better argued interpretations of current events. To this end, there is certainly a place at the journalism table for Sullivan’s passionate opinions, Silver’s statistical analysis, Klein’s wonkish lens, and Greenwald’s political focus. Let’s just hope “old media” organizations will be able to capture the vitality of these branded publications and employ it in the service of continued objective journalism.