Open Source News Is the Future of Journalism

Within an hour of an explosion near a hospital in Gaza, The New York Times home page announced, “Israeli Strike Kills Hundreds in Hospital, Palestinians Say.” Within minutes, media outlets around the world rushed out variations of this same headline. As we now know, in light of evidence that was available to anyone who cared to consider it, the headline was incorrect in every way except for “Palestinians Say.”

Following the publication of the initial Times story, what might have been critical meetings between U.S. President Biden and leaders of Israel’s Arab neighbors were called off by their hosts. These headlines no doubt played a role in these cancellations as did the ensuing outrage on the internet and mass demonstrations on the streets of Cairo, Baghdad, New York, London, Paris, and other cities across the world. In a matter of hours, a false and misleading claim promoted on the front page of the most prestigious news outlet in the world had brought to a halt a superpower’s diplomatic efforts during a rapidly escalating war.

None of the standard accounts of the problems with journalism can adequately explain the catastrophic failure that led to the Times and countless other outlets rushing to publish unverified and inflammatory claims about an event of geopolitical importance. A single responsible editor inside the Times newsroom might have prevented this disaster. They could have pointed out that Hamas was not a reliable source, or that it would be impossible to conduct an accurate assessment of the number of deaths in so little time, or that the question of what caused the hospital explosion could be easily resolved by Hamas producing fragments from the munition that caused the explosion (something they still have not done). The rush to be first and the loss of traditional editorial controls have taken over the newsrooms. This is not a matter of individual failings or a problem unique to the Times.

Something much larger is at play, nothing less than a technology-driven disruption of the way reality is conveyed and news is disseminated. The effects of this same disruption can be seen in the press’s coverage of COVID-19, police brutality, political malfeasance, and virtually every other major geopolitical event in recent memory. Each time we witness a characteristic display of alarmism coming from official sources promoting top-down attempts at narrative control, combined with efforts to police and censor decentralized dissent on the grounds that it is “misinformation.”

At the same time the legacy media was misreporting the incident in Gaza, hundreds of people on social media platforms like X, including a mix of technical and military experts and anonymous accounts going by handles such as @GeoConfirmed and @OSINTtechnical, were hard at work collecting, analyzing, and disseminating research on the cause of the explosion. These people, none of whom were professional journalists, used what intelligence officers call “open source intelligence” or OSINT—information available to anyone on the internet. For example, the first video used widely in news reports to show what was assumed to be the hospital explosion came from footage of a live broadcast on Al Jazeera. Over the course of 48 hours, dozens of videos, stills, maps, images, and chats posted on digital platforms like Telegram were analyzed. The consensus from the open-source analysts was that a failed launch of a rocket fired by one of the terrorist groups in Gaza had crashed into a parking lot next to the hospital and exploded, injuring/killing between 10 and 100 people. A tragedy, but nothing even close to the initial reports of an Israeli missile strike directed at a hospital killing hundreds of civilians.

What the incident showed is that in a race to the truth between professional journalists and self-organizing investigators online, all of whom had access to the same public materials, the journalists lost by a long shot. Journalists did not see the raw feed of information as the primary source or material for their news products the way the internet community at large does. While newsrooms could have turned to some of the same analysts who were publishing their findings online, most chose not to because they view such people as less important than the network of sources that legacy journalism has built up over the years. These sources represent the proprietary materials that a newsroom highly values—their intellectual property—and includes officials, ex-officials, domain experts, and back-channels to government spokespeople, both on and off the record. In this case, the official sources were Hamas, Hamas-aligned government agencies, and nongovernmental organizations sympathetic to Hamas’ claims.

What we are seeing is neither simply ideological manipulation nor even incompetence by the media. Members of the media are practicing journalism as they have always done, using the tools that have always worked, acting with the level of integrity we would all expect. What has changed is the world around them and especially the information environment.

In fact, the news industry as we once knew it has been disrupted by the internet and social media, and instead of adapting, the news industry now tries to cling to an approach that leads to trusting Hamas.

In the late 1990s, the “World Wide Web” was going to be more than enough to disrupt media. Instead of dead trees, we would get our news online from online sources. The media responded with—and I’m not making this up—PDF files of the printed paper or websites updated at 5 p.m. East Coast time with the news for tomorrow. The media survived.

As it turned out, online advertising was also useful for these mainstream media brands, though soon they would see less revenue than they had traditionally seen from dead-tree ads. The fracture in the system was the loss of the geographic lock on news distribution. With a web browser, people could get news from anywhere, and no longer relied on the physical delivery of news.

Then we saw the rise of new forms of commerce on the internet. The arrival of Craigslist and eventually Zillow, LinkedIn, and Tinder started to remove the financial moat the news had long held in classified ads, real estate, job listings, and personals. Again, the death of the media was proclaimed as it was “unbundled,” but instead of dying, the national names traded on the trust of their brands to solidify a base of paid subscribers to augment advertising.

The ascension of user-generated content threatened to create a new generation of personalities to deliver news and other substitutes for the daily paper, such as the entertainment sites. Instead, and more indirectly, the likes of YouTube, Facebook, and Twitter provided platforms to distribute all those videos shot on new smartphones. In 1991 when Rodney King was beaten by police officers in Los Angeles, he was “fortunate” enough that someone across the park was recording with a camcorder. Today, firsthand accounts accompany everything; often there are multiple accounts. There are also body cams, doorbell cams, dash cams, and security cams pointed everywhere, each perhaps capturing moments of any debate (just as some caught the explosion in Gaza). Where a single news outlet would have a temporary monopoly on footage to create a story and proprietary moat around it (perhaps even purchasing rights), in today’s environment people who possess these videos post them on their own platforms for free and leave it to their audience to share, comment, and otherwise provide a narrative.

Then a new level of technology arrived that once was only available to the most well-funded and sophisticated newsrooms or governments. Google Maps evolved to street view and satellite view; it now includes actual satellite imagery of every square inch of the Earth. There is data available on traffic, weather, ocean vessels, air traffic control, and more. Using this data still requires expertise but there are retired experts or part-time experts all over the world who quickly lend their experience to matters that concern them.

Newsrooms have increased staffing in this regard, but a handful of experts hardly compares to the resources available to decentralized networks online. We saw this during the early days of the 2019 pandemic when several websites staffed by data-scraping professionals and aided by epidemiologists began to aggregate the fractured and disparate data sources collected by health departments around the world, including the United Nation’s World Health Organization, to provide whole-world views, drill downs, and pivots that quickly became definitive sources.

Taken together, user-generated content and the data sources described above make up much of the raw material analyzed and processed by the OSINT community. As I know from my own experience as a software professional, this surge of “open source” data is precisely the same force that dramatically changed the software industry starting in the mid-1990s.

Finally, the rise of social networks has forever changed the ability for communities to form and communicate. While online forums and communities are as old as the earliest pre-internet networks (such as USENET, ICQ, and more), today’s networks are instrumental in how communities come together to aggregate, analyze, and collaborate on reaching conclusions from OSINT. Many believe social networks to be the source of misinformation (or disinformation) when in fact, as demonstrated by the Gaza incident, the social networks were the solution to misinformation being spread by trusted official sources—precisely the same pattern that has repeatedly played out over the past five years, as with COVID-19 and other stories of profound importance.

The decentralized OSINT community leverages social networks as a point of collaboration and distribution, which are also the attributes that journalists happen to take advantage of on these platforms. But journalism wants to have it both ways: The media wants to use social networks to publicize and distribute its own content and find sources for its own stories—a single, decade-old tweet can trigger a whole news cycle—while at the same time disparaging social media sources and commentators as spreaders of misinformation when the things they are saying challenge the newsroom’s authority to set the agenda.

This approach dooms modern journalism from seeing the value in OSINT as a primary source. They simply think their old processes and sources are superior.

Disruption is often viewed as a single force that quickly vanquishes an old technology. That is certainly how it came to be characterized in Clay Christensen’s seminal work The Innovator’s Dilemma. Behind the technological changes to news and journalism described above was a chorus of “innovator’s dilemma” converts, touting the disruption of old methods and regimes. With more practice and refinement, we now know two things.

First, disruption is less a theory than a law and it is inevitable. When the right set of technologies come together to create a better or simply more efficient future, that future is going to happen. There is no stopping it. That is the destructive force that is disruptive technology. The destruction is the breakdown and reinvention, rebirth, and recreation of the old way.

Second, disruption is almost never a quick or certain event. A new technology does not necessarily mean that change will happen right away, with old technologies and processes becoming instantly obsolete. Digital photography came into existence in the mid-1990s and started to see limited use. It was not until the mid-2000s, when digital cameras were built into mobile phones that were themselves connected to the internet, that we collectively saw an exponential change in imaging that all but led to the demise of stand-alone cameras.

In the framework of disruption, OSINT is a collection of technologies that taken together will upend the process of journalism. It will do so almost precisely the way that open-source software (OSS) upended the world of commercial software development.

Over the past two decades the journalism industry has been making small changes, believing these would be sufficient to avoid big ones. From those first web pages through online ads through running user-generated content within stories and more, news outlets have adapted to being more in line with the internet and less aligned with traditional print. Perhaps the biggest change has been the rapid-release cycle of news stories and updates. This is so common that we don’t notice it but rather expect it, and newsrooms are proud of their work. Unlike the changes to software, there are no “change logs” or comments from those instigating a change, nor is there any discussion history to accompany the changes. Still, the headline writers for the bombed hospital in Gaza were likely excited that they could quickly alter the headline and subhead as they perceive software makers have—”move fast and break things.”

Yet, as with the changes Microsoft made when facing the disruptive forces of open source software, these turn out to be changes around the edges when added up. Nothing has substantially changed in the process of journalism itself. The defining moment for this might not be the Gaza incident, but it came close. In fact, a cynic or skeptic might even argue that the media raced to make the Gaza incident a defining moment in the war, something like the sinking of the RMS Lusitania. Instead, it became a defining moment for journalism itself.

In the past, the media’s proprietary advantage over nonprofessionals was its army of reporters in the field and its top-tier contact list of sources. Good journalists got scoops by being their first to tell you what they saw or by their ability to call a minister, U.N. official, or executive to get the inside story. While the number of field reporters has declined over the past two decades as newsrooms looked to cut costs and elevated opinion and contextual narratives over traditional just-the-facts coverage, these contacts remain a unique and cherished asset of the traditional media that still distinguished it from most of the people writing on X or Substack. Unfortunately for this process, far more people are recording videos, sharing photos, or procuring online data in a way that breaks the spokesperson’s monopoly on providing a “definitive account” of an event, and thus lessens the value of access.

Many of the biggest newsrooms, such as The New York Times, now have dedicated OSINT teams. But as the Times coverage of the hospital explosion demonstrated, these teams are still subordinate to the traditional news process. We know this to be the case because the initial stories about the hospital all followed the standard old school process—“Palestinians say”—and the subsequent corrections also employed those same techniques, simply augmented by comments from the Israeli side. Eventually the Times and others began to build on OSINT, much of that by following coverage as it developed on X, but by then they had stopped breaking news and were functionally reporting the work done by anonymous accounts.

The problem is not just at the Times. Across the media the OSINT teams are treated more as sideshows than primary drivers of the news process. The BBC, for example, has its own OSINT-based show that is separate (at least in the U.S.) from the regular broadcast. The Washington Post has a team of “tens,” according to what its editor said on a podcast this week, but their coverage tracked precisely with The New York Times. CNN has a single on-air digital reporter who appears to track the OSINT practitioners, but not a substantial dedicated team. This is not unlike the way the “social” teams were adjacent to newsrooms as so vividly portrayed in HBO’s The Newsroom.

The story of disruption is the incumbent attempting to avoid the reality of the new technology as long as possible by doing just enough to prevent real transformation. But if anything good comes out of the Gaza hospital incident, it may be that tens of millions of people got a clear view of just how profoundly and irresponsibly wrong their trusted news sources could be on one of the most important stories in the world. We can only guess at the damage they caused: Maybe President Biden would have had some incredible results with Arab leaders; perhaps more relief would have made it to Palestine sooner. We simply don’t know, but it clearly fit into an established pattern, satirized by the humor site Babylon Bee in a headline that read: “CNN Blames Fog Of War For Errant Reporting On Gaza Bombing, Russian Collusion, Jussie Smollett, Covington Catholic, Hunter’s Laptop, The Steele Dossier, Kyle Rittenhouse, Origins Of Covid (Continued In Article Please Click For More).”

The upside of this devastating failure is that it may perhaps force a reckoning, with more people realizing that the old model of journalism is no longer sustainable in a world of OSINT. As OSS began to gain traction with customers, many would say that it did not apply to everything or that only some software would be OSS. Again, that was the incumbents being defensive. It was the case that no single person had a view for how OSS could in fact do everything. Part of creative destruction is that institutions, technologies, and processes are destroyed before there is a full replacement. Incumbents cling to this reality far too long, when in fact they are in the best position to create a replacement.

There will be OSINT-derived journalism forever now, or at least until it is disrupted by something new. It will change the way we report on everything, from the largest global conflicts to the local school board meetings. This does not mean we will only read news in long-nested X threads by cartography experts. Instead, it means new institutions will form that are OSINT-first and exist to expand OSINT, contribute to it, and package it up for consumption. They will invent techniques for not just reporting, not just relying on OSINT to accompany stories, not just verifying what they are told by official spokespeople, but for literally contextualizing and building on OSINT first.

We know this is the case because of how OSS evolved. OSS did not go the direction most everyone said it would. OSS did not replace Windows and Office, directly. Instead, it built a few new companies that proved to be even bigger. OSS formed the foundation of Google, Facebook, Apple’s iPhone (a huge surprise) and a host of Silicon Valley “unicorns.” Each built OSS-first products by enveloping them in new services, data centers, unique capabilities, and new platforms.

Steve Jobs said of the iPhone on the day it was announced: It was the best phone, the best music player, and best internet communications device. All of those were previously stand-alone industries. Now they are all gone. That is creative destruction.

The press and journalism have a special place in our society that is marked and protected by the First Amendment. With this comes a special responsibility. No entity, not even the federal government, can bring additional transparency or accountability to journalism. Most of the critical outlets for journalism are not answerable to shareholders in any material way. There is no regulatory oversight of the press. There is no professional certification of journalists. The whole of the profession is a calling based on trust. Yet, there is no equivalent of “trust but verify” unless the press as a whole develops a means by itself.

The Washington Post says “Democracy dies in darkness.” Right now, the press operates in darkness when it comes to accountability, process, and methods. There has never been an alternative more powerful than the press itself until OSINT.

The events of the past week should serve as a clarion call for change. If the history of technological disruption is any indication, the change is going to happen whether any incumbent wishes to see it happen or not. Open-source journalism is not a question of if, but when … and who.