The Internet Is Mostly Bots

Look around you, people of the internet. The bots. They’re everywhere.

Most website visitors aren’t humans, but are instead bots—or, programs built to do automated tasks. They are the worker bees of the internet, and also the henchmen. Some bots help refresh your Facebook feed or figure out how to rank Google search results; other bots impersonate humans and carry out devastating DDoS attacks.

Overall, bots—good and bad—are responsible for 52 percent of web traffic, according to a new report by the security firm Imperva, which issues an annual assessment of bot activity online. The 52-percent stat is significant because it represents a tip of the scales since last year’s report, which found human traffic had overtaken bot traffic for the first time since at least 2012, when Imperva began tracking bot activity online. Now, the latest survey, which is based on an analysis of nearly 17 billion website visits from across 100,000 domains, shows bots are back on top. Not only that, but harmful bots have the edge over helper bots, which were responsible for 29 percent and 23 percent of all web traffic, respectively.

“The most alarming statistic in this report is also the most persistent trend it observes,” writes Igal Zeifman, Imperva’s marketing director, in a blog post about the research. “For the past five years, every third website visitor was an attack bot.”

Put another way: More than 94 percent of the 100,000 domains included in the report experienced at least one bot attack over the 90-day period in Imperva’s study.

Websites that are less popular with humans—as measured by traffic—tended to attract more visits from bots. “Simply put,” Zeifman wrote, “good bots will crawl your website and bad bots will try to hack it regardless of how popular it is with the human folk. They will even keep visiting a domain in absence of all human traffic.”

Though bots are interested in websites even when humans are not, bot activity tends to mirror human behavior online. For instance, the most active helper-bot online is what’s known as a “feed fetcher,” and it’s the kind of bot that helps refresh a person’s Facebook feed on the site’s mobile app. Facebook’s feed fetcher, by itself, accounted for 4.4 percent of all website traffic, according to the report—which is perhaps stunning, but not altogether surprising. Facebook is a behemoth, and its bot traffic illustrates as much.

Overall, Feed fetchers accounted for more than 12 percent of web traffic last year. Search engine bots, commercial data-extracting spiders, and website monitoring bots are among the other helpful bots you’re likely to encounter online. (That is, if you consider the collection of your personal data for advertising purposes to be helpful.)

Data-grabbing bots do their work invisibly, while other bots are easier to spot. In fact, bots and people bump into one another often. Spambots show up in comment sections and Twitter bots clog people’s timelines with everything from marketing, to political campaigning, to social activism, to utter nonsense. These sorts of bots aren’t always pleasant, but they aren’t outright dangerous.

For the real villains, we turn to impersonator bots used for DDoS attacks. They accounted for about 24 percent of overall web traffic last year. Top offenders in this category included Nitol malware, a bot called Cyclone meant to mimic Google’s good search-ranking bots, and Mirai malware—a virus that caused mass internet disruptions in the United States in October.

Other bad bots to contend with include unauthorized-data-scrapers, spambots, and scavengers seeking security vulnerabilities to exploit. Together, they made up about 5 percent of web traffic.

And even though the internet is already  mostly bots, we’re only just beginning to see the Bot Age take shape. According to the market-research firm CB Insights, more than a dozen venture-capital-backed bot startups raised their first round of funding last year.