There was a time when a visitor to your website was simply a visitor — a person with a browser and, presumably, a reason for stopping by. That era, it seems, has passed.
The term "stealth crawler" has emerged to describe a category of web-scraping technology that operates, as the name suggests, with a certain deliberate anonymity. These are automated programs that extract content from publishers' sites while taking pains to appear as ordinary human visitors — using generic browser identifiers, residential IP addresses, and browsing patterns designed to avoid detection.
The practice has attracted legislative attention. A bill passed by the New York State Legislature last month, along with legislation introduced in the House of Representatives in July, would require such bots to identify themselves. (What a concept: asking someone to say who they are before they take your things.)
For publishers, the issue is not merely philosophical. Lindsay Van Kirk, senior vice president of innovation at People Inc., described the problem at an IAB Tech Lab event in May.
"We've seen things like our content showing up in applications where we do not have a relationship with an application," Ms. Van Kirk said, noting that much of this unauthorized access flows through home proxy networks that allow crawlers to mask their origins.
Her proposed solution is friction — deliberate slowdowns and obstacles that make scraping more expensive for those who attempt it. "Every scraper who has to pay a home proxy network in order to get access to the content is margin that you are taking out of their business," she said. "That is good for publishing."
The News/Media Alliance has taken a more direct position. "Wagging your finger isn't going to work," said Danielle Coffey, the organization's chief executive. "They're bad actors. They behave as such."
The New York bill awaits Governor Kathy Hochul's signature. The federal legislation needs sponsors. Ms. Coffey predicted the process would take time, though she compared the necessity of the work to inspecting a roof for termites.
Sometimes you have to look under the shingles to see what has been eating the house.
Original story published in Digiday: "WTF is a stealth crawler?", by Sara Guaglione