Web development consultant Jan Moesen alerted the Twitterverse last week about the fact that Apple is running a web crawler, written in the programming language Go.
This is leading many to wonder whether or not Apple is planning to launch its own search engine in the near future. If so, that would be especially bad news for Bing who currently powers Apple’s Siri.
Does anyone know why Apple is running a web crawler (written in @golang, no less)? http://t.co/5JDTZuplQp
— Jan Moesen (@janmoesen) November 6, 2014
On his website, Moesen explains in more detail the information he has gathered so far, saying the first instance of Apple’s web crawler dates back to October 15th:
It looks like someone at Apple is running a web crawler written in Go.
The first hit I could find on <http://catenacycling.com/> was from 17.147.18.33 on the 15th of October:
17.147.18.33 – – [15/Oct/2014:03:05:42 +0200] “GET /robots.txt HTTP/1.1” 301 185 “-” “Mozilla/5.0 (compatible; Fetcher/0.1)”
All IPs starting with “17.” are Apple’s:
NetRange: 17.0.0.0 – 17.255.255.255
CIDR: 17.0.0.0/8
NetName: APPLE-WWNET
A Twitter user by the name of Michael Mittelstadt chimed in saying he has been receiving intermittent amounts of traffic from Apple’s crawler, with short bursts of traffic followed by three week gaps.
That kind of pattern makes it look like Apple is running a search-related experiment, perhaps as R&D for an official search product
Apple is reportedly crawling mobile sites for the most part, while Moesen and others claim to have seen evidence that Apple is crawling the sites of those who have apps in the App Store.
Another possible explanation is that it’s tied to iOS 8 and Apple’s new desktop operating system, Yosemite, and their ability to access web search content.
Of course, everything so far is just speculation until we hear official word from Apple. What do you think it could be?
