Even though they have had their new search out for a few weeks now, Twitter’s new real-time search engine is just now starting to get noticed…which the micro-blogging site thinks is great. The previous back-end was powered by technology based on Summize, but the engineers behind Twitter say that keeping up with all the traffic on that system was getting increasingly hard. They have, because of this fact, switched to using Lucene – a Java-based, open source search engine library.
Twitter’s Engineering Blog did say that they had to do something tweaking to Lucene’s structure to make it a little more real-time search friendly:
“Lucene is great, but in its current form it has several shortcomings for real-time search. That’s why we rewrote big parts of the core in-memory data structures, especially the posting lists, while still supporting Lucene’s standard APIs. This allows us to use Lucene’s search layer almost unmodified. Some of the highlights of our changes include:
• significantly improved garbage collection performance
• lock-free data structures and algorithms
• posting lists, that are traversable in reverse order
• efficient early query termination”
With the sites estimated 12,000 queries per second, the need for a system that could, not only keep up with those demands right now, but allow for further modification if the need arose, was crucial for Twitter. So now that everyone has been made aware of the changes, can you notice them? Do you find it easier to search from things with the new Twitter search engine?
