How to store and analyse a growing number of use cases without too much server stress?

The Elasticsearch & RouteYou Case

Published: December 2016 - Themes: Big Data, Open Source, Elasticsearch.

Do you have a strong, innate desire to rove or travel about? Do you want to discover the most beautiful and interesting trails, tracks, paths, etc. close to home, near your points of interest or, frankly, everywhere you go? Belgian company RouteYou created a platform where you can plan and share your recreational tours as a hiker, jogger, cyclist, biker, inline skater, horseman, etc. Once you’ve chosen your route type, starting point and length, the platform recommends the most beautiful routes in the requested area and on the appropriate soil for your sport. When desired, RouteYou offers even more parameters such as characteristics of the route, theme, event type, score by other users, etc.
_In a few clicks you’ve found the perfect route for your situation. But before you hit the road, don’t forget to click the GPS or print button. Your route will be available in several GPS formats. If you rather take your smartphone with you and let it point out interesting points along your route, just download the RouteYou app. When you’re back home, you’ll be able to share the routes you liked and comment, adapt or improve routes by other users. _“Nice! Let’s go!” shouts your gut, while your technical brain starts shivering in the corner. All that data! Spacial searches! All the stress on the servers in weekends and on holidays! That’s exactly what also RouteYou co-founder Pascal Brackman feared. This, together with RouteYou’s international growing plans and thus hopefully booming user statistics and routes in mind, brought RouteYou to Sizing Servers Lab.

How to search and store precise, spacial and constantly growing data without too much server stress?

Sizing Servers Lab recommended Elasticsearch for RouteYou: Elasticsearch is a distributed, RESTful, open source (Apache License 2.0) search and analytics engine capable of solving a growing number of use cases. It lets you perform and combine many types of searches — structured, unstructured, geo, metric —and Elasticsearch aggregations let you zoom out to explore trends and patterns in your data.

Advantages Elasticsearch

We chose Elasticsearch specifically for this case because on the RouteYou platform you search a complex combination of geographic locations, structured, unstructured and metric data. The data is easy to access and this at a high speed because of the implementation of inverted indices with finite state transducers for full-text querying and BKD trees for storing numeric and geo data. ES can handle thousands of requests per second while the response time still stays under one second because it scales horizontally while automatically managing how indices and queries are distributed across the cluster.

More reasons to work with this open source search and analytics engine: you can interact with Elasticsearch in the programming language you choose, it’s enterprise ready with X-pack (security, monitoring, alerting, reporting, and graph exploration) and there’s an Elasticsearch-Hadoop as well as an Elasticsearch-Spark connector.

Disavantages Elasticsearch

Of course, Elasticsearch is not the perfect answer for every single case. Some downsides were discovered in one of our 2014-2016 research projects “Datamining: From RDBMS to Big Data”. By performing a number of case studies, we collected insights and tried lowering the threshold for SMBs to work with Big Data. RouteYou was one of our cases in this research project.

The first ElasticSearch downside we found was that under (very) high pressure or during network partitions, clients may receive stale data. In other words: new routes will not immediately come though under very high pressure. Sizing Servers Lab defined the maximum stress the platform can handle by stress testing it with our very own stress testing framework vApus. The second ElasticSearch downside was the realization that it takes the same time to upload a brand new document (or route) as it takes to update an already existing document (or route). This means it doesn't deal well with very dynamic data in each document such as "number of times a route is used".

Our conclusion

Apart from these two last points ElasticSearch proves to be a fast and easy to use engine which lets you preform and combine many types of searches. ES growing popularity is no surprise to us at Sizing Servers Lab. But beware to choose the right ES tools for your specific case! Even if your case resembles the RouteYou case, there might be a better solution for you in store. Sizing Servers Lab researches hardware & software performance to provide each individual case with a custom-tailored, cost efficient solution.

Do you want more information about Sizing Servers Lab or this case? Do you want hardware & software recommendations for your specific situation? Do you want to stress test your IT service? Contact Sizing Servers Lab

Like us on Facbook, Follow us on Twitter