Tag Archives: ElasticSearch

Logstash+ElasticSearch+Kibana

Some references for Logstash+ElasticSearch+Kibana

These log files are still hard to use on their own. They are made by a computer for a computer and I am human trying to understand other humans. Logstash, ElasticSearch and Kibana to the rescue! Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (statement humbly borrowed fromhttp://logstash.net). ElasticSearch is a distributed restful search and analytics engine (yet again borrowed, but fromhttp://www.elasticsearch.org/overview/). Kibana is a GUI tool to visualize logs and time-stamped data in realtime (yet again borrowed, but from http://www.elasticsearch.org/overview/kibana/). These three tools make up a pretty good toolkit for creating some graphs and dashboards.

Read more at  http://nhhagen.wordpress.com/2013/11/28/query-log-analysis-using-logstash-elasticsearch-and-kibana/

 

Continue reading Logstash+ElasticSearch+Kibana

HipChat – stats and platform

This is a really good article where they talk about REDIS and ElasticSearch.
http://highscalability.com/blog/2014/1/6/how-hipchat-stores-and-indexes-billions-of-messages-using-el.html

Stats

  • 60 messages per second.

  • 1.2 Billion documents stored

  • 4TB of EBS Raid

  • 8 ElasticSearch servers on AWS

  • 26 front end proxy serves. Double that in backend app servers.

  • 18 people

  • .5 terabytes of search data.

Platform

  • Hosting: AWS EC2 East with 75 Instance currently all Ubuntu 12.04 LTS

  • Database: CouchDB currently for Chat History, transitioning to ElasticSearch.  MySQL-RDS for everything else

  • Caching: Redis

  • Search: ElasticSearch

  • Queue/Workers server: Gearman (queue) and Curler, (worker)

  • Language: Twisted Python (XMPP Server) and PHP (Web front end)

  • System Configure: Open Source Chef + Fabric

  • Code Deployment: Capistrano

  • Monitoring: Sensu and monit pumping alerts to Pagerduty

  • Graphing: statsd + Graphite

    Read more at: http://highscalability.com/blog/2014/1/6/how-hipchat-stores-and-indexes-billions-of-messages-using-el.html

 

Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison

Heres a good – for a n00b guys like me – article about NoSQL databases!

Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison

http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis