Monthly Archives: January 2018

The Infrastructure Behind Twitter: Scale


  • Hadoop: We have multiple clusters storing over 500 PB divided in four groups (real time, processing, data warehouse and cold storage). Our biggest cluster is over 10k nodes. We run 150k applications and launch 130M containers per day.

  • Manhattan(the backend for Tweets, Direct Messages, Twitter accounts, and more): We run several clusters for different use cases such as large multi tenant, smaller for non common, read only, and read/write for heavy write/heavy read traffic patterns. The read/only cluster handles 10s of millions QPS whereas a read/write cluster handles millions of QPS. The highest performance cluster, our observability cluster, which ingests in every datacenter, handles over tens of million writes.

  • Graph: Our legacy Gizzard/MySQL based sharded cluster for storing our graphs. Flock, our social graph, can manage peaks over tens of million QPS, averaging our MySQL servers to 30k – 45k QPS.

  • Blobstore: Our image, video and large file store where we store hundreds of billions objects.

  • Cache: Our Redis and Memcache clusters: caching our users, timelines, tweets and more.

  • SQL: This includes MySQL, PostgreSQL and Vertica. MySQL/PosgreSQL are used where we need strong consistency, managing ads campaign, ads exchange as well as internal tools. Vertica is a column store often used as a backend for Tableau supporting sales and user organisations.

Meltdown & Spectre – CentOS

This is, for what I’v read about, the two main things that we need to have updated…


Check them trought uname -r and dmesg | grep microcode


[[email protected] www]# systemctl status microcode -l
● microcode.service - Load CPU microcode update
 Loaded: loaded (/usr/lib/systemd/system/microcode.service; enabled; vendor preset: enabled)
 Active: inactive (dead) since Fri 2018-01-05 17:43:08 CET; 1 weeks 6 days ago
 Process: 692 ExecStart=/usr/bin/bash -c grep -l GenuineIntel /proc/cpuinfo | xargs grep -l -E "model[[:space:]]*: 79$" > /dev/null || echo 1 > /sys/devices/system/cpu/microcode/reload (code=exited, status=0/SUCCESS)
 Main PID: 692 (code=exited, status=0/SUCCESS)

Jan 05 17:43:08 sd-56969 systemd[1]: Starting Load CPU microcode update...
Jan 05 17:43:08 sd-56969 systemd[1]: Started Load CPU microcode update.




letsencrypt – Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA.

letsencrypt topic about this issue:

This is due to a security incident with LetsEncrypt

Incident Status Security Issue
Components (Production), (Staging), (Staging)
Locations High Assurance Datacenter 1, High Assurance Datacenter 2

Soundcloud changed their streaming format from 128kbps MP3 to 64kbps Opus


Soundcloud recently changed their streaming format from 128kbps MP3 to 64kbps Opus. This drastically reduces the audio quality and can introduce strange artifacts. I made & released a Chrome Extension to restore the 128kbps MP3s. Give it a try.