Have you looked at Google at all? Cloud Bigtable runs the whole of Google’s Advertising Business and could scale per your requirements. * Use a Queue. RabbitMQ is quite good. Instead of writing to files, generate data/tasks on the queue and have them consumed by more than one client.
read full details at http://ift.tt/1LDS8tM