发帖
雷达卡
https://medium.com/@Pinterest_Engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f
中间提及
[color=rgba(0, 0, 0, 0.8)]We had several NoSQL technologies, all of which eventually broke catastrophically
概括的说就是[color=rgba(0, 0, 0, 0.8)] [color=rgba(0, 0, 0, 0.8)]Pinterest engineer, BlackOps发现他们业务发展太迅速，于是尝试应用NoSQL技术，最终都不稳定，容易系统崩溃，于是他们回到MySQL,并且融入分布式的理念，取得了不错的效果。
我的问题是，到底如何评价MySQL vs NoSQL; 另外他们遭遇的NoSQL不稳定的问题是普遍的，还是因为他们自己在实现上不完善呢？欢迎探讨。
**************************************************************
Sharding Pinterest: How we scaled our MySQL fleet
[color=rgba(0, 0, 0, 0.8)]Marty Weiner | Pinterest engineer, BlackOps
“Shard. Or do not shard. There is no try.”
— Yoda
[color=rgba(0, 0, 0, 0.8)]This is a technical dive into how we split our data across many MySQL servers. We finished launching this sharding approach in early 2012, and it’s still the system we use today to store our core data.
[color=rgba(0, 0, 0, 0.8)]Before we discuss how to split the data, let’s be intimate with our data. Mood lighting, chocolate covered strawberries, Star Trek quotes…
[color=rgba(0, 0, 0, 0.8)]Pinterest is a discovery engine for everything that interests you. From a data perspective, Pinterest is the largest human curated interest graph in the world. There are more than 50 billion Pins that have been saved by Pinners onto one billion boards. People repin and like other Pins (roughly a shallow copy), follow other Pinners, boards and interests, and view a home feed of all the Pinners, boards and interests they follow. Great! Now make it scale!
Growing pains[color=rgba(0, 0, 0, 0.8)]In 2011, we hit traction. By some estimates, we were growing faster than any other previous startup. Around September 2011, every piece of our infrastructure was over capacity. We had several NoSQL technologies, all of which eventually broke catastrophically. We also had a boatload of MySQL slaves we were using for reads, which makes lots of irritating bugs, especially with caching. We re-architected our entire data storage model. To be effective, we carefully crafted our requirements.
Requirements- Our overall system needed to be very stable, easy to operate and scale to the moon. We wanted to support scaling the data store from a small set of boxes initially to many boxes as the site grows.
- All Pinner generated content must be site accessible at all times.
- Support asking for N number of Pins in a board in a deterministic order (such as reverse creation time or user specified ordering). Same for Pinner to likes, Pinner to Pins, etc.
- For simplicity, updates will generally be best effort. To get eventual consistency, you’ll need some additional toys on top, such as a distributed transaction log. It’s fun and (not too) easy!
[color=rgba(0, 0, 0, 0.8)]Since we wanted this data to span multiple databases, we couldn’t use the database’s joins, foreign keys or indexes to gather all data, though they can be used for subqueries that don’t span databases.
[color=rgba(0, 0, 0, 0.8)]We also needed to support load balancing our data. We hated moving data around, especially item by item, because it’s prone to error and makes the system unnecessarily complex. If we had to move data, it was better to move an entire virtual node to a different physical node.
[color=rgba(0, 0, 0, 0.8)]In order for our implementation to mature quickly, we needed the simplest usable solution and VERY stable nodes in our distributed data platform.
[color=rgba(0, 0, 0, 0.8)]All data needed to be replicated to a slave machine for backup, with high availability and dumping to S3 for MapReduce. We only interact with the master in production. You never want to read/write to a slave in production. Slaves lag, which causes strange bugs. Once you’re sharded, there’s generally no advantage to interacting with a slave in production.
[color=rgba(0, 0, 0, 0.8)]Finally, we needed a nice way to generate universally unique IDs (UUID) for all of our objects.
提升卡
置顶卡
沉默卡
变色卡
抢沙发
千斤顶
显身卡
京公网安备 11010802022788号
