新浪微博开放平台中的 Redis 实践
16 likes1,272 views
This document discusses using Redis for social networking applications. Some key points: - Redis is an in-memory database that can dump data to disk and supports many useful data structures like hashes. It provides fast read and write performance. - Redis is used to store relationship data like follower/following lists in hashes with user IDs as keys and fields. MySQL is also used to complement Redis for persistence. - Challenges include high memory usage, slow disk writes, lack of high availability, and performance issues with large hashes. - Potential solutions explored are using MySQL with queues and processors to keep Redis in sync, splitting data between Redis and MySQL more intelligently, and implementing Redis data sharding and replication
新浪微博开放平台中的 Redis 实践
- 1. Redis @ http://weibo.com/tangfl http://fulin.org
- 2. • Redis • Redis • • •
- 3. Redis • in memory (database?) • data can dump to disk • many useful data structure • FAST both read and write • we start use from 2.0, now 2.4
- 4. =feed+ +
- 5. =feed+ + • feed • •
- 6. =feed+ +
- 7. =feed+ + • • •
- 8. =feed+ +
- 9. =feed+ + • • • @ • @ •
- 10. =feed+ +
- 11. =feed+ + • • • @ • @ •
- 12. =feed+ +
- 13. =feed+ + • • • @ • @ •
- 14. =feed+ +
- 15. =feed+ + • • • @ • @ •
- 16. =feed+ +
- 17. =feed+ + •feed • mysql • mc
- 18. =feed+ + •
- 19. =feed+ + • mysql: relation.following • fromuid, touid, addtime • addtime desc select * from following where fromuid=? order by • addtime desc select * from following where touid=? order by • fromuid, touid
- 20. =feed+ + • mysql: relation.following relation.follower • fromuid, touid, addtime • addtime desc select * from following where fromuid=? order by • desc select * from follower where touid=? order by addtime •
- 21. =feed+ + • • • • •
- 22. =feed+ + • ta • ta • ta •
- 23. =feed+ + • • c/java • • •
- 24. =feed+ + • redis • hash : • key : user id • fields : friends ids • value : add time
- 25. =feed+ + • redis • hash : • hset fromuid.following touid addtime • hset touid.follower fromuid addtime • hgetAll fromuid. following • hgetAll touid.follower ? • 11,704,598 @Wed Sep 7 21:46:33 CST 2011
- 26. =feed+ + • hash-max-zip-size • 64 -> 256 1/3 • cpu • hgetAll cost too much cpu • add mc • high delay
- 27. =feed+ +
- 28. =feed+ + • redis • cache ? waste too much mem • storage ? • rdb may lost data • aof r/w too slow, recover too slow • all data in mem, waste money • HA : master slave ? NO WAY • memory fragment
- 29. =feed+ +
- 30. =feed+ + • • mysql binlog >> queue >> Java Processor >> redis • mysql binlog >> trigger >> redis
- 31. =feed+ + • • redis@weibo for now: • TB • growing fast
- 32. =feed+ + • •@ mysql modified + innodb • following list of a user in one column • still under dev
- 33. =feed+ + •
- 34. =feed+ + • • • •@ • •
- 35. =feed+ +
- 36. =feed+ + • • • @ • • • js
- 37. =feed+ + • mc + queue + mysql • mysql • mc mysql • mc
- 38. =feed+ + • • • • •
- 39. =feed+ + • redis • k-v , 100 byte per k-v • mc • • hash , hget pipeline slow
- 40. =feed+ + • redis • rdb ? may lost data • aof ? grow too fast (4G/day) • bgsave/bgrewriteaof influence parent
- 41. =feed+ +
- 42. =feed+ + • redis rolling • • • • key
- 43. =feed+ + • • rediscounter @ • array , not linked list • malloc all mem when start • hash key to position • write disk: asyn & slow down • add position to aof file
- 44. =feed+ + • • rediscounter + innodb • auto roll cold data to disk
- 45. =feed+ + • • @TimYang • @XiaoJunHong • • redis (delay) - mc (throughput) - java hash map
- 46. • • cache • storage
- 47. • • • • + •
- 48. • • • / • •
- 49. • • • rdb or aof
- 50. • • • rehash
- 51. • HA / Cluster • Redis • Jedis
- 52. • CPU • Redis • hset with big hash-max-zip-size • hgetAll • mc
- 53. • • • • •
- 54. Thanks @ http://weibo.com/tangfl http://fulin.org
- 55. Q &A PS. We are hiring ! contact me via @