Troubleshooting Redis- DaeMyung Kang, Kakao
7 likes1,255 views
The document discusses troubleshooting Redis issues encountered by a senior software engineer at Kakao, detailing common problems, their causes, and suggested configurations to optimize performance. Key topics include single-threaded operation, memory fragmentation, command performance, and the significance of monitoring Redis instances. It highlights specific troubleshooting cases related to replication failures, memory allocation issues, and the importance of avoiding default configurations for better reliability.
![KEYS – Iterating all Keys
di = dictGetSafeIterator(c->db->dict);
allkeys = (pattern[0] == '*' && pattern[1] == '0');
while((de = dictNext(di)) != NULL) {
……
stringmatchlen(pattern,plen,key,sdslen(key),0)
}](https://image.slidesharecdn.com/wrdrj78qtlwwujw7hu2u-signature-d9cfff659022967bb0c201e325ed3ec45d6902b09018cea1c56a0f3dae1f8009-poli-160527171746/85/Troubleshooting-Redis-DaeMyung-Kang-Kakao-17-320.jpg)
![FlushAll – Deleting all items
for (i = 0; i < ht->size && ht->used > 0; i++) {
dictEntry *he, *nextHe;
if ((he = ht->table[i]) == NULL) continue;
while(he) {
nextHe = he->next;
dictFreeKey(d, he);
dictFreeVal(d, he);
zfree(he);
ht->used--;
he = nextHe;
}
}](https://image.slidesharecdn.com/wrdrj78qtlwwujw7hu2u-signature-d9cfff659022967bb0c201e325ed3ec45d6902b09018cea1c56a0f3dae1f8009-poli-160527171746/85/Troubleshooting-Redis-DaeMyung-Kang-Kakao-18-320.jpg)
Troubleshooting Redis- DaeMyung Kang, Kakao
- 2. About me •Senior Software Engineer in KAKAO •Redis/Twemproxy Contributor •Redis-doc project merger. •Apache Tajo Commiter
- 3. Kakaostory
- 5. Kakaostory 420M API CALL COUNT
- 6. Kakaostory Service Stack •For Storage •MariaDB(Master/Slave for HA) •Hbase •Cassandra •For Cache •Redis •Arcus • (Memcached variant, opensource, supporting collections)
- 7. Redis 5.2TB, 274 Servers (Arcus: 3.3TB, 137 Servers)
- 8. Why Redis? •As lookaside Cache for service data •Example) •User Profile Information •Feeds •Activities •Friends •Notifications
- 9. Agenda •Single Threaded •Memory Fragmentation •Redis Troubleshooting cases •Redis Monitoring •Redis HA
- 10. Single Threaded
- 11. Redis Event Loop Client #1 Client #2 …… Client #N Redis Event Loop I/O Multiplexing Process Command command #1 command #2
- 12. Only One Command at Once
- 15. Why slow?
- 16. O(n)
- 17. KEYS – Iterating all Keys di = dictGetSafeIterator(c->db->dict); allkeys = (pattern[0] == '*' && pattern[1] == '0'); while((de = dictNext(di)) != NULL) { …… stringmatchlen(pattern,plen,key,sdslen(key),0) }
- 18. FlushAll – Deleting all items for (i = 0; i < ht->size && ht->used > 0; i++) { dictEntry *he, *nextHe; if ((he = ht->table[i]) == NULL) continue; while(he) { nextHe = he->next; dictFreeKey(d, he); dictFreeVal(d, he); zfree(he); ht->used--; he = nextHe; } }
- 19. How slow?
- 20. Command Item Count Time flushall 1,000,000 1000ms(1 second) FlushAll
- 21. Delete collections Item Count Time list 1,000,000 1000ms(1 second) set 1,000,000 1000ms(1 second) Sorted set 1,000,000 1000ms(1 second) hash 1,000,000 1000ms(1 second) You can use Xscan commands from 2.8.x
- 22. Using Multiple Instances in a Physical Server (can use more cpus)
- 23. Fork for Creating RDB, AOF Rewrite
- 24. Maximum 2x Memory Disk IO CPU Load/Usage
- 25. CPU 4 core, 32G Memory Mem: 24G Mem: 8G Mem: 8G Mem: 8G more Reliable
- 26. Set CPU Affinity using taskset
- 27. Divide NIC Interrupt CPU and Redis Process CPU
- 30. Memory Fragmentation #2 Used_memory RSS Starting to use Arcus at this case
- 32. Problem #1 KEYS
- 35. Slowlog get 10
- 37. Using Scan
- 39. Scan #1
- 40. Scan #2
- 41. Scan #3
- 42. Problem #2 All Write Commands Fail
- 43. “MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.”
- 44. Reason if (((server.stop_writes_on_bgsave_err && server.saveparamslen > 0 && server.lastbgsave_status == C_ERR) || server.aof_last_write_status == C_ERR) && server.masterhost == NULL && (c->cmd->flags & CMD_WRITE || c->cmd->proc == pingCommand)) { … }
- 46. Problem #3 Using Default Option
- 47. Redis as Cache
- 48. SAVE 900 1 SAVE 300 10 SAVE 60 10000
- 49. Heavy Disk IO High Cpu Load with creating RDB
- 50. Config set SAVE “”
- 51. Problem #4 Using Swap Memory
- 52. Redis using 28G on single 32G machine
- 54. Monitor Redis Server and keep within bounds
- 55. Problem #5 Simultaneous AOF Rewrite
- 56. A 256GB Single Machine Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB
- 57. Simultaneous AOF Rewrite Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB Redis 26GB AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite AOF Rewrite
- 58. Stop all AOF Rewrites
- 59. Turn off Automatic AOF Rewrite
- 61. Manually Run AOF Rewrite
- 62. Problem #6 Replication is Broken with Network Line Failure
- 63. All redis replication are broken by Network line failure
- 64. What Happens if network is recovered
- 65. Replication Master Slave replicationCron Health check Periodically
- 66. All slaves automatically try to reconnect to master.
- 67. Slave of no one
- 69. Permission
- 70. Memory Allocation Fail sysctl vm.overcommit_memory=1
- 73. config set client-output-buffer-limit "slave 1024mb 1024mb 60"
- 74. Problem #8 Hash Table Expansion
- 75. Redis Dict – Hash Table Expansion #1
- 76. Redis Dict – Hash Table Expansion #2
- 77. Redis Dict – Hash Table Expansion #3
- 78. Grows by twice
- 80. 1 Billion items
- 81. 1,000,000,000 * 4 = 4G
- 82. Maxmemory = 16G Used_memory = 12G
- 83. Hash Table Expansion is needed.
- 84. 4G * 2 = 8G. You need 20G(12G + 8G)
- 86. Need a feature that can Set Initial size of Hash Table (Not Supported) https://github.com/antirez/redis/pull/2812
- 87. Redis Monitoring
- 88. Monitoring is important as much as Management
- 89. Redis Monitoring Metrics Factor System or Redis Info CPU Usage, Load System Network Inbound/outbound System Client connections Maxclient setting Info Key size Processed commands Redis Memory Usage, RSS(very Important) Redis Disk Usage, IO System Expired Keys, Evicted Keys Redis
- 90. Redis HA
- 91. Using DNS for Failover
- 92. Private Internal DNS Server with TTL 0
- 93. DNS HA Flow Detect A Redis Failure Change B can write Change DNS A with B Send A Client Kill New clients Will connect to B B Config rewrite
- 97. Zookeeper
- 98. Zookeeper with Redis Information
- 99. Zookeeper with Redis Application Servers ZooKeeper Redis Shard-1 Redis Shard-2 Redis Shard-3 Redis Cluster Monitor Get Redis Shard Information Health Check Update Shard Info Event: Node Add or Remove, Master change
- 100. Summary •Redis is Single Threaded •Creating RDB or AOF Rewrite is expensive •Don’t use KEYS command. •Don’t use default redis configuration. •Monitoring is very importatnt.
- 101. Thanks