Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
0 likes212 views
The document discusses the challenges and current state of implementing Quality of Service (QoS) in storage systems, particularly focusing on iSCSI targets. It details various methods to manage QoS, such as traffic shaping and manipulation of command queue depths, while also highlighting ongoing upstream efforts to integrate QoS into systems like Ceph. Recommendations include waiting for a native solution or selecting existing options tailored to specific use cases.
![# First we need to get the target and initiator name
# e.g. /sys/kernel/config/target/iscsi/{target}
/tpgt_1/acls/{initiator}
if [ ! $1 ]; then
echo "Please provide target name to adjust speed"
exit -1
fi
TARGET=$1
if [ ! $2 ]; then
echo "Please provide initiator name to adjust
speed"
exit -1
fi
INITIATOR=$2
cmdsn_depth Sample Script](https://image.slidesharecdn.com/05-doingqualityofservicewithoutqos-davidbytealexlau-180419045727/85/Doing-QoS-Before-Ceph-Cluster-QoS-is-available-David-Byte-Alex-Lau-7-320.jpg)
![# Check target exist and Check ACL is enable
TARGET_PATH=/sys/kernel/config/target/iscsi/$TARGET
CMD_DEPTH_PATH=$TARGET_PATH/tpgt_1/acls/$INITIATOR/cmdsn_depth
if [ ! -f $TARGET_PATH ]; then
if [ ! -d $TARGET_PATH/tpgt_1/acls ]; then
echo "Target need acl to allow throlle to work"
exit -1
else
if [ ! -f $CMD_DEPTH_PATH ]; then
echo "Initiator throttler controller
doesn't exist"
exit -1
Check target and ACL](https://image.slidesharecdn.com/05-doingqualityofservicewithoutqos-davidbytealexlau-180419045727/85/Doing-QoS-Before-Ceph-Cluster-QoS-is-available-David-Byte-Alex-Lau-8-320.jpg)
![echo "Please enter [min max] to adjust speed ?"
select result in "min" "max"; do
case $result in
"min" ) echo 1 > $CMD_DEPTH_PATH ;
echo "Now $INITIATOR running at slowest speed"
break;;
"max" ) echo 64 > $CMD_DEPTH_PATH ;
echo "Now $2 should be running at fastest speed"
break;;
esac
done
Script to set cmdsn_depth](https://image.slidesharecdn.com/05-doingqualityofservicewithoutqos-davidbytealexlau-180419045727/85/Doing-QoS-Before-Ceph-Cluster-QoS-is-available-David-Byte-Alex-Lau-9-320.jpg)
Doing QoS Before Ceph Cluster QoS is available - David Byte, Alex Lau
- 1. Doing Quality of Service without QoS David Byte Sr. Technical Strategist SUSE Alex Lau Storage Consultant SUSE
- 2. The Challenge Many customers desire Quality of Service. ● Traditional storage provides it ● Modern storage needs it
- 3. Current State of Affairs There isn’t a mechanism in place for providing QoS today and the ISCSI target providers don’t support this directly either. There are multiple way to provide different forms of QoS. ● The client can limit their own read/write/iops. ● Control traffic at the gateway ● Traffic shaping via the network ● Ceph Native QoS
- 4. Upstream efforts In the upstream community, work is ongoing to implement QoS. But a distributed storage QoS is not easy to do. It also involved with the dmclock implementation. https://github.com/ceph/dmclock https://github.com/ceph/ceph/pull/17450
- 5. Possible solutions for now For RGW, load-balancers may provide some functionality For other protocols, there isn’t much… iscsi manipulate cmdsn_depth queue tc (Traffic Control) is built into the Linux kernel and is able to provide weighted queues similar to network QoS. • Option 1 – bandwidth cap • Option 2 – inject latency
- 6. Adjust ISCSI cmdsn_depth queue This is not really QoS, but queue depth controls max I/O ● 1 queue depth size = slowest ● 64 = much faster Pro: Simple script to automate Con: Not exact Minimum may still be too high Adjustment by hand is still necessary
- 7. # First we need to get the target and initiator name # e.g. /sys/kernel/config/target/iscsi/{target} /tpgt_1/acls/{initiator} if [ ! $1 ]; then echo "Please provide target name to adjust speed" exit -1 fi TARGET=$1 if [ ! $2 ]; then echo "Please provide initiator name to adjust speed" exit -1 fi INITIATOR=$2 cmdsn_depth Sample Script
- 8. # Check target exist and Check ACL is enable TARGET_PATH=/sys/kernel/config/target/iscsi/$TARGET CMD_DEPTH_PATH=$TARGET_PATH/tpgt_1/acls/$INITIATOR/cmdsn_depth if [ ! -f $TARGET_PATH ]; then if [ ! -d $TARGET_PATH/tpgt_1/acls ]; then echo "Target need acl to allow throlle to work" exit -1 else if [ ! -f $CMD_DEPTH_PATH ]; then echo "Initiator throttler controller doesn't exist" exit -1 Check target and ACL
- 9. echo "Please enter [min max] to adjust speed ?" select result in "min" "max"; do case $result in "min" ) echo 1 > $CMD_DEPTH_PATH ; echo "Now $INITIATOR running at slowest speed" break;; "max" ) echo 64 > $CMD_DEPTH_PATH ; echo "Now $2 should be running at fastest speed" break;; esac done Script to set cmdsn_depth
- 10. How did I monitor the results? With openATTIC and Grafana Prometheus and node_exporter, we can monitor the iscsi read, write and ops more easily. However the module still currently in PR waiting to get into prometheus. https://github.com/prometheus/procfs/pull/69 https://github.com/prometheus/node_exporter/pull/776
- 11. After dropping it to cmd_depth to 1
- 12. Use tc to control bandwidth Can filter based on source IP address or target IP address tc qdisc add dev eth0 root handle 1: htb default 30 tc class add dev eth0 parent 1: classid 1:1 htb rate 10000mbit burst 15m tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5000mbit burst 15m tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3000mbit burst 15m tc class add dev eth0 parent 1:1 classid 1:30 htb rate 100mbit ceil 9000mbit burst 15m The author then recommends SFQ for beneath these classes: tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10 tc qdisc add dev eth0 parent 1:30 handle 30: sfq perturb 10 #Filter based on destination (iscsi target) IP tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dst 4.3.2.1/32 flowid 1:10 #Filter based on source (iscsi initiator) IP tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip src 1.2.3.4/32 flowid 1:10
- 13. Use tc to inject latency tc qdisc add dev eth0 root handle 1: prio tc qdisk add dev eth0 parent 1:1 handle 10: netem delay .05ms #Filter based on destination (iscsi target) IP tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dst 4.3.2.1/32 flowid 1:1 #Filter based on source (iscsi initiator) IP tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip src 1.2.3.4/32 flowid 1:1
- 14. tc Methods Pros & Cons Pros: Better control for bandwidth Easily managed through salt or ansible Cons: tc is complex Not the easiest to use (hundreds of clients = high complexity) It doesn’t control IOPS Packets can get dropped Thoughts: Use multiple subnets for ISCSI initiators. Each subnet has it’s own filter and thus QoS setting. This only makes sense with injected delays
- 15. Our thoughts and recommendations If possible, wait for upstream to provide a ceph native solution. If not, carefully select, test, and implement a solution that works for your particular use case.