On the Viability of CAPTCHAs for Use in Telephony Systems: A Usability Field Study
1 like679 views
The document presents a usability study on the effectiveness of CAPTCHAs in telephony systems, assessing user performance and convenience. Results show a low accuracy rate for various CAPTCHA types, indicating challenges in solving them over telephony, particularly with increased key presses required. It also outlines user preferences and provides guidelines for improving CAPTCHA usability in this context.
On the Viability of CAPTCHAs for Use in Telephony Systems: A Usability Field Study
- 1. On the Viability of CAPTCHAs for Use in Telephony Systems: A Usability Field Study Niharika Sachdeva*, Nitesh Saxena, Ponnurangam Kumaraguru* University of Alabama, Birmingham*IIIT-‐Delhi InformaDon Security Conference, 2013 (Nov 13 – 15)
- 2. Overview – MoDvaDon – Research quesDon – Study Design – Experimental setup – ParDcipants – Results – Guidelines 2
- 4. CAPTCHA – Completely Automated Public Turing Test to tell Computers and Humans Apart ReCAPTCHA Google Math FuncDon Yahoo
- 5. Is it Really Useful ?? – FrustraDng – Lack of incenDve – Hard to recognize – Difficult to solve – NaDve language E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, and D. Jurafsky. How Good Are Humans at Solving CAPTCHAs? A Large Scale EvaluaDon. SP ’10, pages 399–413. J. Yan, A. Ahmad. Usability of CAPTCHAs Or usability issues in CAPTCHA design. In Symposium On Usable Privacy and Security, pages 44–52, 2008. 5
- 6. But CAPTCHA continues to Rule CAPTCHA for RoboCalls 6
- 7. But CAPTCHA continues to Rule CAPTCHA for RoboCalls 7
- 8. Audio CAPTCHA a solution? – Yahoo – Google – Patent CAPTCHA 8
- 9. Research Question – QuanDfy the amount of inconvenience CAPTCHA causes to users. – How different features of CAPTCHA, e.g. duraDon, size and character set influence the users’ performance? - H1: Close to the expected / correct answers even though the overall CAPTCHA solving accuracy is low. - H2: Accuracy of answering the CAPTCHA correctly on telephony decreases as the number of key presses required increases. - H3: Users will take more Eme responding to a CAPTCHA that requires more key presses than to the one requiring less key presses. 9
- 10. Study Design LaDn Square Polakis, G. Kontaxis, and S. Ioannidis. CAPTCHuring Automated (Smart) Phone Aiacks. In SYSSEC, 2011. 10
- 11. CAPTCHA Features Category Char. Set Word Repeat Duration Nois e Voice Beep Min Max length length Google 0-9 No Yes 34.4 Yes M Yes 5 15 Ebay 0-9 No No 3.7 Yes V No 6 6 Yahoo 0-9 No No 18.0 Yes Child No 6 8 Recaptcha a-z Yes No 10.6 Yes F No 6 6 Slashdot a-z Yes No 2.9 No M No 1 1 CD 1-5 No No 14 Yes M No 1 1 Mathfunction 0-9 No No 6.0 No M No 4 3 RPC 0-9 No No 20.0 No M No 3 2 C+CD 0-9 No No 14.0 No M No 4 3 M = Male ; F = Female; V=Various Voices 11
- 12. Deployment Database IVRS Playing CAPTCHA VOIP Java Application PSTN Linksys Gateway SPA 3102 Cellular Network Source (Legitimate or malicious) File System IP phone Linux Server acting as CAPTCHA Shield (With FreeSWITCH) Architecture Diagram 12
- 13. Participants – 90 ParDcipants – Five ciDes - Delhi - Mumbai - Chennai - Noida - Vellore – Real world deployment 13
- 14. Results: Accuracy CAPTCHA Category Accuracy (%) Skip (%) CD Telephony 18.71 35.67 Math-‐FuncDon Telephony 17.47 26.51 RPC Telephony 15.47 40.33 C+CD Telephony 4.57 40.10 Ebay Web (Number) 8.75 13.13 Google Web (Number) 0.00 43.83 Yahoo Web (Number) 7.74 20.24 ReCaptcha Web (Alphabet) 0.00 46.07 Slashdot Web (Alphabet) 13.73 30.06 14
- 15. Results: Time taken CAPTCHA Category Time (s) CD Telephony 96.11 Math-‐Func Telephony 90.23 RPC Telephony 147.44 C+CD Telephony 109.59 Ebay Web (Number) 80.25 Google Web (Number) 123.49 Yahoo Web (Number) 95.88 ReCaptcha Web (Alphabet) 120.64 Slashdot Web (Alphabet) 122.57 15
- 16. Results:H1 – H1: Close to the expected / correct answers even though the overall CAPTCHA solving accuracy is low. 50 Yahoo! eBay 45 Google Slashdot 40 RPC Number of Captcha 35 Math−Function CD 30 C+CD 25 20 15 10 5 0 1 2 3 4 Edit Distance 5 6 7 16
- 17. Results: H2 Math function, but we noticed a negative relationship with correlation coefficient r = 0.47 for web-based captcha. Finally, we found significant difference (t-test, t-value = 5.30 p-value < 0.001) between Expected Key Press (Average DTMF) and accuracy in statistical results shows that these two were independent of each other. The results mentioned above do not approve our hypothesis H2. • H2: Accuracy of answering the CAPTCHA correctly on telephony decreases as the number of key presses required increases Table 3: Presents the Average DTMF expected for captcha (Avg. DTMF), accuracy, time and Average DTMF input by users (Avg. User DTMF) of each captcha. Scheme Category Avg. Accuracy Time Avg. User DTMF DTMF CD Telephony 1.00 18.71 96.11 1.76 MathTelephony 2.05 17.47 90.23 2.71 function RPC Telephony 3.00 15.47 147.44 3.92 C + CD Telephony 2.06 4.57 109.59 2.65 Ebay Web 6.00 8.75 80.25 3.85 Google Web 6.36 0.00 123.49 4.68 Yahoo Web 7.09 7.74 95.88 4.99 Slashdot Web 15.34 13.73 120.64 6.02 ReCaptcha Web 64.93 0.00 122.57 10.97 H3 – Time vs. Number of key press: Table 3 shows that users spent varying amount of time 17
- 18. Results: H3 – H3: Users will take more Dme responding to a CAPTCHA that requires more key presses than to the one requiring less key presses. 80" 70" 60" 50" 40" 30" 20" 10" 0" Ebay"" Google"" Key"press"(#)" Yahoo"" Accuracy"(%)" Slashdot"" Recaptcha" Avg"play"Gme(sec)" 18
- 19. User Experience – User friendliness Strongly#Agree# Agree# Nuetral# Disagree# #Strongly#disagree# 0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%# Complexity# Frequently#use# Confidence# Technical#help# 19
- 20. 100%# User Experience – User preferred scheme 100%# 90%# 80%# Par$cipants+(%)++ 70%# 60%# 50%# 40%# 30%# 20%# 10%# 0%# 18-24# 25-35# 36-50# 51-65# Age+ #Numeric# #Mathfunc;on# #Alphabets# 20
- 21. Guidelines – One Dme instrucDon – Loss / Error Tolerant – Feedback – Verbal Responses 21
- 23. For any further information, please write to pk@iiitd.ac.in precog.iiitd.edu.in