Towards Android Automation: Screen Vision and Software Touch
1 like837 views
This document discusses automation of Android devices. It describes existing automation methods like AutoHotkey that emulate mouse/keyboard. It also discusses light automation for Android using Tasker to run apps and click locations. The document proposes taking screenshots to analyze screens and using software touches to emulate clicks. It envisions a full automated system that uses screenshots, decision making scripts, and touch emulation to loop through tasks without human interaction. The goal is to automate Android devices for applications like load balancing networks and building infrastructure using mobile devices.
Towards Android Automation: Screen Vision and Software Touch
- 2. . Theory of Android Automation M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 2/10 ... 2/10
- 3. . Traditional Automation (AutoHotkey) • AutoHotkey ◦ mouse click emulation ◦ keyboard emulation ◦ run/switch apps ◦ ... • very useful in research -- software agents replace human operators M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 3/10 ... 3/10
- 4. . Existing Android Automation (light) • "move app to SD" can only be done manually • ...so, as a light form of automation, one can 1. run system.prefs window 2. wait 1s, then, stick a balloon at (x,y) M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 4/10 ... 4/10
- 5. . Android Tasker • often called "android autohotkey", but grossly lacking in function M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 5/10 ... 5/10
- 6. . A Screenshot App for Vision • useful, even if you have a native screenshot function • for example, you can take screenshots with a delay M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 6/10 ... 6/10
- 7. . Task Setup M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 7/10 ... 7/10
- 8. . Full Auto Design • start → tasker + screenshot + PHP/ Python/* script → decision → touch (emulation) → end (→next loop) • you can scipt decision-making using SL4A (I use PHP) M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 8/10 ... 8/10
- 9. . The Big Picture • in my case, basically, I need it for 3G/LTE+WiFiDirect virtualization 01 ◦ load balancing, background processing • usecase 1: throughput maximization for groups at a university campus 03 • usecase 2 : vehicular nets and ultimately clouds 02 . core idea .. . build infrastructure with android devices/OS at the core 01 myself+0 "Virtual Wireless User: ...Parallel MultiConnect Using WiFi Direct..." 10th MobiQuitous Springer LNICS vol.131 (2013) 03 myself+0 "Group Connect in a New Wireless University Campus" IEICE SR研 (2014) 02 myself+0 "The Vobile Cloud : Cars are Better for Mobile Clouds and Group Connect" IEICE ITS研 (2014) M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 9/10 ... 9/10
- 10. . That’s all, thank you ... M.Zhanikeev -- maratishe@gmail.com -- Towards Android Automation: Screen Vision and Software Touch -- http://bit.do/marat150525 10/10 ... 10/10