UIAutomator2.0和AccessibilityService实现分析

0x00 前言

UiAutomator是Android 4.1以上提供的一个UI自动化测试工具,4.3升级到了UiAutomator2.0,实现方式也从UiTestAutomationBridge变成了UiAutomation。

0x01 UiAutomation实现分析

UiAutomation类位于android.app包下面,是API18新增的类。

  1. public final class UiAutomation {
  2. private final IAccessibilityServiceClient mClient;
  3. private final IUiAutomationConnection mUiAutomationConnection;
  4. public void connect();
  5. public void disconnect();
  6. public final boolean performGlobalAction(int action);
  7. public final AccessibilityServiceInfo getServiceInfo();
  8. public AccessibilityNodeInfo getRootInActiveWindow();
  9. public boolean injectInputEvent(InputEvent event, boolean sync);
  10. public boolean setRotation(int rotation);
  11. public AccessibilityEvent executeAndWaitForEvent(Runnable command, AccessibilityEventFilter filter, long timeoutMillis);
  12. public void waitForIdle(long idleTimeoutMillis, long globalTimeoutMillis);
  13. public Bitmap takeScreenshot();
  14. public void setRunAsMonkey(boolean enable);
COPY

以上是UiAutomation类中最重要的成员变量和函数,所有的接口基本都是通过调用mClientmUiAutomationConnection这两个对象实现的。

  1. public UiAutomation(Looper looper, IUiAutomationConnection connection) {
  2. if (looper == null) {
  3. throw new IllegalArgumentException("Looper cannot be null!");
  4. }
  5. if (connection == null) {
  6. throw new IllegalArgumentException("Connection cannot be null!");
  7. }
  8. mUiAutomationConnection = connection;
  9. mClient = new IAccessibilityServiceClientImpl(looper);
  10. }
  11. interface IUiAutomationConnection {
  12. void connect(IAccessibilityServiceClient client);
  13. void disconnect();
  14. boolean injectInputEvent(in InputEvent event, boolean sync);
  15. boolean setRotation(int rotation);
  16. Bitmap takeScreenshot(int width, int height);
  17. boolean clearWindowContentFrameStats(int windowId);
  18. WindowContentFrameStats getWindowContentFrameStats(int windowId);
  19. void clearWindowAnimationFrameStats();
  20. WindowAnimationFrameStats getWindowAnimationFrameStats();
  21. void executeShellCommand(String command, in ParcelFileDescriptor fd);
  22. void grantRuntimePermission(String packageName, String permission, int userId);
  23. void revokeRuntimePermission(String packageName, String permission, int userId);
  24. // Called from the system process.
  25. oneway void shutdown();
  26. }
COPY

UiAutomation构造函数需要传入两个参数,一个是线程Looper对象,用于发送消息,一个是IUiAutomationConnection接口实例。

  1. public void connect() {
  2. if (mHandlerThread.isAlive()) {
  3. throw new IllegalStateException("Already connected!");
  4. }
  5. mHandlerThread.start();
  6. mUiAutomation = new UiAutomation(mHandlerThread.getLooper(),
  7. new UiAutomationConnection());
  8. mUiAutomation.connect();
  9. }
COPY

这是UiAutomationShellWrapper类中的一个方法,正好可以看到UiAutomation如何初始化。构造函数的第二个参数实际使用的是UiAutomationConnection类实例,这个类也是在android.app包下面,正是继承自IUiAutomationConnection.Stub类。

UiAutomation中一个重要的成员变量是mClient,它的类型是IAccessibilityServiceClient

  1. oneway interface IAccessibilityServiceClient {
  2. void init(in IAccessibilityServiceConnection connection, int connectionId, IBinder windowToken);
  3. void onAccessibilityEvent(in AccessibilityEvent event);
  4. void onInterrupt();
  5. void onGesture(int gesture);
  6. void clearAccessibilityCache();
  7. void onKeyEvent(in KeyEvent event, int sequence);
  8. }
COPY

IAccessibilityServiceClient是一个AIDL接口定义,在抽象类AccessibilityService中实现了一个该接口的实现类IAccessibilityServiceClientWrapper。UiAutomation中定义了一个IAccessibilityServiceClientWrapper的子类IAccessibilityServiceClientImpl,主要是重写了onAccessibilityEvent方法,用于获取AccessibilityEvent事件。UiAutomation的构造函数中实例化的正是IAccessibilityServiceClientImpl实例。

UiAutomation的初始化过程主要是在connect方法中。

  1. /**
  2. * Connects this UiAutomation to the accessibility introspection APIs.
  3. *
  4. * @hide
  5. */
  6. public void connect() {
  7. synchronized (mLock) {
  8. throwIfConnectedLocked();
  9. if (mIsConnecting) {
  10. return;
  11. }
  12. mIsConnecting = true;
  13. }
  14. try {
  15. // Calling out without a lock held.
  16. mUiAutomationConnection.connect(mClient);
  17. } catch (RemoteException re) {
  18. throw new RuntimeException("Error while connecting UiAutomation", re);
  19. }
  20. synchronized (mLock) {
  21. final long startTimeMillis = SystemClock.uptimeMillis();
  22. try {
  23. while (true) {
  24. if (isConnectedLocked()) {
  25. break;
  26. }
  27. final long elapsedTimeMillis = SystemClock.uptimeMillis() - startTimeMillis;
  28. final long remainingTimeMillis = CONNECT_TIMEOUT_MILLIS - elapsedTimeMillis;
  29. if (remainingTimeMillis <= 0) {
  30. throw new RuntimeException("Error while connecting UiAutomation");
  31. }
  32. try {
  33. mLock.wait(remainingTimeMillis);
  34. } catch (InterruptedException ie) {
  35. /* ignore */
  36. }
  37. }
  38. } finally {
  39. mIsConnecting = false;
  40. }
  41. }
  42. }
COPY

这里主要是调用了mUiAutomationConnection.connect(mClient)。

  1. public void connect(IAccessibilityServiceClient client) {
  2. if (client == null) {
  3. throw new IllegalArgumentException("Client cannot be null!");
  4. }
  5. synchronized (mLock) {
  6. throwIfShutdownLocked();
  7. if (isConnectedLocked()) {
  8. throw new IllegalStateException("Already connected.");
  9. }
  10. mOwningUid = Binder.getCallingUid();
  11. registerUiTestAutomationServiceLocked(client);
  12. storeRotationStateLocked();
  13. }
  14. }
  15. private void registerUiTestAutomationServiceLocked(IAccessibilityServiceClient client) {
  16. IAccessibilityManager manager = IAccessibilityManager.Stub.asInterface(
  17. ServiceManager.getService(Context.ACCESSIBILITY_SERVICE));
  18. AccessibilityServiceInfo info = new AccessibilityServiceInfo();
  19. info.eventTypes = AccessibilityEvent.TYPES_ALL_MASK;
  20. info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
  21. info.flags |= AccessibilityServiceInfo.FLAG_INCLUDE_NOT_IMPORTANT_VIEWS
  22. | AccessibilityServiceInfo.FLAG_REPORT_VIEW_IDS;
  23. info.setCapabilities(AccessibilityServiceInfo.CAPABILITY_CAN_RETRIEVE_WINDOW_CONTENT
  24. | AccessibilityServiceInfo.CAPABILITY_CAN_REQUEST_TOUCH_EXPLORATION
  25. | AccessibilityServiceInfo.CAPABILITY_CAN_REQUEST_ENHANCED_WEB_ACCESSIBILITY
  26. | AccessibilityServiceInfo.CAPABILITY_CAN_REQUEST_FILTER_KEY_EVENTS);
  27. try {
  28. // Calling out with a lock held is fine since if the system
  29. // process is gone the client calling in will be killed.
  30. manager.registerUiTestAutomationService(mToken, client, info);
  31. mClient = client;
  32. } catch (RemoteException re) {
  33. throw new IllegalStateException("Error while registering UiTestAutomationService.", re);
  34. }
  35. }
COPY

UiAutomationConnection.connect函数接着调用了registerUiTestAutomationServiceLocked(client),而registerUiTestAutomationServiceLocked主要是调用了IAccessibilityManagerregisterUiTestAutomationService注册了一个Accessibility服务。

由此可见,UiAutomation最终也是使用了AccessibilityManagerService。

0x02 如何使用UiAutomation

UiAutomator的常见使用方式是调用uiautomator命令,或是将uiautomator.jar导入到自己的工程中。为了更加自由地使用UiAutomation提供的能力,可以考虑直接创建UiAutomation对象实例使用。由于UiAutomation的构造函数以及其它一些重要方法设置了@hide,因此无法直接使用,需要用反射的方式获取。

  1. Object connection = null;
  2. HandlerThread mHandlerThread = new HandlerThread("UiAutomationThread");
  3. mHandlerThread.start();
  4. try{
  5. Class<?> UiAutomationConnection = Class.forName("android.app.UiAutomationConnection");
  6. Constructor<?> newInstance = UiAutomationConnection.getDeclaredConstructor();
  7. newInstance.setAccessible(true);
  8. connection = newInstance.newInstance();
  9. Class<?> IUiAutomationConnection = Class.forName("android.app.IUiAutomationConnection");
  10. Constructor<?> newUiAutomation = UiAutomation.class.getDeclaredConstructor(Looper.class, IUiAutomationConnection);
  11. UiAutomation mUiAutomation = (UiAutomation)newUiAutomation.newInstance(mHandlerThread.getLooper(), connection);
  12. Method connect = UiAutomation.class.getDeclaredMethod("connect");
  13. connect.invoke(mUiAutomation);
  14. Log.i(TAG, ""+mUiAutomation);
  15. mUiAutomation.waitForIdle(1000, 1000 * 10);
  16. AccessibilityNodeInfo nodeInfo = mUiAutomation.getRootInActiveWindow();
  17. Log.i(TAG, ""+nodeInfo);
  18. }catch(Exception e){
  19. e.printStackTrace();
  20. return;
  21. }
COPY

这是一段最简单的使用代码,编译到apk里面,运行后报错了。

  1. Caused by: java.lang.SecurityException: You do not have android.permission.RETRIEVE\_WINDOW_CONTENT required to call registerUiTestAutomationService from pid=3676, uid=10040
  2. at android.os.Parcel.readException(Parcel.java:1599)
  3. at android.os.Parcel.readException(Parcel.java:1552)
  4. at android.view.accessibility.IAccessibilityManager$Stub$Proxy.registerUiTestAutomationService (IAccessibilityManager.java:352)
  5. at android.app.UiAutomationConnection.registerUiTestAutomationServiceLocked(UiAutomationConnection.java:337)
  6. at android.app.UiAutomationConnection.connect(UiAutomationConnection.java:89)
  7. at android.app.UiAutomation.connect(UiAutomation.java:197)
COPY

看来是需要android.permission.RETRIEVE_WINDOW_CONTENT权限,尝试将该权限添加到AndroidManifest.xml中,但是提示这个是系统权限,普通应用不能申请。因此,应用中不能使用UiAutomation的,同时发现,编译为jar包后使用shell权限执行,是可以正常跑过的。这也是为什么uiautomator工具可以正常运行的原因。

0x03 AccessibilityManagerService

如前所述,UiAutomation底层使用的还是AccessibilityManagerService,这里简单分析一下AccessibilityManagerService的实现。

该类内部包含了一个Service类:

  1. /**
  2. * This class represents an accessibility service. It stores all per service
  3. * data required for the service management, provides API for starting/stopping the
  4. * service and is responsible for adding/removing the service in the data structures
  5. * for service management. The class also exposes configuration interface that is
  6. * passed to the service it represents as soon it is bound. It also serves as the
  7. * connection for the service.
  8. */
  9. class Service extends IAccessibilityServiceConnection.Stub implements ServiceConnection, DeathRecipient;
COPY

从描述上看,每个Service实例代表了一个Accessibility服务,同时它实现了ServiceConnection接口,因此它也是一个Service客户端。

Service类中定义了一个bool类型的变量mIsAutomation,表示当前服务是否是UiAutomation。在AccessibilityManagerService的registerUiTestAutomationService方法中,将当前服务的组件名称设置为sFakeAccessibilityServiceComponentName,而mIsAutomation就是通过组件名称来判断是否是UiAutomation。

  1. accessibilityServiceInfo.setComponentName(sFakeAccessibilityServiceComponentName);
  2. mIsAutomation = (sFakeAccessibilityServiceComponentName.equals(componentName));
COPY

由于UiAutomation与AccessibilityService实现方式不同,因此,AccessibilityManagerService在很多地方都会根据mIsAutomation执行不同的逻辑。

根据前面的分析,在UiAutomation发起connect请求后,会进入AccessibilityManagerService的registerUiTestAutomationService。然后经过一系列的函数调用,进到Service的bindLocked函数。

  1. /**
  2. * Binds to the accessibility service.
  3. *
  4. * @return True if binding is successful.
  5. */
  6. public boolean bindLocked() {
  7. UserState userState = getUserStateLocked(mUserId);
  8. if (!mIsAutomation) {
  9. if (mService == null && mContext.bindServiceAsUser(
  10. mIntent, this,
  11. Context.BIND_AUTO_CREATE | Context.BIND_FOREGROUND_SERVICE_WHILE_AWAKE,
  12. new UserHandle(mUserId))) {
  13. userState.mBindingServices.add(mComponentName);
  14. }
  15. } else {
  16. userState.mBindingServices.add(mComponentName);
  17. mService = userState.mUiAutomationServiceClient.asBinder();
  18. mMainHandler.post(new Runnable() {
  19. @Override
  20. public void run() {
  21. // Simulate asynchronous connection since in onServiceConnected
  22. // we may modify the state data in case of an error but bind is
  23. // called while iterating over the data and bad things can happen.
  24. onServiceConnected(mComponentName, mService);
  25. }
  26. });
  27. userState.mUiAutomationService = this;
  28. }
  29. return false;
  30. }
COPY

bindLocked会发送一个异步消息,调到onServiceConnected回调。

  1. @Override
  2. public void onServiceConnected(ComponentName componentName, IBinder service) {
  3. synchronized (mLock) {
  4. mService = service;
  5. mServiceInterface = IAccessibilityServiceClient.Stub.asInterface(service);
  6. UserState userState = getUserStateLocked(mUserId);
  7. addServiceLocked(this, userState);
  8. if (userState.mBindingServices.contains(mComponentName) || mWasConnectedAndDied) {
  9. userState.mBindingServices.remove(mComponentName);
  10. mWasConnectedAndDied = false;
  11. try {
  12. mServiceInterface.init(this, mId, mOverlayWindowToken);
  13. onUserStateChangedLocked(userState);
  14. } catch (RemoteException re) {
  15. Slog.w(LOG_TAG, "Error while setting connection for service: "
  16. + service, re);
  17. binderDied();
  18. }
  19. } else {
  20. binderDied();
  21. }
  22. }
  23. }
COPY

这里的mServiceInterface其实就是UiAutomation构造函数中实例化的mClient的远程对象。接着,调用了IAccessibilityServiceClient的init函数,该函数的实现位于AccessibilityService.IAccessibilityServiceClientWrapper类中。

  1. public void init(IAccessibilityServiceConnection connection, int connectionId,
  2. IBinder windowToken) {
  3. Message message = mCaller.obtainMessageIOO(DO_INIT, connectionId,
  4. connection, windowToken);
  5. mCaller.sendMessage(message);
  6. }
  7. case DO_INIT: {
  8. mConnectionId = message.arg1;
  9. SomeArgs args = (SomeArgs) message.obj;
  10. IAccessibilityServiceConnection connection =
  11. (IAccessibilityServiceConnection) args.arg1;
  12. IBinder windowToken = (IBinder) args.arg2;
  13. args.recycle();
  14. if (connection != null) {
  15. AccessibilityInteractionClient.getInstance().addConnection(mConnectionId,
  16. connection);
  17. mCallback.init(mConnectionId, windowToken);
  18. mCallback.onServiceConnected();
  19. } else {
  20. AccessibilityInteractionClient.getInstance().removeConnection(
  21. mConnectionId);
  22. mConnectionId = AccessibilityInteractionClient.NO_ID;
  23. AccessibilityInteractionClient.getInstance().clearCache();
  24. mCallback.init(AccessibilityInteractionClient.NO_ID, null);
  25. }
  26. } return;
COPY

这样,初始化完成后,AccessibilityInteractionClient实例中保存了IAccessibilityServiceConnection实例,而AccessibilityManagerService中对应的Service对象中也保存了IAccessibilityServiceClient实例,从而建立起双向的Binder通信。

在发生Accessibility事件后,AccessibilityManagerService会通过IAccessibilityServiceConnection的onAccessibilityEvent方法将事件通知给UiAutomation。

0x04 UiAutomator与AccessibilityService

AccessibilityService是一个继承自Service的抽象服务类,用户在使用时需要实现一个自己的子类。该类很大程度上依赖于AccessibilityInteractionClient类提供的接口,AccessibilityInteractionClient类内部保存了一个静态的LongSparseArray<AccessibilityInteractionClient> sClients对象,用来实现线程相关的单例对象。

  1. /**
  2. * @return The client for the current thread.
  3. */
  4. public static AccessibilityInteractionClient getInstance() {
  5. final long threadId = Thread.currentThread().getId();
  6. return getInstanceForThread(threadId);
  7. }
  8. /**
  9. * <strong>Note:</strong> We keep one instance per interrogating thread since
  10. * the instance contains state which can lead to undesired thread interleavings.
  11. * We do not have a thread local variable since other threads should be able to
  12. * look up the correct client knowing a thread id. See ViewRootImpl for details.
  13. *
  14. * @return The client for a given <code>threadId</code>.
  15. */
  16. public static AccessibilityInteractionClient getInstanceForThread(long threadId) {
  17. synchronized (sStaticLock) {
  18. AccessibilityInteractionClient client = sClients.get(threadId);
  19. if (client == null) {
  20. client = new AccessibilityInteractionClient();
  21. sClients.put(threadId, client);
  22. }
  23. return client;
  24. }
  25. }
COPY

前面提到,AccessibilityInteractionClient对象中保存了IAccessibilityServiceConnection实例,因此,可以调用该接口提供的功能。

  1. /**
  2. * Interface given to an AccessibilitySerivce to talk to the AccessibilityManagerService.
  3. *
  4. * @hide
  5. */
  6. interface IAccessibilityServiceConnection {
  7. void setServiceInfo(in AccessibilityServiceInfo info);
  8. boolean findAccessibilityNodeInfoByAccessibilityId(int accessibilityWindowId,
  9. long accessibilityNodeId, int interactionId,
  10. IAccessibilityInteractionConnectionCallback callback, int flags, long threadId);
  11. boolean findAccessibilityNodeInfosByText(int accessibilityWindowId, long accessibilityNodeId,
  12. String text, int interactionId, IAccessibilityInteractionConnectionCallback callback,
  13. long threadId);
  14. boolean findAccessibilityNodeInfosByViewId(int accessibilityWindowId,
  15. long accessibilityNodeId, String viewId, int interactionId,
  16. IAccessibilityInteractionConnectionCallback callback, long threadId);
  17. boolean findFocus(int accessibilityWindowId, long accessibilityNodeId, int focusType,
  18. int interactionId, IAccessibilityInteractionConnectionCallback callback, long threadId);
  19. boolean focusSearch(int accessibilityWindowId, long accessibilityNodeId, int direction,
  20. int interactionId, IAccessibilityInteractionConnectionCallback callback, long threadId);
  21. boolean performAccessibilityAction(int accessibilityWindowId, long accessibilityNodeId,
  22. int action, in Bundle arguments, int interactionId,
  23. IAccessibilityInteractionConnectionCallback callback, long threadId);
  24. AccessibilityWindowInfo getWindow(int windowId);
  25. List<AccessibilityWindowInfo> getWindows();
  26. AccessibilityServiceInfo getServiceInfo();
  27. boolean performGlobalAction(int action);
  28. oneway void setOnKeyEventResult(boolean handled, int sequence);
  29. }
COPY

与UiAutomation不同的是,AccessibilityService的初始化是和普通的Service一致的。由于AccessibilityService比较特殊的地方在于需要在设置的辅助功能里开启对应的服务,点击开启后,会执行到BindService逻辑,进而执行到AccessibilityService的onBind回调,并触发AccessibilityManagerService中Service的onServiceConnected回调。

  1. /**
  2. * Implement to return the implementation of the internal accessibility
  3. * service interface.
  4. */
  5. @Override
  6. public final IBinder onBind(Intent intent) {
  7. return new IAccessibilityServiceClientWrapper(this, getMainLooper(), new Callbacks() {
  8. @Override
  9. public void onServiceConnected() {
  10. AccessibilityService.this.onServiceConnected();
  11. }
  12. @Override
  13. public void onInterrupt() {
  14. AccessibilityService.this.onInterrupt();
  15. }
  16. @Override
  17. public void onAccessibilityEvent(AccessibilityEvent event) {
  18. AccessibilityService.this.onAccessibilityEvent(event);
  19. }
  20. @Override
  21. public void init(int connectionId, IBinder windowToken) {
  22. mConnectionId = connectionId;
  23. mWindowToken = windowToken;
  24. // The client may have already obtained the window manager, so
  25. // update the default token on whatever manager we gave them.
  26. final WindowManagerImpl wm = (WindowManagerImpl) getSystemService(WINDOW_SERVICE);
  27. wm.setDefaultToken(windowToken);
  28. }
  29. @Override
  30. public boolean onGesture(int gestureId) {
  31. return AccessibilityService.this.onGesture(gestureId);
  32. }
  33. @Override
  34. public boolean onKeyEvent(KeyEvent event) {
  35. return AccessibilityService.this.onKeyEvent(event);
  36. }
  37. });
  38. }
COPY

相对于UiAutomation只能在shell环境中执行,AccessibilityService是可以运行在app环境中的,但是需要用户手动开启服务会略显麻烦。

0x05 总结

UiAutomator和AccessibilityService作为两种不同的实现形式,拥有各自的优缺点,这两年流行的抢红包工具基本也是基于这两种方式实现的。在自动化中使用它们也能起到一些辅助作用。

分享

Gitalking ...