个人主观评分：6.5（满分10分）

阿里的开源水平行业内都知道，也就是装装样子，不过冲着对issue有一部分回复，就值得4分。剩下一分给文章的质量，确实比Ferret-UI写的清楚一些，但是由于文章中对于各个组成部分并无详细的实验细节，也仅仅值得快速读一下的水准。实际上很多公式和文章中给的Prompt也不太对得上，不过代码实现还是很详细。总的来说是一个粗糙度高但值得一度，值得一试的工作

引用一个issue的提问和回答：

Hi,

I am currently reading your paper on Mobile-Agent-v2 and am impressed with the multi-agent architecture proposed.

While going through the paper, I noticed the detailed explanation of the visual perception module's design, including the use of the text recognition tool (ConvNextViT-document), icon recognition tool (GroundingDINO), and icon description tool (Qwen-VL-Int44).

However, I couldn't find any specific data regarding the accuracy of this module in recognizing, localizing, and describing UI elements.

Could you please provide some insights into the performance of the visual perception module?

Thank you for your time and valuable insights! Hello. Since Mobile-Agent-v2 is the first framework to use OCR, SAM and VLLM for mobile icon detection, screen text recognition and mobile icon description, we cannot directly obtain performance from the conclusions of existing work. We have not done specific experiments to quantify the effects of these models. However, from our experience, the current performance of OCR and VLLM is overflowing. On the contrary, the performance of SAM for icon detection is still insufficient. To address this, the solution we adopted is oversampling, that is, using a lower confidence threshold to obtain a large number of icons, and the decision agent will do the screening in the final decision.

实验设置

• 实验设置：文本识别用 ConvNextViT-document，图标识别用 GroundingDINO，图标描述用 Qwen-VL-Int4。规划智能体用 GPT-4，决策和反思智能体用 GPT-4V。采用动态评估法，在 Harmony OS 和 Android OS 系统，针对系统应用、外部应用和多应用场景设计多种指令进行评估，指标包括成功率、完成率、决策准确率和反思准确率 。

• 实验结果：相比 Mobile-Agent，Mobile-Agent-v2 在任务完成率上显著提升，尤其在高级指令场景。知识注入可进一步提升其性能。消融实验表明规划智能体对整体框架影响最大，反思智能体和记忆单元也不可或缺。多智能体架构使 Mobile-Agent-v2 在处理长序列操作时更具优势。

摘要

Introduction

1. 提出多智能体架构Mobile-Agent-v2，缓解移动设备操作任务中，单智能体框架固有的各类导航困难。

2. 设计规划智能体，基于历史操作生成任务进度，辅助决策智能体有效生成操作；同时设计记忆单元与反思智能体，解决聚焦内容导航及反思能力缺失问题。记忆单元由决策智能体更新聚焦内容，反思智能体评估决策智能体操作，不符合预期时生成补救措施。

3. 在多种操作系统、语言环境和应用程序中，对Mobile-Agent-v2进行动态评估。实验表明，该模型性能显著提升，手动操作知识注入可进一步增强其性能。

Related Work

Mobile-Agent-v2

Firstly, the planning agent updates the task progress, allowing the decision agent to navigate the progress of the current task. The decision agent then operates based on the current task progress, current screen state, and the reflection (if the last operation is erroneous). Subsequently, the reflection agent observes the screens before and after the operation to determine if the operation meets expectations.

首先，规划智能体更新任务进度，使决策智能体能够把握当前任务的进展情况。随后，决策智能体依据当前任务进度、当前屏幕状态以及反思结果（若上一次操作有误）来执行操作。接着，反思智能体观察操作前后的屏幕，以判断该操作是否符合预期。

Visual Perception Module

即便对于最先进的多模态大语言模型（MLLMs）而言，端到端处理时，屏幕识别依旧颇具挑战性。因此，我们整合了一个视觉感知模块来提升屏幕识别能力。在这个模块中，我们运用三种工具：文本识别工具、图标识别工具以及图标描述工具。将屏幕截图输入此模块，最终会得到屏幕上呈现的文本和图标信息，以及它们各自的坐标。这一过程可用以下公式表示： ， 代表在第次iteration的感知结果

Memory Unit

The memory unit serves as a short-term memory module that is updated as the task progresses

Planning Agent

Decision Agent

• Ins：用户指令

• :上一步任务进度，

• : 上一步聚焦内容

• ：上一步从反思工具中来的反思结果

• ：第t步的屏幕截图

• ：对第t步的屏幕截图识别结果

• 打开app：如果当前页面是主页面，则打开名为 <app_name> 的应用

• 点击操作：点击位置(x,y)

• 滑动操作：从 位置1(x1,y1) 滑动到 位置2 (x2,y2)

• 打字操作：如果当前键盘被激活，这个操作可以在输入框中输入字符串 <text>

• 回到主界面操作

• 停止操作：如果决策工具认为所有的要求都被完成，则可以通过此操作来结束整个操作流程

Reflection Agent

• 有害操作：将手机当前页面带入到与任务无关的页面中。例如工具本想用短信app联系A，结果打开了B的联系界面。

• 无效操作：操作前后的页面没有产生变化，例如本来想点击某个icon，结果点击位置错了。

• 正确操作：当前操作符合决策工具的预期，有助于进一步完成用户的要求。

实验部分

• OCR：ConvNextViT-document

• ICON Recognition：GroundingDINO

• ICON Description Tool：Qwen-VL-Int4

• planning agent:it does not require screen perception, we use GPT-4

• decision agent: 需要根据页面情况来决策了，用了GPT-4V

• reflection agent:需要根据页面情况来决策了，用了GPT-4V

Evaluation Method

评测范围：

评测指标

结果

SystemPrompt Example

Planning Agent

prompt for Planning agent during the first operation

System You are a helpful AI mobile phone operating assistant. User

### Background ###

There is an user’s instruction which is: {User’s instruction}. You are a mobile phone operating assistant and are operating the user’s mobile phone.

### Hint ###

There are hints to help you complete the user’s instructions. The hints are as follow: If you want to tap an icon of an app, use the action "Open app"

### Current operation ###

To complete the requirements of user’s instruction, you have performed an operation. Your operation thought and action of this operation are as follows: Operation thought: {Last operation thought} Operation action: {Last operation}

### Response requirements ###

Now you need to combine all of the above to generate the "Completed contents". Completed contents is a general summary of the current contents that have been completed. You need to first focus on the requirements of user’s instruction, and then summarize the contents that have been completed.

### Output format ###

Your output format is:

### Completed contents ###

Generated Completed contents. Don’t output the purpose of any operation. Just summarize the contents that have been actually completed in the ### Current operation ###. (Please use English to output)

The prompt for the planning agent during subsequent operations.

System You are a helpful AI mobile phone operating assistant.

User

### Background ###

There is an user’s instruction which is: {User’s instruction}. You are a mobile phone operating assistant and are operating the user’s mobile phone.

### Hint ###

There are hints to help you complete the user’s instructions. The hints are as follow: If you want to tap an icon of an app, use the action "Open app"

### History operations ###

To complete the requirements of user’s instruction, you have performed a series of operations. These operations are as follow: Step-1: [Operation thought: {operation thought 1}; Operation action: {operation 1}] Step-2: [Operation thought: {operation thought 2}; Operation action: {operation 2}] ......

### Progress thinking ###

After completing the history operations, you have the following thoughts about the progress of user’s instruction completion: Completed contents: {Last "Completed contents"}

### Response requirements ###

Now you need to update the "Completed contents". Completed contents is a general summary of the current contents that have been completed based on the ### History operations ###.

### Output format ###

Your output format is:

### Completed contents ###

Updated Completed contents. Don’t output the purpose of any operation. Just summarize the contents that have been actually completed in the ### History operations ###.

Decision Agent‘s prompt

System You are a helpful AI mobile phone operating assistant. You need to help me operate the phone to complete the user’s instruction. User

### Background ###

This image is a phone screenshot. Its width is {Lateral resolution} pixels and its height is {Vertical resolution} pixels. The user’s instruction is: {User’s instruction}.

 

### Screenshot information ###

In order to help you better perceive the content in this screenshot, we extract some information on the current screenshot through system files. This information consists of two parts: coordinates; content. The format of the coordinates is [x, y], x is the pixel from left to right and y is the pixel from top to bottom; the content is a text or an icon description respectively. The information is as follow: (x1, y1); text or icon: text content or icon description ......

### Keyboard status ###

We extract the keyboard status of the current screenshot and it is whether the keyboard of the current screenshot is activated. The keyboard status is as follow: The keyboard has not been activated and you can’t type. or The keyboard has been activated and you can type.

### Hint ###

There are hints to help you complete the user’s instructions. The hints are as follow: If you want to tap an icon of an app, use the action "Open app"

### History operations ###

Before reaching this page, some operations have been completed. You need to refer to the completed operations to decide the next operation. These operations are as follow: Step-1: [Operation thought: {operation thought 1}; Operation action: {operation 1}] ......

### Progress ###

After completing the history operations, you have the following thoughts about the progress of user’s instruction completion: Completed contents: {Task progress from planning agent}

### Response requirements ###

Now you need to combine all of the above to perform just one action on the current page. You must choose one of the six actions below: Open app (app name): If the current page is desktop, you can use this action to open the app named "app name" on the desktop. Tap (x, y): Tap the position (x, y) in current page. Swipe (x1, y1), (x2, y2): Swipe from position (x1, y1) to position (x2, y2). Unable to Type. You cannot use the action "Type" because the keyboard has not been activated. If you want to type, please first activate the keyboard by tapping on the input box on the screen. or Type (text): Type the "text" in the input box. Home: Return to home page. Stop: If you think all the requirements of user’s instruction have been completed and no further operation is required, you can choose this action to terminate the operation process.

### Output format ###

Your output consists of the following three parts: ### Thought ###

Think about the requirements that have been completed in previous operations and the requirements that need to be completed in the next one operation. ### Action ###

You can only choose one from the six actions above. Make sure that the coordinates or text in the "()".

### Operation ###

Please generate a brief natural language description for the operation in Action based on your Thought.

Reflection Agent

System

You are a helpful AI mobile phone operating assistant. User

These images are two phone screenshots before and after an operation. Their widths are {Lateral resolution} pixels and their heights are {Vertical resolution} pixels. In order to help you better perceive the content in this screenshot, we extract some information on the current screenshot through system files. The information consists of two parts, consisting of format: coordinates; content. The format of the coordinates is (x, y), x is the pixel from left to right and y is the pixel from top to bottom; the content is a text or an icon description respectively The keyboard status is whether the keyboard of the current page is activated.

### Before the current operation ###

Screenshot information: (x1, y1); text or icon: text content or icon description ...... Keyboard status: The keyboard has not been activated. or The keyboard has been activated.

 

### After the current operation ###

Screenshot information: (x1, y1); text or icon: text content or icon description ...... Keyboard status: The keyboard has not been activated. or The keyboard has been activated.

 

### Current operation ###

The user’s instruction is: {User’s instruction}. You also need to note the following requirements: If you want to tap an icon of an app, use the action "Open app". In the process of completing the requirements of instruction, an operation is performed on the phone. Below are the details of this operation: Operation thought: {Last operation thought} Operation action: {Last operation}

### Response requirements ###

Now you need to output the following content based on the screenshots before and after the current operation: Whether the result of the "Operation action" meets your expectation of "Operation thought"? A: The result of the "Operation action" meets my expectation of "Operation thought". B: The "Operation action" results in a wrong page and I need to return to the previous page. C: The "Operation action" produces no changes.

### Output format ###

Your output format is:

### Thought ###

Your thought about the question

### Answer ###

A or B or C

代码简读

主函数结构

第一步：加载模型

• qwen-vl-chat

• qwen-vl-chat-int4

第二步：初始化各种需要的参数和路径

主循环

• 截屏文件

• 截屏的感知结果

• 键盘状态感知

• perception_infos, width, height,keyboard

• 文件更新

关键引用函数

get_action_prompt | Decision Agent Prompt

get_memory_prompt

get_reflect_prompt

add_response_two_image

get_process_prompt

• instruction：用户指示

• thought_history：操作推理的详细版[实际没用到，只是用来作为一个数量判断依据]

• summary_history：操作推理的简略版

• action_history：操作历史

• completed_requirements：一个字符串