How do you process AndroidControl Dataset?

[mengzchen]

The raw dataset is TFRecord format: https://github.com/google-research/google-research/blob/master/android_control/README.md

And in your test script, it seems you have done some process on the original data, can you share the scripts?

[lll6gg]

json_file = "anno.json"
output_file = "android_control_train.json"

import json
import re
import os
data = json.load(open(json_file))
action_set = set()
# count = {}
result = []
if os.path.exists(output_file):
    os.remove(output_file)
def extract_param(action_type,action):
    if action_type == "scroll":
        param_match = re.search(r'"direction":"(.*?)"', action)
        return param_match.group(1) if param_match else ""
    elif action_type == "input_text":
        param_match = re.search(r'"text":"(.*?)"', action)
        return param_match.group(1) if param_match else ""
    elif action_type == "open_app":
        param_match = re.search(r'"app_name":"(.*?)"', action)
        return param_match.group(1) if param_match else ""
    return ""
for x in data:
    actions = x['actions']
    screenshots = x['screenshots']
    step_instructions = x['step_instructions']
    for i,action in enumerate(actions):
        match = re.search(r'"action_type"\s*:\s*"([^"]+)"', action)
        action_type = match.group(1) if match else ""
        image_filename = screenshots[i]
        step_instruction = step_instructions[i]
        if action_type != "click" and action_type != "long_press" and action_type != "wait":
            # param = extract_param(action_type,action)
            result.append({'img_filename': image_filename, 'action': action_type, 'instruction': step_instruction  , 'data_type': 'android', 'data_source': 'android'})
        # if action_type not in count:
        #     count[action_type] = 0
        # count[action_type] += 1
with open(output_file, "w") as f:
    json.dump(result, open(output_file, 'w'), ensure_ascii=False, indent=4)

[mengzchen]

sorry, so how do you get anno.json?

[lll6gg]

You could download from Huggingface like ckg/AndroidControlParsed-20k.

[YoloEliwa]

@lll6gg Thanks for your awesome work. It is crucial to align the step instructions in the Android control test set. Can you share the anno.json for test？

[lll6gg]

Sorry to response back so late. We have uploaded the test json for Android Control-Low in dataset/ac_test.json. Please download the images by yourself.