AI-Powered Automation for Text and Image-Based Task Execution
Budget: $250 – $750 USD
Project Description:
We are looking to build a Python-based AI/ML-powered solution capable of executing a sequence of user-defined tasks on a computer. The system should accept two inputs:
1. Text Input:
○ Example 1 (General): "Go to amazon.com and order 1kg of coffee."
○ Example 2 (Detailed): "Open Chrome browser, in the search bar type 'amazon.com', and in the search bar on the opened website look for 1kg of coffee."
2. Image Input:
○ A screenshot of the starting state of the user's desktop or browser.
The program will then:
1. Interpret the text instructions using AI/ML models like NLP (Natural Language Processing).
2. Analyze the provided image to understand the current state of the UI.
3. Perform the actions in a step-by-step manner on the user's computer, taking a screenshot after each step to reassess the current state.
4. Dynamically decide the next steps based on the results of previous actions and the goal specified in the text input.
The solution should function autonomously to achieve the final desired outcome defined in the text input.
Key Requirements:
1. Core Functionality
• Natural Language Understanding:
○ Parse and understand complex user instructions using AI/ML tools like Gemini, Llama, spaCy, OpenAI GPT - preffered solution that can be setup locally like Llama or with free-tier API.
• Image Analysis:
○ Utilize computer vision frameworks such as OpenCV or Tesseract to analyze UI elements on the screenshots.
• Automation Execution:
○ Use tools like PyAutoGUI, or other keyboard/mouse libraries to interact with desktop applications and web browsers.
• All mentioned libraries and tools are just a samples, please use what you find suitable for the task.
2. System Capabilities
• Perform a variety of actions such as opening applications, typing into fields, clicking buttons, scrolling, and navigating web pages. It shall be able to execute that tasks not only on the browser but also other desktop applications.
• Adapt to different starting points (e.g., desktop or specific application) using the provided image input.
• Dynamically update the action plan if a step produces unexpected results (e.g., previous step led to unwanted state or dead-end).
• In case that script hangs and can't find reliable next step, shall pause the exeuction, ask user for the input (e.g. go to correct page) and then resume execution. However it shouldn't be overused and treated as last resort.
3. Input Handling
• Accept and interpret plain text instructions.
• Analyze a starting screenshot to identify key visual elements such as browser windows, search bars, and buttons.
4. Output Requirements
• Provide a log of executed actions for transparency.
• Save screenshots of intermediate and final states during task execution.
• Notify the user upon task completion or if an issue arises (e.g., an unrecognized element).
5. AI/ML Requirements
• Use NLP to extract actions, objects, and intents from the input text.
• Employ pre-trained models as necessary to minimize training efforts (for e.g., identifying buttons, links, text fields).
6. General Requirements
• The solution shall run on Windows/Mac/Linux environments. Mac/Linux can be reconsidered if it brings significant additional cost.
• The system should have a modular design to allow future feature enhancements.
Robust error handling.
We are looking to build a Python-based AI/ML-powered solution capable of executing a sequence of user-defined tasks on a computer. The system should accept two inputs:
1. Text Input:
○ Example 1 (General): "Go to amazon.com and order 1kg of coffee."
○ Example 2 (Detailed): "Open Chrome browser, in the search bar type 'amazon.com', and in the search bar on the opened website look for 1kg of coffee."
2. Image Input:
○ A screenshot of the starting state of the user's desktop or browser.
The program will then:
1. Interpret the text instructions using AI/ML models like NLP (Natural Language Processing).
2. Analyze the provided image to understand the current state of the UI.
3. Perform the actions in a step-by-step manner on the user's computer, taking a screenshot after each step to reassess the current state.
4. Dynamically decide the next steps based on the results of previous actions and the goal specified in the text input.
The solution should function autonomously to achieve the final desired outcome defined in the text input.
Key Requirements:
1. Core Functionality
• Natural Language Understanding:
○ Parse and understand complex user instructions using AI/ML tools like Gemini, Llama, spaCy, OpenAI GPT - preffered solution that can be setup locally like Llama or with free-tier API.
• Image Analysis:
○ Utilize computer vision frameworks such as OpenCV or Tesseract to analyze UI elements on the screenshots.
• Automation Execution:
○ Use tools like PyAutoGUI, or other keyboard/mouse libraries to interact with desktop applications and web browsers.
• All mentioned libraries and tools are just a samples, please use what you find suitable for the task.
2. System Capabilities
• Perform a variety of actions such as opening applications, typing into fields, clicking buttons, scrolling, and navigating web pages. It shall be able to execute that tasks not only on the browser but also other desktop applications.
• Adapt to different starting points (e.g., desktop or specific application) using the provided image input.
• Dynamically update the action plan if a step produces unexpected results (e.g., previous step led to unwanted state or dead-end).
• In case that script hangs and can't find reliable next step, shall pause the exeuction, ask user for the input (e.g. go to correct page) and then resume execution. However it shouldn't be overused and treated as last resort.
3. Input Handling
• Accept and interpret plain text instructions.
• Analyze a starting screenshot to identify key visual elements such as browser windows, search bars, and buttons.
4. Output Requirements
• Provide a log of executed actions for transparency.
• Save screenshots of intermediate and final states during task execution.
• Notify the user upon task completion or if an issue arises (e.g., an unrecognized element).
5. AI/ML Requirements
• Use NLP to extract actions, objects, and intents from the input text.
• Employ pre-trained models as necessary to minimize training efforts (for e.g., identifying buttons, links, text fields).
6. General Requirements
• The solution shall run on Windows/Mac/Linux environments. Mac/Linux can be reconsidered if it brings significant additional cost.
• The system should have a modular design to allow future feature enhancements.
Robust error handling.