Grasp Objects in Pybullet Environment, Apply inverse kinematics algorithm in simulated environment
Budget: $10 – $30 USD
Grasp Objects in Pybullet Environment
1. Project Objective
• Get familiar with 3D robot simulation
• Understand intelligent robot programming
• Apply inverse kinematics algorithm in simulated environment
2. Project Steps
1. Install pybullet with Anaconda (recommended)
◦ Install Anaconda: https://www.anaconda.com/products/individual
◦ Install Pybullet: https://anaconda.org/conda-forge/pybullet
2. (If you have done the first step, please ignore)
Install python3 (optional)
Setup the python3 environment on your system.
◦ Windows 10:
▪ https://phoenixnap.com/kb/how-to-install-python-3-windows
◦ Mac:
▪ https://docs.python-guide.org/starting/install3/osx/
◦ Ubuntu 16.04:
▪ https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04
3. (If you have done the first step, please ignore)
Setup Pybullet Environment:
◦ Run following commands in the terminal:
▪ pip3 install pybullet --upgrade --user
▪ python3 -m pybullet_envs.examples.enjoy_TF_AntBulletEnv_v0_2017may
▪ python3 -m pybullet_envs.examples.enjoy_TF_HumanoidFlagrunHarderBulletEnv_v1_2017jul
▪ python3 -m pybullet_envs.deep_mimic.testrl --arg_file run_humanoid3d_backflip_args.txt
◦ Pybullet Reference: https://github.com/bulletphysics/bullet3
4. Load assigned object in the environment
◦ In sawyerEnv.py, change the variable object_path to your assigned object.
◦ In main.py, update object_path to your assigned object.
5. Record the tray location in the environment
◦ run tune_grasping.py
◦ Click the GUI and Press Q on your keyboard
◦ Record the tray_x and tray_y
◦ update tray_x and tray_y in both sawyerEnv.py and main.py
◦ Please make sure the tray and the object is not conflict.
6. Run tune_grasping.py
◦ Adjust your hand position and grasp type by using the slider
◦ After you have adjusted your hand position. Press Q on your keyboard
◦ you can get 3 outputs
▪ handIReading: the pose of the hand
▪ palmPosition: arm postion
▪ Orientation: arm orientation
7. Find grasping trajectory
◦ Run tune_grasping.py and record parameters for each step in lines 340 – 376 in main.py
▪ handIReading
• handInitial
• handClose
• handOpen
▪ palmPosition and orientation
• grasp_palmPosition, grasp_orientation
• pu_palmPosition, pu_orientation
• final_palmPosition, final_orientation
8. Update the outputs in step 7 to main.py
9. write code for the grasping task
◦ Use Inverse Kinematics Function in main.py to control the position and orientation of the arm
◦ Use Hand Direct Control Function in main.py to control the hand
10. Record a video for successfully trail
3. What you need to submit
• Functional codes (30%)
• Demo illustration (30%)
◦ A video that contains one trial of successfully grasping the target objects.
• Project Report (40%)
◦ A project report that in academic paper format should be written for this project (IEEE conference templates) (1%).
◦ The report should include:
▪ Introduction (5%)
• Background about robotic Object Recognition and Grasping (2%)
• Procedure of the project (2%)
▪ Project Descriptions (10%)
• Describe the procedure (2%)
• Describe the applied algorithm/methods of the project (4%)
• Save the grasping parameters of the hand and fingers into a “.csv” file. For example, if your object is “203”, save your parameters into a “203.csv” including all parameters in step 7 (3%)
• Include a picture of the hand gesture (1%)
▪ Results (10%)
• Trajectory of the object in grasping (5%)
• Snapshots of successful and failed grasps (if there are any) (5%)
▪ Discussion of results (10%)
• Discuss the advantages and disadvantages of the designed hand gesture in grasping the object. Why is it successful? (5%)
• Discuss the designed trajectory of hand movement. What are the key parameters and how are the parameters are selected? (5%)
▪ Conclusion (3%)
• Conclude the report, what you have done (1%), what results you have got (1%), what worked and what did not (1%).
▪ Reference (1%)
1. Project Objective
• Get familiar with 3D robot simulation
• Understand intelligent robot programming
• Apply inverse kinematics algorithm in simulated environment
2. Project Steps
1. Install pybullet with Anaconda (recommended)
◦ Install Anaconda: https://www.anaconda.com/products/individual
◦ Install Pybullet: https://anaconda.org/conda-forge/pybullet
2. (If you have done the first step, please ignore)
Install python3 (optional)
Setup the python3 environment on your system.
◦ Windows 10:
▪ https://phoenixnap.com/kb/how-to-install-python-3-windows
◦ Mac:
▪ https://docs.python-guide.org/starting/install3/osx/
◦ Ubuntu 16.04:
▪ https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04
3. (If you have done the first step, please ignore)
Setup Pybullet Environment:
◦ Run following commands in the terminal:
▪ pip3 install pybullet --upgrade --user
▪ python3 -m pybullet_envs.examples.enjoy_TF_AntBulletEnv_v0_2017may
▪ python3 -m pybullet_envs.examples.enjoy_TF_HumanoidFlagrunHarderBulletEnv_v1_2017jul
▪ python3 -m pybullet_envs.deep_mimic.testrl --arg_file run_humanoid3d_backflip_args.txt
◦ Pybullet Reference: https://github.com/bulletphysics/bullet3
4. Load assigned object in the environment
◦ In sawyerEnv.py, change the variable object_path to your assigned object.
◦ In main.py, update object_path to your assigned object.
5. Record the tray location in the environment
◦ run tune_grasping.py
◦ Click the GUI and Press Q on your keyboard
◦ Record the tray_x and tray_y
◦ update tray_x and tray_y in both sawyerEnv.py and main.py
◦ Please make sure the tray and the object is not conflict.
6. Run tune_grasping.py
◦ Adjust your hand position and grasp type by using the slider
◦ After you have adjusted your hand position. Press Q on your keyboard
◦ you can get 3 outputs
▪ handIReading: the pose of the hand
▪ palmPosition: arm postion
▪ Orientation: arm orientation
7. Find grasping trajectory
◦ Run tune_grasping.py and record parameters for each step in lines 340 – 376 in main.py
▪ handIReading
• handInitial
• handClose
• handOpen
▪ palmPosition and orientation
• grasp_palmPosition, grasp_orientation
• pu_palmPosition, pu_orientation
• final_palmPosition, final_orientation
8. Update the outputs in step 7 to main.py
9. write code for the grasping task
◦ Use Inverse Kinematics Function in main.py to control the position and orientation of the arm
◦ Use Hand Direct Control Function in main.py to control the hand
10. Record a video for successfully trail
3. What you need to submit
• Functional codes (30%)
• Demo illustration (30%)
◦ A video that contains one trial of successfully grasping the target objects.
• Project Report (40%)
◦ A project report that in academic paper format should be written for this project (IEEE conference templates) (1%).
◦ The report should include:
▪ Introduction (5%)
• Background about robotic Object Recognition and Grasping (2%)
• Procedure of the project (2%)
▪ Project Descriptions (10%)
• Describe the procedure (2%)
• Describe the applied algorithm/methods of the project (4%)
• Save the grasping parameters of the hand and fingers into a “.csv” file. For example, if your object is “203”, save your parameters into a “203.csv” including all parameters in step 7 (3%)
• Include a picture of the hand gesture (1%)
▪ Results (10%)
• Trajectory of the object in grasping (5%)
• Snapshots of successful and failed grasps (if there are any) (5%)
▪ Discussion of results (10%)
• Discuss the advantages and disadvantages of the designed hand gesture in grasping the object. Why is it successful? (5%)
• Discuss the designed trajectory of hand movement. What are the key parameters and how are the parameters are selected? (5%)
▪ Conclusion (3%)
• Conclude the report, what you have done (1%), what results you have got (1%), what worked and what did not (1%).
▪ Reference (1%)