Examples: Vision ======================== This page shows how to use Ned's vision set | If you want see more about Ned's vision functions, you can look at :ref:`PyNiryo - Vision` | If you want to see how to do image processing, go check out the Pyniryo (first version) doc. .. note:: Even if you do not own a Vision Set, you can still realize these examples with the Gazebo simulation version .. danger:: If you are using the real robot, make sure the environment around it is clear Needed piece of code ------------------------------- .. important:: In order to achieve following examples, you need to have create a vision workspace. In this page, the workspace used is named ``workspace_1``. To create it, the user should go on Niryo Studio ! As the examples start always the same, add the following lines at the beginning of codes :: # Imports from pyniryo import * # - Constants workspace_name = "workspace_1" # Robot's Workspace Name robot_ip_address = "x.x.x.x" # The pose from where the image processing happens observation_pose = PoseObject( x=0.16, y=0.0, z=0.35, roll=0.0, pitch=1.57, yaw=0.0, ) # Place pose place_pose = PoseObject( x=0.0, y=-0.2, z=0.12, roll=0.0, pitch=1.57, yaw=-1.57 ) # - Initialization # Connect to robot robot = NiryoRobot(robot_ip_address) # Calibrate robot if robot needs calibration robot.arm.calibrate_auto() # Updating tool robot.tool.update_tool() # --- -------------- --- # # --- CODE GOES HERE --- # # --- -------------- --- # robot.end() .. hint:: All the following examples are only of part of what can be made with the API in terms of vision. We advise you to look at :ref:`API - Vision` to understand more deeply Simple Vision Pick & Place ------------------------------- The goal of a Vision Pick & Place is the same as a classical Pick & Place, with a close difference : the camera detects where the robot has to go in order to pick ! This short example show how to do your first vision pick using the :meth:`~.vision.Vision.vision_pick` function : :: robot.move_pose(observation_pose) # Trying to pick target using camera obj_found, shape_ret, color_ret = robot.vision.vision_pick(workspace_name) if obj_found: robot.pick_place.place_from_pose(place_pose) robot.arm.set_learning_mode(True) .. _code_details_simple_vision_pick_n_place: Code Details - Simple Vision Pick and Place ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To execute a Vision pick, we firstly need to go to a place where the robot will be able to see the workspace :: robot.arm.move_pose(observation_pose) Then, we try to perform a vision pick in the workspace with the :meth:`~.vision.Vision.vision_pick` function :: obj_found, shape_ret, color_ret = robot.vision.vision_pick(workspace_name) Variables ``shape_ret`` and ``color_ret`` are respectively of type :class:`~.vision.enums.ObjectShape` and :class:`~.vision.enums.ObjectColor`, and store the shape and the color of the detected object ! We won't use them for this first example. The ``obj_found`` variable is a boolean which indicates whereas an object has been found and picked, or not. Thus, if the pick worked, we can go place the object at the place pose. :: if obj_found: robot.pick_place.place_from_pose(place_pose) Finally, we turn learning mode on:: robot.arm.set_learning_mode(True) .. note:: If you ``obj_found`` variable indicates ``False``, check that : * Nothing obstruct the camera field of view * Workspace's 4 markers are visible * At least 1 object is placed fully inside the workspace First conditioning via Vision ------------------------------------------- In most of use cases, the robot will need to perform more than one Pick & Place. In this example, we will see how to condition multiple objects according to a straight line :: # Initializing variables offset_size = 0.05 max_catch_count = 4 # Loop until enough objects have been caught catch_count = 0 while catch_count < max_catch_count: # Moving to observation pose robot.arm.move_pose(observation_pose) # Trying to get object via Vision Pick obj_found, shape, color = robot.vision.vision_pick(workspace_name) if not obj_found: robot.wait(0.1) continue # Calculate place pose and going to place the object next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size) robot.pick_place.place_from_pose(next_place_pose) catch_count += 1 robot.arm.go_to_sleep() .. _code_details_first_conditionning_via_vision: Code Details - First Conditioning via Vision ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We want to catch ``max_catch_count`` objects, and space each of them by ``offset_size`` meter :: offset_size = 0.05 max_catch_count = 4 We start a loop until the robot has caught ``max_catch_count`` objects :: catch_count = 0 while catch_count < max_catch_count: For each iteration, we firstly go to the observation pose and then, try to make a vision pick in the workspace :: robot.arm.move_pose(observation_pose) obj_found, shape, color = robot.vision.vision_pick(workspace_name) If the vision pick failed, we wait 0.1 second and then, start a new iteration :: if not obj_found: robot.wait(0.1) continue Else, we compute the new place position according to the number of catches, and then, go placing the object at that place :: next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size) robot.pick_place.place_from_pose(next_place_pose) We also increment the ``catch_count`` variable :: catch_count += 1 Once the target catch number is achieved, we go to sleep :: robot.arm.go_to_sleep() Multi Reference Conditioning ------------------------------- During a conditioning task, objects may not always be placed as the same place according to their type. In this example, we will see how to align object according to their color, using the color element :class:`~.vision.enums.ObjectColor` returned by :meth:`~.vision.Vision.vision_pick` function :: # Distance between elements offset_size = 0.05 max_failure_count = 3 # Dict to write catch history count_dict = { ObjectColor.BLUE: 0, ObjectColor.RED: 0, ObjectColor.GREEN: 0, } try_without_success = 0 # Loop until too much failures while try_without_success < max_failure_count: # Moving to observation pose robot.arm.move_pose(observation_pose) # Trying to get object via Vision Pick obj_found, shape, color = robot.vision.vision_pick(workspace_name) if not obj_found: try_without_success += 1 robot.wait(0.1) continue # Choose X position according to how the color line is filled offset_x_ind = count_dict[color] # Choose Y position according to ObjectColor if color == ObjectColor.BLUE: offset_y_ind = -1 elif color == ObjectColor.RED: offset_y_ind = 0 else: offset_y_ind = 1 # Going to place the object next_place_pose = place_pose.copy_with_offsets(x_offset=offset_x_ind * offset_size, y_offset=offset_y_ind * offset_size) robot.pick_place.place_from_pose(next_place_pose) # Increment count count_dict[color] += 1 try_without_success = 0 robot.arm.go_to_sleep() .. _code_details_multi_ref_conditioning: Code Details - Multi Reference Conditioning ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We want to catch objects until Vision Pick failed ``max_failure_count`` times. Each of the object will be put on a specific column according to its color. The number of catches for each color will be store on a dictionary ``count_dict`` :: # Distance between elements offset_size = 0.05 max_failure_count = 3 # Dict to write catch history count_dict = { ObjectColor.BLUE: 0, ObjectColor.RED: 0, ObjectColor.GREEN: 0, } try_without_success = 0 # Loop until too much failures while try_without_success < max_failure_count: For each iteration, we firstly go to the observation pose and then, try to make a vision pick in the workspace :: robot.move_pose(observation_pose) obj_found, shape, color = robot.vision.vision_pick(workspace_name) If the vision pick failed, we wait 0.1 second and then, start a new iteration, without forgetting the increment the failure counter :: if not obj_found: try_without_success += 1 robot.wait(0.1) continue Else, we compute the new place position according to the number of catches, and then, go placing the object at that place :: # Choose X position according to how the color line is filled offset_x_ind = count_dict[color] # Choose Y position according to ObjectColor if color == ObjectColor.BLUE: offset_y_ind = -1 elif color == ObjectColor.RED: offset_y_ind = 0 else: offset_y_ind = 1 # Going to place the object next_place_pose = place_pose.copy_with_offsets(x_offset=offset_x_ind * offset_size, y_offset=offset_y_ind * offset_size) robot.pick_place.place_from_pose(next_place_pose) We increment the ``count_dict`` dictionary and reset ``try_without_success`` :: count_dict[color] += 1 try_without_success = 0 Once the target catch number is achieved, we go to sleep :: robot.arm.go_to_sleep() Sorting Pick with Conveyor ------------------------------- An interesting way to bring objects to the robot, is the use of a Conveyor Belt. In this examples, we will see how to catch only a certain type of object by stopping the conveyor as soon as the object is detected on the workspace :: # Initializing variables offset_size = 0.05 max_catch_count = 4 shape_expected = ObjectShape.CIRCLE color_expected = ObjectColor.RED conveyor_id = robot.conveyor.set_conveyor() catch_count = 0 while catch_count < max_catch_count: # Turning conveyor on robot.conveyor.run_conveyor(conveyor_id) # Moving to observation pose robot.arm.move_pose(observation_pose) # Check if object is in the workspace obj_found, pos_array, shape, color = robot.vision.detect_object(workspace_name, shape=shape_expected, color=color_expected) if not obj_found: robot.wait(0.5) # Wait to let the conveyor turn a bit continue # Stopping conveyor robot.conveyor.stop_conveyor(conveyor_id) # Making a vision pick obj_found, shape, color = robot.visionvision_pick(workspace_name, shape=shape_expected, color=color_expected) if not obj_found: # If visual pick did not work continue # Calculate place pose and going to place the object next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size) robot.pick_place.place_from_pose(next_place_pose) catch_count += 1 # Stopping & unsetting conveyor robot.conveyor.stop_conveyor(conveyor_id) robot.conveyor.unset_conveyor(conveyor_id) robot.arm.go_to_sleep() Code Details - Sort Picking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Firstly, we initialize your process : we want the robot to catch 4 Red Circles. To do so, we set variables ``shape_expected`` and ``color_expected`` with :attr:`ObjectShape.CIRCLE ` and :attr:`ObjectColor.RED ` :: offset_size = 0.05 max_catch_count = 4 shape_expected = ObjectShape.CIRCLE color_expected = ObjectColor.RED We activate the connection with the conveyor and start a loop until the robot has caught ``max_catch_count`` objects :: conveyor_id = robot.set_conveyor() catch_count = 0 while catch_count < max_catch_count: For each iteration, we firstly run the conveyor belt (if the later is already running, nothing will happen), then go to the observation pose :: # Turning conveyor on robot.conveyor.run_conveyor(conveyor_id) # Moving to observation pose robot.arm.move_pose(observation_pose) We then check if an object corresponding to our criteria is in the workspace. If not, we wait 0.5 second and then, start a new iteration :: obj_found, pos_array, shape, color = robot.vision.detect_object(workspace_name, shape=shape_expected, color=color_expected) if not obj_found: robot.wait(0.5) # Wait to let the conveyor turn a bit continue Else, stop the conveyor and try to make a vision pick :: # Stopping conveyor robot.conveyor.stop_conveyor(conveyor_id) # Making a vision pick obj_found, shape, color = robot.vision.vision_pick(workspace_name, shape=shape_expected, color=color_expected) if not obj_found: # If visual pick did not work continue If Vision Pick succeed, compute new place pose, and place the object :: # Calculate place pose and going to place the object next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size) robot.pick_place.place_from_pose(next_place_pose) catch_count += 1 Once the target catch number is achieved, we stop the conveyor and go to sleep :: # Stopping & unsetting conveyor robot.conveyor.stop_conveyor(conveyor_id) robot.conveyor.unset_conveyor(conveyor_id) robot.arm.go_to_sleep()