Tekkotsu relies on color coding to recognize and track objects in the environment.  This method is fast, and can be robust once it has been configured for the lighting conditions and colors that will be used.

Although we release a basic configuration with our software, this is configured for our lab and our tasks.  If you have bright fluorescent lights, and can find similarly colored objects to ours (such as the Aibo's standard pink ball), this may work just fine for you.  Use the Seg Cam button in the ControllerGUI to test how well the segmentation is working.

Otherwise, you will probably need to design a new set of color thresholds using sample images from the Aibo itself.

External Resources

Set Camera Gain and Shutter Speed

1. Launch the Controller and bring up the Raw Cam.
   If you have trouble with this step, take a look at the TekkotsuMon Tutorial.
2. If the image appears too dark or too bright, you'll need to adjust the shutter speed and/or gain.  Use the following commands in the Controller text input field (or on the console):
   
   !set vision.shutter_speed=[fast | mid | slow]
   !set vision.gain=[high | mid | low]
   
   A low shutter speed will brighten the image, but cause more motion blur
   A high gain will brighten the image, but cause more graininess/noise
   
3. Once you've found the proper gain and shutter speed, you should probably make those changes permanent by editing the <project>/ms/config/tekkotsu.cfg file.

Take Sample Images

1. The following commands in the ControllerGUI input box will give you a good configuration for taking sample images. In particular, these commands switch to a high-resolution, no compression, low framerate format, which is good for taking a number of high quality sample images. Normally the camera runs in a relatively low-resolution, high framerate mode.
   !set vision.rawcam_interval=1000
   !set vision.rawcam_transport=tcp
   !set vision.rawcam_y_skip=1
   !set vision.rawcam_uv_skip=1
   !set vision.rawcam_compression=none
   
   After entering these commands, you will need to close and reopen the Raw Cam (if it was already running) so that it will reconnect with TCP (instead of the default UDP).
2. In the Raw Cam, switch to YUV mode.
   YUV mode is the native format of the camera.  Using this format will ensure accurate color.  However, if you have RGB images, you'll still be able to use them later on.
   
3. For each object you want to recognize, you need to take several samples:
   
   1. Hold it in front of the camera
   2. Press the "Freeze Frame" button
   3. Click the "Save Image" button
   4. Press the "Unfreeze" to see the next item
      
   You should take several images of each object, hopefully under varying positions and light conditions. 
   • You could save an image sequence of the object being moved in front of the camera, but don't over do it or you will have too many images to deal with later. (The 'interval' setting above should help prevent this from being a problem by limiting the frame rate.)
   • Save the images in PNG format.  This will provide lossless storage.  JPEG compression will smooth noise, which provides better segmentation in the test images, but will be unrealistic. 
   • However, if you do use JPEG, switch to RGB mode - JPEG uses YUV internally, so if you save in VisionGUI's YUV mode, you actually wind up converting color spaces.
   • Take a sample image(s) of the background as well.  (taking the pictures on a black (or any solid color) background will also simplify things later)

Load Images into EasyTrain

1. In the <Tekkotsu>/tool/easytrain directory, run the following command:
     java EasyTrain -isYUV <image_files>
   If you saved the images as RGB (not recommended unless using JPEG) then pass -isRGB instead of -isYUV.
   If you wish to use the provided sample images, pass test*.png for the files parameter.
   If you get an Out of Memory error either delete some files or try increasing Java's max memory size (try passing it the -X option for more info)
2. Five windows should come up:

   RGB Image View: displays the current image file in RGB color space.	Segmented Viewer: displays the color-segmented version of the current image file. (Initially, it will be solid grey. This figure shows the window after colors have been defined.)
   Color Spectrum Window: displays pixels from selected image regions in a 2D color space representation. Draw a bounding region with the mouse to select pixel clusters for segmentation.
   Control window: contains buttons to add or remove colors, load or save files, etc. Hover over items for tooltips.	Thumbnail window: shows thumbnails of loaded images, allows you to directly jump to specific images by clicking on them.

3. Decide which color/object you want to segment first. Enter the name in the Controls window and hit 'enter'. This should add a new entry in the list with the specified name.
   The colors selected for earlier regions will override those selected for later regions, so you should do the segmentation for "important" things first, so that the leftovers go to the less important colors/objects.
4. Select a region of colors in the spectrum window. Notice that the segmentation window is updated to show those areas of the image which map into that region of color space.

   A yellow region of HSB colorspace selected (left), with corresponding segmented view (bottom right) and original RGB image for reference (top right). Windows are small for readability -- in practice you would want to resize the windows for better visibility.

   • If you draw another region in the spectrum, it will replace the previous one.
   • If you wish to 'add' to the selection, hold down the shift key before you begin a selection, and the new selection will be unioned with the previous selection.
   • If you wish to 'remove' from the current selection, hold down the control key.
   • Use the arrow keys or the thumbnail window to view the results of segmentation on other images you have loaded.
5. Repeat previous two steps to define the colors/objects which interest you.
6. Click the 'save' button in the Control window to generate the .tm (threshold map) and .col (color definition) files, and save selectioned areas. The .tm and .col files will be going to the Aibo's memory stick, so choose a name with 8 or fewer characters so you won't have to rename the files to fit on the memory stick filesystem.
7. If you later wish to re-edit the thresholds you have defined, you can load the .spec file, which stores the selections you made in the spectrum window. (The .tm and .col are deriviatives of this information, and do not store enough information to reconstitute the original selections.)
8. To load the threshold files on the Aibo, see the section "Loading Threshold Files" below.

Segmentation Hints and Tips

• Moving the mouse within the RGB window will display a corresponding "hint" in the Spectrum window to indicate the position of the color the mouse is currently over.
• Selecting a region in the RGB window will show only those pixels in the Spectrum window. (Unless "all pixels" is selected, in which case RGB window selections are ignored.)
• You can undo with Ctrl-Z, redo with Ctrl-Shift-Z.
• Ctrl-A will select all, Ctrl-D to de-select all (clear selection).
• You can resize windows with '+' and '-' to zoom by integer multiples, or '=' to reset to "native" resolution.
• You can use arrow keys in any of the image windows to move between images.
• During runtime, noise removal is done during the run length encoding stage of processing. This will drop small areas of noise, but is responsible for the horizontal banding you will find in highly noisy areas.
• Shadows and hilights are hard to segment. Colors near pure white or black are out of bounds to avoid false positives. (With some further development, these could be mapped to their own colors, but currently they are special cases.)
• There are a number of color spaces available in the Control window. Each has some advantages and disadvantages, and might be better for different operations. However, only one space can be used at a time, and selections cannot be translated between them.
• Use the 'help' button to bring up a condensed version of these notes

Post-Processing: Merging Threshold Files