Few-Shot Incremental Learning

Ali Ayub and Alan R. Wagner

Abstract: For many applications, robots will need to be incrementally trained to recognize the specific objects needed for an application. Imagine, for example, a domestic robot tasked with locating and organizing household items. We would like for this robot to be trained on what items it should organize by its non-expert owner and recognize that the items to be organized might change over time. Although it could be possible to train the system on an enormous corpus of data containing a vast number of objects, hoping that all of the objects that the robot will one day be asked to organize will be in the dataset, this approach seems destined for failure. Ideally, the robot should be taught about important objects incrementally, and, because people will demand quick results, from only a few examples. We seek to develop a practical system that would allow a novice human to teach a robot about different object categories incrementally using only a small set of visual examples provided by the person. We refer to this problem as Few-Shot Incremental Learning (FSIL).

Centroid Based Concept Learning (CBCL)

To solve FSIL, we propose a novel cognitively-inspired method termed Centroid-Based Concept Learning (CBCL). CBCL is inspired by the concept learning model of the hippocampus and the neocortex. CBCL treats each image as an episode and extracts its high-level features. CBCL uses a fixed data representation (ResNet pre-trained on ImageNet) for feature extraction. After feature extraction, CBCL generates a set of concepts in the form of centroids for each class using a cognitively-inspired clustering approach (denoted as Agg-Var clustering). After generating the centroids, to predict the label of a test image, the distance of the feature vector of the test image to the n closest centroids is used. Since CBCL stores the centroids for each class independently of the other classes,the decrease in overall classification accuracy is not catastrophic when new classes are learned.

Performance

Average and standard deviation of classification accuracies (%) on CIFAR-100 dataset with (a) 2, (b) 5, (c) 10, (d) 20 classes per increment with 10 executions. Average incremental accuracies are shown in parenthesis.

Few-Shot Incremental Learning on a Robot

We use CBCL for few-shot incremental learning of objects for a table cleaning task. First, the robot is taught classes of objects to clear from a few examples provided by a human incrementally. The robot then identifies objects belonging to the trained classes from a clutter of objects on the table and organizes the objects by type. We also demonstrate the system’s ability to learn different object arrangements as semantic concepts. After learning the object classes incrementally from the examples provided by the human, the robot then uses the object’s location and incremental classification to learn about the object-arrangements from a single example provided by the human. The robot can then recognize missing or wrong objects in different object arrangements.

(Left) Some example images of object classes toothpaste and hair clip learned by the robot using the camera in the robot’s hand. (Right) An example of Baxter robot cleaning objects belonging to class soap from a set of objects in front of it’s arm camera on the table.

Performance

Comparison of CBCL to the Few-shot Learning Baseline (FLB) and Fine-Tuning (FT) for 5-shot and 10-shot incremental learning in terms of classification accuracy (%). (a) and (b) show results for 5-shot and 10-shot incremental learning with 1 class per increment, while (c) and (d) show results for 5-shot and 10-shot incremental learning with 2 classes per increment. Mean and standard deviations of the classification accuracy is reported for each increment with 10 executions. Average incremental accuracy is shown in brackets for each method.

Related Publications

A. Ayub and A. R. Wagner, “Tell me what this is: Few-Shot Incremental Object Learning by a Robot”, Accepted at IEEE IROS, 2020 [Paper]
A. Ayub and A. R. Wagner, “Cognitively-Inspired Model for Incremental Learning Using a Few Examples”, in The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020. [Paper] [Code][Talk]