Welcome to Continuum’s documentation!¶
Continuum is the library you need for Continual Learning. It supports many datasets and most scenarios (NC, NI, NIC, etc.).
Continuum is made of two parts: Dataset and Scenario. The former is about the actual dataset with sometimes minor modifications to fit in the Continual paradigm. The latter is about the different setting you may encounter in Continual Learning.
Classic Scenarios¶
We propose here a list of classic continual learning scenarios used in the literature. For each, scenarios we show how to create it. For using it, you may look at scenarios documentation
split MNIST: 5 tasks, number of classes per tasks: 2
from continuum import ClassIncremental
from continuum.datasets import MNIST
scenario = ClassIncremental(
MNIST(data_path="my/data/path", download=True, train=True),
increment=2
)
split CIFAR100: 6 tasks, first 50 classes then 10 classes per tasks.
from continuum import ClassIncremental
from continuum.datasets import CIFAR100
scenario = ClassIncremental(
CIFAR100(data_path="my/data/path", download=True, train=True),
increment=10,
initial_increment=50
)
Concatenation of CIFAR10 & CIFAR100, made of 11 tasks of 10 classes each
from continuum import ClassIncremental
from continuum.datasets import CIFARFellowship
scenario = ClassIncremental(
CIFARFellowship(data_path="my/data/path", download=True, train=True),
increment=10,
)
Permut MNIST: 5 tasks with different label space for each task
from continuum import Permutations
from continuum.datasets import MNIST
scenario = Permutations(
MNIST(data_path="my/data/path", download=True, train=True),
nb_tasks=5,
seed=0,
shared_label_space=False
)
Rotations MNIST: 3 tasks, rotation 0-45-90 degrees with different label space for each task
from continuum import Rotations
from continuum.datasets import MNIST
scenario = Rotations(
MNIST(data_path="my/data/path", download=True, train=True),
nb_tasks=3,
list_degrees=[0,45,90]
)
For more info scenarios documentation.
Scenario Generators¶
Scenario generators are object that can produce various version of a certain type of scenarios. For example, they can generate scenario with various task orders. It enables to test easily an algorithm on different flavors of a scenario types. It also make possible to create a never ending sequence of tasks with repetition of task for long training.
Following the different types of generator:
Task Order Generator¶
This generator creates scenarios with various order of tasks:
from continuum.generators import TaskOrderGenerator
# example with ClassIncremental scenario but any type of scenario works
from continuum.scenarios import ClassIncremental
from continuum.datasets import MNIST
dataset = MNIST('my/data/path', train=True)
scenario = ClassIncremental(dataset, increment=2)
scenario_generator = TaskOrderGenerator(scenario)
NB_SCENARIO = 5 # let say we want to generate 5 scenarios
for scenario_id in range(NB_SCENARIO):
# sample with seed for reproducibility
scenario = scenario_generator.sample(seed=scenario_id)
# each scenario has a different task order
# (However, the tasks stays the same.)
Class Order Generator¶
This generator shuffle the class order but keeps the same number of classes per tasks. This generator is suited for ClassIncremental scenarios.
from continuum.generators import ClassOrderGenerator
from continuum.scenarios import ClassIncremental
from continuum.datasets import MNIST
dataset = MNIST('my/data/path', train=True)
scenario = ClassIncremental(dataset, increment=[2, 3, 1, 4])
scenario_generator = ClassOrderGenerator(scenario)
NB_SCENARIO = 5 # let say we want to generate 5 scenarios
for scenario_id in range(NB_SCENARIO):
# sample with seed for reproducibility
scenario = scenario_generator.sample(seed=scenario_id)
# The increment order will stay [2, 3, 1, 4]
# but the classes used in each task will be different
Hash Generator¶
This generator creates scenario from a list of hash. When sampled, it select randomly a hash function and create a scenario from it. The scenario can be create with a fixed number of classes or it can decide automatically the number of tasks depending on the hashes distribution.
from continuum.generators import HashGenerator
from continuum.datasets import CIFAR10
dataset = CIFAR10('my/data/path', train=True)
# list of all hash name that can be used ("CropResistantHash" is very slow)
list_hash = ["AverageHash", "Phash", "PhashSimple", "DhashH", "DhashV", "Whash", "ColorHash"
]
# see documentation about HashScenario for more info about parameters
scenario_generator = HashGenerator(cl_dataset=dataset,
list_hash=list_hash,
nb_tasks=5, # if None the nb_tasks will be automatically set
transformations=your_trsf,
filename_hash_indexes="your_path_to save_ids",
split_task="auto") # [auto or balanced]
for scenario_id in range(NB_SCENARIO):
# sample with seed for reproducibility
scenario = scenario_generator.sample(seed=scenario_id)
# the scenario will be with always the same data but in a different stream order
# data are only ordered based on their hash it will then probably be a NIC scenarios.
Continuum Scenarios¶
Continual learning (CL) aim is to learn without forgetting in the most “satisfying” way. To evaluate the capacity of different CL algorithms the community use different type of benchmarks scenarios.
In Continuum, we implemented the four main types of scenarios used in the community: Class Incremental, Instance Incremental, New Class and Instance Incremental, and, Transformed Incremental. Those scenarios are originally suited for classification but they might be adapted for unsupervised learning, self-supervised learning or reinforcement learning.
For each scenario, there is a finite set of tasks and the goal is to learn the tasks one by one and to be able to generalize at the end to a test set composed of data from all tasks.
For clear evaluation purpose, the data from each tasks can not be found in another tasks. The idea is to be able to assess precisely what the CL algorithms is able to remember or not. Those benchmarks purpose is not to mimic real-life scenarios but to evaluate properly algorithms capabilities.
Here is roughly how continual learning scenarios are created and used:
from torch.utils.data import DataLoader
# First we get a dataset that will be used to compose tasks and the continuum
continual_dataset = MyContinualDataset()
# Then the dataset is provided to a scenario class that will process it to create the sequence of tasks
# the continuum might get specific argument, such as number of tasks.
scenario = MyContinuumScenario(
continual_dataset, SomeOtherArguments
)
# The continuum can then be enumerate tasks
for task_id, taskset in enumerate(scenario):
# taskset can be used as a Pytorch Dataset to load the task data
loader = DataLoader(taskset)
for x, y, t in loader:
# data, label, task index
# train on the task here
break
A practical example with split MNIST:
from torch.utils.data import DataLoader
from continuum import ClassIncremental
from continuum.datasets import MNIST
from continuum.tasks import split_train_val
dataset = MNIST(data_path="my/data/path", download=True, train=True)
# split MNIST with 2 classes per tasks -> 5 tasks
scenario = ClassIncremental(dataset, increment=2)
# The continuum can then be enumerate tasks
for task_id, taskset in enumerate(scenario):
# We use here a cool function to split the dataset into train/val with 90% train
train_taskset, val_taskset = split_train_val(taskset, val_split = 0.1)
train_loader = DataLoader(train_taskset)
# train dataset is a normal data loader like in pytorch that can be used to load the task data
for x, y, t in train_loader:
# data, label, task index
# train on the task here
break
# For testing we need to create another loader (It is importan to keep test a train separate)
dataset_test = MNIST(data_path="my/data/path", download=True, train=False)
# You can also create a test scenario to frame test data as train data.
scenario_test = ClassIncremental(dataset_test, increment=2)
# then iterate through tasks
for task_id, test_taskset in enumerate(scenario_test):
test_loader = DataLoader(test_taskset)
for x, y, t in test_loader:
# something
break
# you can also select specific task(s) in the continuum
# It's just python slicing!
# select task i
i = 2
test_taskset = continuum_test[i]
# select tasks i to i+2
test_taskset = continuum_test[i:i+2]
# select all seen tasks up to the i-th task
test_taskset = continuum_test[:i + 1]
# select all tasks
test_taskset = continuum_test[:]
Classes Incremental¶
In short: Each new task bring instances from new classes only.
Aim: Evaluate the capability of an algorithms to learn concept sequentially, i.e. create representaion able to distinguish concepts and find the right decision boundaries without access to all past data.
Some Details: The continuum of data is composed of several tasks. Each task contains class(es) that is/are specific to this task. One class can not be in several tasks.
One example, MNIST class incremental with five balanced tasks, MNIST has 10 classes then: - task 0 contains data points labelled as 0 and 1 - task 1 contains data points labelled as 2 and 3 … - task 4 contains data points labelled as 8 and 9
The Continual Loader ClassIncremental loads the data and batch it in several tasks, each with new classes. See there some example arguments:
from torchvision.transforms import transforms
from continuum.datasets import MNIST
from continuum import ClassIncremental
continual_dataset = MNIST(data_path="my/data/path", download=True, train=True)
# first use case
# first 2 classes per tasks
scenario = ClassIncremental(
continual_dataset,
increment=2,
transformations=[transforms.ToTensor()]
)
# second use case
# first task with 2 classes then 4 classes per tasks until the end
scenario = ClassIncremental(
continual_dataset,
increment=4,
initial_increment=2,
transformations=[transforms.ToTensor()]
)
# third use case
# first task with 2, second task 3, third 1, ...
scenario = ClassIncremental(
continual_dataset,
increment=[2, 3, 1, 4],
transformations=[transforms.ToTensor()]
)
A very important setting of Class Incremental scenarios is the class ordering. learning ‘dog’, then ‘cat’ may results in vastly different results than learning ‘cat’ then ‘dog’. It’s very simple to change the class ordering:
from continuum import ClassIncremental
continual_dataset = MNIST(data_path="my/data/path", download=True, train=True)
# Default class ordering
scenario_1 = ClassIncremental(
continual_dataset,
increment=2,
class_order=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
)
# Another class ordering
scenario_2 = ClassIncremental(
continual_dataset,
increment=2,
class_order=[1, 8, 3, 6, 5, 4, 10, 2, 9, 7]
)
Note that as for every scenario in continuum, you have to create one scenario for the training set, and one scenario for the testing set. Thus remember to use the same class ordering for both!
Instance Incremental¶
In short: Each new tasks bring new instances from known classes.
Aim: Evaluate the capability of an algorithms to improve its generalization capabilities through new data points, i.e. improve representation without access to all past data.
Some Details:
Tasks are made of new instances. By default the samples images are randomly
shuffled in different tasks, but some datasets provide, in addition of the data x and labels y,
a task id t per sample. For example MultiNLI, a NLP dataset, has 5 classes but
with 10 different domains. Each domain represents a new task.
from continuum import InstanceIncremental
from continuum.datasets import MultiNLI
dataset = MultiNLI("/my/path/where/to/download")
scenario = InstanceIncremental(dataset=dataset)
Likewise, CORe50 provides domain ids which are automatically picked up by the InstanceIncremental scenario:
from continuum import InstanceIncremental
from continuum.datasets import Core50v2_79, Core50v2_196, Core50v2_391
scenario_79 = InstanceIncremental(dataset=Core50v2_79("/my/path"))
scenario_196 = InstanceIncremental(dataset=Core50v2_196("/my/path"))
scenario_391 = InstanceIncremental(dataset=Core50v2_391("/my/path"))
The three available version of CORe50 have respectively 79, 196, and 391 tasks. Each task may bring new classes AND new instances of past classes, akin to the NIC scenario.
Another example could be using different dataset with their original classes as for MNISTFellowship:
from continuum import InstanceIncremental
from continuum.datasets import MNISTFellowship
# We create MNISTFellowship dataset and we keep original labels
dataset = MNISTFellowship("/my/path/where/to/download", update_labels=False)
scenario = InstanceIncremental(dataset=dataset, nb_tasks=3)
In this case, the three dataset MNIST, Fashion-MNIST and KMNIST will be learn with their original labels, e.g. classes 0 of all dataset stay 0. Or with some other dataset:
from continuum import InstanceIncremental
from continuum.datasets import CIFAR100
dataset = CIFAR100("/my/path/where/to/download")
scenario = InstanceIncremental(dataset=dataset, nb_tasks=42)
As you can see, for the last two examples, you need to provide the number of tasks. Because while MultiNLI and CORe50 provide the domain/task ids of each sample, we don’t have this information for other datasets such as MNISTFellowhsip or CIFAR100. In this latter case, you must specify a number of tasks, and then the dataset will be split randomly in this amount of tasks.
Instance incremental scenarios can also be create with transformation as described in next section.
Transformed Incremental¶
In short: Similar to instance incremental, each new tasks bring same instance with a different transformation (ex: images rotations, pixels permutations, …)
Aim: Evaluate the capability of an algorithms to improve its generalization capabilities through new data points, i.e. improve representation without access to all past data.
Some Details: The main difference with instance incremental, is that the scenarios builder has control of the different transformation spaces. It is then easier to evaluate in which transformation space the algorithm is still able to generalize or not.
NB: the transformation used are pytorch.transforms classes
from continuum import TransformationIncremental
list_of_transformation = [Trsf_0, Trsf_1, Trsf_2]
# three tasks continuum, tasks 0 with Trsf_0 transformation
scenario = TransformationIncremental(
dataset=my_continual_dataset,
incremental_transformations=list_of_transformation
)
You can use TransformationIncremental along with the BackgroundSwap transform to create a domain incremental setting with changing backgrounds This is inspired by the Mnist meta-sets from the following paper.
from continuum import TransformationIncremental
from continuum.datasets import CIFAR10, MNIST
from continuum.transforms.bg_swap import BackgroundSwap
import torchvision
cifar = CIFAR10(DATA_PATH, train=True)
mnist = MNIST(DATA_PATH, download=False, train=True)
nb_task = 3
list_trsf = []
for i in range(nb_task):
list_trsf.append([torchvision.transforms.ToTensor(), BackgroundSwap(cifar, bg_label=i, input_dim=(28, 28)),
torchvision.transforms.ToPILImage()])
scenario = TransformationIncremental(mnist, base_transformations=[torchvision.transforms.ToTensor()],
incremental_transformations=list_trsf)
for task_id, task_data in enumerate(scenario):
# Do magic here
Permutations Incremental source
is a famous case of TransformationIncremental class, in this case the transformation is a fixed pixel permutation. Each task has a specific permutation. The scenarios is then to learn a same task in various permutation spaces.
from continuum import Permutations
from continuum.datasets import MNIST
dataset = MNIST(data_path="my/data/path", download=True, train=True)
nb_tasks = 5
seed = 0
# A sequence of permutations is initialized from seed `seed` each task is with different pixel permutation
# shared_label_space=True means that all classes use the same label space
# ex: an image of the zeros digit will be always be labelized as a 0 ( if shared_label_space=False, zeros digit image permutated will got another label than the original one)
scenario = Permutations(cl_dataset=dataset, nb_tasks=nb_tasks, seed=seed, shared_label_space=True)
Rotations Incremental source
is also a famous case of TransformationIncremental class, in this case the transformation is a rotation of image. Each task has a specific rotation or range of rotation. The scenarios is then to learn a same task in various rotations spaces.
from continuum import Rotations
from continuum.datasets import MNIST
nb_tasks = 3
# first example with 3 tasks with fixed rotations
list_degrees = [0, 45, 90]
# second example with 3 tasks with ranges of rotations
list_degrees = [0, (40,50), (85,95)]
dataset = MNIST(data_path="my/data/path", download=True, train=True)
scenario = Rotations(
cl_dataset=dataset,
nb_tasks=nb_tasks,
list_degrees=list_degrees
)
Note that for all TransformationIncremental scenarios (included Rotations and Permutations) you can use advanced indexing (e.g. scenario[2:6], or scenario[:7]). In that case, when sampling multiple tasks together, the same original images will be seen multiple times, but each time with the transformation associated to the task.
New Class and Instance Incremental¶
In short: Each new task bring both instances from new classes and new instances from known classes.
Aim: Evaluate the capability of an algorithms to both create new representation and improve existing ones.
Some Details: NIC setting is a special case of NI setting. For now, only the CORe50 dataset supports this setting.
The New class and instance incremental setting can be created with the Instance incremental class. The t vector which define the task of each data point should be defined by the user or loaded from an existing scenario to create NIC scenario.
# given x data point, y class labels, t tasks labels
# t define the tasks label for each data point.
# Hence, the t vector define the number of tasks for the scenario and the order
# task will be ordered in the croissant order
from continuum import InstanceIncremental
from continuum.datasets import InMemoryDataset
NIC_Dataset = InMemoryDataset(x, y, t)
NIC_Scenario = InstanceIncremental(NIC_Dataset)
Hashed Scenarios¶
In short: Data ordering is determined by an hash function.
Aim: Evaluate the capability of an algorithms to learn new features independently from the labels.
Some Details: This kind of scenario is proposed here to create tasks with related features in classical dataset and evaluate algorithm capabilities in such context. This methodology force features to be different from one task to another making the feature extractor training potentially harder. The resulting scenario (depending on the dataset) is most probably a NIC scenario.
The New class and instance incremental setting can be created with the HashedScenario class. The t vector is generate by the scenario class. It can be saved and reloaded to avoid to compute the hash every time.
# given x data point, y class labels, t tasks labels
# t define the tasks label for each data point.
# Hence, the t vector define the number of tasks for the scenario and the order
# task will be ordered in the croissant order
from continuum import HashedScenario
from continuum.datasets import CIFAR10
dataset = CIFAR10(data_path="my/data/path", train=True)
Hashed_Scenario = HashedScenario(dataset,
nb_tasks=2,
hash_name="AverageHash",
transformations=None,
filename_hash_indexes="hash_indexes_CIFAR10.npy",
split_task = "balanced")
In this example we use “AverageHash” from imagehash library but many other image hash can be used such as: “AverageHash”, “Phash”, “PhashSimple”, “DhashH”, “DhashV”, “Whash”, “ColorHash” and “CropResistantHash”. More information about those hash function in imagehash documentation.
The split_task parameter can either be “balanced” or “auto”, if the amount of data is balanced among tasks or if it is set automatically to create more hash-coherent tasks.
If split_task=”auto” and nb_tasks=None the number of tasks will be automatically estimated with the MeanShift function from scikit-learn.
Incremental Semantic Segmentation¶
Brought by Michieli et al. ICCV-W 2017 and Cermelli et al. CVPR 2020, continual learning for segmentation is very different from previous scenarios.
Semantic segmentation aims at classifying all pixels in an image, therefore multiple classes can co-exist in the same image. This distinction leads to three kinds of scenarios:
Sequential: where for a given task T, with current classes C, the model sees all images that contain at least one pixel labeled as a current classes C. If the image contains future classes, yet unseen, then it is discarded. In the sequential setting, all pixels are labeled, either with a old or current class label, background label (0), or unknown label (255).
Disjoint: It’s the same scenario as Sequential, but on one point. An image’s pixel is only labeled for the current classes. Therefore, during training, if an old class is present in the image, its labels would be 0 (aka background). However, during the test phase, all labels (current + old) are present.
Overlap: It’s the same scenario as Disjoint except that the model can also see images containing a future class, as long as a current class is present.
Here is a quick example on how to do the challenging Overlap 15-1 scenario on Pascal-VOC2012:
from continuum.datasets import PascalVOC2012
from continuum.scenarios import SegmentationClassIncremental
from continuum.transforms.segmentation import ToTensor, Resize
dataset = PascalVOC2012(
data_path="/my/data/path/",
train=True,
download=True
)
scenario = SegmentationClassIncremental(
dataset,
nb_classes=20,
initial_increment=15, increment=1,
mode="overlap",
transformations=[Resize((512, 512)), ToTensor()]
)
NB: Following Cermelli et al., 15-1 means first a task of 15 classes, then followed by multiple tasks made of new 1 class each.
Note that to build the different tasks, SegmentationClassIncremental has to open every ground-truth segmentation maps which can take a few minutes. Therefore, you can provide to the scenario the option save_indexes=”/path/where/to/save/indexes” that will save the computed task indexes. Then, if re-run a second time, the scenario can quickly load the indexes.
Adding Your Own Scenarios with the ContinualScenario Class¶
Continuum is developed to be flexible and easily adapted to new settings. Then you can create a new scenario by providing simply a new dataset framed in an existing scenario such as Classes Incremental, Instance Incremental … You can also create a new class to create your own scenario with your own rules !
You can add it in the scenarios folder in the continuum project and make a pull request!
Scenarios can be seen as a list of tasks , the main thing to define is to define the content of each task to create a meaningful scenario.
You can also create personal scenarios simply by creating your own task label vector t with the ContinualScenario Class. This class is made to just convert a cl_dataset into a scenario without any other processing.
from continuum.datasets import InMemoryDataset
from continuum.scenarios import ContinualScenario
x, y, t = fancy_data_generation_process()
# t should contains the task label for each data point in x.
# t should respect : np.unique(t).sort() == np.arange(len(np.unique(t)))
cl_dataset = InMemoryDataset(x, y, t)
scenario = ContinualScenario(cl_dataset)
Useful functions for scenarios¶
create_subscenario(base_scenario, task_indexes)
This function makes it possible to slice a scenario by selecting only a subset of tasks or to reorder classes with a new order of task indexes.
get_scenario_remapping(scenario)->
This function provided a remapping of class that ensure that labels comes in a continuous increasing order.
It is particularly useful if order of tasks has been changed (for example with create_subscenario).
This function is often use with the function remap_class_vector which will apply the remapping.
The mapping can also be build online with update_remapping(class_vector, mapping) (it is more rigorous but it is also more computationnaly costly and it does not change the final results).
encode_scenario(scenario, model, batch_size, file_name, inference_fct=None)
This function makes it possible to create a scenario with latent representation of a given model. For example, when you have a frozen pretrained model and you want to just train the last layers. With encode_scenario function, you can create a scenario with the data already encoded. This function will save all the latent vectors into a hdf5 files and create the exact same initial scenario with encoded vectors. It reduces the computation footprint and the time spent on encoding data for every experiences.
Taskset Class¶
In continual learning, tasks are empirically designed sets of data. Tasks have generally fixed data distribution or a fixed learning objective. When the learning scenario jumps from one task to another it means that something has changed in the learning process, i.e. the data or the objective.
In Continuum the task_set class is a set of data that can be loaded with pytorch loader and sampled to feed data to an algorithm. The scenarios are composed then by a sequence of task_set. Each task_set defines then a learning problem that the algorithm will have to solve one by one.
In Continuum, the user doesn’t have to create tasks with task_set class, the tasks are created with the scenarios classes. Thus we don’t recommend you to play directly with this API if you’re unsure about what you’re doing.
Nevertheless, even if it is not very useful, it is still possible to create a task.
# create one task with all MNIST
from torch.utils.data import DataLoader
from continuum.datasets import MNIST
from continuum import TaskSet
dataset = MNIST("my/data/path", download=True, train=True)
# get data x, label y and task index t
x, y, t = dataset.get_data()
transform = None
# create the task set
task = TaskSet(x, y, t, transform, data_type=dataset.data_type)
# the task set can be use to create a loader
task_loader = DataLoader(task)
for x, y, t in task_loader:
# sample batch of data to train your model
# t is the task index or task label, if it is not defined t is a batch of -1
break
Tasks are designed to conveniently split the training scenarios into clear learning problems. In real-life scenarios, the separation of the learning process might be impossible, either because there is no clear transition or because we don’t know when they happen, i.e. they are not labeled.
Useful methods¶
get_random_samples(nb_samples)provide random samples from the task, useful for visualization and control.get_raw_samples(indexes)provide samples from the task without applying any transformation, useful for visualization and control.get_classes()provide an array containing all the classes from the task_set object.nb_classes()provide the number of classes.add_samples(x: np.ndarray, y: np.ndarray, t: Union[None, np.ndarray] = None)makes possible to add manually data into the training set, might be useful for rehearsal strategies.
Useful functions¶
split_train_val(dataset, split_val = 0.1)split a task_set into two for validation purpose
Existing Datasets¶
Name |
Nb classes |
Image Size |
Auto Download |
Type |
|---|---|---|---|---|
MNIST |
10 |
28x28x1 |
YES |
Images |
Fashion MNIST |
10 |
28x28x1 |
YES |
Images |
KMNIST |
10 |
28x28x1 |
YES |
Images |
EMNIST |
10 |
28x28x1 |
YES |
Images |
QMNIST |
10 |
28x28x1 |
YES |
Images |
MNIST Fellowship |
30 |
28x28x1 |
YES |
Images |
Rainbow MNIST |
10 |
28x28x3 |
YES |
Images |
Colored MNIST |
2 |
28x28x3 |
YES |
Images |
SVHN |
10 |
32x32x3 |
YES |
Images |
Synbols |
50 |
32x32x3 |
YES |
Images |
CIFAR10 |
10 |
32x32x3 |
YES |
Images |
CIFAR100 |
100 |
32x32x3 |
YES |
Images |
CIFAR Fellowship |
110 |
32x32x3 |
YES |
Images |
STL10 |
10 |
96x96x3 |
YES |
Images |
Omniglot |
964 |
105x105x1 |
YES |
Images |
CTRL minus |
87 |
32x32x3 |
YES |
Images |
CTRL plus |
87 |
32x32x3 |
YES |
Images |
CTRL in |
87 |
32x32x3 |
YES |
Images |
CTRL out |
97 |
32x32x3 |
YES |
Images |
CTRL plastic |
97 |
32x32x3 |
YES |
Images |
FER2013 |
7 |
48x48x1 |
NO |
Images |
TinyImageNet200 |
200 |
64x64x3 |
YES |
Images |
GTSRB |
43 |
~64x64x3 |
YES |
Images |
EuroSAT |
10 |
64x64x3 |
YES |
Images |
ImageNet100 |
100 |
224x224x3 |
NO |
Images |
ImageNet1000 |
1000 |
224x224x3 |
NO |
Images |
CORe50 |
50 |
224x224x3 |
YES |
Images |
CORe50-v2-79 |
50 |
224x224x3 |
YES |
Images |
CORe50-v2-196* |
50 |
224x224x3 |
YES |
Images |
CORe50-v2-391 |
50 |
224x224x3 |
YES |
Images |
Stanford Car196 |
196 |
224x224x3 |
YES |
Images |
Stream-51 |
51 |
224x224x3 |
YES |
Images |
Caltech101 |
101 (102) |
224x224x3 |
YES |
Images |
Caltech256 |
257 |
224x224x3 |
YES |
Images |
FGVC Aircraft |
30/70/100 |
224x224x3 |
YES |
Images |
Food101 |
101 |
224x224x3 |
YES |
Images |
DTD |
47 |
224x224x3 |
YES |
Images |
AwA2 |
50 |
224x224x3 |
YES |
Images |
CUB200 |
200 |
224x224x3 |
YES |
Images |
Terra Incognita |
10 |
224x224x3 |
YES |
Images |
PACS |
7 |
224x224x3 |
NO |
Images |
VLCS |
5 |
224x224x3 |
YES |
Images |
OfficeHome |
65 |
224x224x3 |
NO |
Images |
DomainNet |
345 |
224x224x3 |
YES |
Images |
Pascal-VOC 2007 |
20 |
224x224x3 |
YES |
Images |
Pascal-VOC 2012 |
20 |
512x512x3 |
YES |
Segmentation Maps |
MultiNLI |
5 |
YES |
Text |
|
OxfordFlower102 |
102 |
?x?x3 |
YES |
Images |
OxfordPet |
37 |
?x?x3 |
YES |
Images |
SUN397 |
397 |
?x?x3 |
YES |
Images |
Birdsnap |
500 |
?x?x3 |
YES |
Images |
MetaShift |
410 |
?x?x3 |
YES |
Images |
Fluent Speech |
31 |
? |
YES |
Audio |
All datasets have for arguments train and download, like a torchvision.dataset. Those datasets are then modified to create continuum scenarios.
Once a dataset is created, it is fed to a scenario that will split it in multiple tasks.
Continuum supports many datasets implemented in torchvision in such as MNIST, or CIFAR100:
from continuum import ClassIncremental
from continuum.datasets import MNIST
clloader = ClassIncremental(
MNIST("/my/data/folder", download=True, train=True),
increment=1,
initial_increment=5
)
The data from these small datasets can be automatically downloaded with the option download.
Larger datasets such as ImageNet or CORe50 are also available, although their initialization differ:
from continuum import ClassIncremental
from continuum.datasets import ImageNet1000
dataset_100 = ImageNet1000("/my/data/folder/imagenet/train/", train=True)
dataset_1000 = ImageNet1000("/my/data/folder/imagenet/val/", train=False)
Note that Continuum cannot download ImageNet’s data, it’s on you! We also provide ImageNet100, a subset of 100 classes of ImageNet. The subset meta-data are automatically downloaded, or you can provide them with the option data_subset.
Multiple versions of CORe50 are proposed. For all, the data can automatically be downloaded:
from continuum.datasets import Core50, Core50v2_79, Core50v2_196, Core50v2_391
dataset = Core50("/my/data/folder/", train=True, download=True)
dataset_79 = Core50v2_79("/my/data/folder/", train=True, download=True)
dataset_196 = Core50v2_196("/my/data/folder/", train=True, download=True)
dataset_391 = Core50v2_391("/my/data/folder/", train=True, download=True)
If you wish to learn CORe50 in the class-incremental scenario (NC), Core50 suffices. Although, for instance-incremental scenario (NI and NIC), you need to use Core50v2_79, Core50v2_196, or Core50v2_391 (see our doc about it). Refer to the datatset official webpage for more information about the different versions.
In addition to Computer Vision dataset, Continuum also provide one NLP dataset:
from continuum.datasets import MultiNLI
dataset=MultiNLI("/my/data/folder", train=True, download=True)
The MultiNLI dataset provides text written in different styles and categories. This dataset can be used in Continual Learning in a New Instances (NI) setting where all categories are known from the start, but with styles being incrementally added.
Adding Your Own Datasets¶
The goal of continuum is to propose the most used benchmark scenarios of continual learning but also to make easy the creation of new scenarios through an adaptable framework.
For example, the type of scenarios are easy to use with others dataset:
InMemoryDataset, for in-memory numpy array:
from continuum.datasets import InMemoryDataset
x_train, y_train = gen_numpy_array()
dataset = InMemoryDataset(x_train, y_train)
PyTorchDataset,for datasets defined in torchvision:
from torchvision.datasets import CIFAR10
from continuum.datasets import PyTorchDataset
dataset = PyTorchDataset("/my/data/folder/", dataset_type=CIFAR10, train=True, download=True)
ImageFolderDataset, for datasets having a tree-like structure, with one folder per class:
from continuum.datasets import ImageFolderDataset
dataset_train = ImageFolderDataset("/my/data/folder/train/")
dataset_test = ImageFolderDataset("/my/data/folder/test/")
Fellowship, to combine several continual datasets.:
from torchvision.datasets import CIFAR10, CIFAR100
from continuum.datasets import Fellowship
dataset = Fellowship(datasets=[
CIFAR10(data_path="/my/data/folder1/", train=True),
CIFAR100(data_path="/my/data/folder1/", train=True)
],
update_labels=True
)
The update_labels parameter determines if we want that different datasets have different labels or if we do not care about it. The default value of update_labels is True. Note that Continuum already provide pre-made Fellowship:
from continuum.datasets import MNISTFellowship, CIFARFellowship
dataset_MNIST = MNISTFellowship("/my/data/folder", train=True)
dataset_CIFAR = CIFARFellowship("/my/data/folder", train=True)
You may want datasets that have a different transformation for each new task, e.g. MNIST with different rotations or pixel permutations. Continuum also handles it! However it’s a scenario’s speficic, not dataset, thus look over the Scenario doc.
Supervised setting without Continual¶
Continuum is awesome but you don’t want to do continual learning? Simply want to train a model on a single try on the whole dataset? No problem.
All Continuum datasets can be directly converted to tasksets, which implement the Pytorch Dataset and thus can be directly given to a DataLoader.
Here are an example with MNIST, but all datasets work the same:
from torch.utils.data import DataLoader
from continuum.datasets import MNIST
dataset = MNIST("/my/data/folder", train=True, download=True)
taskset = dataset.to_taskset(
trsf=None # Put your transformations here if you want some
target_trsf=None # Put your target transformations here if you want some
)
loader = DataLoader(taskset, batch_size=32, shuffle=True)
for x, y, t in loader:
pass # Your model here
Multi-target datasets¶
Continuum also allows a dataset to return multiple targets per data point. In class incremental, only the first target is taken in account for designing the increments.
from continuum import ClassIncremental
from continuum.datasets import FluentSpeech
from torch.utils.data import DataLoader
dataset = FluentSpeech("/my/data/folder", train=True)
scenario = ClassIncremental(dataset, increment=1)
for taskset in scenario:
loader = DataLoader(taskset, batch_size=1)
for x, y, t in loader:
print(x.shape, y.shape, t.shape, np.unique(y[:, 0]))
break
In this situation, the FluentSpeech dataset has 4 targets, but only the first one is used to determine the tasks.
The Different Data Types¶
Each dataset has a data type. Here are the ones already implemented:
IMAGE_ARRAY: Image is directly as an array. Example: MNIST dataset.
IMAGE_PATH: Image is stored as a patch to the actual image. Example: ImageNet dataset.
TEXT: For HuggingFace datasets.
TENSOR: Any kind of tensor.
SEGMENTATION: For the Continual Semantic Segmentation datasets.
OBJ_DETECTION: Still WIP.
H5: Storing tensor in a H5 dataset. For the end user, the usage is quite similar to TENSOR.
AUDIO: Storing a string to a audio file. Needs to install the Soundfile library.
Rehearsal¶
A popular and efficient method in continual learning is to do rehearsal. Aka the action of reviewing a limited amount of previous images.
Continuum provides a helper class to manage this memory and functions to samples (“to herd”) examples into the memory.
How to use the memory:
from torch.utils.data import DataLoader
from continuum import ClassIncremental
from continuum.datasets import CIFAR100
from continuum import rehearsal
scenario = ClassIncremental(
CIFAR100(data_path="my/data/path", download=True, train=True),
increment=10,
initial_increment=50
)
memory = rehearsal.RehearsalMemory(
memory_size=2000,
herding_method="barycenter"
)
for task_id, taskset in enumerate(scenario):
if task_id > 0:
mem_x, mem_y, mem_t = memory.get()
taskset.add_samples(mem_x, mem_y, mem_t)
loader = DataLoader(taskset, shuffle=True)
for epoch in range(epochs):
for x, y, t in loader:
# Do your training here
# Herding based on the barycenter (as iCaRL did) needs features,
# so we need to extract those features, but beware to use a loader
# without shuffling.
loader = DataLoader(taskset, shuffle=False)
features = my_function_to_extract_features(my_model, loader)
# Important! Draw the raw samples from `scenario[task_id]` to
# re-generate the taskset, otherwise you'd risk sampling from both new
# data and memory data which is probably not what you want to do.
memory.add(
*scenario[task_id].get_raw_samples(), features
)
Herding¶
We predefine three methods to herd new samples: - random: random (quite efficient despite its simplicity…) - cluster: samples whose features are closest to their class mean feature - barycenter: samples whose features are closest to a moving barycenter as iCaRL did
Note that all these predefined herding methods are done per class.
If you want to define your own herding method and provide it to RehearsalMemory (instead of the string ‘barycenter’ as in the previous example), you should:
Take three parameters in arguments x, y, t, z, nb_per_classes.
x, y, t are the input data, targets and task ids. z is an extra info, that can be whatever you want. For ‘barycenter’ and ‘cluster’ it’s features.
Returns the sampled x, y, t. Note that you could even create new x, y, t and returns those if you want.
Saving and loading¶
Computing rehearsal samples can be slow, and thus if you want to re-start from a checkpoint, you would also want to avoid re-computing the herding. Thus, the memory provide a save and load methods:
memory.save("/my/path/memory.npz")
memory.load("/my/path/memory.npz")
Fixed memory or flexible memory¶
There are currently to way to handle the memory growth, either fixed or flexible.
flexible: Usually in settings where all class increments are of the same size, including the first one, like in iCaRL, all the memory is used: if I have a memory size of 10, and I saw 2 classes over 5, each class can store 5 images. Then, when I see all 5 classes, each class can only store 2 images.
fixed: On the other hand, in settings where the initial class increment is bigger, like in UCIR, each class can store an amout of images that stay fixed for the whole continual training. e.g. if I have a memory size of 10, and I saw 2 classes over 5, each class can store 2 images. Likewise, when I see all 5 classes, each class can still only store 2 images.
The option is fixed_memory=True/False in the initialization of the RehearsalMemory:
from continuum import rehearsal
memory = rehearsal.RehearsalMemory(
memory_size=2000,
herding_method="barycenter",
fixed_memory=True
)
Metrics¶
Continual Learning has many different metrics due to the nature of the task. Continuum proposes a logger module that accumulates the predictions of the model. Then the logger can compute various types of continual learning metrics based on the prediction saved.
Disclaimer: We aim to propose some tools for metric evaluation. Nevertheless, they are many different ways of evaluating CL and different metrics need different information: for example, the forward transfer might need the test accuracy of future tasks or accuracy of the same model trained from scratch. In our metric tools, we assume that we can measure test accuracy on future tasks and that task transitions are clear and available. We do not provide tools for measuring compute efficiency.
Pseudo-code
logger = Logger()
for task in scenario:
for (x,y,t) in tasks:
predictions = model(x,y,t)
logger.add([predictions, y, t])
logger.end_task()
print(f"Metric result: {logger.my_pretty_metric}")
Here is a list of all implemented metrics:
Name |
Code |
↑ / ↓ |
|---|---|---|
Accuracy |
accuracy |
↑ |
Accuracy A |
accuracy_A |
↑ |
Backward Transfer |
backward_transfer |
↑ |
Positive Backward Transfer |
positive_backward_transfer |
↑ |
Remembering |
remembering |
↑ |
Forward Transfer |
forward_transfer |
↑ |
Forgetting |
forgetting |
↓ |
Model Size Growth |
model_size_growth |
↓ |
Accuracy:
Computes the accuracy of a given task.
Accuracy A:
Accuracy as defined in Diaz-Rodriguez and Lomonaco.
Note that it is slightly different from the normal accuracy as it considers
each task accuracy with equal weight, while the normal accuracy considers
the proportion of all targets.
Example:
- Given task 1 with 50,000 images and task 2 with 1,000 images.
- With normal accuracy, task 1 has more importance in the average accuracy.
- With this accuracy A, task 1 has as much importance as task 2.
Reference:
* Don’t forget, there is more than forgetting: newmetrics for Continual Learning
Diaz-Rodriguez and Lomonaco et al. NeurIPS Workshop 2018
Backward Transfer:
Measures the influence that learning a task has on the performance on previous tasks.
Reference:
* Gradient Episodic Memory for Continual Learning
Lopez-paz & ranzato, NeurIPS 2017
Note: To measure backward transfer, the logger has to contains accuracy to past tasks at task t.
Positive Backward Transfer:
Computes the the positive gain of Backward transfer.
Reference:
* Don’t forget, there is more than forgetting: newmetrics for Continual Learning
Diaz-Rodriguez and Lomonaco et al. NeurIPS Workshop 2018
Remembering:
Computes the forgetting part of Backward transfer.
Reference:
* Don’t forget, there is more than forgetting: newmetrics for Continual Learning
Diaz-Rodriguez and Lomonaco et al. NeurIPS Workshop 2018
Forward Transfer:
Measures the influence that learning a task has on the performance of future tasks.
Reference:
* Gradient Episodic Memory for Continual Learning
Lopez-paz & ranzato, NeurIPS 2017
Note: To measure Forward transfer, the logger has to contains accuracy to the future tasks at task t.
Forgetting:
Measures the average forgetting.
Reference:
* Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Chaudhry et al. ECCV 2018
Model Size Growth:
Evaluate the evolution of the model size.
Detailed Example¶
from torch.utils.data import DataLoader
from continuum import ClassIncremental
from continuum.datasets import MNIST
from continuum.metrics import Logger
train_scenario = ClassIncremental(
MNIST(data_path=DATA_PATH, download=True, train=True),
increment=2
)
test_scenario = ClassIncremental(
MNIST(data_path=DATA_PATH, download=True, train=False),
increment=2
)
# model = ...
test_loader = DataLoader(test_scenario[:])
logger = Logger(list_subsets=['train', 'test'])
for task_id, train_taskset in enumerate(train_scenario):
train_loader = DataLoader(train_taskset)
for x, y, t in train_loader:
predictions = y # model(x)
logger.add([predictions, y, None], subset="train")
_ = (f"Online accuracy: {logger.online_accuracy}")
for x_test, y_test, t_test in test_loader:
preds_test = y_test
logger.add([preds_test, y_test, t_test], subset="test")
_ = (f"Task: {task_id}, acc: {logger.accuracy}, avg acc: {logger.average_incremental_accuracy}")
if task_id > 0:
_ = (f"BWT: {logger.backward_transfer}, FWT: {logger.forward_transfer}")
logger.end_task()
Advanced Use of logger¶
The logger is designed to save any type of tensor with a corresponding keyword. For example you may want to save a latent vector at each epoch.
from continuum.metrics import Logger
model = ... Initialize your model here ...
list_keywords=["latent_vector"]
logger = Logger(list_keywords=list_keywords, list_subsets=['train', 'test'])
for tasks in task_scenario):
for epoch in range(epochs)
for x, y, t in task_loader:
# Do here your model training with losses and optimizer...
latent_vector = model.get_latent_vector_fancy_method_you_designed()
logger.add(latent_vector, keyword='latent_vector', subset="train")
logger.end_epoch()
logger.end_task()
If you want to log result to compute metrics AND log you latent vector you can declare and use you logger as following:
# Logger declaration with several keyword
logger = Logger(list_keywords=["performance", "latent_vector"], list_subsets=['train', 'test'])
# [...]
# log test results for metrics
logger.add([x,y,t], keyword='performance', subset="test")
# [...]
# log latent vector while testing
logger.add(latent_vector, keyword='latent_vector', subset="test")
At the end of training or when you want, you can get all the data logged.
logger = Logger(list_keywords=["performance", "latent_vector"], list_subsets=['train', 'test'])
# [... a long training a logging adventure ... ]
logs_latent = logger.get_logs(keyword='latent_vector', subset='test')
# you can explore the logs as follow
for task_id in range(logs_latent):
for epoch_id in range(logs_latent[task_id]):
# the list of all latent vector you saved as task_id and epoch_id by chronological order.
list_of_latent_vector_logged = logs_latent[task_id][epoch_id]
We hope it might be useful for you :)
Introduction¶
In the continual learning literature, several evaluation benchmarks suite have been proposed. Those suites propose a variety of tasks to evaluate algorithms. Continuum aims to reproduce some of their main settings for easy reproducibility.
CORe50
Stream-51
Synbols
CTRL
ALMA
MNIST-360
DomainBed
FluentSpeech
HuggingFace’s NLP datasets
And soon many others
ALMA¶
ALMA (On Anytime Learning At Macroscale) is a new framework, where like (offline) Continual Learning, data arrive sequentially in large (mega)batches over time. See paper https://arxiv.org/abs/2106.09563. Unlike CL however, we do not assume that there is a shift in the underlying distribution. Rather, the goal of ALMA is develop strategies that perform well troughout the learning experience (not just at the end), and that do so efficiently from a compute and memory perspective. ALMA explore a different line of questions arising in this setting, namely :
How long should a model wait and aggregate data before training again ?
Should the model increase its capacity over time to account for the additional data ?
Continuum Scenarios¶
ALMA is a different framing of InstanceIncremental therefore we focus on this scenario.
Instance Incremental
Class and Instances Incremental scenarios are proposed in the scenario from the original paper (next section).
from continuum.scenarios import ALMA
scenario = ALMA(your_dataset, nb_megabatches=50)
CIFAR100¶
CIFAR100 is a famous dataset proposed in “Learning Multiple Layers of Features from Tiny Images (pdf)”. This dataset is mainly used with its 100 class labels type. However, it exists also 20 super classes (coarse labels or category labels). In continuum, we propose to benefit from both to create various types of scenarios.
Continuum Scenarios¶
Class Incremental
We can create a simple class incremental setting with the default parameters, i.e. 100 classes. In this scenario coarse labels are not used.
from continuum.datasets import CIFAR100
from continuum.scenarios import ClassIncremental
dataset = CIFAR100("/your/path", train=True)
# 5 tasks with 20 classes each
scenario = ClassIncremental(dataset, nb_tasks=5)
Or a ClassIncremental with coarse labels (category labels). In this scenario classical labels are not used.
from continuum.datasets import CIFAR100
from continuum.scenarios import ClassIncremental
dataset = CIFAR100("/your/path", train=True, labels_type="category")
# 5 tasks with 4 classes each
scenario = ClassIncremental(dataset, nb_tasks=5)
Instance Incremental (lifelong)
We can create a instance incremental setting with the coarse labels, i.e. 20 classes. Data are labeled with the coarse labels of CIFAR100. However, data are shared between tasks using the original label to ensure a domain drift between tasks , e.g., for the coarse label say{aquatic mammals} the data go from beavers to dolphins to otters to seals to finally whales in separate tasks.
from continuum.datasets import CIFAR100
from continuum.scenarios import ContinualScenario
dataset = CIFAR100("/your/path",
train=True,
labels_type="category",
task_labels="lifelong")
# 5 tasks with 20 labels (coarse labels) each (always the same labels but different classes)
scenario = ContinualScenario(dataset)
Classes and Instances Incremental
In Class and Instances Incremental scenario, the labels are set by category of object but new tasks bring new object. Hence, new task either bring a new object from a known category or a new object from an unknown category.
from continuum.datasets import CIFAR100
from continuum.scenarios import ContinualScenario
# task_labels parameter makes possible to create a task id vector from either classes or categories.
dataset = CIFAR100("/your/path", train=True), task_labels="class", labels_type="category"
# 100 tasks with 1 object each among the 20 categories of coarse labels
# classes are object ids (20 classes then), new tasks might contains new label or known label
scenario = ContinualScenario(dataset)
CORe50¶
Core50 is a dataset proposed in “CORe50: a new Dataset and Benchmark for Continuous Object Recognition”. This dataset proposed small videos of of 50 objects from 10 differents classes with 11 background environments (more info in core50 doc ). This dataset was originally created to propose various continual learning settings. 3 background environments are allocated for test and the remaining for training.
Continuum Scenarios¶
You can create automatically scenarios with continuum by setting the scenario and classification parameter in Core50 dataset. It will provide different types of annotation for targets and tasks. For classification, you can choose to use category (10 classes) annotation or object annotation (50 classes)> For the task ids, you can choose among “classes”, “domains” and “objects” how the task labels will be affected to data.
Class Incremental
We can create a simple class incremental setting.
from continuum.datasets import COre50
# Same as :
# dataset=Core50("/your/path", scenario="classes", classification="object", train=True)
dataset = Core50("/your/path", train=True)
# 5 tasks with 10 classes each
scenario = ClassIncremental(dataset, nb_tasks=5)
Instance Incremental
from continuum.datasets import COre50
dataset = Core50("/your/path", scenario="domains", classification="category", train=True)
# 8 tasks in 1 environment each with 10 classes
scenario = ContinualScenario(dataset, nb_tasks=5)
– Object incremental Scenario – .. code-block:: python
from continuum.datasets import COre50 dataset = Core50(“/your/path”, scenario=”objects”, classification=”object”, train=True) # 50 tasks with 1 object videos in the 8 training environments # classes are object ids (50 classes then) scenario = ContinualScenario(dataset)
Classes and Instances Incremental
Class and Instances Incremental scenarios are proposed in the scenario from the original paper (next section).
from continuum.datasets import COre50
dataset = Core50("/your/path", scenario="objects", classification="category", train=True)
# 50 tasks with 1 object videos in the 8 training environments
# classes are object ids (10 classes then), new tasks might contains new label or known label
scenario = ContinualScenario(dataset)
Original scenarios:¶
CORe50 provides domain ids which are automatically picked up by the InstanceIncremental scenario:
from continuum import InstanceIncremental
from continuum.datasets import Core50v2_79, Core50v2_196, Core50v2_391
scenario_79 = InstanceIncremental(dataset=Core50v2_79("/my/path"))
scenario_196 = InstanceIncremental(dataset=Core50v2_196("/my/path"))
scenario_391 = InstanceIncremental(dataset=Core50v2_391("/my/path"))
The three available version of CORe50 have respectively 79, 196, and 391 tasks. Each task may bring new classes AND new instances of past classes, akin to the NIC scenario.
CTRL benchmark¶
CTRL is a collection of datasets proposed in “Efficient Continual Learning with Modular Networks and Task-Driven Priors” published at ICLR 2021 and co-authored by Veniat (Sorbonne), Denoyer and Ranzato (Facebook Research).
Each of the proposed dataset is combination of multiple small datasets among:
MNIST
SVHN
CIFAR10
Rainbow MNIST
Fashion MNIST
DTD
CTRL proposes different combinations which they name:
CTRL minus
CTRL plus
CTRL in
CTRL out
CTRL plastic
The only way to use those datasets is with the ContinualScenario scenario, which is the most flexible scenario Continuum offers.
You can a better feeling of this set of datasets in this Colab where we display the images of each datasets and their statistics. Pay attention to the class labels!
Usage¶
We can create a simple class incremental setting.
from continuum.datasets import CTRLminus, CTRLplus, CTRLplastic, CTRLin, CTRLout
from continuum import ContinualScenario
scenario_train = ContinualScenario(CTRLminus(path, split="train", download=True))
scenario_val = ContinualScenario(CTRLminus(path, split="val", download=True))
scenario_test = ContinualScenario(CTRLminus(path, split="test", download=True))
for task_id, (train_set, val_set, test_set) in enumerate(zip(scenario_train, scenario_val, scenario_test)):
train_loader = DataLoader(train_set)
val_loader = DataLoader(val_set)
test_loader = DataLoader(test_set)
for x, y, t in train_loader:
# Your model here
Note that contrarly to other datasets, all CTRL datasets have a split option which allow the use of a particularly made validation set.
Custom CTRL¶
Custom CTRL-like scenario (like the CTRL long that Veniat et al. describe) can be generated programatically according to your rules. Beware that most are probably very hard to solve but give it a try :)
from continuum.datasets import CTRL
from continuum.datasets import MNIST, CIFAR10, FashionMNIST, SVHN
class CTRLCustom(CTRL):
def __init__(self, data_path: str = "", split: str = "train", download: bool = True, seed: int = 1):
if split not in ("train", "val", "test"):
raise ValueError(f"Split must be train, val, or test; not {split}.")
train = split in ("train", "val")
datasets = [
CIFAR10(data_path=data_path, train=train, download=download),
MNIST(data_path=data_path, train=train, download=download),
FashionMNIST(data_path=data_path, train=train, download=download),
SVHN(data_path=data_path, train=train, download=download),
CIFAR10(data_path=data_path, train=train, download=download)
CIFAR10(data_path=data_path, train=train, download=download)
]
if split == "train":
proportions = [4000, 400, 400, 400, 400, 400]
elif split == "val":
proportions = [2000, 200, 200, 200, 200, 200]
else:
proportions = None
super().__init__(
datasets=datasets,
proportions=proportions,
class_counter=[0, 10, 20, 30, 0, 40],
class_subsets=[None, None, None, [0, 1, 2], None, None]
seed=seed,
split=split,
target_size=(32, 32)
)
What are the available customizations?
datasets: you can choose any dataset you want, as a long as it’s a Continuum dataset. Beware that they will be loaded in memory (so avoid ImageNet datasets), and all resized to the target_size.
proportions: it restricts the amount of data for train/val/test that will be used. Each class is sampled equally, therefore on CIFAR10, if I’m asking for 400 images, each class will have 40 images. If you don’t want this option for a particular split, set it to None, as we do in the previous example. In this case, all split data will be used.
class_counter: it controls what would be the i-th dataset labels. For example, if the class_counter is 30 for MNIST, then the dataset labels will be between 30 and 39. Thanks to this option, we can choose whether different datasets share the same labels. If you want MNIST and SVHN to share the same labels, they must have the same class_counter. In our code example, the 1-st and 5-th instances of CIFAR10 share the same labels, while the 6-th instance has different labels (although the actual classes are the same for a human).
class_subsets: this options simply allows to select a subset of the classes. In the code example, we use all datasets classes except for SVHN where only the 0, 1, and 2 classes are used.
Now, if we wanted to generated complex random streams such as the CTRLlong of Veniat et al., we can combine those to generate a random stream like this:
import numpy as np
class CTRLCustom(CTRL):
def __init__(self, data_path: str = "", split: str = "train", download: bool = True, seed: int = 1):
if split not in ("train", "val", "test"):
raise ValueError(f"Split must be train, val, or test; not {split}.")
train = split in ("train", "val")
rng = np.random.RandomState(seed=seed)
base_datasets = [
MNIST(data_path=data_path, train=train, download=download),
SVHN(data_path=data_path, train=train, download=download),
FashionMNIST(data_path=data_path, train=train, download=download),
CIFAR10(data_path=data_path, train=train, download=download)
]
svhn_mnist_share_labels = True
if svhn_mnist_share_labels:
task_counter = [0, 0, 10, 10]
else:
task_counter = [0, 10, 20, 30]
proportions_per_class = [1000, 1000, 1000, 500]
dataset_sample_prob = [0.2, 0.2, 0.3, 0.3]
nb_classes = 5
nb_tasks = 30
datasets, class_counter, class_subsets, proportions = [], [], [], []
for _ in range(nb_tasks):
dataset_id = rng.choice([0, 1, 2, 3], p=dataset_sample_prob)
datasets.append(base_datasets[dataset_id])
class_counter.append(task_counter[dataset_id])
class_subsets.append(rng.choice(10, size=nb_classes, replace=False))
if split == "train":
proportions.append(proportions_per_class[dataset_id])
elif split == "val":
proportions.append(proportions_per_class[dataset_id] // 2)
else:
proportions.append(None)
super().__init__(
datasets=datasets,
proportions=proportions,
class_counter=class_counter,
class_subsets=class_subsets,
seed=seed,
split=split,
target_size=(32, 32)
)
But you can do your own stream by choosing the rules.
DomainBed¶
DomainBed is a collection of datasets for domain generalization proposed in “In Search of Lost Domain Generalization”.
Each dataset is made of several “domains” where the pixels distribution change.
In the original paper, the authors tried different permutations where some domains where uniquely in the train or test sets. You can reproduce exactly this setting in Continuum. Although in the following examples, we instead share the same domains for train&test, where each domain is a new task that we cannot review once learned.
We also assume that all classes are shared across domains. A new task doesn’t bring new classes as it does in “ClassIncremental” but rather bring new domains as in “InstanceIncremental” and ContinualScenario.
ColoredMNIST (CMNIST)¶
The dataset is made of two labels: 0 & 1. All the original digits from 0 to 4 are now 0, and the leftover are now 1.
25% of the labels have been randomly flipped.
Each label (0 & 1) has an assigned color (red or green). But flip_color% of the samples have their color flipped. If the model learns the spurrious correlation of the color, then it’ll get super bad.
from continuum.datasets import ColoredMNIST, Fellowship
from continuum import ContinualScenario
dataset = Fellowship([
ColoredMNIST("/your/path", train=True, download=True, flip_color=0.1),
ColoredMNIST("/your/path", train=True, download=True, flip_color=0.2),
ColoredMNIST("/your/path", train=True, download=True, flip_color=0.9),
])
scenario = ContinualScenario(dataset)
RotatedMNIST (RMNIST)¶
MNIST with the same good old 10 digits. Although for each domains, the digits are rotated by a certain amount.
from continuum.datasets import MNIST
from continuum import Rotations
dataset = MNIST("/your/path", train=True, download=True)
scenario = Rotations(dataset, [15, 30, 45, 60])
VLCS¶
A dataset of large images, with 5 classes (bird, car, chair, dog, and person) distributed equally across 4 domains (Caltech101, LabelMe, SUN09, and VOC2007).
from torchvision import transforms
from continuum.datasets import VLCS
from continuum import ContinualScenario
dataset = VLCS("/your/path", train=True, download=True)
scenario = ContinualScenario(
dataset,
transformations=[transforms.Resize((224, 224)), transforms.ToTensor()]
)
PACS¶
A dataset of large images, with 7 classes distributed equally across 4 domains. Note that you need to download yourself this dataset “here”.
from torchvision import transforms
from continuum.datasets import PACS
from continuum import ContinualScenario
dataset = PACS("/your/path", train=True, download=False)
scenario = ContinualScenario(
dataset,
transformations=[transforms.Resize((224, 224)), transforms.ToTensor()]
)
OfficeHome¶
A dataset of large images, with 65 classes distributed equally across 4 domains. Note that you need to download yourself this dataset “here”.
from torchvision import transforms
from continuum.datasets import OfficeHome
from continuum import ContinualScenario
dataset = OfficeHome("/your/path", train=True, download=False)
scenario = ContinualScenario(
dataset,
transformations=[transforms.Resize((224, 224)), transforms.ToTensor()]
)
TerraIncognita¶
A dataset of large images, with 10 classes distributed equally across 4 domains.
from torchvision import transforms
from continuum.datasets import TerraIncognita
from continuum import ContinualScenario
dataset = TerraIncognita("/your/path", train=True, download=False)
scenario = ContinualScenario(
dataset,
transformations=[transforms.Resize((224, 224)), transforms.ToTensor()]
)
DomainNet¶
A dataset of large images, with 345 classes distributed equally across 6 domains.
from torchvision import transforms
from continuum.datasets import DomainNet
from continuum import ContinualScenario
dataset = DomainNet("/your/path", train=True, download=False)
scenario = ContinualScenario(
dataset,
transformations=[transforms.Resize((224, 224)), transforms.ToTensor()]
)
FluentSpeech¶
Audio datasets with multiple targets (class id, action, object, location) that represents short commands.
You need to install the Soundfile library.
You also probably want to add a custom transform to uniformize the length of each audio in order to batch it.
from continuum import ClassIncremental, ContinualScenario
from continuum.datasets import FluentSpeech
from torch.utils.data import DataLoader
dataset = FluentSpeech("/my/data/folder", train=True)
def trunc(x, max_len): # transformationn
l = len(x)
if l > max_len:
x = x[l//2-max_len//2:l//2+max_len//2]
if l < max_len:
x = F.pad(x, (0, max_len-l), value=0.)
return x
# Iterates through the 31 possible classes
scenario = ClassIncremental(
dataset, increment=1, transformations=[partial(trunc, max_len=32000)])
for taskset in scenario:
loader = DataLoader(taskset, batch_size=32)
for x, y, t in loader:
print(x.shape, y.shape, t.shape, np.unique(y[:, 0]))
break
# Iterates through the 77 existing speakers
scenario = ContinualScenario(dataset, transformations=[partial(trunc, max_len=32000)])
for taskset in scenario:
loader = DataLoader(taskset, batch_size=32)
for x, y, t in loader:
print(x.shape, y.shape, t.shape)
break
HuggingFace’s NLP datasets¶
HuggingFace proposes the largest amount of NLP datasets. The full list can be found here <https://huggingface.co/docs/datasets/>__.
Naturally, Continuum can work with those NLP datasets and convert them in continual scenarios.
A Continuum scenario can be iterated to produce task’s dataset. For image-based datasets, those tasks are implemented as Taskset that can be loaded with pytorch with a DataLoader. For text-based datasets, the task’s dataset is instead directly a HuggingFace’s dataset.
HuggingFace Continual¶
HuggingFaceIncremental can allow you to split a HuggingFace dataset according a to field name. For example, in the next code block, I download the MultiNLI dataset with HuggingFace. Then, I’m asking to split this dataset according to the genre field, and I’m also asking to see only 1 genre per task (increment=1):
import datasets # from HuggingFace, do pip install datasets
from continuum.scenarios.hf import HuggingFaceContinual
multi_nli = datasets.load_dataset("multi_nli", split="train")
scenario = HuggingFaceContinual(multi_nli, split_field="genre", increment=1)
print(len(scenario), scenario.nb_classes)
for task_dataset in scenario:
print(task_dataset)
Note that all `task_dataset`s are also HuggingFace’s datasets. So any functions you used to apply one those (filtering, tokenization, etc.) you can still do it.
See below an example of what the “fields” could be in a HuggingFace dataset:
>>> multi_nli
Dataset({
features: ['promptID', 'pairID', 'premise', 'premise_binary_parse', 'premise_parse', 'hypothesis', 'hypothesis_binary_parse', 'hypothesis_parse', 'genre', 'label'],
num_rows: 392702
})
HuggingFace Fellowship¶
HuggingFaceFellowship allows to chain multiple HuggingFace datasets one after the other. Look at the following example where three datasets have been chained:
from continuum.scenarios.hf import HuggingFaceFellowship
scenario = HuggingFaceFellowship(
["squad", "iwslt2017", "cnn_dailymail"],
lazy=True,
train=True
)
for dataset in scenario:
print(dataset)
Each task will be made of only one dataset. With HuggingFaceFellowship, you can specify a list of HuggingFace datasets, or a list of HuggingFace datasets names. The latter is only string, and those names can be found on the HuggingFace documentation page.
You can also ask to load the dataset only on the fly with lazy=True instead of loading all at the initialization.
Note that we proposes two pre-made fellowships that have been used previously in the litterature: AutumnQA and DecaNLP.
MNIST-360¶
MNIST-360 is a scenario proposed in “Dark Experience for General Continual Learning: a Strong, Simple Baseline”
It is composed of 27 tasks with two MNIST classes each. The two classes data have different rotation and the rotation change from one task to another.
Continuum Scenarios¶
We do not propose a click and play scenario for MNIST, however, we show how to implement it with continuum.
import os
import torch
from continuum.scenarios import ClassIncremental
from torchvision import transforms
folder = "tests/samples/mnist360/"
if not os.path.exists(folder):
os.makedirs(folder)
continual_dataset = MNIST(data_path=DATA_PATH, download=False, train=True)
for i in range(3):
start_angle = 120 * i
angle = int(120 / 9)
list_rotation_first = [[transforms.RandomAffine(degrees=[start_angle + angle * j, start_angle + angle * j + 5])]
for j in range(10)]
list_rotation_second = [
[transforms.RandomAffine(degrees=[start_angle + 45 + angle * j, start_angle + 45 + angle * j + 5])] for j in
range(10)]
first_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_first)
second_digit_scenario = ClassIncremental(continual_dataset, increment=1, transformations=list_rotation_second)
for task_id in range(9):
taskset_1 = first_digit_scenario[task_id]
taskset_2 = second_digit_scenario[1 + task_id % 9]
# / ! \ we can not concatenate taskset here, since transformations would not work correctly
loader_1 = DataLoader(taskset_1, batch_size=64)
loader_2 = DataLoader(taskset_2, batch_size=64)
nb_minibatches = min(len(loader_1), len(loader_2))
for minibatch in range(nb_minibatches):
x_1, y_1, t_1 = next(iter(loader_1))
x_2, y_2, t_2 = next(iter(loader_2))
x, y, t = torch.cat([x_1, x_2]), torch.cat([y_1, y_2]), torch.cat([t_1, t_2])
# train here on x, y, t
#### to visualize result ####
# from continuum.viz import visualize_batch
# visualize_batch(batch=x[:100], number=100, shape=[28, 28, 1], path=folder + f"MNIST360_{task_id + 9 * i}.jpg")
MetaShift¶
MetaShift can be described as a dataset of dataset. It divides the original Visual Genome dataset into classes and contexts in order to capture distribution shifts. See paper : https://arxiv.org/abs/2202.06523
In continuum, we use the context information to build various types of scenarios, notably domain incremental scenario.
The user can select a subset of classes to include in the scenario with class_names argument and if all class should appear in all tasks with strict_domain_inc argument.
Images can be associated with several context, then the user can decide if one image appear only once or can appear several time. If the image should appear only once, then the context chosen will be selected randomly.
Visual Genome:¶
To download the Visual Genome dataset in advance :
wget -c https://nlp.stanford.edu/data/gqa/images.zip
unzip images.zip -d MetaShift
Use the visual_genome_path argument in MetaShift constructor if you already have the original Visual Genome dataset downloaded.
Provide it with the path to the folder containing all Visual Genome images.
from continuum.datasets import MetaShift
data = MetaShift(datafolder, visual_genome_path = "some/path/images")
ContinualScenario:¶
Beware that the default configuration will download the full Visual Genome dataset if it is not already present in a folder named “MetaShift” (20GB).
Tasks are defined by the context in which the objects appear. In the following example, dogs and cats with water on the right and with computers on the left.
Default scenario settings:
from continuum.datasets import MetaShift
from continuum.scenarios import ContinualScenario
data = MetaShift(datafolder)
scenario = ContinualScenario(data)
Specific classes:
Select specific classes to appear in the dataset with the argument class_names.
Then specify if all classes should apprear in all tasks with the argument strict_domain_inc. If True, only contexts found in all specified classes will be kept.
from continuum.datasets import MetaShift
from continuum.scenarios import ContinualScenario
from continuum.datasets.metashift import get_all_classes_contexts
all_classes, all_contexts = get_all_classes_contexts(datafolder)
# Use this function to retrieve all metashift classes and contexts.
data = MetaShift(datafolder, class_names = ["cat", "dog"], strict_domain_inc = True)
scenario = ContinualScenario(data) # 81 tasks will be generated.
Specific contexts:
Select specific contexts with the argument context_names.
from continuum.datasets import MetaShift
from continuum.scenarios import ContinualScenario
from continuum.datasets.metashift import get_all_contexts_from_classes
contexts = get_all_contexts_from_classes(datafolder, ["cat", "dog", "horse"])
# Use this function to retreive all metashift contexts for given classes.
data = MetaShift(datafolder, context_names=["water", "ocean"])
scenario = ContinualScenario(data)
Get a unique task for each image:
Tasks can appear in multiple combinations of classes and contexts.
By setting unique_occurence=True, the task in which each image appears will be chosen randomly.
from continuum.datasets import MetaShift
from continuum.scenarios import ContinualScenario
data = MetaShift(datafolder, unique_occurence=True, random_seed=42)
scenario = ContinualScenario(data)
Stream 51¶
Two instance incremental scenarios: based on clip or video
The clip dataset with InstanceIncremental scenario is equivalent to the “instance” scenario of the original Stream-51 paper.
from torchvision.transforms import Resize, ToTensor
from continuum.datasets import Stream51
from continuum.scenarios import InstanceIncremental
dataset = Stream51('../Datasets', task_criterion="clip")
scenario = InstanceIncremental(dataset, transformations=[Resize((224, 224)), ToTensor()])
for task_id, task_set in enumerate(scenario):
task_set.plot(path="Archives/Samples/Stream51",
title="Stream51_InstanceIncremental_{}.jpg".format(task_id),
nb_samples=100)
The video dataset with InstanceIncremental scenario is not present in the original Stream-51 paper. It proposes to learn from sequence of videos without cutting them into clips.
from torchvision.transforms import Resize, ToTensor
from continuum.datasets import Stream51
from continuum.scenarios import InstanceIncremental
dataset = Stream51('../Datasets', task_criterion="video")
scenario = InstanceIncremental(dataset, transformations=[Resize((224, 224)), ToTensor()])
for task_id, task_set in enumerate(scenario):
task_set.plot(path="Archives/Samples/Stream51/video",
title="Stream51_InstanceIncremental_{}.jpg".format(task_id),
nb_samples=100)
The clip dataset with ClassIncremental scenario is equivalent to the “instance_class” scenario of the original Stream-51 paper.
from torchvision.transforms import Resize, ToTensor
from continuum.datasets import Stream51
from continuum.scenarios import ClassIncremental
dataset = Stream51('../Datasets', task_criterion="video")
scenario = ClassIncremental(dataset, transformations=[Resize((224, 224)), ToTensor()])
for task_id, task_set in enumerate(scenario):
task_set.plot(path="Archives/Samples/Stream51/video",
title="Stream51_InstanceIncremental_{}.jpg".format(task_id),
nb_samples=100)
iid scenario
from torchvision.transforms import Resize, ToTensor
from continuum.datasets import Stream51
from continuum.scenarios import InstanceIncremental
dataset = Stream51('/path/to/data')
scenario = InstanceIncremental(dataset, transformations=[Resize((224, 224)), ToTensor()])
unique_task_set = scenario[:]
Synbols¶
Synbols is a tool for rapidly generating new datasets with a rich composition of latent features rendered in low resolution images. Synbols leverages the large amount of symbols available in the Unicode standard and the wide range of artistic font provided by the open font community.
Continuum supports most of the official datasets found in the Synbols repository: https://github.com/ElementAI/synbols-resources/tree/master/datasets/generated under the “dataset_name” argument, as shown next. You can also generate your own dataset (see https://github.com/elementai/synbols).
Here is an usage example for InstanceIncremental learning.
from continuum import InstanceIncremental
from continuum.datasets import Synbols
synbols = InstanceIncremental(dataset=Synbols("/my/path", task_type="char", dataset_name="default_n=100000_2020-Oct-19.h5py"))
We also support the domain incremental case. For example, for the “char” classification scenario, we could introduce a new font in each task:
from continuum import ClassIncremental
from continuum.datasets import Synbols
synbols = ClassIncremental(dataset=Synbols("/my/path", task_type="char", domain_incremental_task="font",
dataset_name="default_n=100000_2020-Oct-19.h5py"))
Quick Example¶
Let’s have a quick look with an example. We want to evaluate our model on a split-MNIST. First, we initialize the dataset; if the data isn’t present at the given path, we’ll download it for you! All datasets but ImageNet have this feature.
Then we specify our dataset, here it is ClassIncremental (New Classes NC) where each task
brings new classes. Our first task is made of 5 classes (initial_increment=5),
then all 5 following tasks will be made of a single new class each:
from torch.utils.data import DataLoader
from continuum import ClassIncremental, split_train_val
from continuum.datasets import MNIST
dataset = MNIST("my/data/path", download=True, train=True)
cl_scenario = ClassIncremental(
dataset,
increment=1,
initial_increment=5
)
print(f"Number of classes in scenario: {cl_scenario.nb_classes}.")
print(f"Number of tasks in scenario: {cl_scenario.nb_tasks}.")
for task_id, train_dataset in enumerate(cl_scenario):
train_dataset, val_dataset = split_train_val(train_dataset, val_split=0.1)
train_loader = DataLoader(train_dataset)
val_loader = DataLoader(val_dataset)
# Do your cool stuff here
The continual loader, here named cl_scenario is an iterable. Each loop provides then
the dataset for a task. We split the dataset into a train and validation subset with
our utility split_train_val.
Note that if we wanted to use the test subset, we need to specify it when creating the scenario.
Finally, after training on the 6 tasks, we want to evaluate our model performance on the test set:
cl_scenario_test = ClassIncremental(
dataset,
increment=1,
initial_increment=5,
train=False
)
From this loader, we can get the first task cl_scenario_test[0], all tasks up to the third
task cl_scenario_test[:3], or even all tasks cl_scenario_test[:]. You can slice
any loaders like you would do with Python’s List. You can also slide the scenario with a list
(for example cl_scenario_test[[0,2,3,4]] ).
This library was developped by Arthur Douillard and Timothée Lesort. If you have any new feature request or want to report a bug, please open an issue here. We are also open to pull requests!