Controlling potato plant diseases is crucial in agriculture because diseased plants can result in a significant loss in crop production. Farmers currently use manual visual inspections to classify potato plants that are diseased with thepotato virus Y. This paper shows the feasibility of using RGB images to automate this process. A small dataset of potato plants pictures was acquired from an outdoor potato field and annotated with polygons to denote the location of the leaves. VGG16 and ResNet50, two convolutional neural networks, were trained to classify individual leaves. Additionally, experiments were conducted by changing the brightness of the images during training. This was done to counteract the different illumination levels present in our dataset. ResNet50’s best performing experimental setup achieved an accuracy of 0.77, while VGG16 achieved an accuracy of 0.70. In general, classifiers with brightness augmentations outperformed other setups. Examining the class activation mappings of shallow layers revealed a focus on either the margins and midrib or the overall texture of the leaf. This corresponds to the characteristics used by domain experts who concentrate oncolour and mosaic patterns on the foliage.