Automated detection of pulsating aurora using deep learning methods

1 Introduction

Auroras are captivating natural light displays that appear (predominantly) in Earth’s polar regions, and that result from a complex interplay between Earth’s upper atmosphere, the near-Earth plasma environment, Earth’s magnetic field, and the solar wind. There are distinct types of auroras, including diffuse, pulsating, discrete, and substorm. The primary objectives of the research presented herein are to detect and classify auroral types from videos captured by a geographically extensive network of all-sky cameras deployed across North America (Donovan et al., 2006; Mende et al., 2008). Our specific focus lies on pulsating aurora, which are caused by magnetically trapped energetic electrons in the Van Allen radiation belt region of the magnetosphere. The electrons are pitch angle scattered into the atmosphere through resonant interaction with magnetospheric plasma waves, resulting in a display of light (Jaynes, 2018). Pulsating aurora appear as diffuse patches of luminosity ranging between 10s to 100s of kilometers in size. These fade, quasi-periodically, in and out of existence with periods typically between 2 and 20 s. A pulsating aurora event can last for hours at a time and include many of these patches, each with a unique size and periodicity (Lessard, 2013; Troyer et al., 2023).

The detection of pulsating auroras is necessary to gain valuable insights into their characteristics, including frequency, duration, occurrence, and seasonality. Additionally, exploring the year-to-year variations in pulsating auroras presents an intriguing avenue for research. However, the vast amount of available data spanning several decades, captured by dozens of all-sky cameras, makes mass-scale manual classification impractical. The unique, long lasting, and easily identifiable nature of pulsating aurora makes them an ideal candidate for machine learning identification. We can use machine learning tools for automatic classification, which provide efficient means to analyze aurora data and conduct statistical investigations.

Machine learning techniques have been used for automatic auroral image classification. For example, Syrjäsuo and Donovan (2002), Syrjäsuo and Donovan (2004) applied the k-nearest neighbors (KNN) algorithm to classify auroral images. Syrjäsuo et al. (2004) used Fourier descriptors to extract auroral features, such as edges and shapes, from auroral images. This method was further studied by Syrjäsuo et al. (2007), and they also developed an automatic classification method based on basic gray level aurora matrices. Moreover, support vector machine (SVM) algorithms were applied for automatic auroral detection (Rao et al., 2014). The effectiveness of those methods has been limited in handling vast amounts of image data. These methods have not gained widespread usage in the field of aurora classification possibly due to their limited generalizability, lower classification accuracy, or inadequate availability of relevant and desired auroral labels (Kvammen et al., 2020). Yang et al. (2012) used uniform local binary pattern operators to extract auroral features and developed a hidden Markov model for classification. Their method incorporates the temporal information in aurora data, which is different from the aforementioned methods focusing on single-frame classification.

Since the early 2010s, there have been remarkable advancements in deep convolutional neural networks (CNNs), driven by breakthrough architectures such as the AlexNet (Krizhevsky et al., 2012) and the ResNet (He et al., 2016a). These developments have propelled the field of image and video classification to new heights. Deep learning methods have exhibited exceptional capabilities in handling image and video data, surpassing the performance of traditional machine learning methods, particularly when dealing with large datasets. Building upon these advancements, our approach leverages deep learning techniques to classify aurora videos, taking advantage of the vast amount of data.

The application of deep learning methods for aurora detection and classification has been explored in prior studies. Notably, Clausen and Nickisch (2018) employed a pretrained CNN model to extract 1,001-dimensional feature vectors from images, subsequently training a ridge classifier on these vectors. Their dataset consisted of 5,824 manually labeled images with six labels (“clear/no aurora”, “cloudy”, “moon”, “arc”, “diffuse”, and “discrete”). Their model resulted in an overall accuracy of 81.7 ± 0.1%. Kvammen et al. (2020) explored six distinct CNN architectures, as well as KNN and SVM methods, for aurora classification. Their classification task encompassed seven classes: “breakup”, “colored”, “arcs”, “discrete”, “patchy”, “edge”, and “faint”. They used a clean, labeled dataset of 3,846 color (RGB) images, carefully devoid of ambiguous auroral forms and noise sources like moonlight and clouds. Their best model achieved an average precision of 92%. Sado et al. (2022) employed a transfer learning approach, capitalizing on pretrained CNN models to extract feature vectors. These vectors were subsequently employed in training SVM classifiers for the classification task. They used the same dataset as Clausen and Nickisch (2018), and achieved up to 91% accuracy.

More recently Johnson et al. (2024) was successful in running an automated machine learning auroral detection algorithm on the entire THEMIS All Sky Imager (ASI) database between 2008 and 2022. They used a Simple framework for Contrastive Learning of Representations (SimCLR) algorithm to perform unsupervised classification of the THEMIS images. They then trained a classifier model using the Oslo Aurora THEMIS (OATH) (Clausen and Nickisch, 2018) dataset to assign labels of Arc, Discrete, Diffuse, Clear/No Aurora, Cloudy, and Moon to the classified images. While this study is groundbreaking there are several points that limit its applicability to pulsating aurora. The first is that while the authors performed a visual inspection of the classification accuracy, they admit for this type of unsupervised learning it is difficult to obtain a rigorous performance metric. The second is that the OATH dataset does not have a separate category for pulsating aurora and these labels are instead combined with the larger diffuse auroral category. Additionally, Johnson et al. (2024) mentions that they noticed issues with their diffuse auroral classifications sometimes getting confused with cloudy periods. In a similar study, Partamies et al. (2024) also used SimCLR to perform unsupervised training of a images from the Kjell Henriksen Observatory on Svalbard, Norway. While they also did not provide metrics for the accuracy of their classifications, they were able to use the algorithm to detect auroral breakup periods.

In comparison with existing research employing deep learning methods for aurora classification, our work introduces the following innovations:

• Utilizing Temporal Information: Unlike previous research that predominantly focuses on classifying individual images, we incorporate temporal information from video data into the aurora classification process. By considering the temporal aspect, our approach captures dynamic patterns and temporal changes, which could enhance the accuracy of aurora classification.
• Extensive Training Dataset: Our models benefit from being trained on a large-scale dataset comprising approximately 100,000 carefully selected representative video frames. This substantial dataset surpasses the typically smaller datasets utilized in other papers, which generally contain fewer than 10,000 labeled images. The large dataset provides us a comprehensive and robust foundation for model training.
• Tailored Image Processing Procedure: In our study, we present an image processing procedure for auroral images. This procedure aims to enhance the visibility of various patterns within the images while effectively mitigating noise, resulting in improved discrimination and classification accuracy.
• Focused Study of Pulsating Auroras: While earlier works often treat pulsating auroras as diffuse auroras (Sado et al., 2022; Johnson et al., 2024), we place specific emphasis on pulsating auroras in our research. This focused attention on pulsating auroras allows us to gain insights into their features and distinguish them from other forms of aurora.
• Realistic and Large-Scale Evaluation: Our method has been tested on an out-of-sample testing data set with 58 videos with each frame labeled, thereby providing realistic performance metrics for prediction. Furthermore, similar to Johnson et al. (2024), we have applied our method to the entire large-scale THEMIS image database.

Through these innovations, we aim to address the limitations in previous research and improve classification performance.

This paper is organized as follows. Section 2 provides an introduction to the dataset, including details on data acquisition and the procedures involved in data processing. Section 3 elaborates on the deep learning models employed in our study, together with the implementation of the models. In Section 4, we present the outcomes obtained from the models. Section 5 concludes the paper, summarizing the findings and highlighting potential avenues for future research and development.

2 Data acquisition and processing

We collect auroral data from two primary sources: the Poker Flat Research Range (PFRR) Digital All-Sky Camera (DASC) managed by the University of Alaska, and the Time History of Events and Macroscale Interactions during Substorms (THEMIS) project. The THEMIS project incorporates 20 ground-based all-sky cameras located across the North American continent, where the exact number of cameras varies based on the year. The THEMIS dataset is managed by the University of Calgary’s Auroral Imaging Group, which comprises a comprehensive directory of daily all-sky video frames captured by the THEMIS cameras over several decades. Both resources are highly valuable, providing us with an extensive and diverse collection of data for our research purposes. Additionally all the camera systems lie with the region of common pulsating aurora for normal geomagnetic conditions Jones et al. (2013).

The raw video frames are 16-bit grayscale images, exhibiting pixel values ranging from 0 to 65,535. Through our experience, the patterns associated with pulsating auroras typically fall within a range of several hundred, such as 200 to 600 in PFRR. For the THEMIS cameras, the corresponding ranges are somewhat different. To improve pattern visibility and distinguishability, we employ a technique called Contrast Limited Adaptive Histogram Equalization (CLAHE) (Pizer et al., 1987; Reza, 2004). CLAHE is an image processing technique that enhances contrast while preventing the over-amplification of noise. In particular, we specify the contrast limiting threshold as 3 and set the grid size for histogram equalization to be (8, 8).

After applying the CLAHE algorithm to the images, we downsize the resulting 16-bit images to 8-bit images using a linear transformation. We have also experimented with other image processing methods. For instance, we have attempted to downsize the 16-bit auroral images using a piecewise linear function known as Rectified Linear Unit (ReLU), which is also commonly used as an activation function in neural networks (Nair and Hinton, 2010). Another approach is to process the 16-bit images using Histogram Equalization (HE) (Jain, 1989), and then linearly downsize them to 8-bit. However, we find that CLAHE is the most suitable method for our purpose. One notable advantage of the downsizing approach is the reduction in memory and computational resource requirements for deep learning. Figure 1 displays some examples of raw images and images processed using different methods. Sado et al. (2022) also downsized the 16-bit aurora images before classification. They employed a linear scaling method spanning the 0.5% and 99.5% intensity percentiles.

For each camera’s data collected on a given day, we save the processed frame sequence and timestamps into HDF5 file format, allowing for efficient storage and retrieval of the downsized images. Additionally, we generate videos based on the processed frames, making it convenient for visually inspecting the dataset.

During the training and testing procedures, we also mask the boundaries of the video frames. Sometimes, aurora patterns occur near the boundaries of the camera’s field of view, making it challenging to determine whether they are genuine auroras and what their types are. The presence of noise around the boundaries (such as building exhaust) can be highly misleading. To address this issue, we apply a masking technique that excludes a 15-degree angular elevation range along the boundary of the circular field of view. Figure 2 displays an illustration of masking the boundaries.

The labeling of auroral images into different categories can vary depending on the different principles. For instance, Clausen and Nickisch (2018) employed a classification scheme consisting of six categories: arc, diffuse, discrete, cloudy, moon, clear/no aurora. Kvammen et al. (2020) utilized a classification scheme comprising seven categories: auroral breakup, colored aurora, auroral arcs, discrete-irregular, patchy aurora, edge aurora, and faint clear. In line with our primary objective, we label the processed videos as belonging to the following four categories:

• Bad Viewing (BV): This category includes cases where approximately 70% of the screen is covered by cloud/snow or exhibits poor contrast/disturbance, making the viewing quality unsatisfactory.
• No Aurora (NA): Under this category, the viewing conditions are good, but no aurora is observed. It is worth noting that stars are generally visible in NA but not in BV.
• Other Aurora (OA): This category includes cases where the viewing conditions are good, and auroras other than pulsating or patchy auroras are observed.
• Pulsating Aurora (PA): The viewing conditions are good, and either pulsating or patchy auroras are observed. We consider patchy aurora to be pulsating aurora that do not visible vary in brightness.

Some sample images for each class are shown in Figure 3.

We utilize keograms to identify the videos that potentially contain auroras and roughly locate the auroras within them. A keogram is generated by selecting a column of pixels from each frame of a video and arranging them chronologically (Grono and Donovan, 2018), which provides a visual representation of auroral intensity. By checking a keogram, we can assess the overall auroral activity throughout a particular night for a given camera. After selecting candidate videos based on the keograms, we manually examine and label the video segments that contain auroras. Additionally, we identify and label the cases for bad viewing conditions and no aurora. To make the training data more representative, we also randomly select some periods of time and label the video frames from those periods.

Manual classification of the video frames presents several challenges, even after preprocessing. The grayscale nature of the video frames can make it difficult to distinguish between auroras and smoke in some cases. Additionally, partial obstructions of auroras caused by clouds can hinder accurate labeling. The presence of various sources of noise, including rain, snow, and the bright moon, can further complicate the classification process. Moreover, it is challenging to identify the beginning and end of an aurora. Specifically, assigning labels to the transitional period between no aurora and pulsating aurora can be difficult. To address these issues, we implemented a confidence level system for labeling. This system utilizes a confidence score scale from 1 to 4, where 4 signifies the highest confidence level and 1 represents the lowest. During the training phase, we exclusively use data with confidence scores of 3 and 4, ensuring high-quality and reliable training samples. The data with lower confidence scores (1 or 2) are set aside for potential future research purposes.

The manual classification for each video includes the start time, end time, and corresponding frame numbers for the observed categories. These labeled segments consist of continuous frames with the same classification. Each segment is then divided into several non-overlapping video clips. The PFRR DASC records approximately one frame every 12 s, while each camera of the THEMIS takes one frame every 3 s. Each video clip derived from the PFRR DASC contains 20 consecutive frames, while each video clip from the THEMIS cameras contains 80 consecutive frames. To ensure consistency, we chose to include all PFRR DASC frames and every 4th frame from the THEMIS video clips.

A video segment of identically labeled frames from the THEMIS cameras could contain more than 10,000 frames, and the video clips from such a segment are generally similar, especially for the BV and NA classes. Due to the similarities, those video clips may not be informative. To tackle this concern, we employed a random selection process to choose a maximum of 45 clips for each BV and NA segment, while retaining all clips for the OA and PA classes. As our primary focus is on classifying videos obtained from the THEMIS cameras, only labeled PA and OA video clips were selected from the PFRR DASC, with the selection restricted to a maximum of 20 clips for each segment. We chose to include the PFRR DASC frames into our training dataset to provide context from a additional system and potentially make the model perform better across different cameras. We would need to perform a detailed model comparison across different cameras to determine if that was actually the case and so leave that work for a future study.

A total of 5405 video clips have been carefully chosen to serve as the training and testing datasets for the RNN-CNN model. In parallel, for the CNN model, frames are extracted from the video clips to create the training and testing datasets. However, during this extraction process, we discovered a minor issue with some repeated frames from the PA class. Specifically, 158 repeated frames were identified, likely stemming from data processing issues. These duplicated frames were removed from the training and testing datasets for the CNN model.

3 Deep learning methods and classification results

3.1 CNN model

This subsection builds upon the ideas presented in Rosebrock (2019), where a CNN model is trained to classify individual video frames and then a prediction averaging algorithm is employed to enhance the classification results for video data. The labeled dataset consists of 107,942 frames, with 35,740 frames classified manually as BV, 17,280 frames as NA, 20,040 frames as OA, and 34,882 frames as PA. The labeled dataset is split into training and testing sets, with 75% allocated for training and 25% for testing purposes. We use the ResNet model (He et al., 2016a) with 50 layers as the base model, whose parameters are pretrained on the ImageNet dataset. By excluding the last fully connected layers of the base model, we are able to leverage the extracted features for our classification. The input images are required to have dimensions of 224 × 224 pixels, and the model produces a 2048-dimensional feature vector. The parameters of the pretrained base model are kept fixed during subsequent training stages.

To construct a custom classifier, we introduce additional layers on top of the base model, which are then trained accordingly. These layers are structured as follows: initially, a dense layer with 512 units and ReLU activation is incorporated. A dense layer is also known as a fully connected layer, where each node (neuron) in the current layer is connected to every node in the preceding layer. Let x = ( x 1 , x 2 , … , x n ) ⊤ be the input neurons and y = ( y 1 , y 2 , … , y m ) ⊤ be the output neurons. The dense layer specifies the relationship,where f ( ⋅ ) is an activation function which introduces non-linearity into the model, w i , j is the weight and b i is the bias. An alternative expression iswhere W is an m × n weight matrix and b is the m-dimensional bias vector.

y i = f ∑ j = 1 n w i , j x j + b i ,

y = f W x + b ,

ReLU is a nonlinear activation function, which is used to deal with the vanishing gradient problem. It takes a scalar as input, and is defined as,

ReLU x = max x , 0 .

Subsequently, a dropout layer is introduced after the dense layer in order to mitigate overfitting. The dropout layer randomly sets a certain proportion of the output features from the dense layer to zero. Lastly, we include another dense layer comprising 4 units, aligning with the number of classes in our classification task. The final dense layer is activated by the softmax function, enabling the generation of class probabilities for each input image. Softmax is an activation function that converts the input vector to class probabilities. Let z = ( z 1 , z 2 , … , z K ) be the input vector, the softmax function is defined as,

softmax z i = exp z i ∑ j = 1 K ⁡ exp z j , i ∈ 1,2 , … , K .

A brief architecture of this neural network is given in Figure 4A. In the training process, we use stochastic gradient descent (SGD) as the optimizer. SGD is an optimization technique that adjusts model parameters iteratively by following stochastic estimates of the gradient of the loss function. The model is compiled utilizing the categorical cross-entropy loss function, which is defined as,in this equation, K is the total number of classes and i indexes the data points. The vector ( p i , 1 , … , p i , K ) represents the predicted probability distribution for data point i generated by the softmax activation function. The vector ( y i , 1 , … , y i , K ) represents the true distribution, where y i , j = 1 if and only if data point i belongs to class j and 0 otherwise. This loss function measures the dissimilarity between the predicted probability distribution and the true probability distribution of the classes, which is suitable for multi-class classification. Another approach, and possible future improvement to our model to increase stability and accuracy, would be to use a weighted loss function such as Aminalragia-Giamini et al. (2021).

L i = − ∑ j = 1 K y i , j ⁡ log p i , j .

Furthermore, to enhance the robustness of our approach, we employ several data augmentation techniques, including rotation, width and height shift, and brightness range adjustments. Specifically, the images undergo random rotation within the range of 0–359°. Horizontally and vertically, the images can be randomly shifted by up to 10% of the total width or height. In addition, image brightness is randomly adjusted within the prescribed range of 0.8–1.3. Due to the fixed field of view of the cameras, other data augmentation methods such as zooming, shearing, and stretching are considered unsuitable and thus not employed.

During the classification process, the classification score for each frame in the video is smoothed by taking the average with equal weights of the classification scores for that frame and the previous 19 frames. This technique effectively mitigates the flickering of predicted labels for consecutive frames, even though it may not fully harness the temporal information in the video sequence. Despite this limitation, it has been observed that this method has achieved practical success in aurora detection and classification.

One possible future improvement for this model would be to perform an exhaustive hyperparameter search. We chose not to do so because the scope of this study was a physics-focused application of machine learning, rather than the development of a new optimized computer vision or video-classification architecture. For the current model implementation our parameters were as follows: A ResNet50 backbone pretrained on ImageNet, with the pretrained backbone kept fixed during training. The classification head consisted of a 512-unit fully connected layer with ReLU activation, dropout regularization with a dropout rate of 0.5, and a four-class softmax output layer. The model was trained using categorical cross-entropy loss and stochastic gradient descent with a learning rate of 1 × 1 0 − 4 and momentum of 0.9. The batch size was 32, and the model was trained for 60 epochs.

3.2 CNN-RNN model

This subsection builds upon the ideas in Paul (2021) for video data classification using a hybrid CNN-RNN architecture. Videos possess both spatial and temporal information, where frames capture spatial details and their sequence conveys temporal dynamics. To effectively capture and represent these aspects, a hybrid model combining CNN and Recurrent Neural Network (RNN) is employed. The labeled dataset consists of 5,405 video clips, where each clip consists of 20 consecutive frames. The individual frames of these videos are what we trained the CNN only model on. The distribution of the labeled dataset across categories is as follows: 1,787 clips manually classified as BV, 864 clips as NA, 1,002 clips as OA, and 1,752 as PA. The labeled dataset is partitioned into training and testing sets, with 75% allocated for training and 25% for testing purposes.

The input frames are resized to 224 × 224 pixels and processed by a ResNet50V2 model (He et al., 2016b) for feature extraction. This model is pretrained on the ImageNet dataset. The last fully connected layer of this model is removed, and the model produces a feature vector of size 2048 for each frame. Consequently, for each video clip, a sequence of 2048-dimensional feature vectors of length 20 is obtained, serving as input for the subsequent RNN processing.

The RNN model here is based on the Gated Recurrent Unit (GRU) architecture, which was first introduced by Cho et al. (2014). Our RNN model consists of two GRU layers, with the first layer comprising 32 units and returning sequences to retain temporal information. The second layer consists of 16 units, which returns the final hidden state of the GRU at the last time step. To prevent overfitting, a dropout layer is applied after the GRU layers. The final dense layer of the model generates class probabilities using the softmax activation function. This model is configured with categorical cross-entropy loss and utilizes the Adam optimizer (Kingma and Ba, 2014). Figure 4B displays the architecture of this model.

Similar to the CNN model, a possible future improvement would be hyperparameter tuning. The parameters we used for this model are as follows: Each frame was processed using an ImageNet-pretrained ResNet50V2 feature extractor, producing a 2048-dimensional feature 1 vector per frame. The recurrent classifier consisted of two GRU layers with 32 and 16 units, respectively. The first GRU layer returned the full sequence, while the second GRU layer summarized the sequence into the final hidden representation. A dropout layer with rate 0.4 was then applied, followed by a 16-unit fully connected layer with ReLU activation and a four-class softmax output layer. The model was trained using sparse categorical cross-entropy loss and the Adam optimizer for 50 epochs, with a batch size of 32.

3.3 Outcomes

The performance of the models is measured by the precision, recall, and F1-score. For a given class, these metrics are defined as

Precision = number of items correctly labelled as belonging to that class number of items labelled as belonging to that class ,

Recall = number of items correctly labelled as belonging to that class number of items truly belonging to that class ,

F 1 - score = 2 Recall − 1 + Precision − 1 .

The following metrics are used to evaluate the overall performance of the model.

Accuracy = number of correct labels total number of labels

The macro averages of precision, recall, and F1-score are calculated as the arithmetic means of the precision, recall, and F1-score values across all classes, respectively. On the other hand, the weighted average takes into account the sample size of each class when computing the average.

The classification reports for the CNN model and the CNN-RNN model are given in Table 1 and Table 2, respectively.

TABLE 1. Performance evaluation on the testing dataset using the CNN model.

TABLE 2. Performance evaluation on the testing dataset using the CNN-RNN model.

Both algorithms demonstrate satisfactory performance across classes based on their precision, recall, and F1-score, particularly for the PA class. The CNN-RNN model exhibits somewhat superior performance compared to the CNN model. However, this may be caused by the sample size discrepancy. While both models have labeled datasets with almost the same number of total frames, the CNN model’s training and testing datasets are nearly twenty times larger than those of the CNN-RNN model. This difference arises from the fact that each data point in the CNN model represents a single frame, whereas in the CNN-RNN model, each data point comprises a sequence of twenty frames.

It is worth noting that the testing dataset shares similarities with the training dataset due to random splitting of the labeled dataset. Also, the labeled dataset consists of frames with relatively high confidence. When applying the methods to classify a different dataset of aurora videos, it is expected that the precision and recall will be lower compared to the values presented in Tables Table 1 and Table 2. This is because those videos may contain some non-ideal clips that are challenging even for manual classification. Therefore, the accurate precision and recall are hard to derive for those cases. Moreover, multiple types of auroras can occur simultaneously, making a single label inadequate for this phenomenon. To address this issue, multi-label classification methods are promising for future studies.

To further compare these two models, we use a new testing dataset consisting of 58 videos. For each video in this dataset, all the frames are manually labeled. However, it is worth noting that the manual classification includes some cases with low confidence levels. The CNN model classifies the videos frame by frame, and the resulting predictions are subsequently smoothed by the averaging algorithm. For the CNN-RNN model, we employ a sliding window algorithm with a step size of 1 frame to predict all the clips of length 80 frames for each video. Each clip’s predicted label is assigned to the last frame of that clip. The first 79 frames of each video are excluded from the classification performance calculation. This adjustment ensures that both methods classify an equivalent number of frames. The classification reports are given in Tables Table 3 and Table 4. The CNN model results in an overall accuracy of 69.1% while the CNN-RNN model achieves an overall accuracy of 55.9%. The CNN model outperforms the CNN-RNN model when testing on a new dataset. This superiority can be attributed to the utilization of data augmentation techniques during the training of the CNN model, enhancing its robustness. When compared to the classification results on the high confidence testing dataset, both models exhibit lower performance. The relatively inferior performance of the CNN-RNN model on the new testing dataset can likely be attributed to a combination of factors, including the relatively small sample size and the potential issue of overfitting. Due to these unknowns we chose to use the CNN model for our classification of the THEMIS ASI dataset and leave a further investigation of the CNN-RNN method to a future study. One possible future improvement to the CNN-RNN model could be to provide an even more homogeneous training dataset which is fine-tuned for individual cameras.

TABLE 3. Performance evaluation on the New testing dataset using the CNN model.

TABLE 4. Performance evaluation on the New testing dataset using the CNN-RNN model.

The results shown in Table 3 and Table 4 using our new testing data set appear worse than results from other similar papers with performance metrics (Clausen and Nickisch, 2018; Kvammen et al., 2020; Sado et al., 2022). However, it is important to note that these papers tested their methods on selected video frames rather than the entire videos. In comparison with their approach, our out-of-sample testing is likely more applicable to real world classifications. First, as auroras are recorded in video format by both the PFRR and the THEMIS, predicting the entire videos is of interest. Secondly, this approach provides practical information of the “confidence” levels in detecting auroras in video data. Lastly, predicting the whole videos could provide a more comprehensive understanding of auroral activities and how they change over time. We believe that this testing method is more representative of real world use. For instance, we do not expect to get classification accuracy in the 90% range when applied to any random ASI image as Tables Table 1 and Table 2 would indicate. Instead, we would expect this to be in the 50%–60% range as shown by Tables Table 3 and Table 4. This is still very useful for pulsating aurora research. We will need to perform additional analysis, outside the scope of this study, to determine if this method can be used to produce a statistical datasets without researcher input. However, at a minimum this will significantly speed up pulsating aurora identification by down sampling the times to search through and will open up the ability to conduct much larger statistical studies of pulsating aurora. To add additional context to the performance metrics of Tables Table 3 and Table 4, we show the corresponding confusion matrices in Table 5 and Table 6.

TABLE 5. Out-of-sample confusion matrix using the CNN model.

TABLE 6. Out-of-sample confusion matrix using the CNN-RNN model.

4 THEMIS dataset results

As an application of the CNN model we processed the entirety of the THEMIS white light ASI dataset. We used the model to predict bad viewing conditions, no aurora, pulsating aurora, or other aurora for each image. In total, this dataset is ∼ 130 Terabytes, is comprised of ∼ 1.05 billion images, and took roughly 14 days to process. We performed the processing by segmenting the dataset into 20 chunks and processing each of those chunks with a separate Red Hat Linux virtual machine created on a server local to the data. These virtual servers had 20 CPU cores and 32 GB of RAM each. To validate the classifications we collected a random sample of 87 days from each of the 20 THEMIS ASI locations. To improve efficiency of our manual classification we split the videos into 20-min sections and made sure that each camera had at least 20 of these sections. We then kept only those sections that an expert auroral observer visually identified as having a continuous 20 min of pulsating aurora or no pulsating aurora. We compared these periods to the deep learning classifications. The results of this analysis are shown in Figure 5. Panel A shows the recall for pulsating aurora observations, Panel B shows the precision, and Panel C the F1 score.

A couple observations stand out in Figure 5. The first is that the classification quality varies drastically between different cameras from unusable to very usable. In certain cameras, we were unable to find any continuous 20-min periods of pulsating aurora, which is why there are no bars for a few locations. There are a few possible reasons for these discrepancies. Physical location likely plays a large role. We have noticed that certain cameras are placed in areas where artificial lights can flash or exhaust from nearby buildings can drift across the field of view. Both of these can “confuse” the automatic classification. Additionally, certain locations experience more rain/snowfall which can obscure the lens and also cause issues in classification. Camera age and physical condition may also play a role in the differences between systems. If the optical dome is scratched, or damaged it would like cause further issues in the classifications. If machine learning techniques will be used in future all sky camera networks, the differences we observed make a case for considering ML applications when selecting sites.

The second observation of Figure 5 is that the precision values are higher than the recall values. This indicates that the machine learning method is prioritizing the quality of each identification of pulsating aurora instead of trying to capture every single instance. However, we believe the values shown in Figure 5 do not necessarily tell the whole story. For instance, in certain cases of continuous pulsating aurora, as identified by our manual classification, we found the machine learning method to “flicker” between pulsating aurora and non-pulsating aurora classifications. While we did implement a rolling average of the classifications to “smooth” them (see Section 3.1 for more detail), it is possible this is still an issue and causing the metrics to skew towards a higher precision. We believe that with some careful analysis the precision and recall values shown in Figure 5 can be improved, but we leave that work and a more complete validation to a future study.

5 Conclusion

This paper advances the field of aurora classification by using both the CNN and CNN-RNN models. By incorporating temporal information and leveraging an extensive training dataset, our methods have achieved satisfactory classification performance for the THEMIS dataset based on our pilot experiments. Moreover, by placing specific emphasis on pulsating auroras, our approach has the potential to provide valuable insights into their features. Applying our method to the entire THEMIS dataset opens up avenues for future statistical analysis and exploration that we are currently working on. By testing this model against a real world representative sample of data we provide context on its true capabilities. We believe this will make it easier for future research to incorporate the model.

Compared to other automatic auroral classification studies such as Johnson et al. (2024) our work is unique for two primary reasons. The first is that we focused specifically on pulsating aurora, a type of aurora which is very common, but included in the broader diffuse aurora class in all other studies we could find. The second is that we provided real world performance metrics to ensure future studies using the classifications have a quantified uncertainty. We also include initial performance of the model broken down by individual THEMIS cameras to help future studies better choose cameras when using the automatic classifications.

There are several potential avenues for further improving the classification performance. For the CNN model, one possible enhancement is the implementation of statistical methods to refine the prediction scores, moving beyond simple averaging. Additionally, the adoption of state-of-the-art video classification techniques, such as the SlowFast networks proposed by Feichtenhofer et al. (2019), holds promise for further improvement.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Yuichi Otsuka, Nagoya University, Japan

Reviewed by: Ze-Jun Hu, Polar Research Institute of China, China

Savvas Raptis, Johns Hopkins University, United States