What could convolutional autoencoders used for in radar time series?
A summary of Thomas di Martino’s thesis
Introduction to the convolutional autoencoders applied to time series
Convolutional autoencoders in radar time series represent a significant advancement for analyzing and interpreting the vast volumes of data generated by satellites such as Sentinel-1. These technologies allow for deciphering the complex information captured over time and space, thus offering a new perspective on terrestrial dynamics.
Unsupervised Learning: This type of learning refers to techniques where the model learns without using specific labels or annotations. It aims to identify intrinsic patterns and hidden structures in the data without direct human intervention.
Autoencoder: An autoencoder is a type of neural network used to learn an efficient representation (encoding) of input data, generally with the aim of dimensionality reduction. Composed of an encoder and a decoder, it seeks to reconstruct the input data after compressing it into a restricted latent space, thereby capturing the essence of the data.
Convolutional Autoencoders: These autoencoders utilize convolutional layers in their encoders and decoders, making them particularly suited for processing images, time series, and spatial data. Convolutional filters preserve the spatial or temporal relationships between data elements.
In Thomas di Martino’s thesis, the use of convolutional autoencoders to analyze radar data amplitude profiles opens new avenues for understanding radar time series. By applying convolutional filters along the temporal axis, the models account for the causal nature of the data. This means that the autoencoders are capable of detecting and leveraging temporal dependencies in radar data, allowing for a finer and more precise analysis of changes and patterns over time.
This approach is particularly relevant given that deep learning excels when fed with large quantities of data. The radar time series provided by satellites like Sentinel-1 offer an unparalleled wealth of data, covering every corner of the Earth with high precision and resolution. Deep learning, through architectures such as convolutional autoencoders, can thus extract meaningful information from these data, ranging from environmental monitoring to disaster management.
How to use this tool ?
To create a beautiful, colorful representation
It is challenging to qualitatively analyze a time series of images without examining them one by one. The data reduction afforded by autoencoding allows for the coding of information from each pixel in 3 dimensions of a latent space, rather than in N temporal data points. This then enables the creation of colorful compositions that highlight different dynamics through various colors.
For clustering
From the latent space, a K-means algorithm allows for unsupervised classification of the results. Here are, in a non-exhaustive manner, scenarios of use:
- a land cover classification
- a binary forest/non-forest classification
- a classification of agricultural parcels
- the delineation of a forest area previously affected by fire. The limitation of this approach, like the previous one, is that it does not provide a semantic interpretation of the obtained cluster.
Correction of Ground Truth
Suppose we have a ground truth, for example a map of annotated agricultural parcels. The annotations can be a source of errors. By comparing this ground truth reference to the result obtained from clustering with the same number of classes as the ground truth, we can then analyze the sources of inconsistency and consider a possible correction of the class type in the original ground truth.
To better extract dates or periods where the analysis of phenology is useful
Temporal variations within vegetation depend on many factors: seasonality, weather conditions, vegetation growth, cuts, or climatic events. The profiles themselves are very difficult to analyze from a qualitative viewpoint. A gradient descent algorithm allows us to identify the temporal portions of the signal that were most important for their reconstruction in the network’s formation. And by analyzing these temporal signal periods, attention is therefore focused on crucial periods. Applied to profiles of boreal forests, this strategy has enabled the identification that the crucial signal portions corresponded to the periods of freezing and thawing of the forests. And that the distinction between different parcels depended strongly on the backscatter jump observed during these periods. It is known that the soil’s permittivity varies significantly between when it is frozen and when it is not. Our interpretation is that the vegetation layer located above the ground attenuates this threshold in intensities more or less strongly/ These periods of freezing and thawing could thus prove to be crucial in an objective of inverting forest biomass.
The two graphs below illustrate the relative importance of freezing or thawing periods in predicting Sentinel-1 temporal profiles over forest parcels.
Detection of a target or an anomaly
If one wishes to detect the presence of very specific objects in a time series of SAR images, assuming their occurrence is very low compared to the total number of pixels in a time series, their detection is amplified by the simple comparison between the original series and the series found after the autoencoding process. The target is indeed then considered an anomaly and neglected in the autoencoding process. A simple mean squared error between the original series and the reconstructed series allows for its fine detection.
To serve as an intermediary for supervised learning
In cases where we have class labels, or a magnitude to infer (biomass, canopy cover, elevation), even if sparse, we can start with a deep autoencoding of the data before implementing supervised learning.
For more information, the Link towards the PhD. Defense
• T. Di Martino, R. Guinvarc’h, L. Thirion-Lefevre and É. Colin, “Beets or Cotton? Blind Extraction of Fine Agricultural Classes Using a Convolutional Autoencoder Applied to Temporal SAR Signatures,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–18, 2022, Art no. 5212018, doi: 10.1109/TGRS.2021.3100637.
• T. Di Martino, R. Guinvarc’h, L. Thirion-Lefevre and É. Colin, “FARMSAR: Fixing AgRicultural Mislabels Using Sentinel-1 Time Series and AutoencodeRs,” in Remote Sensing, vol. 15, 2023. doi: 10.3390/rs15010035.
• T. Di Martino, B. Le Saux, R. Guinvarc’h, L. Thirion-Lefevre and É. Colin, “Detection of Forest Fires through Deep Unsupervised Learning Modeling of Sentinel-1 Time Series,” in ISPRS International Journal of Geo-Information, vol. 12, 2023. doi: 10.3390/ijgi12080332.
• T. Di Martino, R. Guinvarc’h, L. Thirion-Lefevre and É. Colin, “Grad-SLAM: Explaining Convolutional Autoencoders’ Latent Space of Satellite Image Time Series,” in IEEE Geoscience and Remote Sensing Letters, vol. 20, pp. 1–5, 2023, Art no. 8500805, doi:10.1109/LGRS.2023.3302906.
Conference, with proceedings
• T. Di Martino, M. Lenormand and E. Colin, “Multi-Branch Deep Learning Model for Detection of Settlements Without Electricity,” 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium, 2021, pp. 1847–1850,• T. Di Martino, R. Guinvarc’h, L. Thirion-Lefevre and E. Colin, “Convolutional Autoencoder for Unsupervised Representation Learning of PolSAR Time-Series,” 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium, 2021, pp. 3506-
3509,• L. Charrier, T. Di Martino, E. Colin, F. Weissgerber and A. Plyer, “Extracting Relevance from SAR Temporal Profiles on a Glacier and an Alpine Watershed by a Deep Autoencoder,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 43, pp. 1309–1316, 2022
• T. Di Martino, E. Colin, L. Thirion-Lefevre and R. Guinvarc’h, “Modelling of agricultural SAR Time Series using Convolutional Autoencoder for the extraction of harvesting practices of rice fields,” EUSAR 2022; 14th European Conference on Synthetic Aperture Radar, Leipzig, Germany, 2022, pp. 1–6.
• T. Di Martino, R. Guinvarc’h, L. Thirion-Lefevre, and E. Colin, “Towards the understanding of the C-Band temporal signature of boreal forest through physiology parameters retrieval from Sentinel-1 Time Series and Machine Learning,” 2023 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Pasadena, USA, 2023, pp. 1847–1850,
