Corrupted photos (Over Exposition, blur)

Creation of well-trained segmentation model

Extra objects: clouds, smoke

Impossible to classify tree types in winter time

01

AI and ML

02

RealESRGAN

03

ResNeST200

04

Fourier Convolutions

05

Image segmentation using HSV color space and matching with photos of flowering period

SEMANTIC SEGMENTATION WITH NEURAL NETWORKS

IMPLEMENT SUPER RESOLUTION MODEL TO IMPROVE INPUT IMAGES

Used top GAN model (RealESRGAN) to increase input resolution

Model works in unpair mode. Get pairs lowers and higher images – is very complicated task

Used artificial dataset –
(highres photo, compressed
- resized-downsample-noised-photo)

This method allowed
us to increase resolution from 1 meter/per pixel to 25 cm/pixel

We measure dangerous trees square more acurate

Increased resolution made possible to identify types
of trees (pine, birch, spruce) and motivate partners

RESTORED CORRUPTED IMAGES

Satellite image restoration by nonlinear statistical filtering techniques

Bassel Marhaba, 2019

Removed small and medium extra objects with Fourier Convolutions (clouds, fog, smoke, planes, helicopters, baloons)

FEATURES DETERMINATION BASED
ON TRANSFER LEARNING (RESNETST200)

We made
a number
of projects based on satellite photo analysis. Our first project was segmentation
of groups of trees in electric power lines

Firstly we improve quality and resolution images

We get features based on ResNetST200

Based on segmented image we made decision:

Additionally with defined degree of height and density of forest (from 0 to 5).
0 is not needed to clear, 4-5 strongly need
to be removed

Moreover we segmented different kinds
of trees even at winter time

Final adjusted mean iOU is more then 96.5%
(See example on www.cmncosmos.
com)

Not related
to problem

Clear line

Part of
problem line

IMAGE SEGMENTATION USING HSV COLOR SPACE

# convert image to HSV color space
hsv = cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
#H - 0:30, S - 0:150, V - 0:255
lower1 = np.array([0, 0, 0])
upper1 = np.array([30, 150, 255])
mask1 = cv2.inRange(hsv, lower1, upper1)
#первое облако точек

#H - 126:179, S - 0:50, V - 0:255
lower2 = np.array([126, 0, 0])
upper2 = np.array([179, 50, 255])
mask2 = cv2.inRange(hsv, lower2, upper2)
#второе облако точек

OUR (UNEXPECTED) ESG DISCOVERY

We developed our first result to solve problem of short-term forecast of wildfires in Siberia and Far East:
o In this project we used our last networks and Wasserstein
Generative Adversarial Networks (WGAN) for Identification
of wildfires and fill gaps in data under clouds and smoke
o We made two heuristics for training process of WGAN
o We forecasted future air movements and therefore
vectors of fire movements based on LSTM model
o Based on it we estimated future area of wildfires
in short term (4 hours, 8 hours etc.)
o Moreover we estimated value of precipitation. It needs
for understanding degree of danger of spreading fire
o Our plan for nearest future is to retrain our networks based on ViT (visual transformers). This should raise the quality of the model even more

We discover two heuristics for training process of WGANs. We don’t want to show
it totally but we may explain it in common

First is special augmentation of images, which helps us to train our WGAN fast and improve quality

We found a heuristic for strengthening of Kantorovich–Rubinstein duality. It strongly helps us to train. Details in next slides

HOW WE MADE PROJECT WITH FOREST CLEARINGS

We started with manual detection of trees next automated
it with Neural networks segmentation with well precision:
oTook 20 000 photos (20 000 km) from Boeing data. Spring/summer/autumn/ winter in a equal proportions
oPassed photos through ESRGAN to increase resolution
oAnnotated photo dataset in CVAT/PhotoMode (microsoft/intel software) application as polygons (annotated dataset by students)
oThe problem of segmentation of different types of trees is solved using images at different times of the year
oTrained several segmentation models. Metric IOU
oUNET 91,0%,
oDeepLabv3 93,5%,
oResNeST200 96.5%
oCombined GEO tiff segmentation with coordinates of supports of high-voltage lines and safe width standards

So we have got rectangles/bands with dangerous trees segmented. We can calculate area where trees should be cut down (see example www.cmncosmos.com)

WASSERSTEIN DISTANCE AND KANTOROVICH–RUBINSTEIN DUALITY

WGAN is GAN with using Wasserstein metrics as distance

Function f is 1-Lipschitzfunctions according Kantorovich–Rubinstein

We found additional properties
of f and hence adjust standard regularization of loss function

FOREST FIRE MONITORING

DIFFERENT TREE TYPES SEGMENTATION

We combined satellite data, LIDAR data and ground based surveys of forest

We ve collected dataset using
observations of Novosibirsk Oblast (total area is around 20 000 hectares)

Used both
hierarchical classification (aspen/birch
and spruce/pine)
and one-versus-all classification to reach 90,5% of f1 classification score

SATELLITE ECOSYSTEM

PROJECT

SATELLITE MONITORING

Scale:
up to 30 cm in one pixel

Satellite Ecosystem

GOALS:

Creating a Digital Terrain Model (DTM)

Infrastructure monitoring

Various tasks
in agriculture

Oil spill detection

Detecting ships

Monitoring of ice conditions

Cartography

Emergency monitoring

Satellite Ecosystem

INFRASTRUCTURE MONITORING

There are more than 13,000 power transmission lines in Russia

Voltage grade:
110 - 750 kV

Total length:
Over 490,000 km

Annual costs:
EUR 1bn

Measures planned to make power grid more reliable and reduce accidents:

Surveillance of high voltage power line routes

Vegetation clearance
nearby power line routes

Monitoring power line conditions
in inaccessible and remote places

The winning bidder is a company asking for the lowest price

The cost of verifying whether the service was properly provided can exceed 10% of the service cost

You do not know how much money you need to pay for the service without a clear picture of the situation at the site

ROSSETI PJSC'S OFFER
Developing a service for satellite monitoring of woodland near power lines

PROJECT OUTLINE

Satellite Ecosystem

It is impossible to determine the number of trees needed to be cut down
for the power line which is 2,000 km away
from the nearest community

Pilot project:

MVP SOLUTIONS

Satellite Ecosystem

The actual scope of work was estimated, which was not in line with the one announced by the Contractor

1,000 km

⁕ 1 server: 128 cores, 1024GB RAM
⁕ 7 workstations with the following features: Core i7 CPU 3GHz, 6 cores, 16GB RAM
⁕ System for storing work projects must contain 50TB SSD arrays (10 units)
⁕ 2PB HDD storage system
⁕ Network connection 10 Gbit/sec

The task was performed
with the help of the PhotoMod software
Manual labor takes approximately 2 man-hours per km

Pilot project:

MVP SOLUTIONS

Satellite Ecosystem

The task was performed
using the PhotoMod software
Manual labor takes approximately 2 man-hours
per km

1,000 km

⁕1 server: 128 cores, 1024 GB RAM
⁕27 workstations with the following features: Core i7 CPU 3 GHz, 6 cores, 16 GB RAM
⁕System for storing work projects must contain 50TB SSD arrays (10 units)
⁕2PB HDD storage system
⁕Network connection 10 Gbit/sec

Satellite Ecosystem

Comprehensive scaling up of the service
on prospective inspection of 500,000 km power lines is unfeasible due to the following bottlenecks:
⁕Low level of automation
⁕High level of manual labor

1st and 2nd stage takeaways

The turnout for vegetation clearance tenders was low, as it became known that verification of work performance will be mandatory and reliable

Pilot project:

MVP SOLUTIONS

Satellite Ecosystem

The actual amount and type of usable timber to be obtained during project implementation was determined.
The forecast accuracy level is 73%

1,000 km

⁕ 1 server: 128 cores, 1024GB RAM
⁕17 workstations with the following features: Core i7 CPU 3GHz, 6 cores, 16GB RAM
⁕System for storing work projects must contain 50TB SSD arrays (10 units)
⁕2PB HDD storage system
⁕Network connection 10 Gbit/sec

The task was performed
with the help of the PhotoMod software
Manual labor takes approximately 12 man-hours
per km

Satellite Ecosystem

Providing a comprehensive service is unfeasible due to the following bottlenecks:
⁕ Low level of automation
⁕ High level of manual labor

3rd stage takeaways

To continue with the project, the process of service provision needs to be automated

MVP SOLUTIONS

Satellite Ecosystem

*Segmentation grids have evolved in the
following way:
UNET > DeepLabv3 > transformers

Images on which all the objects are identified are subject to the following tasks:

1.Track hazardous objects in an improper place
(e.g., fires or enemy ships)
2.Examine the difference
to the previous frame
to estimate the rate of change
3.Compare the objects' positions, e.g. "to signal that a tree has grown in the power line area"
4.Recalculate transport routes
5.3D reconstruction
and photogrammetry (Google
Earth, 2GIS, Microsoft Flight Simulator, JSC Racurs)
Other tasks

Image segmentation

Determining where the objects of interest are located
(forests, fire sources/fires, swamps, and other objects that are significant
for other purposes: roads,
buildings, vehicles,
ships, oil spills)

Toolkit:

• segmentation grids*
• color threshold filters

Satellite Ecosystem

The team has learned how to train a neural network
in the context of solving a task of clearing forest belts

4th stage takeaways

The tasks from
the site monitoring category
can be accomplished

The ecosystem imaging technology can be widely used all over the world