Note

You can run this notebook interactively: Binder, or view & download the original on GitHub.

Stacking 6 years of imagery into a GIF

We’ll load all the Landsat-8 (Collection 2, Level 2) data that’s available from Microsoft’s Planetary Computer over a small region on the coast of Cape Cod, Massachusetts, USA.

Using nothing but standard xarray syntax, we’ll mask cloudy pixels with the Landsat QA band and reduce the data down to biannual median composites.

Animated as a GIF, we can watch the coastline move over the years due to longshore drift.

Planetary Computer is Microsoft’s open Earth data initiative. It’s particularly nice to use, since they also maintain a STAC API for searching all the data, as well as a browseable data catalog. It’s free for anyone to use, though you have to sign your requests with the planetary_computer package to prevent abuse. If you sign up, you’ll get faster reads.

[1]:
import coiled
import distributed
import dask
import pystac_client
import planetary_computer as pc
import ipyleaflet
import IPython.display as dsp
import geogif
import stackstac

Using a cluster will make this much faster. Particularly if you’re not in Europe, which is where this data is stored.

You can sign up for a Coiled account and run clusters for free at https://cloud.coiled.io/ — no credit card or username required, just sign in with your GitHub or Google account.

[2]:
cluster = coiled.Cluster(
    name="stackstac-eu",
    software="gjoseph92/stackstac",
    backend_options={"region": "eu-central-1"},
    # ^ Coiled doesn't yet support Azure's West Europe region, so instead we'll run on a nearby AWS data center in Frankfurt
    n_workers=20,
    protocol="wss",  # remove this line when not running on Binder
)
client = distributed.Client(cluster)
client
Using existing cluster: 'stackstac-eu'
[2]:

Client

Client-e79eb2e6-5268-11ec-9843-acde48001122

Connection method: Cluster object Cluster type: coiled.Cluster
Dashboard: http://18.184.228.92:8787

Cluster Info

Interactively pick the area of interest from a map. Just move the map around and re-run all cells to generate the timeseries somewhere else!

[3]:
m = ipyleaflet.Map(scroll_wheel_zoom=True)
m.center = 41.64933994767867, -69.94438630063088
m.zoom = 12
m.layout.height = "800px"
m
[4]:
bbox = (m.west, m.south, m.east, m.north)

Search for STAC items

Use pystac-client to connect to Microsoft’s STAC API endpoint and search for Landsat-8 scenes.

[5]:
catalog = pystac_client.Client.open('https://planetarycomputer.microsoft.com/api/stac/v1')

search = catalog.search(
    collections=['landsat-8-c2-l2'],
    bbox=bbox,
)

Load and sign all the STAC items with a token from Planetary Computer. Without this, loading the data will fail.

[6]:
%%time
items = pc.sign(search)
len(items)
CPU times: user 584 ms, sys: 41.1 ms, total: 625 ms
Wall time: 7.03 s
[6]:
375

These are the footprints of all the items we’ll use:

[7]:
dsp.GeoJSON(items.to_dict())
<IPython.display.GeoJSON object>

Create an xarray with stacksatc

Set bounds_latlon=bbox to automatically clip to our area of interest (instead of using the full footprints of the scenes).

[8]:
%%time
stack = stackstac.stack(items, bounds_latlon=bbox)
stack
CPU times: user 311 ms, sys: 10.5 ms, total: 321 ms
Wall time: 319 ms
[8]:
<xarray.DataArray 'stackstac-bae570f2cbe4987fda83bcd5371b5a40' (time: 375, band: 19, y: 773, x: 1158)>
dask.array<fetch_raster_window, shape=(375, 19, 773, 1158), dtype=float64, chunksize=(1, 1, 773, 1024), chunktype=numpy.ndarray>
Coordinates:
  * time                         (time) datetime64[ns] 2013-03-22T15:19:00.54...
    id                           (time) <U31 'LC08_L2SP_011031_20130322_02_T1...
  * band                         (band) <U13 'SR_B1' 'SR_B2' ... 'SR_QA_AEROSOL'
  * x                            (x) float64 4.04e+05 4.04e+05 ... 4.387e+05
  * y                            (y) float64 4.623e+06 4.623e+06 ... 4.6e+06
    proj:epsg                    int64 32619
    instruments                  object {'tirs', 'oli'}
    view:sun_elevation           (time) float64 44.94 50.04 51.92 ... 29.5 27.64
    landsat:collection_category  (time) <U2 'T1' 'T1' 'T1' ... 'T1' 'T1' 'T1'
    landsat:scene_id             (time) <U21 'LC80110312013081LGN02' ... 'LC8...
    view:sun_azimuth             (time) float64 149.6 148.3 ... 163.0 163.2
    description                  (band) <U91 'Collection 2 Level-2 Coastal/Ae...
    landsat:wrs_path             (time) <U3 '011' '012' '012' ... '011' '012'
    platform                     <U9 'landsat-8'
    eo:cloud_cover               (time) float64 92.94 0.77 11.81 ... 0.07 1.59
    landsat:wrs_type             <U1 '2'
    landsat:cloud_cover_land     (time) float64 90.98 1.51 15.5 ... 0.54 2.08
    landsat:wrs_row              <U3 '031'
    landsat:collection_number    <U2 '02'
    landsat:processing_level     <U4 'L2SP'
    view:off_nadir               int64 0
    gsd                          (band) float64 30.0 30.0 30.0 ... 30.0 30.0
    title                        (band) <U46 'Coastal/Aerosol Band (B1)' ... ...
    common_name                  (band) object 'coastal' 'blue' ... None None
    center_wavelength            (band) object 0.44 0.48 0.56 ... None None None
    full_width_half_max          (band) object 0.02 0.06 0.06 ... None None None
    epsg                         int64 32619
Attributes:
    spec:        RasterSpec(epsg=32619, bounds=(403980.0, 4599690.0, 438720.0...
    crs:         epsg:32619
    transform:   | 30.00, 0.00, 403980.00|\n| 0.00,-30.00, 4622880.00|\n| 0.0...
    resolution:  30.0

And that’s it for stackstac! Everything from here on is just standard xarray operations.

[9]:
# use common_name for bands
stack = stack.assign_coords(band=stack.common_name.fillna(stack.band).rename("band"))
stack.band
[9]:
<xarray.DataArray 'band' (band: 19)>
array(['coastal', 'blue', 'green', 'red', 'nir08', 'swir16', 'swir22', 'ST_QA',
       'lwir11', 'ST_DRAD', 'ST_EMIS', 'ST_EMSD', 'ST_TRAD', 'ST_URAD',
       'QA_PIXEL', 'ST_ATRAN', 'ST_CDIST', 'QA_RADSAT', 'SR_QA_AEROSOL'],
      dtype=object)
Coordinates:
  * band                       (band) object 'coastal' ... 'SR_QA_AEROSOL'
    proj:epsg                  int64 32619
    instruments                object {'tirs', 'oli'}
    description                (band) <U91 'Collection 2 Level-2 Coastal/Aero...
    platform                   <U9 'landsat-8'
    landsat:wrs_type           <U1 '2'
    landsat:wrs_row            <U3 '031'
    landsat:collection_number  <U2 '02'
    landsat:processing_level   <U4 'L2SP'
    view:off_nadir             int64 0
    gsd                        (band) float64 30.0 30.0 30.0 ... 30.0 30.0 30.0
    title                      (band) <U46 'Coastal/Aerosol Band (B1)' ... 'A...
    common_name                (band) object 'coastal' 'blue' ... None None
    center_wavelength          (band) object 0.44 0.48 0.56 ... None None None
    full_width_half_max        (band) object 0.02 0.06 0.06 ... None None None
    epsg                       int64 32619

See how much input data there is for just RGB. This is the amount of data we’ll end up processing

[10]:
stack.sel(band=["red", "green", "blue"])
[10]:
<xarray.DataArray 'stackstac-bae570f2cbe4987fda83bcd5371b5a40' (time: 375, band: 3, y: 773, x: 1158)>
dask.array<getitem, shape=(375, 3, 773, 1158), dtype=float64, chunksize=(1, 1, 773, 1024), chunktype=numpy.ndarray>
Coordinates:
  * time                         (time) datetime64[ns] 2013-03-22T15:19:00.54...
    id                           (time) <U31 'LC08_L2SP_011031_20130322_02_T1...
  * band                         (band) object 'red' 'green' 'blue'
  * x                            (x) float64 4.04e+05 4.04e+05 ... 4.387e+05
  * y                            (y) float64 4.623e+06 4.623e+06 ... 4.6e+06
    proj:epsg                    int64 32619
    instruments                  object {'tirs', 'oli'}
    view:sun_elevation           (time) float64 44.94 50.04 51.92 ... 29.5 27.64
    landsat:collection_category  (time) <U2 'T1' 'T1' 'T1' ... 'T1' 'T1' 'T1'
    landsat:scene_id             (time) <U21 'LC80110312013081LGN02' ... 'LC8...
    view:sun_azimuth             (time) float64 149.6 148.3 ... 163.0 163.2
    description                  (band) <U91 'Collection 2 Level-2 Red Band (...
    landsat:wrs_path             (time) <U3 '011' '012' '012' ... '011' '012'
    platform                     <U9 'landsat-8'
    eo:cloud_cover               (time) float64 92.94 0.77 11.81 ... 0.07 1.59
    landsat:wrs_type             <U1 '2'
    landsat:cloud_cover_land     (time) float64 90.98 1.51 15.5 ... 0.54 2.08
    landsat:wrs_row              <U3 '031'
    landsat:collection_number    <U2 '02'
    landsat:processing_level     <U4 'L2SP'
    view:off_nadir               int64 0
    gsd                          (band) float64 30.0 30.0 30.0
    title                        (band) <U46 'Red Band (B4)' ... 'Blue Band (...
    common_name                  (band) object 'red' 'green' 'blue'
    center_wavelength            (band) object 0.65 0.56 0.48
    full_width_half_max          (band) object 0.04 0.06 0.06
    epsg                         int64 32619
Attributes:
    spec:        RasterSpec(epsg=32619, bounds=(403980.0, 4599690.0, 438720.0...
    crs:         epsg:32619
    transform:   | 30.00, 0.00, 403980.00|\n| 0.00,-30.00, 4622880.00|\n| 0.0...
    resolution:  30.0

Mask cloudy pixels using the QA band

Use the bit values of the Landsat-8 QA band to mask out bad pixels. We’ll mask pixels labeled as dilated cloud, cirrus, cloud, or cloud shadow. (By “mask”, we mean just replacing those pixels with NaNs).

See page 14 on this PDF for the data table describing which bit means what.

[11]:
# Make a bitmask---when we bitwise-and it with the data, it leaves just the 4 bits we care about
mask_bitfields = [1, 2, 3, 4]  # dilated cloud, cirrus, cloud, cloud shadow
bitmask = 0
for field in mask_bitfields:
    bitmask |= 1 << field

bin(bitmask)
[11]:
'0b11110'
[12]:
qa = stack.sel(band="QA_PIXEL").astype("uint16")
bad = qa & bitmask  # just look at those 4 bits

good = stack.where(bad == 0)  # mask pixels where any one of those bits are set
[13]:
# What's the typical interval between scenes?
good.time.diff("time").dt.days.plot.hist();
../_images/examples_gif_23_0.png

Make biannual median composites

The Landsat-8 scenes appear to typically be 5-15 days apart. Let’s composite that down to a 6-month interval.

Since the cloudy pixels we masked with NaNs will be ignored in the median, this should give us a decent cloud-free-ish image for each.

[14]:
# Make biannual median composites (`2Q` means 2 quarters)
composites = good.resample(time="2Q").median("time")
composites
[14]:
<xarray.DataArray 'stackstac-bae570f2cbe4987fda83bcd5371b5a40' (time: 19, band: 19, y: 773, x: 1158)>
dask.array<stack, shape=(19, 19, 773, 1158), dtype=float64, chunksize=(1, 1, 773, 512), chunktype=numpy.ndarray>
Coordinates:
  * time                       (time) datetime64[ns] 2013-03-31 ... 2022-03-31
  * band                       (band) object 'coastal' ... 'SR_QA_AEROSOL'
  * x                          (x) float64 4.04e+05 4.04e+05 ... 4.387e+05
  * y                          (y) float64 4.623e+06 4.623e+06 ... 4.6e+06
    proj:epsg                  int64 32619
    instruments                object {'tirs', 'oli'}
    platform                   <U9 'landsat-8'
    landsat:wrs_type           <U1 '2'
    landsat:wrs_row            <U3 '031'
    landsat:collection_number  <U2 '02'
    landsat:processing_level   <U4 'L2SP'
    view:off_nadir             int64 0
    epsg                       int64 32619

Pick the red-green-blue bands to make a true-color image.

[15]:
rgb = composites.sel(band=["red", "green", "blue"])
rgb
[15]:
<xarray.DataArray 'stackstac-bae570f2cbe4987fda83bcd5371b5a40' (time: 19, band: 3, y: 773, x: 1158)>
dask.array<getitem, shape=(19, 3, 773, 1158), dtype=float64, chunksize=(1, 1, 773, 512), chunktype=numpy.ndarray>
Coordinates:
  * time                       (time) datetime64[ns] 2013-03-31 ... 2022-03-31
  * band                       (band) object 'red' 'green' 'blue'
  * x                          (x) float64 4.04e+05 4.04e+05 ... 4.387e+05
  * y                          (y) float64 4.623e+06 4.623e+06 ... 4.6e+06
    proj:epsg                  int64 32619
    instruments                object {'tirs', 'oli'}
    platform                   <U9 'landsat-8'
    landsat:wrs_type           <U1 '2'
    landsat:wrs_row            <U3 '031'
    landsat:collection_number  <U2 '02'
    landsat:processing_level   <U4 'L2SP'
    view:off_nadir             int64 0
    epsg                       int64 32619

Some final cleanup to make a nicer-looking animation:

  • Forward-fill any NaN pixels from the previous frame, to make the animation look less jumpy.

  • Also skip the first frame, since its NaNs can’t be filled from anywhere.

[16]:
cleaned = rgb.ffill("time")[1:]

Render the GIF

Use GeoGIF to turn the stack into an animation. We’ll use dgif to render the GIF on the cluster, so there’s less data to send back. (GIFs are a lot smaller than NumPy arrays!)

[17]:
client.wait_for_workers(20)
[18]:
%%time
gif_img = geogif.dgif(cleaned).compute()
CPU times: user 3.44 s, sys: 309 ms, total: 3.75 s
Wall time: 49.9 s
[19]:
# we turned ~7GiB of data into a 4MB GIF!
dask.utils.format_bytes(len(gif_img.data))
[19]:
'4.38 MiB'
[21]:
gif_img
[21]:
../_images/examples_gif_34_0.png