Earth Observation on AWS, Part 1: Stream Satellite data from Orbit to your S3 bucket

The missing piece for an end-to-end data pipeline, entirely on AWS, from space.

Camping in the North Cascades growing up, stargazing was a treasured pastime for me. My dad and I would often read between the lines of stars and planets to look for satellites… nowadays, they’re so common, it’s hard to stargaze WITHOUT seeing one (except here in New York… tragically, the stars are overshadowed by city lights).

That’s because, over the last ten years, satellites started handling more and more of our data infrastructure. We use these mechanical vantage points to predict wildfires, uphold national security, and provide global gigabit-down internet to every coordinate on the Earth’s surface. Most of these satellites are either privately operated or handle restricted workloads, and are therefore closed off to the public. If you want to launch your own satellite and retrieve data from it, however, you’ll need control over an antenna that is both (a) equipped to track the satellite, and (b) positioned directly under the satellite’s orbit to ensure repeating and reliable contact windows.

That’s a lot of infra to manage, and expensive to own up front. But what if you could rent the infrastructure needed to communicate with a satellite the same way you “rent” a server rack when you launch an EC2 instance? That’s where AWS Ground Station comes in.

Since I do not have the money for a SpaceX launch at the moment, I’ll demonstrate this workflow by capturing downlink from JPSS-1, a weather satellite operated by the National Oceanic and Atmospheric Administration (NOAA) broadcasting data freely to the public. In this article, we will onboard the satellite to our account, schedule a contact window, collect the data the next time the satellite passes over our selected ground station, and view the data in our S3 bucket.

NOTE: This article is meant as a proof-of-concept for developers interested in building a truly source-to-screen geoprocessing pipeline. NOAA provides thousands of its datasets for free, including some on AWS Open Data Registry, and reserving a contact slot to downlink data can cost anywhere from ~$20 to ~$100 USD per session.

BEFORE ANYTHING ELSE: Download and deploy CloudFormation template from docs

Before we touch the console, let’s get the foundational infrastructure in place. AWS provides a pre-built CloudFormation template that creates everything you need in one shot: an S3 bucket for your downlinked data, IAM roles, antenna configs, recording configs, and a mission profile that ties them all together. This saves you from manually wiring up each resource yourself.

Prerequisites:

• An AWS account with the AWS CLI installed and configured (aws configure or aws sso login). You can check if your AWS CLI is installed and authenticated by running “aws sts get-caller-identity” in your terminal (should return your account ID, only performs a read).

• Your CLI pointed at us-east-2 (Ohio), since that’s where our ground station is:
  

aws configure set region us-east-2

Enter fullscreen mode Exit fullscreen mode

a. Download the template:

This pulls the official Ground Station CloudFormation template for public broadcast satellites (JPSS-1, NOAA-20, SNPP, Terra, Aqua) from the AWS-provided S3 bucket:==

aws s3 cp s3://groundstation-cloudformation-templates-us-east-2/AquaSnppJpss-1TerraDigIfS3DataDelivery.yml .

Enter fullscreen mode Exit fullscreen mode

b. Deploy the stack:

This creates all the Ground Station resources in your account. The CAPABILITY_IAM flag authorizes CloudFormation to create the IAM role that grants Ground Station write access to your S3 bucket:

aws cloudformation deploy 
  --template-file AquaSnppJpss-1TerraDigIfS3DataDelivery.yml
  --stack-name gs-satellite-demo
  --capabilities CAPABILITY_IAM
  --region us-east-2

Enter fullscreen mode Exit fullscreen mode

c. Verify the deployment succeeded:

aws cloudformation describe-stacks --stack-name gs-satellite-demo --region us-east-2
 --query "Stacks[0].StackStatus"

Enter fullscreen mode Exit fullscreen mode

You should see CREATE_COMPLETE. If it failed, check the Events tab in the CloudFormation console for details.

d. Record your mission profile ARN (you’ll need this in Step 3 when scheduling contacts):

aws groundstation list-mission-profiles --region us-east-2 --output table

Enter fullscreen mode Exit fullscreen mode

Copy the ARN of the profile the stack just created. We’ll use this later.

For the full reference, see Public broadcast satellite utilizing Amazon S3 data delivery.

1: Submit onboarding request for your satellites

This will likely be the most time-consuming part of the process, but unfortunately it is required for setup. Be warned that getting onboarded is a manual process, and may take up to a week. Keep reading to see the end result, then come back once your account has been approved to finish the pipeline yourself.

Click the “Onboard” button, and you’ll see this questionnaire. For our purposes, leave all optional fields blank, list “JPSS-1” as your satellite, and list the satellite owner as “NOAA”. For company / point of contact, put your school or actual company, and put your own contact info for the point of contact.

If you’re only requesting the public broadcast satellites (PBS), you should get approved pretty quickly. If you’re onboarding your company’s satellite, they may reach out with follow-up questions.

Verify your satellite is onboarded:

Once approved, you can confirm your satellite shows up in your account through the AWS CLI:==

aws groundstation list-satellites --region us-east-2 --output table
  --query "satellites[].{NORAD:noradSatelliteID,Stations:groundStations[0],ID:satelliteId}"

Enter fullscreen mode Exit fullscreen mode

Look for NORAD ID 43013 (NOAA-20 / JPSS-1) with a ground station listed. Save the satelliteId UUID. You’ll need it to schedule contacts.

2: Create a Mission Profile

If you deployed the CloudFormation template above, your mission profile was created as part of the CloudFormation deploy, and you can safely skip this step. I’ll leave a bit of info below describing the Mission Profile.

The Mission Profile provides AWS Ground Station with the parameters required to contact a satellite.

You’ll need to select:

1. A name
2. A config (async s3 vs synchronous EC2 processing — we will start out simple with S3)
3. A minimum contact duration (how long you need to transmit / downlink data)
4. Set your streamsKmsKey and streamsKmsRole, controlling how your data is encrypted in transit from Ground Station to S3.
5. Set your dataflows. Create your dataflows to match your carrier signals using the configs you created in the previous step.

AWS Ground Station provides some pre-built configs and mission plans, including a .yml template (the CloudFormation template we deployed earlier) that can fully configure the necessary resources on your AWS account to receive downlinks from the public satellites.

3: Reserve a time slot with a Ground Station antenna

Select a satellite, ground station, and mission profile, and select “Available” in the Availability box, to get a list of eligible “Contacts”. Each contact represents a period where the satellite is capturing and broadcasting data within range of your selected ground station.

Once you’ve reserved your time slot, it’s time to wait and let the capture happen. In this example, I reserved a timeslot with a Ground Station, owned and operated by AWS in us-east-2 (Ohio), to contact the satellite at 2am last night, over a 7 minute contact period where the satellite was flying within communication range of the ground station.

Intermission: What Ground Station is doing
AWS Ground Station is:

1. Verifying that your profile has an attached license for the specific satellite you’re contacting
2. The antenna physically moves to track the satellite while it’s visible to the antenna (line-of-sight not blocked by the Earth)
3. Downlinking data as the satellite broadcasts it. The data has been modulated, as most RF signals from satellites are, to improve signal clarity. This means that the signal needs to be demodulated, a.k.a “decoded” or transformed back into the original data.

4: Peek into your S3 bucket

What happened overnight?
I woke up and checked my S3 bucket, with the eagerness of a kid checking under their pillow for a note from the Tooth Fairy. I first confirmed that the satellite contact event executed with no errors:

I went into the completed Contact event’s details, and retrieve the config ID:

I copied the Config ID, then searched for it in my Config panel in AWS Ground Station:

I looked at the Config details to get the S3 bucket ARN:

I took the S3 bucket name out of the ARN, and searched for it to find the bucket.

Sure enough, this guy got his wish; 15 beautiful .pcap files (packet capture files containing the raw satellite transmission) streamed directly into my S3 bucket. These .pcap files contain infrared scans of a pass over the Earth, measuring the energy emitted by the surface of the Earth in the path of the satellite while it was within range of the antenna.

What we covered

So far, we walked through reserving a time slot with an antenna, authorizing our account to ingest data from a satellite, and streaming data straight from space into an S3 bucket. The next steps are to demodulate the signal, visualize the data, identify use cases for what we see, and compute analyses based on it.

Stay tuned as we build out this truly end-to-end geospatial pipeline on AWS to sharpen your skills.