PO.DAAC Data Migrating to Earthdata Cloud in AWS
With the impending arrival of high-volume data from new missions such as the Surface Water and Ocean Topography (SWOT), the need to effectively archive and distribute significantly larger data volumes is critically important for PO.DAAC operations. To meet these needs, PO.DAAC is migrating its data archive to the NASA Earthdata Cloud, hosted in Amazon Web Services (AWS).
As PO.DAAC and other Earth Science data move into the Cloud, end users will be able to do more than ever, enabling multidisciplinary and interdisciplinary science and its application through large-scale analytics and cloud computing. For more information visit Earthdata Cloud Evolution page.
PO.DAAC is taking steps to help our users with this cloud data migration. This is the first of several upcoming announcements. It lays out the data migration timeline and the collected resources for better understanding the PO.DAAC cloud data access. Below you will find the necessary details to start the transition of your PO.DAAC data access endpoints to utilize data from the NASA Earthdata Cloud in AWS. Migration Timeline:
PO.DAAC data will be migrated to the cloud in four phases. The datasets for each phase and corresponding migration dates are listed on the PO.DAAC Cloud Data Migration page, but may be refined over time. For up-to-date information, please subscribe to the PO.DAAC mailing list.
PO.DAAC will continue to distribute data from the on-premise archive in addition to making our data available via Earthdata Cloud endpoints. However, on-premise access will be discontinued for migrated datasets at the end of each transition phase, as noted on the Migration section of the PO.DAAC Cloud Data page.
We encourage all PO.DAAC users to complete their transition of legacy data access scripts and methods to conform to cloud data access endpoints at their earliest convenience. To ensure a successful transition for all PO.DAAC users, our User Services team is standing by to address all technical issues and concerns. A PO.DAAC in the CLOUD Forum has also been set up. Information Resources:
PO.DAAC Cloud Data page is the centralized location for all information about cloud data access and resources, and will be continuously updated. It offers access to resources to help guide data users in discovering, accessing, and utilizing cloud data, including tutorials, data recipes, and webinars in the Resources section, Frequently Asked Questions (FAQ), connections to Access Data via tools and services, Migration information, and more.
While the move to the cloud is intended to enable big data (on the order of petabytes) analytics near the data, PO.DAAC understands that the use of the data, in any fashion, is paramount. Data will always be available through traditional mechanisms, such as downloading entire files, at no charge to the end-user. PO.DAAC will continue to add value-added services to data as well, which can be utilized within or outside of the cloud. These services, such as subsetting and regridding, will minimize both the amount of data needed to be transferred and the work required by users to integrate PO.DAAC data into their use cases. Data access and transformation services like subsetting and aggregation will soon be available for cloud-archived data. Cloud Data Access - Getting Started:
For help getting started preparing for the migration, please visit the PO.DAAC Cloud Data page. Among other resources, a video tutorial and accompanying python script guides users on scripted access to PO.DAAC data in the Earthdata Cloud, while this short tutorial walks users through steps on how to bulk-download data using wget.
Please note that each user must create a free Earthdata Login Account in order to download files or access PO.DAAC data through the Earthdata Cloud. Resources* Summary
- Cloud Data page with About, FAQs, Resources, Access Data and Migration information
- PO.DAAC in the CLOUD Forum
- Scripted Access video and python script
- Bulk Download guide
- PO.DAAC Github
- Obtain Earthdata Login Account
- Earthdata Common Metadata Repository (CMR) API
- Earthdata Search Client
- PO.DAAC Cloud Earthdata Search Portal
- Earthdata Cloud Primer documents
*Note: These will continue to evolve and grow as we move forward with the cloud migration.