Commit adc18b06 authored by SILVANT's avatar SILVANT

Create the structure for the Downloader.

- add the first step for mirroring NHC's FTP
parent 55e7e1d5
# atcf_mirror
# Official ATCF mirroring from remote sources
## installation
The installation is done with the command :
> pip install -e .
in a Conda environment.
## Create a mirror of all official ATCF Tracks
Create a config file named `atcf_mirror/etc/atcf_mirror.conf`, and fill it with working user and password :
> jtwc_id="xxxxx"
jtwc_password="xxxxx"
*If needed, call the Cyclobs team at Ifremer to get an access*
Then, execute the command :
> atcf_mirror.sh
## Explanation
__The script `bin/atcf_mirror.sh` will launch the two scripts :__
- `bin/mirror_nhc.sh` who retrieve via LFTP the NHC data from ftp://ftp.nhc.noaa.gov/atcf/
- `bin/mirror_jtwc.py` who scrap via a web request the JTWC data from https://pzal.metoc.navy.mil/php/rds/m2m/index.php/nrlhhc/
*This way the project propose a mirror of all official ATCF tracks that is maintained up to date.*
## Work in progress
- `bin/mirror_nhc.sh` isn't complete, the archives and current storms are still separated.
- `bin/mirror_jtwc.py` is still to be completed
\ No newline at end of file
# TODO : Use the two script to update every remote data
mirror_nhc.sh
mirror_jtwc.py
\ No newline at end of file
#!/usr/bin/env python
import logging
import os
logging.basicConfig()
logger = logging.getLogger(os.path.basename(__file__))
logger.setLevel(logging.INFO)
if __name__ == "__main__":
a = 0
\ No newline at end of file
#!/bin/bash
# Notes :
# A conf file must be created on
# Example
# lftp ftp://identifiant:mot_de_passe@site_de_connexion -e "mirror -e -x dossier_ignoré -x dossier_ignoré /emplacement_distant /emplacement_local ; quit"
# FTP URL
protocol="ftp" # "sftp" for ssl
user="anonymous"
pass="pass"
host="ftp.nhc.noaa.gov"
ftpurl="$protocol://$user:$pass@$host"
# LFTP parameters
lftp_sets="set cmd:fail-exit yes;"
remote_dir="/atcf/"
# TODO : change the local path
local_dir="/home3/homedir7/perso/vsilvant/Documents/test/nhc_ftp/"
# === Download Current === #
# Mirror : Forecast
remotecd=$remote_dir"fst/"
localcd=$local_dir"fst/"
lftp -e "$lftp_sets open '$ftpurl'; lcd $localcd; cd $remotecd;
mirror --continue --only-newer --delete --verbose=3 --no-perms --parallel=8; quit;"
# Mirror : Best-Track
remotecd=$remote_dir"btk/"
localcd=$local_dir"btk/"
lftp -e "$lftp_sets open '$ftpurl'; lcd $localcd; cd $remotecd;
mirror --continue --only-newer --delete --verbose=3 --no-perms --parallel=8; quit;"
# Mirror : Fix
remotecd=$remote_dir"fix/"
localcd=$local_dir"fix/"
lftp -e "$lftp_sets open '$ftpurl'; lcd $localcd; cd $remotecd;
mirror --continue --only-newer --delete --verbose=3 --no-perms --parallel=8; quit;"
# === Download Archive === #
# Mirror the years of archives we want
remotecd=$remote_dir"archives/"
localcd=$local_dir"archives/"
includes="2020/"
lftp -e "$lftp_sets open '$ftpurl'; lcd $localcd; cd $remotecd;
mirror --exclude '.*' --exclude '.*/' --continue --only-newer --delete --verbose=3 --no-perms --parallel=8 --include $includes; quit;"
# Then, sort them between forecast and best-track
\ No newline at end of file
from setuptools import setup
import glob
setup(name='atcf_mirror',
description='Downloader of ATCF Best, Fix and Forecast Tracks from NHC and JTWC',
url='https://gitlab.ifremer.fr/cyclobs/atcf_mirror',
author = "Virgile Silvant",
author_email = "Virgile.Silvant@ifremer.fr",
license='GPL',
scripts=glob.glob('bin/**'),
use_scm_version=True,
setup_requires=['setuptools_scm'],
packages=['atcf_mirror'],
install_requires=[],
zip_safe=False
)
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment