OCEAN MONITORING
and FORECASTING
Providing Products and Services for all marine applications.
Technical FAQ
June 2013
- Do I need to register before downloading MyOcean products?
- How to browse the catalogue?
- How to download MyOcean products?
- Can I automate downloads of MyOcean products?
- How to write and run the script to download MyOcean products through Subset or Direct download mechanisms?
- What are the advantages of each MyOcean download mechanism?
- Is a download limited in terms of volume?
- How to download a large amount of data using Subset or Direct download mechanisms?
- Can you give a few examples of command lines to download?
- Do I need to ask for access codes each time I want to download a new product?
- I have trouble downloading data: I enter my username and password but the page refreshes without allowing access to my data. What is wrong?
- I am having trouble downloading data using Internet Explorer. What's wrong?
- Which tools can I use to manipulate the MyOcean netCDF files?
- How can I store the list of products resulting from an online product research?
- Can I get a direct access to a product and/or send its description to somebody?
- How to quickly visualize a product?
- How to get information in real time in the productions events?
- Where can I get more documents information on products?
___________________________________________________________________________________________________________
1. Do I need to register before downloading MyOcean products?
YES, you need to be registered before downloading MyOcean product.
Please find below the registration steps:
The registration form can be found in the "USER CORNER" section and in the "SERVICES" section. This registration form is a part of the MyOcean agreement named "Service Level Agreement" (SLA).
Read the MyOcean service commitments and licence
Fill in, sign and click on send button the Service Level Agreement on-line form.
If you have any problems with the on-line SLA, please use this PDF Form.
If you have any problems with the "SEND" button of the SLA (PDF form), save the form and send it by e-mail to servicedesk@myocean.eu.org.
You will receive a Login & password from the Service Desk within a maximum of 2 working days. This means that the SLA is accepted.
You can download your products (see FAQ3)
If you have sent the form and not received any response within 2 working days, please contact servicedesk@myocean.eu.org.
You are registered to use MyOcean when the service desk has given you all necessary login/password information. You can now download any MyOcean products.
2. HOW TO BROWSE THE CATALOGUE?
In order to browse the web portal catalogue, use a web browser and connect to the MyOcean web site.
Click on “Access the catalogue”, in the User Corner, on the right hand side of page.
There are three ways to enter the catalogue:
- Access full catalogue online
- Download the latest MyOcean Catalogue (PDF)
- Search for a product selecting an area, a parameter, a product type.
Select a product (with the search mode or by browsing the full catalogue)
Click on “Data Access”
Click on one of the different “Go” buttons proposed. According to product selected, MyOcean propose different solutions (See FAQ6). Select one.
Login with your login/passwd provided by the MyOcean service desk. You are redirected to the chosen product.
3. HOW TO DOWNLOAD MYOCEAN PRODUCTS?
You first need to register (see FAQ1).
MyOcean is offering 3 different authenticated Download Mechanisms:
- Subset Download (HTTP protocol, subset the files)
- Direct Download (HTTP protocol, save file as is)
- Myocean FTP (standard FTP protocol, save file as is)
See FAQ6 for more detail on each Download Mechanisms characteristics.
Login with your login/passwd provided by the MyOcean service desk. You are redirected to the chosen product.
- Subset Download
Select dataset by clicking on dataset name link.
1. Select your geographical area
2. Select your time window
3. Select parameters
4. Click on “Download” button
A message informs you about download status and a pop-up window opens asking you to confirm download.
- Direct Download
Select dataset by clicking on dataset name link.
1. Select the time range
2. Click on “Download”
A message informs you about download status and a pop-up window opens asking you to confirm download.
- MyOcean FTP Download
Select dataset by clicking on dataset name link.
A pop-up window opens asking you to confirm download.
You can also use your favorite FTP client.
4. CAN I AUTOMATE DOWNLOAD OF MYOCEAN PRODUCT?
YES , command lines can be particularly useful:
• if you want to download MyOcean data regularly and automatically (without having to connect manually to the web portal each time)
• if you want to download a large amount of data because each downloading command is volume limited
If you choose the SUBSET or DIRECT download mechanism (see FAQ3) you can build command lines using a python script. It can be launched under different environments: Windows (possibly through Cygwin), Linux and MacOS. See also FAQ5.
If you choose the MyOcean FTP download mechanism (see FAQ3), you can download products using FTP tools or on-line FTP command. You can also use script with FTP command.
5 . HOW TO WRITE AND RUN THE SCRIPT TO DOWNLOAD MYOCEAN PRODUCTS
THROUGH SUBSET OR DIRECT DOWNLOAD MECHANISMS?
The MyOcean web portal allows you generating a template command line using the python script for downloading product you are interested in. This command line integrates the extraction parameters. In order to first create your command line, you have to navigate as if you wanted to download the data through the web portal.
After selecting product in catalogue, click on “Go” in the “Data access” and log in (as shown in FAQ3). The interface will propose you all the datasets available. Select the required dataset.
The next page allows you to use selections in your download request, then download the data or generate a command line. Click on “Script” button in order to generate a command line corresponding to your request:
The generated command line is opened in a new tab of your Web Browser.
If it is the first time you are using script, you first need to download motu python client and python. Then you need to cut and paste the above command line and change local parameters: your password, the output file name, the output directory, and eventually the proxy parameters if you have to pass a proxy to access internet.
Note: if the dataset has an hourly temporal resolution, you may have to define the temporal parameter in the command line with quotes.
Example : python motu-client.py -u your_login -p your_password -m http://data.ncof.co.uk/mis-gateway-servlet/Motu -s http://purl.org/myocean/ontology/service/database#SST_GLO_SST_L4_NRT_OBSERVATIONS_010_001_a -d METOFFICE-GLO-SST-L4-NRT-OBS-SST -x -179.97500610351562 -X 20 -y 0 -Y 70 -t “2011-06-23 12:00:00” -T “2011-06-23 12:00:00” -v analysed_sst -o your_output_directory -f your_output_file_name
- How to run the command line
If it is the first time you are using download scripts, you need to download and install python, and the client script called “motu-client”.
Then you will find python here http://www.python.org/getit/. Please download python 2.7.3. Note that, due to a compatibility issue, you have to use a python version 2.x and not 3.x. You will find lastest version of the script “motu-client” here: http://sourceforge.net/projects/cls-motu/files/client/motu-client-python/.
The last version is highly recommended.
- Running command line on windows
The operating system Microsoft Windows is not the most convenient operating system to run command lines. But it is of course feasible to do so, by using DOS command, or installing Cygwin if you are familiar with unix environment.
Click on “Start” => “Accessories” => “Command line”
Go in the directory where you saved the motu client script.
cd C:your_current_directory
path_to_your_python_binarypython motu-client-pythonmotu-client.py -u XXX -p XXX -m http://misgw-qo-externe.cls.fr/misgw-qo-servlet/Motu -s http://purl.org/myocean/ontology/service/database#misgw-qo-thr -d LR_MOD -x -180.0 -X 180 -y -89 -Y 90 -t 2011-01-02 -T 2011-01-02 -z 0 -Z 0 -v salin-ity temperature -o . -f test.nc
- Running command line on Linux or MacOS
These operating systems are well adapted to run command line. Just open a terminal, go to the directory where you saved the motu client script and run the command.
For any questions and further assistance you can contact servicedesk@myocean.eu.org.
6 . WHAT ARE THE ADVANTAGES OF EACH MYOCEAN DOWNLOAD MECHANISM?
MyOcean is offering 3 different authenticated Download Mechanisms:
- Subset Download (HTTP protocol, subset the files)
- Direct Download (HTTP protocol, save file as is)
- MyOcean FTP (standard FTP protocol, save file as is)
For each product, at least one of the three above download mechanism is offered.
For all of them, the user can authenticate with his MyOcean login and password. For the 2 first ones, the authentication is totally secured (https protocol). All download mechanisms are presented to the user on the MyOcean Web Catalogue of products. The user can also access the download mechanism through a web browser, and through command line.
For a given product, how to recognize which Download Mechanism is available?
Once in the catalogue, click on “DATA ACCESS” link, you will be presented up to 3 “GO” buttons on top of which an introductive sentence will tell you which Download Mechanism it connects to. They can be one of the following:
- Subset Download : The “Subset download” allows you to subset the data, and should be used if you wish to download a small part of the total dataset.
- Direct Download : The “Direct download” directly transfers the file as stored on the server. It is able to deliver compressed data and should be used if you wish to download larger datasets.
- MyOcean FTP : The “FTP download” allows to download an entire dataset directly from the FTP server at the supply center.
What are the advantages of each Download Mechanism?
SUBSET DOWNLOAD: User can retrieve a subset of gridded datasets through http and https protocol. The main idea is: download exactly what user needs. User can retrieve the needed variable, the geospatial coverage, the temporal coverage. There are two ways to access this service: via GUI (through the web portal), or via a machine to machine interface (script). The advantages of this service are:
user minimizes the volume of data transiting on the network, and the volume of data to be stored on his computer.
user selects the different parameters of his request in a “easy to understand” way: variable can be selected via their names or standard names (from CF convention standard name table), date format is harmonized at MyOcean level (ex: 2013-02-22).
Information on dataset (notably the updated temporal coverage) and on the request (volume of the request), can be provided to user before running the request (via GUI or via script).
Security of the authentication (https, CAS authentication with a ticket).
DIRECT DOWNLOAD: User can retrieve data files by selecting a temporal coverage through http and https protocol. There are two ways to access this service: via GUI (through the web portal), or via a machine to machine interface (script). The advantages of this service are:
Download at once the entire geospatial coverage and all variable(s) of dataset.
Select temporal coverage in a “easy to understand” way: date format is harmonized at MyOcean level (ex: 2013-02-22).
Information on dataset (and notably the updated temporal coverage) and on the download request (notably the volume of the request), can be provided to user before running the request (via GUI or via script).
Security of the authentication (https, CAS authentication with ticket).
MYOCEAN FTP DOWNLOAD: The user can retrieve data files by selecting files in a directory through FTP protocol. There are three ways to access this service: via a web browser, via an FTP client with GUI (user shall install this client on his computer before), or via a machine to machine interface. The advantages of this service are:
Download at once the entire geospatial coverage and all variable(s) of dataset.
Performances of the server depend only on disk access performances and bandwidth.
FTP protocol is well known among users.
7. IS A DOWNLOAD LIMITED IN TERMS OF VOLUME?
Volume limitations for a single request have been set up on Subset and Direct Download mechanisms in order to guaranty good performances. The maximum amount of data you can download is 1 Gigabyte for a single request. If your single request is beyond this size, an error message will be immediately displayed: "The size of the data to download - ... Megabytes- exceeds the maximum allowed - 1024 Megabytes". You can also encounter a "proxy error 502". This error message appears when a request lasts too long.
In order to overcome the limited volume associated to a single request, you need to use multiple requests (i.e.: a loop within a script command line) hence splitting the data volume into smaller pieces at a time. For example in your script command line, you can change the temporal extraction windows.
FTP download does not offer this limitation, however the way of downloading data (no subset / full data download) can generate large volumes of data transfer that has impacts on bandwidth and significantly increase overall download time.
8. HOW TO DOWNLOAD A LARGE AMOUNT OF DATA?
One download request results into one file, so downloading a very large amount of data (i.e. more than 1Gb) with a single request is not allowed. Moreover, it is not recommended to manipulate files larger than 1Gb, for usability purpose.
Downloading a very large amount of data, especially for downloading large time coverage (like 10 days or 20 years) is possible through the MyOcean download service : The simplest way is to perform a loop on dates. You need to split your download requests into several download requests.
For example, below you can find a loop coded in a simple Unix/Linux shell script.
# Definition of the dates
DATES=(2011-01-01 2011-01-02 2011-01-03 2011-01-04 2011-01-05 2011-01-06)
# Loop on the dates
for (( i = 0 ; i < ${#DATES[@]} ; i++ ))
do
#Download request (should be adapted)
python motu-client-python/motu-client.py -u XXX -p XXX -m http://misgw-qo-externe.cls.fr/misgw-qo-servlet/Motu -s http://purl.org/myocean/ontology/service/database#misgw-qo-thr -d LR_MOD -x -180.0 -X 180 -y -89 -Y 90 -t ${DATES[$i]} -T ${DATES[$i]} -z 0 -Z 0 -v salinity temperature -o ./ -f test.nc
done
The array DATES shall be adapted to each situation (time coverage of the dataset and your own need). Of course, depending on the way you want to code the loop (python, perl, java…), you will have to adapt the code. The line in blue is a command line to download one time step of data.
9. CAN YOU GIVE A FEW EXAMPLES OF COMMAND LINES TO DOWNLOAD?
- Ocean Colour Product
python motu-client-python/motu-client.py -u XXX -p XXX -m http://myocean.artov.isac.cnr.it/mis-gateway-servlet/Motu -s http://purl.org/myocean/ontology/service/database/OCEANCOLOUR_MED_CHL_SEAWIFS_L3_RAN_OBSERVATIONS_009_025_a-TDS -d OCEANCOLOUR_MED_CHL_SEAWIFS_L3_RAN_OBSERVATIONS_009_025_a -o . -t 2009-12-04 -T 2009-12-04 --out-name test_color.nc
python motu-client-python/motu-client.py -u XXX -p XXX -m http://myocean.acri.fr/mis-gateway-servlet/Motu? -s http://purl.org/myocean/ontology/service/database/OCEANCOLOUR_ARC_KD490_MERIS_MODIS_L _KD490_MERIS_MODIS_L3_L4_NRT_OBSERVATIONS_009_010_m -d dataset-oc-acri-arctic-1km-kd490-daily-nrt -o ./ -t 2010-09-03 -T 2010-09-03 -y 0 -Y 68 -x 0 -X 100 -z 0 -Z 50 -v volume_attenuation_coefficient_of_downwelling_radiative_flux_in_sea_water --proxy-server http://proxyURL:port --out-name test_color.nc
- Global analysis and forecast product
python motu-client-python/motu-client.py -u XXX -p XXX -m http://atoll.mercator-ocean.fr/mfcglo-mercator-gateway-servlet/Motu? -s http://purl.org/myocean/ontology/service/database#GLOBAL_ANALYSIS_FORECAST_PHYS_001_002-TDS -d global-analysis-forecast-phys-001-002 -o ./ -t 2013-01-21 -T 2013-01-22 -y 0 -Y 68 -x 0 -X 100 -z 0 -Z 50 -v sea_water_salinity --proxy-server http://proxyURL:port --out-name test_mercator.nc
- Mediterranean Sea product
An example of shell script with a loop:
DATES=(2005-01-01 2005-01-02) #etc...
# Loop on the dates
for (( i = 0 ; i < ${#DATES[@]} ; i++ ))
do
#Download request
python motu-client-python/motu-client.py -u XXX -p XXX -m
http://gnoodap.bo.ingv.it:4080/mis-gateway-servlet/Motu? -s
done
- Sea Level product
Please note that concerning the Sea Level products, you receive a zip file containing some netCDF files, and not the netCDF file directly.
10. DO I NEED TO ASK FOR ACCESS CODES EACH TIME I WANT TO DOWNLOAD A NEW
PRODUCT?
NO
With your login/password delivered by Service Desk when you first registered, you can access all products from the online catalogue of products. However, few products might remain unavailable, in which case you need to contact the service desk by email.
11. I HAVE TROUBLE DOWNLOADING DATA: I ENTER MY USERNAME AND PASSWORD BUT
THE PAGE REFRESHES WITHOUT ALLOWING ACCESS TO MY DATA. WHAT'S WRONG?
This might be due to your computers security settings: the direct download service requires you to accept cookies from a third party website (http://coremis-cas-int.cls.fr/cas/).
The easiest way to enable this is to access your browsers security settings and select ‘accept third party cookies’.
Note that the default security settings do not allow downloading from third party cookies, which is needed to access MyOcean data through the web portal. In this situation a small icon appears on the status bar to inform the user that a cookie has been rejected.
Decreasing the security settings will prevent this issue. Hence we recommend:
1. Select ‘Tools’ from the top menu bar
2. Select ‘Internet Options’
3. Select the ‘Security’ tab
4. Move the slider all the way down to ‘accept all cookies’.
In some cases this may not be possible, for example if internet security settings are centrally controlled.
Please be aware that the MyOcean team is working on better solutions for this issue.
12. I AM HAVING TROUBLE WITH INTERNET EXPLORER. WHAT'S WRONG?
The MyOcean Web Portal shows a bug for Internet Explorer with version lower to 7, which makes it impossible to see the web catalogue, and that has nothing to do with cookies. Note that MyOcean WebPortal is only compatible with Internet Explorer version 7 or higher.
Please accept our apologies and be aware that the MyOcean team is working on this issue.
We recommend accessing the MyOcean WebPortal using other browsers than Internet Explorer (Such as Mozilla Firefox or Google Chrome).
13. WHICH TOOLS CAN I USE TO MANIPULATE THE MYOCEAN NETCDF FILES?
- A- DEFINITION: what is Netcdf?
http://www.unidata.ucar.edu/software/netcdf/
- B- UNCOMPRESS
(non exhaustive list)
Bzip2 decompressor (windows: http://www.7-zip.org/) or Bzip2 project pages at http://www.bzip.org/
- C- VIEW
(non exhaustive list of visualization tools, i.e. easy to install)
Ncview (http://meteora.ucsd.edu/~pierce/ncview_home_page.html) : Simple visualization and analysis (easy to install on unix/linux platforms, possible on windows)
Ferret (http://ferret.wrc.noaa.gov/Ferret/) Advanced visualization and analysis (you need to develop your own program)
Matplotlib, python library (http://matplotlib.sourceforge.net/)
ncl : http://www.ncl.ucar.edu
idv : http://www.unidata.ucar.edu/software/idv (Advanced view with 3D animation)
- D- EXTRACT/MANIPULATE (Concatenation/Computing the mean/Add variable and attributs ....)
(non exhaustive list)
List of the NetCDF software packages, http://www.unidata.ucar.edu/software/netcdf/software.html
NetCDF software package, including libraries for programming languages such as Fortran and C++, can be freely downloaded, please see: http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#howtoget. This package is the best way for performing detailed processing of NetCDF files, including conversion to ASCII or other file formats.
A powerful set of tools (for Extraction/Concatenation/Computing the mean/Add variable and attributes) for working with NetCDF files are the NetCDF operators (NCO), which can be freely obtained from http://nco.sourceforge.net/
cdo: https://code.zmaw.de/projects/cdo
Marine Geospatial Ecology Tools (http://code.env.duke.edu/projects/mget), a free add-on to ArcGIS, that allows converting formats such as NetCDF to ASCII and ArcGrid.
- E- CONVERT (from Netcdf to GRIB for example)
ncl: http://www.ncl.ucar.edu
14. HOW CAN I STORE THE LIST OF PRODUCTS RESULTING FROM AN ONLINE PRODUCT
RESEARCH?
There are three ways to enter the catalogue. If you perform a search for a product using criteria or by using keywords you get a list of products corresponding to the result of your request.
It is possible to store the results of your web search. In order to do so, once search performed and results of search displayed, you can bookmark or copy/paste the URL of the page appearing on top of you web browser.
In order to access this same list later, you will be able to use the markup or copy/paste the saved URL into your web browser. This will directly allow displaying the same list of products resulting from your initial search.
15. CAN I GET A DIRECT ACCESS TO A PRODUCT AND/OR SEND ITS DESCRIPTION TO
SOMEBODY?
Each product in the online catalogue of products has its own individual URL than allow a direct access to a product sheet. This allow you to target directly the needed product when browsing through the catalogue, save this address for further reference or send it to somebody else.
16. HOW TO QUICKLY VISUALIZE A PRODUCT?
In less than ten seconds, you can quickly visualize a product using the on-line Viewer. We illustrate here the viewing for BLACKSEA-ANALYSIS-FORECAST-PHYS-007-001 product.
On the Web Portal, in the 'User Corner' section, click on 'Access the Catalogue' and search the BLACKSEA-ANALYSIS-FORECAST-PHYS-007-001 product information.
Click on the VIEW link to start the product VIEWER. A pop-up window opens.
The Viewer interface allows you to:
- Zoom on the map by double clicking.
- Move on the map center by holding down the left mouse button and moving the mouse.
- Update the map color and opacity using the right color bar buttons
- Change the parameter, the date and the depth level by using the top-level buttons.
17. HOW TO GET INFORMATION IN REAL TIME ON THE PRODUCTION EVENTS?
You can get real-time information on the production events which can have an impact on your activities (product availability issues, quality issues, services issues) on the web portal, on the News Flash pages.
On the web portal, in the User Corner, select the News Flash link. You are redirected to the News/Flash information. Information is provided on the form of a historical table summarizing all events with the recording date, the subject and the status.
You can search information by selecting “Area”, “Parameters” and “Type” criterions on the table header.
You can get detailed information for each event by clicking on the subject.
You can also subscribe to RSS feeds by clicking on the RSS link. It will allow you to visualize all event information from your web browser.
You can also get event information by email by asking the Service Desk by email.
18. WHERE CAN I GET MORE DOCUMENTS INFORMATION ON PRODUCTS?
You can find more information on products on-line with downloading Product User Manual and Quality Manual documents.
On the Web Portal, in the User Corner section, click on ‘Access the Catalogue’ link and select your product of interest.
You get the the product information summary pages. To get more information on the product, select the INFO button.
The Product User Manuals and Quality Information Documents:
Documentation on products, i.e. Product User Manual and Quality Documents (PUM and QUID) are available by clicking on 'Documentation' tab in the on-line product info page.
