Commit 4e00d38a authored by PONCELET's avatar PONCELET

update XSF Documentation template

parent 516c2ba8
== The XSF convention
#
NetCDF4 is a data model, application programming interface (API) library, and file
format for storing and managing data, developed and maintained by Unidata. Unidata
is part of the US University Corporation for Atmospheric Research (UCAR) Community
......@@ -21,7 +20,7 @@ The XSF format focuses on describing the acoustic data of bathymetry sounder, ie
# Hierachical Structure
=== Hierachical Structure
NetCDF4 has two main organizational concepts: (i) the variable, which can contain a
variety of data structures, and (ii) groups, being a collection of variables. Groups and
......@@ -46,7 +45,7 @@ the data. Depending on the context and in order to inherit from dimension defini
These groups contain variables and variable attributes with prescribed names and
contents.
## Obligations and missing data
=== Obligations and missing data
Some variables and attributes in SONAR-netCD4 are mandatory; these form the minimal set of data required to quantitatively use backscattering amplitude data. The remaining variables and attributes have various levels of optionality and provide enhanced context and information about the sonar data. The obligations are:
|===
|M| mandatory
......@@ -61,37 +60,36 @@ The set of mandatory variables and attributes has been chosen so that classical
The _FillValue attribute should be used to indicate missing data in variables. For floating point values, the IEEE 754 not-a-number (NaN) is the preferred fill value as this is convenient for commonly-used analysis packages (e.g. Python, Matlab, R). By default for float and double values NaN is used as a _FillValue.
# Metadata and authorities
=== Metadata and authorities
As possible the SONAR-netcdf naming and convention have been reused for this document. Where relevant, attribute and variable names have
been reused from this convention.
The NetCDF Climate and Forecast (CF) Metadata Conventions (Eaton et al., 2017) have been used where sensible (the efficient storage of unprocessed active sonar data has not been a design goal of the CF convention) along with the ESIP Attribute Convention for Data Discovery (ACDD). Terms and concepts from other metadata conventions have also been used where appropriate. The NOAA NCEI NetCDF Templates v2.0 has been used whenever they were applicable. For bathymetry, most of the namings are inspired by the Kongsberg Kmall format document specification.
# Units
=== Units
All relevant variables and attributes are required to have a textual netCDF4 attribute with the name “units” that specifies the units.
The International
System of Units (SI) is used. For simplicity, the data format mandates the use of particular units and their textual form, as per the definitions and conventions of the UDUNITS-2 package (UCAR, 2014).
Decibels are commonly used in underwater acoustic quantities, but UDUNITS does not currently include such a unit. However, as the CF convention does permit decibel (with symbols: “dB”, “db”, and “dbel”), it is used in the SONAR-netCDF4 convention.
# Datatypes
=== Datatypes
Each item has a suggested datatype, chosen to have sufficient range and precision to contain the expected data. Alternative datatypes can be used if necessary, but are discouraged.
The “string” type should contain text in the UTF-8 encoding and should be
treated as case-sensitive during any comparisons. Enumerated datatypes are used for some of the controlled vocabularies.
# Main differences with SONAR-netcdf4
=== Main differences with SONAR-netcdf4
The XSF format definition try to be as closed as possible to the SONAR-netcdf4. But due to inherent diffences between bathymetry multibeam echo sounder some diffences appears between the two formats. This chapter describes the main diffences between the two formats.
## Platform definition
==== Platform definition
Bathymetry Echo sounder are made of several antennas located at different position on the vessel. They are usually made up with one emitting antenna (Tx) and one or several receiving antennas (Rx). Therefore descriptions and links for those antenna where added in the file format.
## Sensor definition
==== Sensor definition
Usually a vessel can contains several sensors linked to the platform, each sensor kind was added to a subgroup in the Platform group. In case of several sensor of the same kind (for exemple several position sensors), subgroup are created with their respective id.
## Technical choice for variable lenght attribute
==== Technical choice for variable lenght attribute
The technical choice of variable length attribute for storage if sample data is a very elegant choice in the netcdf convention but it has very poor performances with respect to ragged array attribute storage. This is why, especially for water column samples, the choice was made to use instead ragged array and associate them with an ancillary variable indicating the variable count. This is one of the main discripency between the SONAR-netcdf format and XSF format.
# File-naming convention
==== File-naming convention
SONAR-netCDF4 files should always end with a “.xsf” suffix to indicate that they are a XSF netCDF file. As for SONAR-netcdf convention it is recommended that the filename should sort alphanumerically into chronological order (e.g. date and time of the first ping in the file; thus: YYYYMMDDHHMMSS.nc). This facilitates file management and use in analysis systems
# Versioning
= Coordinate System Definiton
== Coordinate System Definiton
:imagesdir: images
:stem: latexmath
:toc:
Describe the different coordinates systems and convention used into the XSF file format definition. For definitions are inspired from the Kongsberg Kmall format definition revision -e.
# Position coordinates Systems
## Vessel Coordinate System (VCS)
=== Position coordinates Systems
==== Vessel Coordinate System (VCS)
Origin of the VCS is the vessel reference point. The VCS is defined according to the right hand rule
image:RightHandCRS.png[]
......@@ -20,31 +20,31 @@ Pitch - rotation around the y-axis.
Heading - rotation around the z-axis. Heading as input in depth calculations is sensor data referred to true north.
## Array Coordinate System (ACS)
==== Array Coordinate System (ACS)
Origin of the ACS is at the centre of the array face. The ACS is defined according to the right hand rule.
x-axis pointing forward along the array.
y-axis pointing starboard along the array plane.
z-axis pointing down orthogonal to the array plane.
## Surface Coordinate System (SCS)
==== Surface Coordinate System (SCS)
Origin of the SCS is the vessel reference point at the time of transmission. The SCS is defined according to the right hand rule.
x-axis pointing forward along the horizontal projection of the vessel main axis.
y-axis pointing horizontally to starboard, orthogonal to the horizontal projection of the vessel main axis.
z-axis pointing down along the g-vector.
To move SCS into the waterline, use reference point height corrected for roll and pitch at the time of transmission.
## Fixed Coordinate System (FCS)
==== Fixed Coordinate System (FCS)
Origin of the FCS is fixed somewhere in the nominal sea surface. The FCS is defined according to the right hand rule.
x-axis pointing north.
y-axis pointing east.
z-axis pointing down along the g-vector.
## Overview of Coordinates System
==== Overview of Coordinates System
image::CRSOverviews.png[]
## Coordinates system transformations
=== Coordinates system transformations
### VCS to SCS
==== VCS to SCS
In VCS csr, heading is not take into account, this system remaining in the vessel axis
Given latexmath:[ X_{vcs}] position coordinates of X point in vcs referential
......@@ -96,7 +96,7 @@ X_{vcs}
====
### SCS to VCS
==== SCS to VCS
[NOTE]
====
......@@ -122,7 +122,7 @@ X_{scs}
]
====
### ACS to VCS
==== ACS to VCS
Given latexmath:[ X_{vcs}] position coordinates of X point in vcs referential
......@@ -175,7 +175,7 @@ MRU_{installation offset} \\
]
====
### VSC to ACS
==== VSC to ACS
Given latexmath:[ X_{vcs}] position coordinates of X point in vcs referential
Given latexmath:[ X_{acs}] position coordinates of X point in acs referential
......@@ -198,7 +198,7 @@ MRU_{installation offset} \\
====
### ACS to SCS
==== ACS to SCS
Given latexmath:[ X_{scs}] position coordinates of X point in scs referential
......@@ -228,7 +228,7 @@ MRU_{installation offset} \\
====
### SCS to FCS
==== SCS to FCS
Taking into account tide and dynamic draught is done through a projection in FCS referential
......@@ -270,25 +270,25 @@ X_{scs} \\
====
# Angle Coordinate Systems
=== Angle Coordinate Systems
Usually in acoustics sounder indication about angle are given by the constructors. When defined an angle value should refers to which axis they are refered.
One can distinguished several Angle Coordinate System allowing to which axis angle are referred to
## FACS (Fixed Angle Coordinate System)
==== FACS (Fixed Angle Coordinate System)
Angles are expressed according to a SCS coordinate system, having it origin set at the center of the considered transducer. Angle is positive in the pitch direction and mesured according to Z axis of FCS system (i.e following the gravity direction)
## TACS (Transducer Angle Coordinate System)
==== TACS (Transducer Angle Coordinate System)
Angles are referred to a coordinate system having its having it origin set at the center of the considered transducer, angle is positive in the positive roll direction and mesured according to a vector normal to the transducer face.
## VACS (Vessel Angle Coordinate System)
==== VACS (Vessel Angle Coordinate System)
Angles are referred to a coordinate system having its having it origin set at the center of the considered transducer, angle is positive in the positive roll direction and mesured according to the Z axis of the VSC referential.
## Angle Coordinates system transformations
=== Angle Coordinates system transformations
Installation angles being small enough, we can deduced the following formulas
### Rx antennas
==== Rx antennas
Given latexmath:[RxAngle_{tacs}] an accross angle in TACS coordinate system
......@@ -298,16 +298,16 @@ Given latexmath:[RxAngle_{facs}] an accross angle in FACS coordinate system
Given latexmath:[RxTransducer_{installation roll offset}] the installation offset of the transducer in roll
#### TACS to VACS
===== TACS to VACS
latexmath:[RxAngle_{vacs} = RxAngle_{tacs} + RxTransducer_{installation roll offset} ]
#### VACS to FACS
===== VACS to FACS
latexmath:[RxAngle_{facs} = RxAngle_{vacs} + Roll(t)]
### Tx Antennas
==== Tx Antennas
Given latexmath:[TxAngle_{tacs}] an along angle in TACS coordinate system
Given latexmath:[TxAngle_{vacs}] an along angle in VACS coordinate system
......@@ -316,14 +316,14 @@ Given latexmath:[TxAngle_{facs}] an along angle in FACS coordinate system
Given latexmath:[RxTransducer_{installation pitch offset}] the installation offset of the transducer in pitch
#### TACS to VACS
===== TACS to VACS
latexmath:[TxAngle_{vacs} = TxAngle_{tacs} + TxTransducer_{installation pitch offset} ]
#### VACS to FACS
===== VACS to FACS
latexmath:[TxAngle_{facs} = TxAngle_{vacs} + Pitch(t)]
# Time coordinates systems
=== Time coordinates systems
For bathymetry data, time stamp for multibeam datagrams are the time of the midpoint of the first transmitted pulse of the ping. All delays are referred to this time if not specifically stated
## root (/root)
=== root (/root)
root
......@@ -27,7 +27,7 @@ root
|===
### environment (/root/environment)
=== environment (/root/environment)
......@@ -56,7 +56,7 @@ root
|===
#### sound_speed_profile (/root/environment/sound_speed_profile)
=== sound_speed_profile (/root/environment/sound_speed_profile)
......@@ -136,7 +136,7 @@ root
|===
#### surface_sound_speed (/root/environment/surface_sound_speed)
=== surface_sound_speed (/root/environment/surface_sound_speed)
......@@ -165,7 +165,7 @@ root
|===
#### tide (/root/environment/tide)
=== tide (/root/environment/tide)
......@@ -193,7 +193,7 @@ root
|===
#### dynamic_draught (/root/environment/dynamic_draught)
=== dynamic_draught (/root/environment/dynamic_draught)
......@@ -220,7 +220,7 @@ root
|===
### platform (/root/platform)
=== platform (/root/platform)
......@@ -338,7 +338,7 @@ root
|===
#### position (/root/platform/position)
=== position (/root/platform/position)
a group by sensor ID
......@@ -349,7 +349,7 @@ a group by sensor ID
|*Variables*||
|===
##### position_sub_group (/root/platform/position/position_sub_group)
=== position_sub_group (/root/platform/position/position_sub_group)
......@@ -415,7 +415,7 @@ a group by sensor ID
|===
#### attitude (/root/platform/attitude)
=== attitude (/root/platform/attitude)
a group by sensor ID
......@@ -426,7 +426,7 @@ a group by sensor ID
|*Variables*||
|===
##### attitude_sub_group (/root/platform/attitude/attitude_sub_group)
=== attitude_sub_group (/root/platform/attitude/attitude_sub_group)
......@@ -496,7 +496,7 @@ a group by sensor ID
|===
### provenance (/root/provenance)
=== provenance (/root/provenance)
......@@ -517,7 +517,7 @@ a group by sensor ID
|===
### sonar (/root/sonar)
=== sonar (/root/sonar)
......@@ -541,7 +541,7 @@ a group by sensor ID
|*Variables*||
|===
#### beam (/root/sonar/beam)
=== beam (/root/sonar/beam)
Example of a beam group. Include as many subgroups as necessary for different beam groups. Use unique group names, preferably of the form Beam_groupX where X is an integer
......@@ -552,7 +552,7 @@ Example of a beam group. Include as many subgroups as necessary for different be
|*Variables*||
|===
##### Beam_group1 (/root/sonar/beam/Beam_group1)
=== Beam_group1 (/root/sonar/beam/Beam_group1)
......@@ -875,7 +875,7 @@ Example of a beam group. Include as many subgroups as necessary for different be
|===
###### Vendor_specific (/root/sonar/beam/Beam_group1/Vendor_specific)
=== Vendor_specific (/root/sonar/beam/Beam_group1/Vendor_specific)
The vendor specific group contains information about the sonar and data specific to the sonar. It is described as a subgroup of beam group in order to inherit from its defined dimensions
......
ifdef::env-github,env-browser[:outfilesuffix: .adoc]
# Introduction
== Introduction
Swath surveying systems have become the predominant tool for collection of high‐resolution bathymetry and imagery data.
The systems leaded to a variety of constructor data proprietary formats. With the increasing sophistication of these sonars has resulted an expand in the data stream and a significant amount of effort being spent determining which version of a particular sonar was used to collect a given set of data and to maintain library able to read it.
......@@ -8,7 +8,7 @@ The systems leaded to a variety of constructor data proprietary formats. With th
The XSF format definition is a way to convert them in a usable fashion, standardized, self-described, easy to connect to technical computing environment (java, c++, python, R , Matlab) .
## Background
=== Background
Many purpose-built file formats exist to store and exchange data from scientific and
industrial equipment. Formats have been created by sonar and echosounder manufacturers;
in addition, more generic acoustic data formats such as the Generic Water
......@@ -26,8 +26,9 @@ The SONAR-netCDF4 format presents a convention for the storage and exchange of f
IMPORTANT: Given the work accomplished by the working group and the matching between their storage requirement, technical solutions and the ones we had for multibeam storage a decision was made to base as much as possible the XSF format on SONAR-netcdf4. As a consequence, much of the naming, technical choices, group organisation are inherited from this format.
# Versioning
=== Versioning
This document and the convention that it describes will change over time to implement enhancements and to correct errors and omissions. To accommodate this, the convention always requires a version number. This document has a separate version number, allowing for revised versions of the document to describe
the same version of the convention.
The convention version number will always be included in the title of the document.
The document version number will always be found in <<Revisionhistory.adoc#Revision History, Revision History>>.
\ No newline at end of file
The document version number will always be found in <<_revision_history, Revision History>>.
# Revision History
== Revision History
|===
|Document Revision | XSF Version | Date | Changes
......
# XSF
= XSF Format Definition
:toc:
:toc: left
:toclevels: 4
:imagesdir: content
:docinfo: shared
_This document is the entry point for the XSF format definition_
_This document is the XSF format definition document_
== Index of XSF Format definition
include::RevisionHistory.adoc[RevisionHistory]
. link:Introduction.adoc[introduction]
. link:Convention.adoc[convention]
. link:Coordinates.adoc[coordinates system]
. link:Groups.adoc[groups definition]
include::Introduction.adoc[introduction]
include::Convention.adoc[convention]
include::Coordinates.adoc[coordinates system]
== Groups definition
include::Groups.adoc[groups definition]
......@@ -55,7 +55,7 @@ public class AsciiDoctorGenerator {
try(FileWriter fileOut = new FileWriter(output))
{
context.out = fileOut;
writeGroup("##","", root, context);
writeGroup("===","", root, context);
}
}
......@@ -79,7 +79,7 @@ public class AsciiDoctorGenerator {
writeVariables( group.getVariables(), context);
endTable(context);
for (Group subgroup : group.getSubGroups()) {
writeGroup(level+"#",path+"/"+ group.getName(), subgroup, context);
writeGroup(level,path+"/"+ group.getName(), subgroup, context);
}
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment