Data Storage

DataTamer uses the Hilltop file format to store data. This offers compactness and high performance; a standard office computer can read data from a Hilltop file at 100,000 rows per second.  The Hilltop format puts time series data, water quality data, cross section readings and ratings in the one file.  This makes it easier to manage a big system.

Data is often collected and stored in other systems, and DataTamer can read from many commercial scada, telemetry and archiving systems.  Interfaces have been written for TimeStudio, Hydsys, Tideda, OSI PI, HydroTel, Magpie, PC208W, Citect, Datran, Intellution Fix and DHI dsf0 files.  DataTamer also has a connection method whereby a DBA can provide rules to let DataTamer read from any relational database.  The DataTamer programs connect to the various data systems directly.  There is no data import step, and the programs behave as if they were reading data from the native file format.  This transparency extends to the OLE DB Provider in Prep and means programmers can use SQL to fetch data from systems that don't offer this feature themselves.

Data Transfer

Data transfer is done via XML and this ensures the meta data travels with the actual data.  DataTamer Manager makes it easy send data from one organisation to another.  The method is to make Manager copy the required data to a temporary Hilltop file and then use a File Send command  built into Manager.  This command converts the temporary file into XML, writes this into a zip file, starts your mailing software and attaches the zip file to an e-mail.  Your sole remaining task is to name the recipient and press the Send button.  At the other end, the person receiving the e-mail can save the attached file, and then use the File Open command in Manager or Hydro to open the zip file.  There is no need to unzip the data file.

Quality Markers, Comments and Audit Trail

Quality markers can be attached to data stored in Hilltop files.  The markers are numbers that you assign meaning to as part of your office practice.  DataTamer can colour the graphs based on the quality markers, and can also draw the quality markers as an independent time series.  You can store comments as part of the data, or in a separate relational database.  The separate database is useful for occasional comments such as site visits.  Comments are highlighted as a blue box at the top a Manager graph, and you can click on the box to see the comment.  Comments stored in the data can be listed out alongside the data values.

Data files can have an audit trail associated with them.  This is a relational database that has records for all operations performed on the data.  Copies to and from the data file are recorded, as are data adds, edits and deletes.  The details of any ramp corrections made to the data are also stored in the audit trail.