®20827 N.W. Cornell Road #100, Hillsboro, OR 97124
Ph: +1-503-601-2099 Fax: +1-503-601-6878
Copyright © 2001 - 2004 Obvius, All rights reserved.
$Revision: 1.16 $
Contact Us
| Data Collection: |
| Home | Products | Documentation | About Obvius | Contact Us |
Now that you have installed your AcquiSuite and configured it to collect data from your meters and sensors, you will want to collect the data from your AcquiSuite. This document will provide an overview for collecting data from the AcquiSuite. Processing the data for billing reports, summaries, and other data processing are beyond the scope of this document.
Three basic ways to collect data from the AcquiSuite:
Once data has been exported from the BMO website or the AcquiSuite directly, read the Data Format Notes for information on the specific file format.
Steps for win98 command prompt ftp.
Make a directory such as c:\as_xxxxx. You should create one directory for each AcquiSuite you wish to collect data from. This will prevent one data file from overwriting one from another AcquiSuite.
Use the following dos commands.
cd as_xxxx change local dir for file storage ftp 192.168.40.50 Use the IP address of your AcquiSuite username: root password: admin The same as the admin password cd /var/log/modbus dir view a list of log files. bin request binary file transfer get mb-001.log use the file names reported in the directory listing. del mb-001.log remove the log file to free up flash space bye
Now that you have the log file on your hard drive, you can open the file using excel, or any other tool you wish. You may wish to read the faq entry that describes the log file format available from the AcquiSuite and from the BMO website.
Batch mode transfer: To transfer multiple files quickly, use the following commands in place of the "get" command in the previous example.
prom turn off interactive mode mget *.gz download all files that end in .gz from the directory.
Note: this example only downloads the compressed/rotated log files from the AcquiSuite. use "mget *" to download all files.
For automated log data transfer by ftp, we recommend the free "wget" utility. The wget command can be used in Windows for automated batch ftp transfers. The software is available at ftp://sunsite.dk/projects/wget/windows/ When downloading the software, be sure to download ssllibs.zip and wget-1.8.2b.zip (or the latest version as available) Unzip all the files from both downloads and put them all in one directory. Complete documentation and a list of alternative download sites are available at http://www.gnu.org/manual/wget/.
The following two examples show how the wget command can be used with the AcquiSuite.
wget -nc --timeout=60 -x --cut-dirs=3 -a 192.168.40.50/transfer.log -b ftp://admin:admin@192.168.40.50//var/log/modbus/*
or
wget -nc --timeout=60 -x --cut-dirs=3 ftp://admin:admin@192.168.40.50//var/log/modbus/*
(each example should read as one line)
The command options are as follows, complete details on each option are available at the wget homepage noted above.
-nc No Clobber: don't retrieve a file if a file by the same name is already on the local computer. Be careful with this feature. The .log files are rotated each day so you should re-fetch these. the .gz files are static however. It is recommended that your import/postprocessing tool delete the .log files but leave the .gz files behind.
Another solution would be to use the "-R rejlist" feature which allows you to specify a list of files that are not to be transferred. You could keep a list rather than the data files to save space. For example, after a batch transfer, unzip and process each file. If successful, append the file name to the rejlist, if not successful, delete the file and let wget re-fetch the file in the next pass. again, don't add the .log files to the list since those change each day.
--timeout=60 terminate after 60 seconds of inactivity. This is not 60 seconds of run-time, it is 60 seconds where no data was returned from the remote host. As long as data keeps flowing, wget will keep resetting the timeout even if you are only getting a few bytes per second. This will help speed things up if the network connection is completely down.
-x Create a target directory matching the url, ie \192.168.40.50\ rather than putting the files in the current directory. This will help you keep a unique subdirectory for each acquisuite.
--cut-dirs=3 Don't create 3 levels of subdirectories. In our example, this will place log files in \192.168.40.50\* rather than in the fully qualified directory \192.168.40.50\var\log\modbus\*
-a 192.168.40.50/transfer.log Append a transfer log to this file name. hint, if using multiple processes, keep separate transfer logs. This option will cause wget to terminate before fetching any files if the specified directory \192.168.40.50\ does not exist; it does not automatically create the directory. The file "transfer.log" will be created automatically in that directory if it does not exist. Delete the transfer.log file weekly or as needed. (the transfer log useful for debugging)
You should also look at the -c option, which will try to re-transfer a file if the transfer was stopped in the middle. The -t option will specify a number of retries. If you have an ethernet link it will probably be easier to retranser the entire file.
-b run in background. You should not use this option until you work out the details of your script. -b will let you run multiple concurrent processes to fetch data from many AcquiSuites at once, but there doesn't appear to be any easy way to tell when these have all completed.
ftp://admin:admin@192.168.40.50//var/log/modbus/* The url to fetch. Note the username/password on the front of the url. Also note the double // between the ip address and directory. this is required to make the directory refer to an absolute directory rather than a relative directory.
The wget utility also has the ability to process batches of urls (each AcquiSuite needs one) so you can create a file with all the URLs to fetch and then launch the wget on the entire batch. For concurrent processing, you could create two, or three files and launch a copy of wget for each url listing file. The total number of concurrent processes that you may want to run depends on the speed of your connection(s) as well as the memory and speed of your computer. In most situations, wget will be fast enough that you will not need to run more than one process at once. This greatly simplifies the operation of wget, and makes it much easier to find problems if they should happen.
The AcquiSuite has the ability to export log file data to a web browser directly from the setup web pages. To use this feature, access the AcquiSuite setup menu with your web browser. Select the "Log File Export" page from the Log File Data menu on the left side of the page.
For each device, a separate log file will be saved. Select the device from the dropdown list provided. Specify Comma or Tab delimited data, and indicate if column headers are required. Click the download button to download the data.
The log file format is the same as is provided by FTP and by the BMO web site. Note: All timestamps are in UTC.
Once you have downloaded the data from the AcquiSuite, you will need to process it. The file is an ASCII text file with comma delimited data. One line is recorded for each log cycle. The file will look something like the following example from a 4A4P-M IO Module.
'2002-01-31 18:30:00',0,0,0,-0.00,89.29,39.05,49.11,0,0,0,0
'2002-01-31 18:35:00',0,0,0x02,-0.01,104.24,39.05,49.11,0,0,0,0
'2002-01-31 18:40:00',0,0,0,NULL,87.82,39.05,49.11,0,0,0,0
'2002-01-31 18:45:00',139,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL
In this example, the first line shows a regular log entry. The second line shows a high range alarm for data point 2. The third column shows point 1 as invalid, and the fourth line shows the modbus device is not responding.
Notice the following columns are shown.
Comments:
Date/Time: The date/time column reports the time at which data was logged from the modbus device. Note: this is not necessarily the time at the start of the log cycle. Users with large number of devices may notice some of the modbus devices at higher address numbers report 1 or 2 seconds after the first modbus devices. This is because the 9600baud speed of the modbus loop is not fast enough to log all modbus devices in less than 1 second. If a device does not respond during a log cycle, the AcquiSuite will attempt to query the device several more times. If a subsequent query is successful, the time of the successful query will be used in the log file but may be several seconds after the initial log cycle.
The log date/time is in Universal time. (UTC) This means that the data is logged with no shift for daylight savings time, and must be converted to local time for reporting purposes. When importing into a database, data should be kept in UTC time for ease of use. When reporting the final data, you should adjust the time to reflect the local timezone for which you are reporting. The data export option on the BMO website will prompt you for a timezone when you export the data, and will adjust the log date/time accordingly. Note: UTC is sometimes referred to as GMT. (Greenwich Mean Time)
Modbus Error: If the error value is not 0, the remainder of the columns for this line will be reported as NULL.
Low/High Range Alarms: This number is a hex representation showing which data points are out of range. 0x01 shows data point 1 in alarm state. 0x04 shows data point 3 in alarm state.
Data Points: Data points are shown in the same order as the modbus device configuration display in the modbus/device list page. Data is displayed in floating point form. When a specific point is unavailable (4A4P-M broken wire alarm) or the point is not supported as part of the configuration (phase B current on a single phase H8163 meter hookup) the point will be logged as NULL. This preserves the column structure of the file, and allows notation of invalid data. SQL databases often accept NULL as a valid entry in a data table to represent invalid data.
For data exported from the BMO website, the columns that are invalid (NULL) are reported as blank fields. This makes it easier to import into MS Excel as blank cells. At some point in the future, the AcquiSuite will be converted to report blank fileds rather than "NULL" to make direct import of data from the AcquiSuite easier, as well as reduce the file size. Developers indending to use data files from the AcquiSuite should handle both the word "NULL" as well as a blank column as indications of an invalid data point.
| ® 20827 N.W. Cornell Road #100, Hillsboro, OR 97124 Ph: +1-503-601-2099 Fax: +1-503-601-6878 Copyright © 2001 - 2004 Obvius, All rights reserved. |
$Date: 2004/09/23 20:41:53 $ $Revision: 1.16 $ Contact Us |