Visit our Popular Forums

rgleason · 21-05-2013, 13:51

Thomas, for a moment there I thought the currents were negative rather than positive, but no longer think that. But I think it is very likely something like that because as far as I can tell the weather_routing looks pretty good, but hasn't really been tested that much, so we'll be learning things.

People are just getting used to having these wonderful tools, which we have to thank Sean for!

rgleason · 21-05-2013, 15:18

Here are two Grib Weather_Routings using the same 8 day Noaa Saildocs grib file wiithout currents.

The first screenshot is using QtVLM and a hand recorded/estimated polar for Bristol 32 that is faster, 7 day trip. Uses a straight line route.

The next two screenshots are using Opencpn Grib & Weather_Routing plugins, with an adjusted integral Polar that is set slower (perhaps too slow) in this run and it results in a 15 day trip. The first screenshot shows as many of the settings as possible, and the second one has a partial of the Route Manager screen. Results in a dogleg route much like what the climatology plugin provided.

seandepagnier · 21-05-2013, 22:16

Quote:

Originally Posted by cagney

I had to test the Atlantic Route between Antigua and Gran Is he plugin using winds 180 degrees out? Or is the polars turned upside down?
Whatever....., the result is badly wrong!

bug which is now fixed.

Now I add automatic generation of world cruising routes eventually with generated commentary specific for your boat and requirements.

Quote:

Originally Posted by ivica

just reply on my part of the post...
depending on resolution you want, i.e. I have 2km resolution so it is not the same getting whole domain every 3hours or not. Moreover if you are on the boat with mobile connection.
My model data, and majority of other models are not compressed netCDF, i.e. version 4, but could be say bz2 which is again bunzip2ed for you at the server side, not locally. So at the end you do not notice this, it's done behind. Possibly you can save "scaled netcdf", with short format variables and use offset, scaling factor, there are many possibilities to reduce size. Compare double precision with signed(or even unsigned) integer (say pressure, temp, wind are ideal for that).
Cheers,
Ivica

I already am packing to 1 or 2 byte integers with scale and offset instead of floating point. This also allows much better compression to work than many close but not identical floating point values, when you drop the noise out.

Netcdf is not really an optimal format because of limitations, like fixed size per item. Sometimes I maybe use less bytes to store some wind data on the grid than others.

As far as decompressing on the server side.. that is not so good because we want to download compressed to have smaller file transfer. It should be possible to break the world data down into smaller slices, but also keep in mind I didn't use a fixed size per element, so computing the index is not so simple. If I did, I would have a much larger file size.

rgleason · 22-05-2013, 02:24

Sean should I git in Linux? Does data need recompile?

cagney · 22-05-2013, 10:06

Quote:

Originally Posted by boat_alexandra

bug which is now fixed.

......

Yeah, OK now. But to test I had to decompress the data files as the compressed files are no longer recognized. Intentional or a mistake?

Thanks for all your great new plugins and "Bon Voyage"!

Thomas

rgleason · 22-05-2013, 10:47

Thomas,
Yes, the 'current' and 'wind' files have to be uncompressed into 12 separate gz files if you have downloaded the single data file. 12 wind and 12 current, plus some others.

Note to others:

Sean has added several data types to the Climatology_pi plugin, (precipitation, temp, etc) therefore we will have to update the data files along with the plugin files.
Pavel advises that the data processing and compilation cannot be done in Windows, so Linux is the only way to do it.
This is being done now, with complete redo of Wind and Currents, because Sean said he had made the processing more efficient and smaller.
Hopefully the pesky window bugs will also disappear and this plugin will compile ok.

(I do hope Sean has not sailed yet, Climatology_pi is close to working in Windows, and windows gets very picky, while it runs fine in Linux. I can readily understand why it feels like you're trying to tie shoelaces using a 40' pole, when trying to debug windows through a third party, from a linux platform. -Frustrating for everyone. Wish I was confident/skilled enough to make my own changes.)

cagney · 22-05-2013, 11:02

Rick

A .gz file is of course compressed, and presently not recognized.
From the log.

Quote:

17:54:42 CEST: climatology_pi: failed to read file: /usr/local/share/opencpn/plugins/climatology/data/wind04
17:54:42 CEST: climatology_pi: failed to read file: /usr/local/share/opencpn/plugins/climatology/data/wind05
17:54:42 CEST: climatology_pi: failed to read file: /usr/local/share/opencpn/plugins/climatology/data/wind06

As you can see O is looking for wind04, not wind04.gz.
I decompressed wind04.gz etc, and then the plugin worked, using git as of a couple of hours ago.

Thomas

rgleason · 22-05-2013, 11:17

Okay, I think I know what you are saying. There are several steps to all this, some of it in the background to most people (to create a makefile, add paths and add other files, and finally process the data with "make -k" from the gendata directory).

I think you are saying that the packaged data files have to be uncompressed for the plugin to read them properly..... that is a change. Now I think I recall reading a note from Sean to that effect, but I can't find it... are you sure you have the libraries needed? -Yes now I see, it is from the LOG!

I'm currently processing the 180gb of data and it seems to read gz files OK.

Quote:

/media/sf_Data-Dart/Climate-Winds-6hr/uv19880710.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880711.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880712.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880713.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880714.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880715.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880716.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880717.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880718.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880719.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880720.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880721.be.gz
reading file: /media/sf_Data-Dart/Climate-Winds-6hr/uv19880722.be.gz

have created up to Wind07.gz now.

seandepagnier · 22-05-2013, 16:19

Quote:

Originally Posted by cagney

Rick

A .gz file is of course compressed, and presently not recognized.
From the log.

As you can see O is looking for wind04, not wind04.gz.
I decompressed wind04.gz etc, and then the plugin worked, using git as of a couple of hours ago.

Thomas

I think I made a mistake and broke it. Should be fixed now.

I have a slight dilemma. gz is not the smallest compression, so is not ideal for internet transfers, and 7z or bz2 takes so long to decompress it is too slow to load the plugin.

I thought maybe it's ok to use 7z and just to store the files locally uncompressed which causes more disk space usage, but this is way less than a set of charts. But then gz hardly takes any time at all to decompress and has huge savings..

If compressed first with gz, then with 7z, the resulting file is larger than the gz. That would be the easy way, but 7z is not smart enough to recognize gz, extract, recompress.. and transparently deal with compressed files better.

So if compressed with 7z, for network and stored in gz for disk usage, then the user must compress to gz when plugin runs for first time.

rgleason · 22-05-2013, 16:43

Thomas you're right, various data files must be uncompressed from gz now, otherwise in Linux with the most recent plugin, the data will not show and the data type checkbox is greyed. I do not know if it is a simple matter of uncompressing the old data set.

There is additional data in this version of the plugin, so I'll be uploading a new data set shortly as well as new Windows Plugin dll's for Climatology and Weather_Routing, provided they compile and work reasonably well.

Under Linux, after git clone climatology and Weather_Routing and processing and compiling the data, testing of Sean's improvements was possible:

1. With Climatology and WX-Rte open, and Climatology checked in the Wx-Rte Config a routing from Nassau to Newport was run using the Month of July data with Wind and Current checked. This time the routing stay straight along the rhumb line for awhile but quickly jumped over towards FL to take advantage of the Gulf stream, eventually saving more than a day by the time it had isochrones up to Newport. This is working well in my opinion, recall last time the route was a dogleg out.and back with no recognition of the currents available. See first screenshot.

2. The next run was with Grib plugin and a 4 day grib with wind and no currents. The route was virtually on the rhumb line until the grib ran out, before making it to Newport. No screenshot.

3. The next run (and second screenshot) used Wx-Rte with both Grib (4 day grib) and Climatology, with Wx--Rte Config climatology checked and grib checked. In this configuration the Grib wind is used until it runs out, and then Climatollogy wind is used to finish the routing. Since the currents were checked in Climatology, they are used by Wx-Rte for the routing because the Grib does not have current informaiton. You can see that the Grib winds create a very different Isochrone layout, but result in a route which is similar to the first all Climatology routing, taking advantage of the Gulf Stream along the coast. Once the data for the grib winds runs out, the wind component comes from the monthly climatology winds, during the last 4 days going into Newport.

4. After "Export Route" from Wx-Rte, the Route Manager is opened, the Track tab selected and the Track highlighted, then open Properties and see the waypoints and times in the routing. The third screenshot shows this configuration.

These are significant improvements and features that we should all thank Sean for! There are a couple of little data problems in the Linux version, which will probably be fixed next time, some of the new data types are greyed out because the data is not available yet.

rgleason · 22-05-2013, 17:25

Sean, we're both working on it same time I think. Just saw your message about compression.

Quote:

I thought maybe it's ok to use 7z and just to store the files locally uncompressed which causes more disk space usage, but this is way less than a set of charts. But then gz hardly takes any time at all to decompress and has huge savings..

Okay for Internet we'll use 7z

Quote:

So if compressed with 7z, for network and stored in gz for disk usage, then the user must compress to gz when plugin runs for first time.

Ok, can we have plugiin suck in 7z and put out gz on first initialization? Then it is the best of both worlds as you said?

Quote:

I think I made a mistake and broke it. Should be fixed now.

Let me know what you are doing. I was in the middle of upload of a new data set.
Maybe I should wait, until you have a chance to get it all squared away. --I will do that.

In Linux, Climatology pluigin there are three data types that are greyed out, due to not getting data I think. They are new ones. Sea temp, Air Temp and Sea Depth, ---you probably know that.

It takes quite awhile to get the data done (don't know if the data format has changed for wind and currents). I've learned how to parse the makefile (I copy it and delete the parts I don't want to run (already done) and just run the new or missing data. Speeds things up, but it is still a process...
--What do the All statements at the beginning of the gendata Makefile do?

I'm going to try Windows Git and see if it is better, but won't upload data or plugins until you let me know you are ready and how you want the data.

rgleason · 22-05-2013, 19:34

Interesting evening. I git fetch and git pull origin master branch, git clone wx-rte, clim & watchman. Make sure crashrpt lines are commented out, copy new data into proper location, then star MSVC and rebuild, then Debug ---> immediate memory error. Try continue, stop, start, contine, stop etch # times same error. Shutdown reboot, do it all over. same error. - Some problem with the master branch. I CANNOT COMPILE anything. I hope this gets fixed somehow.

So MSVC built for Release. Copied data and copied the release plugins into Ocpn3.3.424. Start it up. I can do routing with climatology, see below. Good results.

Several problems - Isobars are greyed out now, no isobars available, but the plugin works. Certain data is not available, sea temp, air temp and sea depth, they are greyed out and do not show with cursor moving.

Also cyclones still are not working. They were beffore, I wonder what has changed.

Hope we can get MSVC debuging going, but this plugin is getting much better anyway. See the dogleg route from last week, now the wx-rte uses both climate current and wind to arrive at a very fast solution. Notice how the current saves about 1 day.

rgleason · 23-05-2013, 05:09

Email from a friend

Quote:

...This is amazing. The final track taking advantage of the currents is basically the same as Jimmy Cornell's and the final track --- selected. What was the final time and total distance for the track running up the coast?
There is still the choice of selecting the NE or NW Providence Channel on departure from Nassau. The comment I fed back to ----was that this looks great but what about shipping traffic. I don't know if he has AIS, but he does have radar. It is a pretty quick process of determining track of most of the ships coming North or South. They move at 18-20 knts and from either direction they will be passing us. [north is in stream, south is inboard out of stream] The pilot chart shows the tracks veer off to the East at about the latitude of Hatteras. [marinetraffic.com might also help with ship traffic routes] This checks with my last BI-Norfolk trip, when we saw only one ship until we got close to the Chesapeake. There are several sites on line that show historical ship tracks digitally. A neat capability would be to have a color gradient swath showing the density of the shipping along that route -- one, to alert the skipper that he is running into high density shipping and two, to permit minor course changes to get off to the edge of the shipping where the skipper is likely to get more sleep.

Perhaps have a shipping overlay as suggested? Not climatology, but certainly affects routing considerations. Sort of fits with the idea of hiistoric trade routes and Jimmy Cornell routes. I'll ask him for those URLs.