Routes and Mark Duplication

[capn_billl]

I don't know if this is the correct, or easiest way, but when I see a duplicate waypoint, I add the new waypoint to existing routes, and remove the old one. (very tedious), then I delete the duplicate, and I don't get a warning because it's no longer used in any route.

[wingssail]

Originally Posted by RhythmDoctor

I've used OpenCPN for about 10 years and its amazing features make the occasional glitch worth it. I have it talking to all my navigation equipment through Bluetooth and nicely integrating the data onto a dashboard on my Windows tablet.

One of the annoying glitches is the sharing of waypoints between different routes. O seems to offer to do this whenever you click a new route waypoint near an existing one. In short, don't ever accept the offer to use an existing waypoint - create a new one instead.

I save every route individually in its own .GPX file (filename gives origin and destination) and re-use them years later. When opening several routes (which may have been edited over the years), unexpected things happen that can cause one or more of your routes to divert over land. I'm still editing routes from many years ago to eliminate these shared waypoints.

I've never made a flyspray request for this because the issue is easily avoided by not sharing any route waypoints.

Originally Posted by capn_billl

I don't know if this is the correct, or easiest way, but when I see a duplicate waypoint, I add the new waypoint to existing routes, and remove the old one. (very tedious), then I delete the duplicate, and I don't get a warning because it's no longer used in any route.

capn-bill andRhythmDoctor,

The reason for using existing waypoints instead of creating a new waypoint for a location that is used in several routes, aside from being tedious is that if the waypoint needs to be changed, such as a navigation aid which is relocated, you then have to move it in every route.

For JeffSpSailing and myself, who use OpenCPN for race courses, it really does not work to create multiple waypoints for each race mark which a race committee might set. For example, in my area the race committees of various clubs have established a list of marks which we might be sent to. There are a few dozen. They are given names and set at locations specified. Then they establish courses, each going around several marks. It is not unusual to have 20-30 possible courses used in a season. All utilizing sunsets of the same marks. Being able to reuse existing waypoints makes this process infinitely easier. AND when the race committe says, "The Normal W1 windward mark is now at xxxlat/xxxlon for all courses" we are able to move that mark in O and all the courses (routes) are updated.

I also personally have a large number of waypoints I use for cruising, for example the edge of a nearby reef. Many routes I establish each year would go around that reef. It is tedious and dangerous (dangerous due to the possibility of errors) to make a new one each time.

I understand the way O is meant to work and that is how I use it. I don't know if the problem I reported is a glitch or a user error (something I am doing wrong) but when another user makes a similar complaint I think it's worth reporting and looking into.

[RhythmDoctor]

Originally Posted by fgd3

It what way is that a glitch?

The part you quoted isn't the glitch - you only quoted my suggested way to avoid the glitch. But this part that you edited out of the quote is a glitch (or a "feature" if you want to debate it):

Originally Posted by RhythmDoctor

I save every route individually in its own .GPX file (filename gives origin and destination) and re-use them years later. When opening several routes (which may have been edited over the years), unexpected things happen that can cause one or more of your routes to divert over land. I'm still editing routes from many years ago to eliminate these shared waypoints.

I've never made a flyspray request for this because the issue is easily avoided by not sharing any route waypoints.

Originally Posted by fgd3

Rather than dogmatically using or rejecting shared waypoints it is more sensible to spend a little time understanding how they work so you can use them when appropriate and avoid them when they don't meet your needs.

The nice thing about free internet advice is that you're free to ignore it. After 10 years I have spent a lot of time understanding how most of the features work, including this one. O keeps track of changes to shared route waypoints as long as you keep all the routes with shared waypoints open. Since I started using O long before they added the "eyeball" hide icon, I built a library of over 100 routes in separate .gpx files. And I know exactly why this glitch occurs: If you keep your routes in separate .gpx files and change a route waypoint in one route while the other routes that share the waypoint are not in the active list, unexpected things can happen.

As someone else pointed out, unexpected things can also happen when you try to share waypoints imported using .gpx files from other programs or chartplotters.

In this case OP is a new user who hasn't had time to figure out these intricacies, and whose frustration level became so high that he was considering abandoning the program. So my advice for new users is to walk before you try to run. Avoid shared waypoints until you know the unintended consequences (aka "glitches") of certain features.

Dogmatic? If that's what you think, feel free to ignore my advice. I promise you that the heavens won't reach down and strike you with lightning. However, you might find yourself in skinny water someday:

[JeffS-Sailing]

Interesting discussion.
fgd3's response may point the way to a way to use a single set of marks in multiple courses. I may try that at some point.

If assigning a unique ID to a mark is a requirement of OpenCPN it seems like OpenCPN should make that assigment when a mark is selected for multiple use rather than assuming that a user will know that importing marks from another source requires exporting and re-importing marks before using them for multiple courses.

I need marks to be used in multiple courses for the same reason that was mentioned somewhere in this thread. I need to have access to many racing marks that are used in many race courses. Sometimes a mark's location will change. An update to the location of a mark must result in that new location being reflected in all courses that the mark is used in.

I manage a fair number of marks and courses.
Since OpenCPN has mark and course management features, I had hoped that it would be a tool that is the central point for managing all marks and courses that I use, allowing me to selectively download what I need for a given race series into the onboard nav system that will be used.
Given the issues I'm reading here and the ones I've encountered myself, it seems that the mark / course management in OpenCPN isn't up to that task currently.

It also occurred to me that the developers may not have thought of race course and race mark management when building OpenCPN. Given that this discussion is on a Cruiser's forum, I suspect that's the case.

It seems like OpenCPN is very close to being able to manage race courses and race marks without requiring it's users to discover what doesn't work and find work-arounds to do course and mark management outside of the system.

I hope this discussion moves OpenCPN in that direction.
Thanks all.
Jeff

[wingssail]

Originally Posted by JeffS-Sailing

...It also occurred to me that the developers may not have thought of race course and race mark management when building OpenCPN. Given that this discussion is on a Cruiser's forum, I suspect that's the case...

The developers of OpenCPN may not have been thinking of race course and race mark management but they designed and built a system exactly well suited to that purpose, except that some usability or technical issues have limited its utility. The work-arounds presented above are un-necessary. What is needed, and I am confident that it will be found, is some fix or procedural change so that the system works as designed. Then it will be perfect for the race course and race mark management needs.

[fgd3]

When you create a mark in OpenCPN the program assigns that mark a unique identifier (GUID). When that mark is exported to a GPX file the GUID is exported with it. When you opt to use a nearby waypoint OpenCPN uses the GUID of the nearby waypoint. OpenCPN "knows" it is a shared waypoint because it has the same GUID.

If you get a GPX file from a non-OpenCPN source or even from someone else's installation of OpenCPN the GUIDs will either not be present (in the case of a non-OpenCPN source) or will be different from the GUIDs generated by your installation of OpenCPN.

If you are importing waypoints which are completely new to your OpenCPN installation from a source which does not provide GUIDs there will be no problem as OpenCPN will assign the GUIDs when the waypoints are imported. If you subsequently import the original GPX file again the waypoints will be assigned new GUIDs and you will get duplicate waypoints. One way to avoid that problem is once you import a GPX file which does not include GUIDs immediately export that data overwriting the original file. Now you will have an OpenCPN-created GPX file that has GUIDs to identify each object in the file.

If you import a GPX file created by a different OpenCPN installation there is a very slight risk that the GUIDs assigned to the objects in that file may collide with GUIDs assigned to objects created in your OpenCPN installation. Since part of the GUID is the date and time it was created the risk of assigning the same GUID to different object is extremely slight.

If you understand how OpenCPN uses GUIDs to differentiate objects and pay attention to the consequences of how GUIDs work I don't believe OpenCPN will display unexpected behavior. I think it is only unexpected because you missed some subtle consequence of the logic of GUIDs.

RythmDoctor, when you wrote "don't ever accept the offer to use an existing waypoint - create a new one instead" it wasn't my impression you intended that as temporary advice to a beginner.

You wrote "O keeps track of changes to shared route waypoints as long as you keep all the routes with shared waypoints open." That isn't correct. First, OpenCPN doesn't keep track of changes at all. If you import a GPX file containing a shared waypoint, move that waypoint, export to a new GPX file, and delete the object from OpenCPN (1) the original GPX file will remain unchanged, (2) the new GPX file will record the new position of the waypoint but will have no record of the former position, and (3) the position of the waypoint in question which OpenCPN shows will depend on which GPX file is imported first. If you import the GPX file with the modified position first and subsequently imported GPX files which share that waypoint will show it in the new position. On the other hand, if you import a GPX file with the old position first and later import the file with the new position, OpenCPN will show the waypoint in the old position.


You wrote "As someone else pointed out, unexpected things can also happen when you try to share waypoints imported using .gpx files from other programs or chartplotters." Once again, nothing unexpected happens if you understand how OpenCPN uses GUIDs. If an object comes into OpenCPN without a GUID OpenCPN assigns it one. If an object comes into OpenCPN with a GUID OpenCPN uses that GUID. If OpenCPN already has an object with that GUID on its chart it assumes the newly imported object is another instance of the one it already has and, if the location of the object is different, it assumes the object should be moved to the location of the already existing object. If you're seeing objects moving in unexpected ways its because you either don't understand these rules or you already have an object on your chart you've forgotten about.


JeffS-Sailing, in your original post you mentioned importing a list of waypoints. I asked where that list came from and never got an answer. If the GPX file didn't have GUIDs (because it didn't come from OpenCPN) and you imported it twice you would wind up with two sets of waypoints. They would appear as duplicates to you but would have different GUIDs (because each time they were imported they were assigned new GUIDs). That's not a bug in OpenCPN, that's operator error. Once you import a non-OpenCPN GPX file never import it again. Delete it or export the data back over it (thus transforming it into an OpenCPN-generated GPX file) if that's what it takes to prevent accidentally importing it a second time.


I hope this has helped clear up some misunderstandings regarding shared waypoints in OpenCPN. It's covered in the documentation but until you've had some experience with shared waypoints the significance of some of the logic may not be apparent from even a careful reading of the documentation.

[fgd3]

Originally Posted by wingssail

some usability or technical issues have limited its utility. The work-arounds presented above are un-necessary. What is needed, and I am confident that it will be found, is some fix or procedural change so that the system works as designed.

I think what limits OpenCPN's utility in this area is a lack of user understanding of how the program was designed. The system does work as designed, but users have to understand how the design works and conform to its logic.

[bdbcat]

All...
Some investigation under minimum reasonable conditions leads to the following tentative conclusions:


1. Building routes with shared points works more or less as expected, as long as all of the routes using shared waypoints are left in a "visible" state.
2. If one "hides" a route containing shared waypoints, one gets the message:
"Do you also want to make the shared waypoints being part of this route invisible?"


Two choices:


"No"
This is probably the logical choice for most users to make, especially in the case of using multiple racing courses around fixed marks. Unfortunately, this response can produce an inconsistent internal state, such that the visibility of the affected waypoints cannot then be toggled. This is the effect seen in FS2795, referred to above.



"Yes"
This choice immediately hides the shared waypoints. In the route/manager dialog, the affected waypoints are shown correctly with red "X", and may have their visibility toggled manually. There is no inconsistency in this state, although the UX is awkward.



My conclusions:
3. This logic is at best a non-intuitive glitch, and at worst a real behavioral bug, depending on your use case.


I propose the following change to the logic:
a) When user elects to "hide" a route, drop the non-intuitive message box "Do you also want..."
b) If a route to be hidden contains no shared waypoints, then simply hide the route and all points. This covers the majority of use cases involving single use routes. This is also current behavior of OCPN 5.x
c) Otherwise, hide only the unshared waypoints in a route. Ensure that any un-hidden (i.e. shared) points may later be individually hidden, as desired, but only if all routes using such points are also hidden. This is new proposed behavior.


I invite comments.
Thanks
Dave

[wingssail]

Originally Posted by fgd3

I think what limits OpenCPN's utility in this area is a lack of user understanding of how the program was designed. The system does work as designed, but users have to understand how the design works and conform to its logic.

fgd3, your explanations up thread are excellent and useful however we are dealing with either a bug or a user error in the observed behavior of waypoints in flyspray 2795. It has nothing to do with importation of GPX files.

Waypoints are appearing on the chart when they have been made invisible in the route and mark manager. They appear on the chart but cannot be used in a route ("use nearby waypoint" does not appear).

The waypoints in question have previously worked perfectly but no longer work. It is not all waypoints.

Bobcat has asked for a GPX file with some marks which behave this way but cannot recreate the error. He has asked for the total navobj file but I cannot upload it due to size.

I am a OpenCPN booster and use it regardless of the outcome of this issue.

But it should be fixed. There are a lot of implications

[bdbcat]

WIngssail...


As I say above, I'm pretty sure that the FS2795 issue is just this:
You probably have a route using the affected marks, and that route was hidden using the "No" answer to the "Do you also want..." question.
It really has nothing to do with waypoint import, or the state of the import GPX file.



Solution/workaround: Temporarily enable visibility on ALL routes (manually, in route/manager dialog). When you want to hide a route, choose the "Yes" option. Then manually re-enable visibility on the desired waypoints.


As I say above, this workaround leads to an awkward UX. We are together trying to come up with a workable solution for the next version of OCPN, without breaking the model for existing users.


Dave

[nohal]

The main "problem" here is that OpenCPN tries to give the user as much freedom as possible and rely on open standard like GPX to allow for easy migration both to and from. In GPX, the paradigm of a route passing a waypoint simply does not exist, waypoints (marks) are a completely different entity than the points forming routes there and there is absolutely no relation between them at all.

"Fortunately" the GPX format allows vendor specific extensions, which we use to assign GUIDs to our objects to be able to identify them with increased precision. But of course that is not a perfect solution. While that works reasonably well in one OpenCPN instance with a concious operator, the GUIDs (and all the other vendor-specific information) are ignored, and usually removed, by every other tool that touches the GPX file. And even if not, what are we supposed to do when reimporting a point with a known GUID that has meanwhile changed position in the running system?

Or importing several GPX files where a seemingly identical point GUID-wise has a different position in each, or name, or symbol? Should we ask when unsure? If we should, will users importing 10000 such objects for whichever reason thank us for doing it?

And what has precedence - an object with a GUID we "know" or an object with identical position we already have with different GUID? And is it still the same object? Are two object with same position and different name one? Which one is "correct"? And were the route points with same coordinates as known waypoints present in the imported routes shared or not? And what is the same position? Exactly? One meter apart? 10 meters? 100? And is it always the same? Isn't the difference caused just by the fact the positions passed through some software that rounded them with a lower number of decimal points? And then another that again rounded them to a higher number of them?
And is the logic crosschecking everything still able to handle a GPX file with a 1000000 points with acceptable performance?

So many questions and no answers to them

What we have now is a result of quite a few years of refining the waypoint and route related logic based on user input.
The current implementation is not completely bug free and I certainly do not think the logic as designed is perfect and there is no more room for improvement (like for example also extend the vendor specific metadata we store with the timestamp of last modification and use it to decide which version of the seemingly "same", but actually different, object is "correct"), but honestly, as far as I can tell over time it got reasonably usable and performant for pretty much everyone willing to understand the limitations and slightly adjust their workflow in case it is necessary.
Please try to understand that what from your point might and does seem like bugs, even obvious and straightforward to fix, when changed, will very likely make other people with different workflows and needs think we have broken them.

Still many thanks for the concious Flyspray entries resulting from this thread as they should make our next iterarion even better.

Pavel

[wingssail]

Originally Posted by bdbcat

...I propose the following change to the logic:
a) When user elects to "hide" a route, drop the non-intuitive message box "Do you also want..."


b) If a route to be hidden contains no shared waypoints, then simply hide the route and all points. This covers the majority of use cases involving single use routes. This is also current behavior of OCPN 5.x

The problem I see with this is: Suppose I create a number of waypoints that I will use in routes. Then I create a route that uses five of them. When I hide that route none of these waypoints will remain visible or available for the next route I create. If I proceed to create a second route which uses three of these waypoints they are unavailable ( or they appear but cannot be selected) unless I go into the Waypoint and Mark Manager and make all of them visible again. If I miss one then I must "end route" and go find that waypoint and make it visible.

c) Otherwise, hide only the unshared waypoints in a route. Ensure that any un-hidden (i.e. shared) points may later be individually hidden, as desired, but only if all routes using such points are also hidden. This is new proposed behavior.

Again, when I create the first route none of the waypoints are yet shared, so all will be hidden and unavailable for the next route.

I invite comments.

Thanks
Dave

I'd propose:

1. That you do not hide pre-existing waypoints except in the Route and Waypoint Manager. I don't care if they are shared or not. If I created them prior to creating the route, leave them alone when hiding a route. You can refer to these preexisting waypoints as "permanent". (I understand that there is or was a distinction between route-points and waypoints. Would that be useful?)
2. If I "drop a mark" it becomes "permanent" by default
3. Further to enhance this, that you add a "permanant" flag to waypoints so I can add create a waypoint while creating a route and go to the properties dialog and I can mark it "permanent" in which case it does not disappear when hiding the route.
4. Neither of these two changes affect the Route and mark manager "visibility" function. Any waypoint, even one flagged as permanent can be made invisible by toggling the eyeball.
5. None of these changes would affect the other options of the Route and Waypoint Manager. A waypoint can still have it's properties changed, and still be deleted.

I know this is a more extensive change than what you proposed, but it would be better. The "shared" designation does little to keep marks available as we start to use them but a "permanent" designation would.

[bdbcat]

Wingssail....


Terminology differences.
In the context of OCPN, a "shared" waypoint is just what you'd label as a "permanent" waypoint. The word "shared" means that this waypoint exists as an entity, but is is "shared" with a route. Imported waypoints are "shared". Dropped waypoints are "shared".

In fact, the only "waypoints" that are not shared are the ones automatically dropped during route creation when not using a nearby existing point. We might call these "routepoints", for clarity. Anyway, these are the default diamond shaped marks.



I do not really want to add another flag to a waypoint (e.g. "permanent"). I think the existing internal "shared" designation does the job.


So, our ideas are almost congruent.

Agree?
Dave

[wingssail]

Originally Posted by bdbcat

Wingssail....


Terminology differences.
In the context of OCPN, a "shared" waypoint is just what you'd label as a "permanent" waypoint. The word "shared" means that this waypoint exists as an entity, but is is "shared" with a route. Imported waypoints are "shared". Dropped waypoints are "shared".

In fact, the only "waypoints" that are not shared are the ones automatically dropped during route creation when not using a nearby existing point. We might call these "routepoints", for clarity. Anyway, these are the default diamond shaped marks.



I do not really want to add another flag to a waypoint (e.g. "permanent"). I think the existing internal "shared" designation does the job.


So, our ideas are almost congruent.

Agree?
Dave

Dave, I am agreeable. After all I am using the system the way it is and have been happy, and a supporter.

I actually just want the waypoints which have an "X" in the Route and Mark Manger to not remain on the chart, and if it is on the chart, allow me to "select a nearby waypoint" and if I say "No" to "Make shared waypoints invisible", it doesn't and it doesn't set up this illogical situation I refer to 2795.

Those are the important issues for me.

Please don't make wholesale changes to lots of mark's visibilities so that I have a lot of manual effort to do to get them back on my screen and usable.

[bdbcat]

Wingssail...
"make wholesale changes to lots of mark's visibilities so that I have a lot of manual effort"
Be assured, we will not do that. Thousands of users coming at us with torches and pitchforks sounds like no fun. No thank you, sir.



The trick, as Nohal notes, is to "improve" the UX without breaking existing workflows, nor hurting performance.


Anyway, I understand your use case now. I expect you will see some logic improvements here when we start the next Beta test cycle, which is "real soon now". That will give us a chance to iron out the bugs.


Thanks for the feedback.
Dave