Signal K

[bcn]

If you want to squeeze the navobj.xml into this context you are adding dynamite.

Keep it simple, please.

Hubert

[motu42]

Originally Posted by bcn

If you want to squeeze the navobj.xml into this context you are adding dynamite.

Keep it simple, please.

Hubert

Sorry, I dont follow, whats navobj.xml?

[rgleason]

Motu42, Please don't miss Dave's questions, because we have asked other questions. Thanks.
navobj.xml is a file that contains all routes and waypoints. navobj.xml is backed up 5 times with navobj1.xml, navobj2.xml etc.

Originally Posted by bdbcat

Motu42...
I think we are on the same wavelength.
But tell me more about:

What proved very useful in the Artemis server was exposing these as a stream over a tcp/ip port.

Do I read that you did NOT use SK/JSON as the "language" of the stream, but defined a proprietary binary package for inter-process/inter-machine communication instead? If so, this seems a step backwards from the SK idea...

Dave

[motu42]

@bdbcat - i see looking back I didnt answer your question well.


I used a binary format for efficiency to avoid constantly converting to/from strings - a problem mentioned by others. I think this is fine for iterprocess/interhost communications if those processes and hosts are in a trusted wired network. eg all on the same ethernet hub. It assumes high speed, low latency reliable connection, probably one packet per key:value etc.



Once you start connecting with arbitrary hosts across the internet the streaming binary format has its own set of problems. Since latency is usually > 100ms, it needs message buffering, etc to be viable. At that point a signalk delta (json) message is better.


Json (as a string)was chosen deliberately for signalk since we could not forsee what transport mediums would be used in the future. Binary is a problem on serial for instance. Text strings are pretty much universal. Its proven a good choice as IoT has developed (MQTT/COAP/STOMP/etc). Even email and paper works!

[motu42]

Re navobj.xml - if its designed to hold/archive waypoint and route data, then it can be replaced by signal k messages (it has keys for waypoints/routes/etc). That automatically means its also synchronised to all other interested devices.



This already works on the freeboard-sk browser based chartplotter. Create a waypoint on one browser and watch it pop up on the other :-)



Does that remove the dynamite danger? :-)


The real point is that integrating signal k as I have described above both simplifies opencpn, while opening up a whole range of future development directions for opencpn functionality

[rgleason]

RE: bdbcat's questions to motu42, I found this project and some links:

Artemis Server
Artemis Server Design
ActiveMQ - Power Your Microservices, High-performance Messaging for Highly Scalable Microservices
Security

[bdbcat]

Motu42....

I remain unconvinced about the need for a private binary packet distribution model, even within one host. Are we really certain that parsing JSON on our target hardware is a back-breaking task? Has this actually been prototyped and measured?


The model that I would like to see goes something like this:
1. SK server exists somewhere, accessible by TCP/IP.
2. SK server "owns" the serial NMEA and N2K interfaces, maybe by plugins to the server.
3. OCPN core is a simple client of the SK server, subscribing to and parsing incoming JSON containing only data it cares about (a relatively small set)
4. OCPN plugins are also simple clients of the SK server, also receiving data by subscription.
5. OCPN and its plugins communicate closely for GUI interaction, easy enough since they run in the same process. But plugins manage their sensor data requirements independent of core.
6. OCPN core and plugins use whatever private internal data structure they choose.
7. OCPN core retains its currently implemented serial and network NMEA multiplexer, for legacy environments.
8. OCPN core does NOT retransmit anything received by subscription to the SK server.

I'll stop there and await comments.
Dave

[motu42]

bdbcat,

I agree mostly with your model, thats pretty much what I see as a nice clean architecture to move forward on. What you describe with simple subscriptions to the server using the current signalk formats will work fine for most use cases.

BTW You will find that OpenCPN will need to send data out. The simple case of setting a waypoint to the autohelm will require a message to be sent to the signalk server (which will message the autohelm, etc), or acknowledging an alarm.

In Artemis server I started implementing fine grain security and found that I had to decode and test each incoming json string on a per key/per subscriber basis. I didnt want to build a connected system (like cars and phones) where you can hack into critical devices easily.

That produced the key:value binary message and worked well. I then found its way easier and faster to send that direct to plugins. And that gave me the idea to offer it over TCP/IP.

I did test the binary key:value format. It made quite a difference when I ramped up the volume on an RPi3. Individually parsing a json delta message is not much work, even for an RPi2 or 3. Lets say it takes 500ns.

When you get a higher volume of messages, eg 200/sec, 100ms/sec is required just for parsing. But you typically have to parse a larger message (many keys), filter for need and security, then recreate delta string message to pass on and it all adds up. When you push to high loads (eg 1500 msg/sec), then you cant afford any extra cost per message at all.

You will find you need a key:value format internally anyway, as has been discussed before. Using the same one is a nice win all round.

So either way will work, just a question of required performance.

[Axel_G]

Originally Posted by motu42

bdbcat,


Individually parsing a json delta message is not much work, even for an RPi2 or 3. Lets say it takes 500ns.

When you get a higher volume of messages, eg 200/sec, 100ms/sec is required just for parsing. But you typically have to parse a larger message (many keys), filter for need and security, then recreate delta string message to pass on and it all adds up. When you push to high loads (eg 1500 msg/sec), then you cant afford any extra cost per message at all.

.

At 200 msg/sec and 500ns per message it is only 100µs/s (Micro, not Milli) for parsing. Or 0.01%.

[bdbcat]

motu42 (or anyone)....


I have not actually worked with N2K and SK, so my practical experience is minimal.
Please describe an environment that might generate 1500 messages/second in real life.


Thanks
Dave

[stevead]

For a NMEA 2000 network probably a bus running at 60% capacity. Very unlikely on a cruising yacht with standard speed, wind, depth, gps and autopilot and mfd devices and similarly unlikely with other toys such as all the engine, tank, hvac, entertainment & electrical sensors. About half the sensors transmit at 1 Hz the other half at 10Hz.

Possibly likely on racing yacht with higher speed wind & speed sensors coupled with gyros (heel, yaw & pitch), rate compasses, the high end autopilots and strain and compression sensors.

And lets not forget that coming down the pipe will be OneNet (all the NMEA 2000 stuff over ethernet)

I guess in reality it shouldn't be a problem with the current hardware that is in use today (including a Pi), although if poorly architected, 500ns to process a JSON string could result in dropped NMEA 2000 frames.

But let be voice my dissenting opinion yet again on using strings (be it NMEA 183 or JSON) to pass data between OpenCPN and its plugins.

Take your normal cruising yacht with perhaps 10 - 20 % load on the NMEA 2000 network and add 2 or 3 tablets or smartphones running some sort of Signsl K client. Now for a single data value (eg depth) , you're now performing the same JSON parsing 4 or 5 times. Then take an extreme case either a VHF DSC message or AIS message.

[stevead]

1. SK server exists somewhere, accessible by TCP/IP.

Making for a complex network configuration that 'Joe Boater' may have difficulty configuring.

2. SK server "owns" the serial NMEA and N2K interfaces, maybe by plugins to the server.

Which Signal K Server implementation? An expensive iKommunicate box ? An inexpensive RaspberryPi running OpenPlotter which non Windows/Mac users may find difficult ?


Or Signal K Server running on Windows (Node JS etc) and only able to connect to NMEA 2000 networks with an Actisense NGT-1 ? And what of our poor Mac user base?

I love the idea of Signal K schema & JSON to communicate between display devices. However I'm not convinced today that a Signal K Server is the panacea.

From reading this thread there are a lot of developers implementing technology for technology's sake and not fully understanding our user's needs.

[bdbcat]

Steve...


Take your normal cruising yacht with perhaps 10 - 20 % load on the NMEA 2000 network and add 2 or 3 tablets or smartphones running some sort of Signsl K client. Now for a single data value (eg depth) , you're now performing the same JSON parsing 4 or 5 times.



This is the part of your argument I don't get. Assuming the server is keeping up with the pace, why does a PC on the net running O care what the tablets are doing? Sure, the tablets and PC have to individually parse JSON. But why, other than overall system efficiency, do we care?


I'm suggesting trading away "overall system efficiency" in favor of "extensibility and readability".



Dave

[bdbcat]

Steve...


Making for a complex network configuration that 'Joe Boater' may have difficulty configuring.


Read back some. I would like to see a new SK server implemented as a OCPN plugin, installed like any other OCPN plugin. Minimal interaction with OCPN core, just forks and runs a separate "server process", probably. Possibly interacts with other OCPN plugins (TBD) that own the serial ports and Canbus interface.



I envision this as the "entry level" SK/N2K support for OCPN. Advanced users will no doubt want to add external hardware for SK server, unloading their Nav PC. No problem...


Dave

[teppokurki]

Originally Posted by stevead

Or Signal K Server running on Windows (Node JS etc) and only able to connect to NMEA 2000 networks with an Actisense NGT-1 ? And what of our poor Mac user base?

Please don't spread misinformation.

SK Node Server runs just fine on macs, on practically any modern linux system and Windows. Pick your poison.

As for NMEA2000 it supports practically any CAN device out there: NGT-1, iKonvert, YDNU-02, YDGW-02, N2K over NMEA0183 (PCDIN) and socketcan on various hardware (linux).

What we are missing is one click installers for the different systems.