FormatConversion

Editor's Notes

• #5: Why?: Environmental: reef protection, bilge water dumping, oil spills, but most importantly illegal fishing… Logistical: Port authorities, logistics companies, scheduling Security: surveillance, smuggling, piracy
• #6: As a ship captain, how do you prevent collisions with other vessels? People tend to jump immediately to SONAR/RADAR, but there are a few major problems to that: The equipment is expensive The equipment requires a lot of power These systems are actually quite difficult to read. They require some skill to operate. The most common method was simply to visually observe the other ships and try to estimate their speed, course, heading, and acceleration. Then you would do the same for your vessel and figure out the calculus to determine if you are going to collide or not. Obviously, this I also tricky, and fails in situations like: Night time Stormy weather When you are going around a tight curve in a waterway, and can’t see what’s coming at you Ships don’t stop on a dime. In fact, some of these vessels take in the neighbourhood of 20 minutes to stop. So sometimes, you will have two ships that know they are going to collide well in advance of the collision, but they can’t turn or stop fast enough to do anything about it. So the problem of ship collisions is what triggered the creation of AIS.
• #7: All vessels transmit a “Hear I am! Please don’t hit me” to the other vessels in the area.
• #8: Here are some of the fields that are transmitted. In the Type 1,2,3 messages, we have position information. These are transmitted every few seconds while the vessel is moving. In other message types, we have more static information that doesn’t change very often, like registration and destination and ETA.
• #22: The first thing you do when you have a lot of formats: Create a new format! We created a new internal format that we called EEA. Not only could it hold all of the AIS message fields, but it also had “extension fields” where we would shove all the fields that might be specific to a format. For example, GNM has some metadata fields that are specific to GNM. We put those into a GNMMetadata field within the EEA spec. So now we have a format that can capture all of the complexity of all the AIS formats. It was written to be faithful to the AIS spec, which means that it handles the full complexity of the AIS spec.
• #23: A side benefit of redesigning our format (and our in-memory representation) with EEA is that we got to fix some of the issues developers were accidentally creating. AIS is a complicated spec. For example, look at the definition for speed over ground. There are 1024 bits, and while most of the bit values can be interpreted as doubles, there are two reserved values that can not. There is 1023 = Not Available, for when the ship doesn’t know how fast it is going. There is also the 1022 value which means 102.2 knots or greater. The problem with all the ad hoc parsers is that developers of them would often get the parsing of these fields wrong. They would often just parse the field as a double, not realizing that these special values existed, and then perform mathematical operations on the fields. So they would do things like average the values of the speed over ground, so you would end up with massive values when a large number of vessels were reporting “Not available”. With EEA, we fixed that, changing the type of the field to be either a double, or one of two special values. This prevents mathematical operations being applied on the field, and forces the developer to stop and think about how they want to actually handle the math.
• #24: The next step was to define a common interface of functions we apply when parsing all formats. We eventually settled on this interface. You start with a “PAYLOAD” which is a collection of bytes representing messages, which might be the contents of a file for example. You then call tokenize() which finds the boundaries of the messages within the payload and splits the payload on those boundaries. You still haven’t parsed the messages, so you don’t know what they say yet. You only know the bytes that make up each message. We called this “UNPARSED” in this diagram. You can then call deserialize, which actually parses the bytes of the message and gives you an in-memory representation of the message. Most commonly, this was the new EEA format. Then on the reverse direction, we take individual messages and call serialize() on them to return them to the UNPARSED tokens we had before. And finally we call merge(), which writes the messages, one after another, into a payload. So these four functions are pretty much universal to format parsing. They also form a graph, perhaps a sort of state-machine where the data parse levels are the nodes, and the functions are the edges.
• #25: We went and implemented these four functions for all of data formats. And when you put all the conversion graphs beside one another, you notice something.
• #26: The PARSED node for most of the formats is EEA.
• #27: It’s the same format. Really, it’s all the same node. Conceptually, you could add edges between them with a NO-OP function.
• #29: So now if you wanted to convert between GNM and NM4, you can just follow the edges from GNM PAYLOAD to NMEA4 PAYLOAD
• #35: And you suddenly have the conversion steps.
• #37: In order to represent the graph in code, we need a graphing library. I came across NetworkX and have had no complaints. It allows you to create nodes and edges in a graph, and then gives you an easy way to do algorithms like shortest path across graphs.
• #38: Here’s what it looks like in code: We have our conversions object, which is just a networkX graph object. Then on each line, we add an edge between each of our parse levels. On each edge, we also supply the function to perform. Notice in the last line, that this is an example where we jump between formats that are already parsed into the EEA in-memory representation. Therefore the function is a simple NO-OP.