NAVTEQ and MASSGIS

Editor's Notes

• #4: MassGIS brokered a partnership involving three state agencies and a private company to develop roads data for emergency response. Each organization used its strengths to improve the final product.The three agencies involved are:Executive Office of Energy and Environmental Affairs (EOEEA)Executive Office of Public Safety and Security (EOPSS)Executive Office of Transportation and Public Works (EOTPW)The commercial company is NAVTEQ, a worldwide provider of roads data for geocoding and navigation.Massachusetts essentially has no county government and the Public Safety Answering Points (PSAPs) in Massachusetts are not regionalized in any other systematic way. While there is some ad-hoc regionalization, particularly in the more sparsely populated area, there are still currently 278 PSAPs covering the 351 cities and towns in Massachusetts.The Massachusetts Statewide Emergency Telecommunications Board (SETB), under the Executive Office of Public Safety and Security (EOPSS), was created by the Massachusetts legislature to coordinate and administer the implementation of Enhanced 911 (E-911) across the state.Each partner’s role is discussed in more detail in the following slides.
• #5: As part of the E-911 effort, SETB set out three years ago to add mapping to the PSAPs. While this mapping would give added information for land-line calls, the real driver was Wireless Phase II 911 compliance. In Phase II, the PSAP receives the wireless phone number plus a lat/long location accurate to within 50-300 meters, depending on the handset and carrier service. Mapping in the PSAPs would provide reverse geocoding of this lat/long to provide a nearby address for response and would also provide an on-screen circular buffer showing the area of location uncertainty.The PSAPs would be using the MapStar mapping program from Plant Equipment, Inc., to provide the mapping. The SETB needed to develop or acquire a roads data layer with geocoding support. They do not have an in-house GIS department and did not want to create one from scratch. At the time they considered purchasing a commercial data set license for each of the 278 PSAPs.
• #6: The Commonwealth has its own statewide 1:5000 roads data, but this data does not contain address ranges for geocoding. In addition, this data is maintained by the Executive Office of Transportation and is oriented toward data collection for highway maintenance and federal reporting standards. EOT was not in a position to do the work required to maintain a dataset for geocoding. Although they do have in-house GIS expertise, their mandate is for transportation projects, so they were not able to directly assist the SETB with the GIS work required for generating the roads database.EOT had its own needs for a geocoding layer, however, particularly to improve the accuracy of crash mapping. At the time that SETB was planning the mapping project, EOT was considering purchasing a commercial dataset for this purpose.
• #7: The Massachusetts Office of Geographic and Environmental Information (MassGIS) under the Executive Office of Energy and Environmental Affairs (EOEEA) does have extensive and long-time expertise in GIS. It also has existing support infrastructure, including an enterprise geodatabase that could be used to perform any required data manipulation. Finally, they have a mandate to apply their experience to help other government entities meet their GIS needs in a cost effective manner, to “foster cooperation,” and to improve data. Specifically “…the duties of said office shall include, but not be limited to:fostering cooperation among local, state, regional and federal government agencies, academic institutions and the private sector in order to improve the quality, access, cost-effectiveness and utility of geographical and environmental information as a strategic resource for the state; (b) coordinating data sharing and executing data sharing agreements among all levels of government and private users;(c) identifying, developing, correcting, updating, distributing and assembling geographical and environmental data;…“Recognizing the synergy between the needs of the SETB and EOT, MassGIS offered to work with both agencies to coordinate their efforts, and to provide the necessary GIS support to the SETB. SETB agreed to fund three positions at MassGIS to perform the initial GIS work required to improve and install the GIS roads data for the PSAPs.
• #9: In addition to providing the data, and perhaps even more importantly, NAVTEQ and MassGIS forged an agreement in which MassGIS would provide data update requests (DURs) directly to the local NAVTEQ field office in bulk and NAVTEQ would provide the field crew to verify, update, and improve the data. NAVTEQ committed to perform this work in a fixed time frame from receipt of each batch of DURs. NAVTEQ was the only vendor we spoke with that was promising to look at the ground reality to resolve issues. This included using GPS to capture new streets, and verifying names on street signs.This part of the agreement provides the Commonwealth with ground verification capability at no extra cost that it never would have had otherwise. For their part, NAVTEQ gets improved and more complete data by having their data set checked against other sources that they wouldn’t normally use.
• #10: Example of statistics ;2008 totals by % (3005 entities)41.1% - Correct in NT (1240)22.5% - Fixed in NT (676)17.1% - Doesn’t Exist/Paper Street (517)12.8% - Added to NT (385)6.1% - Further Research needed (187)0.4% - Other (15)
• #12: The Master Street Address Guide (MSAG) is the database maintained by the phone company for validation of addresses for emergency response. The MSAG contains all streets in each community along with the low and high address number for each street, as well as some 911-specific fields (Emergency Service Zone or ESZ, Emergency Service Number or ESN, etc.).The addresses in the NAVTEQ database needed to be translated into MSAG format to improve the address matching in the mapping software. This included normalization to the MSAG address standard and the creation of an alias table for standard MSAG abbreviations.Dealing with the MSAG standardization is a difficult task that is still ongoing. In particular the fact that the ST is dropped from streets can make translation of single-field addresses difficult, and also creates some ambiguity in parsing MSAG addresses.
• #13: The MSAG in Massachusetts contains 29 communities which are not one of the 351 cities or towns or make up only a subset of a given city or town (with the rest being covered by other communities). For these cases, each road segment in the database needed to be assigned to its correct MSAG community. Like most other neighborhood boundaries, these are a bit amorphous and have evolved over time independent of other boundaries. This example shows the streets in Boston symbolized by their MSAG community values. There are 12 MSAG communities in Boston. The red lines show the boundaries of the quasi-official Boston planning department boundaries to illustrate the difference. Using ZIP code polygons and other available boundaries provided much the same result, so in order to correctly assign the community values the street names and often the address ranges in the MSAG had to be checked against each segment.
• #14: The next step in the data verification process was to identify all of the streets in the MSAG database that were missing from the NAVTEQ database. These were sent to NAVTEQ for research and field collection.This identification was done for the whole state, town-by-town, first using automated methods to reduce the candidates, then by manually checking each candidate.
• #15: We also verified the NAVTEQ database against every other dataset we had at our disposal, using a combination of spatial queries and attribute matching to further identify missing streets as well as possible attribute errors in the NAVTEQ data. We checked the NAVTEQ streets against the EOT roads statewide, and used municipal GIS street and parcel data wherever they were available. This process also involved creating an alias table to track valid differences in street names between the datasets. All potential data issues were sent to NAVTEQ for research and field verification.
• #16: In order to make the labor intensive comparison process easier, we created several custom tools in-house and had one built for us by a third party contractor. Two examples are shown above.The toolbar on the top contains a collection of frequently used built-in commands along with a custom search box that allows the user to perform an attribute search on several different feature classes at once using the same wildcard search pattern. The tool on the bottom allows users to quickly cycle through the features in a feature class and mark them with attribute values that allow them to be filtered in a later step in the process.
• #17: Since EOT had purchased the NAVTEQ linework, in addition to updating and improving the NAVTEQ database we were also able to compared the NAVTEQ to the EOT in the other direction in order to identify errors and omissions in the EOT data. Errors that were identified were sent to EOT for corrections, and missing streets were edited by MassGIS to EOT centerline standards and delivered as a GIS dataset to be copied in bulk directly into the EOT database.As mentioned previously, the NAVTEQ attributes were also conflated to the EOT lines. MassGIS maintains the NAVTEQ attribute link to the EOT data inside the EOT geodatabase. Using EOT’s enterprise geodatabase system we are able to add and delete entries to this table with little interference and to maintain the existing data without interruption of EOT’s day-to-day operations.
• #18: Long term there is still a wish for the Commonwealth to own as much of this data as possible, so MassGIS contracted a conflation project to link the NAVTEQ addresses ranges onto the EOT road segments. Integration of the conflation results is nearly complete. This will eventually allow us to use EOT geometry with the NAVTEQ attributes and might also allow us to integrate datasets from the local municipalities into the PSAP mapping product.MassGIS maintains the NAVTEQ attribute link to the EOT data inside the EOT geodatabase. Using EOT’s enterprise geodatabase system we are able to add and delete entries to this table with little interference and to maintain the existing data without interruption of EOT’s day-to-day operations.
• #19: The biggest obstacles we faced through the project were: MSAG Community boundary delineation (discussed previously) Standardization of MSAG street names, particularly the “ST” issue (discussed previously) Reconciling the NAVTEQ schema with the schema required by the PSAP mapping application, including: Alias handling in the NAVTEQ data (duplicate arcs vs. separate alias table) Address range handling in the NAVTEQ (primary vs. secondary address range) Mixed parity address rangesThe mixed-parity issue required us to make a small change in the address ranges themselves as a workaround. NAVTEQ stores address ranges with a high and low value for each side of the street plus a separate field to indicate whether the side is odd, even, or mixed. In the mixed case, often both sides of the segment would have the same address range as shown in the top segment above. The problem is that none of our geocoding engines have the ability to look at this extra field to determine the parity for the side. Instead they look at the low address number for each side to determine the parity for that side. In the top segment above, the low address for both sides is odd, so the geocoders will assume that both sides are odd and even addresses will not be geocoded.As a workaround, we added one to the low address (and just for symmetry subtracted one from the high address) for the mixed parity segments as shown in the bottom segment above. This forces one side to be odd and one to be even, but doesn’t effectively alter the geocode location since in the top case we don’t know what side the address is on anyway.
• #20: With any large project, the unexpected hurdles are what get you in trouble. At the beginning of this project it was hard to exactly see how the details of each individual dataset to be used would affect the comparison of the datasets to each other.Another difficult idea to come to terms with was that there may be no truly “right” answer when multiple datasets disagree and each considers themselves to be correct by their own standards. The goal driving this project was to match the MSAG as best as possible, even when it contains its own mistakes.A project like this can also take a large amount of time – 18 months to put the partnership together and 2 years to get the data updated. In the end, though the payoff was there and the end results more than justify the time invested.
• #21: We now receive quarterly updates from NAVTEQ which need to be processed for installation in the PSAPs. In Massachusetts these updates include the results of the data update requests sent to NAVTEQ by MassGIS in the previous quarter in addition to any changes and updates they have made through their own processes.At a very high level, our process consists of standardizing and reformatting the data for the PSAP mapping software, verifying the data against all other available data sources, and sending any discrepancies to NAVTEQ for field verification.This is an ongoing maintenance process that will provide continuous improvement to the dataset.
• #22: The above diagram illustrates the flow of data from the NAVTEQ quarterly releases. We are not yet at the point where we can push a data update out to all 278 PSAPs automatically, so the PSAP upgrades are currently on an as-needed basis, with rolling upgrades as time allows.In addition, we are currently working on a process to synchronize the EOT and NAVTEQ data on a quarterly basis. Since EOT is also making changes to their data based on their own processes, this synchronization is necessarily two-way.
• #23: In addition to continuing to check the NAVTEQ releases against existing datasets, we also wanted to have a way for the PSAPs to report data issues directly to MassGIS. This would be on top of other required reporting, but gives local users direct access to us and in turn direct access to NAVTEQ. To that end we have create a web-based data issue reporting tool. This tool allows the user to put a point on a map of the NAVSTREETS data to show the location of the issue, and to add attribute information to describe the issue.This tool uses the MassGIS web services technology based on Geoserver and the OGC WMS/WFS standards.
• #25: Our goal is still to move the EOT linework into the PSAP application, and we are currently putting the final touches on the conflation to help us realize this goal. This will not change the relationship with NAVTEQ, however, as we will still be getting the attribute data from their database and we’ll still be using their lines to verify our database (and vice-versa).As discussed earlier, we currently have a setup where EOT maintains their own geometry and all of their own attributes while we maintain the address ranges.
• #26: We are also working on an MSAG standardization engine that will allow us to standardize any incoming address to MSAG format regardless of the source. This is the initial goal, which meets a short-term need, but ultimately we would like to migrate all of our internal address storage to a standard format such as the FGDC standard sponsored by URISA and NENA, and translate to other formats from that common point as needed.
• #27: The next step will be to integrate point data into the PSAP mapping application where available. We have successfully completed a pilot project in the City of Cambridge. The mapping software will look first look for a matching point address and will resort to interpolated geocoding against the lines if one is not found.Currently we have point address data (or parcel data from which it can be derived) in less than half the state, so a big part of this phase will be partnering again with other agencies to build out the statewide parcel data.Although we have a parcel standard written by MassGIS, not all existing parcel datasets are compliant. The address standardization discussed on the previous slide will help us to manage the varying address formats coming in as we move to point-based geocoding.The final issue with using point data will be rolling it out to 278 locations. A statewide point dataset will be too unwieldy and isn’t really necessary, meaning that we’ll need to devise some way to regionalize the point data while still delivering the line data to all locations for occasional geocoding outside the region.
• #28: In the end, this partnership is a win-win for all parties.MassGIS furthered their mandate to provide centralized GIS support and bring together other state entities, and also improved the roads data for all state users.SETB saved money on the initial purchase of the NAVTEQ data, saved money on GIS by not having to build an in-house capability, and got better data.EOT lowered their ongoing costs by not having to pay the annual maintenance on the NAVTEQ data, got better data for their geocoding needs, and improved their own data.NAVTEQ of course got better data.