GeospatialPresentationWASUP_RobHall
- 1. Geospatial Analytics using SAS and Publically Available Information Rob Hall WASUP – October 2012
- 2. Overview Visualisation of geographic data is essential to understanding the distribution of attributes across Australia. Retail Credit Risk has developed software to enable easy generation of maps for any desired statistic through SAS Enterprise Guide. Once the data is prepared based on postcode, a fully automated process will generate a desired map in under a minute. The program can be easily extended enabling analysts to view data from many angles. It is an essential tool which can be easily added to existing MI packs. SUCCESS STORY: One of our reports rarely received feedback from Executives. Once we placed a map visualising the data, we immediately received requests for more information!
- 3. Visual inspection of data is essential to understanding regional trends and hotspots Visualising Data Large result tables are difficult to place into perspective, and don’t allow the reader to see the big picture. The ability to visualise data adds meaning to reports and facilitates the communication of a message. Comparing the map above with the table, we can easily see where areas are experiencing stress. This information is lost in the table. It is difficult to place all the results in one table and communicate the intended message. Area Number of Accounts Delinquent Accounts Percentage Delinquent Ave Delinquent LVR Ave Delinquent Total Limits Ave Delinquent EOM Balance Ave Delinquent Amount TUGUN - QLD 35 6 17.10% 66 262,803 -284,825 -22,022 ADELAIDE - SA 85 7 8.24% 81 501,052 -519,162 -18,110 AUSTINVILLE - QLD 143 11 7.69% 84 259,973 -232,103 27,870 BLACKSTONE HEIGHTS - TAS 94 7 7.45% 84 174,336 -180,224 -5,888 BERKSHIRE PARK - NSW 74 5 6.76% 56 257,738 -275,449 -17,711 ASCOT - QLD 85 5 5.88% 83 261,538 -268,410 -6,872 BARANGAROO - NSW 139 8 5.76% 69 570,543 -893,595 -65,000 BADGIN - WA 98 5 5.10% 66 175,961 -183,021 -7,061 BENOWA - QLD 139 7 5.04% 88 273,636 -281,545 -7,909 ABERCROMBIE - NSW 127 6 4.72% 82 221,895 -227,025 -5,131 BALLIMORE - NSW 147 6 4.08% 86 206,946 -215,589 -8,643 AROONA - QLD 202 8 3.96% 90 315,982 -326,191 -10,209 BAMARANG - NSW 102 4 3.92% 55 224,548 -239,354 -14,806 COLLINGWOOD PARK - QLD 104 4 3.85% 68 288,195 -302,357 -14,162 BRANDITT - VIC 131 5 3.82% 84 375,463 -389,270 -13,808 CLEAR ISLAND WATERS - QLD 131 5 3.82% 83 422,059 -437,013 -14,954 MACGREGOR - QLD 162 6 3.70% 85 261,405 -265,832 -4,427 BASIN POCKET - QLD 191 7 3.66% 71 212,036 -220,254 -8,219 BELGRAVIA - NSW 221 8 3.62% 73 244,405 -249,963 -5,558 ASHCROFT - NSW 196 7 3.57% 80 316,575 -323,112 -6,536 ARUNDEL - QLD 198 7 3.54% 71 246,028 -254,343 -8,315 BROADBEACH - QLD 143 5 3.50% 79 319,676 -334,795 -15,119 CONISTON - NSW 116 4 3.45% 58 222,348 -227,042 -4,693 AMBERLEY - QLD 119 4 3.36% 65 234,315 -247,949 -13,634 CRESTMEAD - QLD 125 4 3.20% 51 173,490 -177,129 -3,639 FRENCHS FOREST - NSW 161 5 3.11% 78 247,681 -253,731 -6,050 NORMANHURST - NSW 198 6 3.03% 80 323,569 -342,623 -19,054 COLLAROY - NSW 133 4 3.01% 85 283,678 -291,542 -7,864 HMAS PLATYPUS - NSW 100 3 3.00% 80 929,211 -965,459 -36,248 GILBERTON - QLD 100 3 3.00% 92 420,057 -445,315 -25,258 DINGLEY VILLAGE - VIC 137 4 2.92% 45 373,832 -380,187 -6,355 BURLEIGH DC - QLD 138 4 2.90% 84 384,508 -889,992 -70,000 PADSTOW - NSW 138 4 2.90% 58 148,621 -155,076 -6,455 COOLAROO - VIC 138 4 2.90% 80 243,244 -249,687 -6,443 KINGSTON - QLD 141 4 2.84% 85 345,028 -359,125 -14,097 PARALOWIE - SA 211 6 2.84% 82 178,641 -192,291 -13,649 ALBURY - NSW 142 4 2.82% 69 260,287 -267,349 -7,062 ALBION - VIC 287 8 2.79% 81 200,088 -207,630 -7,542 KUNYUNG - VIC 109 3 2.75% 0 90,000 -120,629 -30,629 ABBOTSFORD - QLD 291 8 2.75% 56 198,639 -203,405 -4,766 CANNON HILL - QLD 146 4 2.74% 79 444,927 -471,691 -26,764 GARDEN CITY - VIC 147 4 2.72% 73 706,434 -758,754 -52,320 ADVANCETOWN - QLD 331 9 2.72% 79 332,900 -341,796 -8,897 CALAMVALE - QLD 148 4 2.70% 71 381,881 -394,281 -12,400 AMAMOOR - QLD 112 3 2.68% 27 221,188 -229,098 -7,910 MAWSON LAKES - SA 112 3 2.68% 92 156,710 -159,847 -3,137 MILDURA - VIC 149 4 2.68% 85 210,203 -214,821 -4,618 ALFREDTOWN - NSW 150 4 2.67% 67 242,365 -253,479 -11,114 ALFRED COVE - WA 374 1 0.27% 60 20,000 -20,278 -278 CLOVERDALE - WA 480 1 0.21% 70 267,700 -270,841 -3,141 KINGSLEY - WA 1206 3 0.21% 54 229,127 -233,828 -4,701 /// Another 500 Lines ///
- 4. Development of SAS Geospatial software allows easy mapping any data containing Australian Postcodes SAS Geospatial Software The development of geospatial software was created in- house. Based on mapping information freely available from the ABS and Australia Post, we have developed a user friendly approach to very quickly produce an Australian map many different statistics. The analyst only needs to produce a table containing postcode level data, including the information required for charting. Once this data is linked into the software, it takes less that 1 minute to produce a map. Maps can be displayed at State, Fitch Region or Postcode level. Different maps can also be displayed for each state. Sales Targets By Postcode Sales Targets By Fitch Region
- 5. Example: Distribution by Fitch Region Maps can be easy generated To create a map, all that is required is two columns of data - the Postcode, and the attribute you wish to map. The attribute need not represent numerical data, but can also include categorical information. Examples of Maps Include: • Product sales or application volumes • Credit Card Fraud • Default Rates • Scorecard Results • Cross Sale Volume • Loan Utilization Australian Delinquency Rates By Fitch Region
- 6. Data can be summarised at various levels and the analyst can easily create options to create different desired maps. Customisation Through SAS Enterprise Guide An Enterprise Guide project can easily be set up to handle any segmentation or statistics. In this case, the project has been created for easy re-use. The analyst can select the map attributes, including the statistic (mean, count, min, max), the display level (Postcode, State or Fitch Region) and a few other options to create the map desired. This gives the analyst a chance to consider results from many perspectives. The process of producing maps can then be performed without changing any code, making a short learning curve for new starters.
- 7. Allows zooming in of maps dynamically to see greater detail Zooming In Dynamic maps can be produced that enable the analyst to zoom in on regions for finer detail. This is done through a web browser using the Graph Toolbar. Australia Zoom to Central East NSW Zoom Further to Sydney Area
- 8. Auto RAG Vs User Defined RAG Status RAG (Red Amber Green, or Traffic Lights) Status RAG status can be set to predefined values in order to line up with existing triggers. However, often it is necessary to assign a meaningful RAG status for data we have not visualised before. The software can automatically generate RAG status based on percentiles. White: 0 observations Green: Less that 75th percentile Amber: Less than the 90th percentile Red: Higher than the 90th percentile Once the analyst has reviewed the outputs, the RAG status can be manually set at the levels desired with little programming. Auto RAG User-Defined RAG
- 9. Information Sources: ABS: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1270.0.55.003July%202011?OpenDocument Google Maps API: http://maps.googleapis.com/maps/api/geocode... The MapInfo Interchange File (MIF): Format Specification, October 1999, (Mapinfo_Mif.pdf), MapInfo Corporation Australia Post for Postcode Mappings: http://auspost.com.au What we need to do: • Decode the mapinfo file • Link in the Australia Post postal zones • Create a valid GMAP dataset • Create a DENSITY variable • Link in our postcode data we wish to plot • Run proc gmap WHY NOT USE PROC MAPIMPORT TO PROCESS THE MAPINFO FILE? PROC MAPIMPORT will import native Shapefiles. However, there are limits on the size of the input data it will accept. The ABS map files are too big! Therefore we must reverse engineer the files and create our own data.
- 10. SAS GMap Data Format We use PROC GMAP to create our maps. SAS Expects the mapping data in a particular format. • Boundaries are represented by polygons • The polygon coordinates must be in the order in which they are drawn. I created an index variable, which allows me to sort the polygon data regardless of the subset chosen. • Boundaries between joining polygons must align • Polygon boundaries must end with X and Y = missing • See SAS Help – The GMAP procedure for further specifications. GREDUCE is used to determine common boundaries and create a density variable which enables you to specify the level of detail of your map. 1= very granular, 6 = high detail. • The input map data set must be a traditional map data set and contain these variables: • a numeric variable named X that contains the horizontal coordinates of the map boundary points. • a numeric variable named Y that contains the vertical coordinates of the map boundary points. • X and Y correspond to the Latitude and longitude of the map • one or more identification variables that uniquely identify the unit areas in the map. These variables are listed in the ID statement. It also can contain: • one or more variables that identify groups of unit areas (for BY-group processing) • the variable SEGMENT, which distinguishes noncoterminous segments of the unit areas. • Any other variables in the input map data set do not affect the GREDUCE procedure. Source: SAS Help – GREDUCE Procedure Perfect alignment here
- 11. Mapinfo Data Format VERSION 450 DELIMITER "," CoordSys Earth Projection 1, 116 Bounds (96,-45) (160,-8) COLUMNS 4 SSC_CODE_2011 char(5) SSC_NAME_2011 char(45) CONF_VALUE char(12) AREA_ALBERS_SQKM float DATA BRUSH(1,0) REGION 1 6 149.849264992 -36.6253990025 149.848946016 -36.6258159925 149.848335008 -36.626610993 149.84918 -36.6242170005 149.849251008 -36.624721995 149.849264992 -36.6253990025 BRUSH(1,0) REGION 2 6 148.934595296 -36.2314241865 148.934515232 -36.231766048 148.935468832 -36.233815737 153.342782016 -29.480945996 153.342276 -29.481940001 153.342455008 -29.4828790055 BRUSH(1,0) REGION 3 5 152.113704064 -32.7755481515 152.114273696 -32.775713449 152.114248096 -32.774133364 152.11393104 -32.7749443485 152.113704064 -32.7755481515 5 152.080612992 -32.789413994 152.080636 -32.7895320055 152.080721984 -32.7893500025 152.080646016 -32.789302994 152.080612992 -32.789413994 3 152.079690016 -32.789695009 152.07966 -32.7895769975 152.079690016 -32.789695009 BRUSH(1,0) NONE NONE There are 2 Mapinfo files - .mid and .mif. We are only interested in the .mif. This is a text file which consists of: • Header • Various pen commands (control colour etc) • Commands to designate each region • The polygon definitions for each region – each region can have several polygons (e.g. there may be a lake inside the main polygon). • A number on a line by its own, designating the number of points in the polygon. • Ends with a NONE command To process the file we: • Ignore the header and BRUSH commands • SCAN each line • If we find a REGION label, increment out REGION counter • If we find an integer on its own, increment a POLYGON counter, but keep it associated with the current region • Any other case, extract the LAT and LONG coordinates • Stop processing when we find the NONE label. These regions only have 1 polygon This region consists of 3 polygons
- 12. Creating Your Map Using the GMAP procedure PROC SORT DATA = WORK.MAP_DATA OUT = WORK.MAP_DATA_SORTED; BY STATE REGION INDEX; RUN; PROC GMAP MAP = WORK.MAP_DATA_SORTED ; ID REGION; CHORO MAP_DATA_FORMATED / MISSING COUTLINE= GREY; BY STATE ; RUN; Standard Map With a little extra coding you can created animated maps Melbourne
- 13. Using Google Maps API to plot street addresses If you don’t have the coordinate data for customer addresses, you can feed it to Google Maps . You can then overlay the points on a map; Advantages: Free, the API will generally interpret misspelling correctly Disadvantages: Can only download 1,200 addresses per day – subject to a misuse policy. Sample address to submit to Google API: http://maps.googleapis.com/maps/api/geocode/xml?address=5 jessel place,duncraig,wa,AUSTRALIA&sensor=true Returns XML data containing the Lat and Long of the address <?xml version="1.0" encoding="UTF-8" ?> - <GeocodeResponse> <status>OK</status> - <result> <type>street_address</type> <formatted_address>5 Jessel Pl, Duncraig WA 6023, Australia</formatted_address> - <address_component> <long_name>5</long_name> <short_name>5</short_name> <type>street_number</type> </address_component> BLAH BLAH BLAH - <southwest> <lat>-31.8431920</lat> <lng>115.7773300</lng> </southwest> - <northeast> <lat>-31.8404940</lat> <lng>115.7800280</lng> </northeast> </viewport> </geometry> </result> </GeocodeResponse>
- 14. Using Google Maps API Using Excel and VBA to download addresses: Using VBA to automate the download of addresses is straight forward using the QueryTables collection object. The results can be fed into a SAS table and overlayed on a map using the ANNOTATE option to GMAP. Sub getGeoData() For Each q In Sheets("google").QueryTables Debug.Print q.Name q.Delete Next q startRow = 26510 resultsStart = 27763 Address = Data.Range("H" & startRow) conn = "URL;http://maps.googleapis.com/maps/api/geocode/xml?address=" & Address & "&sensor=true" With Google.QueryTables.Add(Connection:=conn, Destination:=Google.Range("B1")) .Refresh (False) End With For r = startRow To 30000 If Data.Range("H" & r) <> Data.Range("H" & r - 1) Then Address = Data.Range("H" & r) conn = "URL;http://maps.googleapis.com/maps/api/geocode/xml?address=" & Address & "&sensor=true" With Google.QueryTables(1) .Connection = "URL;http://maps.googleapis.com/maps/api/geocode/xml?address=" & Address & "&sensor=true" .Refresh (False) End With If Google.Cells(3, 2).Value = "OVER_QUERY_LIMIT" Then Exit Sub If Google.Cells(3, 2).Value <> "ZERO_RESULTS" Then Results.Cells(resultsStart, 1) = Data.Cells(r, 1) Results.Cells(resultsStart, 2) = Google.Cells(50, 1) Results.Cells(resultsStart, 3) = Google.Cells(51, 1) resultsStart = resultsStart + 1 End If Google.Cells(52, 1) = r Google.Cells(53, 1) = r - startRow End If DoEvents Next r End Sub %MACRO dot( x1, y1, rad, colin, fill ); /*--------------------------------------------------------------------------*/ /* Draw a circle with center at ( X1,Y1 ) of radius RAD. */ /*--------------------------------------------------------------------------*/ X = &x1; Y = &y1; LINE = 0; ANGLE = 0.00; ROTATE = 360.00; SIZE = &rad; STYLE = "&fill"; IF "&colin" =: '*' THEN ; ELSE color = "&colin"; FUNCTION = "PIE"; output; %MEND dot; %let size_cur = 0.05; %let size_30 = 0.09; %let size_60 = 0.11; %let size_90 = 0.15; data ANNO; length function style color $ 8 position $ 1 ; retain xsys ysys "2" hsys "3" when "a" ; set WORK.CORD_MARKER; position = "E"; segment = 1; if days_buck = -1 then do; %dot( X, Y, &size_cur , green, solid ); end; else if days_buck = 1 then do; %dot( X, Y, &size_30 , orange, solid ); end; else if days_buck = 30 then do; %dot( X, Y, &size_30 , blue, solid ); end; else if Days_Buck = 60 then do; %dot( X, Y, &size_60 , purple, solid ); end; else do; %dot( X, Y, &size_90 , red, solid ); end; run;
- 15. Issues: - Gaps in map where there is a national park (e.g. 1/3 of Tasmania missing!) - Some SSD to Postcode mappings are incorrect - Some postcodes overlap states - No feature maps How much work was involved While the build took place over 2 years (not full time!), development is quite straight forward. The hardest part was finding the best map file on ABS, then linking that data to the Australia Post postcode file. There are inconsistencies in the ABS data which had to be rectified (e.g. postcodes in the wrong position or overlapping). Once the base geospatial files were finalised, it is a relatively short process to create a semi-automated solution in Enterprise Guide.