Choosing the Right Transformer for Your Data Challenge
3 likes7,373 views
The document outlines a webinar featuring a team of experts discussing FME transformers, user-submitted scenarios, and solutions for various data processing challenges. It includes resources and guides for utilizing over 400 transformers, as well as specific user scenarios that highlight key transformers used to solve complex problems. Additionally, it encourages users to participate in the FME Knowledge Center for more learning and collaboration.
Choosing the Right Transformer for Your Data Challenge
- 2. Team of Experts Dale Lutz Co-Founder and VP of Software Development Iris Gutowski Web Developer, Web GUI Team Lead Mark Ireland FME Evangelist
- 3. Team of Experts (on Q&A) Trent Kading FME Technology Expert Nathan Hildebrand FME Technology Expert Debbi Leung FME Desktop Technology Expert Stephanie Warner Webinar Expert
- 4. Topics Covered ● Resources on FME Transformers ● Top 10+ Transformers - Expert recommended! ● Data specific Transformers and FME user submitted scenarios ● Live Q+A in today’s webinar #SafeWebinar
- 5. Resources Quick Add in FME Workbench Transformer Gallery: safe.com/transformers Transformer Guide Booklet fme.ly/TransformerGuide18 Great article for merging transformers: fme.ly/mergingtransformers The best places to learn about FME’s 400+ transformers
- 6. New to FME? On the best place to learn FME: Our Free training courses on our virtual machines On transformer essentials: Review the top 30 transformers listed here: fme.ly/top30 On dealing with a specific type of data? Try typing that on the Workbench canvas to search (e.g. PointCloud, Raster) Or maybe you just want to refresh the basics. Here’s what our experts have to say:
- 7. ARE YOU UP FOR A CHALLENGE? Calling all users in the FME Knowledge Center Sign up for free at knowledge.safe.com
- 8. User Scenarios Uploaded Movie Magik 1) Grab list of video files from a webpage using many html extractors and list expanders. 2) Compare that list to list of files in a folder 3) Connect to an FTP Server and grab the missing files 4) Connect to YouTube’s API and post the new video to youtube XML eXtravagance “Read online XML - Create as points, remove irrelevant fields etc, export to ESRI geodatabase feature class, or if possible to several Oracle Databases representing our different environments using in ArcGIS Portal. Add metadata to the feature class, updating an XML file which already contains the metadata for the layer in INSPIRE format, use a transformer to update a field in metadata with today’s date….” Do you know which transformers to use? Go to knowledge.safe.com to share! 1 2
- 9. Rad Radar “I would like to be able to analyze raster images of historical radar from NOAA. I would compare the value in the radar image (a geotiff) with a linear feature (electric line). The radar images are stored in 5 minute intervals so I would like to know how often the linear feature intersected with a certain level of radar activity.” User Submitted Scenario #1
- 10. Rad Radar User Submitted Scenario #1 Key transformers used to solve the problem NumericRasterizer: Create a raster from vector input Use for: Calculating statistics based on a designated attribute or set of attributes of the incoming features. Alternative: Convert Raster to vector and determining intersections in Raster Dmitri says: Always stay in raster as long as possible.
- 12. Dynamic Duo “I have been able to create a workspace which can achieve the above, but it is not dynamic. If a new table gets created in the database then the workspace has to be edited to include it. New tables are commonly created so there needs to be a workspace solution which does not reference specific database tables, but the answer is beyond me!” User Submitted Scenario #2
- 13. Dynamic Duo User Submitted Scenario #2 Key transformer used to solve the problem #SafeWebinar
- 14. Anonymous Automation “So the scenario is: 1) how do we read as many as 13 different csv files (all with the same file name) 2) decide which feature type we are dealing with in each csv file 3) write those records for each csv file to the appropriate set of fields in a MS SQL database” User Submitted Scenario #3
- 15. Anonymous Automation User Submitted Scenario #3 Key transformers used to solve the problem Filters features by test conditions to one or more output ports. Removes all attributes and list attributes, except the specific ones you specify to be retained.
- 16. PDF Palooza “Objective: to produce a multi-page PDF – one page for each photo - each page containing 4 objects (plus Title and signature) 1) A photo 2) Large scale location plan 1:5000 3) Small scale location plan 1:100,000 4) Relevant row from points shape file attribute table giving details of featured photo.” User Submitted Scenario #4 #SafeWebinar
- 17. PDF Palooza User Submitted Scenario #4 Key transformers used to solve the problem. Use to: Prepare features for output to PDF Use when: You want to detect a geometric relationship containment, intersection, etc between 2 features Also Consider: HTML ReportGenerator to make dynamic web reports
- 19. SCENARIOS SOLVED? Check in with the FME Knowledge Center Sign up for free at knowledge.safe.com
- 20. Winds of Change “I need to convert this to two new raster bands U and V. I.e. a vector equivalent to the input bands. But how would I apply this calculation to NetCDF files? Next I need to convert this raster file to SQL Inserts that can be used to load data into a database.” User Submitted Scenario #9 #SafeWebinar
- 21. Winds of Change User Submitted Scenario #9 Key transformers used to solve the problem Use to calculate a new raster from values from existing rasters Use to convert raster cell values to individual points Use to turn a points x,y values into attributes
- 23. LAS Vegas “I have about 4000 LAS files and I need to generate DEM’s and Hillshades from them. So far, I can only process them one at a time. I’d like to see how I can automate this process through FME to process all 4k tiles, using the tile name as an output name.” User Submitted Scenario #12 #SafeWebinar
- 24. LAS Vegas User Submitted Scenario #12 Key transformers used to solve the problem Draws input point, line and polygon features onto a numeric raster filled with the background value. Generates a shaded relief effect, useful for visualizing terrain. Constructs a Delaunay triangulation based on input points and breaklines. Mosaics multiple raster features into a single raster feature.
- 25. Path Production “I am trying to take an area feature class representing trails and convert it to a line feature class. The CenterLineReplacer has given me some results, but the data needs a lot of clean up and this is where I am struggling.” User Submitted Scenario #8 #SafeWebinar
- 26. Path Production User Submitted Scenario #8 Key transformers used to solve the problem Use for: Replacing an area feature with its medial axis or a straight skeleton. This transformer works best with long, narrow areas. Use for: Transforming or measuring geometry features based on a specified algorithm.
- 27. List Limbo aka Overlapping Overlay “A. I have 8 different polygon datasets with flood model meshes (which basically are triangles with a depth value) B. I have another 5 polygon datasets with land use, which are not complementary, i.e. they do overlap in places The question to be answered is: how much land (number and area) is affected by flooding ? ...which transformer should I use?” User Submitted Scenario #14
- 28. List Limbo User Submitted Scenario #14 Key transformers used to solve the problem Alters multiple attributes through adding, renaming, copying, deleting and re-ordering. Dissolves area features by removing common boundaries to create larger areas. AreaOnAreaOverlayer: Performs an area-on-area overlay (intersection of polygons) so that all input areas are intersected against each other and resultant area features are created and output Use Tester when: - You want to make a different decision based on an Attributes Value - To split the logic flow into 2 cases IE Test Pass Test Fail
- 29. CAD Conversion The requested workspace would convert a folder of Bentley MicroStation DGNs to ESRI Geodatabases with the following parameters: 1. Create one GDB per DGN with the new GDB having the same name as the original DGN 2. The DGNs have a local coordinate system but have an associated world file in the folder 3. Convert annotation, points, polygons, and polylines while honoring the original levels in the DGN without attaching the annotation to the points, polygons, and polylines. (I need Annotation as a separate Feature Class) 4. The Annotation must honor the sizing in an MXD as it was in the original DGN User Submitted Scenario #11 #SafeWebinar
- 30. CAD Conversion User Submitted Scenario #11 Key transformers used to solve the problem Routes a feature based on its geometry type. Adds one or more attributes to the feature and optionally assigns a value derived from constants, attribute values, and expressions.
- 31. Common theme in Dealing with large data volumes user submitted scenarios: Video on Bulk Data Processing: fme.ly/bulk-data Dynamic Workflow: fme.ly/dynamic-workflow
- 32. Q&A Thank you! If we didn’t get to your scenario, please post it to knowledge.safe.com or tweet us @SafeSoftware
- 33. Transformer Reference Guide Thank you! Get our free fme.ly/TransformerGuide18
- 34. Bonus Notes (Not included in webinar)
- 35. Tester Use tester when: >You want to make a different decision based on an Attributes Value > to split the logic flow into 2 cases IE Test Pass Test Fail Also Consider: TestFilter > If you want to consider branching more than 2 ways consider a TestFilter
- 36. AttributeCreator Use when you want to: Create attributes based on the value of other attributes Use over AttributeManager if you want simplicity and less options Also Consider: AttributeManager Pro Secret: AttributeCreator and Manager are the same under the covers
- 37. FeatureMerger Use when you want to bring data together Allows for a basic join on matching attribute values Can take a combination of ports to produce inner, left right and full outer joins Offers complex ways to assemble geometry as either aggregates or reassembling Use when you need an all - purpose tool Also Consider FeatureJoiner for performance SQL Caller/DBJoiner- Merging in the DB often faster can use SQL InlineQuerier if you know sql for more flexibility can use SQL on non -dbs
- 38. AttributeManager Use when you need to do anything attribute Create, delete, rename and copy Do all your attribute manipulation in 1 place! Keep your workspaces tidy
- 39. Reprojector Use when you want to convert a geometries coordinates to a new coord system you want the actual coord values to change (actual data transformation) Don’t confuse with CoordinateSystemSetter when you have the right coordinates but just need to define a new system Also consider ESriReprojector CSMapReprojector which use different engines
- 40. Creator Use when you need a “dummy” feature to trigger something Want to create a simple test geometry Creator ->Http Caller> … Can bring in a wkt geometry easily Also consider RasterCreator, PointCloudCreator
- 41. AttributeRenamer Does a basic rename Consider combining with other tasks in the AttributeManager
- 42. TestFilter Use when you need to split out features into multiple 2+ paths depending on attribute values IE branch a table out to different writers
- 43. AttributeKeeper Use when you need to clean up Cut your schema down to the set of things you want Can improve performance and reduce memory overhead Also Consider AttributeRemover to specify what to get rid of instead of what to keep
- 44. Clipper Use when you want to clip a geometry to a predefined area Also consider AreaOnAreaOverlayer
- 46. Logger Can be used to troubleshoot tricky data type problems
- 47. Transformer Reference Guide Thank you! Get our free fme.ly/TransformerGuide18