List Manipulation in FME
4 likes6,241 views
The document discusses list manipulation techniques in FME, highlighting the use of various transformers such as listexploder, listbuilder, and custom transformers. It outlines methods for processing list elements, including 'the classic', 'the loop', and 'the python', each with its benefits and drawbacks. The document also addresses challenges related to nested lists and includes performance benchmarks for different processing methods.
List Manipulation in FME
- 1. List Manipulation in FME Joanna Hobbins
- 2. JOANNA HOBBINS Subject Matter Expert: Content Creation Global Engineering, Immersive Environment
- 3. UNMATCHED ON A GLOBAL SCALE COUNTRIES 35+ TRAINING CENTRES 65+ LOCATIONS 160 EMPLOYEES 8,000+
- 4. Lists Lists are everywhere in FME: • 15 transformers are dedicated to list manipulation • Over 80 transformers can produce lists But there’s no such thing as a list data type in FME Working with Wait… What?
- 5. With a specific naming convention Lists are simply attributes As long as the attribute name contains {#} FME will treat it as a list. You can use regular attribute manipulators to create a list.
- 6. Vocabulary A Little Basic listSimple Complex Nested List with sub-elements _animals{0} Cat _animals{1} Dog _pets{0}.Name Felix _pets{0}.Type Cat _pets{1}.Name Fido _pets{1}.Type Dog Multi-level list _relationships{0}.pass{0} Contains _relationships{0}.pass{1} Intersects _relationships{1}.pass{0} Intersects _relationships{2}.pass{0} Intersects
- 7. Processing Every List Element There are lots of situations where you might want to process every list element individually. We’re going to examine three different methods of doing so. The Classic The Loop The Python
- 8. The Classic Pros • Easy to setup • Access to all transformers Cons • Loss of geometry (ListBuilder) or duplication of geometry (Aggregator) • Special handling required for non-list attributes • Does not scale well
- 9. The Classic Methodology 1. Create unique ID 2. Explode the list: ListExploder 3. Perform the manipulation(s) on each feature 4. Recombine the list: ListBuilder or Aggregator Grouping by unique ID 5. Restore geometry/non-list attributes
- 10. The Loop Pros • No issues with geometry or non list attributes • More efficient than classic method for shorter lists Cons • Requires a linked custom transformer if element manipulation involves blocking transformers • Inefficient with large lists
- 11. The Loop Methodology 1. Create a Custom Transformer 2. Create an attribute to hold the current index 3. Check the index is less than the length of the list 4. Perform the manipulation(s) using the current index 5. Increment the index 6. Loop to Step 3
- 12. Place your screenshot here The Python Pros • Extremely efficient • No attribute or geometry issues Cons • Requires coding skills • Not all transformers have an equivalent python method
- 13. Place your screenshot here The Python Methodology 1. Get the list attribute 2. Perform the manipulations 3. Set the list attribute 4. Expose the list attribute if creating a new list Is it really that simple?
- 14. Place your screenshot here Gotchas: Types feature.getAttribute() will often return numbers as strings. You may have to explicitly cast variables back and forth between types.
- 15. Place your screenshot here Gotchas: Types feature.setAttribute() will not work directly on numeric lists. You can either cast the list to string or use a for loop.
- 16. Place your screenshot here Gotchas: Nested List Nested lists cannot be accessed directly. You can retrieve the sublist for a specific element of the main list, or a specific element of the sublist for all elements of the main list. To use all elements of both lists,you need to loop through the main list. But the length of the list cannot be accessed directly.
- 17. Place your screenshot here Gotchas: Nested List Hybrid Solution • Use a ListElementCounter prior to the PythonCaller • Use a “for i in range” loop to retrieve the subList
- 18. Benchmarks Features List Items Processing Time in Seconds ListBuilder Aggregator Loop Python 10 10 1.5 0.5 0.5 0.6 10 100 1.5 0.6 0.6 0.6 10 1000 2.8 1.8 2.1 1.0 100 10 2.3 0.7 0.7 0.6 100 100 3.1 1.5 1.7 0.7 100 1000 12.2 9.9 12.8 1.5 1000 10 8.5 2.7 1.8 0.8 1000 100 17.8 12.6 12.3 1.2 1000 1000 101.2 91.5 120.5 6.4 10000 10 73.5 22.8 13.2 1.9 10000 100 156.0 102.4 119.2 6.2 10000 1000 1001.9 916.7 1199.0 53.6
- 19. Custom Transformers involving lists • ListAppender • ListComparer • ListElementRemover • ListExpressionEvaluator • HistogramSummer • AngularLineJoiner* • FeatureSegmentator* • LongestEdgeCalculator* • OverlapPercentList* • VisvalinghamSimplifier* • ListElementFilter • ListSlicer • ListStatisticsCalculator • ListStringReplacer • ListSubsetEnumerator • ListZipper • ListCombiner • ListElementExtractor • ListMerger • ListMissingElementFiller • ListSampler • AttributeListExploder • AttributeListPopulator
- 21. Longest Edge Calculator Uses a variation of the HistogramSummer to get the longest cumulative edge within the Angle Tolerance.
- 22. VisvalingamSimplifier Uses a list to store the triplet triangle areas of the feature, and the ListElementRemover to iteratively reduce vertices.