How to Build Complex Calls to APIs (Without Writing Any Code)
0 likes2,280 views
This document provides an overview of how to build complex calls to APIs without writing code using FME. It discusses features of the HTTPCaller like cookies, concurrency, rate limiting, building SQL-like queries, uploading/downloading files, pagination, and creating custom web tools with custom transformers. Examples are given around traffic cameras, photo processing, and retrieving all jobs from an API. Tips discussed include using other tools like Postman to test workflows and starting simple before adding complexity.
How to Build Complex Calls to APIs (Without Writing Any Code)
- 1. How to Build Complex Calls to APIs (Without Writing Any Code)
- 2. Just getting started with REST APIs? How to Connect to Any REST API (Without Writing Any Code)
- 3. Meet the Presenters Sanae Mendoza FME Server Support Specialist Sienna Emery FME Server Support Specialist
- 4. Agenda What we’re covering today. ● Intro to FME ● Did you know the HTTPCaller could do that? ● Building HTTP “SQL” Queries ● Turning cURL into HTTP Calls ● Building Custom Web Tools ● Pagination in the HTTPCaller
- 5. Our Mission To help you maximize the value of your data.
- 6. FME® Integration Platform Connect. Transform. Automate. FME Desktop Build & Run Data Workflows Get a free trial of FME Desktop and FME Server at safe.com FME Server Automate Data Workflows FME Cloud Hosted FME Server FME Mobile Deliver the Power of FME to Mobile
- 7. Connect Your Data Sources Transform Your Data Automate Your Workflows Integrate and convert information across 400+ applications. 450+ powerful transformers safe.com/transformers | hub.safe.com Build event based workflows in a visual interface
- 8. Did you know the HTTPCaller could do that? New and Advanced Features.
- 9. Cookies A small chunk of data given a user’s web browser by the server. The “cookie” is exchanged back and forth, so that the server can track that user’s requests.
- 10. All About Cookies Cookies are Stateful. Both the client and server keep a record of the ongoing “session” with a unique cookie. Typical Uses for Cookies Authentication. Keeping you logged in. User Personalization. Dark mode or light mode. Tracking. Following you around the internet.
- 11. Maintaining a Session inside a Workspace The HTTPCaller Response Headers (Feature Information)
- 12. Concurrency Multiple requests can be sent and processed at the same time (make your workflows faster!)
- 13. Concurrency / Looping Options Control how many HTTP requests you send at one time Sending one request at a time might slow down your workspace. By increasing the number of concurrent requests allowed, you can process multiple requests at the same time. This can provide significant performance benefits. (Just make sure the HTTP server allows it!)
- 14. HTTPCaller Racing 50 requests 1 request
- 15. Rate Limiting The number of HTTP requests allowed to be made within a given time period.
- 16. Working within server rate limits Some servers limit how many requests can be sent in a time frame. Exceeding those limits can result in an error or temporary block. Control the maximum number of requests that can be made within a time frame to make sure your workspaces always run smoothly. Rate Limiting What is the API rate limit here? 1 per year?
- 17. Building HTTP “SQL” Queries Dynamically build HTTP queries in your workspaces
- 18. API “Databases” Socrata Open Data Portal Online platform for government, non-profit, and NGO datasets. Socrata Open Data API (SODA) A tabular “database” that can be accessed programmically. Webinar: Open Data… Automated! Using FME with Socrata Don’t forget the Socrata Reader and Writer
- 19. Combining HTTP parameters with Structured Query Language (SQL) SoQL (Socrata Query Language) allows for complex queries, functions, or analysis. (See also: GraphQL, SOQL (Salesforce Query Language), PostgREST….) SoQL queries are sent with REST calls just like parameters. Append a query to an API endpoint to pull specific data from an API accessible dataset. API Query Languages e.g. ROUND(123.4545, 2) = 123.4500
- 20. Traffic Camera Demo The Goal I want to pull traffic cameras along my routes, I need to build a SoQL clause to use the Socrata API. The Obstacles I’m not familiar with API query languages and the traffic camera dataset is very large. The Solution I can use FME to dynamically build SoQL clauses so that I can pull just the data I need each time. The Result Socrata is queried for each route. I can leverage SoQL queries to make my workflows more efficient.
- 21. Turning cURL into HTTP Calls What to do when the API documentation is written in cURL
- 22. API Documentation Ever get confused by API documentation like this? Some API documentation may be written in cURL or a programming language Question of the Week: Web Services, cURL, and the HTTPCaller
- 23. cURL ● Client URL ● A command line tool that is most likely already installed on your machine ● Works for HTTP calls, FTP, and much more! ● FME uses it in the HTTPCaller Want to learn more?
- 24. Using Postman Using Postman (a free tool!) cURL can be automatically transformed into HTTP requests
- 25. Understanding cURL https://idratherbewriting.com/learnapidoc/docapis_understand_curl.html cURL Description -X HTTP Method -H Header -F Form content. This causes curl to POST data using the Content-Type multipart/form-data Need more?
- 26. Demo Intro Colorize through deep learning API
- 27. Uploading Attachments There are many ways to upload files using the HTTPCaller. Read the API documentation to determine which to use!
- 28. Downloading Attachments Files can be downloaded directly in the HTTPCaller! This sends the response body to a file instead of an attribute.
- 29. Photo Processing Demo The Goal I have a series of photos I’d like to send to an API for processing. The Obstacles I’m not a programmer and I’m unsure how to use cURL. The Solution I can use FME and Postman to break down this cURL into an HTTP Request I can easily use in the HTTPCaller The Result I can now automatically process photos without writing any code or use a command line tool!
- 30. Building Custom Web Tools Introducing Custom Transformers
- 31. A sequence of standard transformers becomes… a custom transformer!
- 32. 5 reasons to Custom Transformers 1. Organization. Separate workspace content into tabs on the canvas. 2. Flexibility. Use existing custom transformers as a prototype, or tweak your own. 3. Efficiency. Reuse sequences of transformers across workspaces. 4. The FMX file. Share it with others or publish it to FME Hub. 5. Custom. Develop personalized FME tools that fit your workflows.
- 33. Embedded transformers are stored with the workspace. Linked transformers’ definitions are stored as an external file (.fmx). If the definition changes, all workspaces will be updated to reflect those changes. Where do I find them? The Transformer Gallery in FME Workbench or The FME Hub, a library of custom transformers and resources. How do I make one? Select a group of transformers, right click and select “Create Custom Transformer....” Give it a name, description, and set up user parameters.
- 34. Building Custom Web Tools in FME The Goal I need a tool that sends text messages via API so that I can quickly add it to my organization's workspaces. The Obstacles There is not a transformer that fits my needs and it is not convenient to configure the API request each time. The Solution I can “wrap up” a sequence of transformers, with an API request, into a convenient package. The Result I can reuse and share a custom transformer that accepts, sends, and processes my API requests.
- 35. Pagination in the HTTPCaller Getting all the data you need!
- 36. Pagination If an API is returning many results, the results are often broken down into pages so the responses from the API are kept short. This is similar to a web page displaying results on different pages to not overwhelm users.
- 37. Pagination Variables Just like this page on FME Hub, HTTP calls often let you select the number of results per page and how many pages you’d like to return.
- 38. Using pagination in FME I’d like to retrieve FME Server job statistics, but the FME Server REST API will only let me retrieve 1000 jobs at a time. How would I retrieve all jobs from a certain date in FME?
- 39. Example attributes for pagination Attribute Definition limit (integer) limits how many items are returned by this call. Could set it to 10 to only retrieve 10 jobs at a time. offset (integer) Offset of this results page
- 40. To loop or not to loop Typically, getting all the results from every page in a REST call requires a loop in a custom transformer. However, you don’t necessarily need a loop if you know how many pages you need in advance.
- 41. Okay, I need to loop ● In FME looping functionality can be found in custom transformers ● Loops can be added by right clicking on the canvas and selecting Insert Transformer Input, then Insert Transformer Loop
- 42. Looping Considerations ● Without the right logic, it is very easy to create an infinite loop! ● Avoid blocking transformers if possible. Blocking transformers can only be used in linked transformers which are much more difficult to troubleshoot ● Set the HTTPCaller concurrency to 1
- 43. Pagination demo The Goal I’d like to pull data from all FME Server jobs around a certain date The Obstacles I need to make multiple calls to the FME Server API to get all results The Solution I can use a loop in a custom transformer to get all the results The Result Using FME, I can get all the results I need in an easy way
- 44. Tips & Tricks ● The documentation should have all you need to know about working with REST APIs ● External tools like Postman can be an easy way to test the workflow ● Start small and build complexity ● Developer tools in a browser can be used to help understand the calls being made ● Fiddler can be used to track requests and response
- 45. Which REST API service are you using? How can we help you get connected?
- 46. Poll: Which REST API training content are you most interested in? ● Creating your own Web Services ● Debugging HTTP errors ● Web Mapping and Applications ● Parsing JSON ● Other (chat out anything missed!)
- 47. Getting Started Download FME 2021.0 Download the workspaces from this webinar Try connecting to a REST API! Free Trial | Upgrade
- 48. Webinars How to Connect to Any API (Without Writing Any Code) A Step-By-Step Guide to Building Codeless Web Apps Automating the Enterprise with Event-Driven Workflows Useful Links More Resources... Getting Started with APIs FME Server REST API Training Creating Web Connections FME Hub FME Community
- 49. Q&A
- 50. Thank you! Try out your REST API workflow today! Connect with us for more FME