Automating Cloud Operations - Everything you wanted to know about cURL and REST APIs

MICHIGAN ORACLE USERS SUMMIT 2020
MONDAY OCTOBER 26 – THURSDAY OCTOBER 29, 2020
VIRTUAL EVENT
AUTOMATING CLOUD OPERATIONS
EVERYTHING YOU WANTED TO KNOW ABOUT CURL AND REST APIS
PRESENTER NAME: AHMED ABOULNAGA
COMPANY:

© Revelation Technologies Group, Inc. 2020 | All rights reserved. Slide 4 of 56
@Revelation_Tech
Table of Contents
Introduction 3
REST and Cloud 6
Introduction to REST 9
Introduction to cURL 23
Creating an Oracle Autonomous Database 30
Figuring Out Authentication and Headers 41
Demo 56

@Revelation_Tech
INTRODUCTION

@Revelation_Tech
About Me
Ahmed Aboulnaga
• Master’s degree in Computer Science from George Mason University
• Recent emphasis on cloud, DevOps, middleware, security in current projects
• Oracle ACE Pro, OCE, OCA
• Author, Blogger, Presenter
• @Ahmed_Aboulnaga

@Revelation_Tech
REST AND CLOUD

@Revelation_Tech
Cloud APIs
• All cloud vendors provide some type of API to their services.
• This allows for programmatic access to cloud services.
• A basic understanding of cURL, REST, and JSON is helpful.
• Most cloud providers use the REST architectural style for their APIs.
Client REST API Cloud Service
JSON / XML
GET / POST / PUT / DELETE

@Revelation_Tech
Goal of this Presentation
• Better understanding REST and JSON.
• Understand and interpret REST API documentation.
• Familiarization with authorization when calling REST APIs.
• Become capable of automating your cloud operations through REST APIs.
Target
Audience
https://globalacademy-eg.com/training-courses/oracle-dba.htm

@Revelation_Tech
MY STORY

@Revelation_Tech
My Use Case
What I was trying to do?
• Create an Identity Domain in
Oracle Access Manager.
• OAM 12.2.1.3 had a web-
based console for all OAuth
configuration.

@Revelation_Tech
My Use Case
What was my challenge?
• OAM 12.2.1.4 provided
no web-based interface
and only supported a
REST API.
https://docs.oracle.com/en/middleware/idm/access-manager/12.2.1.3/oroau/op-oam-services-rest-ssa-api-v1-oauthpolicyadmin-oauthidentitydomain-post.html

@Revelation_Tech
Start with the Documentation
Was the documentation helpful?
• Not all REST API
documentation is created
equally.
• Fortunately, the OAM REST
API documentation provided
an example request and
example response.

@Revelation_Tech
Preparing to Login
What did I do first?
• Passwords can be passed as a
“username:password”
combination or encoded.
• Encoding converts literal text to
a humanly unreadable format.
• Used online to encode
“weblogic:welcome1”.
Authentication Options:
curl -u 'weblogic':'welcome1'
curl -H 'Authorization:Basic d2VibG9naWM6d2VsY29tZTE='
https://www.base64encode.org

@Revelation_Tech
Did it work?
• Initial invocation failed command:
curl -H 'Content-Type: application/x-www-form-urlencoded'
-H 'Authorization:Basic d1VibG9naWM6d2VsY29tZTE='
--request POST
http://soadev.revtech.com:7701/oam/services/rest/ssa/api
/v1/oauthpolicyadmin/oauthidentitydomain
-d '
{
"name" : "AhmedWebGateDomain",
"identityProvider" : "AhmedOUDStore",
"description" : "Ahmed OIDC Domain"
}
'
• Output:
Mandatory param not found. Entity - IdentityDomain,
paramName - name
• The documentation states that only “name” is mandatory.
So what’s the problem?

@Revelation_Tech
What was the successful outcome?
• Final successful command:
curl -H 'Content-Type: application/x-www-form-urlencoded' -H 'Authorization:Basic
d1VibG9naWM6d2VsY29tZTE='
'http://soadev.revtech.com:7701/oam/services/rest/ssa/api/v1/oauthpolicyadmin/oauthidentit
ydomain' -d '{"name":"AhmedWebGateDomain", "identityProvider":"AhmedOUDStore",
"description":"Ahmed OIDC WebGate Domain",
"tokenSettings":[{"tokenType":"ACCESS_TOKEN","tokenExpiry":3600,"lifeCycleEnabled":false,"
refreshTokenEnabled":false,"refreshTokenExpiry":86400,"refreshTokenLifeCycleEnabled":false
},
{"tokenType":"AUTHZ_CODE","tokenExpiry":3600,"lifeCycleEnabled":false,"refreshTokenEnabled
":false,"refreshTokenExpiry":86400,"refreshTokenLifeCycleEnabled":false},
{"tokenType":"SSO_LINK_TOKEN","tokenExpiry":3600,"lifeCycleEnabled":false,"refreshTokenEna
bled":false,"refreshTokenExpiry":86400,"refreshTokenLifeCycleEnabled":false}],
"errorPageURL":"/oam/pages/servererror.jsp", "consentPageURL":"/oam/pages/consent.jsp"}’
• Impossible to determine accurate command without support, examples, or thorough
documentation.
• Required Oracle Support assistance to get these details.

@Revelation_Tech
INTRODUCTION TO REST

@Revelation_Tech
What is REST?
• REpresentational State Transfer.
• Architectural style for distributed hypermedia system.
• Proposed in 2000 by Roy Fielding in his dissertation.
• Web Service implemented with REST is called RESTful web service.
• REST is not a protocol like SOAP
. It is rather an architectural style.
• REST services typically use HTTP/HTTPS, but can be implemented with other
protocols like FTP
.

@Revelation_Tech
REST Architectural Considerations
Uniform interface: Easy to understand and readable results and
can be consumed by any client or programming language over
basic protocols.
URI-based access: Using the same approach to a human
browsing a website where all resource are linked together.
Stateless communication: Extremely scalable since no client
context is stored on the server between requests.

@Revelation_Tech
REST Methods
• The HTTP protocol provides multiple methods which you can utilize for RESTful web services.
• The table maps the HTTP method to the typical REST operation.
• Some firewalls may limit some HTTP methods for security reasons.
HTTP Method REST Operation
GET Read
POST Create
PUT Update
DELETE Delete
OPTIONS List of available methods
HEAD Get version
PATCH Update property/attribute
Most common
in web
applications
Most common in
REST to provide
CRUD
functionality

@Revelation_Tech
Resources
• Requests are sent to resources (i.e., URLs).
• Each resource represents an object which identified by a noun (e.g., employee, phone,
address, etc.).
• Each resource has a unique URL.
• Typically when performing a POST (create) or PUT (update), you must pass additional values.
Resource HTTP Method REST Output
https://hostname/hr/employee GET Retrieve a list of all employees
https://hostname/hr/employee/12 GET Retrieve details for employee #12
https://hostname/hr/employee POST Create a new employee
https://hostname/hr/employee/12 PUT Update employee #12
https://hostname/hr/employee/12 DELETE Delete employee #12
https://hostname/hr/employee/12/address GET Retrieve address for employee #12

@Revelation_Tech
HTTP Response Codes
• HTTP response codes determine the overall response of the REST invocation.
HTTP Code Status Description
2XX (200, 201, 204) OK Data was received and operation was performed
3XX (301, 302) Redirect Request redirected to another URL
4XX (403, 404) Client Error Resource not available to client
5XX (500) Server Error Server error

@Revelation_Tech
JSON
• JavaScript Object Notation.
• Pronounced “Jason”.
• An object surrounded by { }.
• An array or ordered list.
• REST can support both
JSON and XML.
• Less verbose than XML, but
lacks metadata support.
//JSON Object
{
"employee": {
"id": 12,
"name": "Kobe",
"location": "USA"
}
}
//JSON Array
{
"employees": [
{
"id": 12,
"name": "Kobe",
"location": "USA"
},
{
"id": 13,
"name": "Jordan",
"location": "Canada"
},
{
"id": 14,
"name": "Barkley",
"location": "USA"
}
]
}

@Revelation_Tech
INTRODUCTION TO CURL

@Revelation_Tech
What is cURL?
• Open-source command-line tool.
• Supports more than 22 different
protocols (e.g., HTTP
, HTTPS, FTP
, etc.).
• For HTTP
, supports all methods (e.g.,
GET, POST, PUT, DELETE, etc.).
• Very useful for testing RESTful web
services.
• Other advanced tools available include
Postman, SoapUI, Oracle SQL Developer,
etc.
• Example service:
https://api.weather.gov/alerts/active?area=MI

@Revelation_Tech
Viewing Header Attributes
• Fetch the HTTP header attributes only using the –i or -I option:

@Revelation_Tech
Authentication Options
• Many authentication options are supported; access token, bearer token, OAuth,
basic auth, username/password, client certificate, etc.
Authentication Examples:
curl -H 'token:T3jaD5aDHJLr3idjCma894DAWzk49opp'
curl -u 'weblogic':'welcome1'
curl -H 'Authorization:Basic d2VibG9naWM6d2VsY29tZTE='
Token-based Authentication:

@Revelation_Tech
Postman
• Popular API client.
• Free version available.
• www.postman.com
• Numerous features that
include:
- Create API
documentation
- Automated testing
- Design and mock APIs
- Monitor APIs
- Etc.

@Revelation_Tech
Benefits of cURL
• Free.
• Command-line based tool.
• Useful for non-interactive scripts.
• Can pass HTTP headers, cookies, and authentication information.
• Support for SSL, proxy, numerous protocols (e.g., LDAP
, SMB, SCP
, IMAP
, FILE,
TELNET, etc.), etc.

@Revelation_Tech
Recap
• Recap of previous cURL request:
curl -H 'Content-Type: application/x-www-form-urlencoded'
-H 'Authorization:Basic d1VibG9naWM6d2VsY29tZTE='
-X POST
http://revtech.com:7701/oam/services/rest/ssa/api/v1/oauthpolicyadmin/oauthidentitydomain
-d '
{
"name" : "AhmedWebGateDomain",
"identityProvider" : "AhmedOUDStore",
"description" : "Ahmed OIDC Domain"
}
'

@Revelation_Tech
CREATING AN AUTONOMOUS DATABASE

@Revelation_Tech
Manually Create an Autonomous Database (1 of 4)

@Revelation_Tech

@Revelation_Tech
Navigate to Documentation
• CreateAutonomousDatabase Reference
https://docs.cloud.oracle.com/en-us/iaas/api/#/en/database/20160918/AutonomousDatabase/CreateAutonomousDatabase
Note:
• API reference
• CreateAutonomous
Database operation
• REST API endpoint
• API version

@Revelation_Tech
View Resource Details and Example
• The resource details provides a list of all required parameters, often beyond what is
demonstrated in the example.
• Use the example as a starting point.
{
"compartmentId" : "ocid1.tenancy.oc1..d6cpxn…dbx",
"displayName" : "MOUS DB 2020 Auto",
"dbName" : "MOUSDBAUTO",
"adminPassword" : "Kobe_24_24_24",
"cpuCoreCount" : 1,
"dataStorageSizeInTBs" : 1
}

@Revelation_Tech
First Attempt… Failed!
• Unable to authenticate upon first try despite all parameters/settings correct per
the documentation…

@Revelation_Tech
Execute Command
• Eventually got it working…

@Revelation_Tech
Provisioning…

@Revelation_Tech
Available!

@Revelation_Tech
FIGURING OUT AUTHENTICATION &HEADERS

@Revelation_Tech
Discovery is Painful
API References and
Endpoints
https://docs.cloud.oracle.com/en-
us/iaas/api/#/en/database/20160918/
Oracle Cloud Infrastructure
Documentation
https://docs.cloud.oracle.com/en-
us/iaas/Content/API/Concepts/apisigningkey.htm#How
Managing Autonomous
Data Warehouse Using oci-
curl
https://blogs.oracle.com/datawarehousing/managing-
autonomous-data-warehouse-using-oci-curl
Oracle Cloud Infrastructure
(OCI) REST call
walkthrough with curl
https://www.ateam-oracle.com/oracle-cloud-
infrastructure-oci-rest-call-walkthrough-with-curl
But why didn’t
the docs point
me to this?
Found this on my
own, has some
helpful info… I don’t want
to use “oci-
curl”!
This is
complicated!
Thank God they
have a script!
Blog?
Another
blog?

@Revelation_Tech
Piecing It Together
If you want to use cURL to invoke OCI REST APIs…
1. Get information from the OCI Console
a. Get the Tenancy ID
b. Get the User ID
2. Generate and configure an API Signing Key
a. Create public/private key pair
b. Get the fingerprint of the key
c. Get the public key from the private key in PEM format
d. Add the API Key to the OCI user (by uploading the public key)
3. Prepare and execute script
a. Ensure private key is available
b. Create the JSON request
c. Update the custom script
d. Execute!

@Revelation_Tech
1a. Get the Tenancy ID

@Revelation_Tech
1b. Get the User ID

@Revelation_Tech
2a. Create public/private key pair
• Use ssh-keygen to create a public/private key pair.
• The public key will be added as an “API Key” to your OCI account.
• The private key will be used by your client (i.e., cURL).

@Revelation_Tech
2b. Get the fingerprint of the key
• Use openssl to view the X.509 MD5 PEM certificate fingerprint.

@Revelation_Tech
2c. Get public key from the private key in PEM format
• OCI requires that the public key is imported in PEM format.
• Use openssl to get the public key in PEM format.

@Revelation_Tech
2d. Add the API Key to the OCI user
• The public key is added to the OCI user’s API Key.
• Must be in PEM format.
• Can be uploaded or pasted.

@Revelation_Tech
3a. Ensure private key is available
• The private key created earlier (and in PEM format) is used when invoking the
REST service.

@Revelation_Tech
3b. Create the JSON request
• The payload is created in JSON format.

@Revelation_Tech
3c. Update custom script
• The various elements (tenancy
id, user id, private key, key
fingerprint, etc.) are
parameterized.
• OCI’s REST API requires
additional calculated elements,
all taken care of here (oci-curl
takes care of all of this for you).
• The cURL command is eventually
called using a combination of
static and dynamic values.

@Revelation_Tech
3d. Execute!
• The cURL command
is expanded.
• The cURL command
is executed.
• The HTTP status
code is observed to
obtain the result of
the invocation.

@Revelation_Tech
Provisioning…

@Revelation_Tech
Available!

@Revelation_Tech
DEMO

@Revelation_Tech

Automating Cloud Operations - Everything you wanted to know about cURL and REST APIs

More Related Content

What's hot (20)

Similar to Automating Cloud Operations - Everything you wanted to know about cURL and REST APIs (20)

More from Revelation Technologies (15)

Recently uploaded (20)

Automating Cloud Operations - Everything you wanted to know about cURL and REST APIs