SlideShare a Scribd company logo

curl better

Daniel Stenberg, profile picture
Daniel Stenberg

Curl is an open source command line tool and library for transferring data using various internet protocols. The document discusses curl's history, features, usage, and options. Curl supports many protocols including HTTP, HTTPS, FTP, FTPS, and more. It has over 230 command line options and has been ported to many operating systems. The document provides examples of common curl commands and options for making HTTP requests, handling cookies and redirects, uploading and downloading files, and more.

Technology
1 of 58
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
curl bettercurl better March 30, foss-north 2020March 30, foss-north 2020
Daniel Stenberg @bagderhttps://daniel.haxx.se
Daniel Stenberg @bagder
what curl iswhat curl is option how-tooption how-to Q&AQ&A @bagder@bagder
An open source project that makes a command line tool and a library for transferring data using Internet protocols @bagder@bagder
Features! DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP TLS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, HTTP/HTTPS/SOCKS proxy, cookies, authentication (Basic, Digest, NTLM, Negotiate, Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP compression and much more @bagder@bagder
@bagder@bagder
2,200 contributors Who makes curl curl 770 authors 150 authors per year 12 regulars Daniel @bagder@bagder (The boxes are not drawn to scale)
@bagder@bagder Conquered the world Right time Open Source Portable Backwards compatible Stable
Many places curl has been around since spring 1998 MacOS (since 2001) Windows 10 (since 2018) Linux distros (since 1999) BSDs, other Unixes, MS-DOS, older systems Side-loaded - from the official site or elsewhere Built from source @bagder@bagder
Many flavors Millions of build combinations Thirty different dependencies in combinations Frequent releases Rarely the same in two places! @bagder@bagder
Many protocols and options Today we only talk about the tool named curl curl knows many protocols curl features 230 command line options I stick to HTTP(S) in this presentation Using a small subset of flags @bagder@bagder
Number of command line options @bagder@bagder First curl release 24 230
HTTP RequestRequest - method + path- method + path - headers- headers - body- body ResponseResponse - response code- response code - headers- headers - body- body @bagder@bagder
Under the hood GET / HTTP/1.1 Host: www.example.com Accept: */* User-Agent: HTTP-eats-the-world/2019 HTTP/1.1 200 OK Date: Thu, 09 Nov 2018 14:49:00 GMT Server: my-favorite v3 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT Content-Length: 12345 Set-Cookie: this-is-simple=yeah-really; Content-Type: text/html [content] @bagder@bagder
Minimal behavior by default Basic set of headers No fancy functionality Let users enable more when wanted Toggle features on/off one-by-one @bagder@bagder
Long and short options All short options have a long alternative -s can also be set with –-silent I will use the short options today Many options only exist as long options @bagder@bagder
$ curl example.com <html> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> @bagder@bagder Plain curl HTTP GET
$ curl -i https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder GET and show me the response headers
$ curl https://example.com/json | jq { "id": "kr09ddfgbfsf", "name": "Issues, PRs, Dashboard, Projects", "status": "operational", "created_at": "2017-01-31T20:01:46.638Z", "updated_at": "2019-09-26T14:00:45.562Z", "position": 3, "description": "Fake JSON for an example", "group": false, "only_show_if_degraded": false }, @bagder@bagder GET and show JSON nicely
$ curl -I https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 @bagder@bagder HEAD only shows the response headers HEAD returns no body!
$ curl -I https://example.com/redirected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html location: /a/separate/place content-length: 306 server: server/3000 @bagder@bagder Curl doesn’t follow redirects by default HEAD returns no body!
$ curl -I -L https://example.com/redi rected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT ... HTTP/1.1 200 OK date: Wed, 19 Oct 2019 11:23:06 GMT content-type: text/html content-length: 4676 server: server/3000 @bagder@bagder Please follow redirects HEAD returns no body!
URL Globbing $ curl https://example.com/[1-9].html $ curl https://example.com/[01-99].html $ curl https://example.com/[a-z].html $ curl https://example.com/[1-9:2].html $ curl https://example.com/[a-z:3].html $ curl https://example.com/[1-9].html -o save_#1.html $ curl https://example.com/{ham,cheese,pineapple}.jpg -o hawaii_#1.jpg $ curl http://archive.example/issue[1996-1999]/vol[1-4]/part{a,b,c}.html
$ curl -v https://example.com/ -o /dev/null * Trying 93.184.216.34:443... * Connected to example.com (93.184.216.34) port 443 (#0) … * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org … > GET / HTTP/2 > Host: example.com > User-Agent: curl/7.66.0 > Accept: */* > < HTTP/2 200 < date: Wed, 09 Oct 2019 11:16:06 GMT < content-type: text/html < content-length: 306 < server: server/3000 @bagder@bagder Verbose shows more from under the hood
One -v is enough. More adds nothing!
$ curl https://example.com/ -H "Magic: disc0" $ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7" $ curl https://example.com/ -H "User-agent:" $ curl https://example.com/ -H "User-agent;" @bagder@bagder Pass in custom HTTP headers
$ curl -d name=Daniel -i https://example.com/receiver HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder POST some basic data to the remote
$ curl -d @file https://example.com/receiver -o saved $ ls -l | curl -d @- https://example.com/receiver -o saved $ ls -l | curl --data-binary @- https://example.com/receiver -o saved $ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com @bagder@bagder POST a file
$ curl -T localfile -i https://example.com/remote_name HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder PUT a file
$ curl -T localfile -X SWOOSH https://example.com/remote_name -o save @bagder@bagder With -X we change the method string
curl -X in vain leads to pain https://xkcd.com/386/ @bagder@bagder
$ curl -d postdata -X POST https://example.com/remote_name -o save $ curl -d postdata -X POST https://example.com/remote_name -o save -L @bagder@bagder Bad -X leads to pain Probably not what you wanted!
@bagder@bagder Bad -X -X means use this method in all requests -L means follow redirects Redirects instruct the client what method to use in the next request -X overrides the method the server wants the client to use Use -X if you want a different method than curl would use
@bagder@bagder Avoid -XAvoid -X
@bagder@bagder Cookies are name value pairs Cookies are name=value pairs sent by servers Sent back by clients when the URL matches curl doesn’t know cookies unless you tell it to curl has separate options for reading and writing cookie files
$ curl -c cookiejar.txt https://example.com/ $ curl -b cookiejar.txt https://example.com/ $ cat cookiejar.txt # Netscape HTTP Cookie File # https://curl.haxx.se/docs/http-cookies.html # This file was generated by libcurl! Edit at your own risk. example.com TRUE / FALSE 1602699857 duidd182180ebab48 @bagder@bagder Cookies!
$ curl -c cookiejar.txt https://example.com/login_form $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234 $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile @bagder@bagder Cookies in a login
$ curl https://127.0.0.1/ $ curl -k https://127.0.0.1/ $ curl -k https://127.0.0.1/ -H "Host: example.com" $ curl https://example.com/ --resolve example.com:443:127.0.0.1 $ curl https://example.com/ --connect-to example.com:443:host.tld:8443 @bagder@bagder HTTPS obstacles, locally hosted named site Causes certificate problems Ducks for the cert problem Doesn’t work with cookies Works with cookies Doesn’t work with virtual servers Does TLS and certs correctly Works with cookies Works with virtual servers
curl -k really?
@bagder@bagder Avoid -kAvoid -k
$ curl https://example.com/ -d sendthisdata --trace-ascii - … => Send header, 143 bytes (0x8f) 0000: POST / HTTP/2 000f: Host: example.com 0022: User-Agent: curl/7.66.0 003b: Accept: */* 0048: Content-Length: 12 005c: Content-Type: application/x-www-form-urlencoded 008d: => Send SSL data, 5 bytes (0x5) 0000: ....& => Send SSL data, 1 bytes (0x1) 0000: . => Send data, 12 bytes (0xc) 0000: sendthisdata == Info: We are completely uploaded and fine … @bagder@bagder More details!
Extracting transfer meta-data --write-out (-w) is your friend --write-out ‘text and %{variables} to see’ Variables include:  content_type  http_version  remote_ip  local_ip  speed_download %{json}%{json} Coming soon! Coming soon!
$ curl https://example.com/ -o saved -w ‘%{json}’ | jq @bagder@bagder --write-out %{json} { "url_effective": "https://example.com/", "http_code": 200, "response_code": 200, "http_connect": 0, "time_total": 0.658048, "time_namelookup": 0.001, "time_connect": 0.153556, "time_appconnect": 0.501105, "time_pretransfer": 0.501223, "time_starttransfer": 0.657869, "size_header": 331, "size_request": 77, "size_download": 1256,
$ curl https://example.com/ --libcurl sourcecode.c $ gcc sourcecode.c -lcurl -o ./myapp $ ./myapp $ cat sourcecode.c @bagder@bagder Convert it into an application? /********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0"); curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L); curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS); curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts"); curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L); ...
@bagder@bagder Mimic that browser – copy as curl
$ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/ 20100101 Firefox/69.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en- US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure- Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb- 59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers' $ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/ OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/ *;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb- 59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed @bagder@bagder Mimic that browser – copy as curl
SSLKEYLOGFILE Why trust your tools when you can snoop on them Wireshark is your friend Tell curl or your browser to store its secrets in SSLKEYLOGFILE Tell Wireshark where the secrets are located Run curl or your browser @bagder@bagder
$ export SSLKEYLOGFILE=$HOME/tmp/tlskey $ curl https://example.com @bagder@bagder Snoop on curl
The Wireshark end @bagder@bagder
Many HTTP versions – but they look the same HTTP/1.0 shipped in 1996 We’ve learned how HTTP headers look like HTTP/2 shipped in 2015 HTTP/3 is “coming soon” curl pretends all headers look and work like HTTP/1 @bagder@bagder
$ curl –-http1.0 https://example.com $ curl –-http1.1 https://example.com $ curl –-http2 https://example.com $ curl –-http3 https://example.com @bagder@bagder HTTP versions HTTP/3 support is experimental!
$ curl -d user=daniel https://example.com https://another.example.com $ curl -d user=daniel https://example.com –-next https://another.example.com $ curl https://example.com –-next -d user=daniel https://another.example.com @bagder@bagder Different requests on the same command line!
All this, and much much more... @bagder@bagder
https://curl.haxx.se/book.html @bagder@bagder
56 You can help!You can help! @bagder@bagder
Daniel Stenberg @bagder https://daniel.haxx.se/ Thank you!Thank you! Questions?Questions? @bagder@bagder
License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder

More Related Content

PDF
HTTP/3 is next generation HTTP
Daniel Stenberg
 
PDF
curl roadmap 2020
Daniel Stenberg
 
PDF
HTTP/3 in curl
Daniel Stenberg
 
PDF
HTTP/3 for everyone
Daniel Stenberg
 
PDF
Http3 fullstackfest-2019
Daniel Stenberg
 
PDF
HTTP/3 in curl 2020
Daniel Stenberg
 
PDF
Just curl it!
Daniel Stenberg
 
PDF
Getting started with libcurl
Daniel Stenberg
 
HTTP/3 is next generation HTTP
Daniel Stenberg
 
curl roadmap 2020
Daniel Stenberg
 
HTTP/3 in curl
Daniel Stenberg
 
HTTP/3 for everyone
Daniel Stenberg
 
Http3 fullstackfest-2019
Daniel Stenberg
 
HTTP/3 in curl 2020
Daniel Stenberg
 
Just curl it!
Daniel Stenberg
 
Getting started with libcurl
Daniel Stenberg
 

What's hot (20)

PDF
common mistakes when using libcurl
Daniel Stenberg
 
PDF
HTTP/3, QUIC and streaming
Daniel Stenberg
 
PDF
curl - a hobby project that conquered the world
Daniel Stenberg
 
PDF
Testing curl for security
Daniel Stenberg
 
PDF
HTTP/3
Daniel Stenberg
 
PDF
Landing code in curl
Daniel Stenberg
 
PDF
DNS over HTTPS
Daniel Stenberg
 
PDF
Curl with rust
Daniel Stenberg
 
PDF
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
NETWAYS
 
PPTX
Re-thinking Performance tuning with HTTP2
Vinci Rufus
 
PPT
Zombilizing The Web Browser Via Flash Player 9
thaidn
 
PDF
Altitude SF 2017: QUIC - A low-latency secure transport for HTTP
Fastly
 
ODP
Testing Wi-Fi with OSS Tools
All Things Open
 
PPT
Dos threats and countermeasures
n|u - The Open Security Community
 
PDF
Demystifying Docker Networking Devoxx MA 2017
Imad Hsissou
 
PPTX
Масштабируя TLS / Артём Гавриченков (Qrator Labs)
Ontico
 
PDF
XFLTReat: a new dimension in tunnelling
Shakacon
 
PPTX
Ruby Proxies for Scale, Performance, and Monitoring
Ilya Grigorik
 
PPT
securing_syslog_onFreeBSD
webuploader
 
PPTX
Dock ir incident response in a containerized, immutable, continually deploy...
Shakacon
 
common mistakes when using libcurl
Daniel Stenberg
 
HTTP/3, QUIC and streaming
Daniel Stenberg
 
curl - a hobby project that conquered the world
Daniel Stenberg
 
Testing curl for security
Daniel Stenberg
 
HTTP/3
Daniel Stenberg
 
Landing code in curl
Daniel Stenberg
 
DNS over HTTPS
Daniel Stenberg
 
Curl with rust
Daniel Stenberg
 
stackconf 2020 | Speeding up Linux disk encryption by Ignat Korchagin
NETWAYS
 
Re-thinking Performance tuning with HTTP2
Vinci Rufus
 
Zombilizing The Web Browser Via Flash Player 9
thaidn
 
Altitude SF 2017: QUIC - A low-latency secure transport for HTTP
Fastly
 
Testing Wi-Fi with OSS Tools
All Things Open
 
Dos threats and countermeasures
n|u - The Open Security Community
 
Demystifying Docker Networking Devoxx MA 2017
Imad Hsissou
 
Масштабируя TLS / Артём Гавриченков (Qrator Labs)
Ontico
 
XFLTReat: a new dimension in tunnelling
Shakacon
 
Ruby Proxies for Scale, Performance, and Monitoring
Ilya Grigorik
 
securing_syslog_onFreeBSD
webuploader
 
Dock ir incident response in a containerized, immutable, continually deploy...
Shakacon
 
Ad

Similar to curl better (20)

PPTX
Next Level Curl
Nordic APIs
 
PDF
mastering the curl command line.pdf
DanielStenberg7
 
PDF
Broom not included curling the modern way
DanielStenberg7
 
PDF
mastering libcurl part 1
Daniel Stenberg
 
PDF
curl manual
Anthony Yuan , PMP
 
PDF
Ilugc curl
Akilan Ram
 
PDF
Curl
manasystest
 
PPTX
Curl Tutorial
Ankireddy Polu
 
PDF
From zero to almost rails in about a million slides...
david_e_worth
 
PPTX
Curl
penetration Tester
 
PDF
Ethernet Shield
Tinker London
 
PDF
Ethernet Shield
Tinker
 
PDF
Internet all the things - curl everywhere!
Daniel Stenberg
 
PPTX
wget, curl and scp
Gaurav Mishra
 
ODP
Fscons scalable appplication transfers
Daniel Stenberg
 
PDF
Let me tell you about curl
Daniel Stenberg
 
PPTX
Httpbasics 1207412539273264-9-converted
computerorganization
 
PPT
Http request&response
Aswin Krishnamoorthy
 
PPTX
Web technologies: HTTP
Piero Fraternali
 
PPTX
www and http services
Jenica Salmorin
 
Next Level Curl
Nordic APIs
 
mastering the curl command line.pdf
DanielStenberg7
 
Broom not included curling the modern way
DanielStenberg7
 
mastering libcurl part 1
Daniel Stenberg
 
curl manual
Anthony Yuan , PMP
 
Ilugc curl
Akilan Ram
 
Curl
manasystest
 
Curl Tutorial
Ankireddy Polu
 
From zero to almost rails in about a million slides...
david_e_worth
 
Curl
penetration Tester
 
Ethernet Shield
Tinker London
 
Ethernet Shield
Tinker
 
Internet all the things - curl everywhere!
Daniel Stenberg
 
wget, curl and scp
Gaurav Mishra
 
Fscons scalable appplication transfers
Daniel Stenberg
 
Let me tell you about curl
Daniel Stenberg
 
Httpbasics 1207412539273264-9-converted
computerorganization
 
Http request&response
Aswin Krishnamoorthy
 
Web technologies: HTTP
Piero Fraternali
 
www and http services
Jenica Salmorin
 
Ad

More from Daniel Stenberg (16)

PDF
What comes after world domination with Daniel Stenberg, April 2025
Daniel Stenberg
 
PDF
digital infrastruktur är open source-1.pdf
Daniel Stenberg
 
PDF
Tightening every bolt at FOSDEM 2025 by Daniel Stenberg
Daniel Stenberg
 
PDF
curl security by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
PDF
rust in curl by Daniel Stenberg from- curl up 2024
Daniel Stenberg
 
PDF
trurl 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
PDF
curl future 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
PDF
The state of curl 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
PDF
mastering libcurl part 2
Daniel Stenberg
 
PDF
curl - openfourm europe.pdf
Daniel Stenberg
 
PDF
curl experiments - curl up 2022
Daniel Stenberg
 
PDF
curl security - curl up 2022
Daniel Stenberg
 
PDF
HTTP/3 in curl - curl up 2022
Daniel Stenberg
 
PDF
The state of curl 2022
Daniel Stenberg
 
PDF
The state of curl 2020
Daniel Stenberg
 
PDF
HTTP/3 over QUIC. All is new but still the same!
Daniel Stenberg
 
What comes after world domination with Daniel Stenberg, April 2025
Daniel Stenberg
 
digital infrastruktur är open source-1.pdf
Daniel Stenberg
 
Tightening every bolt at FOSDEM 2025 by Daniel Stenberg
Daniel Stenberg
 
curl security by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
rust in curl by Daniel Stenberg from- curl up 2024
Daniel Stenberg
 
trurl 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
curl future 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
The state of curl 2024 by Daniel Stenberg from curl up 2024
Daniel Stenberg
 
mastering libcurl part 2
Daniel Stenberg
 
curl - openfourm europe.pdf
Daniel Stenberg
 
curl experiments - curl up 2022
Daniel Stenberg
 
curl security - curl up 2022
Daniel Stenberg
 
HTTP/3 in curl - curl up 2022
Daniel Stenberg
 
The state of curl 2022
Daniel Stenberg
 
The state of curl 2020
Daniel Stenberg
 
HTTP/3 over QUIC. All is new but still the same!
Daniel Stenberg
 

Recently uploaded (20)

PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
KodekX | Application Modernization Development
KodekX
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Approach and Philosophy of On baking technology
30anthuong17
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Cloud computing and distributed systems.
kkkkkhan
 
Teaching material agriculture food technology
LiaRayya
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 

curl better

  • 1. curl bettercurl better March 30, foss-north 2020March 30, foss-north 2020
  • 4. what curl iswhat curl is option how-tooption how-to Q&AQ&A @bagder@bagder
  • 5. An open source project that makes a command line tool and a library for transferring data using Internet protocols @bagder@bagder
  • 6. Features! DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP TLS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, HTTP/HTTPS/SOCKS proxy, cookies, authentication (Basic, Digest, NTLM, Negotiate, Kerberos), HTTP/2, HTTP/3, alt-svc:, happy eyeballs, file transfer resume, proxy tunneling, DNS-over-HTTPS, HTTP compression and much more @bagder@bagder
  • 8. 2,200 contributors Who makes curl curl 770 authors 150 authors per year 12 regulars Daniel @bagder@bagder (The boxes are not drawn to scale)
  • 9. @bagder@bagder Conquered the world Right time Open Source Portable Backwards compatible Stable
  • 10. Many places curl has been around since spring 1998 MacOS (since 2001) Windows 10 (since 2018) Linux distros (since 1999) BSDs, other Unixes, MS-DOS, older systems Side-loaded - from the official site or elsewhere Built from source @bagder@bagder
  • 11. Many flavors Millions of build combinations Thirty different dependencies in combinations Frequent releases Rarely the same in two places! @bagder@bagder
  • 12. Many protocols and options Today we only talk about the tool named curl curl knows many protocols curl features 230 command line options I stick to HTTP(S) in this presentation Using a small subset of flags @bagder@bagder
  • 13. Number of command line options @bagder@bagder First curl release 24 230
  • 14. HTTP RequestRequest - method + path- method + path - headers- headers - body- body ResponseResponse - response code- response code - headers- headers - body- body @bagder@bagder
  • 15. Under the hood GET / HTTP/1.1 Host: www.example.com Accept: */* User-Agent: HTTP-eats-the-world/2019 HTTP/1.1 200 OK Date: Thu, 09 Nov 2018 14:49:00 GMT Server: my-favorite v3 Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT Content-Length: 12345 Set-Cookie: this-is-simple=yeah-really; Content-Type: text/html [content] @bagder@bagder
  • 16. Minimal behavior by default Basic set of headers No fancy functionality Let users enable more when wanted Toggle features on/off one-by-one @bagder@bagder
  • 17. Long and short options All short options have a long alternative -s can also be set with –-silent I will use the short options today Many options only exist as long options @bagder@bagder
  • 18. $ curl example.com <html> <body> <div> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </div> </body> </html> @bagder@bagder Plain curl HTTP GET
  • 19. $ curl -i https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder GET and show me the response headers
  • 20. $ curl https://example.com/json | jq { "id": "kr09ddfgbfsf", "name": "Issues, PRs, Dashboard, Projects", "status": "operational", "created_at": "2017-01-31T20:01:46.638Z", "updated_at": "2019-09-26T14:00:45.562Z", "position": 3, "description": "Fake JSON for an example", "group": false, "only_show_if_degraded": false }, @bagder@bagder GET and show JSON nicely
  • 21. $ curl -I https://example.com/ HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 @bagder@bagder HEAD only shows the response headers HEAD returns no body!
  • 22. $ curl -I https://example.com/redirected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html location: /a/separate/place content-length: 306 server: server/3000 @bagder@bagder Curl doesn’t follow redirects by default HEAD returns no body!
  • 23. $ curl -I -L https://example.com/redi rected HTTP/1.1 302 OK date: Wed, 09 Oct 2019 11:16:06 GMT ... HTTP/1.1 200 OK date: Wed, 19 Oct 2019 11:23:06 GMT content-type: text/html content-length: 4676 server: server/3000 @bagder@bagder Please follow redirects HEAD returns no body!
  • 24. URL Globbing $ curl https://example.com/[1-9].html $ curl https://example.com/[01-99].html $ curl https://example.com/[a-z].html $ curl https://example.com/[1-9:2].html $ curl https://example.com/[a-z:3].html $ curl https://example.com/[1-9].html -o save_#1.html $ curl https://example.com/{ham,cheese,pineapple}.jpg -o hawaii_#1.jpg $ curl http://archive.example/issue[1996-1999]/vol[1-4]/part{a,b,c}.html
  • 25. $ curl -v https://example.com/ -o /dev/null * Trying 93.184.216.34:443... * Connected to example.com (93.184.216.34) port 443 (#0) … * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=Los Angeles; O=Corporation; CN=www.example.org … > GET / HTTP/2 > Host: example.com > User-Agent: curl/7.66.0 > Accept: */* > < HTTP/2 200 < date: Wed, 09 Oct 2019 11:16:06 GMT < content-type: text/html < content-length: 306 < server: server/3000 @bagder@bagder Verbose shows more from under the hood
  • 26. One -v is enough. More adds nothing!
  • 27. $ curl https://example.com/ -H "Magic: disc0" $ curl https://example.com/ -H "User-agent: disc0 1n th3 n1gh7" $ curl https://example.com/ -H "User-agent:" $ curl https://example.com/ -H "User-agent;" @bagder@bagder Pass in custom HTTP headers
  • 28. $ curl -d name=Daniel -i https://example.com/receiver HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder POST some basic data to the remote
  • 29. $ curl -d @file https://example.com/receiver -o saved $ ls -l | curl -d @- https://example.com/receiver -o saved $ ls -l | curl --data-binary @- https://example.com/receiver -o saved $ curl --data-binary @file.json -H "Content-Type: application/json" https://example.com @bagder@bagder POST a file
  • 30. $ curl -T localfile -i https://example.com/remote_name HTTP/1.1 200 OK date: Wed, 09 Oct 2019 11:16:06 GMT content-type: text/html content-length: 306 server: server/3000 <body> <h1>Example Domain</h1> <p>This domain is established to be used for illustrative examples. <p><a href="http://www.iana.org/domains/example">More information...</a></p> </body> @bagder@bagder PUT a file
  • 31. $ curl -T localfile -X SWOOSH https://example.com/remote_name -o save @bagder@bagder With -X we change the method string
  • 32. curl -X in vain leads to pain https://xkcd.com/386/ @bagder@bagder
  • 33. $ curl -d postdata -X POST https://example.com/remote_name -o save $ curl -d postdata -X POST https://example.com/remote_name -o save -L @bagder@bagder Bad -X leads to pain Probably not what you wanted!
  • 34. @bagder@bagder Bad -X -X means use this method in all requests -L means follow redirects Redirects instruct the client what method to use in the next request -X overrides the method the server wants the client to use Use -X if you want a different method than curl would use
  • 36. @bagder@bagder Cookies are name value pairs Cookies are name=value pairs sent by servers Sent back by clients when the URL matches curl doesn’t know cookies unless you tell it to curl has separate options for reading and writing cookie files
  • 37. $ curl -c cookiejar.txt https://example.com/ $ curl -b cookiejar.txt https://example.com/ $ cat cookiejar.txt # Netscape HTTP Cookie File # https://curl.haxx.se/docs/http-cookies.html # This file was generated by libcurl! Edit at your own risk. example.com TRUE / FALSE 1602699857 duidd182180ebab48 @bagder@bagder Cookies!
  • 38. $ curl -c cookiejar.txt https://example.com/login_form $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/login -d user=daniel -d password=1234 $ curl -b cookiejar.txt -c cookiejar.txt https://example.com/profile @bagder@bagder Cookies in a login
  • 39. $ curl https://127.0.0.1/ $ curl -k https://127.0.0.1/ $ curl -k https://127.0.0.1/ -H "Host: example.com" $ curl https://example.com/ --resolve example.com:443:127.0.0.1 $ curl https://example.com/ --connect-to example.com:443:host.tld:8443 @bagder@bagder HTTPS obstacles, locally hosted named site Causes certificate problems Ducks for the cert problem Doesn’t work with cookies Works with cookies Doesn’t work with virtual servers Does TLS and certs correctly Works with cookies Works with virtual servers
  • 42. $ curl https://example.com/ -d sendthisdata --trace-ascii - … => Send header, 143 bytes (0x8f) 0000: POST / HTTP/2 000f: Host: example.com 0022: User-Agent: curl/7.66.0 003b: Accept: */* 0048: Content-Length: 12 005c: Content-Type: application/x-www-form-urlencoded 008d: => Send SSL data, 5 bytes (0x5) 0000: ....& => Send SSL data, 1 bytes (0x1) 0000: . => Send data, 12 bytes (0xc) 0000: sendthisdata == Info: We are completely uploaded and fine … @bagder@bagder More details!
  • 43. Extracting transfer meta-data --write-out (-w) is your friend --write-out ‘text and %{variables} to see’ Variables include:  content_type  http_version  remote_ip  local_ip  speed_download %{json}%{json} Coming soon! Coming soon!
  • 44. $ curl https://example.com/ -o saved -w ‘%{json}’ | jq @bagder@bagder --write-out %{json} { "url_effective": "https://example.com/", "http_code": 200, "response_code": 200, "http_connect": 0, "time_total": 0.658048, "time_namelookup": 0.001, "time_connect": 0.153556, "time_appconnect": 0.501105, "time_pretransfer": 0.501223, "time_starttransfer": 0.657869, "size_header": 331, "size_request": 77, "size_download": 1256,
  • 45. $ curl https://example.com/ --libcurl sourcecode.c $ gcc sourcecode.c -lcurl -o ./myapp $ ./myapp $ cat sourcecode.c @bagder@bagder Convert it into an application? /********* Sample code generated by the curl command line tool ********** * All curl_easy_setopt() options are documented at: * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html ************************************************************************/ #include <curl/curl.h> int main(int argc, char *argv[]) { CURLcode ret; CURL *hnd; hnd = curl_easy_init(); curl_easy_setopt(hnd, CURLOPT_BUFFERSIZE, 102400L); curl_easy_setopt(hnd, CURLOPT_URL, "https://example.com"); curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L); curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.66.0"); curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L); curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_2TLS); curl_easy_setopt(hnd, CURLOPT_SSH_KNOWNHOSTS, "/home/user/.ssh/known_hosts"); curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L); ...
  • 47. $ curl 'https://curl.haxx.se/' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:69.0) Gecko/ 20100101 Firefox/69.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en- US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure- Requests: 1' -H 'If-Modified-Since: Thu, 17 Oct 2019 02:05:02 GMT' -H 'If-None-Match: "21eb- 59511a53fa694-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: Trailers' $ curl 'https://curl.haxx.se/' -H 'authority: curl.haxx.se' -H 'cache-control: max-age=0' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/ OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Mobile Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/ *;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: sv,en-US;q=0.9,en;q=0.8' -H 'if-none-match: "21eb- 59511a53fa694-gzip"' -H 'if-modified-since: Thu, 17 Oct 2019 02:05:02 GMT' --compressed @bagder@bagder Mimic that browser – copy as curl
  • 48. SSLKEYLOGFILE Why trust your tools when you can snoop on them Wireshark is your friend Tell curl or your browser to store its secrets in SSLKEYLOGFILE Tell Wireshark where the secrets are located Run curl or your browser @bagder@bagder
  • 49. $ export SSLKEYLOGFILE=$HOME/tmp/tlskey $ curl https://example.com @bagder@bagder Snoop on curl
  • 51. Many HTTP versions – but they look the same HTTP/1.0 shipped in 1996 We’ve learned how HTTP headers look like HTTP/2 shipped in 2015 HTTP/3 is “coming soon” curl pretends all headers look and work like HTTP/1 @bagder@bagder
  • 52. $ curl –-http1.0 https://example.com $ curl –-http1.1 https://example.com $ curl –-http2 https://example.com $ curl –-http3 https://example.com @bagder@bagder HTTP versions HTTP/3 support is experimental!
  • 53. $ curl -d user=daniel https://example.com https://another.example.com $ curl -d user=daniel https://example.com –-next https://another.example.com $ curl https://example.com –-next -d user=daniel https://another.example.com @bagder@bagder Different requests on the same command line!
  • 54. All this, and much much more... @bagder@bagder
  • 56. 56 You can help!You can help! @bagder@bagder
  • 58. License This presentation and its contents are licensed under the Creative Commons Attribution 4.0 license: http://creativecommons.org/licenses/by/4.0/ @bagder@bagder