Category: HTTP

curl HTTPS Over an SSH Tunnel

Posted on by rchapin

If you want to execute curl commands on your local machine and connect to an HTTPS server that is only reachable from a bastion or other host through which you can only get to via SSH, the following is how you set up the SSH tunnel and execute the curl command.

The following will not work

# Create ssh tunnel
#
ssh -L localhost:8443:example.com:443 user@bastion.example.com

# Attempt to hit the endpoint otherwise accessible from bastion.example.com
# with curl -X GET https://example.com/v1/endpoint
#
curl -X GET https://localhost:8443/v1/endpoint

The reason that this does not work is that with the port forwarded ssh tunnel curl is unable to resolve the IP of the example.com HTTPS server on the other side of the connection on bastion.example.com and the connection fails.

If we execute curl -v we can see the details. Notice that curl is connecting to localhost:8443 and not resolving example.com

$ curl -v https://localhost:8443/subjects
*   Trying ::1:8443...
* Connected to localhost (::1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.example.com
*  start date: Aug  3 00:00:00 2022 GMT
*  expire date: Sep  1 23:59:59 2023 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x560827a1b2c0)
> GET /subjects HTTP/2
> Host: localhost:8443
> user-agent: curl/7.74.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404 
< server: awselb/2.0
< date: Wed, 08 Feb 2023 15:20:04 GMT
< content-type: text/plain; charset=utf-8
< content-length: 0

Instead of just forwarding TCP packets we need to tell ssh client to setup a SOCKS5 proxy through which we will tunnel traffic.

ssh -D 8443 -f -C -q -N bastion.example.com

The -D option creates a SOCKS5 proxy server listening on port 8443 which tunnels the traffic over ssh to bastion.example.com from which hostnames for the destination webserver can be resolved. This creates a proxy server that enables you to connect to “dynamic” destinations on the other side of the tunnel.

The other options

-D 8443- start a SOCKS server listening on port 8443 on the localhost
-f - fork the process, running it in the background
-C - compress data
-q - quite mode
-N - indicate to the ssh client that there are no commands to be sent over the tunnel

Once you create the proxy and tunnel you can then execute curl commands as follows on the localhost telling curl to use the SOCKS5 proxy listening on localhost:8443

curl -v -x socks5h://0:8443 https://example.com/v1/endpoint

The verbose output shows that the SOCKS5 proxy is the connecting to example.com:443 over the tunnel and remotely resolving the IP to the correct HTTPS server on the other side of the tunnel.

$ curl -v -x socks5h://0:8443 https://example.com/v1/endpoint
*   Trying 0.0.0.0:8443...
* SOCKS5 connect to example.com:443 (remotely resolved)
* SOCKS5 request granted.
* Connected to 0 (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.example.com
*  start date: Aug  3 00:00:00 2022 GMT
*  expire date: Sep  1 23:59:59 2023 GMT
*  subjectAltName: host "example.com" matched cert's "*.example.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562ada1d52c0)
> GET /subjects HTTP/2
> Host: example.com
> user-agent: curl/7.74.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< date: Wed, 08 Feb 2023 15:22:08 GMT
< content-type: application/vnd.schemaregistry.v1+json
< content-length: 1937
< vary: Accept-Encoding, User-Agent

Mocking an HTTPS RESTful endpoint with Netcat

Posted on by rchapin

Netcat is generally known as a TCP/IP Swiss Army Knife and is incredibly helpful for both debugging and mocking up network services

Following is an example on how to setup a mock RESTful service that communicates over HTTPS.

On the “server” side, run the following command.  The -l command instructs Netcat to listen.

while true; do { echo -e “HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>” |  nc -vl –ssl 8080; } done

On the “client” side, run the following to PUT a sample json document.

curl https://localhost:8080/foo/blah -k -XPUT -d @sample.json

Alternatively, you can also generate a key cert pair to use if you have to test importing of certs

To do so, first generate a self-signed cert and an ssl key without a passphrase for your nc “server”.  Place the server.key and server.cert file in /var/tmp/server-cert

openssl req -nodes -new -x509 -keyout server.key -out server.cert

Then run nc as follows:

while true; do { echo -e “HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>” |  nc -vl –ssl 8080 –ssl-key /var/tmp/server-cert/server.key –ssl-cert /var/tmp/server-cert/server.cert; } done

Using netcat to Mock a RESTful Webservice that Returns JSON

Posted on by rchapin

Let’s say that you are working on a part of a project that needs to consume some JSON data from a forthcoming HTTP service.  That sub-system is being developed by one of your colleagues and is not yet ready for you to stand-up to develop against.

You can use netcat to mock the webservice and return some static JSON data for which you can develop and test against with a simple one-liner.

First, put together your JSON in a file, ‘my.json’ and then run the following command in a terminal:

while true; do echo -e "HTTP/1.1 200 OK\r\n\r\n$(cat my.json)" | nc -l 9998; done

An hitting http://localhost:9998 will return your test JSON data.

How to Use Credentials That Contain Special Characters with curl

Posted on by rchapin

In order to execute curl commands to endpoints with passwords that contain special characters, the cleanest way that I have found to do so is to Base64 encode the authentication string for curl and then pass an Authorization request header along with the request.

In this example the credentials are uid ‘rchapin’ and passwd ‘abc123!@#’.  Normally we would pass this to curl as follows:

$ curl -u rchapin:abc123!@# -X GET https://some-endpoint:443

However, this will not work and the password will need to be sent in some other fashion other than ASCII.

Following are the steps to pass the credentials as Base64:

1. Using your favorite Base64 encoder, generate and ASCII string of the entire ‘username:password’ string that you would normally pass with the -u option for curl

rchapin:abc123!@# converted = cmNoYXBpbjphYmMxMjMhQCM=

2. Now modify the curl command as follows:

$ curl -H "Authorization: Basic cmNoYXBpbjphYmMxMjMhQCM=" -X GET https://some-endpoint:443

Do Not Use Symlinks in Jetty’s webapps Directory

Posted on by rchapin

I set up a development instance of Jetty on my local machine and have been happily coding, compiling and deploying via a shell script.  The script copies the war from my user’s target directory to the jetty users’s home dir changes the permissions and then moves it to the /webapp dir creating a symlink to the name of the .war that is referenced in a number of config files.

This was working just fine until I did a merge with another developers code and a significant portion of it changed that Jetty had problems determining whether or not is should dump it’s local cache of .class files and configs and got into a very funky state.  Unfortunately, not one in which it was obviously funky, but where, all of the sudden I could not look up defined DataSources in JNDI.

Ultimately, after looking through many other different things, I decided to delete the symlink and, rename the .war in the /webapps dir.

Bingo, worked right away.

© 2005 Ryan Chapin