Principal Software Engineer/Architect, motorcyclist, drummer, and artist
I have a Google Workspace account for my domain and sometimes it is useful to have messages sent from or sent too other addresses in my domain also delivered to my address.
To set it up, login to your Google Admin console, then in the left-hand navigation click on Google Workspace, and then Gmail.
Look for Routing and click on it to enter the routing configurations.
You can then add, modify, delete, enable, or disable routing rules for different organizational units in your domain.
If you want to execute curl commands on your local machine and connect to an HTTPS server that is only reachable from a bastion or other host through which you can only get to via SSH, the following is how you set up the SSH tunnel and execute the curl command.
The following will not work
# Create ssh tunnel
#
ssh -L localhost:8443:example.com:443 user@bastion.example.com
# Attempt to hit the endpoint otherwise accessible from bastion.example.com
# with curl -X GET https://example.com/v1/endpoint
#
curl -X GET https://localhost:8443/v1/endpointThe reason that this does not work is that with the port forwarded ssh tunnel curl is unable to resolve the IP of the example.com HTTPS server on the other side of the connection on bastion.example.com and the connection fails.
If we execute curl -v we can see the details. Notice that curl is connecting to localhost:8443 and not resolving example.com
$ curl -v https://localhost:8443/subjects
* Trying ::1:8443...
* Connected to localhost (::1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.example.com
* start date: Aug 3 00:00:00 2022 GMT
* expire date: Sep 1 23:59:59 2023 GMT
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x560827a1b2c0)
> GET /subjects HTTP/2
> Host: localhost:8443
> user-agent: curl/7.74.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< server: awselb/2.0
< date: Wed, 08 Feb 2023 15:20:04 GMT
< content-type: text/plain; charset=utf-8
< content-length: 0
Instead of just forwarding TCP packets we need to tell ssh client to setup a SOCKS5 proxy through which we will tunnel traffic.
ssh -D 8443 -f -C -q -N bastion.example.comThe -D option creates a SOCKS5 proxy server listening on port 8443 which tunnels the traffic over ssh to bastion.example.com from which hostnames for the destination webserver can be resolved. This creates a proxy server that enables you to connect to “dynamic” destinations on the other side of the tunnel.
The other options
-D 8443- start a SOCKS server listening on port 8443 on the localhost
-f - fork the process, running it in the background
-C - compress data
-q - quite mode
-N - indicate to the ssh client that there are no commands to be sent over the tunnelOnce you create the proxy and tunnel you can then execute curl commands as follows on the localhost telling curl to use the SOCKS5 proxy listening on localhost:8443
curl -v -x socks5h://0:8443 https://example.com/v1/endpointThe verbose output shows that the SOCKS5 proxy is the connecting to example.com:443 over the tunnel and remotely resolving the IP to the correct HTTPS server on the other side of the tunnel.
$ curl -v -x socks5h://0:8443 https://example.com/v1/endpoint
* Trying 0.0.0.0:8443...
* SOCKS5 connect to example.com:443 (remotely resolved)
* SOCKS5 request granted.
* Connected to 0 (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.example.com
* start date: Aug 3 00:00:00 2022 GMT
* expire date: Sep 1 23:59:59 2023 GMT
* subjectAltName: host "example.com" matched cert's "*.example.com"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562ada1d52c0)
> GET /subjects HTTP/2
> Host: example.com
> user-agent: curl/7.74.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Wed, 08 Feb 2023 15:22:08 GMT
< content-type: application/vnd.schemaregistry.v1+json
< content-length: 1937
< vary: Accept-Encoding, User-Agent
If you do not have web console permissions to do so, but have the ability to activate a service account that has the viewer permissions or IAM permissons to list IAM roles in a given project, the following is how you can list the roles for a given user or service account.
gcloud projects get-iam-policy <gcp-project> \
--filter="bindings.members:<email-address>" \
--flatten="bindings[].members" --format="table(bindings.role)"Following is the syntax and bq load command that you need to issue if you want to load data in avro file into a partitioned BigQuery table based on avro field defined as a logicalType.
Given the following schema
{
"type" : "record",
"name" : "logicalType",
"namespace" : "com.ryanchapin.tests",
"fields" : [ {
"name" : "id",
"type" : [ "null", "string" ],
"default" : null
}, {
"name" : "value",
"type" : [ "null", "long" ],
"default" : null
}, {
"name" : "day",
"type" : {
"type" : "int",
"logicalType" : "date"
}
}
}And the following BigQuery schema
[
{
"name": "id",
"mode": "NULLABLE",
"type": "STRING"
},
{
"name": "value",
"mode": "NULLABLE",
"type": "INT64"
},
{
"name": "day",
"mode": "REQUIRED",
"type": "DATE"
}
]Assuming that you have a correct avro data file (an exercise for the reader) that contains records that include values in the day column that are the number of days since the epoch, you can run the following bq load command to load that data into your table.
bq --project_id my_project load --source_format=AVRO --time_partitioning_type=DAY --time_partitioning_field=day --use_avro_logical_types my_dataset.my_table gs://my_bucket/*.avro
If you are setting up Thunderbird to use your Gmail account you may find that when Thunderbird opens a new window to a Google web portal into which you are to provide your email address and password that it will keep asking you over and over again for your email and never enable to you to enter the password.
This occurs when Thunderbird’s privacy settings do not allow it to store cookies.
First, ensure that your gmail account has Allow insecure apps off. Unfortunately, it may take some time for this setting to propagate to your account.
Than, go to Preferences > Privacy and under Web Content check the Accept cookies from sites checkbox.
Return to your account settings when when prompted you should now be able to enter your credentials to grant Thunderbird access to your account.
Netcat is generally known as a TCP/IP Swiss Army Knife and is incredibly helpful for both debugging and mocking up network services
Following is an example on how to setup a mock RESTful service that communicates over HTTPS.
On the “server” side, run the following command. The -l command instructs Netcat to listen.
while true; do { echo -e “HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>” | nc -vl –ssl 8080; } done
On the “client” side, run the following to PUT a sample json document.
curl https://localhost:8080/foo/blah -k -XPUT -d @sample.json
Alternatively, you can also generate a key cert pair to use if you have to test importing of certs
To do so, first generate a self-signed cert and an ssl key without a passphrase for your nc “server”. Place the server.key and server.cert file in /var/tmp/server-cert
openssl req -nodes -new -x509 -keyout server.key -out server.cert
Then run nc as follows:
while true; do { echo -e “HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>” | nc -vl –ssl 8080 –ssl-key /var/tmp/server-cert/server.key –ssl-cert /var/tmp/server-cert/server.cert; } done
Let’s say that you are working on a part of a project that needs to consume some JSON data from a forthcoming HTTP service. That sub-system is being developed by one of your colleagues and is not yet ready for you to stand-up to develop against.
You can use netcat to mock the webservice and return some static JSON data for which you can develop and test against with a simple one-liner.
First, put together your JSON in a file, ‘my.json’ and then run the following command in a terminal:
while true; do echo -e "HTTP/1.1 200 OK\r\n\r\n$(cat my.json)" | nc -l 9998; doneAn hitting http://localhost:9998 will return your test JSON data.
In order to execute curl commands to endpoints with passwords that contain special characters, the cleanest way that I have found to do so is to Base64 encode the authentication string for curl and then pass an Authorization request header along with the request.
In this example the credentials are uid ‘rchapin’ and passwd ‘abc123!@#’. Normally we would pass this to curl as follows:
$ curl -u rchapin:abc123!@# -X GET https://some-endpoint:443However, this will not work and the password will need to be sent in some other fashion other than ASCII.
Following are the steps to pass the credentials as Base64:
1. Using your favorite Base64 encoder, generate and ASCII string of the entire ‘username:password’ string that you would normally pass with the -u option for curl
rchapin:abc123!@# converted = cmNoYXBpbjphYmMxMjMhQCM=2. Now modify the curl command as follows:
$ curl -H "Authorization: Basic cmNoYXBpbjphYmMxMjMhQCM=" -X GET https://some-endpoint:443I am putting together a chrooted installation of named and ran into a problem whereby attempting to generate an rndc.key with rndc-confgen just hangs, never returning and not generating a key.
After doing some searching I discovered that I needed to run the command as follows:
rndc-confgen -a -r /dev/urandom -t /var/named/chrootWhich outputs the following, generating the key file that I expected.
wrote key file "/etc/rndc.key"
wrote key file "/var/named/chroot/etc/rndc.key"Google requires that you login with a Google account before you can install any Chrome extensions.
The following is how to install an extension without logging in (under Windows. The same should work under Linux and Mac):