Category: Cloud

List the Roles for a User or Service Account in a Specific GCP Project

Posted on by rchapin

If you do not have web console permissions to do so, but have the ability to activate a service account that has the viewer permissions or IAM permissons to list IAM roles in a given project, the following is how you can list the roles for a given user or service account.

gcloud projects get-iam-policy <gcp-project> \
--filter="bindings.members:<email-address>" \
--flatten="bindings[].members" --format="table(bindings.role)"

Using bq load Command to Load logicalType Partitioned Data into a BigQuery Table

Posted on by rchapin

Following is the syntax and bq load command that you need to issue if you want to load data in avro file into a partitioned BigQuery table based on avro field defined as a logicalType.

Given the following schema

{
  "type" : "record",
  "name" : "logicalType",
  "namespace" : "com.ryanchapin.tests",
  "fields" : [ {
    "name" : "id",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "value",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "day",
    "type" : {
      "type" : "int",
      "logicalType" : "date"
    }
  }
}

And the following BigQuery schema

[
  {
    "name": "id",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "value",
    "mode": "NULLABLE",
    "type": "INT64"
  },
  {
    "name": "day",
    "mode": "REQUIRED",
    "type": "DATE"
  }
]

Assuming that you have a correct avro data file (an exercise for the reader) that contains records that include values in the day column that are the number of days since the epoch, you can run the following bq load command to load that data into your table.

 bq --project_id my_project load --source_format=AVRO --time_partitioning_type=DAY --time_partitioning_field=day --use_avro_logical_types my_dataset.my_table gs://my_bucket/*.avro

s3cmd ‘ERROR: Test failed: 403 (AccessDenied): Access Denied’ and ‘ERROR: Config: verbosity level ’20’ is not valid’ [SOLVED]

Posted on by rchapin

I’m working on a project that includes sending data via Amazon Simple Storage Service (S3) and was having some problems configuring and using the s3cmd client.

The first thing I discovered about s3cmd is not to trust what it tells you when invoking s3cmd –configure to get things set up to use the bucket.

$ s3cmd -v --configure s3://some-bucket/some-prefix/

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: thisisanaccesskey
Secret Key: thisisasecretkey

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: Y

New settings:
  Access Key: thisisanaccesskey
  Secret Key: thisisasecretkey
  Encryption password: 
  Path to GPG program: None
  Use HTTPS protocol: True
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list bucket: s3://some-bucket/some-prefix/
ERROR: Test failed: 403 (AccessDenied): Access Denied

Retry configuration? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/home/rchapin/.s3cfg'

As you can see, when I ran configure and opted to test the configs, I got a 403 error.  At that point, I assumed that I didn’t have acces to the bucket and went back to the client to try and figure out if I had the right key, if they set up the bucket with the right permissions, blah, blah, blah.

It turns out, that s3cmd simply gave me incorrect information, or the command that it was using to test it wasn’t valid, or it was trying to do something with the bucket that I didn’t have permission to do.

After running the config above, I tried:

$ s3cmd put test.txt s3://some-bucket/some-prefix/
ERROR: Config: verbosity level '20' is not valid
test.txt -> s3://some-bucket/some-prefix/test.txt  [1 of 1]
 15 of 15   100% in    0s    67.58 B/s  done

Turns out that I have access after all.

$ s3cmd ls s3://some-bucket/some-prefix/
ERROR: Config: verbosity level '20' is not valid
2014-03-01 00:25         0   s3://some-bucket/some-prefix/
2014-03-19 14:06        15   s3://some-bucket/some-prefix/test.txt

It also turns out that appending the ‘-v’ arg when configuring s3cmd causes it to throw the ‘ERROR: Config: verbosity level ’20’ is not valid’ error.

If you delete the .s3cfg file in your home dir, and re-run s3cmd –configure without the -v command it should work as expected.

Just don’t trust the s3cmd –configure test . . . test it yourself and you might find that you have access already.

© 2005 Ryan Chapin