Category: GCP

List the Roles for a User or Service Account in a Specific GCP Project

Posted on by rchapin

If you do not have web console permissions to do so, but have the ability to activate a service account that has the viewer permissions or IAM permissons to list IAM roles in a given project, the following is how you can list the roles for a given user or service account.

gcloud projects get-iam-policy <gcp-project> \
--filter="bindings.members:<email-address>" \
--flatten="bindings[].members" --format="table(bindings.role)"

Using bq load Command to Load logicalType Partitioned Data into a BigQuery Table

Posted on by rchapin

Following is the syntax and bq load command that you need to issue if you want to load data in avro file into a partitioned BigQuery table based on avro field defined as a logicalType.

Given the following schema

{
  "type" : "record",
  "name" : "logicalType",
  "namespace" : "com.ryanchapin.tests",
  "fields" : [ {
    "name" : "id",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "value",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "day",
    "type" : {
      "type" : "int",
      "logicalType" : "date"
    }
  }
}

And the following BigQuery schema

[
  {
    "name": "id",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "value",
    "mode": "NULLABLE",
    "type": "INT64"
  },
  {
    "name": "day",
    "mode": "REQUIRED",
    "type": "DATE"
  }
]

Assuming that you have a correct avro data file (an exercise for the reader) that contains records that include values in the day column that are the number of days since the epoch, you can run the following bq load command to load that data into your table.

 bq --project_id my_project load --source_format=AVRO --time_partitioning_type=DAY --time_partitioning_field=day --use_avro_logical_types my_dataset.my_table gs://my_bucket/*.avro
© 2005 Ryan Chapin