Using bq load Command to Load logicalType Partitioned Data into a BigQuery Table

Following is the syntax and bq load command that you need to issue if you want to load data in avro file into a partitioned BigQuery table based on avro field defined as a logicalType.

Given the following schema

{
  "type" : "record",
  "name" : "logicalType",
  "namespace" : "com.ryanchapin.tests",
  "fields" : [ {
    "name" : "id",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "value",
    "type" : [ "null", "long" ],
    "default" : null
  }, 
Continue reading “Using bq load Command to Load logicalType Partitioned Data into a BigQuery Table”

s3cmd ‘ERROR: Test failed: 403 (AccessDenied): Access Denied’ and ‘ERROR: Config: verbosity level ’20’ is not valid’ [SOLVED]

I’m working on a project that includes sending data via Amazon Simple Storage Service (S3) and was having some problems configuring and using the s3cmd client.

The first thing I discovered about s3cmd is not to trust what it tells you when invoking s3cmd –configure to get things set up to use the bucket.

$ s3cmd -v --configure s3://some-bucket/some-prefix/

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access 
Continue reading “s3cmd ‘ERROR: Test failed: 403 (AccessDenied): Access Denied’ and ‘ERROR: Config: verbosity level ’20’ is not valid’ [SOLVED]”