Category: Uncategorized

How to Add Calcite to Your Home Acid Neutralizer

Posted on by rchapin

If your incoming water supply is acidic you will need to have an acid neutralizer and a water softener installed. The medium in the acid neutralizer is calcite and it slowly dissolves as it treats the water that flows through it. When you first purchase your unit and once it is flushed and in service you should shine a bright light through one side of it to see how full it is and mark it with a piece of tape or magic marker.

Over time, depending on the acidity of your water, you will need to replace the media as it dissolves.

Following are the steps to do so which includes servicing two different types of units; one with an inspection port (or domehole) and one without that requires unscrewing the head.

Supplies

  1. A helper is a must, doing it by yourself is very difficult
  2. A couple 50 lbs. bags of calcite. Having more than you think is better than not having enough. If you can find it at a retail store, or a company that services water treatment systems, it will be much cheaper since you will not have to pay shipping. Any type of calcite that is for acid neutralizers will work.
  3. N95 dusk masks
  4. A bunch of towels
  5. A approximately four or five foot section of hose or tubing
  6. A few five gallon buckets
  7. A large funnel
  8. Some food grade silicone lubricant for plumbing
  9. A short ladder or step ladder
  10. A bright flashlight
  11. A three foot dowel or other rigid rod you can use to check the calcite level as you fill it

For Both

Depressurize it. Hopefully you have sufficient bypass valves in the plumbing to and from the unit. Turn off the valve upstream to the unit and open either a drain valve or a faucet to relieve the pressure. Run the water until it is just a trickle.

For A Unit With an Inspection Port

  1. Mark with painters tape, or a magic marker, a corner of the cap, and mark the canister right by the tape. This will allow you to know how far the cap was screwed in so that you do not over tighten it when putting it back together.
  2. Hold a towel under/around the cap and slowly unscrew it while your helper is at the ready with the section of hose and a bucket.
  3. Once you get the inspection port off, put one end of the tube into the canister and suck on the other end to create a vacuum and siphon out about a 5 gallon bucket’s worth of water.

For A Unit Without an Inspection Port

  1. Unplug the electrical connection.
  2. Disconnect the head from the plumbing.
  3. Mark with painters tape, or a magic marker, a part of the head, and mark the canister right by the tape. This will allow you to know how far the head was screwed in so that you do not over tighten it when putting it back together.
  4. Have a helper hold a towel at the base of the head and unscrew it. Do this slowly. Unscrew it 1/4 of a turn and then back 1/8 of a turn. What you are trying to do is break any seal between the down pipe that goes to the bottom of the canister from the head so that you do not lift it out when you unscrew it. If you do happen do lift the down pipe, it makes it VERY difficult to reassembly and you will have to either trim the pipe, or lay the canister on it’s side to try to re-seat the down pipe and its base.

Adding the Calcite

  1. Use the hose to siphon out about as much water as you think you will need to add calcite.
  2. Have your helper hold the funnel in either the inspection port, or in the top of the canister. For units where you removed the head ensure that you DO NOT put calcite down the down pipe.
  3. One person will hold the funnel and other will climb up the step ladder and pour in the calcite. Wear your dust mask for this otherwise you will breath in a lot of very fine particles which are very bad for your lungs.
  4. Stop regularly to check the calcite level. Insert the rod into the canister to measure the level. You can also use a flashlight by shining against the opposite side of the canister, but if the calcite that you are replacing is a different color than the existing one you can accidentally overfill it. OVERFILLING IT IS A HUGE PAIN TO FIX. Ask me how I know this. If you overfill it, when it back washes the calcite will get stuck in the head and clog the drain tube and you will have to take it outside, and lay it on its side to dump out the excess calcite. All while ensuring that the down pipe does not come out. So, just add calcite slowly, checking regularly, until it is about two-thirds full.

Reassembly

  1. Ensure that the threads for either the inspection port or the head are clean by using a clean toothbrush and pouring water over them while gently scrubbing.
  2. Lube the top of the down pipe, if applicable, and any o-rings with food-grade silicone grease and re-assemble ensuring not to cross thread it.
  3. Screw until it starts to get snug and then re-align the tape marks made before disassembling it. If you can easily screw it past that point, you will need to go another 360 degrees to get to the correct torque.

Start-up Procedure for a Non Back Washing Head

  1. Start with both in and out flow bypass valves off.
  2. Turn a downstream faucet or hose bib on. Typically, these systems are plumbed with a ball valve overflow directly downstream from the unit.
  3. Turn the upstream water on.
  4. Open the downstream bypass valve all of the way
  5. Slowly open the inflow bypass valve to allow it to fill with water and push the air out.
  6. Once it is completely full and you can no longer hear any air moving through the system turn the downstream faucet off 85% and then completely open the inflow so that now both bypasses are completely open.
  7. Then connect a hose to the same downstream faucet or hose bib and open it about 30% and run it until the water is clear and all of the calcite dust is rinsed out.

Start-up Procedure for an Electronic Head

  1. Reconnect the flush outlet tube.
  2. Start with both in and out flow bypass valves off.
  3. Set the unit into manual regeneration/backwash mode. This will allow you to purge the air from the unit when turning on the water.
  4. Slowly open the inflow bypass until all of the air is purged from the system and only water is flowing out of the flush outlet tube.
  5. Once all air is purged open the inflow bypass all of the way and let the regen/flush cycle complete.
  6. Open the outflow bypass valve.
  7. You may have to run the regen cycle a few times until the water runs clear through the unit.

Clack WS1 EE

  • To enter the user programming mode press and hold NEXT and UP together for three seconds
Clack-4-Button-EE-OEM-ManualDownload

Mount External, Encrypted, LMV drives

Posted on by rchapin

First decrypt the drive. When I plugged it in GNOME displayed a dialog and asked for the key. I’m guessing that cryptsetup luksOpen /dev/device device_crypt would work as well.

Run lvdisplay to get the LV Path for the logical volume that you want to mount.

Then, activate the volumes , there is a way to activate a specific one volume, but the following activates them all.

vgchange -ay

Then mount it

mount <lv-path> /mnt

My jq cheat sheet

Posted on by rchapin

jq is pure awesomeness. If every you want to “grep” through JSON data on the command line, this is the tool for you. Unfortunately, I don’t use it everyday, so when I do go to use it I forget the details for doing some basic operations.

Sample Data

[
    {
        "name": "Homer Simpson",
        "occupation": "Nuclear Safety Inspector"
    },
    {
        "name": "Marge Simpson",
        "occupation": "Mayor"
    },
    {
        "name": "Milhouse Van Houten",
        "occupation": "Student"
    },
    {
        "name": "Homeboy Van Somebody",
        "occupation": "Lawyer"
    }
]

Filtering

Often you will have json and you’ll want to filter out all objects that do not meet some criteria.

Selecting objects with a specific value

❯ jq '.[] | select (.name == "Homer Simpson")' sample.json 
{
  "name": "Homer Simpson",
  "occupation": "Nuclear Safety Inspector"
}

Outputting a new JSON object based on the input data

❯ jq '.[] | select (.name == "Homer Simpson") | {character_name: .name, job: .occupation}' sample.json 
{
  "character_name": "Homer Simpson",
  "job": "Nuclear Safety Inspector"
}

Using a regex

❯ jq '.[] | select (.name | test("Home"))' sample.json 
{
  "name": "Homer Simpson",
  "occupation": "Nuclear Safety Inspector"
}
{
  "name": "Homeboy Van Somebody",
  "occupation": "Lawyer"
}

Vegetarian Shepherd’s Pie

Posted on by rchapin

This is a recipe by Samantha Seneviratne.

Ingredients

Topping

  • 2 tablespoons kosher salt
  • 2 pounds russet potatoes, peeled and quartered (about 3 large)
  • 6 tablespoons unsalted butter
  • 1/2 cup whole milk
  • 1/2 cup sour cream
  • 1/2 cup packed grated Parmigiano-Reggiano cheese or vegetarian Parmesan (optional)
  • Freshly ground black pepper

Filling

  • 3/4 cup French lentils
  • 4 sprigs thyme
  • 3 1/2 cups vegetable broth
  • 1 teaspoon kosher salt
  • 4 tablespoons butter
  • 8 ounces sliced mixed mushrooms, such as button, cremlnl and shiitake
  • 1 large leek, white part only, thinly sliced (about 1 1/2 cups)
  • 2 medium carrots, peeled and diced (about 1 cup)
  • 2 cloves garlic, minced
  • Freshly ground black pepper
  • 2 tablespoons tomato paste
  • 2 tablespoons flour
  • 1 cup frozen peas, thawed
  • 1 to 2 teaspoons fresh lemon juice

Directions

  1. In a large pot, bring a gallon of water and 2 tablespoons salt to a boil over high heat. Add potatoes to boiling water and boil for about 15 to 20 minutes, until soft; a knife should go in with almost no resistance.
  2. Meanwhile, prepare the filling: In a 10-inch oven-safe skillet with high sides or an enameled cast-iron braiser over medium-high heat, bring the lentils, thyme and 2 cups of the broth to a simmer with 1 teaspoon salt. Reduce the heat and continue to cook the lentils, partly covered, until they are tender and most of the liquid is absorbed, about 20 minutes. Transfer the lentils to a bowl.
  3. Finish the topping: In a small saucepan or a microwave oven, heat 6 tablespoons of the butter and milk together until butter melts. Drain potatoes well and return to pot. Using a masher or a ricer, mash hot potatoes until smooth. Mix in the hot butter mixture and sour cream just until blended. Stir in 1/2 of the Parmigiano. Season to taste with salt and pepper. Cover and set aside.
  4. Finish the filling: Heat the oven to 375 degrees. Melt the butter in the 10-inch skillet over medium-high heat. Add the mushrooms and cook, stirring often with a wooden spoon, until they are deep golden brown, about 8 to 10 minutes. Reduce the heat to medium and add leeks, carrots and garlic, and continue to cook until tender, another 10 minutes. Season to taste with salt and pepper. Add the tomato paste and stir, cooking until it is well combined, another 2 to 3 minutes. Sprinkle flour over the mixture, stir and cook for 1 minute. Add remaining 1 1/2 cups broth, cooked lentils and peas, and cook until thickened. Remove thyme stems and stir in lemon juice to taste.
  5. Top the mixture with dollops of the mashed potatoes, then spread them out over the top. (Or transfer the lentil mixture to a 3-quart casserole dish and spread into an even layer, and top with potatoes.) Sprinkle with the remaining Parmigiano. Transfer to the oven and, if the mixture is at the top edges of your pan, set a foil-lined baking sheet underneath the pan to catch any drips. Bake the pie until the potatoes have begun to brown and the edges are bubbling, about 30 minutes. Let stand at room temperature for 15 minutes before serving.

PostgreSQL Query to Find Tables With Name LIKE

Posted on by rchapin

Sometimes you are working with a PostgreSQL database with A LOT of tables and looking for tables that contain a sub-string in their name. Following is a query that you can run that will return all of the tables that have the string in their name:

SELECT
  table_schema,
  table_name
FROM
  information_schema.tables
WHERE
      table_name LIKE '%<string>%'
  AND table_schema not in ('information_schema', 'pg_catalog')
  AND table_type = 'BASE TABLE'
ORDER BY
  table_name, table_schema
;

helm Cheat Sheet

Posted on by rchapin

Development Tips and Tricks

Test Template Rendering

Run the following. Instead of it installing the chart it will render the template and display the output

helm install --debug --dry-run <release-name> <path-to-chart-dir>

To test with an overriding value

helm install <release-name> <path-to-chart-dir> --dry-run --debug --set k=v

Deployments

  • List releases: helm list
  • Get the manifest for a release: helm get manifest <release-name> [flags]

Running GUI apps locally as root in a non-root session

Posted on by rchapin

There are instances when you need to run an X Window application. For me this is often running a terminator instance as root so that I can create tabs and split the window as still be root in each of those terminals.

In order for the root user to be able to connect to the X server you need to provide it with “credentials”. In this case it is on the same box and not over the network so the use of cookie authentication is acceptable.

As the user that is already authenticated to the X server run the following command to get the cookie used to connect to the the current $DISPLAY

rchapin@leviathan:~$ xauth list $DISPLAY
leviathan/unix:0  MIT-MAGIC-COOKIE-1  5fb2c0e68f4618ee4fa2202e1e4ae937

su to root and then run the following to add that cookie to roots authorization file

root@leviathan:~# xauth add leviathan/unix:0 MIT-MAGIC-COOKIE-1 5fb2c0e68f4618ee4fa2202e1e4ae937

You should now be able to run X windows applications from that root terminal.

Setting Per File Type Tab Configurations in VSCode

Posted on by rchapin

If you would like to have different tab configurations (tabs or spaces) along with the number of tab chars for different file types you can update your user settings.

The first thing you need to do is figure out what the file type code thinks the file that you want to change is. Open the file in vscode and then look at the bottom right of your window. In my case, I’m looking at an avro schema (.avsc) file:

In this case, code sees this file type as “JSON with Comments” and is configured to use spaces with 2 spaces per “tab”.

In order to change that, press CTRL+Shift+P and select “Preferences: Configure Language Specific Settings”. You will then be presented with a host of file types. Scroll down to the entry you want (JSON with comments) and take note of of the string within the parenthesis. That is the key that you will need to make an entry into settings.json. If you click on that entry it will open up your user settings.json. I was baffled by what to do next until I saw this stackoverflow post.

Once I read that, I knew what to add to the settings.json. Simply add a new key, defined at the root of the JSON object that is the string displayed in the parenthesis that you saw for the file type in the dropdown that was displayed after selecting “Preferences: Configure Language Specific Settings”.

The JSON in settings.json will look like the following

   "[jsonc]": {
        "editor.tabSize": 2,
        "editor.detectIndentation": false,
        "editor.insertSpaces": true,
        "editor.quickSuggestions": {
            "strings": true
        },
        "editor.suggest.insertMode": "replace",
        "gitlens.codeLens.scopes": [
            "document"
        ]
    }

Check the language specific settings docs for the configuration options.

[SOLVED] Upgrading Apache Kafka 2.7 to Java 11 Changes authenticationID sent to ZooKeeper Enabling Only 1 Kafka Broker to r/w znodes

Posted on by rchapin

The title of this post is a bit of mouthful and requires a bit more explanation.

I am running a pure open-source version of Kafka (currently running 2.7) and am using SASL/GSSAPI connections between all of the brokers and ZooKeeper. Currently, the whole system, including ZooKeeper, is running Java 8 and it is long-overdue to be upgraded to Java 11.

Upgrading Kafka to Java 11 causes the server to send an incorrect authenticationID String to ZooKeeper which results in the ACLs on the znodes being set to the hostname of the first Kafka server that connects to ZooKeeper. This results in only one of the Kafka hosts being able to r/w the znodes.

Prior to upgrading the following are the logs from one of the ZooKeeper nodes:

2021-05-17 14:47:35,345 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.19.65.22:41698
2021-05-17 14:47:35,357 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.19.65.22:41698
2021-05-17 14:47:35,367 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x1797ac46a0f0000 with negotiated timeout 18000 for client /172.19.65.22:41698
2021-05-17 14:47:35,470 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET;  authorizationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET.
2021-05-17 14:47:35,474 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: kafka
2021-05-17 14:47:35,474 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@964] - adding SASL authorization for authorizationID: kafka

The following is the corresponding logs from the Kafka host making the connection:

[2021-05-17 14:47:37,512] INFO Successfully authenticated client: authenticationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET; authorizationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET. (org.apache.kafka.common.secur
ity.authenticator.SaslServerCallbackHandler)

The most important part of this is that ZooKeeper sees the authenticationID as kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET. ZooKeeper is configured with the following options so that the host and realm will be removed from the authenticationID. This will leave the service name, ‘kafka‘, and result in all of the Kakfa znodes with kafka as the owner of the node. This then enables all of the hosts in the Kafka cluster r/w access to those znodes. Following are the configurations in ZooKeeper that enable this behavior.

kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true

The resulting znodes ACLs are

[zk: localhost:2181(CONNECTED) 5] getAcl /brokers
'world,'anyone
: r
'sasl,'kafka
: cdrwa

System Configurations

Following are all of the configurations for both the ZooKeeper and Kafka cluster:

ZooKeeper

zoo.cfg

maxClientCnxns=50
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/data_dir
dataLogDir=/zk/data_log_dir
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
clientPort=2181
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true

server.1=zk2-01.dv.quasar.nadops.net:2888:3888
server.2=zk2-02.dv.quasar.nadops.net:2888:3888
server.3=zk2-03.dv.quasar.nadops.net:2888:3888

jaas.conf

Server {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/zookeeper/conf/vmwqsrdvz201.dv.quasar.nadops.net.keytab"
  storeKey=true
  useTicketCache=false
  principal="zookeeper/vmwqsrdvz201.dv.quasar.nadops.net@EXAMPLE.NET";
};

Kafka

broker-jaas.conf

KafkaServer {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/usr/local/kafka/config/vmwqsrdvk01.dv.quasar.nadops.net.keytab"
  storeKey=true
  principal="kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET";
};

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/usr/local/kafka/config/vmwqsrdvk01.dv.quasar.nadops.net.keytab"
  storeKey=true
  serviceName="zookeeper"
  principal="kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET";
};

server.properties

Only the relevant configs are included below.

listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,BROKER_LISTENER:SASL_SSL

inter.broker.listener.name=BROKER_LISTENER
listeners=SASL_SSL://:9092,BROKER_LISTENER://:9093

ssl.keystore.location=/usr/local/kafka/config/kafka01.dv.quasar.nadops.net-keystore.jks
ssl.keystore.password=***
ssl.key.password=***
ssl.truststore.location=/usr/local/kafka/config/kafka01.dv.quasar.nadops.net-truststore.jks
ssl.truststore.password=***
sasl.kerberos.service.name=kafka
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
zookeeper.set.acl=true
super.users=User:kafka
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=false

Upgrading Kafka to Java 11

Once Kafka (leaving ZooKeeper on Java 8) is upgraded to Java 11 Kafka no longer sends the expected authorizationID String to ZooKeeper.

Following are the ZooKeeper logs of the same authentication:

2021-05-18 14:31:01,627 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.19.65.23:53374
2021-05-18 14:31:01,631 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.19.65.23:53374
2021-05-18 14:31:01,636 [myid:3] - INFO  [CommitProcessor:3:ZooKeeperServer@617] - Established session 0x3797fd27e300004 with negotiated timeout 18000 for client /172.19.65.23:53374
2021-05-18 14:31:01,728 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=vmwqsrdvk02@EXAMPLE.NET;  authorizationID=kafka/vmwqsrdvk02.dv.quasar.nadops.net@EXAMPLE.NET.
2021-05-18 14:31:01,728 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: vmwqsrdvk02
2021-05-18 14:31:01,728 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@964] - adding SASL authorization for authorizationID: vmwqsrdvk02

Following are the logs from Kafka that results in an exception being thrown because it is unable to write to the ZooKeeper znode:

[2021-05-18 14:31:01,410] INFO Initiating client connection, connectString=vmwqsrdvz201.dv.quasar.nadops.net:2181,vmwqsrdvz202.dv.quasar.nadops.net:2181,vmwqsrdvz203.dv.quasar.nadops.net:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@39b43d60 (org.apache.zookeeper.ZooKeeper)
[2021-05-18 14:31:01,414] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-05-18 14:31:01,419] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,421] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-18 14:31:01,556] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,573] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,577] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-05-18 14:31:01,604] INFO TGT valid starting at:        Tue May 18 14:31:01 UTC 2021 (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,604] INFO TGT expires:                  Wed May 19 00:31:01 UTC 2021 (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,605] INFO TGT refresh sleeping until: Tue May 18 22:32:58 UTC 2021 (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,606] INFO Opening socket connection to server vmwqsrdvz203.dv.quasar.nadops.net/172.19.65.21:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,614] INFO Socket connection established, initiating session, client: /172.19.65.23:53374, server: vmwqsrdvz203.dv.quasar.nadops.net/172.19.65.21:2181 (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,622] INFO Session establishment complete on server vmwqsrdvz203.dv.quasar.nadops.net/172.19.65.21:2181, sessionid = 0x3797fd27e300004, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,626] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-18 14:31:01,731] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /brokers/ids
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564)
        at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1662)
        at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1560)
        at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1(KafkaZkClient.scala:1552)
        at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1$adapted(KafkaZkClient.scala:1552)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1552)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:467)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
[2021-05-18 14:31:01,733] INFO shutting down (kafka.server.KafkaServer)

The authorizationID now provided by Kafka is now vmwqsrdvk02@EXAMPLE.NET, only the hostname and realm, and no longer the String that ZooKeeper expects. The resulting, derived authorizationID is vmwqsrdvk02 which means that only that Kafka host has r/w to any of the znodes which it creates.

Solution

After exhausting search options and finding multiple pages indicating that I have everything configured correctly, even for JDK 11, I posted to the #apache-kafka channel on chat.freenode.net and got a hint. Evidently, there are some JDKs that have bugs in their Kerberos implementation.

I was running java-11-openjdk-11.0.8.10-1.el7.x86_64 from CentOS 7 Updates. I double-checked and there was an updated JDK available.

After updating to the latest Java 11 JDK everything worked as expected. If the particular JDK that you are using is still not working try AdoptOpenJDK.

Creating a Counter or Progress Bar for a Python Program

Posted on by rchapin

I’ve written a number of Python apps where I would like it to print some sort of counter or progress bar to STDOUT to let me know that it is still running instead of locked up or died somehow without me being able to see it.

I had tried using a couple of different existing progress bar related modules but none of them really worked except in a very specific use case.

So, after a bit of futzing around I came up with a very simple way to print out a updating counter to STDOUT. The synchronization code was gleaned from this post here, thank you Daniel.

Following is a working example for Python 3.7. You could write your own implementation of the progress function to render whatever you want to STDOUT.

import threading
import sys
import time

def synchronized(func):
    func.__lock__ = threading.Lock()
    def synced_func(*args, **kws):
        with func.__lock__:
            return func(*args, **kws)
    return synced_func

total = 0

@synchronized
def progress(count):
    global total
    total += count
    sys.stdout.write(f'\rtotal: [{total}]')
    sys.stdout.flush()

for i in range(500):
    progress(1)
    time.sleep(0.01)

print(f'\nFinished...')

Running this will generate the following output with the total value dynamically updating as the application runs:

(progressbar) rchapin@leviathan:progressbar$ python progressbar.py 
total: [500]
Finished...
© 2005 Ryan Chapin