Using awk and SQLite

I have recently had the opportunity to use awk and sqlite on a project that I’m working on.

My first thought was that awk should be able to do that. I found some code that someone else did that parses quote and comma delimited.

I started off installing sqlite on cygwin.

After getting sqlite installed, I felt that I needed some test data in order to work with so I did a search and found this site:

https://www.briandunning.com/sample-data/

I grabbed the free file for testing.

The file is comma delimited file that has these fields:

  • First Name
  • Last Name
  • Company
  • Address
  • City
  • County (where applicable)
  • State/Province (where applicable)
  • ZIP/Postal Code
  • Phone 1
  • Phone 2
  • Email
  • Web

Here I used the SQLite command line to issue the command to create the table.

I attempted to import the csv file but got an error.

I can tell that the records are from a Macintosh system because the file has a carriage return record delimiter.

so I wanted to view a hexdump of the file.  I used cat, tr , head and hexdump.

You can see in the HEX dump below that the file now has hex 09 line feeds.

The problem is how to handle quote comma delimited files. I did a search to see if someone already had a solution for that and found this:

AWK CSV Parser

The code contains a function called parse_csv(). You can look at the usage of the parameters on the link above. The important part is how to call this function.

I’m using AWK to convert the CSV file into a pipe delimited file.

I then imported the data into the database.

I’m seeing the first row has the column titles. I’ll look into how to import without that line later.

What is Twitter shadow ban?

It is the practice of making tweets invisible to the general public in an attempt to censor a particular user. That is the truth. A more politically correct answer is that it is the technique of blocking a user that so that they don’t know that they are being blocked. It is pure deception.

Originally, the concept was intended to reduce spam or undesirable posts from a user that was considered a problem. However, there is a potential that this practice can be abused and used for nefarious purposes. Who judges whether a user is a problem or not and what motives are behind that decision?

I really became interested in this because after the topic came up on twitter. A well known cartoonist named Scott Adams wrote this blog post about shadow banning.

  • http://allnewspipeline.com/Smoking_Gun_Twitter_Shadow_Ban_List.php
  • http://www.breitbart.com/tech/2017/02/16/twitter-introduces-account-limiting-for-abusive-behavior/

Sample net capacity SQL

A handy way of calculating based on a transaction file.