How to check normality of the data

All parametric tests make certain assumptions about the data. Most of the parametric tests like F-test, Z-test assume the data is normally distributed. So it is always useful to test the assumption of normality before we proceed. Sharing my notes about normality tests in this post.

At high level I would generalize tests into two categories

• Visual test
• Statistical test

Visual tests: These might not be the best way to check for normality and can be ambiguous and/or misleading sometimes. Lets get a high-level overview of how to use them.

Histogram: We can plot a histogram of observed data and check for

• If it looks like a bell shaped curve
• Not skewed in any direction.

Useful Unix commands for exploring data

While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files. Also sometimes it is not convenient to load the data file in Python or R in order to perform few basic checks and exploratory analysis. Unix commands are pretty handy in these scenarios and often takes significantly less time in execution.

Lets consider movie data set from some parallel universe (with random values) for this assignment. There are 8 fields in total,

What is Data profiling?

What is data profiling?

Data profiling is process of exploring the data available in order to get some idea about it. We can perform several tasks in this step like,

• Find number of missing values from each column (count)
• Find null values from each column (these are not missing value)
• Find count of useful values from each column (not-Null and not-missing)
• Find percentage of useful values from each column
• Find cardinality of each column (number unique values)
• Find distinctness as percent calculated by dividing cardinality with actual value count

R programming: Four power-commands to explore data

This is a simple tutorial demonstrating 4 useful commands to explore data. 

Requirement:

1. You have R installed
2. You have this data file 

Go to R command line interface by just entering R in terminal. It should look something like following,

Exploring Data - 1: Descriptive statistics using Excel

Excel might be one of the easiest way to get familiar with the raw data. Lets consider the US household income by state data set from here for 2009.

Before we start the tutorial make sure Analysis toolpack excel add-in is enabled. You can check it by File > Options > Add-ins.

Now make sure the data we need is in Excel file. Go to Data  >  Data Analysis.