Signal and Noise: Optimizing the predictive model

Signal gives us useful information required and we would like to maximize its input.
Noise gives us useless information and we would like to remove all the possible noise as input.

Unfortunately in real life scenarios every input to model contains both signal and noise. So we have optimize the balance between two.

For any predictive model we can determine if the inputs really affect the output by association tests. Lower the p-value, stronger the association. The next step is we would like to arrange predictors in decreasing order of association and add them one by one to build the predictive model. Every time we add a predictor, a new model on training data should be built. Then it should be evaluated against the testing data.

Lets use 'Area Under the Curve' or AUC as evaluator of the model. 

Predictive Modelling project : Workflow and learnings

I got the opportunity to work on a project aimed at building predictive model for a disorder X. So sharing few observations and procedures here. Due to some constraints wont be getting into details.

Aim: Build a generic predictive model for disorder X in all ethnic groups. The predictive model works better if developed for a specific ethnic group. However we were trying to build a generic model which means we might not get too good results.

Data set: The data gathered was from three different studies and involving three ethnic groups. Every source was a case-control study. Case is a record with positive outcome (here for disorder X) and control is observation with negative outcome (for disorder X).  

Time division: It was a year long project and we spent around 80% of the time in data pre-processing (gathering, profiling, cleaning and formatting). Also most of the heavy processing was done in this part. The remaining 20% time was spend on building and evaluating different predictive models.