Showing posts with label Data Mining. Show all posts
Showing posts with label Data Mining. Show all posts

Sunday, November 10, 2019

Data Mining on ClinicalTrials.gov's XML database

The clinicaltrials.gov is one of the biggest clinical trials registry. However the search functionality offered has its limitations.

For example, if you are trying to search for 'double-hit lymphoma' completed trials then ct.gov returns zero results. One could assume based on the results that the data doesn't exists or get the hands dirty to double-check the results.

You could do this using clinicaltrials.gov api or the xml database download. Although the api is a better way to scale things, the learning curve could be steep. So the second way using xml download might be better suited for beginners.

Luckily clinicaltrials.gov offers xml download of all of its data. So you can,
1. Download the data
2. Create your own non-relational database
3. Create queries to validate the default search engine results

Part 1:

The first step is pretty straight forward. All you have to do it paste the following link in the browser and hit enter: https://clinicaltrials.gov/AllPublicXML.zip.

This will download a single huge zip file with all the information we need. If you extraction it you can find individual study study in a tree-like xml structure.

In order to add more filtering on the data you can explore the clinicaltrials.gov's download documentation here.