Aleksandra Badaczewska

Aleksandra Badaczewska

Alex is a Research Scientist IV at the Genome Informatics Facility at Iowa State University. Her academic background is in Chemistry and Biotechnology, with a Ph.D. in Computational Biology and broad experience in programming and designing web applications. She develops a comprehensive collection of highly customizable visualization solutions for Bioinformatics and supports software optimization for the USDA Geospatial analyses.

DataScience Workbook / 07: Data Acquisition and Wrangling / 2. Data manipulation

Introduction

Data manipulation refers to the process of transforming and organizing data to make it more useful for analysis, reporting, meeting specific research requirements. It is a crucial step in data analysis, as it enables researchers to extract valuable insights from raw data. This process often involves cleaning, merging, restructuring, or summarizing data, among other tasks.

There are many tools and techniques available to help researchers effectively manipulate and analyze their data. Whether working with Excel, Python, R, SQL, or other specialized tools, researchers should carefully consider their data manipulation needs and choose the tools and techniques that best suit the research goals.

Try out any of the tools quickly online!

EXCEL
Google Sheets online ⤴
Microsoft Excel GUI

SQL
www.programiz.com/sql ⤴
SQL programming language

02_data_manipulation_tools1.png

PYTHON
www.python.org/shell/ ⤴
Python programming language

R
https://rdrr.io/snippets/ ⤴
R programming language

02_data_manipulation_tools2.png

Excel

One of the most common tools with a graphical user interface (GUI) used for data manipulation and analysis is Microsoft Excel ⤴. Excel offers a wide range of built-in features for organizing and analyzing data, including:

  • sorting,
  • filtering,
  • merging and splitting data, and
  • calculating functions.

By using these features, researchers can quickly and easily manipulate data sheets to create summaries, charts, and reports.

…about manipulating Excel data sheets, you can visit the following website: Basic tasks in Excel ⤴ by Microsoft. Explore practical examples in the hands-on tutorials available in this workbook:

  • INTRODUCTION:
  • TUTORIALS:

Python

Python ⤴ is a powerful programming language widely used for advanced data analysis, statistics, and interactive visualization, particularly for working with large text files. Python offers a rich set of libraries and modules that can be used to manipulate text files, including:

library description tutorial
Pandas a Python library for efficient data structure manipulation and analysis Tutorial
NumPy a Python library for computing and data structure transformation Tutorial
SciPy a Python library for scientific computing and statistics Tutorial
Math a Python module with various mathematical functions Tutorial
Regex a Python module for working with regular expressions Comming soon
JSON a Python module for working with data in JSON format Tutorial

With these libraries, you can manipulate large text files, clean and transform data, perform statistical analysis, and format the output.

…more about manipulating large text files with Python, you can visit the following tutorials in this workbook:

  • INTRODUCTION:
  • TUTORIALS:

R

R ⤴ is another popular programming language used for statistical analysis and simple data visualization. It provides several libraries and functions for data manipulation, including:

library description tutorial
dplyr an R package for data filtering, selecting, arranging, and summarizing Tutorial
tidyverse an R package for data cleaning, reshaping and tidying Tutorial
data.table an R package for aggregation and manipulation of large data sets span class=”c-alert”>Comming soon</span>

With these libraries, you can manipulate and transform data, filter, select, and summarize data.

…more about manipulating data for a quick statistical analysis with R, you can visit the following tutorials in this workbook:

  • INTRODUCTION:
  • TUTORIALS:

SQL

SQL ⤴ (Structured Query Language) is a powerful language used for managing and manipulating relational databases. It provides several functions and commands for data manipulation, including SELECT, INSERT, UPDATE, and DELETE. With SQL, you can manipulate data stored in a database, extract data, and perform statistical analysis.

…about SQL from the website https://www.w3schools.com/sql/default.asp

Other tools for data manipulation in research projects

In addition to the above tools and languages, there are several other specialized software for data manipulation in research projects, including:

software description
MATLAB ⤴ programming language and environment for numerical computing and data analysis
SAS ⤴ statistical software suite for data management, analysis, and visualization
OpenRefine ⤴ tool for cleaning and transforming messy data
Tableau ⤴ tool for data visualization and manipulation