07: Data Acquisition and Wrangling

Information is the foundation of the learning process. Data acquisition and wrangling are crucial parts of Data Science that lead to extracting knowledge from the information. With large, difficult to transfer data, remote access is the rule, almost exclusively via a command-line interface. Luckily for you, knowing a few tricks make it easy to access and visualize data in a friendly way on a remote machine. As you explore this section, you will also learn how to manage Excel spreadsheets and efficiently manipulate massive data with Python.

Table of contents

1. Remote Data Access

• 1.1 Remote Data Transfer
  • 1.1.1 Copying Data using Graphical Interface: Globus
  • 1.1.2 Copying Data via SSH using Command Line: scp, rsync
  • 1.1.3 File Transfer using irods
• 1.2 Remote Data Download
  • 1.2.1 Downloading Online Data using WGET
  • 1.2.2 Downloading Online Data using Web Scraping
  • 1.2.3 Downloading Online GitHub Repos using GIT
  • 1.2.4 Downloading Single Folder from GitHub using SVN
• 1.3 Remote Data Preview without Downloading
  • 1.3.1 Viewing Text Files using UNIX commands
  • 1.3.2 Viewing PDF Files using X11 SSH connection
  • 1.3.3 Viewing graphics in a terminal as the text-based ASCII art
  • 1.3.4 Mounting Remote Folder on Local Machine

2. Data Manipulation

• 2.1 Manipulating Excel Data Sheets
  • 2.1.1 Create Workbook from Multiple Text Files
  • 2.1.2 Export Multiple Worksheets as Separate Text Files
  • 2.1.3 Create Index for All Worksheets
  • 2.1.4 Merge Two Spreadsheets using a Common Column
• 2.2 Manipulating Text Files with Python
  • 2.2.1 Read, Write, Split, Select Data

3. Data Wrangling: ready-made apps

• 3.1 Merge files by common column (python)
• 3.2 Aggregate data over slicing variations (python)

---

Homepage Prior Section Next Section