Data Science. Programming

at a company now?
How many U.S. states have they visited?
Their favorite unhealthy food… ?
Do they have any "Data Science" background? (statistics, machine learning, CS)

Where?

working at a company now?
How many U.S. states have they visited?
Their favorite unhealthy food… ?
Do they have any "Data Science" background? (statistics, machine learning, CS)

Zachary Dodds

Pittsburgh, PA

Harvey Mudd

Where?

44

mostly CS for me…

M&Ms

working at a company now?
How many U.S. states have they visited?
Their favorite unhealthy food… ?
Do they have any "Data Science" background? (statistics, machine learning, CS)

Zachary Dodds

Pittsburgh, PA

Harvey Mudd

Where?

44

mostly CS for me…

M&Ms

be sure to set up your login + profile for the submission site…

This class is truly seminar-style: I'm here, as you are, in order to gain insights into this very new field… .

http://www.quora.com/Neuroscience-1/How-much-data-can-the-human-brain-store

(2002) 5 EB: http://www2.sims.berkeley.edu/research/projects/how-much-info-2003/execsum.htm

1 Petabyte == 1000 TB

2002

2009

(2009) 800 EB: http://www.emc.com/collateral/analyst-reports/idc-digital-universe-are-you-ready.pdf

(2015) 8 ZB: http://www.emc.com/collateral/analyst-reports/idc-extracting-value-from-chaos-ar.pdf

2006

2011

(2006) 161 EB: http://www.emc.com/collateral/analyst-reports/expanding-digital-idc-white-paper.pdf

(2011) 1.8 ZB: http://www.emc.com/leadership/programs/digital-universe.htm

(life in video) 60 PB: in 4320p resolution, extrapolated from 16MB for 1:21 of 640x480 video (w/sound) – almost certainly a gross overestimate, as sleep can be compressed significantly!

5 EB

161 EB

800 EB

1.8 ZB

8.0 ZB

14 PB

60 PB

Data produced each year

100-years of HD video + audio

Human brain's capacity

Data, data everywhere…

References

1 TB = 1000 GB

120 PB

logarithmic scale

true that Data Science isn't a science – but that doesn't mean it's not useful!

our insights – large and small, permanent and ephemeral, natural and artificial – come about through the integration of lots of data.
Data Science simply recognizes that the rules and skills behind those insights are widely applicable…

Thought award, 2009

… Data Science is at the heart of computer science

and how do we succeed?

science

Stanford's Autonomous Vehicles project (Thrun et al.)

Learning to Powerslide

science

"my summer was finding that red line"

Learning ground from obstacles

Prize

Finding Nemo =
Lord of the Rings =

(I don't know this guy)

1.22
.75

??
??

Some films are difficult to predict…

Dynamite =
Batman Begins =

Finding Nemo =
Lord of the Rings =

1.22
.75

.67
.42

Some films are difficult to predict… and others are easier!

Netflix Prize

language)

Experience with several statistical analyses (descriptive statistics)

Experience with predictive statistics (modeling) and machine learning algorithms

whatever datasets you encounter in the future – on your own or as part of a team.

R statistical environment (and the S programming language)

Experience with several statistical analyses (descriptive statistics)

Experience with predictive statistics (modeling) and machine learning algorithms

Final project ~ open-ended with datasets of your choice

assignment: Getting started with R

Programming: R

Textbook

An introduction to Data Science

jsresearch.net/groups/teachdatascience/

www.r-project.org/

Grab both of these now…

freely available online

and many online resources…

two links…

the following week by 11:59 pm.

Assignment 1 due Tuesday, February 5.

1 week + 1 day…

the work at the keyboard evenly!

Submitting programs: at the submission website

Today's Lab:

install software ensure accounts are working

try out R - the first HW is officially due on 2/5

Assignments

~ 2-5 problems/week ~ 100 points extra credit, often

Due Tuesday of the following week by 11:59 pm.

Assignment 1 due Tuesday, February 5.

machines (SVMs)

random forests

k-means algorithm

nearest neighbors (NN)

Weeks 11-15

approximate!

Final Project

No breaks?!

grade = "A-"
if score >= 0.86: grade = "B+"

the last ~4 weeks will work towards a larger, final project

there will be a short design phase and a short final presentation

I'd encourage you to connect R and our Data Science techniques to other datasets or projects that you use/need/like, etc.

Based on points percentage

~ 800 points for assignments

see the course syllabus for the full list...

~ 400 points for the final project

choose your own problem to study (I'll have some suggestions, too.)

Academic Honesty policies…
Your work must be your own. This must be true for the whole team, if you're working in a pair.
Consulting with others (except team members or myself) is encouraged, but has to be limited to discussion and debugging of problems. Sharing of written, electronic, or verbal solutions/files/code is a violation of CGU’s academic honesty policy.
A reasonable guideline: Work is your own if you could delete all of it and recreate it yourself.

Stata, SPSS are preferred by those in business intelligence

vs. Statistics

Download and install R

3) Run R and try some basic commands at the prompt:

6 * 7

rnorm(10)

x <- 380

7, the "5 minute example session"
3) Try out the commands in section 2.2 to get started…
4) When you finish, save your session and submit it!

This is problem 1 this week

as… (Mac) in order to save your current console session into hw1

This is problem 1 this week

1) Create a folder named hw1, perhaps on your desktop

3) Name that file pr1.txt

4) From your operating system, open up that file in order to confirm it contains your whole session!

site link.

You've completed Problem 1!

1) Zip up hw1 into hw1.zip

3) Choose a submission site login name & let me know!

4) Once your account is made, login, change your password to something you know, and submit hw1.zip

This webserver can be spacey -- I should know!

troubles? email me!

5) You can submit again – all copies are saved…

variable.

of element

c ~ concatenate

of element

c ~ concatenate

the colon : also creates vectors

vectors

a boolean vector to subset another vector

going on here?

The function is.na( ) tests for NA

going on here?

The function is.na( ) tests for NA

This uses subsetting to remove NA values!

introduce them…

welcome you to stay for the lab, but it is not required.

Today's lab:

Work through Santorico and Shin's Tutorial for the R Statistical Package and submit the console sessions as pr2_1.txt, pr2_1.txt, pr2_1.txt, pr2_1.txt, and pr2_1.txt.

This is a nice reinforcement of vectors, introduction to data frames, and a look at the graphics that R supports.

data-analysis introduction from pr2's tutorial.

Problem 4: Introduction to Data Science, early chapters

This is a fuller background on R and the field of data science

(submit your console session for both of these…)

specifics

and data

inheritance

Week 13

Week 14

Week 15

final projects

final projects

final exam

at a company now?
How many U.S. states have they visited?
Their favorite unhealthy food… ?
Do they have any "Data Science" (statistics, machine learning, CS) background?

Where?

working at a company now?
How many U.S. states have they visited?
Their favorite unhealthy food… ?
Do they have any "Data Science" (statistics, machine learning, CS) background?

Zachary Dodds

Pittsburgh, PA

Harvey Mudd

Where?

44

mostly CS for me…

M&Ms