ISYE 6414
ISYE 6414 Final Project
Sign In

Datasets

Key Definitions

Dataset:A collection of data, typically a single file or table (e.g., a CSV file with unemployment rates by state).
Data Source:The organization or entity that aggregates, assembles, and publishes data (e.g., the Bureau of Labor Statistics).

Requirements

  • At least 3,000 rows in the data you actually analyze
    You need 3,000+ rows in the data you actually model — measured after cleaning and filtering, not before. Starting from 50,000 rows and filtering down to 400 does not meet this. You need enough left to split into training, validation and test sets and still fit something meaningful.
  • Enough columns for 10+ predictors per model
    Your data must support 10+ predictors per model (a categorical variable counts as 1, regardless of its number of levels).
    The point is to see meaningful variable selection applied: start broad, then let the data guide you toward a parsimonious final model.
  • How many datasets?
    One dataset is fine if it meets both thresholds. Join additional sources when they add information you actually need — not to hit a quota.

Example

A Group's Combined Dataset

Joining is often the right call. A combined dataset might look like:

DatasetData SourceRole
Housing prices (50,000+ rows)Zillow ResearchCore dataset
Unemployment ratesBureau of Labor StatisticsJoined by region + date
Population dataUS Census BureauJoined by region
✓ Valid — but so is the Zillow dataset on its own, if 3,000+ rows survive cleaning and it supports 10+ predictors.