Causality, Research Design, Survey Methods, and Measurement
The University of Sydney
Semester 2, 2026
I would like to acknowledge the Traditional Owners of Australia and recognise their continuing connection to land, water and culture. The University of Sydney is located on the land of the Gadigal people of the Eora Nation. I pay my respects to their Elders, past and present.
Textbook: Kellstedt, Whitten, and Tuch (2022)
This week’s chapters:
Chapters 3-6: Building Research Foundations
By the end of today, you will be able to:
Modern sociology fundamentally revolves around establishing whether there are causal relationships between important concepts.
— Kellstedt, Whitten, and Tuch (2022, chap. 3)
Remember
Finding a relationship ≠ Finding a causal relationship
In social science:
To claim X causes Y, we must cross four hurdles:
Question: How and why would X cause Y?
Think Through the Process
Example: School choice → Test scores
Question: Could Y cause X instead?
Sometimes easy to rule out:
Sometimes difficult:
Question: Do X and Y vary together?
Important Caveat
No bivariate relationship? Don’t give up yet! A confounding variable Z might be suppressing the relationship.
Question: Have we controlled for other causes of Y?
Track answers with: [hurdle1 hurdle2 hurdle3 hurdle4]
In groups, evaluate these claims using the scorecard:
For each claim:
The four hurdles:
Good research design helps us:
Important
Bad design → Bad statistics (no matter how sophisticated!)
Key features:
Example: Testing teaching methods
Challenges:
Example: Effect of poverty on crime
Middle ground:
Example: Voting law changes
| Design | Mechanism | Reverse | Covariation | Confounders |
|---|---|---|---|---|
| Experimental | Test | ✓✓✓ | Test | ✓✓✓ |
| Quasi-exp. | Test | ✓✓ | Test | ✓✓ |
| Observational | Test | ✓ | Test | ✓ |
Better design → More hurdles cleared
In groups of 3-4:
For each research scenario, identify:
Scenarios on handout
Surveys are a primary tool for:
But: Survey design matters!
The Goal:
Key Question
Is your sample representative of the population?
Tip
Always consider: What biases might affect these survey results?
The measurement process:
Concept (abstract idea)
↓
Operationalisation (how to measure)
↓
Variable (actual data)
Each choice has implications!
Two fundamental properties of good measurement:
Central question: Does this measure actually capture the concept I care about?
The Core Issue
A measure can be reliable but not valid!
Example: Measuring “intelligence” by shoe size - Very reliable (shoe size doesn’t change much) - Not valid (shoe size isn’t intelligence)
Concept: How much do people know about politics?
Central question: Would we get the same result if we measured again?
Key Principle
Without reliability, we can’t have validity!
If a thermometer gives random readings, it can’t accurately measure temperature.
Definition: Do you get the same score if you measure the same person twice?
Definition: Do different observers/coders give the same score?
Definition: Do multiple items measuring the same concept give similar results?
Single Item: “How satisfied are you with your life?” (1-10 scale)
Multiple Items: Life Satisfaction Scale
All items should correlate (internal consistency)
Four possible scenarios:
| Reliable | Unreliable | |
|---|---|---|
| Valid | ✓✓ IDEAL Accurate & consistent |
✗ Problematic Can’t be valid without reliability |
| Invalid | ✗ Precise but wrong (like a biased scale) |
✗✗ WORST Random noise |
Tip
Think of a dartboard: - Valid & Reliable: Darts clustered around bullseye - Reliable but Invalid: Darts clustered, but in wrong spot - Unreliable: Darts scattered everywhere
When developing measures, ask:
Good measurement is the foundation of good research!
Three measurement levels:
Why care about measurement level?
In JASP
You’ll need to specify: Nominal, Ordinal, or Scale
What You See When You Start JASP
Two Main Ways to Get Data
.csv file (you can also open Excel files)CSV File Format: - Plain text file - Comma-separated values - Can be created in Excel, Google Sheets, etc. - First row = variable names
.jasp fileFirst Dataset: titanic.csv
Three Variable Types in JASP
JASP uses symbols in column headers to show variable type:
| Type | Description | Example |
|---|---|---|
| Scale | Continuous numerical data | Age, Height, Income |
| Ordinal | Ordered categories | Education level (1=Low, 2=Medium, 3=High) |
| Nominal | Unordered categories | Gender, Color, Country |
IMPORTANT
Two Methods:
Important Tips:
Your Task:
titanic.csv from Canvastitanic.csv and open itExpected: 891 rows, 12 columns
The Titanic Dataset Variables:
Look at your data and identify which type each variable should be:
| Variable | Current Type? | Should Be? |
|---|---|---|
| PassengerId | ? | ? |
| Survived | ? | ? |
| Pclass | ? | ? |
| Name | ? | ? |
| Sex | ? | ? |
| Age | ? | ? |
| Fare | ? | ? |
Your Task:
| Variable | Should Be | Why? |
|---|---|---|
| PassengerId | Nominal | ID number (not meaningful numerically) |
| Survived | Nominal | Binary: 0 = No, 1 = Yes |
| Pclass | Nominal or Ordinal | Ticket class: 1, 2, 3 (categories) |
| Name | Nominal | Text label (categorical) |
| Sex | Nominal | male or female (categorical) |
| Age | Scale | Continuous number (0-80 years) |
| Fare | Scale | Continuous number (price paid) |
Key Decisions:
Making Your Data More Readable
Make the Survived Variable More Interpretable
Step-by-Step Instructions:
Summarizing Your Data
Central tendency: Where is the “middle”? - Mean, Median, Mode
Spread: How spread out are the values? - Standard deviation, Variance, Range - Minimum, Maximum, Quartiles
Frequencies: How many in each category? - Counts and percentages
Why Use Them? - Understand your data before analysis - Spot errors and outliers - Describe your sample in reports
Finding the “Middle” of Your Data
Which to use?
How Variable Is Your Data?
Step-by-Step Guide
Your Task:
Calculate descriptive statistics for the Age variable:
Expected Results (approximately):
| Statistic | Value | Interpretation |
|---|---|---|
| Mean | ~29.7 | Average age around 30 years |
| Median | ~28.0 | Middle age is 28 |
| SD | ~14.5 | Typical deviation is ±14-15 years |
| Min | ~0.42 | Youngest passenger was an infant |
| Max | ~80 | Oldest was 80 years old |
| Range | ~80 | Ages span 80 years |
Observations:
Counting Categories
| Value | Count | Percent |
|---|---|---|
| 0 (No) | 549 | 61.6% |
| 1 (Yes) | 342 | 38.4% |
| Value | Count | Percent |
|---|---|---|
| male | 577 | 64.8% |
| female | 314 | 35.2% |
| Value | Count | Percent |
|---|---|---|
| 1 (First) | 216 | 24.2% |
| 2 (Second) | 184 | 20.7% |
| 3 (Third) | 491 | 55.1% |
Insights:
Understanding Survival Patterns Across Groups
The Question: Did survival rates differ by passenger class? By sex?
Part A: Survival by Passenger Class
What You See:
Observe:
Now You Try:
Using the same approach:
Questions to Answer:
Note
Discuss: What story do these two analyses tell us about survival on the Titanic?
Adding Plots to Descriptives
Under Basic Plots/Customizable Plots in Descriptive Statistics:
Your Task:
Create visualizations for the Age variable:
Understanding Box-and-Whisker Plots
● ← Outlier (beyond 1.5 × IQR)
|
┬ ← Maximum (within 1.5 × IQR)
|
┼──┐
│ │
│──│ ← Median (thick line)
│ │
└──┼
|
┴ ← Minimum (within 1.5 × IQR)
|
● ← Outlier
Quick Assessment: - Symmetric box → Symmetric distribution - Long upper whisker → Right skew - Dots → Potential outliers to investigate
Comparing Groups
What if we want to compare subgroups?
Example: Age statistics separately for males and females
Result: - Separate statistics for each group - Easy comparison across groups - Can also split plots!
Your Task:
Compare age distributions for survivors vs non-survivors:
Expected Pattern:
| Survived | Mean Age | Median Age | Observations |
|---|---|---|---|
| 0 (Died) | ~30.6 | 28 | Slightly older |
| 1 (Survived) | ~28.3 | 28 | Slightly younger |
This is where hypothesis testing comes in!
Your Challenge:
Perform a complete descriptive analysis of Fare:
Transforming Data in JASP
Common Uses: - Log transformations → Normalize skewed data - Z-scores → Standardize variables - Sum scores → Combine multiple items - Difference scores → Change from baseline - Means → Average across variables
Let’s create two new variables together
First variable: FARE + 1
Then use this new variable to create: log(FARE + 1)
What’s the point of taking the log of FARE + 1?
This is necessary because some passengers paid 0 as FARE. And \log(0) = -\infty.
Of course, -\infty would be impossible to visualize.
Adding 1 to variables that contain zeros is a common, quick-and-dirty trick to still return meaningful visualizations.
What Did We Just Do?
The Formula: log(Fare+1) - Takes the natural logarithm (base e) of each Fare+1 value - Compresses large values more than small values - Result: more symmetric distribution
Example Transformations:
| Original Fare | log(Fare+1) | Interpretation |
|---|---|---|
| £1 | 0.00 | Log of 1 is 0 |
| £10 | 2.30 | |
| £100 | 4.61 | Less difference at high end |
| £500 | 6.21 |
Notice: The gap between £1 and £100 is huge in original scale, but only 4.61 units in log scale. Meanwhile, £100 to £500 (5x increase) is only 1.6 log units.
Now let’s see the difference:
Saving Your Work
Tip
Pro Tip: Save as .jasp file regularly to preserve your work!
The research process:
Next week: Statistical Analysis
Before Next Week
Familiarise yourself with the JASP interface. Explore datasets on your own!
Key takeaways:
Preparing for your own research:
Think about your research question - which design would work? What variables would you need?
See you next week!
Resources: