Session 1 · de-code data analysis

Democratizing Coding with AI — Mon 5 Oct

Author

nils.holmberg@iko.lu.se

Topics

1 Loading data

1.1 Load from GitHub

The sample dataset is a small simulated survey: 100 respondents, one group variable and six questionnaire items on a 1–5 scale, combined into three indices. The file lives in the course repository on GitHub. Its raw URL returns the plain file, so pandas can read it directly from the web — nothing to download first.

The file is tab-separated (.tsv), so we tell read_csv that the separator is a tab (\t).

import pandas as pd

url = "https://raw.githubusercontent.com/nils-holmberg/de-code/main/csv/komp014-simulated-data-n100.tsv"
df = pd.read_csv(url, sep="\t")

print(df.shape)  # (rows, columns)
df.head()
(100, 10)
IV_CAT_GROUP IV1_ITEM1 IV1_ITEM2 IV2_ITEM1 IV2_ITEM2 DV_ITEM1 DV_ITEM2 IV_INDEX1 IV_INDEX2 DV_INDEX
0 category2 4 3 1 1 1 1 3.5 1.0 1.0
1 category1 4 5 3 3 2 3 4.5 3.0 2.5
2 category2 1 2 1 1 1 1 1.5 1.0 1.0
3 category2 3 2 5 5 2 2 2.5 5.0 2.0
4 category1 2 2 1 2 1 1 2.0 1.5 1.0

1.2 Alternative: upload the file to Colab

If you have the file on your own computer, you can upload it to the Colab session instead. This cell is not run on the course website; in Colab, run it and pick the file in the dialog. Uploaded files end up in /content/, the working folder of the Colab session.

from google.colab import files

files.upload()  # choose komp014-simulated-data-n100.tsv
df = pd.read_csv("/content/komp014-simulated-data-n100.tsv", sep="\t")
df.head()

1.3 Colab: install pyreadstat

SPSS files (.sav) need the pyreadstat package, which Colab does not include. Run this cell once per Colab session before the next step.

%pip install pyreadstat

1.4 Same data, four formats

The same table can be stored in different file formats. Here we save df as CSV, TSV, Excel and SPSS, and attach a short label to each variable in the SPSS file.

from pathlib import Path

import pyreadstat

labels = {
    "IV_CAT_GROUP": "Group (category1 / category2)",
    "IV1_ITEM1": "IV1 item 1 (1-5)",
    "IV1_ITEM2": "IV1 item 2 (1-5)",
    "IV2_ITEM1": "IV2 item 1 (1-5)",
    "IV2_ITEM2": "IV2 item 2 (1-5)",
    "DV_ITEM1": "DV item 1 (1-5)",
    "DV_ITEM2": "DV item 2 (1-5)",
    "IV_INDEX1": "IV1 index, mean of IV1 items",
    "IV_INDEX2": "IV2 index, mean of IV2 items",
    "DV_INDEX": "DV index, mean of DV items",
}

out = Path("data")
out.mkdir(exist_ok=True)

df.to_csv(out / "komp014.csv", index=False)
df.to_csv(out / "komp014.tsv", sep="\t", index=False)
df.to_excel(out / "komp014.xlsx", index=False)
pyreadstat.write_sav(df, out / "komp014.sav",
                     column_labels=[labels[c] for c in df.columns])

sorted(p.name for p in out.iterdir())
['komp014.csv', 'komp014.sav', 'komp014.tsv', 'komp014.xlsx', 'run1.txt']

1.5 Compare the formats

Now we read each file back and compare it with the original: how big is the file, are the values the same, and did the column types (whole numbers, decimals, text) survive?

readers = {
    "csv": lambda p: pd.read_csv(p),
    "tsv": lambda p: pd.read_csv(p, sep="\t"),
    "xlsx": lambda p: pd.read_excel(p),
    "sav": lambda p: pyreadstat.read_sav(p)[0],
}

rows = []
for fmt, read in readers.items():
    path = out / f"komp014.{fmt}"
    back = read(path)
    rows.append({
        "format": fmt,
        "bytes": path.stat().st_size,
        # cast the original to the read-back types, then compare cell by cell
        "same values": back.equals(df.astype(back.dtypes.to_dict())),
        "same types": back.dtypes.astype(str).equals(df.dtypes.astype(str)),
        "text file": fmt in ("csv", "tsv"),
    })

pd.DataFrame(rows)
format bytes same values same types text file
0 csv 3500 True True True
1 tsv 3500 True True True
2 xlsx 8663 True True False
3 sav 10037 True False False
  • CSV / TSV are plain text: open them in any editor, small, universal. Column types are guessed again on every read, and there is no place for labels.
  • Excel is a zipped binary format: handy for people, larger on disk, keeps types.
  • SPSS .sav is binary and stores metadata with the data. Whole numbers come back as decimals (SPSS has one numeric type), but the variable labels travel with the file:
_, meta = pyreadstat.read_sav(out / "komp014.sav")
dict(zip(meta.column_names, meta.column_labels))
{'IV_CAT_GROUP': 'Group (category1 / category2)',
 'IV1_ITEM1': 'IV1 item 1 (1-5)',
 'IV1_ITEM2': 'IV1 item 2 (1-5)',
 'IV2_ITEM1': 'IV2 item 1 (1-5)',
 'IV2_ITEM2': 'IV2 item 2 (1-5)',
 'DV_ITEM1': 'DV item 1 (1-5)',
 'DV_ITEM2': 'DV item 2 (1-5)',
 'IV_INDEX1': 'IV1 index, mean of IV1 items',
 'IV_INDEX2': 'IV2 index, mean of IV2 items',
 'DV_INDEX': 'DV index, mean of DV items'}

1.6 Give an LLM context

A chat assistant cannot see your data. Paste a short description like the one below, followed by your question, and it can write code that fits your df. This cell only stores the text; it is not run on the course website.

prompt = """
I am analysing a survey dataset in Python with pandas. It is already loaded as `df`.

- 100 rows (respondents), 10 columns, no missing values
- IV_CAT_GROUP: text, two groups, "category1" and "category2" (50 each)
- IV1_ITEM1, IV1_ITEM2, IV2_ITEM1, IV2_ITEM2, DV_ITEM1, DV_ITEM2:
  whole numbers, questionnaire items on a 1-5 scale
- IV_INDEX1, IV_INDEX2, DV_INDEX: decimals 1-5, each the mean of its two items
  (IV_INDEX1 = mean of IV1_ITEM1/2, and so on)
- DV_INDEX is the outcome (dependent variable); IV_CAT_GROUP, IV_INDEX1 and
  IV_INDEX2 are the predictors (independent variables)

Continue from here. Write short, commented Python code and explain the results
in plain language.
"""
print(prompt)

2 Describing each variable

2.1 Descriptive statistics

describe() summarises every numeric column: how many values (count), the average (mean), the spread (std), the smallest and largest values, and the quartiles (25%, 50% = median, 75%). We add skewness (is the distribution lopsided?) and kurtosis (are the tails heavy?). Values near 0 mean roughly symmetric and normal-shaped.

numeric = df.select_dtypes("number")

stats = numeric.describe().T
stats["skew"] = numeric.skew()
stats["kurtosis"] = numeric.kurt()
stats.round(2)
count mean std min 25% 50% 75% max skew kurtosis
IV1_ITEM1 100.0 3.09 1.22 1.0 2.00 3.00 4.00 5.0 -0.01 -0.86
IV1_ITEM2 100.0 3.05 1.24 1.0 2.00 3.00 4.00 5.0 -0.06 -0.94
IV2_ITEM1 100.0 2.84 1.16 1.0 2.00 3.00 3.00 5.0 0.32 -0.53
IV2_ITEM2 100.0 2.78 1.31 1.0 2.00 3.00 4.00 5.0 0.17 -1.06
DV_ITEM1 100.0 2.57 1.20 1.0 2.00 2.00 3.25 5.0 0.42 -0.79
DV_ITEM2 100.0 2.56 1.17 1.0 2.00 3.00 3.00 5.0 0.34 -0.58
IV_INDEX1 100.0 3.07 1.11 1.0 2.38 3.00 4.00 5.0 -0.03 -0.89
IV_INDEX2 100.0 2.81 1.15 1.0 2.00 2.75 3.50 5.0 0.29 -0.79
DV_INDEX 100.0 2.56 1.09 1.0 1.88 2.50 3.50 5.0 0.46 -0.52

2.2 Frequencies

For the group variable and the 1–5 items, it is more useful to count how often each value occurs. The first table counts the two groups; the second shows, for each item, the share of respondents (in %) who chose each answer from 1 to 5.

df["IV_CAT_GROUP"].value_counts()
IV_CAT_GROUP
category2    50
category1    50
Name: count, dtype: int64
items = ["IV1_ITEM1", "IV1_ITEM2", "IV2_ITEM1", "IV2_ITEM2", "DV_ITEM1", "DV_ITEM2"]

freq = df[items].apply(lambda s: s.value_counts(normalize=True)).fillna(0) * 100
freq.index.name = "answer"
freq.round(0)
IV1_ITEM1 IV1_ITEM2 IV2_ITEM1 IV2_ITEM2 DV_ITEM1 DV_ITEM2
answer
1 11.0 13.0 12.0 21.0 20.0 22.0
2 21.0 21.0 28.0 23.0 35.0 26.0
3 32.0 28.0 36.0 25.0 20.0 33.0
4 20.0 24.0 12.0 19.0 18.0 12.0
5 16.0 14.0 12.0 12.0 7.0 7.0

2.3 Distributions

A histogram shows how the values of one variable are spread out: each bar counts the respondents in a range of values. The smooth line is a density curve, an estimate of the shape of the distribution. Items have only five possible values, so their bars are discrete; the indices take more values and look smoother.

import seaborn as sns

sns.set_theme(style="whitegrid")

fig, axes = plt.subplots(3, 3, figsize=(9, 7.5))
for ax, col in zip(axes.flat, numeric.columns):
    sns.histplot(df[col], bins=9, kde=True, color="#2f9e44", ax=ax)
    ax.set_title(col)
    ax.set_xlabel("")
fig.tight_layout()
plt.show()

2.4 Categorical variable

The group variable is text, so it gets a bar chart of counts instead of a histogram. Both groups have 50 respondents: the design is balanced.

fig, ax = plt.subplots(figsize=(4.5, 3.2))
sns.countplot(data=df, x="IV_CAT_GROUP", color="#2f9e44", ax=ax)
ax.set_xlabel("")
plt.show()

3 Relating the outcome to the predictors

3.1 Group comparison table

Now we look at two variables at a time: the outcome DV_INDEX against each predictor. We use the indices rather than the single items, since each index summarises its two items. First, the outcome split by group: does the average differ between the groups?

df.groupby("IV_CAT_GROUP")["DV_INDEX"].agg(["count", "mean", "std", "median"]).round(2)
count mean std median
IV_CAT_GROUP
category1 50 2.87 1.04 2.75
category2 50 2.26 1.06 2.00

3.2 Correlation table

For two numeric variables, the correlation (Pearson’s r) measures how closely they move together, from −1 (opposite) through 0 (unrelated) to +1 (in step).

df[["DV_INDEX", "IV_INDEX1", "IV_INDEX2"]].corr().round(2)
DV_INDEX IV_INDEX1 IV_INDEX2
DV_INDEX 1.00 0.41 0.41
IV_INDEX1 0.41 1.00 0.32
IV_INDEX2 0.41 0.32 1.00

3.3 Visualizations

Left: the outcome per group as a box plot (median, middle half and range) with each respondent as a dot. Middle and right: the outcome against each index, one dot per respondent, with a fitted straight line and its uncertainty band.

import numpy as np
np.random.seed(1)   # same random jitter and uncertainty band on every run

fig, axes = plt.subplots(1, 3, figsize=(10, 3.6))

sns.boxplot(data=df, x="IV_CAT_GROUP", y="DV_INDEX", color="#d3f9d8", ax=axes[0])
sns.stripplot(data=df, x="IV_CAT_GROUP", y="DV_INDEX", color="#2f9e44",
              alpha=0.6, jitter=0.2, ax=axes[0])
axes[0].set_xlabel("")

for ax, iv in zip(axes[1:], ["IV_INDEX1", "IV_INDEX2"]):
    sns.regplot(data=df, x=iv, y="DV_INDEX", color="#2f9e44",
                scatter_kws={"alpha": 0.6}, ax=ax, seed=1)

fig.tight_layout()
plt.show()

3.4 Significance tests

A significance test asks: could a difference or relationship this large plausibly appear by chance if there were none in the population? A small p-value (by convention below 0.05) says chance alone is an unlikely explanation. The effect size says how large the effect is, which p does not.

  • Group → DV: Welch’s t-test compares the two group means; Cohen’s d is the difference in standard-deviation units (around 0.2 small, 0.5 medium, 0.8 large).
  • Index → DV: Pearson’s correlation test; r itself is the effect size (around 0.1 small, 0.3 medium, 0.5 large).
from scipy import stats as st

g1, g2 = (df.loc[df["IV_CAT_GROUP"] == g, "DV_INDEX"] for g in ["category1", "category2"])
t = st.ttest_ind(g1, g2, equal_var=False)
pooled_sd = ((g1.var() + g2.var()) / 2) ** 0.5

results = [{
    "predictor": "IV_CAT_GROUP", "test": "Welch t-test",
    "statistic": t.statistic, "p": t.pvalue,
    "effect size": (g1.mean() - g2.mean()) / pooled_sd, "measure": "Cohen's d",
}]
for iv in ["IV_INDEX1", "IV_INDEX2"]:
    r = st.pearsonr(df[iv], df["DV_INDEX"])
    results.append({
        "predictor": iv, "test": "Pearson correlation",
        "statistic": r.statistic, "p": r.pvalue,
        "effect size": r.statistic, "measure": "r",
    })

results = pd.DataFrame(results).round(3)
results["p"] = results["p"].map(lambda p: "< .001" if p < 0.001 else f"{p:.3f}")
results
predictor test statistic p effect size measure
0 IV_CAT_GROUP Welch t-test 2.905 0.005 0.581 Cohen's d
1 IV_INDEX1 Pearson correlation 0.412 < .001 0.412 r
2 IV_INDEX2 Pearson correlation 0.411 < .001 0.411 r

The data are simulated, so these effects were built in on purpose; with real survey data, read them as associations, not proof of cause and effect.

3.5 Check the AI

AI assistants write convincing code, and sometimes convincing mistakes. Before you trust an AI-produced result, check one number yourself:

  1. Paste the context prompt from section 1.6 into a chat assistant.
  2. Ask it for the mean of DV_INDEX in each group, or the correlation between DV_INDEX and IV_INDEX1.
  3. Run its code and compare the answer with the tables above. Do they match? If not, which one is right — and why?