I’m poking at a little dataset from my own life: hours of sleep vs my score on next-day practice quizzes. The scatterplot looks like a curved hill – low scores with very little sleep, higher scores around 7–8 hours, then lower again when I oversleep. But when I compute Pearson’s correlation, it’s basically 0, which feels wrong because the relationship looks real and strong, just not a straight line. Am I misunderstanding what correlation is actually measuring? Is Pearson only capturing linear patterns? Should I be using Spearman instead, or would Spearman also miss this since the pattern isn’t monotonic? Follow-up: would transforming a variable (like squaring hours of sleep or taking a log) be an appropriate way to make correlation tell me something useful here, or does standardizing/transforming not fix this kind of issue?
Ready to make maths more enjoyable, accessible, and fun? Join a friendly community where you can explore puzzles, ask questions, track your progress, and learn at your own pace.
By becoming a member, you unlock:
3 Responses
You’re seeing exactly what Pearson’s r is (and isn’t) built to do: it measures linear association, so a strong curved relationship can have r ≈ 0 even when the connection is real. In fact, a perfectly “inverted U” that’s symmetric around its peak has zero Pearson correlation. Simple example: take x = 0,1,2,3,4,5,6 and y = −(x−3)², which is a perfect hill. The mean of x is 3 and of y is −4; if you compute the covariance Σ(x−3)(y+4), the positive and negative parts cancel to 0, so r = 0 despite a perfect deterministic relationship. Spearman’s rank correlation looks for monotonic (always increasing or always decreasing) patterns, so it will also be near 0 for a hill shape. What to do instead: model the curve. A straightforward approach is a quadratic regression, score ≈ β0 + β1·sleep + β2·sleep², or a spline/LOESS smoother; these capture the peak and give you an R² that reflects the strong non-linear fit. If you prefer a quick summary statistic aligned with your story (“best around 7–8 hours, worse as you stray”), transform sleep into “distance from optimal,” d = |sleep − h*|, and correlate score with d or −d²; that’s a legitimate, interpretable transformation because it encodes the hypothesized mechanism. Note that standardizing (z-scoring) or monotone transforms like logs won’t fix a non-monotonic shape; they can’t turn a hill into a line. If you want a form-agnostic dependence measure, consider distance correlation or the maximal information coefficient, which can detect general non-linear associations. For a clear refresher on what Pearson’s r measures and why non-linear patterns can yield r ≈ 0, see Khan Academy’s overview: https://www.khanacademy.org/math/statistics-probability/describing-relationships-quantitative-data/more-on-correlation/v/pearson-correlation-coefficient.
Yep-Pearson only has eyes for straight lines. A hump-shaped “best-at-the-middle” pattern will often give you r ≈ 0 because the positive slope on the left cancels the negative slope on the right. Spearman is no rescue here either: it measures monotonic association (always going up or always going down). Your scores go up then down, so the rank correlation also washes out. Zero correlation doesn’t mean “no relationship”; it means “no linear (or monotonic) relationship.”
What to do: model the curve, not the line. Two easy options. (1) Fit a quadratic: score ≈ a + b1·hours + b2·hours². If your hill is real, b2 will be negative and the R² will jump. (2) Re-express hours as “distance from best sleep”: d = hours − 7.5 (or wherever the peak is), then correlate score with −d² (or just use d and d² in a regression). Simple example: take x = −2, −1, 0, 1, 2 and y = −x². The relation is perfect and curved, but Pearson r(x, y) = 0 because left and right cancel. However, r(x², y) = −1 exactly, because y is a perfect negative multiple of x². That’s your data in spirit: center hours at your peak, square the distance, and you’ll see the relationship loud and clear. Standardizing or logging won’t fix a U-shape; centering and adding a squared term will.
Yes-Pearson’s correlation measures the strength of a linear relationship, so a symmetric “hill” (low–high–low) can average to a slope of about zero even when the relationship is strong. Think of it like hiking up and then down a hill: your net elevation change is zero, but you clearly did a lot of climbing. Spearman’s rank correlation captures monotonic trends; a rise-then-fall is not monotonic, so Spearman will also be near zero. A better approach is to model the curvature directly: fit a regression with a quadratic term, score ≈ β0 + β1·sleep + β2·sleep^2, or use a smooth curve (e.g., loess/splines). This will quantify the U-shape and estimate the peak around 7–8 hours. If you want a single-number summary, correlate score with the absolute deviation from optimal sleep, |sleep − x0|, where x0 can be estimated from the quadratic fit; that should be negatively related to score. Standardizing does not fix this, and generic monotone transforms (like logs) won’t turn a hill into a line. Transformations can help only if they reflect a plausible model (e.g., including both sleep and sleep^2), and you should check fit and avoid ad hoc fishing-cross-validation or holdout checks are sensible.