From 5acf3d19a4906e2d8849d6fa338b7b32196db4e1 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 16 Jul 2026 12:19:44 +0100 Subject: [PATCH 1/2] feat: convert tidy data chapter from pandas to Polars --- data-tidy.ipynb | 265 ++++++++++++++++++++++++++---------------------- 1 file changed, 143 insertions(+), 122 deletions(-) diff --git a/data-tidy.ipynb b/data-tidy.ipynb index 157081f..8ce90e0 100644 --- a/data-tidy.ipynb +++ b/data-tidy.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Tidy Data {#sec-data-tidy}\n", @@ -19,7 +19,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "1", "metadata": { "tags": [ "remove-cell" @@ -38,17 +38,17 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package." ] }, { "cell_type": "markdown", - "id": "8b5f8255", + "id": "3", "metadata": {}, "source": [ "## Tidy Data\n", @@ -68,12 +68,12 @@ "1. There's a general advantage to picking one consistent way of storing data.\n", " If you have a consistent data structure, it's easier to learn the tools that work with it because they have an underlying uniformity. Some tools, for example data visualisation package **seaborn**, are designed with tidy data in mind.\n", "\n", - "2. There's a specific advantage to placing variables in columns because it allows you to take advantage of **pandas**' vectorised operations (operations that are more efficient).\n", + "2. There's a specific advantage to placing variables in columns because it allows you to take advantage of **polars**' vectorised operations (operations that are more efficient).\n", "\n", "\n", "Tidy data aren't going to be appropriate *every* time and in every case, but they're a really, really good default for tabular data. Once you use it as your default, it's easier to think about how to perform subsequent operations.\n", "\n", - "Having said that tidy data are great, they are, but one of **pandas**' advantages relative to other data analysis libraries is that it isn't *too* tied to tidy data and can navigate awkward non-tidy data manipulation tasks happily too.\n", + "Having said that tidy data are great, they are, but one of **polars**' advantages relative to other data analysis libraries is that it isn't *too* tied to tidy data and can navigate awkward non-tidy data manipulation tasks happily too.\n", "\n", "There are two common problems you find in data that are ingested that make them not tidy:\n", "\n", @@ -89,20 +89,20 @@ }, { "cell_type": "markdown", - "id": "deb8cf13", + "id": "4", "metadata": {}, "source": [ - "## Tools to Make Data Tidy with **pandas**" + "## Tools to Make Data Tidy with **polars**" ] }, { "cell_type": "markdown", - "id": "bd134b04", + "id": "5", "metadata": {}, "source": [ - "### Melt\n", + "### Unpivot\n", "\n", - "`melt()` can help you go from \"wider\" data to \"longer\" data, and is a *really* good one to remember.\n", + "`unpivot()` can help you go from \"wider\" data to \"longer\" data, and is a *really* good one to remember.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_melt.png)\n", "\n", @@ -112,13 +112,13 @@ { "cell_type": "code", "execution_count": null, - "id": "0f9fbf5a", + "id": "6", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", " \"first\": [\"John\", \"Mary\"],\n", " \"last\": [\"Doe\", \"Bo\"],\n", @@ -130,16 +130,21 @@ "print(\"\\n Unmelted: \")\n", "print(df)\n", "print(\"\\n Melted: \")\n", - "df.melt(id_vars=[\"first\", \"last\"], var_name=\"quantity\", value_vars=[\"height\", \"weight\"])" + "df.unpivot(\n", + " index=[\"first\", \"last\"],\n", + " variable_name=\"quantity\",\n", + " value_name=\"value\",\n", + " on=[\"height\", \"weight\"],\n", + ")" ] }, { "cell_type": "markdown", - "id": "e62978fd", + "id": "7", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", - "Perform a `melt()` that uses `job` as the id instead of `first` and `last`.\n", + "Perform a `unpivot()` that uses `job` as the id instead of `first` and `last`.\n", ":::\n", "\n", "How does this relate to tidy data? Sometimes you'll have a variable spread over multiple columns that you want to turn tidy. Let's look at this example that uses cases of [tuburculosis from the World Health Organisation](https://www.who.int/teams/global-tuberculosis-programme/data).\n", @@ -150,11 +155,11 @@ { "cell_type": "code", "execution_count": null, - "id": "bfa121cf", + "id": "8", "metadata": {}, "outputs": [], "source": [ - "df_tb = pd.read_parquet(\n", + "df_tb = pl.read_parquet(\n", " \"https://github.com/aeturrell/python4DS/raw/refs/heads/main/data/who_tb_cases.parquet\"\n", ")\n", "df_tb.head()" @@ -162,7 +167,7 @@ }, { "cell_type": "markdown", - "id": "583e419d", + "id": "9", "metadata": {}, "source": [ "You can see that we have two columns for a single variable, year. Let's now melt this." @@ -171,21 +176,21 @@ { "cell_type": "code", "execution_count": null, - "id": "dc03ccd9", + "id": "10", "metadata": {}, "outputs": [], "source": [ - "df_tb.melt(\n", - " id_vars=[\"country\"],\n", - " var_name=\"year\",\n", - " value_vars=[\"1999\", \"2000\"],\n", + "df_tb.unpivot(\n", + " index=[\"country\"],\n", + " variable_name=\"year\",\n", " value_name=\"cases\",\n", + " on=[\"1999\", \"2000\"],\n", ")" ] }, { "cell_type": "markdown", - "id": "74d81b30", + "id": "11", "metadata": {}, "source": [ "We now have one observation per row, and one variable per column: tidy!" @@ -193,31 +198,31 @@ }, { "cell_type": "markdown", - "id": "488f5f10", + "id": "12", "metadata": {}, "source": [ - "### A simpler wide to long\n", + "### Reshaping Wide Data with Multiple Variables\n", "\n", - "If you don't want the headscratching of `melt()`, there's also `wide_to_long()`, which is really useful for typical data cleaning cases where you have data like this:" + "Sometimes you have multiple variables spread across columns in a wide format. Here's how to handle that in polars:" ] }, { "cell_type": "code", "execution_count": null, - "id": "293768c1", + "id": "13", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", - " \"A1970\": {0: \"a\", 1: \"b\", 2: \"c\"},\n", - " \"A1980\": {0: \"d\", 1: \"e\", 2: \"f\"},\n", - " \"B1970\": {0: 2.5, 1: 1.2, 2: 0.7},\n", - " \"B1980\": {0: 3.2, 1: 1.3, 2: 0.1},\n", - " \"X\": dict(zip(range(3), np.random.randn(3))),\n", - " \"id\": dict(zip(range(3), range(3))),\n", + " \"A1970\": [\"a\", \"b\", \"c\"],\n", + " \"A1980\": [\"d\", \"e\", \"f\"],\n", + " \"B1970\": [2.5, 1.2, 0.7],\n", + " \"B1980\": [3.2, 1.3, 0.1],\n", + " \"X\": np.random.randn(3),\n", + " \"id\": [0, 1, 2],\n", " }\n", ")\n", "df" @@ -225,147 +230,154 @@ }, { "cell_type": "markdown", - "id": "fb6c09e6", + "id": "14", "metadata": {}, "source": [ - "i.e. data where there are different variables and time periods across the columns. Wide to long is going to let us give info on what the stubnames are ('A', 'B'), the name of the variable that's always across columns (here, a year), any values (X here), and an id column." + "To reshape this data, we first unpivot, then extract the year and variable:" ] }, { "cell_type": "code", "execution_count": null, - "id": "a9ca2fa8", + "id": "15", "metadata": {}, "outputs": [], "source": [ - "pd.wide_to_long(df, stubnames=[\"A\", \"B\"], i=\"id\", j=\"year\")" + "df_long = df.unpivot(\n", + " index=[\"id\", \"X\"],\n", + " variable_name=\"variable_year\",\n", + " value_name=\"value\",\n", + " on=[\"A1970\", \"A1980\", \"B1970\", \"B1980\"],\n", + ")\n", + "\n", + "df_long = df_long.with_columns(\n", + " [\n", + " pl.col(\"variable_year\").str.slice(0, 1).alias(\"variable\"),\n", + " pl.col(\"variable_year\").str.slice(1, 4).cast(pl.Int64).alias(\"year\"),\n", + " ]\n", + ")\n", + "\n", + "df_final = df_long.drop(\"variable_year\").pivot(\n", + " index=[\"id\", \"X\", \"year\"],\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ")\n", + "\n", + "df_final.sort([\"id\", \"year\"])" ] }, { "cell_type": "markdown", - "id": "1a6bf17c", + "id": "16", "metadata": {}, "source": [ "### Stack and Unstack\n", "\n", - "Stack, `stack()` is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", + "Polars uses unpivot() and pivot() for reshaping. \n", + "\n", + "Stack is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_stack.png)\n", "\n", - "Unstack, `unstack()` unsurprisingly does the same operation, but in reverse.\n", + "To go back to wider format (unstack)\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_unstack.png)\n", "\n", - "Let's define a multi-index data frame to demonstrate this:" + "Here's how to work with hierarchical data:" ] }, { "cell_type": "code", "execution_count": null, - "id": "2b791dd1", + "id": "17", "metadata": {}, "outputs": [], "source": [ - "tuples = list(\n", - " zip(\n", - " *[\n", - " [\"bar\", \"bar\", \"baz\", \"baz\", \"foo\", \"foo\", \"qux\", \"qux\"],\n", - " [\"one\", \"two\", \"one\", \"two\", \"one\", \"two\", \"one\", \"two\"],\n", - " ]\n", - " )\n", + "# Create a DataFrame with multiple columns\n", + "df = pl.DataFrame(\n", + " {\n", + " \"first\": [\"bar\", \"bar\", \"baz\", \"baz\", \"foo\", \"foo\", \"qux\", \"qux\"],\n", + " \"second\": [\"one\", \"two\", \"one\", \"two\", \"one\", \"two\", \"one\", \"two\"],\n", + " \"A\": np.random.randn(8),\n", + " \"B\": np.random.randn(8),\n", + " }\n", ")\n", - "index = pd.MultiIndex.from_tuples(tuples, names=[\"first\", \"second\"])\n", - "df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=[\"A\", \"B\"])\n", "df" ] }, { "cell_type": "markdown", - "id": "38e7c349", + "id": "18", "metadata": {}, "source": [ - "Let's stack this to create a tidy dataset:" + "To go to a longer format (stack):" ] }, { "cell_type": "code", "execution_count": null, - "id": "d25eb012", + "id": "19", "metadata": {}, "outputs": [], "source": [ - "df = df.stack()\n", - "df" - ] - }, - { - "cell_type": "markdown", - "id": "edeef123", - "metadata": {}, - "source": [ - "This has automatically created a multi-layered index but that can be reverted to a numbered index using `df.reset_index()`" + "# Unpivot to stack the A and B columns\n", + "df_stacked = df.unpivot(\n", + " index=[\"first\", \"second\"],\n", + " variable_name=\"variable\",\n", + " value_name=\"value\",\n", + " on=[\"A\", \"B\"],\n", + ")\n", + "df_stacked" ] }, { "cell_type": "markdown", - "id": "6b1e4e03", + "id": "20", "metadata": {}, "source": [ - "Now let's see unstack but, instead of unstacking the 'A', 'B' variables we began with, let's unstack the 'first' column by passing `level=0` (the default is to unstack the innermost index). This diagram shows what's going on:\n", - "\n", - "![](https://pandas.pydata.org/docs/_images/reshaping_unstack_0.png)\n", - "\n", - "And here's the code:" + "then to wider format (unstack)" ] }, { "cell_type": "code", "execution_count": null, - "id": "b6742a54", + "id": "21", "metadata": {}, "outputs": [], "source": [ - "df.unstack(level=0)" - ] - }, - { - "cell_type": "markdown", - "id": "185eae1f", - "metadata": {}, - "source": [ - "::: {.callout-tip title=\"Exercise\"}\n", - "What happens if you unstack to `level=1` instead? What about applying `unstack()` twice?\n", - ":::" + "# Pivot back to wider format\n", + "df_unstacked = df_stacked.pivot(\n", + " index=[\"first\", \"second\"],\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ")\n", + "df_unstacked" ] }, { "cell_type": "markdown", - "id": "39a210bb", + "id": "22", "metadata": {}, "source": [ - "### Pivoting data from long to wide\n", + "### Pivoting from Long to Wide\n", "\n", - "`pivot()` and `pivot_table()` help you to sort out data in which a single observation is scattered over multiple rows.\n", + "pivot() helps you to sort out data in which a single observation is scattered over multiple rows.\n", + "\n", + "![](https://pandas.pydata.org/docs/_images/reshaping_pivot.png)\n", "\n", - "![](https://pandas.pydata.org/docs/_images/reshaping_pivot.png)\n" - ] - }, - { - "cell_type": "markdown", - "id": "8acbcedf", - "metadata": {}, - "source": [ "Here's an example dataframe where observations are spread over multiple rows:" ] }, { "cell_type": "code", "execution_count": null, - "id": "fa612456", + "id": "23", "metadata": {}, "outputs": [], "source": [ - "df_tb_cp = pd.read_parquet(\n", + "df_tb_cp = pl.read_parquet(\n", " \"https://github.com/aeturrell/python4DS/raw/refs/heads/main/data/who_tb_case_and_pop.parquet\"\n", ")\n", "df_tb_cp.head()" @@ -373,49 +385,47 @@ }, { "cell_type": "markdown", - "id": "0d4a4077", - "metadata": {}, - "source": [ - "You see that we have, for each year-country, \"case\" and \"population\" in different rows." - ] - }, - { - "cell_type": "markdown", - "id": "e7c9ed1b", + "id": "24", "metadata": {}, "source": [ + "You see that we have, for each year-country, \"case\" and \"population\" in different rows.\n", + "\n", "Now let's pivot this to see the difference:" ] }, { "cell_type": "code", "execution_count": null, - "id": "e584cf37", + "id": "25", "metadata": {}, "outputs": [], "source": [ "pivoted = df_tb_cp.pivot(\n", - " index=[\"country\", \"year\"], columns=[\"type\"], values=\"count\"\n", - ").reset_index()\n", + " index=[\"country\", \"year\"],\n", + " columns=\"type\",\n", + " values=\"count\",\n", + " aggregate_function=\"first\",\n", + ").sort([\"country\", \"year\"])\n", "pivoted" ] }, { "cell_type": "markdown", - "id": "d82307c0", + "id": "26", "metadata": {}, "source": [ - "Pivots are especially useful for time series data, where operations like `shift()` or `diff()` are typically applied assuming that an entry in one row follows (in time) from the one above. When we do `shift()` we often want to shift a single variable in time, but if a single observation (in this case a date) is over multiple rows, the timing is going go awry. Let's see an example." + "Pivots are especially useful for time series data, where operations like `shift()` are typically applied assuming that an entry in one row follows (in time) from the one above. When we do `shift()` we often want to shift a single variable in time, but if a single observation (in this case a date) is over multiple rows, the timing goes awry. Let's see an example." ] }, { "cell_type": "code", "execution_count": null, - "id": "97c6d139", + "id": "27", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", + "import pandas as pd\n", "\n", "data = {\n", " \"value\": np.random.randn(20),\n", @@ -425,31 +435,42 @@ " + list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", " ),\n", "}\n", - "df = pd.DataFrame(data, columns=[\"date\", \"variable\", \"value\"])\n", + "df = pl.DataFrame(data)\n", "df.sample(5)" ] }, { "cell_type": "markdown", - "id": "90a5b930", + "id": "28", "metadata": {}, "source": [ - "If we just run `shift()` on the above, it's going to shift variable B's and A's together even though these overlap in time and are different variables. So we pivot to a wider format (and then we can shift in time safely)." + "If we just run `shift()` on the above, it's going to shift variable B's and A's together even though they overlap in time and are different variables. So we pivot to a wider format to shift safely:" ] }, { "cell_type": "code", "execution_count": null, - "id": "04f2bd28", + "id": "29", "metadata": {}, "outputs": [], "source": [ - "df.pivot(index=\"date\", columns=\"variable\", values=\"value\").shift(1)" + "df_pivoted = df.pivot(\n", + " index=\"date\",\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ").with_columns(\n", + " [\n", + " pl.col(\"A\").shift(1),\n", + " pl.col(\"B\").shift(1),\n", + " ]\n", + ")\n", + "df_pivoted" ] }, { "cell_type": "markdown", - "id": "841c4821", + "id": "30", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -471,7 +492,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 896c842b6af01df029f855af2fbbe77870a5f99e Mon Sep 17 00:00:00 2001 From: Your Name Date: Sat, 25 Jul 2026 00:42:40 +0100 Subject: [PATCH 2/2] updated convert data-tidy.ipynb to polars --- data-tidy.ipynb | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/data-tidy.ipynb b/data-tidy.ipynb index 8ce90e0..a1c90b2 100644 --- a/data-tidy.ipynb +++ b/data-tidy.ipynb @@ -11,7 +11,7 @@ "\n", "In this chapter, you will learn a consistent way to organise your data in Python using the principle known as *tidy data*. Tidy data is not appropriate for everything, but for a lot of analysis and a lot of tabular data it will be what you need. Getting your data into this format requires some work up front, but that work pays off in the long term. Once you have tidy data, you will spend much less time munging data from one representation to another, allowing you to spend more time on the data questions you care about.\n", "\n", - "In this chapter, you'll first learn the definition of tidy data and see it applied to simple toy dataset. Then we'll dive into the main tool you'll use for tidying data: melting. Melting allows you to change the form of your data, without changing any of the values. We'll finish up with a discussion of usefully untidy data, and how you can create it if needed.\n", + "In this chapter, you'll first learn the definition of tidy data and see it applied to simple toy dataset. Then we'll dive into the main tool you'll use for tidying data: unpivoting. Unpivoting allows you to change wide data into a long format without altering any underlying values. We'll finish up with a discussion of usefully untidy data, and how you can create it if needed.\n", "\n", "If you particularly enjoy this chapter and want to learn more about the underlying theory, you can learn more in the [Tidy Data](https://www.jstatsoft.org/article/view/v059i10) paper published in the Journal of Statistical Software.\n" ] @@ -127,9 +127,9 @@ " \"weight\": [130, 150],\n", " }\n", ")\n", - "print(\"\\n Unmelted: \")\n", + "print(\"\\n Wide DataFrame: \")\n", "print(df)\n", - "print(\"\\n Melted: \")\n", + "print(\"\\n Unpivoted (Long) DataFrame: \")\n", "df.unpivot(\n", " index=[\"first\", \"last\"],\n", " variable_name=\"quantity\",\n", @@ -170,7 +170,7 @@ "id": "9", "metadata": {}, "source": [ - "You can see that we have two columns for a single variable, year. Let's now melt this." + "You can see that we have two columns for a single variable, year. Let's now unpivot this." ] }, { @@ -259,7 +259,7 @@ "\n", "df_final = df_long.drop(\"variable_year\").pivot(\n", " index=[\"id\", \"X\", \"year\"],\n", - " columns=\"variable\",\n", + " on=\"variable\",\n", " values=\"value\",\n", " aggregate_function=\"first\",\n", ")\n", @@ -272,11 +272,11 @@ "id": "16", "metadata": {}, "source": [ - "### Stack and Unstack\n", + "### Reshaping Multi-Column Data\n", "\n", "Polars uses unpivot() and pivot() for reshaping. \n", "\n", - "Stack is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", + "In **polars**, reshaping between long and wide tabular formats is performed seamlessly using `unpivot()` (to go long) and `pivot()` (to go wide), without needing row indices or complex MultiIndexes.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_stack.png)\n", "\n", @@ -349,7 +349,7 @@ "# Pivot back to wider format\n", "df_unstacked = df_stacked.pivot(\n", " index=[\"first\", \"second\"],\n", - " columns=\"variable\",\n", + " on=\"variable\",\n", " values=\"value\",\n", " aggregate_function=\"first\",\n", ")\n", @@ -402,7 +402,7 @@ "source": [ "pivoted = df_tb_cp.pivot(\n", " index=[\"country\", \"year\"],\n", - " columns=\"type\",\n", + " on=\"type\",\n", " values=\"count\",\n", " aggregate_function=\"first\",\n", ").sort([\"country\", \"year\"])\n", @@ -425,15 +425,21 @@ "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", + "\n", + "dates = list(\n", + " pl.date_range(\n", + " start=pl.date(2000, 1, 31),\n", + " end=pl.date(2000, 10, 31),\n", + " interval=\"1mo\",\n", + " eager=True,\n", + " )\n", + ")\n", "\n", "data = {\n", " \"value\": np.random.randn(20),\n", " \"variable\": [\"A\"] * 10 + [\"B\"] * 10,\n", - " \"date\": (\n", - " list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", - " + list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", - " ),\n", + " \"date\": dates + dates,\n", "}\n", "df = pl.DataFrame(data)\n", "df.sample(5)" @@ -456,7 +462,7 @@ "source": [ "df_pivoted = df.pivot(\n", " index=\"date\",\n", - " columns=\"variable\",\n", + " on=\"variable\",\n", " values=\"value\",\n", " aggregate_function=\"first\",\n", ").with_columns(\n", @@ -474,12 +480,12 @@ "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", - "Why is the first entry NaN?\n", + "Why is the first entry `null`?\n", ":::\n", "\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", - "Perform a `pivot()` that applies to both the `variable` and `category` columns in the example from above where category is defined such that `df[\"category\"] = np.random.choice([\"type1\", \"type2\", \"type3\", \"type4\"], 20). (Hint: remember that you will need to pass multiple objects via a list.)\n", + "Perform a `pivot()` that applies to both the `variable` and `category` columns in the example from above where category is defined such that `df = df.with_columns(category=pl.Series(np.random.choice([\"type1\", \"type2\", \"type3\", \"type4\"], 20)))`. (Hint: pass multiple column names to the `on` parameter as a list.)\n", ":::" ] } @@ -492,7 +498,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" },