From 03e70f993c41b4a953156f45669e8c10900f740c Mon Sep 17 00:00:00 2001 From: Your Name Date: Sun, 12 Jul 2026 05:00:56 +0100 Subject: [PATCH 01/30] refactor: replace pandas with polars in vis-layers.ipynb data processing examples --- vis-layers.ipynb | 205 +++++++++++++++++++++++------------------------ 1 file changed, 101 insertions(+), 104 deletions(-) diff --git a/vis-layers.ipynb b/vis-layers.ipynb index 6a92e84..1807a29 100644 --- a/vis-layers.ipynb +++ b/vis-layers.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Layers {#sec-vis-layers}\n" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "9fdd3b8a", + "id": "1", "metadata": {}, "source": [ "## Introduction\n", @@ -30,12 +30,12 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "You will need to install the **letsplot** package for this chapter, as well as **pandas**.\n", + "You will need to install the **letsplot** package for this chapter, as well as **polars**.\n", "\n", "In your Python session, import the libraries we'll be using:" ] @@ -43,11 +43,12 @@ { "cell_type": "code", "execution_count": null, - "id": "a86fb211", + "id": "3", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "\n", + "import polars as pl\n", "from lets_plot import *\n", "\n", "LetsPlot.setup_html()" @@ -55,7 +56,7 @@ }, { "cell_type": "markdown", - "id": "55b00fde", + "id": "4", "metadata": {}, "source": [ "## Aesthetic mappings\n", @@ -68,35 +69,29 @@ { "cell_type": "code", "execution_count": null, - "id": "39a6d993", + "id": "5", "metadata": {}, "outputs": [], "source": [ - "mpg = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n", + "mpg = pl.read_csv(\n", + " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\"\n", ")\n", "\n", - "mpg = mpg.astype(\n", - " {\n", - " \"manufacturer\": \"category\",\n", - " \"model\": \"category\",\n", - " \"displ\": \"double\",\n", - " \"year\": \"int64\",\n", - " \"cyl\": \"int64\",\n", - " \"trans\": \"category\",\n", - " \"drv\": \"category\",\n", - " \"cty\": \"double\",\n", - " \"hwy\": \"double\",\n", - " \"fl\": \"category\",\n", - " \"class\": \"category\",\n", - " }\n", - ")\n", + "mpg = mpg.with_columns([\n", + " pl.col(\"manufacturer\").cast(pl.Categorical),\n", + " pl.col(\"model\").cast(pl.Categorical),\n", + " pl.col(\"trans\").cast(pl.Categorical),\n", + " pl.col(\"drv\").cast(pl.Categorical),\n", + " pl.col(\"fl\").cast(pl.Categorical),\n", + " pl.col(\"class\").cast(pl.Categorical),\n", + "])\n", + "\n", "mpg.head()" ] }, { "cell_type": "markdown", - "id": "6d6f1307", + "id": "6", "metadata": {}, "source": [ "Among the variables in `mpg` are:\n", @@ -118,7 +113,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fe77349a", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -128,7 +123,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e77b5640", + "id": "8", "metadata": {}, "outputs": [], "source": [ @@ -137,7 +132,7 @@ }, { "cell_type": "markdown", - "id": "53e51510", + "id": "9", "metadata": {}, "source": [ "Similarly, we can map `class` to `size` or `alpha` aesthetics as well, which control the shape and the transparency of the points, respectively." @@ -146,7 +141,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ef221330", + "id": "10", "metadata": {}, "outputs": [], "source": [ @@ -156,7 +151,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d042255e", + "id": "11", "metadata": {}, "outputs": [], "source": [ @@ -165,7 +160,7 @@ }, { "cell_type": "markdown", - "id": "ec07349f", + "id": "12", "metadata": {}, "source": [ "While we are able to do it, mapping an unordered discrete (categorical) variable (`class`) to an ordered aesthetic variable (`size` or `alpha`) is generally not a good idea because it implies a ranking that does not in fact exist.\n", @@ -182,7 +177,7 @@ { "cell_type": "code", "execution_count": null, - "id": "618edcb4", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -191,7 +186,7 @@ }, { "cell_type": "markdown", - "id": "52611640", + "id": "14", "metadata": {}, "source": [ "Here, the colour doesn't convey information about a variable, but only changes the appearance of the plot.\n", @@ -206,7 +201,7 @@ }, { "cell_type": "markdown", - "id": "37d49ff8", + "id": "15", "metadata": {}, "source": [ "So far we have discussed aesthetics that we can map or set in a scatterplot, when using a point geom.\n", @@ -217,7 +212,7 @@ }, { "cell_type": "markdown", - "id": "3f1da019", + "id": "16", "metadata": {}, "source": [ "1. Create a scatterplot of `hwy` vs. `displ` where the points are pink filled in triangles.\n", @@ -240,7 +235,7 @@ }, { "cell_type": "markdown", - "id": "83aa98f0", + "id": "17", "metadata": {}, "source": [ "## Geometric objects\n", @@ -251,7 +246,7 @@ { "cell_type": "code", "execution_count": null, - "id": "277a4c0f", + "id": "18", "metadata": {}, "outputs": [], "source": [ @@ -261,7 +256,7 @@ { "cell_type": "code", "execution_count": null, - "id": "07247ba9", + "id": "19", "metadata": {}, "outputs": [], "source": [ @@ -270,7 +265,7 @@ }, { "cell_type": "markdown", - "id": "58824a10", + "id": "20", "metadata": {}, "source": [ "Both plots contain the same x variable, the same y variable, and both describe the same data.\n", @@ -283,7 +278,7 @@ }, { "cell_type": "markdown", - "id": "0db26c6c", + "id": "21", "metadata": {}, "source": [ "Every geom function in **lets-plot** takes a `mapping` argument, either defined locally in the geom layer or globally in the `ggplot()` layer.\n", @@ -299,7 +294,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4b20c825", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -309,7 +304,7 @@ { "cell_type": "code", "execution_count": null, - "id": "84df3e78", + "id": "23", "metadata": {}, "outputs": [], "source": [ @@ -318,7 +313,7 @@ }, { "cell_type": "markdown", - "id": "7b114911", + "id": "24", "metadata": {}, "source": [ "Here, `geom_smooth()` separates the cars into three lines based on their `drv` value, which describes a car's drive train.\n", @@ -331,7 +326,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c9e8d92f", + "id": "25", "metadata": {}, "outputs": [], "source": [ @@ -344,7 +339,7 @@ }, { "cell_type": "markdown", - "id": "b6392da1", + "id": "26", "metadata": {}, "source": [ "Notice that this plot contains two geoms in the same graph.\n", @@ -363,7 +358,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b3916558", + "id": "27", "metadata": {}, "outputs": [], "source": [ @@ -372,7 +367,7 @@ }, { "cell_type": "markdown", - "id": "88708546", + "id": "28", "metadata": {}, "source": [ "You can use the same idea to specify different data for each layer.\n", @@ -383,23 +378,23 @@ { "cell_type": "code", "execution_count": null, - "id": "38870eb5", + "id": "29", "metadata": {}, "outputs": [], "source": [ "(\n", " ggplot(mpg, aes(x=\"displ\", y=\"hwy\"))\n", " + geom_point()\n", - " + geom_point(data=mpg.loc[mpg[\"class\"] == \"2seater\", :], color=\"red\", size=2)\n", + " + geom_point(data=mpg.filter(pl.col(\"class\") == \"2seater\"), color=\"red\", size=2)\n", " + geom_point(\n", - " data=mpg.loc[mpg[\"class\"] == \"2seater\", :], shape=1, size=3, color=\"red\"\n", + " data=mpg.filter(pl.col(\"class\") == \"2seater\"), shape=1, size=3, color=\"red\"\n", " )\n", ")" ] }, { "cell_type": "markdown", - "id": "35a3d017", + "id": "30", "metadata": {}, "source": [ "Geoms are the fundamental building blocks of **lets-plot**.\n", @@ -415,7 +410,7 @@ }, { "cell_type": "markdown", - "id": "39a12f36", + "id": "31", "metadata": {}, "source": [ "### Exercises\n", @@ -443,7 +438,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ae75c5c1", + "id": "32", "metadata": { "tags": [ "remove-cell" @@ -460,7 +455,7 @@ }, { "cell_type": "markdown", - "id": "4c1d45ab", + "id": "33", "metadata": {}, "source": [ "## Facets\n", @@ -471,7 +466,7 @@ { "cell_type": "code", "execution_count": null, - "id": "cb651300", + "id": "34", "metadata": {}, "outputs": [], "source": [ @@ -480,7 +475,7 @@ }, { "cell_type": "markdown", - "id": "0cd9ef67", + "id": "35", "metadata": {}, "source": [ "To facet your plot with the combination of two variables, switch from `facet_wrap()` to `facet_grid()`." @@ -489,7 +484,7 @@ { "cell_type": "code", "execution_count": null, - "id": "61481052", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -498,7 +493,7 @@ }, { "cell_type": "markdown", - "id": "4f502141", + "id": "37", "metadata": {}, "source": [ "By default each of the facets share the same scale and range for x and y axes.\n", @@ -509,7 +504,7 @@ { "cell_type": "code", "execution_count": null, - "id": "adcd9079", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -523,7 +518,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ceb2a354", + "id": "39", "metadata": {}, "outputs": [], "source": [ @@ -532,7 +527,7 @@ }, { "cell_type": "markdown", - "id": "5e3a949f", + "id": "40", "metadata": {}, "source": [ "### Exercises\n", @@ -602,7 +597,7 @@ }, { "cell_type": "markdown", - "id": "cacf1fb5", + "id": "41", "metadata": {}, "source": [ "## Statistical transformations\n", @@ -616,34 +611,36 @@ { "cell_type": "code", "execution_count": null, - "id": "f379e31b", + "id": "42", "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv\",\n", - " index_col=0,\n", + "diamonds = pl.read_csv(\n", + " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv\"\n", ")\n", "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", - "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n", - " pd.CategoricalDtype(categories=diamonds_cut_order, ordered=True)\n", + "diamonds = diamonds.with_columns(\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order))\n", ")\n", + "\n", "diamonds.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "d8faf1ab", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "(ggplot(diamonds, aes(x=\"cut\")) + geom_bar())" + "diamonds_sorted = diamonds.sort(\"cut\")\n", + "\n", + "(ggplot(diamonds_sorted, aes(x=\"cut\")) + geom_bar())" ] }, { "cell_type": "markdown", - "id": "a27666cf", + "id": "44", "metadata": {}, "source": [ "On the x-axis, the chart displays `cut`, a variable from `diamonds`.\n", @@ -666,7 +663,7 @@ }, { "cell_type": "markdown", - "id": "62519f73", + "id": "45", "metadata": {}, "source": [ "You can learn which stat a geom uses by inspecting the default value for the `stat` argument.\n", @@ -680,22 +677,25 @@ { "cell_type": "code", "execution_count": null, - "id": "ca772dd5", + "id": "46", "metadata": {}, "outputs": [], "source": [ + "diamonds_counts = (\n", + " diamonds.group_by(\"cut\")\n", + " .agg(pl.len().alias(\"counts\"))\n", + " .sort(\"cut\") \n", + ")\n", + "\n", "(\n", - " ggplot(\n", - " diamonds.value_counts(\"cut\").reset_index(name=\"counts\"),\n", - " aes(x=\"cut\", y=\"counts\"),\n", - " )\n", + " ggplot(diamonds_counts, aes(x=\"cut\", y=\"counts\"))\n", " + geom_bar(stat=\"identity\")\n", ")" ] }, { "cell_type": "markdown", - "id": "e365aaaf", + "id": "47", "metadata": {}, "source": [ "## Position adjustments\n", @@ -707,7 +707,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f8da7d91", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -717,7 +717,7 @@ { "cell_type": "code", "execution_count": null, - "id": "088e7550", + "id": "49", "metadata": {}, "outputs": [], "source": [ @@ -726,7 +726,7 @@ }, { "cell_type": "markdown", - "id": "c3f7a0a3", + "id": "50", "metadata": {}, "source": [ "Note what happens if you map the fill aesthetic to another variable, like `class`: the bars are automatically stacked.\n", @@ -736,7 +736,7 @@ { "cell_type": "code", "execution_count": null, - "id": "181c70d2", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -745,7 +745,7 @@ }, { "cell_type": "markdown", - "id": "b3f74621", + "id": "52", "metadata": {}, "source": [ "The stacking is performed automatically using the **position adjustment** specified by the `position` argument.\n", @@ -759,7 +759,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a8e9c378", + "id": "53", "metadata": {}, "outputs": [], "source": [ @@ -768,7 +768,7 @@ }, { "cell_type": "markdown", - "id": "2aeffbf7", + "id": "54", "metadata": {}, "source": [ "The identity position adjustment is more useful for 2d geoms, like points, where it is the default.\n", @@ -780,7 +780,7 @@ { "cell_type": "code", "execution_count": null, - "id": "14205000", + "id": "55", "metadata": {}, "outputs": [], "source": [ @@ -789,7 +789,7 @@ }, { "cell_type": "markdown", - "id": "3a8e72bd", + "id": "56", "metadata": {}, "source": [ "- `position = \"dodge\"` places overlapping objects directly *beside* one another.\n", @@ -799,7 +799,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c33c4a03", + "id": "57", "metadata": {}, "outputs": [], "source": [ @@ -808,7 +808,7 @@ }, { "cell_type": "markdown", - "id": "cd46f22b", + "id": "58", "metadata": {}, "source": [ "There's one other type of adjustment that's not useful for bar charts, but can be very useful for scatterplots.\n", @@ -819,7 +819,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ba4161de", + "id": "59", "metadata": {}, "outputs": [], "source": [ @@ -828,7 +828,7 @@ }, { "cell_type": "markdown", - "id": "28a3bfc7", + "id": "60", "metadata": {}, "source": [ "The underlying values of `hwy` and `displ` are rounded so the points appear on a grid and many points overlap each other.\n", @@ -844,7 +844,7 @@ { "cell_type": "code", "execution_count": null, - "id": "414ce7af", + "id": "61", "metadata": {}, "outputs": [], "source": [ @@ -853,7 +853,7 @@ }, { "cell_type": "markdown", - "id": "96277926", + "id": "62", "metadata": {}, "source": [ "Adding randomness seems like a strange way to improve your plot, but while it makes your graph less accurate at small scales, it makes your graph *more* revealing at large scales.\n", @@ -867,7 +867,7 @@ }, { "cell_type": "markdown", - "id": "90d09736", + "id": "63", "metadata": {}, "source": [ "### Exercises\n", @@ -879,7 +879,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9bc38aef", + "id": "64", "metadata": {}, "outputs": [], "source": [ @@ -888,7 +888,7 @@ }, { "cell_type": "markdown", - "id": "57b6523f", + "id": "65", "metadata": {}, "source": [ "2. What, if anything, is the difference between the two plots?\n", @@ -908,7 +908,7 @@ }, { "cell_type": "markdown", - "id": "a3138f75", + "id": "66", "metadata": {}, "source": [ "3. What parameters to `geom_jitter()` control the amount of jittering?\n", @@ -919,7 +919,7 @@ }, { "cell_type": "markdown", - "id": "7d302c31", + "id": "67", "metadata": {}, "source": [ "## The layered grammar of graphics\n", @@ -954,7 +954,7 @@ }, { "cell_type": "markdown", - "id": "c0f7a7ff", + "id": "68", "metadata": {}, "source": [ "At this point, you would have a complete graph, but you could further adjust the positions of the geoms within the coordinate system (a position adjustment) or split the graph into subplots (faceting).\n", @@ -971,9 +971,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -981,7 +978,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 8a67182ecf0a0a64e494012c55c7abfc7b1c52a7 Mon Sep 17 00:00:00 2001 From: Your Name Date: Mon, 13 Jul 2026 02:59:23 +0100 Subject: [PATCH 02/30] refactor: migrate exploratory data analysis examples to polars --- exploratory-data-analysis.ipynb | 374 ++++++++++++++++++-------------- 1 file changed, 209 insertions(+), 165 deletions(-) diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index 071a96b..eae03ec 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Exploratory Data Analysis {#sec-exploratory-data-analysis}\n", @@ -21,7 +21,7 @@ "\n", "### Prerequisites\n", "\n", - "For doing EDA, we'll use the **pandas**, **skimpy**, and **pandas-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", + "For doing EDA, we'll use the **polars**, **skimpy**, and **polars-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", "\n", "As ever, we begin by loading these packages that we'll use:" ] @@ -29,14 +29,13 @@ { "cell_type": "code", "execution_count": null, - "id": "a3377aa6", + "id": "1", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "from lets_plot import *\n", "from lets_plot.mapping import as_discrete\n", - "from pandas.api.types import CategoricalDtype\n", "from skimpy import skim\n", "\n", "LetsPlot.setup_html()" @@ -45,7 +44,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "2", "metadata": { "tags": [ "remove-cell" @@ -56,14 +55,13 @@ "import matplotlib.pyplot as plt\n", "import matplotlib_inline.backend_inline\n", "\n", - "# Plot settings\n", "plt.style.use(\"https://github.com/aeturrell/python4DS/raw/main/plot_style.txt\")\n", "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")" ] }, { "cell_type": "markdown", - "id": "e4ddb863", + "id": "3", "metadata": {}, "source": [ "## Questions\n", @@ -90,7 +88,7 @@ }, { "cell_type": "markdown", - "id": "6adeff41", + "id": "4", "metadata": {}, "source": [ "## Variation\n", @@ -103,27 +101,28 @@ { "cell_type": "code", "execution_count": null, - "id": "069caa7c", + "id": "5", "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", + "diamonds = pl.read_csv(\n", " \"https://github.com/mwaskom/seaborn-data/raw/master/diamonds.csv\"\n", ")\n", - "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n", - " CategoricalDtype(\n", - " categories=[\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"], ordered=True\n", - " )\n", - ")\n", - "diamonds[\"color\"] = diamonds[\"color\"].astype(\n", - " CategoricalDtype(categories=[\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"], ordered=True)\n", - ")\n", + "\n", + "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", + "diamonds_color_order = [\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"]\n", + "\n", + "diamonds = diamonds.with_columns([\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", + " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", + "])\n", + "\n", "diamonds.head()" ] }, { "cell_type": "markdown", - "id": "01f5979c", + "id": "6", "metadata": {}, "source": [ "Since `\"carat\"` is a numerical variable, we can use a histogram:" @@ -132,7 +131,7 @@ { "cell_type": "code", "execution_count": null, - "id": "97900f58", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -141,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "2307ba7c", + "id": "8", "metadata": {}, "source": [ "Now that you can visualize variation, what should you look for in your plots?\n", @@ -173,18 +172,18 @@ { "cell_type": "code", "execution_count": null, - "id": "20d75550", + "id": "9", "metadata": {}, "outputs": [], "source": [ - "smaller_diamonds = diamonds.query(\"carat < 3\").copy()\n", + "smaller_diamonds = diamonds.filter(pl.col(\"carat\") < 3)\n", "\n", "(ggplot(smaller_diamonds, aes(x=\"carat\")) + geom_histogram(binwidth=0.01))" ] }, { "cell_type": "markdown", - "id": "ba20a0e2", + "id": "10", "metadata": {}, "source": [ "This histogram suggests several interesting questions:\n", @@ -211,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "0626d35a", + "id": "11", "metadata": {}, "source": [ "### Unusual values\n", @@ -226,7 +225,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d9d7e995", + "id": "12", "metadata": {}, "outputs": [], "source": [ @@ -235,7 +234,7 @@ }, { "cell_type": "markdown", - "id": "05bcf733", + "id": "13", "metadata": {}, "source": [ "There are so many observations in the common bins that the rare bins are very short, making it very difficult to see them (although maybe if you stare intently at 0 you'll spot something).\n", @@ -245,7 +244,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ea8f8bf3", + "id": "14", "metadata": {}, "outputs": [], "source": [ @@ -258,7 +257,7 @@ }, { "cell_type": "markdown", - "id": "ba2b2c79", + "id": "15", "metadata": {}, "source": [ "`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis.\n", @@ -270,17 +269,17 @@ { "cell_type": "code", "execution_count": null, - "id": "e81ffb55", + "id": "16", "metadata": {}, "outputs": [], "source": [ - "unusual = diamonds.query(\"y < 3 or y > 20\").loc[:, [\"x\", \"y\", \"z\", \"price\"]]\n", + "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select([\"x\", \"y\", \"z\", \"price\"])\n", "unusual" ] }, { "cell_type": "markdown", - "id": "c9321f36", + "id": "17", "metadata": {}, "source": [ "The `\"y\"` variable measures one of the three dimensions of these diamonds, in mm.\n", @@ -297,7 +296,7 @@ }, { "cell_type": "markdown", - "id": "142d21d7", + "id": "18", "metadata": {}, "source": [ "### Exercises\n", @@ -321,7 +320,7 @@ }, { "cell_type": "markdown", - "id": "c51e8d6f", + "id": "19", "metadata": {}, "source": [ "## Unusual Values\n", @@ -345,18 +344,16 @@ { "cell_type": "code", "execution_count": null, - "id": "ecf345a7", + "id": "20", "metadata": {}, "outputs": [], "source": [ - "diamonds2 = diamonds.copy()\n", - "condition = (diamonds2[\"y\"] < 3) | (diamonds2[\"y\"] > 20)\n", - "diamonds2.loc[condition, \"y\"] = pd.NA" + "diamonds2 = diamonds.filter(~((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)))" ] }, { "cell_type": "markdown", - "id": "d26d922e", + "id": "21", "metadata": {}, "source": [ "It's not obvious where you should plot missing values, so **lets-plot** doesn't include them in the plot:" @@ -365,7 +362,7 @@ { "cell_type": "code", "execution_count": null, - "id": "15a43255", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -374,7 +371,7 @@ }, { "cell_type": "markdown", - "id": "e8ae854b", + "id": "23", "metadata": {}, "source": [ "Other times you want to understand what makes observations with missing values different to observations with recorded values.\n", @@ -386,28 +383,34 @@ { "cell_type": "code", "execution_count": null, - "id": "0a4ea922", + "id": "24", "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", + "import polars as pl\n", + "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights = pd.read_csv(url)\n", + "flights_pd = pd.read_csv(url)\n", + "flights = pl.from_pandas(flights_pd)\n", "flights.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "6849f4d9", + "id": "25", "metadata": {}, "outputs": [], "source": [ - "flights2 = flights.assign(\n", - " cancelled=lambda x: pd.isna(x[\"dep_time\"]),\n", - " sched_hour=lambda x: x[\"sched_dep_time\"] // 100,\n", - " sched_min=lambda x: x[\"sched_dep_time\"] % 100,\n", - " sched_dep_time=lambda x: x[\"sched_hour\"] + x[\"sched_min\"] / 60,\n", - ")\n", + "flights2 = flights.with_columns([\n", + " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", + " ((pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60).alias(\"sched_dep_time\"),\n", + "])\n", + "\n", + "from lets_plot import *\n", "\n", "(\n", " ggplot(flights2, aes(x=\"sched_dep_time\"))\n", @@ -417,7 +420,7 @@ }, { "cell_type": "markdown", - "id": "b97e453a", + "id": "26", "metadata": {}, "source": [ "However this plot isn't great because there are many more non-cancelled flights than cancelled flights.\n", @@ -436,7 +439,7 @@ }, { "cell_type": "markdown", - "id": "f63d0b9f", + "id": "27", "metadata": {}, "source": [ "## Covariation\n", @@ -453,7 +456,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e1719d8f", + "id": "28", "metadata": {}, "outputs": [], "source": [ @@ -465,7 +468,7 @@ }, { "cell_type": "markdown", - "id": "20387b96", + "id": "29", "metadata": {}, "source": [ "The default appearance of `geom_freqpoly()` is not that useful here because the height, determined by the overall count, differs so much across cuts, making it hard to see the differences in the shapes of their distributions.\n", @@ -477,7 +480,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9388e24b", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -489,7 +492,7 @@ }, { "cell_type": "markdown", - "id": "157f63dd", + "id": "31", "metadata": {}, "source": [ "There’s something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that’s because density plots are a little hard to interpret - there’s a lot going on in this plot.\n", @@ -500,7 +503,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a3f333a6", + "id": "32", "metadata": {}, "outputs": [], "source": [ @@ -509,7 +512,7 @@ }, { "cell_type": "markdown", - "id": "b9d9ef00", + "id": "33", "metadata": {}, "source": [ "We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counter-intuitive finding that better quality diamonds are typically cheaper! In the exercises, you’ll be challenged to figure out why.\n", @@ -522,21 +525,22 @@ { "cell_type": "code", "execution_count": null, - "id": "6949db81", + "id": "34", "metadata": {}, "outputs": [], "source": [ - "mpg = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n", + "mpg = pl.read_csv(\n", + " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\"\n", ")\n", - "mpg[\"class\"] = mpg[\"class\"].astype(\"category\")\n", + "\n", + "mpg = mpg.with_columns(pl.col(\"class\").cast(pl.Categorical))\n", "\n", "(ggplot(mpg, aes(x=\"class\", y=\"hwy\")) + geom_boxplot())" ] }, { "cell_type": "markdown", - "id": "871aaf1c", + "id": "35", "metadata": {}, "source": [ "To make the trend easier to see, we can reorder class based on the median value of `\"hwy\"`:" @@ -545,7 +549,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a5b1ed09", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -554,7 +558,7 @@ }, { "cell_type": "markdown", - "id": "dde59236", + "id": "37", "metadata": {}, "source": [ "If you have long variable names, geom_boxplot() will work better if you flip it 90°. You can do that by adding `coord_flip()`." @@ -563,7 +567,7 @@ { "cell_type": "code", "execution_count": null, - "id": "920a4268", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -576,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "299635df", + "id": "39", "metadata": {}, "source": [ "#### Exercises\n", @@ -598,7 +602,7 @@ }, { "cell_type": "markdown", - "id": "6e6a0b46", + "id": "40", "metadata": {}, "source": [ "### Two categorical variables\n", @@ -609,20 +613,22 @@ { "cell_type": "code", "execution_count": null, - "id": "68d330d2", + "id": "41", "metadata": {}, "outputs": [], "source": [ - "ct_cut_color = pd.melt(\n", - " pd.crosstab(diamonds[\"cut\"], diamonds[\"color\"]).reset_index(),\n", - " id_vars=[\"cut\"],\n", - " value_vars=diamonds[\"color\"].unique(),\n", - ")" + "ct_cut_color = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", + ")\n", + "\n", + "ct_cut_color_long = ct_cut_color.melt(id_vars=\"cut\", variable_name=\"color\", value_name=\"value\")" ] }, { "cell_type": "markdown", - "id": "c2ebc0c6", + "id": "42", "metadata": {}, "source": [ "Followed by visualising it with `geom_tile()`:" @@ -631,16 +637,16 @@ { "cell_type": "code", "execution_count": null, - "id": "e858cd22", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "(ggplot(ct_cut_color, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" + "(ggplot(ct_cut_color_long, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" ] }, { "cell_type": "markdown", - "id": "8ccd5c1b", + "id": "44", "metadata": {}, "source": [ "### Exercises\n", @@ -654,7 +660,7 @@ }, { "cell_type": "markdown", - "id": "b2f19afc", + "id": "45", "metadata": {}, "source": [ "### Two numerical variables\n", @@ -668,7 +674,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2afe2535", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -677,7 +683,7 @@ }, { "cell_type": "markdown", - "id": "9d8034f2", + "id": "47", "metadata": {}, "source": [ "(In this section we'll use the `smaller_diamonds` dataset to stay focused on the bulk of the diamonds that are smaller than 3 carats)\n", @@ -689,7 +695,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b55707a9", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -698,7 +704,7 @@ }, { "cell_type": "markdown", - "id": "351c22bd", + "id": "49", "metadata": {}, "source": [ "But using transparency can be challenging for very large datasets. In that case, we recommend a *binscatter*, or binned scatterplot. A binned scatterplot divides the conditioning variable, `\"carat\"` in our example, into equally sized bins or quantiles, and then plots the conditional mean of the dependent variable, `\"price\"` in our example, within each bin. Bin scatters often come with confidence intervals too. A good bin scatter package in Python is [**binsreg**](https://nppackages.github.io/binsreg/). However, bin scatters are an advanced topic, and we won't cover them here." @@ -706,7 +712,7 @@ }, { "cell_type": "markdown", - "id": "14158b48", + "id": "50", "metadata": {}, "source": [ "## **pandas** built-in tools for EDA\n", @@ -721,7 +727,7 @@ { "cell_type": "code", "execution_count": null, - "id": "13079065", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -730,7 +736,7 @@ }, { "cell_type": "markdown", - "id": "db57dbb7", + "id": "52", "metadata": {}, "source": [ "Although helpful, that sure is hard to read! We can improve this by using the `round()` method too:\n" @@ -739,17 +745,23 @@ { "cell_type": "code", "execution_count": null, - "id": "b4144440", + "id": "53", "metadata": {}, "outputs": [], "source": [ - "sum_table = diamonds.describe().round(1)\n", + "sum_table = (\n", + " diamonds.describe()\n", + " .with_columns(\n", + " pl.selectors.numeric().round(1)\n", + " )\n", + ")\n", + "\n", "sum_table" ] }, { "cell_type": "markdown", - "id": "8063b8f5", + "id": "54", "metadata": {}, "source": [ "Published summary statistics tables often list one variable per row, and if your data frame has many variables, `describe()` can quickly get too wide to read easily. You can transpose it using the `T` property (or the `transpose()` method):" @@ -758,17 +770,17 @@ { "cell_type": "code", "execution_count": null, - "id": "cd2f8772", + "id": "55", "metadata": {}, "outputs": [], "source": [ - "sum_table = sum_table.T\n", - "sum_table" + "sum_table_long = sum_table.melt(id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\")\n", + "sum_table_long" ] }, { "cell_type": "markdown", - "id": "e67104d3", + "id": "56", "metadata": {}, "source": [ "Of course, the stats provided in this pre-built table are not very customised. So what do we do to get the table that we actually want? Well, the answer is to draw on the contents of the previous data chapters, particularly the introduction to data analysis. Groupbys, merges, aggregations: use all of them to produce the EDA table that you want.\n", @@ -781,24 +793,25 @@ { "cell_type": "code", "execution_count": null, - "id": "5afcacbc", + "id": "57", "metadata": {}, "outputs": [], "source": [ - "(\n", - " diamonds.groupby([\"cut\", \"color\"])[\"price\"]\n", - " .mean()\n", - " .unstack()\n", - " .apply(lambda x: x / 1e3)\n", - " .fillna(\"-\")\n", - " .style.format(precision=2)\n", - " .set_caption(\"Sale price (thousands)\")\n", - ")" + "import polars.selectors as cs\n", + "\n", + "price_summary = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.col(\"price\").mean())\n", + " .pivot(index=\"cut\", on=\"color\", values=\"price\")\n", + " .with_columns(cs.numeric().round(2))\n", + ")\n", + "\n", + "price_summary" ] }, { "cell_type": "markdown", - "id": "d16e0fc1", + "id": "58", "metadata": {}, "source": [ "Although a neater one than we've seen, this is still a drab table of numbers. The eye is not immediately drawn to it!\n", @@ -811,16 +824,28 @@ { "cell_type": "code", "execution_count": null, - "id": "21e65189", + "id": "59", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.background_gradient(cmap=\"plasma\")" + "ct_cut_color = (\n", + " diamonds.group_by([\"color\", \"cut\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(\n", + " index=\"color\",\n", + " on=\"cut\",\n", + " values=\"count\",\n", + " aggregate_function=\"sum\",\n", + " )\n", + " .with_columns(\n", + " cs.numeric().fill_null(0)\n", + " )\n", + ")" ] }, { "cell_type": "markdown", - "id": "b7c6aa71", + "id": "60", "metadata": {}, "source": [ "By default, `background_gradient()` highlights each number relative to the others in its column; you can highlight by row using `axis=1` or relative to all table values using `axis=0`. And of course `plasma` is just one of [many available colormaps](https://matplotlib.org/stable/tutorials/colors/colormaps.html)!\n", @@ -837,38 +862,41 @@ { "cell_type": "code", "execution_count": null, - "id": "bb0162ba", + "id": "61", "metadata": {}, "outputs": [], "source": [ "(\n", - " pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"])\n", - " .style.format(precision=0)\n", + " ct_cut_color\n", + " .to_pandas()\n", + " .style\n", + " .format(precision=0)\n", " .bar(color=\"#d65f5f\")\n", ")" ] }, { "cell_type": "markdown", - "id": "e36d4795", + "id": "62", "metadata": {}, "source": [ - "Use `.hightlight_max()`, and similar commands, to show important entries:" + "Use `max_counts()`, and similar commands, to show important entries:" ] }, { "cell_type": "code", "execution_count": null, - "id": "5d19072c", + "id": "63", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.highlight_max().format(\"{:.0f}\")" + "max_counts = ct_cut_color.select(pl.exclude(\"color\").max())\n", + "max_counts" ] }, { "cell_type": "markdown", - "id": "63075da4", + "id": "64", "metadata": {}, "source": [ "You can find a full set of styling commands [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Styling)." @@ -876,10 +904,10 @@ }, { "cell_type": "markdown", - "id": "d8b6d225", + "id": "65", "metadata": {}, "source": [ - "### Exploratory Plotting with **pandas**\n", + "### Exploratory Plotting with **polars**\n", "\n", "**pandas** has some built-in plotting options to help you look at data quickly. These can be accessed via `.plot.*` or `.plot()`, depending on the context. Let's make a quick `.plot()` using a dataset on taxis." ] @@ -887,48 +915,46 @@ { "cell_type": "code", "execution_count": null, - "id": "b479d5b1", + "id": "66", "metadata": {}, "outputs": [], "source": [ - "taxis = pd.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", - "# turn the pickup time column into a datetime\n", - "taxis[\"pickup\"] = pd.to_datetime(taxis[\"pickup\"])\n", - "# set some other columns types\n", - "taxis = taxis.astype(\n", - " {\n", - " \"dropoff\": \"datetime64[ns]\",\n", - " \"pickup\": \"datetime64[ns]\",\n", - " \"color\": \"category\",\n", - " \"payment\": \"category\",\n", - " \"pickup_zone\": \"string\",\n", - " \"dropoff_zone\": \"string\",\n", - " \"pickup_borough\": \"category\",\n", - " \"dropoff_borough\": \"category\",\n", - " }\n", - ")\n", + "taxis = pl.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", + "\n", + "taxis = taxis.with_columns(pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\"))\n", + "\n", + "taxis = taxis.with_columns([\n", + " pl.col(\"color\").cast(pl.Categorical),\n", + " pl.col(\"payment\").cast(pl.Categorical),\n", + " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", + " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", + "])\n", + "\n", "taxis.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "ee971c9c", + "id": "67", "metadata": {}, "outputs": [], "source": [ - "taxis.info()" + "taxis.describe()" ] }, { "cell_type": "code", "execution_count": null, - "id": "2015b1dc", + "id": "68", "metadata": {}, "outputs": [], "source": [ + "taxis_pd = taxis.to_pandas()\n", + "taxis_pd[\"pickup\"] = pd.to_datetime(taxis_pd[\"pickup\"])\n", + "\n", "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[\"total\"]\n", " .mean()\n", " .plot(\n", @@ -936,12 +962,13 @@ " xlabel=\"\",\n", " ylabel=\"Fare (USD)\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "b9c0f990", + "id": "69", "metadata": {}, "source": [ "Again, if you can get the data in the right shape, you can plot it. The same function works with multiple lines\n" @@ -950,12 +977,12 @@ { "cell_type": "code", "execution_count": null, - "id": "51e86185", + "id": "70", "metadata": {}, "outputs": [], "source": [ "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[[\"fare\", \"tip\", \"tolls\"]]\n", " .mean()\n", " .plot(\n", @@ -964,12 +991,13 @@ " xlabel=\"\",\n", " ylabel=\"USD\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "cc227b81", + "id": "71", "metadata": {}, "source": [ "Now let's see some of the other quick `.plot.*` options." @@ -977,7 +1005,7 @@ }, { "cell_type": "markdown", - "id": "1c4278ac", + "id": "72", "metadata": {}, "source": [ "A bar chart (use `barh` for horizontal orientation; `rot` sets rotation of labels):" @@ -986,16 +1014,32 @@ { "cell_type": "code", "execution_count": null, - "id": "79ceca92", + "id": "73", "metadata": {}, "outputs": [], "source": [ - "taxis.value_counts(\"payment\").sort_index().plot.bar(title=\"Counts\", rot=0);" + "payment_counts = (\n", + " taxis\n", + " .filter(pl.col(\"payment\").is_not_null())\n", + " .group_by(\"payment\")\n", + " .agg(pl.len().alias(\"count\"))\n", + " .sort(\"payment\")\n", + ")\n", + "\n", + "payment_counts.to_pandas().plot.bar(\n", + " x=\"payment\",\n", + " y=\"count\",\n", + " legend=False,\n", + " rot=0,\n", + " title=\"Counts\",\n", + ")\n", + "\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "4e6f1cc2", + "id": "74", "metadata": {}, "source": [ "This next one, uses `.plot.hist()` to create a histogram." @@ -1004,16 +1048,17 @@ { "cell_type": "code", "execution_count": null, - "id": "5efc5817", + "id": "75", "metadata": {}, "outputs": [], "source": [ - "taxis[\"tip\"].plot.hist(bins=30, title=\"Tip\");" + "(taxis_pd[\"tip\"].plot.hist(bins=30, title=\"Tip\"))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "3f15bfa7", + "id": "76", "metadata": {}, "source": [ "Boxplot:" @@ -1022,16 +1067,17 @@ { "cell_type": "code", "execution_count": null, - "id": "0b735d15", + "id": "77", "metadata": {}, "outputs": [], "source": [ - "(taxis[[\"fare\", \"tolls\", \"tip\"]].plot.box());" + "(taxis_pd[[\"fare\", \"tolls\", \"tip\"]].plot.box())\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "67017bef", + "id": "78", "metadata": {}, "source": [ "Scatter plot:" @@ -1040,16 +1086,17 @@ { "cell_type": "code", "execution_count": null, - "id": "66adada2", + "id": "79", "metadata": {}, "outputs": [], "source": [ - "taxis.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None));" + "(taxis_pd.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None)))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "9766d2c4", + "id": "80", "metadata": {}, "source": [ "## Other tools for EDA\n", @@ -1059,7 +1106,7 @@ }, { "cell_type": "markdown", - "id": "813ec70a", + "id": "81", "metadata": {}, "source": [ "### **skimpy** for summary statistics\n", @@ -1072,7 +1119,7 @@ { "cell_type": "code", "execution_count": null, - "id": "32796b5f", + "id": "82", "metadata": {}, "outputs": [], "source": [ @@ -1081,7 +1128,7 @@ }, { "cell_type": "markdown", - "id": "f2810c9e", + "id": "83", "metadata": {}, "source": [ "## Summary\n", @@ -1095,9 +1142,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -1105,7 +1149,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 3ec138cd42332f0aba34ea854279aa1b987b333d Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 14 Jul 2026 22:24:10 +0100 Subject: [PATCH 03/30] style: fix pre-commit issues in exploratory data analysis notebook --- exploratory-data-analysis.ipynb | 85 +++++++++++++++++---------------- 1 file changed, 43 insertions(+), 42 deletions(-) diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index eae03ec..5841632 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -112,10 +112,12 @@ "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", "diamonds_color_order = [\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"]\n", "\n", - "diamonds = diamonds.with_columns([\n", - " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", - " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", - "])\n", + "diamonds = diamonds.with_columns(\n", + " [\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", + " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", + " ]\n", + ")\n", "\n", "diamonds.head()" ] @@ -273,7 +275,9 @@ "metadata": {}, "outputs": [], "source": [ - "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select([\"x\", \"y\", \"z\", \"price\"])\n", + "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select(\n", + " [\"x\", \"y\", \"z\", \"price\"]\n", + ")\n", "unusual" ] }, @@ -403,15 +407,20 @@ "metadata": {}, "outputs": [], "source": [ - "flights2 = flights.with_columns([\n", - " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", - " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", - " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", - " ((pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60).alias(\"sched_dep_time\"),\n", - "])\n", - "\n", "from lets_plot import *\n", "\n", + "flights2 = flights.with_columns(\n", + " [\n", + " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", + " (\n", + " (pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60\n", + " ).alias(\"sched_dep_time\"),\n", + " ]\n", + ")\n", + "\n", + "\n", "(\n", " ggplot(flights2, aes(x=\"sched_dep_time\"))\n", " + geom_freqpoly(aes(color=\"cancelled\"), binwidth=1 / 4)\n", @@ -529,9 +538,7 @@ "metadata": {}, "outputs": [], "source": [ - "mpg = pl.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\"\n", - ")\n", + "mpg = pl.read_csv(\"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\")\n", "\n", "mpg = mpg.with_columns(pl.col(\"class\").cast(pl.Categorical))\n", "\n", @@ -623,7 +630,9 @@ " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", ")\n", "\n", - "ct_cut_color_long = ct_cut_color.melt(id_vars=\"cut\", variable_name=\"color\", value_name=\"value\")" + "ct_cut_color_long = ct_cut_color.melt(\n", + " id_vars=\"cut\", variable_name=\"color\", value_name=\"value\"\n", + ")" ] }, { @@ -749,12 +758,7 @@ "metadata": {}, "outputs": [], "source": [ - "sum_table = (\n", - " diamonds.describe()\n", - " .with_columns(\n", - " pl.selectors.numeric().round(1)\n", - " )\n", - ")\n", + "sum_table = diamonds.describe().with_columns(pl.selectors.numeric().round(1))\n", "\n", "sum_table" ] @@ -774,7 +778,9 @@ "metadata": {}, "outputs": [], "source": [ - "sum_table_long = sum_table.melt(id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\")\n", + "sum_table_long = sum_table.melt(\n", + " id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\"\n", + ")\n", "sum_table_long" ] }, @@ -837,9 +843,7 @@ " values=\"count\",\n", " aggregate_function=\"sum\",\n", " )\n", - " .with_columns(\n", - " cs.numeric().fill_null(0)\n", - " )\n", + " .with_columns(cs.numeric().fill_null(0))\n", ")" ] }, @@ -866,13 +870,7 @@ "metadata": {}, "outputs": [], "source": [ - "(\n", - " ct_cut_color\n", - " .to_pandas()\n", - " .style\n", - " .format(precision=0)\n", - " .bar(color=\"#d65f5f\")\n", - ")" + "(ct_cut_color.to_pandas().style.format(precision=0).bar(color=\"#d65f5f\"))" ] }, { @@ -921,14 +919,18 @@ "source": [ "taxis = pl.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", "\n", - "taxis = taxis.with_columns(pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\"))\n", + "taxis = taxis.with_columns(\n", + " pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\")\n", + ")\n", "\n", - "taxis = taxis.with_columns([\n", - " pl.col(\"color\").cast(pl.Categorical),\n", - " pl.col(\"payment\").cast(pl.Categorical),\n", - " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", - " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", - "])\n", + "taxis = taxis.with_columns(\n", + " [\n", + " pl.col(\"color\").cast(pl.Categorical),\n", + " pl.col(\"payment\").cast(pl.Categorical),\n", + " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", + " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", + " ]\n", + ")\n", "\n", "taxis.head()" ] @@ -1019,8 +1021,7 @@ "outputs": [], "source": [ "payment_counts = (\n", - " taxis\n", - " .filter(pl.col(\"payment\").is_not_null())\n", + " taxis.filter(pl.col(\"payment\").is_not_null())\n", " .group_by(\"payment\")\n", " .agg(pl.len().alias(\"count\"))\n", " .sort(\"payment\")\n", From a1e1d0dbb66aef63bb21ee9061cf2960289098d4 Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 14 Jul 2026 22:35:02 +0100 Subject: [PATCH 04/30] style: run pre-commit on vis-layers notebook to format and clean code --- vis-layers.ipynb | 38 ++++++++++++++------------------------ 1 file changed, 14 insertions(+), 24 deletions(-) diff --git a/vis-layers.ipynb b/vis-layers.ipynb index 1807a29..cf6a92d 100644 --- a/vis-layers.ipynb +++ b/vis-layers.ipynb @@ -47,7 +47,6 @@ "metadata": {}, "outputs": [], "source": [ - "\n", "import polars as pl\n", "from lets_plot import *\n", "\n", @@ -73,19 +72,19 @@ "metadata": {}, "outputs": [], "source": [ - "mpg = pl.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\"\n", + "mpg = pl.read_csv(\"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\")\n", + "\n", + "mpg = mpg.with_columns(\n", + " [\n", + " pl.col(\"manufacturer\").cast(pl.Categorical),\n", + " pl.col(\"model\").cast(pl.Categorical),\n", + " pl.col(\"trans\").cast(pl.Categorical),\n", + " pl.col(\"drv\").cast(pl.Categorical),\n", + " pl.col(\"fl\").cast(pl.Categorical),\n", + " pl.col(\"class\").cast(pl.Categorical),\n", + " ]\n", ")\n", "\n", - "mpg = mpg.with_columns([\n", - " pl.col(\"manufacturer\").cast(pl.Categorical),\n", - " pl.col(\"model\").cast(pl.Categorical),\n", - " pl.col(\"trans\").cast(pl.Categorical),\n", - " pl.col(\"drv\").cast(pl.Categorical),\n", - " pl.col(\"fl\").cast(pl.Categorical),\n", - " pl.col(\"class\").cast(pl.Categorical),\n", - "])\n", - "\n", "mpg.head()" ] }, @@ -619,9 +618,7 @@ " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv\"\n", ")\n", "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", - "diamonds = diamonds.with_columns(\n", - " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order))\n", - ")\n", + "diamonds = diamonds.with_columns(pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)))\n", "\n", "diamonds.head()" ] @@ -681,16 +678,9 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds_counts = (\n", - " diamonds.group_by(\"cut\")\n", - " .agg(pl.len().alias(\"counts\"))\n", - " .sort(\"cut\") \n", - ")\n", + "diamonds_counts = diamonds.group_by(\"cut\").agg(pl.len().alias(\"counts\")).sort(\"cut\")\n", "\n", - "(\n", - " ggplot(diamonds_counts, aes(x=\"cut\", y=\"counts\"))\n", - " + geom_bar(stat=\"identity\")\n", - ")" + "(ggplot(diamonds_counts, aes(x=\"cut\", y=\"counts\")) + geom_bar(stat=\"identity\"))" ] }, { From b31661a639043177418cc53c022f84e5d3712ac7 Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 15 Jul 2026 22:25:11 +0100 Subject: [PATCH 05/30] Convert categorical operations from pandas to Polars --- categorical-data.ipynb | 247 ++++++++++++------ workflow-style.ipynb | 558 ++++++++++++++++++++--------------------- 2 files changed, 444 insertions(+), 361 deletions(-) diff --git a/categorical-data.ipynb b/categorical-data.ipynb index ab6ba30..c391411 100644 --- a/categorical-data.ipynb +++ b/categorical-data.ipynb @@ -2,25 +2,21 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "cell_000", "metadata": {}, "source": [ "# Categorical Data {#sec-categorical-data}\n", "\n", "## Introduction\n", "\n", - "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **pandas** [documentation](https://pandas.pydata.org/).\n" + "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **polars** [documentation](https://docs.pola.rs/api/python/stable/reference/index.html).\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "51a55374", - "metadata": { - "tags": [ - "remove-cell" - ] - }, + "id": "cell_001", + "metadata": {}, "outputs": [], "source": [ "# remove cell\n", @@ -34,248 +30,265 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "cell_002", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package.\n" ] }, { "cell_type": "markdown", - "id": "8757f369", + "id": "cell_003", "metadata": {}, "source": [ "## The Category Datatype\n", "\n", - "Everything in Python has a type, even the data in **pandas** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical`. There are some benefits to using categorical variables (where appropriate):\n", + "Everything in Python has a type, even the data in **polars** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical` and `Enum`. There are some benefits to using categorical variables (where appropriate):\n", "\n", "- they can keep track even when elements of the category isn't present, which can sometimes be as interesting as when they are (imagine you find no-one from a particular school goes to university)\n", "- they can use vastly less of your computer's memory than encoding the same information in other ways\n", "- they can be used efficiently with modelling packages, where they will be recognised as potential 'dummy variables', or with plotting packages, which will treat them as discrete values\n", - "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\")\n", + "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\") using pl.Enum\n", + "\n", + "All values of categorical data for a **polars** column are either in the given categories or take the value `None`.\n", + "\n", + "Polars has two categorical types: \n", "\n", - "All values of categorical data for a **pandas** column are either in the given categories or take the value `np.nan`. \n", + "- pl.Categorical: Unordered categories (like pandas category)\n", + "\n", + "- pl.Enum: Ordered categories with a fixed set of values (similar to pandas ordered categorical)\n", "\n", "## Creating Categorical Data\n", "\n", - "Let's create a categorical column of data:" + "Let's create a categorical column of data:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "535ef959", + "id": "cell_004", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", + "df = pl.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", "\n", - "df[\"A\"] = df[\"A\"].astype(\"category\")\n", + "df = df.with_columns(pl.col(\"A\").cast(pl.Categorical))\n", "df[\"A\"]" ] }, { "cell_type": "markdown", - "id": "62db4dec", + "id": "cell_005", "metadata": {}, "source": [ - "Notice that we get some additional information at the bottom of the shown series: we get told that not only is this a categorical column type, but it has three values 'a', 'b', and 'c'.\n", + "Notice that polars tracks the categorical type, though the display is more compact than pandas.\n", "\n", - "You can also use special functions, such as `pd.cut()`, to groups data into discrete bins. Here's an example where specify the labels for the categories directly:\n" + "You can also use special expression methods, such as pl.col().cut(), to group data into discrete bins. Here's an example where we specify the labels for each category directly.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "358c83bb", + "id": "cell_006", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", + "df = pl.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", "labels = [f\"{i} - {i+9}\" for i in range(0, 100, 10)]\n", - "df[\"group\"] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)\n", + "df = df.with_columns(\n", + " pl.col(\"value\")\n", + " .cut(breaks=list(range(10, 100, 10)), labels=labels, left_closed=True)\n", + " .alias(\"group\")\n", + ")\n", "df.head()" ] }, { "cell_type": "markdown", - "id": "87bc56b6", + "id": "cell_007", "metadata": {}, "source": [ "In the example above, the `group` column is of categorical type.\n", "\n", - "Another way to create a categorical variable is directly using the `pd.Categorical()` function:" + "Another way to create a categorical variable is directly using the `pl.Categorical` or `pl.Enum` types:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "fb389105", + "id": "cell_008", "metadata": {}, "outputs": [], "source": [ - "raw_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"], categories=[\"b\", \"c\", \"d\"]\n", + "raw_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"b\", \"c\", \"d\"]), strict=False\n", ")\n", "raw_cat" ] }, { "cell_type": "markdown", - "id": "ebfdf815", + "id": "cell_009", "metadata": {}, "source": [ - "We can then enter this into a data frame:" + "We can then enter this into a data frame:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "0497fc16", + "id": "cell_010", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(raw_cat, columns=[\"cat_type\"])\n", + "df = pl.DataFrame({\"cat_type\": raw_cat})\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "76dc9dd6", + "id": "cell_011", "metadata": {}, "source": [ - "Note that NaNs appear for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values." + "Note that `None` appears for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values.\n" ] }, { "cell_type": "markdown", - "id": "aa76a795", + "id": "cell_012", "metadata": {}, "source": [ - "You can also create ordered categories:" + "You can also create ordered categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f7520d3d", + "id": "cell_013", "metadata": {}, "outputs": [], "source": [ - "ordered_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"],\n", - " categories=[\"a\", \"b\", \"c\", \"d\"],\n", - " ordered=True,\n", + "ordered_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"a\", \"b\", \"c\", \"d\"])\n", ")\n", "ordered_cat" ] }, { "cell_type": "markdown", - "id": "258cc6f5", + "id": "cell_014", "metadata": {}, "source": [ "## Working with Categories\n", "\n", - "Categorical data has a `categories` and a `ordered` property; these list the possible values and whether the ordering matters or not respectively. These properties are exposed as `.cat.categories` and `.cat.ordered`. If you don’t manually specify categories and ordering, they are inferred from the passed arguments.\n", + "Categorical data in polars has categories and ordering properties. For `pl.Enum`, the categories are fixed and ordered. For `pl.Categorical`, categories are not ordered and can be more flexible.\n", "\n", - "Let's see some examples:" + "Let's see some examples:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "2caba354", + "id": "cell_015", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.categories" + "df[\"cat_type\"].dtype.categories" ] }, { "cell_type": "code", "execution_count": null, - "id": "5f1fe093", + "id": "cell_016", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.ordered" + "isinstance(df[\"cat_type\"].dtype, pl.Enum)" ] }, { "cell_type": "markdown", - "id": "694c07b0", + "id": "cell_017", "metadata": {}, "source": [ - "If categorical data is ordered (ie `.cat.ordered == True`), then the order of the categories has a meaning and certain operations are possible: you can sort values (with `.sort_values`), and apply `.min` and `.max`." + "If categorical data is ordered (ie `pl.Enum`), then the order of the categories has a meaning and certain operations are possible: you can sort values, and apply `.min()` and `.max()`.\n" ] }, { "cell_type": "markdown", - "id": "0a6d7a90", + "id": "cell_018", "metadata": {}, "source": [ "### Renaming Categories\n", "\n", - "Renaming categories is done via the `rename_categories()` method (which works with a list or a dictionary)." + "Renaming categories in polars requires re-casting with new categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "097171b8", + "id": "cell_019", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.rename_categories([\"alpha\", \"beta\", \"gamma\"])" + "df = df.with_columns(\n", + " pl.col(\"cat_type\")\n", + " .cast(pl.String)\n", + " .replace({\"b\": \"alpha\", \"c\": \"beta\", \"d\": \"gamma\"})\n", + " .cast(pl.Enum([\"alpha\", \"beta\", \"gamma\"]))\n", + ")" ] }, { "cell_type": "markdown", - "id": "3de7fd23", + "id": "cell_020", "metadata": {}, "source": [ - "Quite often, you'll run into a situation where you want to add a category. You can do this with `.add_categories()`:" + "To add categories, you need to re-cast with an extended Enum:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4ae6df38", + "id": "cell_021", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.add_categories([\"delta\"])\n", + "df = df.with_columns(\n", + " pl.col(\"cat_type\").cast(pl.Enum([\"alpha\", \"beta\", \"gamma\", \"delta\"]))\n", + ")\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "ba525491", + "id": "cell_022", "metadata": {}, "source": [ - "Similarly, there is a `.remove_categories()` function and a `.remove_unused_categories()` function. `.set_categories` adds and removes categories in one fell swoop. One of the nice properties of set categories is that Remember that you need to do `df[\"columnname\"].cat` before calling any cat(egory) functions though." + "Because Enum categories are fixed in Polars, you cannot remove categories directly. Instead, ensure that the data contains only the desired values, then re-cast the column to a new Enum containing just those categories.\n" ] }, { "cell_type": "markdown", - "id": "76b38a2a", + "id": "cell_023", "metadata": {}, "source": [ "## Operations on Categories\n", "\n", - "As noted, ordered categories will already undergo some operations. But there are some that work on any set of categories. Perhaps the most useful is `value_counts()`" + "`Enum` columns behave like any other Polars column, but they also retain the complete set of allowed categories. This means Polars remembers categories even if no rows currently contain them.\n", + "\n", + "For example, suppose we define an `Enum` with five possible categories, but only use four of them:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "19f5bdda", + "id": "cell_024", "metadata": {}, "outputs": [], "source": [ @@ -284,42 +297,112 @@ }, { "cell_type": "markdown", - "id": "a36db155", + "id": "cell_025", + "metadata": {}, + "source": [ + "Notice that \"delta\" is not shown because value_counts() only reports values that actually occur in the data.\n", + "\n", + "However, the Enum still knows that \"delta\" is a valid category.\n", + "\n", + "You can inspect the complete set of categories stored in the Enum:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_026", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"cat_type\"].dtype.categories" + ] + }, + { + "cell_type": "markdown", + "id": "cell_027", + "metadata": {}, + "source": [ + "If you want a frequency table that includes unused categories, combine the stored category list with the observed counts:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_028", + "metadata": {}, + "outputs": [], + "source": [ + "all_categories = pl.DataFrame(\n", + " {\"cat_type\": df[\"cat_type\"].dtype.categories.cast(df[\"cat_type\"].dtype)}\n", + ")\n", + "all_categories.join(df[\"cat_type\"].value_counts(), on=\"cat_type\", how=\"left\").fill_null(\n", + " 0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "cell_029", "metadata": {}, "source": [ - "Note that even though 'delta' doesn't appear at all, it gets a count (of zero). This tracking of missing values can be quite handy.\n", + "This demonstrates one advantage of `Enum`: the complete set of allowed categories is retained, even when some categories are not present in the data.\n", "\n", - "`mode()` is another one:" + "You can also inspect the datatype of the column:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f52d5d0d", + "id": "cell_030", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].mode()" + "df[\"cat_type\"].dtype" ] }, { "cell_type": "markdown", - "id": "d28f34d8", + "id": "cell_031", "metadata": {}, "source": [ - "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example," + "You can compute the most frequent category:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4d43e94d", + "id": "cell_032", "metadata": {}, "outputs": [], "source": [ - "time_df = pd.DataFrame(\n", - " pd.Series(pd.date_range(\"2015/05/01\", periods=5, freq=\"M\"), dtype=\"category\"),\n", - " columns=[\"datetime\"],\n", + "df[\"cat_type\"].drop_nulls().mode()" + ] + }, + { + "cell_type": "markdown", + "id": "cell_033", + "metadata": {}, + "source": [ + "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_034", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "time_df = pl.DataFrame(\n", + " {\n", + " \"datetime\": pl.date_range(\n", + " datetime.date(2015, 5, 31), datetime.date(2015, 9, 30), \"1mo\", eager=True\n", + " )\n", + " .cast(pl.String)\n", + " .cast(pl.Categorical)\n", + " }\n", ")\n", "time_df" ] @@ -327,29 +410,29 @@ { "cell_type": "code", "execution_count": null, - "id": "db697f86", + "id": "cell_035", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].dt.month" + "time_df[\"datetime\"].cast(pl.String).str.to_date().dt.month()" ] }, { "cell_type": "markdown", - "id": "c202f0d1", + "id": "cell_036", "metadata": {}, "source": [ - "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.cat.codes` to get unique codes for each value." + "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.to_physical()` to get unique codes for each value.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "13e7bc66", + "id": "cell_037", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].cat.codes" + "time_df[\"datetime\"].to_physical()" ] } ], diff --git a/workflow-style.ipynb b/workflow-style.ipynb index 3de114b..ac02375 100644 --- a/workflow-style.ipynb +++ b/workflow-style.ipynb @@ -1,280 +1,280 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "95f0a171", - "metadata": {}, - "source": [ - "# Workflow: Style {#sec-workflow-style}\n", - "\n", - "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", - "\n", - "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", - "\n", - "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", - "\n", - "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" - ] - }, - { - "cell_type": "markdown", - "id": "c8111c37", - "metadata": {}, - "source": [ - "## Names\n", - "\n", - "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", - "\n", - "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", - "\n", - "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", - "\n", - "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", - "\n", - "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", - "\n", - "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", - "\n", - "In summary:\n", - "\n", - "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", - "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", - "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", - "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", - "\n", - " ```python\n", - " import random\n", - "\n", - " # This is bad\n", - " def roll():\n", - " return random.randint(0, 36) # magic number!\n", - "\n", - " # This is good\n", - " MAX_INT_VALUE = 36\n", - "\n", - " def roll():\n", - " return random.randint(0, MAX_INT_VALUE)\n", - " ```\n", - "\n", - "- use verbs for function names, eg `get_regression()`\n", - "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", - "- variable names should be snake_case and all lowercase, eg `first_name`\n", - "- class names should be CamelCase, eg `MyClass`\n", - "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", - "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", - "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", - "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" - ] - }, - { - "cell_type": "markdown", - "id": "b552c344", - "metadata": {}, - "source": [ - "## Whitespace\n", - "\n", - "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", - "\n", - "```python\n", - "this_is_a_var = 5\n", - "```\n", - "\n", - "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", - "\n", - "```python\n", - "list_var = [1, 2, 3, 4]\n", - "```\n", - "\n", - "rather than\n", - "\n", - "```python\n", - "list_var = [1,2,3,4]\n", - "# or\n", - "list_var = [1 , 2 , 3 , 4]\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "54048af1", - "metadata": {}, - "source": [ - "## Code Comments\n", - "\n", - "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", - "\n", - "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", - "\n", - "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", - "\n", - "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", - "\n", - "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", - "\n", - "```python\n", - "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", - " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", - " Args:\n", - " df (pl.DataFrame): Input dataframe\n", - " Returns:\n", - " pl.DataFrame: Dataframe of mean value of each numeric column.\n", - " \"\"\"\n", - " df_mean = df.select(S.numeric()).mean()\n", - " return df_mean\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "861f3102", - "metadata": {}, - "source": [ - "## Line width and line continuation\n", - "\n", - "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", - "\n", - "```python\n", - "def function(input_one, input_two,\n", - " input_three, input_four):\n", - " result = (input_one,\n", - " + input_two,\n", - " + input_three,\n", - " + input_four)\n", - " return result\n", - "```\n", - "\n", - "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f0f5bb37", - "metadata": {}, - "outputs": [], - "source": [ - "import polars as pl\n", - "\n", - "df = pl.DataFrame(\n", - " data={\n", - " \"col0\": [0, 0, 0, 0],\n", - " \"col1\": [0, 0, 0, 0],\n", - " \"col2\": [0, 0, 0, 0],\n", - " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", - " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", - " },\n", - ")\n", - "\n", - "\n", - "# Chaining inside parentheses works\n", - "\n", - "results = df.group_by([\"col3\", \"col4\"]).agg(\n", - " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", - ")\n", - "\n", - "results" - ] - }, - { - "cell_type": "markdown", - "id": "1d6f3bf8", - "metadata": {}, - "source": [ - "And this is what _not_ to do:\n", - "\n", - "```python\n", - "results = df\n", - " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", - "\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "d016d530", - "metadata": {}, - "source": [ - "## Principles of Clean Code\n", - "\n", - "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", - "\n", - "### Do not repeat yourself (DRY)\n", - "\n", - "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", - "\n", - "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", - "\n", - "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", - "\n", - "::: {.callout-tip title=\"Tip\"}\n", - "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", - ":::\n", - "\n", - "### KISS (Keep It Simple, Stupid)\n", - "\n", - "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", - "\n", - "### SoC (Separation of Concerns) / Make it Modular\n", - "\n", - "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", - "\n", - "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", - "\n", - "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", - "\n", - "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", - "\n", - "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", - "\n", - "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", - "\n", - "```python\n", - "# This is bad\n", - "def transform(text, uppercase):\n", - " if uppercase:\n", - " return text.upper()\n", - " else:\n", - " return text.lower()\n", - "\n", - "# This is good\n", - "def uppercase(text):\n", - " return text.upper()\n", - "\n", - "def lowercase(text):\n", - " return text.lower()\n", - "```\n" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, - "jupytext": { - "cell_metadata_filter": "-all", - "encoding": "# -*- coding: utf-8 -*-", - "formats": "md:myst", - "main_language": "python" - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - }, - "toc-showtags": true - }, - "nbformat": 4, - "nbformat_minor": 5 -} + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Workflow: Style {#sec-workflow-style}\n", + "\n", + "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", + "\n", + "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", + "\n", + "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", + "\n", + "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## Names\n", + "\n", + "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", + "\n", + "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", + "\n", + "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", + "\n", + "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", + "\n", + "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", + "\n", + "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", + "\n", + "In summary:\n", + "\n", + "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", + "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", + "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", + "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", + "\n", + " ```python\n", + " import random\n", + "\n", + " # This is bad\n", + " def roll():\n", + " return random.randint(0, 36) # magic number!\n", + "\n", + " # This is good\n", + " MAX_INT_VALUE = 36\n", + "\n", + " def roll():\n", + " return random.randint(0, MAX_INT_VALUE)\n", + " ```\n", + "\n", + "- use verbs for function names, eg `get_regression()`\n", + "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", + "- variable names should be snake_case and all lowercase, eg `first_name`\n", + "- class names should be CamelCase, eg `MyClass`\n", + "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", + "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", + "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", + "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" + ] + }, + { + "cell_type": "markdown", + "id": "2", + "metadata": {}, + "source": [ + "## Whitespace\n", + "\n", + "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", + "\n", + "```python\n", + "this_is_a_var = 5\n", + "```\n", + "\n", + "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", + "\n", + "```python\n", + "list_var = [1, 2, 3, 4]\n", + "```\n", + "\n", + "rather than\n", + "\n", + "```python\n", + "list_var = [1,2,3,4]\n", + "# or\n", + "list_var = [1 , 2 , 3 , 4]\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## Code Comments\n", + "\n", + "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", + "\n", + "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", + "\n", + "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", + "\n", + "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", + "\n", + "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", + "\n", + "```python\n", + "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", + " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", + " Args:\n", + " df (pl.DataFrame): Input dataframe\n", + " Returns:\n", + " pl.DataFrame: Dataframe of mean value of each numeric column.\n", + " \"\"\"\n", + " df_mean = df.select(S.numeric()).mean()\n", + " return df_mean\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "4", + "metadata": {}, + "source": [ + "## Line width and line continuation\n", + "\n", + "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", + "\n", + "```python\n", + "def function(input_one, input_two,\n", + " input_three, input_four):\n", + " result = (input_one,\n", + " + input_two,\n", + " + input_three,\n", + " + input_four)\n", + " return result\n", + "```\n", + "\n", + "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5", + "metadata": {}, + "outputs": [], + "source": [ + "import polars as pl\n", + "\n", + "df = pl.DataFrame(\n", + " data={\n", + " \"col0\": [0, 0, 0, 0],\n", + " \"col1\": [0, 0, 0, 0],\n", + " \"col2\": [0, 0, 0, 0],\n", + " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", + " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", + " },\n", + ")\n", + "\n", + "\n", + "# Chaining inside parentheses works\n", + "\n", + "results = df.group_by([\"col3\", \"col4\"]).agg(\n", + " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", + ")\n", + "\n", + "results" + ] + }, + { + "cell_type": "markdown", + "id": "6", + "metadata": {}, + "source": [ + "And this is what _not_ to do:\n", + "\n", + "```python\n", + "results = df\n", + " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", + "\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## Principles of Clean Code\n", + "\n", + "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", + "\n", + "### Do not repeat yourself (DRY)\n", + "\n", + "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", + "\n", + "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", + "\n", + "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", + "\n", + "::: {.callout-tip title=\"Tip\"}\n", + "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", + ":::\n", + "\n", + "### KISS (Keep It Simple, Stupid)\n", + "\n", + "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", + "\n", + "### SoC (Separation of Concerns) / Make it Modular\n", + "\n", + "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", + "\n", + "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", + "\n", + "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", + "\n", + "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", + "\n", + "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", + "\n", + "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", + "\n", + "```python\n", + "# This is bad\n", + "def transform(text, uppercase):\n", + " if uppercase:\n", + " return text.upper()\n", + " else:\n", + " return text.lower()\n", + "\n", + "# This is good\n", + "def uppercase(text):\n", + " return text.upper()\n", + "\n", + "def lowercase(text):\n", + " return text.lower()\n", + "```\n" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" + }, + "jupytext": { + "cell_metadata_filter": "-all", + "encoding": "# -*- coding: utf-8 -*-", + "formats": "md:myst", + "main_language": "python" + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + }, + "toc-showtags": true + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file From 5acf3d19a4906e2d8849d6fa338b7b32196db4e1 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 16 Jul 2026 12:19:44 +0100 Subject: [PATCH 06/30] feat: convert tidy data chapter from pandas to Polars --- data-tidy.ipynb | 265 ++++++++++++++++++++++++++---------------------- 1 file changed, 143 insertions(+), 122 deletions(-) diff --git a/data-tidy.ipynb b/data-tidy.ipynb index 157081f..8ce90e0 100644 --- a/data-tidy.ipynb +++ b/data-tidy.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Tidy Data {#sec-data-tidy}\n", @@ -19,7 +19,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "1", "metadata": { "tags": [ "remove-cell" @@ -38,17 +38,17 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package." ] }, { "cell_type": "markdown", - "id": "8b5f8255", + "id": "3", "metadata": {}, "source": [ "## Tidy Data\n", @@ -68,12 +68,12 @@ "1. There's a general advantage to picking one consistent way of storing data.\n", " If you have a consistent data structure, it's easier to learn the tools that work with it because they have an underlying uniformity. Some tools, for example data visualisation package **seaborn**, are designed with tidy data in mind.\n", "\n", - "2. There's a specific advantage to placing variables in columns because it allows you to take advantage of **pandas**' vectorised operations (operations that are more efficient).\n", + "2. There's a specific advantage to placing variables in columns because it allows you to take advantage of **polars**' vectorised operations (operations that are more efficient).\n", "\n", "\n", "Tidy data aren't going to be appropriate *every* time and in every case, but they're a really, really good default for tabular data. Once you use it as your default, it's easier to think about how to perform subsequent operations.\n", "\n", - "Having said that tidy data are great, they are, but one of **pandas**' advantages relative to other data analysis libraries is that it isn't *too* tied to tidy data and can navigate awkward non-tidy data manipulation tasks happily too.\n", + "Having said that tidy data are great, they are, but one of **polars**' advantages relative to other data analysis libraries is that it isn't *too* tied to tidy data and can navigate awkward non-tidy data manipulation tasks happily too.\n", "\n", "There are two common problems you find in data that are ingested that make them not tidy:\n", "\n", @@ -89,20 +89,20 @@ }, { "cell_type": "markdown", - "id": "deb8cf13", + "id": "4", "metadata": {}, "source": [ - "## Tools to Make Data Tidy with **pandas**" + "## Tools to Make Data Tidy with **polars**" ] }, { "cell_type": "markdown", - "id": "bd134b04", + "id": "5", "metadata": {}, "source": [ - "### Melt\n", + "### Unpivot\n", "\n", - "`melt()` can help you go from \"wider\" data to \"longer\" data, and is a *really* good one to remember.\n", + "`unpivot()` can help you go from \"wider\" data to \"longer\" data, and is a *really* good one to remember.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_melt.png)\n", "\n", @@ -112,13 +112,13 @@ { "cell_type": "code", "execution_count": null, - "id": "0f9fbf5a", + "id": "6", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", " \"first\": [\"John\", \"Mary\"],\n", " \"last\": [\"Doe\", \"Bo\"],\n", @@ -130,16 +130,21 @@ "print(\"\\n Unmelted: \")\n", "print(df)\n", "print(\"\\n Melted: \")\n", - "df.melt(id_vars=[\"first\", \"last\"], var_name=\"quantity\", value_vars=[\"height\", \"weight\"])" + "df.unpivot(\n", + " index=[\"first\", \"last\"],\n", + " variable_name=\"quantity\",\n", + " value_name=\"value\",\n", + " on=[\"height\", \"weight\"],\n", + ")" ] }, { "cell_type": "markdown", - "id": "e62978fd", + "id": "7", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", - "Perform a `melt()` that uses `job` as the id instead of `first` and `last`.\n", + "Perform a `unpivot()` that uses `job` as the id instead of `first` and `last`.\n", ":::\n", "\n", "How does this relate to tidy data? Sometimes you'll have a variable spread over multiple columns that you want to turn tidy. Let's look at this example that uses cases of [tuburculosis from the World Health Organisation](https://www.who.int/teams/global-tuberculosis-programme/data).\n", @@ -150,11 +155,11 @@ { "cell_type": "code", "execution_count": null, - "id": "bfa121cf", + "id": "8", "metadata": {}, "outputs": [], "source": [ - "df_tb = pd.read_parquet(\n", + "df_tb = pl.read_parquet(\n", " \"https://github.com/aeturrell/python4DS/raw/refs/heads/main/data/who_tb_cases.parquet\"\n", ")\n", "df_tb.head()" @@ -162,7 +167,7 @@ }, { "cell_type": "markdown", - "id": "583e419d", + "id": "9", "metadata": {}, "source": [ "You can see that we have two columns for a single variable, year. Let's now melt this." @@ -171,21 +176,21 @@ { "cell_type": "code", "execution_count": null, - "id": "dc03ccd9", + "id": "10", "metadata": {}, "outputs": [], "source": [ - "df_tb.melt(\n", - " id_vars=[\"country\"],\n", - " var_name=\"year\",\n", - " value_vars=[\"1999\", \"2000\"],\n", + "df_tb.unpivot(\n", + " index=[\"country\"],\n", + " variable_name=\"year\",\n", " value_name=\"cases\",\n", + " on=[\"1999\", \"2000\"],\n", ")" ] }, { "cell_type": "markdown", - "id": "74d81b30", + "id": "11", "metadata": {}, "source": [ "We now have one observation per row, and one variable per column: tidy!" @@ -193,31 +198,31 @@ }, { "cell_type": "markdown", - "id": "488f5f10", + "id": "12", "metadata": {}, "source": [ - "### A simpler wide to long\n", + "### Reshaping Wide Data with Multiple Variables\n", "\n", - "If you don't want the headscratching of `melt()`, there's also `wide_to_long()`, which is really useful for typical data cleaning cases where you have data like this:" + "Sometimes you have multiple variables spread across columns in a wide format. Here's how to handle that in polars:" ] }, { "cell_type": "code", "execution_count": null, - "id": "293768c1", + "id": "13", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", - " \"A1970\": {0: \"a\", 1: \"b\", 2: \"c\"},\n", - " \"A1980\": {0: \"d\", 1: \"e\", 2: \"f\"},\n", - " \"B1970\": {0: 2.5, 1: 1.2, 2: 0.7},\n", - " \"B1980\": {0: 3.2, 1: 1.3, 2: 0.1},\n", - " \"X\": dict(zip(range(3), np.random.randn(3))),\n", - " \"id\": dict(zip(range(3), range(3))),\n", + " \"A1970\": [\"a\", \"b\", \"c\"],\n", + " \"A1980\": [\"d\", \"e\", \"f\"],\n", + " \"B1970\": [2.5, 1.2, 0.7],\n", + " \"B1980\": [3.2, 1.3, 0.1],\n", + " \"X\": np.random.randn(3),\n", + " \"id\": [0, 1, 2],\n", " }\n", ")\n", "df" @@ -225,147 +230,154 @@ }, { "cell_type": "markdown", - "id": "fb6c09e6", + "id": "14", "metadata": {}, "source": [ - "i.e. data where there are different variables and time periods across the columns. Wide to long is going to let us give info on what the stubnames are ('A', 'B'), the name of the variable that's always across columns (here, a year), any values (X here), and an id column." + "To reshape this data, we first unpivot, then extract the year and variable:" ] }, { "cell_type": "code", "execution_count": null, - "id": "a9ca2fa8", + "id": "15", "metadata": {}, "outputs": [], "source": [ - "pd.wide_to_long(df, stubnames=[\"A\", \"B\"], i=\"id\", j=\"year\")" + "df_long = df.unpivot(\n", + " index=[\"id\", \"X\"],\n", + " variable_name=\"variable_year\",\n", + " value_name=\"value\",\n", + " on=[\"A1970\", \"A1980\", \"B1970\", \"B1980\"],\n", + ")\n", + "\n", + "df_long = df_long.with_columns(\n", + " [\n", + " pl.col(\"variable_year\").str.slice(0, 1).alias(\"variable\"),\n", + " pl.col(\"variable_year\").str.slice(1, 4).cast(pl.Int64).alias(\"year\"),\n", + " ]\n", + ")\n", + "\n", + "df_final = df_long.drop(\"variable_year\").pivot(\n", + " index=[\"id\", \"X\", \"year\"],\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ")\n", + "\n", + "df_final.sort([\"id\", \"year\"])" ] }, { "cell_type": "markdown", - "id": "1a6bf17c", + "id": "16", "metadata": {}, "source": [ "### Stack and Unstack\n", "\n", - "Stack, `stack()` is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", + "Polars uses unpivot() and pivot() for reshaping. \n", + "\n", + "Stack is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_stack.png)\n", "\n", - "Unstack, `unstack()` unsurprisingly does the same operation, but in reverse.\n", + "To go back to wider format (unstack)\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_unstack.png)\n", "\n", - "Let's define a multi-index data frame to demonstrate this:" + "Here's how to work with hierarchical data:" ] }, { "cell_type": "code", "execution_count": null, - "id": "2b791dd1", + "id": "17", "metadata": {}, "outputs": [], "source": [ - "tuples = list(\n", - " zip(\n", - " *[\n", - " [\"bar\", \"bar\", \"baz\", \"baz\", \"foo\", \"foo\", \"qux\", \"qux\"],\n", - " [\"one\", \"two\", \"one\", \"two\", \"one\", \"two\", \"one\", \"two\"],\n", - " ]\n", - " )\n", + "# Create a DataFrame with multiple columns\n", + "df = pl.DataFrame(\n", + " {\n", + " \"first\": [\"bar\", \"bar\", \"baz\", \"baz\", \"foo\", \"foo\", \"qux\", \"qux\"],\n", + " \"second\": [\"one\", \"two\", \"one\", \"two\", \"one\", \"two\", \"one\", \"two\"],\n", + " \"A\": np.random.randn(8),\n", + " \"B\": np.random.randn(8),\n", + " }\n", ")\n", - "index = pd.MultiIndex.from_tuples(tuples, names=[\"first\", \"second\"])\n", - "df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=[\"A\", \"B\"])\n", "df" ] }, { "cell_type": "markdown", - "id": "38e7c349", + "id": "18", "metadata": {}, "source": [ - "Let's stack this to create a tidy dataset:" + "To go to a longer format (stack):" ] }, { "cell_type": "code", "execution_count": null, - "id": "d25eb012", + "id": "19", "metadata": {}, "outputs": [], "source": [ - "df = df.stack()\n", - "df" - ] - }, - { - "cell_type": "markdown", - "id": "edeef123", - "metadata": {}, - "source": [ - "This has automatically created a multi-layered index but that can be reverted to a numbered index using `df.reset_index()`" + "# Unpivot to stack the A and B columns\n", + "df_stacked = df.unpivot(\n", + " index=[\"first\", \"second\"],\n", + " variable_name=\"variable\",\n", + " value_name=\"value\",\n", + " on=[\"A\", \"B\"],\n", + ")\n", + "df_stacked" ] }, { "cell_type": "markdown", - "id": "6b1e4e03", + "id": "20", "metadata": {}, "source": [ - "Now let's see unstack but, instead of unstacking the 'A', 'B' variables we began with, let's unstack the 'first' column by passing `level=0` (the default is to unstack the innermost index). This diagram shows what's going on:\n", - "\n", - "![](https://pandas.pydata.org/docs/_images/reshaping_unstack_0.png)\n", - "\n", - "And here's the code:" + "then to wider format (unstack)" ] }, { "cell_type": "code", "execution_count": null, - "id": "b6742a54", + "id": "21", "metadata": {}, "outputs": [], "source": [ - "df.unstack(level=0)" - ] - }, - { - "cell_type": "markdown", - "id": "185eae1f", - "metadata": {}, - "source": [ - "::: {.callout-tip title=\"Exercise\"}\n", - "What happens if you unstack to `level=1` instead? What about applying `unstack()` twice?\n", - ":::" + "# Pivot back to wider format\n", + "df_unstacked = df_stacked.pivot(\n", + " index=[\"first\", \"second\"],\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ")\n", + "df_unstacked" ] }, { "cell_type": "markdown", - "id": "39a210bb", + "id": "22", "metadata": {}, "source": [ - "### Pivoting data from long to wide\n", + "### Pivoting from Long to Wide\n", "\n", - "`pivot()` and `pivot_table()` help you to sort out data in which a single observation is scattered over multiple rows.\n", + "pivot() helps you to sort out data in which a single observation is scattered over multiple rows.\n", + "\n", + "![](https://pandas.pydata.org/docs/_images/reshaping_pivot.png)\n", "\n", - "![](https://pandas.pydata.org/docs/_images/reshaping_pivot.png)\n" - ] - }, - { - "cell_type": "markdown", - "id": "8acbcedf", - "metadata": {}, - "source": [ "Here's an example dataframe where observations are spread over multiple rows:" ] }, { "cell_type": "code", "execution_count": null, - "id": "fa612456", + "id": "23", "metadata": {}, "outputs": [], "source": [ - "df_tb_cp = pd.read_parquet(\n", + "df_tb_cp = pl.read_parquet(\n", " \"https://github.com/aeturrell/python4DS/raw/refs/heads/main/data/who_tb_case_and_pop.parquet\"\n", ")\n", "df_tb_cp.head()" @@ -373,49 +385,47 @@ }, { "cell_type": "markdown", - "id": "0d4a4077", - "metadata": {}, - "source": [ - "You see that we have, for each year-country, \"case\" and \"population\" in different rows." - ] - }, - { - "cell_type": "markdown", - "id": "e7c9ed1b", + "id": "24", "metadata": {}, "source": [ + "You see that we have, for each year-country, \"case\" and \"population\" in different rows.\n", + "\n", "Now let's pivot this to see the difference:" ] }, { "cell_type": "code", "execution_count": null, - "id": "e584cf37", + "id": "25", "metadata": {}, "outputs": [], "source": [ "pivoted = df_tb_cp.pivot(\n", - " index=[\"country\", \"year\"], columns=[\"type\"], values=\"count\"\n", - ").reset_index()\n", + " index=[\"country\", \"year\"],\n", + " columns=\"type\",\n", + " values=\"count\",\n", + " aggregate_function=\"first\",\n", + ").sort([\"country\", \"year\"])\n", "pivoted" ] }, { "cell_type": "markdown", - "id": "d82307c0", + "id": "26", "metadata": {}, "source": [ - "Pivots are especially useful for time series data, where operations like `shift()` or `diff()` are typically applied assuming that an entry in one row follows (in time) from the one above. When we do `shift()` we often want to shift a single variable in time, but if a single observation (in this case a date) is over multiple rows, the timing is going go awry. Let's see an example." + "Pivots are especially useful for time series data, where operations like `shift()` are typically applied assuming that an entry in one row follows (in time) from the one above. When we do `shift()` we often want to shift a single variable in time, but if a single observation (in this case a date) is over multiple rows, the timing goes awry. Let's see an example." ] }, { "cell_type": "code", "execution_count": null, - "id": "97c6d139", + "id": "27", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", + "import pandas as pd\n", "\n", "data = {\n", " \"value\": np.random.randn(20),\n", @@ -425,31 +435,42 @@ " + list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", " ),\n", "}\n", - "df = pd.DataFrame(data, columns=[\"date\", \"variable\", \"value\"])\n", + "df = pl.DataFrame(data)\n", "df.sample(5)" ] }, { "cell_type": "markdown", - "id": "90a5b930", + "id": "28", "metadata": {}, "source": [ - "If we just run `shift()` on the above, it's going to shift variable B's and A's together even though these overlap in time and are different variables. So we pivot to a wider format (and then we can shift in time safely)." + "If we just run `shift()` on the above, it's going to shift variable B's and A's together even though they overlap in time and are different variables. So we pivot to a wider format to shift safely:" ] }, { "cell_type": "code", "execution_count": null, - "id": "04f2bd28", + "id": "29", "metadata": {}, "outputs": [], "source": [ - "df.pivot(index=\"date\", columns=\"variable\", values=\"value\").shift(1)" + "df_pivoted = df.pivot(\n", + " index=\"date\",\n", + " columns=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ").with_columns(\n", + " [\n", + " pl.col(\"A\").shift(1),\n", + " pl.col(\"B\").shift(1),\n", + " ]\n", + ")\n", + "df_pivoted" ] }, { "cell_type": "markdown", - "id": "841c4821", + "id": "30", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -471,7 +492,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 2998f9b60dc90edadcfe35aa3ecdca106dabdced Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 16 Jul 2026 14:47:21 +0100 Subject: [PATCH 07/30] migrate: convert joins.ipynb from pandas to polars --- joins.ipynb | 268 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 217 insertions(+), 51 deletions(-) diff --git a/joins.ipynb b/joins.ipynb index d494d84..8ec4902 100644 --- a/joins.ipynb +++ b/joins.ipynb @@ -11,7 +11,7 @@ "\n", "It's rare that a data analysis involves only a single data frame. Typically you have many data frames, and you must *join* them together to answer the questions that you're interested in.\n", "\n", - "**pandas** has a really rich set of options for combining one or more data frames, with the two most important being concatenate and merge. Some of the examples in this chapter show you how to join a pair of data frames. Fortunately this is enough, since you can combine three data frames by combining two pairs." + "**polars** has a really rich set of options for combining one or more data frames, with the two most important being concatenate and joins. Some of the examples in this chapter show you how to join a pair of data frames. Fortunately this is enough, since you can combine three data frames by combining two pairs." ] }, { @@ -41,7 +41,7 @@ "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package." ] }, { @@ -51,15 +51,13 @@ "source": [ "## Concatenate\n", "\n", - "If you have two or more data frames with the same index or the same columns, you can glue them together into a single data frame using `pd.concat()`.\n", + "If you have two or more data frames with the same columns, you can glue them together into a single data frame using pl.concat().\n", "\n", "![](https://pandas.pydata.org/docs/_images/08_concat_row.svg)\n", "\n", - "For the same columns, pass `axis=0` to glue the index together; for the same index, pass `axis=1` to glue the columns together. The concatenate function will typically be used on a list of data frames.\n", + "For the same columns, use pl.concat() to stack rows vertically. For the same index, you can use pl.concat() with how=\"horizontal\".\n", "\n", - "If you want to track where the original data came from in the final data frame, use the `keys` keyword.\n", - "\n", - "Here's an example using data on two different states' populations that also makes uses of the `keys` option:" + "Here's an example using data on two different states' populations:" ] }, { @@ -69,7 +67,7 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", "base_url = (\n", " \"https://github.com/aeturrell/coding-for-economists/raw/refs/heads/main/data/\"\n", @@ -77,32 +75,66 @@ "state_codes = [\"ca\", \"il\"]\n", "end_url = \"pop.dta\"\n", "\n", - "# This grabs the two dataframes, one for each state\n", - "list_of_state_dfs = [pd.read_stata(base_url + state + end_url) for state in state_codes]\n", - "# Show example of first entry in list of dataframes\n", + "# Read the Stata files and convert to polars\n", + "list_of_state_dfs = []\n", + "for state in state_codes:\n", + " import pandas as pd\n", + "\n", + " temp_df = pd.read_stata(base_url + state + end_url)\n", + " list_of_state_dfs.append(pl.from_pandas(temp_df))\n", + "\n", + "# shows example of the 2 dataframes\n", "print(list_of_state_dfs[0])\n", + "print(list_of_state_dfs[1])\n", "\n", - "# Concatenate the list of dataframes\n", - "df = pd.concat(list_of_state_dfs, keys=state_codes, axis=0)\n", + "# concatenate the list of dataframes\n", + "df = pl.concat(list_of_state_dfs)\n", "df" ] }, { "cell_type": "markdown", - "id": "e0c343b0", + "id": "79bbd002", "metadata": {}, "source": [ - "Note that the `keys` argument is optional, but is useful for keeping track of origin data frames within the merged data frame.\n", + "Note that when concatenating, you can track the origin of rows by adding a column before concatenation:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2adb960a", + "metadata": {}, + "outputs": [], + "source": [ + "# Add a state identifier column to each dataframe\n", + "list_with_state = []\n", + "for state, df_state in zip(state_codes, list_of_state_dfs):\n", + " df_state = df_state.with_columns(pl.lit(state).alias(\"state\"))\n", + " list_with_state.append(df_state)\n", "\n", + "# Concatenate with state identifiers\n", + "df_with_state = pl.concat(list_with_state)\n", + "df_with_state" + ] + }, + { + "cell_type": "markdown", + "id": "e0c343b0", + "metadata": {}, + "source": [ "::: {.callout-tip title=\"Exercise\"}\n", "Concatenate the follow two data frames:\n", "\n", "```python\n", - "df1 = pd.DataFrame([['a', 1], ['b', 2]],\n", - " columns=['letter', 'number'])\n", + "df1 = pl.DataFrame({\n", + " 'letter': ['a', 'b'],\n", + " 'number': [1, 2]\n", + "})\n", "\n", - "df2 = pd.DataFrame([['c', 3], ['d', 4]],\n", - " columns=['letter', 'number'])\n", + "df2 = pl.DataFrame({\n", + " 'letter': ['c', 'd'],\n", + " 'number': [3, 4]\n", "```\n", "\n", ":::" @@ -113,17 +145,25 @@ "id": "f6e91777", "metadata": {}, "source": [ - "## Merge\n", + "## Joins\n", "\n", - "There are so many options for merging data frames using `pd.merge(left, right, on=..., how=...` that we won't be able to cover them all here. The most important features are: the two data frames to be merged, what variables (aka keys) to merge on (and these can be indexes) via `on=`, and *how* to do the merge (eg left, right, outer, inner) via `how=`. This diagram shows an example of a merge using keys from the left-hand data frame:\n", + "There are many options for joining data frames using `pl.DataFrame.join()`. The most important features are: the two data frames to be joined, what variables (aka keys) to join on via `on=`, and *how* to do the join (eg left, right, outer, inner) via `how=`. This diagram shows an example of a left join:\n", "\n", "![](https://pandas.pydata.org/docs/_images/08_merge_left.svg)\n", "\n", "The `how=` keyword works in the following ways:\n", - "- `how='left'` uses keys from the left data frame only to merge.\n", - "- `how='right'` uses keys from the right data frame only to merge.\n", - "- `how='inner'` uses keys that appear in both data frames to merge.\n", - "- `how='outer'` uses the cartesian product of keys in both data frames to merge on.\n", + "\n", + "- how=\"left\" uses keys from the left data frame only to join.\n", + "\n", + "- how=\"right\" uses keys from the right data frame only to join.\n", + "\n", + "- how=\"inner\" uses keys that appear in both data frames to join.\n", + "\n", + "- how=\"outer\" uses all keys from both data frames to join on.\n", + "\n", + "- how=\"semi\" returns rows from the left where keys appear in the right.\n", + "\n", + "- how=\"anti\" returns rows from the left where keys do NOT appear in the right.\n", "\n", "Let's see examples of some of these:" ] @@ -135,7 +175,7 @@ "metadata": {}, "outputs": [], "source": [ - "left = pd.DataFrame(\n", + "left = pl.DataFrame(\n", " {\n", " \"key1\": [\"K0\", \"K0\", \"K1\", \"K2\"],\n", " \"key2\": [\"K0\", \"K1\", \"K0\", \"K1\"],\n", @@ -143,16 +183,32 @@ " \"B\": [\"B0\", \"B1\", \"B2\", \"B3\"],\n", " }\n", ")\n", - "right = pd.DataFrame(\n", + "right = pl.DataFrame(\n", " {\n", " \"key1\": [\"K0\", \"K1\", \"K1\", \"K2\"],\n", " \"key2\": [\"K0\", \"K0\", \"K0\", \"K0\"],\n", " \"C\": [\"C0\", \"C1\", \"C2\", \"C3\"],\n", " \"D\": [\"D0\", \"D1\", \"D2\", \"D3\"],\n", " }\n", - ")\n", - "# Right merge\n", - "pd.merge(left, right, on=[\"key1\", \"key2\"], how=\"right\")" + ")" + ] + }, + { + "cell_type": "markdown", + "id": "cfe5aa30", + "metadata": {}, + "source": [ + "Right join:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "595f41e6", + "metadata": {}, + "outputs": [], + "source": [ + "left.join(right, on=[\"key1\", \"key2\"], how=\"right\")" ] }, { @@ -160,7 +216,7 @@ "id": "8af0241e", "metadata": {}, "source": [ - "Note that the key combination of K2 and K0 did not exist in the left-hand data frame, and so its entries in the final data frame are NaNs. But it *does* have entries because we chose the keys from the right-hand data frame.\n", + "Note that the key combination of K2 and K0 did not exist in the left-hand data frame, and so its entries in the final data frame are null values. But it does have entries because we chose the keys from the right-hand data frame.\n", "\n", "What about an inner merge?" ] @@ -172,7 +228,7 @@ "metadata": {}, "outputs": [], "source": [ - "pd.merge(left, right, on=[\"key1\", \"key2\"], how=\"inner\")" + "left.join(right, on=[\"key1\", \"key2\"], how=\"inner\")" ] }, { @@ -182,7 +238,7 @@ "source": [ "Now we see that the combination K2 and K0 are excluded because they didn't exist in the overlap of keys in both data frames.\n", "\n", - "Finally, let's take a look at an outer merge that comes with some extra info via the `indicator` keyword:" + "Finally, let's take a look at an outer join:" ] }, { @@ -192,48 +248,158 @@ "metadata": {}, "outputs": [], "source": [ - "pd.merge(left, right, on=[\"key1\", \"key2\"], how=\"outer\", indicator=True)" + "left.join(right, on=[\"key1\", \"key2\"], how=\"outer\")" ] }, { "cell_type": "markdown", - "id": "67ca901b", + "id": "b984a202", + "metadata": {}, + "source": [ + "Polars doesn't include a built-in indicator parameter to track which side contributed each row in a join. However, you can achieve the same effect by comparing the two DataFrames after the join:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "574c4508", "metadata": {}, + "outputs": [], "source": [ - "Now we can see that the products of all key combinations are here. The `indicator=True` option has caused an extra column to be added, called '_merge', that tells us which data frame the keys on that row came from.\n", + "# Perform an outer join\n", + "joined = left.join(right, on=[\"key1\", \"key2\"], how=\"outer\")\n", "\n", + "# Add indicator column manually\n", + "# Rows with nulls in B only came from right\n", + "# Rows with nulls in D only came from left\n", + "# Rows with no nulls came from both\n", + "joined = joined.with_columns(\n", + " pl.when(pl.col(\"B\").is_null() & pl.col(\"D\").is_not_null())\n", + " .then(pl.lit(\"right_only\"))\n", + " .when(pl.col(\"B\").is_not_null() & pl.col(\"D\").is_null())\n", + " .then(pl.lit(\"left_only\"))\n", + " .otherwise(pl.lit(\"both\"))\n", + " .alias(\"_merge\")\n", + ")\n", + "joined" + ] + }, + { + "cell_type": "markdown", + "id": "67ca901b", + "metadata": {}, + "source": [ "::: {.callout-tip title=\"Exercise\"}\n", "\n", - "Merge the following two data frames using the `left_on` and `right_on` keyword arguments to specify a join on `lkey` and `rkey` respectively:\n", + "Join the following two data frames using the `left_on` and `right_on` keyword arguments to specify a join on lkey and rkey respectively:\n", "\n", "```python\n", - "df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],\n", - " 'value': [1, 2, 3, 5]})\n", - "df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],\n", - " 'value': [5, 6, 7, 8]})\n", + "df1 = pl.DataFrame({\n", + " 'lkey': ['foo', 'bar', 'baz', 'foo'],\n", + " 'value': [1, 2, 3, 5]\n", + "})\n", + "df2 = pl.DataFrame({\n", + " 'rkey': ['foo', 'bar', 'baz', 'foo'],\n", + " 'value': [5, 6, 7, 8]\n", + "})\n", "```\n", ":::\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", "\n", - "Merge the following two data frames on `\"a\"` using `how=\"left\"` as a keyword argument:\n", + "Join the following two data frames on `\"a\"` using `how=\"left\"`:\n", "\n", "```python\n", - "df1 = pd.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]})\n", - "df2 = pd.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]})\n", + "df1 = pl.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]})\n", + "df2 = pl.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]})\n", "```\n", "\n", - "What do you notice about the position `.loc[1, \"c\"]` in the merged data frame? \n", - ":::\n", + "What do you notice about the resulting row for `\"baz\"`?\n", + ":::\n" + ] + }, + { + "cell_type": "markdown", + "id": "5d09e145", + "metadata": {}, + "source": [ + "## Semi and Anti Joins\n", + "\n", + "Polars also supports semi and anti joins, which are useful for filtering:\n", + "\n", + "Semi join: Keep rows from the left where keys appear in the right:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "33207e8a", + "metadata": {}, + "outputs": [], + "source": [ + "# Semi join - only keeps left rows that have matches in right\n", + "left.join(right, on=[\"key1\", \"key2\"], how=\"semi\")" + ] + }, + { + "cell_type": "markdown", + "id": "d55603d0", + "metadata": {}, + "source": [ + "Anti join: Keep rows from the left where keys do NOT appear in the right:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "92218fbb", + "metadata": {}, + "outputs": [], + "source": [ + "# Anti join - only keeps left rows without matches in right\n", + "left.join(right, on=[\"key1\", \"key2\"], how=\"anti\")" + ] + }, + { + "cell_type": "markdown", + "id": "355135ce", + "metadata": {}, + "source": [ + "## Handling Duplicate Column Names\n", + "\n", + "When joining, if both DataFrames have columns with the same name (other than the join keys), polars will add suffixes to differentiate them. By default it adds `_right`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a042820b", + "metadata": {}, + "outputs": [], + "source": [ + "# Create DataFrames with overlapping column names\n", + "left2 = pl.DataFrame(\n", + " {\"key\": [\"K0\", \"K1\", \"K2\"], \"value\": [1, 2, 3], \"shared\": [\"L0\", \"L1\", \"L2\"]}\n", + ")\n", + "\n", + "right2 = pl.DataFrame(\n", + " {\"key\": [\"K0\", \"K1\", \"K2\"], \"value\": [4, 5, 6], \"shared\": [\"R0\", \"R1\", \"R2\"]}\n", + ")\n", "\n", - "For more on the options for merging, see **pandas**' comprehensive [merging documentation](https://pandas.pydata.org/docs/user_guide/merging.html#database-style-dataframe-or-named-series-joining-merging)." + "# Join with default suffix\n", + "left2.join(right2, on=\"key\", suffix=\"_right\")" + ] + }, + { + "cell_type": "markdown", + "id": "aa6e6071", + "metadata": {}, + "source": [ + "For more information about the available join types and options, see **polars'** [joining documentation](https://docs.pola.rs/api/python/stable/reference/index.html)." ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -241,7 +407,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 179dd052455c7ecd362b60a9152e19154d57a41f Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 16 Jul 2026 21:38:24 +0100 Subject: [PATCH 08/30] fix(datetime): add time_zone='UTC' to to_datetime() to resolve Polars parsing error --- data-transform.ipynb | 2 +- workflow-style.ipynb | 558 +++++++++++++++++++++---------------------- 2 files changed, 280 insertions(+), 280 deletions(-) diff --git a/data-transform.ipynb b/data-transform.ipynb index 8dfc2c1..ccc75f0 100644 --- a/data-transform.ipynb +++ b/data-transform.ipynb @@ -164,7 +164,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights.with_columns(pl.col(\"time_hour\").str.to_datetime())" + "flights.with_columns(pl.col(\"time_hour\").str.to_datetime(time_zone=\"UTC\"))" ] }, { diff --git a/workflow-style.ipynb b/workflow-style.ipynb index ac02375..98b5469 100644 --- a/workflow-style.ipynb +++ b/workflow-style.ipynb @@ -1,280 +1,280 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "0", - "metadata": {}, - "source": [ - "# Workflow: Style {#sec-workflow-style}\n", - "\n", - "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", - "\n", - "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", - "\n", - "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", - "\n", - "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" - ] - }, - { - "cell_type": "markdown", - "id": "1", - "metadata": {}, - "source": [ - "## Names\n", - "\n", - "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", - "\n", - "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", - "\n", - "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", - "\n", - "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", - "\n", - "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", - "\n", - "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", - "\n", - "In summary:\n", - "\n", - "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", - "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", - "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", - "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", - "\n", - " ```python\n", - " import random\n", - "\n", - " # This is bad\n", - " def roll():\n", - " return random.randint(0, 36) # magic number!\n", - "\n", - " # This is good\n", - " MAX_INT_VALUE = 36\n", - "\n", - " def roll():\n", - " return random.randint(0, MAX_INT_VALUE)\n", - " ```\n", - "\n", - "- use verbs for function names, eg `get_regression()`\n", - "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", - "- variable names should be snake_case and all lowercase, eg `first_name`\n", - "- class names should be CamelCase, eg `MyClass`\n", - "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", - "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", - "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", - "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" - ] - }, - { - "cell_type": "markdown", - "id": "2", - "metadata": {}, - "source": [ - "## Whitespace\n", - "\n", - "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", - "\n", - "```python\n", - "this_is_a_var = 5\n", - "```\n", - "\n", - "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", - "\n", - "```python\n", - "list_var = [1, 2, 3, 4]\n", - "```\n", - "\n", - "rather than\n", - "\n", - "```python\n", - "list_var = [1,2,3,4]\n", - "# or\n", - "list_var = [1 , 2 , 3 , 4]\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "3", - "metadata": {}, - "source": [ - "## Code Comments\n", - "\n", - "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", - "\n", - "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", - "\n", - "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", - "\n", - "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", - "\n", - "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", - "\n", - "```python\n", - "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", - " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", - " Args:\n", - " df (pl.DataFrame): Input dataframe\n", - " Returns:\n", - " pl.DataFrame: Dataframe of mean value of each numeric column.\n", - " \"\"\"\n", - " df_mean = df.select(S.numeric()).mean()\n", - " return df_mean\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "4", - "metadata": {}, - "source": [ - "## Line width and line continuation\n", - "\n", - "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", - "\n", - "```python\n", - "def function(input_one, input_two,\n", - " input_three, input_four):\n", - " result = (input_one,\n", - " + input_two,\n", - " + input_three,\n", - " + input_four)\n", - " return result\n", - "```\n", - "\n", - "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5", - "metadata": {}, - "outputs": [], - "source": [ - "import polars as pl\n", - "\n", - "df = pl.DataFrame(\n", - " data={\n", - " \"col0\": [0, 0, 0, 0],\n", - " \"col1\": [0, 0, 0, 0],\n", - " \"col2\": [0, 0, 0, 0],\n", - " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", - " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", - " },\n", - ")\n", - "\n", - "\n", - "# Chaining inside parentheses works\n", - "\n", - "results = df.group_by([\"col3\", \"col4\"]).agg(\n", - " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", - ")\n", - "\n", - "results" - ] - }, - { - "cell_type": "markdown", - "id": "6", - "metadata": {}, - "source": [ - "And this is what _not_ to do:\n", - "\n", - "```python\n", - "results = df\n", - " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", - "\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "7", - "metadata": {}, - "source": [ - "## Principles of Clean Code\n", - "\n", - "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", - "\n", - "### Do not repeat yourself (DRY)\n", - "\n", - "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", - "\n", - "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", - "\n", - "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", - "\n", - "::: {.callout-tip title=\"Tip\"}\n", - "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", - ":::\n", - "\n", - "### KISS (Keep It Simple, Stupid)\n", - "\n", - "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", - "\n", - "### SoC (Separation of Concerns) / Make it Modular\n", - "\n", - "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", - "\n", - "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", - "\n", - "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", - "\n", - "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", - "\n", - "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", - "\n", - "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", - "\n", - "```python\n", - "# This is bad\n", - "def transform(text, uppercase):\n", - " if uppercase:\n", - " return text.upper()\n", - " else:\n", - " return text.lower()\n", - "\n", - "# This is good\n", - "def uppercase(text):\n", - " return text.upper()\n", - "\n", - "def lowercase(text):\n", - " return text.lower()\n", - "```\n" - ] - } - ], - "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, - "jupytext": { - "cell_metadata_filter": "-all", - "encoding": "# -*- coding: utf-8 -*-", - "formats": "md:myst", - "main_language": "python" - }, - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.13" - }, - "toc-showtags": true - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# Workflow: Style {#sec-workflow-style}\n", + "\n", + "Good coding style is like correct punctuation: you can manage without it, butitsuremakesthingseasiertoread. Even as a very new programmer it's a good idea to work on your code style. Use a consistent style makes it easier for others (including future-you!) to read your work, and is particularly important if you need to get help from someone else.\n", + "\n", + "This chapter will introduce you to some important style points drawn from [Clean Code in Python](https://testdriven.io/blog/clean-code-python/), [Tips for Better Coding](https://aeturrell.github.io/coding-for-economists/code-best-practice.html) from _Coding for Economists_, the UK Government Statistical Service’s [Quality Assurance of Code for Analysis and Research](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) guidance, and the bible of Python style guides, [PEP 8 — Style Guide for Python Code](https://peps.python.org/pep-0008/).\n", + "\n", + "Styling your code will feel a bit tedious to start with, but if you practice it, it will soon become second nature. Additionally, there are some great tools to quickly restyle existing code, like the [Black](https://black.readthedocs.io/) Python package (\"you can have any colour you like, as long as it's black\").\n", + "\n", + "Once you've installed Black by running `uv tool install black`, you can use it on the command line (aka the terminal) within Visual Studio Code. Open up a terminal by clicking 'Terminal -> New Terminal' and then run `black *.py` to apply a standard code style to all Python scripts in the current directory.\n" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## Names\n", + "\n", + "First, names matter. Use meaningful names for variables, functions, or whatever it is you're naming. Avoid abbreviations that you understand _now_ but which will be unclear to others, or future you. For example, use `real_wage_hourly` over `re_wg_ph`. I know it's tempting to use `temp` but you'll feel silly later when you can't for the life of you remember what `temp` does or is. A good trick when naming booleans (variables that are either true or false) is to use `is` followed by what the boolean variable refers to, for example `is_married`.\n", + "\n", + "As well as this general tip, Python has conventions on naming different kinds of variables. The naming convention for almost all objects is lower case separated by underscores, e.g. `a_variable=10` or ‘this_is_a_script.py’. This style of naming is also known as snake case. There are different naming conventions though—[Allison Horst](https://twitter.com/allison_horst) made this fantastic cartoon of the different conventions that are in use.\n", + "\n", + "![Different naming conventions. Artwork by @allison_horst.](https://github.com/aeturrell/coding-for-economists/raw/main/img/in_that_case.jpg) Different naming conventions. Artwork by \\@allison_horst.\n", + "\n", + "There are three exceptions to the snake case convention: classes, which should be in camel case, eg `ThisIsAClass`; constants, which are in capital snake case, eg `THIS_IS_A_CONSTANT`; and packages, which are typically without spaces or underscores and are lowercase `thisisapackage`.\n", + "\n", + "For some quick shortcuts to re-naming columns in **polars** data frames or other string variables, try the unicode-friendly [**slugify**](https://github.com/un33k/python-slugify) library or the `clean_headers()` function from the [**dataprep**](https://docs.dataprep.ai/index.html) library.\n", + "\n", + "The better named your variables, the clearer your code will be--and the fewer comments you will need to write!\n", + "\n", + "In summary:\n", + "\n", + "- use descriptive variable names that reveal your intention, eg `days_since_treatment`\n", + "- avoid using ambiguous abbreviations in names, eg use `real_wage_hourly` over `rw_ph`\n", + "- always use the same vocabulary, eg don't switch from `worker_type` to `employee_type`\n", + "- avoid 'magic numbers', eg numbers in your code that set a key parameter. Set these as named constants instead. Here's an example:\n", + "\n", + " ```python\n", + " import random\n", + "\n", + " # This is bad\n", + " def roll():\n", + " return random.randint(0, 36) # magic number!\n", + "\n", + " # This is good\n", + " MAX_INT_VALUE = 36\n", + "\n", + " def roll():\n", + " return random.randint(0, MAX_INT_VALUE)\n", + " ```\n", + "\n", + "- use verbs for function names, eg `get_regression()`\n", + "- use consistent verbs for function names, don't use `get_score()` and `grab_results()` (instead use `get` for both)\n", + "- variable names should be snake_case and all lowercase, eg `first_name`\n", + "- class names should be CamelCase, eg `MyClass`\n", + "- function names should be snake_case and all lowercase, eg `quick_sort()`\n", + "- constants should be snake_case and all uppercase, eg `PI = 3.14159`\n", + "- modules should have short, snake_case names and all lowercase, eg `pandas`\n", + "- single quotes and double quotes are equivalent so pick one and be consistent—most automatic formatters prefer `\"`\n" + ] + }, + { + "cell_type": "markdown", + "id": "2", + "metadata": {}, + "source": [ + "## Whitespace\n", + "\n", + "Surrounding bits of code with whitespace can significantly enhance readability. One such convention is that functions should have two blank lines following their last line. Another is that assignments are separated by spaces\n", + "\n", + "```python\n", + "this_is_a_var = 5\n", + "```\n", + "\n", + "Another convention is that a space appears after a `,`, for example in the definition of a list we would have\n", + "\n", + "```python\n", + "list_var = [1, 2, 3, 4]\n", + "```\n", + "\n", + "rather than\n", + "\n", + "```python\n", + "list_var = [1,2,3,4]\n", + "# or\n", + "list_var = [1 , 2 , 3 , 4]\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## Code Comments\n", + "\n", + "As mentioned previously, Python will ignore any text after `#`. This allows to you to write **comments**, text that is ignored by Python but can be read by other humans. Comments can be helpful for briefly describing what the subsequent code does: use them to provide extra contextual information that _isn't_ conveyed by function and variable names.\n", + "\n", + "Actually, well-written code needs _fewer_ comments because it's more evidence what's going on. And it's tempting not to update comments even when code changes. So do comment, but see if you can make the code tell its own story first.\n", + "\n", + "Also, avoid \"noise\" comments that tell you what you already know from just looking at the code.\n", + "\n", + "![Picture of a cat wearing a label that says cat](https://i.postimg.cc/t4SV5NQg/catto.png)\n", + "\n", + "Finally, functions come with their own special type of comments called a doc string. Here's an example that tells us all about the functions inputs and outputs, including the type of input and output (here a data frame, also known as `pd.DataFrame`).\n", + "\n", + "```python\n", + "def get_numeric_mean(df: pl.DataFrame) -> pl.DataFrame:\n", + " \"\"\"Returns the mean of each numeric column in the DataFrame.\n", + " Args:\n", + " df (pl.DataFrame): Input dataframe\n", + " Returns:\n", + " pl.DataFrame: Dataframe of mean value of each numeric column.\n", + " \"\"\"\n", + " df_mean = df.select(S.numeric()).mean()\n", + " return df_mean\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "4", + "metadata": {}, + "source": [ + "## Line width and line continuation\n", + "\n", + "For quite arbitrary historical reasons, PEP8 also suggests 79 characters for each line of code. Some find this too restrictive, especially with the advent of wider monitors. But it is good to split up very long lines. Anything that is contained in parenthesis can be split into multiple lines like so:\n", + "\n", + "```python\n", + "def function(input_one, input_two,\n", + " input_three, input_four):\n", + " result = (input_one,\n", + " + input_two,\n", + " + input_three,\n", + " + input_four)\n", + " return result\n", + "```\n", + "\n", + "When using _method chaining_ (something you can see in action in [](data-transform)) it's necessary to put the chain inside parentheses and it's good practice to use a new line for every method. The code snippet below gives an example of what good looks like:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5", + "metadata": {}, + "outputs": [], + "source": [ + "import polars as pl\n", + "\n", + "df = pl.DataFrame(\n", + " data={\n", + " \"col0\": [0, 0, 0, 0],\n", + " \"col1\": [0, 0, 0, 0],\n", + " \"col2\": [0, 0, 0, 0],\n", + " \"col3\": [\"a\", \"b\", \"b\", \"a\"],\n", + " \"col4\": [\"alpha\", \"gamma\", \"gamma\", \"gamma\"],\n", + " },\n", + ")\n", + "\n", + "\n", + "# Chaining inside parentheses works\n", + "\n", + "results = df.group_by([\"col3\", \"col4\"]).agg(\n", + " pl.col(\"col1\").count(), pl.col(\"col2\").mean()\n", + ")\n", + "\n", + "results" + ] + }, + { + "cell_type": "markdown", + "id": "6", + "metadata": {}, + "source": [ + "And this is what _not_ to do:\n", + "\n", + "```python\n", + "results = df\n", + " .group_by([\"col3\", \"col4\"]).agg(pl.col(\"col1\").count(), pl.col(\"col2\").mean())\n", + "\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## Principles of Clean Code\n", + "\n", + "While automation can help apply style, it can't help you write _clean code_. Clean code is a set of rules and principles that helps to keep your code readable, maintainable, and extendable. Writing code is easy; writing clean code is hard! However, if you follow these principles, you won't go far wong.\n", + "\n", + "### Do not repeat yourself (DRY)\n", + "\n", + "The DRY principle is 'Every piece of knowledge or logic must have a single, unambiguous representation within a system.' Divide your code into re-usable pieces that you can call when and where you want. Don't write lengthy methods, but divide logic up into clearly differentiated chunks.\n", + "\n", + "This saves having to repeat code, having no idea whether it's this or that version of the same function doing the work, and will help your debugging efforts no end.\n", + "\n", + "Some practical ways to apply DRY in practice are to use functions, to put functions or code that needs to be executed multiple times by multiple different scripts into another script (eg called `utilities.py`) and then import it, and to think carefully if another way of writing your code would be more concise (yet still readable).\n", + "\n", + "::: {.callout-tip title=\"Tip\"}\n", + "If you're using Visual Studio Code, you can [automatically send code into a function](https://code.visualstudio.com/docs/editor/refactoring) by right-clicking on code and using the 'Extract to method' option.\n", + ":::\n", + "\n", + "### KISS (Keep It Simple, Stupid)\n", + "\n", + "Most systems work best if they are kept simple, rather than made complicated. This is a rule that says you should avoid unnecessary complexity. If your code is complex, it will only make it harder for you to understand what you did when you come back to it later.\n", + "\n", + "### SoC (Separation of Concerns) / Make it Modular\n", + "\n", + "Do not have a single file that does everything. If you split your code into separate, independent modules it will be easier to read, debug, test, and use. You can check the basics of coding chapter to see how to create and import functions from other scripts. But even within a script, you can still make your code modular by defining functions that have clear inputs and outputs.\n", + "\n", + "A good rule of thumb is that if a code that achieves one end goes longer than about 30 lines, it should probably go into a function. Scripts longer than about 500 lines are ripe for splitting up too.\n", + "\n", + "Relatedly, do not have a single function that tries to do everything. Functions should have limits too; they should do approximately one thing. If you're naming a function and you have to use 'and' in the name then it's probably worth splitting it into two functions.\n", + "\n", + "Functions should have no 'side effects' either; that is, they should only take in value(s), and output value(s) via a return statement. They shouldn't modify global variables or make other changes.\n", + "\n", + "Another good rule of thumb is that each function shouldn't have lots of separate arguments.\n", + "\n", + "A final tip for modularity and the creation of functions is that you shouldn't use 'flags' in functions (aka boolean conditions). Here's an example:\n", + "\n", + "```python\n", + "# This is bad\n", + "def transform(text, uppercase):\n", + " if uppercase:\n", + " return text.upper()\n", + " else:\n", + " return text.lower()\n", + "\n", + "# This is good\n", + "def uppercase(text):\n", + " return text.upper()\n", + "\n", + "def lowercase(text):\n", + " return text.lower()\n", + "```\n" + ] + } + ], + "metadata": { + "interpreter": { + "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" + }, + "jupytext": { + "cell_metadata_filter": "-all", + "encoding": "# -*- coding: utf-8 -*-", + "formats": "md:myst", + "main_language": "python" + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + }, + "toc-showtags": true + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 630e0677e0bd06e9281e2b4dfc369b53bdeb0254 Mon Sep 17 00:00:00 2001 From: Uchenna Ugoh Date: Thu, 16 Jul 2026 22:28:18 +0100 Subject: [PATCH 09/30] Updated Categorical-data notebook with minor edits --- categorical-data.ipynb | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/categorical-data.ipynb b/categorical-data.ipynb index c391411..123f03d 100644 --- a/categorical-data.ipynb +++ b/categorical-data.ipynb @@ -52,9 +52,9 @@ "- they can be used efficiently with modelling packages, where they will be recognised as potential 'dummy variables', or with plotting packages, which will treat them as discrete values\n", "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\") using pl.Enum\n", "\n", - "All values of categorical data for a **polars** column are either in the given categories or take the value `None`.\n", + "All values of categorical data for a **polars** column are either in the given categories or take the value `null`.\n", "\n", - "Polars has two categorical types: \n", + "Polars has two categorical types:\n", "\n", "- pl.Categorical: Unordered categories (like pandas category)\n", "\n", @@ -135,6 +135,14 @@ "cell_type": "markdown", "id": "cell_009", "metadata": {}, + "source": [ + "`pl.Enum` is preferred when the specific categories are known upfront. `pl.Categorical` infers categories as it reads the dataframe column.\n" + ] + }, + { + "cell_type": "markdown", + "id": "5decf775", + "metadata": {}, "source": [ "We can then enter this into a data frame:\n" ] @@ -155,7 +163,7 @@ "id": "cell_011", "metadata": {}, "source": [ - "Note that `None` appears for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values.\n" + "Note that `null` appears for any value that _isn't_ in the categories we specified—you can find more on this in @sec-missing-values.\n" ] }, { @@ -383,7 +391,7 @@ "id": "cell_033", "metadata": {}, "source": [ - "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example:\n" + "And if your categorical column happens to consist of _elements_ that can undergo operations, those same operations will still work. For example:\n" ] }, { @@ -437,9 +445,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -447,7 +452,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "python4ds (3.12.12.final.0)", "language": "python", "name": "python3" }, @@ -461,7 +466,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.13" + "version": "3.12.12" }, "toc-showtags": true }, From 9fad4304bd9102161b6bcdb2a0c535a966e3c938 Mon Sep 17 00:00:00 2001 From: Your Name Date: Fri, 17 Jul 2026 09:24:51 +0100 Subject: [PATCH 10/30] migrate: convert numbers.ipynb from pandas to polars --- numbers.ipynb | 194 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 130 insertions(+), 64 deletions(-) diff --git a/numbers.ipynb b/numbers.ipynb index 0b96fbb..e73d803 100644 --- a/numbers.ipynb +++ b/numbers.ipynb @@ -13,7 +13,7 @@ "\n", "### Prerequisites\n", "\n", - "This chapter mostly uses functions from **pandas**, which you are likely to already have installed but you can install using `uv add pandas` in the terminal. We'll use real examples from nycflights13, as well as toy examples made with fake data.\n", + "This chapter mostly uses functions from **pandas**, which you are likely to already have installed but you can install using `uv add polars` in the terminal. We'll use real examples from nycflights13, as well as toy examples made with fake data.\n", "\n", "Let's first load up the NYC flights data\n" ] @@ -44,10 +44,11 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights = pd.read_csv(url)" + "\n", + "flights = pl.read_csv(url, use_pyarrow=True)" ] }, { @@ -57,7 +58,7 @@ "source": [ "### Counts\n", "\n", - "It's surprising how much data science you can do with just counts and a little basic arithmetic, so **pandas** strives to make counting as easy as possible with `.count()` and `.value_counts()`. The former just provides a straight count of all the non NA items:" + "It's surprising how much data science you can do with just counts and a little basic arithmetic, so **polars** strives to make counting as easy as possible with `.count()` and `.value_counts()`. The former provides a straight count of all the non-null items:" ] }, { @@ -104,12 +105,14 @@ "outputs": [], "source": [ "(\n", - " flights.groupby([\"dest\"])\n", + " flights.group_by(\"dest\")\n", " .agg(\n", - " mean_delay=(\"dep_delay\", \"mean\"),\n", - " count_flights=(\"dest\", \"count\"),\n", + " [\n", + " pl.col(\"dep_delay\").mean().alias(\"mean_delay\"),\n", + " pl.col(\"dest\").count().alias(\"count_flights\"),\n", + " ]\n", " )\n", - " .sort_values(by=\"count_flights\", ascending=False)\n", + " .sort(\"count_flights\", descending=True)\n", ")" ] }, @@ -128,7 +131,7 @@ "metadata": {}, "outputs": [], "source": [ - "(flights.groupby(\"tailnum\").agg(miles=(\"distance\", \"sum\")))" + "flights.group_by(\"tailnum\").agg(pl.col(\"distance\").sum().alias(\"miles\"))" ] }, { @@ -136,7 +139,7 @@ "id": "1b489765", "metadata": {}, "source": [ - "You can count missing values by combining `sum()` and `isnull()`. In the flights dataset this represents flights that are cancelled. Note that because there isn't a simple string name for applying `.isnull()` followed by `.sum()` (unlike just running `sum()`, which would be given by the string \"sum\"), we need to use a lambda function in the below:" + "You can count missing values by combining sum() and is_null(). In the flights dataset this represents flights that are cancelled:" ] }, { @@ -146,7 +149,7 @@ "metadata": {}, "outputs": [], "source": [ - "(flights.groupby(\"dest\").agg(n_cancelled=(\"dep_time\", lambda x: x.isnull().sum())))" + "flights.group_by(\"dest\").agg(pl.col(\"dep_time\").is_null().sum().alias(\"n_cancelled\"))" ] }, { @@ -160,11 +163,11 @@ "\n", "Basic number arithmatic is achieved by `+` (addition), `-` (subtraction), `*` (multiplication), `/` (division), `**` (powers), `%` (modulo), and `@` (tensor product). Most of these functions don't need a huge amount of explanation because you'll be familiar with them already (and you can look up the others when you do need them).\n", "\n", - "When you have two numeric columns of equal length and you add or subtract them, it's pretty obvious what's going to happen. But we do need to talk about what happens when there is a variable involved that is *not* as long as the column. This is important for operations like `flights.assign(air_time = air_time / 60)` because there are 336,776 numbers on the left of `/` but only one on the right. In this case, **pandas** will understand that you'd like to divide *all* values of air time by 60. This is sometimes called 'broadcasting'. Below is a digram that tries to explain what's going on:\n", + "When you have two numeric columns of equal length and you add or subtract them, it's pretty obvious what's going to happen. But we do need to talk about what happens when there is a variable involved that is *not* as long as the column. This is important for operations like `flights.assign(air_time = air_time / 60)` because there are 336,776 numbers on the left of `/` but only one on the right. In this case, **polars** will understand that you'd like to divide *all* values of air time by 60. This is sometimes called 'broadcasting'. Below is a digram that tries to explain what's going on:\n", "\n", "![](https://numpy.org/doc/stable/_images/broadcasting_1.png)\n", "\n", - "You can find out much more about [broadcasting on the **numpy** documentation](https://numpy.org/doc/stable/user/basics.broadcasting.html). **pandas** is built on top of **numpy** and inherits some of its functionality. \n" + "You can find out much more about [broadcasting on the **numpy** documentation](https://numpy.org/doc/stable/user/basics.broadcasting.html). **polars** is built on top of **numpy** and inherits some of its functionality. \n" ] }, { @@ -172,7 +175,7 @@ "id": "adb10890", "metadata": {}, "source": [ - "When operating on two columns, **pandas** compares their shapes element-wise. Two columns are compatible when they are equal, or one of them is a scalar. If these conditions are not met, you will get an error.\n", + "When operating on two columns, **polars** compares their shapes element-wise. Two columns are compatible when they are equal, or one of them is a scalar. If these conditions are not met, you will get an error.\n", "\n" ] }, @@ -201,7 +204,7 @@ "id": "dd24faa5", "metadata": {}, "source": [ - "Sometimes, you'd like to look at the maximum or minimum value across rows *or* columns. As often is the case with **pandas**, you can specify rows or columns to apply functions to by passing `axis=0` (index) or `axis=1` (columns) to that function. The axis designation can be confusing: remember that you are asking which dimension you wish to aggregate over, leaving you with the other dimension. So if we wish to find the minimum in each row, we aggregate / collapse columns, so we need to pass `axis=1`." + "Sometimes, you'd like to look at the maximum or minimum value across rows or columns. To find the minimum in each row, you can use the horizontal functions:" ] }, { @@ -211,7 +214,7 @@ "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame({\"x\": [1, 5, 7], \"y\": [3, 2, pd.NA]})\n", + "df = pl.DataFrame({\"x\": [1, 5, 7], \"y\": [3, 2, None]})\n", "df" ] }, @@ -230,7 +233,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.min(axis=1)" + "df.select(pl.min_horizontal(\"x\", \"y\"))" ] }, { @@ -273,9 +276,11 @@ "metadata": {}, "outputs": [], "source": [ - "flights.assign(\n", - " hour=lambda x: x[\"sched_dep_time\"] // 100,\n", - " minute=lambda x: x[\"sched_dep_time\"] % 100,\n", + "flights.with_columns(\n", + " [\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"minute\"),\n", + " ]\n", ")" ] }, @@ -300,8 +305,11 @@ "\n", "starting = 100\n", "interest = 1.05\n", - "money = pd.DataFrame(\n", - " {\"year\": 2000 + np.arange(1, 51), \"money\": starting * interest ** np.arange(1, 51)}\n", + "money = pl.DataFrame(\n", + " {\n", + " \"year\": 2000 + np.arange(1, 51),\n", + " \"money\": starting * interest ** np.arange(1, 51),\n", + " }\n", ")\n", "money.head()" ] @@ -321,7 +329,10 @@ "metadata": {}, "outputs": [], "source": [ - "money.plot(x=\"year\", y=\"money\");" + "# Convert to pandas for plotting\n", + "money_pd = money.to_pandas()\n", + "money_pd.plot(x=\"year\", y=\"money\")\n", + "plt.show()" ] }, { @@ -339,7 +350,8 @@ "metadata": {}, "outputs": [], "source": [ - "money.plot(x=\"year\", y=\"money\", logy=True);" + "money_pd.plot(x=\"year\", y=\"money\", logy=True)\n", + "plt.show()" ] }, { @@ -371,7 +383,7 @@ "metadata": {}, "outputs": [], "source": [ - "money.head().round(2)" + "money.select(pl.all().round(2)).head()" ] }, { @@ -379,7 +391,7 @@ "id": "2b26e633", "metadata": {}, "source": [ - "This can also be applied to individual columns or differentially to columns via a dictionary:" + "This can also be applied to individual columns:" ] }, { @@ -389,7 +401,12 @@ "metadata": {}, "outputs": [], "source": [ - "money.head().round({\"year\": 0, \"money\": 1})" + "money.with_columns(\n", + " [\n", + " pl.col(\"year\").round(0),\n", + " pl.col(\"money\").round(1),\n", + " ]\n", + ").head()" ] }, { @@ -407,7 +424,12 @@ "metadata": {}, "outputs": [], "source": [ - "money.tail().round({\"year\": 0, \"money\": -2})" + "money.with_columns(\n", + " [\n", + " pl.col(\"year\").round(),\n", + " ((pl.col(\"money\") / 100).round() * 100).alias(\"money\"),\n", + " ]\n", + ").head()" ] }, { @@ -415,7 +437,7 @@ "id": "6544b94e", "metadata": {}, "source": [ - "Sometimes, you'll want to round according to significant figures rather than decimal places. There's not a really easy way to do this but you can define a custom function to do it. Here's an example of rounding to 2 significant figures (change the `2` in the below to round to a different number of significant figures):" + "Sometimes, you'll want to round according to significant figures rather than decimal places. There's not a really easy way to do this but you can define a custom function to do it. Here's an example of rounding to 2 significant figures (change the `2` in the code block below to round to a different number of significant figures):" ] }, { @@ -425,7 +447,9 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].head().apply(lambda x: float(f'{float(f\"{x:.2g}\"):g}'))" + "money.with_columns(\n", + " pl.col(\"money\").map_elements(lambda x: float(f'{float(f\"{x:.2g}\"):g}'))\n", + ")" ] }, { @@ -490,7 +514,7 @@ "id": "c22b0eda", "metadata": {}, "source": [ - "Remember that you can always apply **numpy** functions to **pandas** data frame columns like so:" + "Remember that you can always apply numpy functions to **polars** data frame columns using `map_elements()`:" ] }, { @@ -500,7 +524,7 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].head().apply(np.ceil)" + "money.with_columns(pl.col(\"money\").map_elements(np.ceil)).head()" ] }, { @@ -510,7 +534,7 @@ "source": [ "### Cumulative and Rolling Aggregates\n", "\n", - "**pandas** has several cumulative functions, including `.cumsum()`, `.cummax()` and `.cummin()`, and `.cumprod()`. " + "**polars** has several cumulative functions, including `.cumsum()`, `.cummax()` and `.cummin()`, and `.cumprod()`. " ] }, { @@ -520,15 +544,24 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].tail().cumsum()" - ] - }, - { - "cell_type": "markdown", - "id": "c4b569cb", - "metadata": {}, - "source": [ - "As ever, this can be applied across rows too by passing `axis=1`." + "import numpy as np\n", + "import polars as pl\n", + "\n", + "# Create the money DataFrame\n", + "starting = 100\n", + "interest = 1.05\n", + "money = pl.DataFrame(\n", + " {\n", + " \"year\": 2000 + np.arange(1, 51),\n", + " \"money\": starting * interest ** np.arange(1, 51),\n", + " }\n", + ")\n", + "\n", + "# Show the data\n", + "money.head()\n", + "\n", + "# Cumulative sum\n", + "money[\"money\"].cum_sum().tail()" ] }, { @@ -542,7 +575,7 @@ "\n", "### Ranking\n", "\n", - "**pandas**' rank function is `.rank()`. Let's look back at the data we made earlier and rank it:" + "**polars**' rank function is `.rank()`. Let's look back at the data we made earlier and rank it:" ] }, { @@ -562,7 +595,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.rank()" + "df.select(pl.all().rank())" ] }, { @@ -570,7 +603,7 @@ "id": "85345760", "metadata": {}, "source": [ - "Of course, there's no change here because the items were already ranked! We can also do a pct rank by passing the keyword argument `pct=True`." + "We can also do a percent rank by passing the keyword argument `method=\"average\"` and then scaling:" ] }, { @@ -580,7 +613,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.rank(pct=True)" + "df.select(pl.all().rank(method=\"average\") / df.height)" ] }, { @@ -602,9 +635,15 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money_lag_5\"] = money[\"money\"].shift(5)\n", - "money[\"money_lead_10\"] = money[\"money\"].shift(-10)\n", - "money.set_index(\"year\").plot();" + "money_with_shifts = money.with_columns(\n", + " [\n", + " pl.col(\"money\").shift(5).alias(\"money_lag_5\"),\n", + " pl.col(\"money\").shift(-10).alias(\"money_lead_10\"),\n", + " ]\n", + ")\n", + "money_with_shifts_pd = money_with_shifts.to_pandas()\n", + "money_with_shifts_pd.set_index(\"year\").plot()\n", + "plt.show()" ] }, { @@ -636,7 +675,7 @@ "source": [ "## More Useful Summary Statistics\n", "\n", - "We've seen how useful `.mean()`, `.count()`, and `.value_counts()` can be for analysis. **pandas** has a great many more built-in summary statistics functions, however. These include `.median()` (you may find it interesting to compare the mean vs the median when looking at the hourly departure delay in the flights data), `.mode()`, `.min()`, and `.max()`.\n", + "We've seen how useful `.mean()`, `.count()`, and `.value_counts()` can be for analysis. **polars** has a great many more built-in summary statistics functions, however. These include `.median()`, `.mode()`, `.min()`, and `.max()`.\n", "\n", "A class of useful summary statistics are provided by the `.quantile` function, which is the same as `median` for `.quantile(0.5)`. The quantile at x% is the value that x% of values are below. (Note that under this definition, `.quantile(1)` will be the same as `.max()`.) Let's see an example with the 25th percentile." ] @@ -656,7 +695,7 @@ "id": "02c212f4", "metadata": {}, "source": [ - "Sometimes you don't just want one percentile, but a bunch of them. **pandas** makes this very easy by allowing you to pass a list of quantiles:" + "Sometimes you don't just want one percentile, but a bunch of them. **polars** makes this very easy by allowing you to pass a list of quantiles:" ] }, { @@ -666,7 +705,30 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].quantile([0, 0.25, 0.5, 0.75])" + "# Create a struct with quantile values\n", + "money.select(\n", + " [\n", + " pl.struct(\n", + " [\n", + " pl.lit(0).alias(\"0%\"),\n", + " pl.lit(0.25).alias(\"25%\"),\n", + " pl.lit(0.5).alias(\"50%\"),\n", + " pl.lit(0.75).alias(\"75%\"),\n", + " ]\n", + " ).alias(\"quantiles\")\n", + " ]\n", + ")\n", + "\n", + "# Create a DataFrame with the quantile results\n", + "quantile_df = money.select(\n", + " [\n", + " pl.lit(money[\"money\"].quantile(0)).alias(\"0%\"),\n", + " pl.lit(money[\"money\"].quantile(0.25)).alias(\"25%\"),\n", + " pl.lit(money[\"money\"].quantile(0.5)).alias(\"50%\"),\n", + " pl.lit(money[\"money\"].quantile(0.75)).alias(\"75%\"),\n", + " ]\n", + ")\n", + "quantile_df" ] }, { @@ -690,12 +752,16 @@ "outputs": [], "source": [ "(\n", - " flights.groupby([\"origin\", \"dest\"])\n", + " flights.group_by([\"origin\", \"dest\"])\n", " .agg(\n", - " distance_sd=(\"distance\", lambda x: x.quantile(0.75) - x.quantile(0.25)),\n", - " count=(\"distance\", \"count\"),\n", + " [\n", + " (\n", + " pl.col(\"distance\").quantile(0.75) - pl.col(\"distance\").quantile(0.25)\n", + " ).alias(\"distance_sd\"),\n", + " pl.col(\"distance\").count().alias(\"count\"),\n", + " ]\n", " )\n", - " .query(\"distance_sd > 0\")\n", + " .filter(pl.col(\"distance_sd\") > 0)\n", ")" ] }, @@ -718,7 +784,9 @@ "metadata": {}, "outputs": [], "source": [ - "flights[\"dep_delay\"].plot.hist(bins=50, title=\" Distribution: length of delay\");" + "flights_pd = flights.to_pandas()\n", + "flights_pd[\"dep_delay\"].plot.hist(bins=50, title=\"Distribution: length of delay\")\n", + "plt.show()" ] }, { @@ -728,9 +796,10 @@ "metadata": {}, "outputs": [], "source": [ - "flights.query(\"dep_delay <= 120\")[\"dep_delay\"].plot.hist(\n", - " bins=50, title=\" Distribution: length of delay\"\n", - ");" + "flights_pd.query(\"dep_delay <= 120\")[\"dep_delay\"].plot.hist(\n", + " bins=50, title=\"Distribution: length of delay\"\n", + ")\n", + "plt.show()" ] }, { @@ -768,9 +837,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -778,7 +844,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 930ed44b8ba80c2a8b794aabdbd532d6a3bfe1f8 Mon Sep 17 00:00:00 2001 From: Your Name Date: Fri, 17 Jul 2026 12:42:16 +0100 Subject: [PATCH 11/30] migrate: convert missing-values.ipynb from pandas to polars --- missing-values.ipynb | 181 ++++++++++++++++++++++++++----------------- 1 file changed, 110 insertions(+), 71 deletions(-) diff --git a/missing-values.ipynb b/missing-values.ipynb index 67e95ef..37d46e8 100644 --- a/missing-values.ipynb +++ b/missing-values.ipynb @@ -9,7 +9,7 @@ "\n", "## Introduction\n", "\n", - "In this chapter, we'll look at the tools and tricks for dealing with missing values. We'll start by discussing some general tools for working with missing values recorded as `NA`s. We'll then explore the idea of implicitly missing values, values are that are simply absent from your data, and show some tools you can use to make them explicit.\n", + "In this chapter, we'll look at the tools and tricks for dealing with missing values. We'll start by discussing some general tools for working with missing values recorded as `null` values. We'll then explore the idea of implicitly missing values, values are that are simply absent from your data, and show some tools you can use to make them explicit.\n", "We'll finish off with a related discussion of empty groups, caused by categories that don't appear in the data." ] }, @@ -40,7 +40,7 @@ "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package." ] }, { @@ -50,13 +50,11 @@ "source": [ "## Explicit Missing Values\n", "\n", - "To begin, let's explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see an `NA` or `nan`.\n", + "To begin, let's explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see an `null` value.\n", "\n", "### Types of Missing Values\n", "\n", - "One thing to note about missing values in **pandas** is that they are not all created alike!\n", - "\n", - "For example, real numbers in **pandas** (such as the `float64` dtype) uses a 'nan' (aka, Not a Number):" + "In polars, missing values are represented by `null`:" ] }, { @@ -67,9 +65,9 @@ "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame([5, 27.3, np.nan, -16], columns=[\"numbers\"])\n", + "df = pl.DataFrame({\"numbers\": [5.0, 27.3, None, -16.0]})\n", "df" ] }, @@ -78,7 +76,7 @@ "id": "4b1c9ea7", "metadata": {}, "source": [ - "But this isn't the only way! We can also use Python's built-in `None` value (which, here, gets converted to a NaN because the valid values are all floating point numbers) and **pandas**' `pd.NA`:" + "Polars also handles Python's built-in None values seamlessly:" ] }, { @@ -88,7 +86,7 @@ "metadata": {}, "outputs": [], "source": [ - "numbers = pd.DataFrame([pd.NA, 27.3, np.nan, -16, None], columns=[\"numbers\"])\n", + "numbers = pl.DataFrame({\"numbers\": [None, 27.3, np.nan, -16.0, None]})\n", "numbers" ] }, @@ -97,7 +95,7 @@ "id": "b2dda456", "metadata": {}, "source": [ - "However, with the object data type (the default for strings), the types can co-exist:" + "For string columns, missing values are also represented by `null`:" ] }, { @@ -107,9 +105,7 @@ "metadata": {}, "outputs": [], "source": [ - "fruits = pd.DataFrame(\n", - " [\"orange\", np.nan, \"apple\", None, \"banana\", pd.NA], columns=[\"fruit\"]\n", - ")\n", + "fruits = pl.DataFrame({\"fruit\": [\"orange\", None, \"apple\", None, \"banana\", None]})\n", "fruits" ] }, @@ -118,7 +114,7 @@ "id": "8b5a2849", "metadata": {}, "source": [ - "Both of these types of missing value can be found using **pandas** `.isna()` function. This returns a new column of boolean values that are `True` if the value is any kind of missing value." + "Both types of missing value can be found using the `.is_null()` method, which returns a new column of boolean values that are `True` if the value is missing:" ] }, { @@ -128,7 +124,7 @@ "metadata": {}, "outputs": [], "source": [ - "fruits.isna()" + "fruits.select(pl.col(\"fruit\").is_null())" ] }, { @@ -136,7 +132,17 @@ "id": "0fd6764d", "metadata": {}, "source": [ - "As a convenience, there is also a `notna()` function." + "As a convenience, there is also an .is_not_null() method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3fb8082", + "metadata": {}, + "outputs": [], + "source": [ + "fruits.select(pl.col(\"fruit\").is_not_null())" ] }, { @@ -146,7 +152,7 @@ "source": [ "### Dealing with Explicit Missing Values\n", "\n", - "There are various options for dealing with missing values. The `fillna()` function achieves this. Let's take a look at it with some test data:" + "There are various options for dealing with missing values. The `fill_null()` method achieves this. Let's take a look at it with some test data:" ] }, { @@ -156,17 +162,15 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df = pd.DataFrame(\n", - " [\n", - " [np.nan, 2, None, 0],\n", - " [3, 4, np.nan, 1],\n", - " [5, np.nan, np.nan, pd.NA],\n", - " [np.nan, 3, np.nan, 4],\n", - " ],\n", - " columns=list(\"ABCD\"),\n", + "null_df = pl.DataFrame(\n", + " {\n", + " \"A\": [None, 3, 5, None],\n", + " \"B\": [2, 4, None, 3],\n", + " \"C\": [None, None, None, None],\n", + " \"D\": [0, 1, None, 4],\n", + " }\n", ")\n", - "\n", - "nan_df" + "null_df" ] }, { @@ -184,7 +188,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(0)" + "null_df.fill_null(0)" ] }, { @@ -192,7 +196,7 @@ "id": "af8e2cb7", "metadata": {}, "source": [ - "This can be done on a by-column basis; here we replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1, 2, and 3 respectively." + "This can be done on a by-column basis by passing a dictionary:" ] }, { @@ -202,7 +206,14 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(value={\"A\": 0, \"B\": 1, \"C\": 2, \"D\": 3})" + "null_df.with_columns(\n", + " [\n", + " pl.col(\"A\").fill_null(0),\n", + " pl.col(\"B\").fill_null(1),\n", + " pl.col(\"C\").fill_null(2),\n", + " pl.col(\"D\").fill_null(3),\n", + " ]\n", + ")" ] }, { @@ -210,7 +221,7 @@ "id": "2544fe5a", "metadata": {}, "source": [ - "We can also propagate non-null values forward or backward (relative to the index)" + "We can also propagate non-null values forward or backward:" ] }, { @@ -220,7 +231,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(method=\"ffill\")" + "null_df.fill_null(strategy=\"forward\")" ] }, { @@ -230,7 +241,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(method=\"bfill\")" + "null_df.fill_null(strategy=\"backward\")" ] }, { @@ -246,7 +257,7 @@ "id": "ca7e4ca9", "metadata": {}, "source": [ - "Another feature of all of these functions is that you can limit the number of NaNs that get replaced using the `limit=` keyword argument." + "Another feature of all of these functions is that you can limit the number of nulls that get replaced using the `limit=` keyword argument." ] }, { @@ -256,7 +267,8 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(value={\"A\": 0, \"B\": 1, \"C\": 2, \"D\": 3}, limit=1)" + "# Forward fill with limit of 1\n", + "null_df.fill_null(strategy=\"forward\", limit=2)" ] }, { @@ -264,7 +276,7 @@ "id": "99b94a01", "metadata": {}, "source": [ - "Of course, another option might be just to filter out the missing values altogether. There are a couple of ways to do this depending if you want to remove entire rows (`axis=0`) or columns (`axis=1`—but in this case, as there is at least one NaN in each column though, there will be no data left!)" + "Of course, another option might be just to filter out the missing values altogether. The `.drop_nulls()` method removes rows with any missing values:" ] }, { @@ -274,7 +286,15 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df[\"A\"].dropna(axis=0) # on a single column" + "null_df.drop_nulls()" + ] + }, + { + "cell_type": "markdown", + "id": "710bcb2c", + "metadata": {}, + "source": [ + "You can also drop rows where all values are null using a custom filter:" ] }, { @@ -284,25 +304,34 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.dropna(axis=1)" + "null_exp_df = pl.DataFrame(\n", + " {\n", + " \"A\": [None, 3, 5, None, None],\n", + " \"B\": [2, 4, None, 3, None],\n", + " \"C\": [None, None, None, None, None],\n", + " \"D\": [0, 1, None, 4, None],\n", + " }\n", + ")\n", + "\n", + "null_exp_df" ] }, { - "cell_type": "markdown", - "id": "9bc57652", + "cell_type": "code", + "execution_count": null, + "id": "f7b38107", "metadata": {}, + "outputs": [], "source": [ - "`dropna()` takes some keyword arguments too; for example `how=\"all\"` only drops a column or row if *all* of the values are NA." + "null_exp_df.filter(~pl.all_horizontal(pl.all().is_null()))" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "3296ea35", + "cell_type": "markdown", + "id": "2b058955", "metadata": {}, - "outputs": [], "source": [ - "nan_df.dropna(how=\"all\")" + "Notice that the only the last all null column was removed." ] }, { @@ -318,7 +347,7 @@ "id": "095f53a5", "metadata": {}, "source": [ - "Another way to filter out nans is to use the same filtering methods you would use normally, via boolean columns, in combination with the `.notna()` function. In the below example, we see all columns for the rows for which A is not NA." + "Another way to filter out nulls is to use the same filtering methods you would use normally, via boolean columns, in combination with the `.is_not_null()` method. In the below example, we see the rows for which column A is not null:" ] }, { @@ -328,7 +357,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df[nan_df[\"A\"].notna()]" + "null_df.filter(pl.col(\"A\").is_not_null())" ] }, { @@ -340,7 +369,7 @@ "\n", "Sometimes you'll hit the opposite problem where some concrete value actually represents a missing value. This typically arises in data generated by older software that doesn't have a proper way to represent missing values, so it must instead use some special value like 99 or -999.\n", "\n", - "If possible, handle this when reading in the data, for example, by using the `na_values=` keyword argument when calling `pd.read_csv()`. If you discover the problem later, or your data source doesn't provide a way to handle it on reading the file, you can use a range of options to replace the given data:" + "If possible, handle this when reading in the data, for example, by using the `null_values=` keyword argument when calling `pl.read_csv()`. If you discover the problem later, or your data source doesn't provide a way to handle it on reading the file, you can use a range of options to replace the given data:" ] }, { @@ -350,8 +379,13 @@ "metadata": {}, "outputs": [], "source": [ - "stata_df = pd.DataFrame([[3, 4, 5], [-7, 4, -99], [-99, 6, 5]], columns=list(\"ABC\"))\n", - "\n", + "stata_df = pl.DataFrame(\n", + " {\n", + " \"A\": [3, -7, -99],\n", + " \"B\": [4, 4, 6],\n", + " \"C\": [5, -99, 5],\n", + " }\n", + ")\n", "stata_df" ] }, @@ -360,7 +394,7 @@ "id": "81685807", "metadata": {}, "source": [ - "The easiest option is probably `.replace()`:" + "The easiest option is probably the expression-level `replace()` method:" ] }, { @@ -370,7 +404,7 @@ "metadata": {}, "outputs": [], "source": [ - "stata_df.replace({-99: pd.NA})" + "stata_df.with_columns(pl.all().replace(-99, None))" ] }, { @@ -378,7 +412,7 @@ "id": "9d4d3d3d", "metadata": {}, "source": [ - "Because `.replace()` accepts a dictionary, it's possible to replace several values at once:" + "Because `replace()` accepts a dictionary, it's possible to replace several values at once:" ] }, { @@ -388,7 +422,7 @@ "metadata": {}, "outputs": [], "source": [ - "stata_df.replace({-99: pd.NA, -7: pd.NA})" + "stata_df.with_columns(pl.all().replace({-99: None, -7: None}))" ] }, { @@ -418,11 +452,11 @@ "metadata": {}, "outputs": [], "source": [ - "stocks = pd.DataFrame(\n", + "stocks = pl.DataFrame(\n", " {\n", " \"year\": [2020, 2020, 2020, 2020, 2021, 2021, 2021],\n", " \"qtr\": [1, 2, 3, 4, 2, 3, 4],\n", - " \"price\": [1.88, 0.59, 0.35, np.nan, 0.92, 0.17, 2.66],\n", + " \"price\": [1.88, 0.59, 0.35, None, 0.92, 0.17, 2.66],\n", " }\n", ")\n", "stocks" @@ -467,7 +501,7 @@ "metadata": {}, "outputs": [], "source": [ - "stocks.pivot(columns=\"qtr\", values=\"price\", index=\"year\")" + "stocks.pivot(index=\"year\", on=\"qtr\", values=\"price\", aggregate_function=\"first\")" ] }, { @@ -497,14 +531,16 @@ "metadata": {}, "outputs": [], "source": [ - "health = pd.DataFrame(\n", + "health = pl.DataFrame(\n", " {\n", " \"name\": [\"Ikaia\", \"Oletta\", \"Leriah\", \"Dashay\", \"Tresaun\"],\n", " \"smoker\": [\"no\", \"no\", \"previously\", \"no\", \"yes\"],\n", " \"age\": [34, 88, 75, 47, 56],\n", " }\n", ")\n", - "health[\"smoker\"] = health[\"smoker\"].astype(\"category\")" + "health = health.with_columns(\n", + " pl.col(\"smoker\").cast(pl.Enum([\"no\", \"previously\", \"yes\"]))\n", + ")" ] }, { @@ -522,7 +558,7 @@ "metadata": {}, "outputs": [], "source": [ - "health_cut = health.iloc[:-1, :]\n", + "health_cut = health[:-1]\n", "health_cut" ] }, @@ -531,7 +567,7 @@ "id": "7ae95b5f", "metadata": {}, "source": [ - "The value 'yes' for smoker now doesn't (seem to) appear anywhere in our data frame. But if we run `value_counts()` to get a count of the number of each type of category, you'll see that the data frame 'remembers' that there's a 'yes' category that isn't currently present:" + "The value 'yes' for smoker now doesn't (seem to) appear anywhere in our data frame. Because we used an `Enum` type, the column still remembers that 'yes' is a valid category. We can inspect the categories of the column's datatype:" ] }, { @@ -541,7 +577,7 @@ "metadata": {}, "outputs": [], "source": [ - "health_cut[\"smoker\"].value_counts()" + "health_cut[\"smoker\"].dtype.categories" ] }, { @@ -549,7 +585,7 @@ "id": "bde48398", "metadata": {}, "source": [ - "You'll see the same thing happen with a `groupby()` operation:" + "If we perform aggregation operations, Polars will only return results for groups present in the data by default. If we need to include all categories (even those with no observations), we can join the categories list back to the aggregated result:" ] }, { @@ -559,7 +595,13 @@ "metadata": {}, "outputs": [], "source": [ - "health_cut.groupby(\"smoker\")[\"age\"].mean()" + "all_smokers = pl.DataFrame({\"smoker\": health_cut[\"smoker\"].dtype.categories})\n", + "means = (\n", + " health_cut.group_by(\"smoker\")\n", + " .agg(pl.col(\"age\").mean())\n", + " .with_columns(pl.col(\"smoker\").cast(pl.Utf8))\n", + ")\n", + "all_smokers.join(means, on=\"smoker\", how=\"left\")" ] }, { @@ -567,14 +609,11 @@ "id": "5a2af19e", "metadata": {}, "source": [ - "You can see here that, because we took the mean of a number that doesn't exist, we got a NaN in place of a real value for the yes row (but there is a 'yes' row)." + "You can see here that, because we took the mean of a number that doesn't exist, we got a null in place of a real value for the yes row (but there is a 'yes' row)." ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -582,7 +621,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 5eecfaed8cfa6470dfcdd0366b8bb7354b858ca3 Mon Sep 17 00:00:00 2001 From: Your Name Date: Fri, 17 Jul 2026 16:03:37 +0100 Subject: [PATCH 12/30] migrate: convert strings.ipynb from pandas to polars --- strings.ipynb | 108 ++++++++++++++++++++++---------------------------- 1 file changed, 47 insertions(+), 61 deletions(-) diff --git a/strings.ipynb b/strings.ipynb index b1a00fe..bab55e9 100644 --- a/strings.ipynb +++ b/strings.ipynb @@ -768,7 +768,7 @@ "source": [ "We've seen how to work with individual strings. But often we want to work with a group of strings, otherwise known as a corpus, that is a collection of texts. It could be a collection of words, sentences, paragraphs, or some domain-based grouping (eg job descriptions). Just like any other Python object, you can put strings into a list (or other iterable).\n", "\n", - "And, fortunately, many of the methods that we have seen deployed on a single string can be straightforwardly scaled up to hundreds, thousands, or millions of strings using **pandas** or other tools. This scaling up is achieved via *vectorisation*, in analogy with going from a single value (a scalar) to multiple values in a list (a vector).\n", + "And, fortunately, many of the methods that we have seen deployed on a single string can be straightforwardly scaled up to hundreds, thousands, or millions of strings using **polars** or other tools. This scaling up is achieved via *vectorisation*, in analogy with going from a single value (a scalar) to multiple values in a list (a vector).\n", "\n", "As a very minimal example, here is capitalisation of names vectorised using a list comprehension:\n" ] @@ -788,7 +788,7 @@ "id": "730df2c8", "metadata": {}, "source": [ - "A **pandas** series can be used in place of a list. Let's create the series first:" + "A **polars** series can be used in place of a list. Let's create the series first:" ] }, { @@ -798,11 +798,10 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "dfs = pd.Series(\n", + "dfs = pl.Series(\n", " [\"ada lovelace\", \"adam smith\", \"elinor ostrom\", \"grace hopper\", \"jean bartik\"],\n", - " dtype=\"string\",\n", ")\n", "dfs" ] @@ -812,7 +811,7 @@ "id": "222fbb51", "metadata": {}, "source": [ - "Now we use the syntax series.str.function to change the text series:\n" + "Now we use the syntax `.str.function` to change the text series:\n" ] }, { @@ -822,7 +821,7 @@ "metadata": {}, "outputs": [], "source": [ - "dfs.str.title()" + "dfs.str.to_titlecase()" ] }, { @@ -840,8 +839,8 @@ "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(dfs, columns=[\"names\"])\n", - "df[\"names\"].str.title()" + "df = pl.DataFrame({\"names\": dfs})\n", + "df.with_columns(pl.col(\"names\").str.to_titlecase())" ] }, { @@ -849,42 +848,35 @@ "id": "2e430dd9", "metadata": {}, "source": [ - "The table below shows a non-exhaustive list of the string methods that are available in **pandas**.\n", + "The table below shows a non-exhaustive list of the string methods that are available in **polars**.\n", "\n", "| Function (preceded by `.str.`) | What it does |\n", "|-----------------------------|-------------------------|\n", "| `len()` | Length of string. |\n", - "| `lower()` | Put string in lower case. |\n", - "| `upper()` | Put string in upper case. |\n", - "| `capitalize()` | Put string in leading upper case. |\n", - "| `swapcase()` | Swap cases in a string. |\n", - "| `translate()` | Returns a copy of the string in which each character has been mapped through a given translation table. |\n", - "| `ljust()` | Left pad a string (default is to pad with spaces) |\n", - "| `rjust()` | Right pad a string (default is to pad with spaces) |\n", - "| `center()` | Pad such that string appears in centre (default is to pad with spaces) |\n", - "| `zfill()` | Pad with zeros |\n", + "| `to_lower()` | Put string in lower case. |\n", + "| `to_upper()` | Put string in upper case. |\n", + "| `to_titlecase()` | Put string in leading upper case. |\n", "| `strip()` | Strip out leading and trailing whitespace |\n", "| `rstrip()` | Strip out trailing whitespace |\n", "| `lstrip()` | Strip out leading whitespace |\n", "| `find()` | Return the lowest index in the data where a substring appears |\n", "| `split()` | Split the string using a passed substring as the delimiter |\n", - "| `isupper()` | Check whether string is upper case |\n", - "| `isdigit()` | Check whether string is composed of digits |\n", - "| `islower()` | Check whether string is lower case |\n", "| `startswith()` | Check whether string starts with a given sub-string |\n", + "| `ends_with()` | Check whether string ends with a given sub-string |\n", + "| `contains()` | Check whether string contains a given pattern |\n", + "| `replace()` | Replace occurrences of pattern with another string |\n", + "| `replace_all()` | Replace all occurrences of pattern |\n", + "| `slice()` | Extract a substring by position |\n", "\n", - "Regular expressions can also be scaled up with **pandas**. The below table shows vectorised regular expressions.\n", + "Regular expressions can also be scaled up with **polars**. The below table shows vectorised regular expressions.\n", "\n", "| Function | What it does |\n", "|-|----------------------------------|\n", - "| `match()` | Call `re.match()` on each element, returning a boolean. |\n", - "| `extract()` | Call `re.match()` on each element, returning matched groups as strings. |\n", - "| `findall()` | Call `re.findall()` on each element |\n", + "| `contains()` | Check if pattern exists, returning a boolean |\n", "| `replace()` | Replace occurrences of pattern with some other string |\n", - "| `contains()` | Call `re.search()` on each element, returning a boolean |\n", - "| `count()` | Count occurrences of pattern |\n", - "| `split()` | Equivalent to `str.split()`, but accepts regexes |\n", - "| `rsplit()` | Equivalent to `str.rsplit()`, but accepts regexes |\n", + "| `replace_all()` | Replace all occurrences of pattern |\n", + "| `extract()` | Extract matched groups as strings |\n", + "| `extract_all()` | Extract all matched groups |\n", "\n", "\n", "Let's see a couple of these in action. First, splitting on a given sub-string:" @@ -897,7 +889,7 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.split(\" \")" + "df.with_columns(pl.col(\"names\").str.split(\" \"))" ] }, { @@ -905,7 +897,7 @@ "id": "597c0d23", "metadata": {}, "source": [ - "It's fairly common that you want to split out strings and save the results to new columns in your data frame. You can specify a (max) number of splits via the `n=` kwarg and you can get the columns using `expand`" + "It's fairly common that you want to split out strings and save the results to new columns in your data frame. You can get the columns using `list` and `to_struct()`:" ] }, { @@ -915,7 +907,9 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.split(\" \", n=2, expand=True)" + "df.with_columns(\n", + " pl.col(\"names\").str.split(\" \").list.to_struct(fields=[\"first\", \"last\"])\n", + ").unnest(\"names\")" ] }, { @@ -927,7 +921,7 @@ "Using vectorised operations, create a new column with the index position where the first vowel occurs for each row in the `names` column.\n", ":::\n", "\n", - "Here's an example of using a regex function with **pandas**:" + "Here's an example of using a regex function with **polars**:" ] }, { @@ -937,7 +931,7 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.extract(\"(\\w+)\", expand=False)" + "df.with_columns(pl.col(\"names\").str.extract(r\"(\\w+)\", 1))" ] }, { @@ -949,19 +943,13 @@ "\n", "| Method | Description |\n", "|-|-|\n", - "| `get()` | Index each element |\n", "| `slice()` | Slice each element |\n", - "| `slice_replace()` | Replace slice in each element with passed value |\n", - "| `cat()` | Concatenate strings |\n", + "| `concat()` | Concatenate strings |\n", "| `repeat()` | Repeat values |\n", - "| `normalize()` | Return Unicode form of string |\n", - "| `pad()` | Add whitespace to left, right, or both sides of strings |\n", - "| `wrap()` | Split long strings into lines with length less than a given width |\n", - "| `join()` | Join strings in each element of the Series with passed separator |\n", - "| `get_dummies()` | extract dummy variables as a data frame |\n", + "| `contains()` | Check if pattern exists |\n", "\n", "\n", - "The `get()` and `slice()` methods give access to elements of the lists returned by `split()`. Here's an example that combines `split()` and `get()`:\n", + "The `slice()` method gives access to substrings by position:\n", "\n" ] }, @@ -972,7 +960,7 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.split().str.get(-1)" + "df.with_columns(pl.col(\"names\").str.slice(0, 5))" ] }, { @@ -980,7 +968,7 @@ "id": "4ce15edb", "metadata": {}, "source": [ - "If we have a column with tags split by a symbol, we can use the `get_dummies()` function to split it out. For example, let's create a data frame with a single column that mixes subject and nationality tags:\n" + "If we have a column with tags split by a symbol, we can use the `str.split()` to split it out. For example, let's create a data frame with a single column that mixes subject and nationality tags:\n" ] }, { @@ -990,7 +978,7 @@ "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", " \"names\": [\n", " \"ada lovelace\",\n", @@ -1010,7 +998,8 @@ "id": "f44b0d84", "metadata": {}, "source": [ - "If we now use `str.get_dummies()` and split on `;` we can get a data frame of dummies." + "If we now use `str.split()` and then list operations we can get separate columns:\n", + "\n" ] }, { @@ -1020,7 +1009,9 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"tags\"].str.get_dummies(\";\")" + "df.with_columns(\n", + " pl.col(\"tags\").str.split(\"; \").list.to_struct(fields=[\"country\", \"subject\"])\n", + ").unnest(\"tags\")" ] }, { @@ -1040,34 +1031,29 @@ " text_of_book = f.read()\n", "```\n", "\n", - "You can also read a text file directly into a **pandas** data frame using \n", + "You can also read a text file directly into a **polars** data frame using \n", "\n", "```python\n", - "df = pd.read_csv('book.txt', delimiter = \"\\n\")\n", + "df = pl.read_csv('book.txt', separator=\"\\n\", has_header=False)\n", "```\n", "\n", "In the above, the delimiter for different rows of the data frame is set as \"\\n\", which means new line, but you could use whatever delimiter you prefer.\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", - "Download the file 'smith_won.txt' using this [link](https://github.com/aeturrell/coding-for-economists/blob/main/data/smith_won.txt) (use right-click and save as). Then read the text in using **pandas**.\n", + "Download the file 'smith_won.txt' using this [link](https://github.com/aeturrell/coding-for-economists/blob/main/data/smith_won.txt) (use right-click and save as). Then read the text in using **polars**.\n", ":::\n", "\n", "### CSV file\n", "\n", - "CSV files are already split into rows. By far the easiest way to read in csv files is using **pandas**,\n", + "CSV files are already split into rows. By far the easiest way to read in csv files is using **polars**,\n", "\n", "```python\n", - "df = pd.read_csv('book.csv')\n", - "```\n", - "\n", - "Remember that **pandas** can read many other file types too." + "df = pl.read_csv('book.csv')\n", + "```" ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -1075,7 +1061,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From c4b0a4c0c75e004d67a7b8b70c69eff3291ad982 Mon Sep 17 00:00:00 2001 From: Your Name Date: Sat, 18 Jul 2026 02:03:12 +0100 Subject: [PATCH 13/30] migrate: convert dates-and-times.ipynb from pandas to polars --- dates-and-times.ipynb | 454 ++++++++++++++++++------------------------ 1 file changed, 189 insertions(+), 265 deletions(-) diff --git a/dates-and-times.ipynb b/dates-and-times.ipynb index 47b53ad..bc612f6 100644 --- a/dates-and-times.ipynb +++ b/dates-and-times.ipynb @@ -57,7 +57,7 @@ "uv run pip install --pre seaborn\n", "```\n", "\n", - "We will also be using the **pandas** package and numerical package **numpy**." + "We will also be using the **polars** package and numerical package **numpy**." ] }, { @@ -69,7 +69,7 @@ "\n", "A point in time as represented in data science is composed of a clock time and a date. These two elements are brought together as a *datetime*.\n", "\n", - "The datetime object is the fundamental time object in Python. It’s useful to know about these before moving on to datetime operations using **pandas** (which you’re far more likely to use in practice). Python's *datetime* objects capture the year, month, day, hour, second, and microsecond. Let’s import the class that deals with datetimes (whose objects are of type datetime.datetime) and take a look at it." + "The datetime object is the fundamental time object in Python. It’s useful to know about these before moving on to datetime operations using **polars** (which you’re far more likely to use in practice). Python's *datetime* objects capture the year, month, day, hour, second, and microsecond. Let’s import the class that deals with datetimes (whose objects are of type datetime.datetime) and take a look at it." ] }, { @@ -415,7 +415,7 @@ "id": "c0a974aa", "metadata": {}, "source": [ - "One word of warning with **numpy** and datetimes though: the more precise you go, and you can go down to femtoseconds ($10^{-15}$ seconds), the more precise you go the smaller the range of dates you can hit. A popular choice of precision is `datetime64[ns]`, which can encode times from 1678 AD to 2262 AD. Working with seconds gets you 2.9$\\times 10^9$ BC to 2.9$\\times 10^9$ AD." + "One word of warning with **numpy** and datetimes though: the more precise you go, and you can go down to femtoseconds ($10^{-15}$ seconds), the smaller the range of dates you can hit. A popular choice of precision is `datetime64[ns]`, which can encode times from 1678 AD to 2262 AD. Working with seconds gets you 2.9$\\times 10^9$ BC to 2.9$\\times 10^9$ AD." ] }, { @@ -431,7 +431,7 @@ "id": "82a4a3f3", "metadata": {}, "source": [ - "[**pandas**](https://pandas.pydata.org/) is the workhorse of time series analysis in Python. The basic object is a *timestamp*. The `pd.to_datetime()` function creates timestamps from strings that could reasonably represent datetimes. Let's see an example of using `pd.to_datetime()` to create a timestamp and then take a look at it." + "[**polars**](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html) is the workhorse of time series analysis in Python. The basic object is a *Datetime* type. Let's create a datetime from a string: " ] }, { @@ -441,9 +441,9 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "date = pd.to_datetime(\"16th of February, 2020\")\n", + "date = pl.Series([\"2020-02-16\"]).str.strptime(pl.Datetime, \"%Y-%m-%d\")\n", "date" ] }, @@ -452,7 +452,7 @@ "id": "8b083619", "metadata": {}, "source": [ - "This is of type `Timestamp` and you can see that it has many of the same properties as the built-in Python `datetime.datetime` class from the previous chapter. As with that, the default setting for `tz` (timezone) and `tzinfo` is `None`. There are some extra properties, though, such as `freq` for frequency, which will be very useful when it comes to manipulating time *series* as opposed to just one or two datetimes." + "This creates a column of type `Datetime`. As with Python's datetime, the default timezone is `None`." ] }, { @@ -462,9 +462,9 @@ "source": [ "### Creating and Using Time Series\n", "\n", - "There are two main scenarios in which you might be creating time series using **pandas**: i) creating one from scratch or ii) reading in data from a file. Let's look at a few ways to do i) first. \n", + "There are two main scenarios in which you might be creating time series using **polars**: i) creating one from scratch or ii) reading in data from a file. Let's look at a few ways to do i) first. \n", "\n", - "You can create a time series with **pandas** by taking a date as created above and extending it using **pandas** timedelta function:" + "You can create a time series with **polars** using `pl.datetime_range()`:" ] }, { @@ -474,7 +474,7 @@ "metadata": {}, "outputs": [], "source": [ - "date + pd.to_timedelta(np.arange(12), \"D\")" + "pl.datetime_range(start=\"2018-01-01\", end=\"2018-01-08\", interval=\"1d\")" ] }, { @@ -482,9 +482,7 @@ "id": "3ba587ff", "metadata": {}, "source": [ - "This has created a datetime index of type `datetime65[ns]` (remember, an index is a special type of **pandas** column), where \"ns\" stands for nano-second resolution.\n", - "\n", - "Another method is to create a range of dates (pass a frequency using the `freq=` keyword argument):" + "Another method is to specify the number of periods and the frequency:" ] }, { @@ -494,7 +492,14 @@ "metadata": {}, "outputs": [], "source": [ - "pd.date_range(start=\"2018/1/1\", end=\"2018/1/8\")" + "from datetime import datetime\n", + "\n", + "pl.datetime_range(\n", + " start=datetime(2018, 1, 1, 0, 0),\n", + " end=datetime(2018, 1, 1, 2, 0),\n", + " interval=\"1h\",\n", + " eager=True,\n", + ")" ] }, { @@ -502,7 +507,7 @@ "id": "3f6c8a4f", "metadata": {}, "source": [ - "Another way to create ranges is to specify the number of periods and the frequency:" + "Polars also supports timezone operations:" ] }, { @@ -512,26 +517,16 @@ "metadata": {}, "outputs": [], "source": [ - "pd.date_range(\"2018-01-01\", periods=3, freq=\"h\")" - ] - }, - { - "cell_type": "markdown", - "id": "ce8c06d8", - "metadata": {}, - "source": [ - "Following the discussion of the previous chapter on timezones, you can also localise timezones directly in **pandas** data frames:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "6703682c", - "metadata": {}, - "outputs": [], - "source": [ - "dti = pd.date_range(\"2018-01-01\", periods=3, freq=\"h\").tz_localize(\"UTC\")\n", - "dti.tz_convert(\"US/Pacific\")" + "from datetime import datetime\n", + "\n", + "dti = pl.datetime_range(\n", + " start=datetime(2018, 1, 1, 0, 0),\n", + " end=datetime(2018, 1, 1, 2, 0),\n", + " interval=\"1h\",\n", + " eager=True,\n", + ").dt.replace_time_zone(\"UTC\")\n", + "\n", + "dti.dt.convert_time_zone(\"US/Pacific\")" ] }, { @@ -539,7 +534,7 @@ "id": "4d3e7668", "metadata": {}, "source": [ - "Now let's see how to turn data that has been read in with a non-datetime type into a vector of datetimes. This happens *all the time* in practice. We'll read in some data on job vacancies for information and communication jobs, ONS code UNEM-JP9P, and then try to wrangle the given \"date\" column into a **pandas** datetime column." + "Now let's see how to turn data that has been read in with a non-datetime type into a vector of datetimes. This happens *all the time* in practice. We'll read in some data on job vacancies for information and communication jobs, ONS code UNEM-JP9P, and then try to wrangle the given \"date\" column into a **polars** datetime column." ] }, { @@ -553,12 +548,15 @@ "\n", "url = \"https://api.beta.ons.gov.uk/v1/data?uri=/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/timeseries/jp9z/lms/previous/v108\"\n", "\n", - "# Get the data from the ONS API:\n", + "# Get the data from the ONS API\n", "json_data = requests.get(url).json()\n", - "df = pd.DataFrame(pd.json_normalize(json_data[\"months\"]))\n", - "df[\"value\"] = pd.to_numeric(df[\"value\"])\n", - "df = df[[\"date\", \"value\"]]\n", - "df = df.rename(columns={\"value\": \"Vacancies (ICT), thousands\"})\n", + "df = pl.DataFrame(json_data[\"months\"])\n", + "df = df.select(\n", + " [\n", + " pl.col(\"date\"),\n", + " pl.col(\"value\").cast(pl.Float64).alias(\"Vacancies (ICT), thousands\"),\n", + " ]\n", + ")\n", "df.head()" ] }, @@ -567,7 +565,7 @@ "id": "1be10bf8", "metadata": {}, "source": [ - "We have the data in. Let's look at the column types that arrived." + "We have the data in. Let's look at the column types:" ] }, { @@ -577,7 +575,17 @@ "metadata": {}, "outputs": [], "source": [ - "df.info()" + "df.schema" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "924ce7e4", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"date\"].head(10)" ] }, { @@ -585,7 +593,7 @@ "id": "71bbbb02", "metadata": {}, "source": [ - "This is the default 'object' type, but we want the date column to have `datetime64[ns]`, which is a datetime type. Again, we use `pd.to_datetime()`:" + "We want the date column to have `Datetime` type. We can use `str.strptime()`:" ] }, { @@ -595,7 +603,10 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"date\"] = pd.to_datetime(df[\"date\"])\n", + "df = df.with_columns(\n", + " pl.col(\"date\").str.to_titlecase().str.strptime(pl.Date, format=\"%Y %b\")\n", + ")\n", + "\n", "df[\"date\"].head()" ] }, @@ -604,7 +615,7 @@ "id": "ca0b1727", "metadata": {}, "source": [ - "In this case, the conversion from the format of data that was put in of \"2001 MAY\" to datetime worked out-of-the-box. `pd.to_datetime` will always take an educated guess as to the format, but it won't always work out.\n", + "In this case, the conversion from the format of data that was put in of \"2001 MAY\" to datetime worked out-of-the-box.\n", "\n", "What happens if we have a more tricky-to-read-in datetime column? This frequently occurs in practice so it's well worth exploring an example. Let's create some random data with dates in an unusual format with month first, then year, then day, eg \"1, '19, 29\" and so on." ] @@ -616,7 +627,7 @@ "metadata": {}, "outputs": [], "source": [ - "small_df = pd.DataFrame({\"date\": [\"1, '19, 22\", \"1, '19, 23\"], \"values\": [\"1\", \"2\"]})\n", + "small_df = pl.DataFrame({\"date\": [\"1, '19, 22\", \"1, '19, 23\"], \"values\": [\"1\", \"2\"]})\n", "small_df[\"date\"]" ] }, @@ -625,7 +636,7 @@ "id": "b1e6f80c", "metadata": {}, "source": [ - "Now, if we were to run this via `pd.to_datetime` with no further input, it would misinterpret, for example, the first date as `2022-01-19`. So we must pass a bit more info to `pd.to_datetime` to help it out. We can pass a `format=` keyword argument with the format that the datetime takes. Here, we'll use `%m` for month in number format, `%y` for year in 2-digit format, and `%d` for 2-digit day. We can also add in the other characters such as `'` and `,`. You can find a list of datetime format identifiers above or over at [https://strftime.org/](https://strftime.org/)." + "Now, if we were to run this via `str.strptime()` with no further input, it would misinterpret the format. So we must pass a `format=` keyword argument with the format that the datetime takes. Here, we'll use `%m` for month in number format, `%y` for year in 2-digit format, and `%d` for 2-digit day. We can also add in the other characters such as `'` and `,`." ] }, { @@ -635,7 +646,7 @@ "metadata": {}, "outputs": [], "source": [ - "pd.to_datetime(small_df[\"date\"], format=\"%m, '%y, %d\")" + "small_df.with_columns(pl.col(\"date\").str.strptime(pl.Datetime, format=\"%m, '%y, %d\"))" ] }, { @@ -645,7 +656,7 @@ "source": [ "### Datetime Offsets\n", "\n", - "Our data, currently held in `df`, were read in as if they were from the *start* of the month but these data refer to the month that has passed and so should be for the *end* of the month. Fortunately, we can change this using a time offset." + "Our data, currently held in `df`, were read in as if they were from the *start* of the month but these data refer to the month that has passed and so should be for the *end* of the month. Fortunately, we can change this using date offset operations. In polars, you can add durations:" ] }, { @@ -655,18 +666,10 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"date\"] = df[\"date\"] + pd.offsets.MonthEnd()\n", + "df = df.with_columns((pl.col(\"date\") + pl.duration(days=1)).dt.offset_by(\"1mo\"))\n", "df.head()" ] }, - { - "cell_type": "markdown", - "id": "61ef45b4", - "metadata": {}, - "source": [ - "While we used the `MonthEnd` offset here, there are many different offsets available. You can find a [full table of date offsets here](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects)." - ] - }, { "cell_type": "markdown", "id": "5afd54be", @@ -674,7 +677,7 @@ "source": [ "### The `.dt` accessor\n", "\n", - "When you have a datetime column, you can use the `.dt` accessor to grab lots of useful information from it such as the `minute`, `month`, and so on. Some that are functions, rather than just accessors of underlying properties, are followed by brackets, `()`, because they are functions. Here are a few useful examples:" + "When you have a datetime column, you can use the `.dt` accessor to grab lots of useful information from it such as the `minute`, `month`, and so on. Here are a few useful examples:" ] }, { @@ -684,12 +687,12 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"Using `dt.day_name()`\")\n", - "print(df[\"date\"].dt.day_name().head())\n", - "print(\"Using `dt.isocalendar()`\")\n", - "print(df[\"date\"].dt.isocalendar().head())\n", - "print(\"Using `dt.month`\")\n", - "print(df[\"date\"].dt.month.head())" + "print(\"Using `dt.to_string()`\")\n", + "print(df[\"date\"].dt.to_string(\"%A\").head())\n", + "print(\"Using `dt.weekday()`\")\n", + "print(df[\"date\"].dt.weekday().head())\n", + "print(\"Using `dt.month()`\")\n", + "print(df[\"date\"].dt.month().head())" ] }, { @@ -699,7 +702,7 @@ "source": [ "### Creating a datetime Index and Setting the Frequency\n", "\n", - "For the subsequent parts, we'll set the datetime column to be the index of the data frame. *This is the standard setup you will likely want to use when dealing with time series.*" + "For time series operations in polars, you typically don't set an index like in pandas. Instead, you keep the datetime as a column and use it in group operations. However, you can sort by the datetime column:" ] }, { @@ -709,7 +712,7 @@ "metadata": {}, "outputs": [], "source": [ - "df = df.set_index(\"date\")\n", + "df = df.sort(\"date\")\n", "df.head()" ] }, @@ -718,7 +721,7 @@ "id": "8d5017ed", "metadata": {}, "source": [ - "Now, if we look at the first few entries of the index of data frame (a datetime index) using `head` as above, we'll see that the `freq=` parameter is set as `None`." + "Polars supports resampling operations using `group_by_dynamic()`:" ] }, { @@ -728,58 +731,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.index[:5]" - ] - }, - { - "cell_type": "markdown", - "id": "3bd0fe54", - "metadata": {}, - "source": [ - "This can be set for the whole data frame using the `asfreq()` function:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9146c99d", - "metadata": {}, - "outputs": [], - "source": [ - "df = df.asfreq(\"M\")\n", - "df.index[:5]" - ] - }, - { - "cell_type": "markdown", - "id": "8f024684", - "metadata": {}, - "source": [ - "Although most of the time it doesn't matter about the fact that `freq=None`, some aggregation operations need to know the frequency of the time series in order to work and it's good practice to set it if your data *are* regular. You can use `asfreq` to go from a higher frequency to a lower frequency too: the last entry from the higher frequency that aligns with the lower frequency will be taken, for example in going from months to years, December's value would be used.\n", - "\n", - "Note that trying to set the frequency when your datetime index doesn't match up to a particular frequency will cause errors or problems.\n", - "\n", - "A few useful frequencies to know about are in the table below; all of these can be used with `pd.to_datetime()` too.\n", - "\n", - "| Code | Represents |\n", - "|-------|---------------------------------------------------------------------|\n", - "| D | Calendar day |\n", - "| W | Weekly |\n", - "| M | Month end |\n", - "| Q | Quarter end |\n", - "| A | Year end |\n", - "| H | Hours |\n", - "| T | Minutes |\n", - "| S | Seconds |\n", - "| B | Business day |\n", - "| BM | Business month end |\n", - "| BQ | Business quarter end |\n", - "| BA | Business year end |\n", - "| BH | Business hours |\n", - "| MS | Month start |\n", - "| QS | Quarter start |\n", - "| W-SUN | Weeks beginning with Sunday (similar for other days) |\n", - "| 2M | Every 2 months (works with other combinations of numbers and codes) |" + "# Resample to annual frequency with mean\n", + "df_resampled = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", + " pl.col(\"Vacancies (ICT), thousands\").mean()\n", + ")\n", + "df_resampled" ] }, { @@ -789,7 +745,7 @@ "source": [ "## Making Quick Time Series Plots\n", "\n", - "Having managed to put your time series into a data frame, perhaps converting a column of type string into a colume of type datetime in the process, you often just want to see the thing! We can achieve this using the `plot()` command, as long as we have a datetime index.\n" + "Having managed to put your time series into a data frame, perhaps converting a column of type string into a column of type datetime in the process, you often just want to see the thing! We can achieve this by converting to pandas for plotting:\n" ] }, { @@ -799,7 +755,10 @@ "metadata": {}, "outputs": [], "source": [ - "df.plot();" + "# Convert to pandas for plotting\n", + "df_pd = df.to_pandas()\n", + "df_pd.set_index(\"date\").plot()\n", + "plt.show()" ] }, { @@ -813,7 +772,7 @@ "\n", "### Resampling\n", "\n", - "Quite frequently, there is a situation in which one would like to change the frequency of a given time series. A time index-based data frame makes this easy via the `resample()` function. `resample()` must be told *how* you'd like to resample the data, for example via the mean or median. Here's an example resampling the monthly data to annual and taking the mean:" + "Quite frequently, there is a situation in which one would like to change the frequency of a given time series. Polars makes this easy via the `group_by_dynamic()` function. Here's an example resampling the monthly data to annual and taking the mean:" ] }, { @@ -823,25 +782,10 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample(\"A\").mean()" - ] - }, - { - "cell_type": "markdown", - "id": "d320a934", - "metadata": {}, - "source": [ - "As resample is just a special type of aggregation, it can work with all of the usual functions that aggregations do, including in-built functions or user-defined functions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fbbbcdff", - "metadata": {}, - "outputs": [], - "source": [ - "df.resample(\"5YE\").agg([\"mean\", \"std\"]).head()" + "df.resample = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", + " pl.col(\"Vacancies (ICT), thousands\").mean()\n", + ")\n", + "df.resample" ] }, { @@ -849,7 +793,7 @@ "id": "c3b45c36", "metadata": {}, "source": [ - "Resampling can go up in frequency (up-sampling) as well as down, but we no longer need to choose an aggregation function, we must now choose how we'd like to fill in the gaps for the frequencies we didn't have in the original data. In the example below, they are just left as NaNs." + "Resampling can go up in frequency (up-sampling) as well as down, but we no longer need to choose an aggregation function, we must now choose how we'd like to fill in the gaps for the frequencies we didn't have in the original data." ] }, { @@ -859,7 +803,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample(\"D\").asfreq()" + "# Upsample to daily frequency\n", + "daily = df.group_by_dynamic(\"date\", every=\"1d\").agg(\n", + " pl.col(\"Vacancies (ICT), thousands\").first()\n", + ")\n", + "daily" ] }, { @@ -867,7 +815,7 @@ "id": "c08e1dad", "metadata": {}, "source": [ - "Options to fill in missing time series data include using `bfill` or `ffill` to fill in the blanks based on the next or last available value, respectively, or `interpolate()` (note how only the first 3 NaNs are replaced using the `limit` keyword argument):" + "Options to fill in missing time series data include using forward or backward fill:" ] }, { @@ -877,7 +825,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample(\"D\").interpolate(method=\"linear\", limit_direction=\"forward\", limit=3)[:6]" + "# Forward fill missing values\n", + "daily_filled = daily.with_columns(\n", + " pl.col(\"Vacancies (ICT), thousands\").fill_null(strategy=\"forward\")\n", + ")\n", + "daily_filled.head()" ] }, { @@ -895,22 +847,25 @@ "metadata": {}, "outputs": [], "source": [ - "# Get stock market data\n", "import yfinance as yf\n", "\n", - "xf = yf.download(\"AAPL\", start=\"2017-01-01\", end=\"2019-06-01\")\n", - "xf = xf.sort_index()\n", - "xf.columns = xf.columns.droplevel(1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ab1efae3", - "metadata": {}, - "outputs": [], - "source": [ - "plt.rcParams[\"axes.prop_cycle\"]" + "xf = yf.download(\n", + " \"AAPL\",\n", + " start=\"2017-01-01\",\n", + " end=\"2019-06-01\",\n", + " multi_level_index=False,\n", + ")\n", + "\n", + "xf = xf.reset_index()\n", + "\n", + "xf_pl = pl.from_pandas(xf).rename(\n", + " {\n", + " \"Date\": \"date\",\n", + " \"Close\": \"close\",\n", + " }\n", + ")\n", + "\n", + "xf_pl.head()" ] }, { @@ -922,127 +877,89 @@ "source": [ "from itertools import cycle\n", "\n", - "fig, ax = plt.subplots()\n", - "data = xf.iloc[:10, 3]\n", - "colour_wheel = cycle(plt.rcParams[\"axes.prop_cycle\"])\n", - "\n", - "data.asfreq(\"D\").plot(ax=ax, marker=\"o\", linestyle=\"None\", zorder=3)\n", - "data.asfreq(\"D\", method=\"bfill\").plot(\n", - " ax=ax, style=\"-.o\", lw=1, color=next(colour_wheel)[\"color\"]\n", - ")\n", - "data.asfreq(\"D\", method=\"ffill\").plot(\n", - " ax=ax, style=\"--o\", lw=1, color=next(colour_wheel)[\"color\"]\n", - ")\n", - "ax.set_ylabel(\"Close ($)\")\n", - "ax.legend([\"original\", \"back-fill\", \"forward-fill\"]);" - ] - }, - { - "cell_type": "markdown", - "id": "235f6b1c", - "metadata": {}, - "source": [ - "### Rolling Window Functions\n", + "import matplotlib.dates as mdates\n", + "import matplotlib.pyplot as plt\n", "\n", - "The `rolling()` and `ewm()` methods are both rolling window functions. The first includes functions of the sequence\n", + "# Create a daily series\n", + "data = xf_pl.select([\"date\", \"close\"]).sort(\"date\").head(10)\n", "\n", - "$$\n", - "y_t = f(\\{x_{t-i} \\}_{i=0}^{i=R-1})\n", - "$$\n", + "# Original observations\n", + "original = data\n", "\n", - "where $R$ is the number of periods to use for the rolling window. For example, if the function is the mean, then $f$ takes the form $\\frac{1}{R}\\displaystyle\\sum_{i=0}^{i=R-1} x_{t-i}$.\n", + "# Daily frequency with missing dates\n", + "daily = data.upsample(time_column=\"date\", every=\"1d\")\n", "\n", - "The example below is a 2-period rolling mean:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1ddc4fb2", - "metadata": {}, - "outputs": [], - "source": [ - "df.rolling(2).mean()" - ] - }, - { - "cell_type": "markdown", - "id": "488f3c7c", - "metadata": {}, - "source": [ - "The `ewm()` includes the class of functions where data point $x_{t-i}$ has a weight $w_i = (1-\\alpha)^i$. As $0 < \\alpha < 1$, points further back in time are given less weight. For example, an exponentially moving average is given by\n", + "# Back-fill\n", + "bfill = daily.fill_null(strategy=\"backward\")\n", "\n", - "$$\n", - "y_t = \\frac{x_t + (1 - \\alpha)x_{t-1} + (1 - \\alpha)^2 x_{t-2} + ...\n", - "+ (1 - \\alpha)^t x_{0}}{1 + (1 - \\alpha) + (1 - \\alpha)^2 + ...\n", - "+ (1 - \\alpha)^t}\n", - "$$\n", + "# Forward-fill\n", + "ffill = daily.fill_null(strategy=\"forward\")\n", "\n", - "The example below shows the code for the exponentially weighted moving average:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "0ea9c8ce", - "metadata": {}, - "outputs": [], - "source": [ - "df.ewm(alpha=0.2).mean()" - ] - }, - { - "cell_type": "markdown", - "id": "8d9eb44c", - "metadata": {}, - "source": [ - "Let's see these methods together on the stock market data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "0af7b5e6", - "metadata": {}, - "outputs": [], - "source": [ "fig, ax = plt.subplots()\n", - "roll_num = 28\n", - "alpha = 0.03\n", - "xf[\"Close\"].plot(label=\"Raw\", alpha=0.5)\n", - "xf[\"Close\"].expanding().mean().plot(label=\"Expanding Average\", style=\":\")\n", - "xf[\"Close\"].ewm(alpha=alpha).mean().plot(\n", - " label=f\"EWMA ($\\\\alpha=${alpha:.2f})\", style=\"--\"\n", + "\n", + "colour_wheel = cycle(plt.rcParams[\"axes.prop_cycle\"])\n", + "\n", + "# Original observations\n", + "ax.plot(\n", + " original[\"date\"],\n", + " original[\"close\"],\n", + " \"o\",\n", + " linestyle=\"None\",\n", + " label=\"original\",\n", + " zorder=3,\n", ")\n", - "xf[\"Close\"].rolling(roll_num).mean().plot(label=f\"{roll_num}D MA\", style=\"-.\")\n", + "\n", + "# Back-fill\n", + "ax.plot(\n", + " bfill[\"date\"],\n", + " bfill[\"close\"],\n", + " \"-.o\",\n", + " color=next(colour_wheel)[\"color\"],\n", + " linewidth=1,\n", + " label=\"back-fill\",\n", + ")\n", + "\n", + "# Forward-fill\n", + "ax.plot(\n", + " ffill[\"date\"],\n", + " ffill[\"close\"],\n", + " \"--o\",\n", + " color=next(colour_wheel)[\"color\"],\n", + " linewidth=1,\n", + " label=\"forward-fill\",\n", + ")\n", + "\n", + "ax.set_ylabel(\"Close ($)\")\n", "ax.legend()\n", - "ax.set_ylabel(\"Close ($)\");" + "\n", + "# Show only the day of the month on the x-axis\n", + "ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%b %d\"))\n", + "fig.autofmt_xdate(rotation=0)\n", + "\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "6f795baa", + "id": "235f6b1c", "metadata": {}, "source": [ - "For more tools to analyse stocks, see the [**Pandas TA**](https://twopirllc.github.io/pandas-ta/) package.\n", + "### Rolling Window Functions\n", "\n", - "We can also use `rolling()` as an intermediate step in creating more than one type of aggregation:" + "Polars supports rolling window functions with .rolling():" ] }, { "cell_type": "code", "execution_count": null, - "id": "134199ae", + "id": "1ddc4fb2", "metadata": {}, "outputs": [], "source": [ - "roll = xf[\"Close\"].rolling(50, center=True)\n", - "\n", - "fig, ax = plt.subplots()\n", - "m = roll.agg([\"mean\", \"std\"])\n", - "m[\"mean\"].plot(ax=ax)\n", - "ax.fill_between(m.index, m[\"mean\"] - m[\"std\"], m[\"mean\"] + m[\"std\"], alpha=0.2)\n", - "ax.set_ylabel(\"Close ($)\");" + "# 2-period rolling mean\n", + "df.with_columns(\n", + " pl.col(\"Vacancies (ICT), thousands\").rolling_mean(window_size=2).alias(\"rolling_2\")\n", + ")" ] }, { @@ -1052,7 +969,7 @@ "source": [ "### Shifting\n", "\n", - "Shifting can move series around in time; it's what we need to create leads and lags of time series. Let's create a lead and a lag in the data. Remember that a lead is going to shift the pattern in the data to the left (ie earlier in time), while the lag is going to shift patterns later in time (ie to the right)." + "Shifting can move series around in time; it's what we need to create leads and lags of time series. Let's create a lead and a lag in the data. Remember that a lead is going to shift the pattern in the data to the left (i.e. earlier in time), while the lag is going to shift patterns later in time (i.e. to the right)." ] }, { @@ -1064,10 +981,14 @@ "source": [ "lead = 12\n", "lag = 3\n", - "orig_series_name = df.columns[0]\n", - "df[f\"lead ({lead} months)\"] = df[orig_series_name].shift(-lead)\n", - "df[f\"lag ({lag} months)\"] = df[orig_series_name].shift(lag)\n", - "df.head()" + "orig_series = \"Vacancies (ICT), thousands\"\n", + "df_shifted = df.with_columns(\n", + " [\n", + " pl.col(orig_series).shift(-lead).alias(f\"lead ({lead} months)\"),\n", + " pl.col(orig_series).shift(lag).alias(f\"lag ({lag} months)\"),\n", + " ]\n", + ")\n", + "df_shifted.head()" ] }, { @@ -1077,7 +998,10 @@ "metadata": {}, "outputs": [], "source": [ - "df.iloc[100:300, :].plot();" + "# Plot the shifted series\n", + "df_shifted_pd = df_shifted.to_pandas()\n", + "df_shifted_pd.set_index(\"date\").iloc[100:300].plot()\n", + "plt.show()" ] } ], @@ -1089,7 +1013,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 1e752fb0f68adf437d4b472ed574c9ebee1ab821 Mon Sep 17 00:00:00 2001 From: Your Name Date: Sat, 18 Jul 2026 11:45:12 +0100 Subject: [PATCH 14/30] migrate: convert boolean-data.ipynb from pandas to polars --- boolean-data.ipynb | 55 +++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/boolean-data.ipynb b/boolean-data.ipynb index d389d25..0bd94c2 100644 --- a/boolean-data.ipynb +++ b/boolean-data.ipynb @@ -481,11 +481,11 @@ "id": "8e06930b", "metadata": {}, "source": [ - "## Booleans in **pandas** data frames\n", + "## Booleans in **polars** DataFrames\n", "\n", "### Operations on booleans in data frames\n", "\n", - "Quite often, you will run into a scenario where you're working with data that have True or False values in a data frame. It is easy to create a column of booleans in a **pandas** data frame:" + "Quite often, you will run into a scenario where you're working with data that have True or False values in a data frame. It is easy to create a column of booleans in a **polars** data frame:" ] }, { @@ -495,9 +495,9 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame.from_dict(\n", + "df = pl.DataFrame(\n", " {\n", " \"bool_col_1\": [False] * 3 + [True, True],\n", " \"bool_col_2\": [True, False, True, False, True],\n", @@ -521,7 +521,7 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"bool_col_1\"] | df[\"bool_col_2\"]" + "df.select((pl.col(\"bool_col_1\") | pl.col(\"bool_col_2\")).alias(\"result\"))" ] }, { @@ -529,7 +529,7 @@ "id": "25dede69", "metadata": {}, "source": [ - "Quite often, it's useful to have a count of the number of true values. If you take the sum of boolean columns in a **pandas** data frame, it will tot up the number of `True` values:" + "Quite often, it's useful to have a count of the number of true values. If you take the sum of boolean columns in a **polars** data frame, it will tot up the number of `True` values:" ] }, { @@ -539,7 +539,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.sum()" + "df.select(pl.col(\"*\").sum())" ] }, { @@ -557,8 +557,8 @@ "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame.from_dict({\"bool_col\": [0, 1, 0, 1, 1]})\n", - "df[\"bool_col\"].astype(bool)" + "df = pl.DataFrame({\"bool_col\": [0, 1, 0, 1, 1]})\n", + "df.with_columns(pl.col(\"bool_col\").cast(pl.Boolean))" ] }, { @@ -579,7 +579,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", + "diamonds = pl.read_csv(\n", " \"https://github.com/mwaskom/seaborn-data/raw/master/diamonds.csv\"\n", ")\n", "diamonds.head()" @@ -596,12 +596,16 @@ { "cell_type": "code", "execution_count": null, - "id": "7a27f0a0", + "id": "6b739dc5", "metadata": {}, "outputs": [], "source": [ - "diamonds[\"expensive\"] = diamonds[\"price\"] > 1000\n", - "diamonds.sample(10)" + "(\n", + " diamonds.with_columns((pl.col(\"price\") > 1000).alias(\"expensive\"))\n", + " # tweaked to display a mix of true and false values.\n", + " .select([\"cut\", \"carat\", \"color\", \"clarity\", \"price\", \"expensive\"])\n", + " .sample(10, shuffle=True)\n", + ")" ] }, { @@ -609,7 +613,7 @@ "id": "8383cab5", "metadata": {}, "source": [ - "Of course, this could also have been achieved in a call to assign:" + "Of course, this could also have been achieved with `select()` or `with_columns()`:" ] }, { @@ -619,7 +623,10 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds.assign(expensive=lambda x: x[\"price\"] > 1000).head()" + "diamonds.select(\n", + " pl.all(),\n", + " (pl.col(\"price\") > 1000).alias(\"expensive\"),\n", + ").head()" ] }, { @@ -627,7 +634,7 @@ "id": "622ff486", "metadata": {}, "source": [ - "Another use of booleans that is quite useful when it comes to data frames is the `.isin()` function. For example, if you just want some True or False values for whether a set of columns is in a data frame:" + "Another use of booleans that is quite useful when it comes to data frames is checking if a set of columns exists:" ] }, { @@ -637,7 +644,8 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds.columns.isin([\"x\", \"y\", \"z\"])" + "cols_to_check = [\"x\", \"y\", \"z\"]\n", + "[col in diamonds.columns for col in cols_to_check]" ] }, { @@ -647,7 +655,7 @@ "source": [ "### any() and all() in data frames\n", "\n", - "A **pandas** column of booleans behaves a lot like a list of booleans, and we can apply the same logic to it via **pandas** built-in `.any()` and `.all()` methods. We expect some entries for `\"expensive\"` to be true, so `any()` should return true:" + "A **polars** column of booleans behaves a lot like a list of booleans, and we can apply the same logic to it via **polars** built-in `.any()` and `.all()` methods. We expect some entries for `\"expensive\"` to be true, so `any()` should return true:" ] }, { @@ -657,7 +665,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds[\"expensive\"].any()" + "diamonds.select((pl.col(\"price\") > 1000).any().alias(\"any_expensive\"))" ] }, { @@ -677,7 +685,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds[diamonds[\"x\"] > diamonds[\"y\"]]" + "diamonds.filter(pl.col(\"x\") > pl.col(\"y\"))" ] }, { @@ -685,14 +693,11 @@ "id": "9b5906ba", "metadata": {}, "source": [ - "The expression `diamonds[\"x\"] > diamonds[\"y\"]` creates a column of booleans that is used to filter to just the rows where the condition is true." + "The expression `pl.col(\"x\") > pl.col(\"y\")` creates a column of booleans that is used to filter to just the rows where the condition is true." ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -700,7 +705,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From db40f45f7bcc8be8b3532df9ab987192e6f02890 Mon Sep 17 00:00:00 2001 From: Your Name Date: Sat, 18 Jul 2026 14:34:03 +0100 Subject: [PATCH 15/30] refactor: migrate data analysis examples and documentation from pandas to polars --- prerequisites.ipynb | 6 ++---- quarto.qmd | 13 +++++++------ workflow-help.qmd | 13 ++++++------- workflow-help.quarto_ipynb_1 | 13 ++++++------- 4 files changed, 21 insertions(+), 24 deletions(-) diff --git a/prerequisites.ipynb b/prerequisites.ipynb index 07119a8..7d7a058 100644 --- a/prerequisites.ipynb +++ b/prerequisites.ipynb @@ -44,7 +44,7 @@ "\n", "### Packages\n", "\n", - "A Python **package** is a collection of functions, data, and documentation that extends the capabilities of an installed version of Python. Using packages is key to most data science because most of the functionality we'll need comes from extra packages. You'll see statetments like `import numpy as np` at the start of many Python code scripts—these are instructions to use an installed package (here one called `numpy`) and to give it a shortened name (`np`, for convenience) in the rest of the script. The functions in the `numpy` package are then accessed through syntax like `np.`; for example, you can take logs with `np.log(x)` where `x` is a variable containing a number. You need only install packages once.\n", + "A Python **package** is a collection of functions, data, and documentation that extends the capabilities of an installed version of Python. Using packages is key to most data science because most of the functionality we'll need comes from extra packages. You'll see statetments like `import polars as pl` at the start of many Python code scripts—these are instructions to use an installed package (here one called `polars`) and to give it a shortened name (`pl`, for convenience) in the rest of the script. The functions in the `polars` package are then accessed through syntax like `pl.`; for example, you can take logs with `pl.read_csv('file.csv')`. You need only install packages once.\n", "\n", "### Typical workflow\n", "\n", @@ -75,8 +75,6 @@ "\n", "### Installing Python\n", "\n", - "### Installing Python\n", - "\n", "To download and install Python, we'll use the uv \"distribution\" of Python, which is available on all major operating systems. To install it, follow the instructions at [this website](https://docs.astral.sh/uv/getting-started/installation/#installation-methods). Unlike installing normal programmes, we're going to use the *command line* to install Python. Linux, Mac, and Windows all have built-in command lines: search for *Terminal* on Mac or Linux, and *Powershell* on Windows. These apps will bring up boxes that you can type commands in. As of the time of writing, the commands are:\n", "\n", "```bash\n", @@ -257,7 +255,7 @@ "uv add packagename\n", "```\n", "\n", - "and hit return. In the above, `packagename` might be `pandas`, for example. If you have problems installing, make sure that you are connected to the internet, and that [PyPI](https://pypi.org/) (the Python package index) isn't blocked by your firewall or proxy. You could try installing the data analysispackage **polars** this way, by running `uv add polars`. We'll see how to use **polars** in later chapters, but if you want to know if it installed correctly, just look for the message saying \"Successfully installed polars\" followed by the version number.\n", + "and hit return. In the above, `packagename` might be `polars`, for example. If you have problems installing, make sure that you are connected to the internet, and that [PyPI](https://pypi.org/) (the Python package index) isn't blocked by your firewall or proxy. You could try installing the data analysispackage **polars** this way, by running `uv add polars`. We'll see how to use **polars** in later chapters, but if you want to know if it installed correctly, just look for the message saying \"Successfully installed polars\" followed by the version number.\n", "\n", "You can see what packages you have previously installed by entering `uv pip list` into the command line.\n", "\n", diff --git a/quarto.qmd b/quarto.qmd index 7d2561d..87f01f0 100644 --- a/quarto.qmd +++ b/quarto.qmd @@ -144,11 +144,12 @@ For an example of a code output where the input is not shown, the code below wil ```` ```{{python}} #| echo: false -import pandas as pd +import polars as pl import seaborn as sns -df = sns.load_dataset("penguins") -pd.crosstab(df["species"], [df["island"], df["sex"]], margins=True) +df = pl.from_pandas(sns.load_dataset("penguins")) +ct = df.group_by(["species", "island", "sex"]).agg(pl.len().alias("count")) +ct.pivot(index="species", columns=["island", "sex"], values="count", aggregate_function="sum") ``` ```` @@ -186,17 +187,17 @@ For an example of a code output where the input is not shown, the code below wil ```{{python}} #| echo: false from IPython.display import display, Markdown -import pandas as pd +import polars as pl import seaborn as sns -df = sns.load_dataset("penguins") +df = pl.from_pandas(sns.load_dataset("penguins")) big_pen = df["body_mass_g"].max() number = len(df) ``` ### The Heaviest Penguin -We find that the heaviest penguin, out of a total of `python f"{number}"` penguins, has a mass of `python f"{big_pen:.2f}"` kilograms. +We find that the heaviest penguin, out of a total of `python f"{number}"` penguins, has a mass of `python f"{big_pen:.2f}"` grams. ```` diff --git a/workflow-help.qmd b/workflow-help.qmd index 443bc32..d96321d 100644 --- a/workflow-help.qmd +++ b/workflow-help.qmd @@ -30,7 +30,7 @@ A good reprex makes it easier for other people to help you, and often you'll fig - First, you need to make your code reproducible. This means that you need to capture everything, i.e., include any packages you used and create all necessary objects. The easiest way to make sure you've done this is to use the [**watermark**](https://github.com/rasbt/watermark) package alongside whatever else you are doing: ```{python} -import pandas as pd +import polars as pl import numpy as np import watermark.watermark as watermark @@ -52,16 +52,15 @@ There are several things you need to include to make your example reproducible: - **Packages** and their versions. These should be loaded at the top of the script, so it's easy to see which ones the example needs. By using **watermark** with the above configuration, you will also print the package versions. This is a good time to check that you're using the latest version of each package; it's possible you've discovered a bug that's been fixed since you installed or last updated the package. -- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **pandas**, it's quite easy to generate data from code; here's an example: +- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **polars**, it's quite easy to generate data from code; here's an example: ```{python} -df = pd.DataFrame( +df = pl.DataFrame( data=np.reshape(range(36), (6, 6)), - index=["a", "b", "c", "d", "e", "f"], - columns=["col" + str(i) for i in range(6)], - dtype=float, + schema=["col" + str(i) for i in range(6)], + orient="row", ) -df["random_normal"] = np.random.normal(size=6) +df = df.with_columns(pl.Series("random_normal", np.random.normal(size=6))) df ``` diff --git a/workflow-help.quarto_ipynb_1 b/workflow-help.quarto_ipynb_1 index e7bebf8..d475b24 100644 --- a/workflow-help.quarto_ipynb_1 +++ b/workflow-help.quarto_ipynb_1 @@ -41,7 +41,7 @@ "cell_type": "code", "metadata": {}, "source": [ - "import pandas as pd\n", + "import polars as pl\n", "import numpy as np\n", "import watermark.watermark as watermark\n", "\n", @@ -70,7 +70,7 @@ "\n", "- **Packages** and their versions. These should be loaded at the top of the script, so it's easy to see which ones the example needs. By using **watermark** with the above configuration, you will also print the package versions. This is a good time to check that you're using the latest version of each package; it's possible you've discovered a bug that's been fixed since you installed or last updated the package.\n", "\n", - "- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **pandas**, it's quite easy to generate data from code; here's an example:" + "- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **polars**, it's quite easy to generate data from code; here's an example:" ], "id": "c4ac60b4" }, @@ -78,13 +78,12 @@ "cell_type": "code", "metadata": {}, "source": [ - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " data=np.reshape(range(36), (6, 6)),\n", - " index=[\"a\", \"b\", \"c\", \"d\", \"e\", \"f\"],\n", - " columns=[\"col\" + str(i) for i in range(6)],\n", - " dtype=float,\n", + " schema=[\"col\" + str(i) for i in range(6)],\n", + " orient=\"row\",\n", ")\n", - "df[\"random_normal\"] = np.random.normal(size=6)\n", + "df = df.with_columns(pl.Series(\"random_normal\", np.random.normal(size=6)))\n", "df" ], "id": "d1e4562c", From d6acc11d69bd6d501cd75c22ac9dc312542b3d3c Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 22 Jul 2026 12:15:27 +0100 Subject: [PATCH 16/30] fix: address review feedback for joins.ipynb migration --- data-transform.ipynb | 2 +- joins.ipynb | 75 ++++++++++++++++++++++---------------------- 2 files changed, 39 insertions(+), 38 deletions(-) diff --git a/data-transform.ipynb b/data-transform.ipynb index 8dfc2c1..ccc75f0 100644 --- a/data-transform.ipynb +++ b/data-transform.ipynb @@ -164,7 +164,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights.with_columns(pl.col(\"time_hour\").str.to_datetime())" + "flights.with_columns(pl.col(\"time_hour\").str.to_datetime(time_zone=\"UTC\"))" ] }, { diff --git a/joins.ipynb b/joins.ipynb index 8ec4902..e6cdeb0 100644 --- a/joins.ipynb +++ b/joins.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Joins {#sec-joins}\n", @@ -17,7 +17,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "1", "metadata": { "tags": [ "remove-cell" @@ -36,17 +36,17 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **polars** data analysis package." + "This chapter will use the **polars** data analysis package. For reading Stata (.dta) files, we'll use pandas as a bridge (pd.read_stata()) and then convert to polars with pl.from_pandas()." ] }, { "cell_type": "markdown", - "id": "80710c6e", + "id": "3", "metadata": {}, "source": [ "## Concatenate\n", @@ -55,7 +55,7 @@ "\n", "![](https://pandas.pydata.org/docs/_images/08_concat_row.svg)\n", "\n", - "For the same columns, use pl.concat() to stack rows vertically. For the same index, you can use pl.concat() with how=\"horizontal\".\n", + "For the same columns, use pl.concat() to stack rows vertically. Polars doesn't have a row index like pandas; `pl.concat(how=\"horizontal\")` joins DataFrames by position, aligning rows in order.\n", "\n", "Here's an example using data on two different states' populations:" ] @@ -63,10 +63,11 @@ { "cell_type": "code", "execution_count": null, - "id": "f5ef4f37", + "id": "4", "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", "import polars as pl\n", "\n", "base_url = (\n", @@ -78,8 +79,6 @@ "# Read the Stata files and convert to polars\n", "list_of_state_dfs = []\n", "for state in state_codes:\n", - " import pandas as pd\n", - "\n", " temp_df = pd.read_stata(base_url + state + end_url)\n", " list_of_state_dfs.append(pl.from_pandas(temp_df))\n", "\n", @@ -94,7 +93,7 @@ }, { "cell_type": "markdown", - "id": "79bbd002", + "id": "5", "metadata": {}, "source": [ "Note that when concatenating, you can track the origin of rows by adding a column before concatenation:" @@ -103,7 +102,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2adb960a", + "id": "6", "metadata": {}, "outputs": [], "source": [ @@ -120,7 +119,7 @@ }, { "cell_type": "markdown", - "id": "e0c343b0", + "id": "7", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -142,7 +141,7 @@ }, { "cell_type": "markdown", - "id": "f6e91777", + "id": "8", "metadata": {}, "source": [ "## Joins\n", @@ -159,7 +158,7 @@ "\n", "- how=\"inner\" uses keys that appear in both data frames to join.\n", "\n", - "- how=\"outer\" uses all keys from both data frames to join on.\n", + "- how=\"full\" uses all keys from both data frames to join on (full outter join).\n", "\n", "- how=\"semi\" returns rows from the left where keys appear in the right.\n", "\n", @@ -171,7 +170,7 @@ { "cell_type": "code", "execution_count": null, - "id": "53c66d5d", + "id": "9", "metadata": {}, "outputs": [], "source": [ @@ -195,7 +194,7 @@ }, { "cell_type": "markdown", - "id": "cfe5aa30", + "id": "10", "metadata": {}, "source": [ "Right join:" @@ -204,7 +203,7 @@ { "cell_type": "code", "execution_count": null, - "id": "595f41e6", + "id": "11", "metadata": {}, "outputs": [], "source": [ @@ -213,18 +212,18 @@ }, { "cell_type": "markdown", - "id": "8af0241e", + "id": "12", "metadata": {}, "source": [ "Note that the key combination of K2 and K0 did not exist in the left-hand data frame, and so its entries in the final data frame are null values. But it does have entries because we chose the keys from the right-hand data frame.\n", "\n", - "What about an inner merge?" + "What about an inner join?" ] }, { "cell_type": "code", "execution_count": null, - "id": "5e73608f", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -233,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "67ac74c0", + "id": "14", "metadata": {}, "source": [ "Now we see that the combination K2 and K0 are excluded because they didn't exist in the overlap of keys in both data frames.\n", @@ -244,30 +243,32 @@ { "cell_type": "code", "execution_count": null, - "id": "5d209fb9", + "id": "15", "metadata": {}, "outputs": [], "source": [ - "left.join(right, on=[\"key1\", \"key2\"], how=\"outer\")" + "left.join(right, on=[\"key1\", \"key2\"], how=\"full\")" ] }, { "cell_type": "markdown", - "id": "b984a202", + "id": "16", "metadata": {}, "source": [ - "Polars doesn't include a built-in indicator parameter to track which side contributed each row in a join. However, you can achieve the same effect by comparing the two DataFrames after the join:" + "This performs a full outer join, keeping all rows from both data frames.\n", + "\n", + "Polars doesn't include a built-in indicator parameter to track which side contributed each row in a join. On a full join, polars keeps the join keys from both sides with suffixes. However, you can achieve the same effect by comparing the two DataFrames after the join:" ] }, { "cell_type": "code", "execution_count": null, - "id": "574c4508", + "id": "17", "metadata": {}, "outputs": [], "source": [ "# Perform an outer join\n", - "joined = left.join(right, on=[\"key1\", \"key2\"], how=\"outer\")\n", + "joined = left.join(right, on=[\"key1\", \"key2\"], how=\"full\")\n", "\n", "# Add indicator column manually\n", "# Rows with nulls in B only came from right\n", @@ -286,7 +287,7 @@ }, { "cell_type": "markdown", - "id": "67ca901b", + "id": "18", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -320,7 +321,7 @@ }, { "cell_type": "markdown", - "id": "5d09e145", + "id": "19", "metadata": {}, "source": [ "## Semi and Anti Joins\n", @@ -333,7 +334,7 @@ { "cell_type": "code", "execution_count": null, - "id": "33207e8a", + "id": "20", "metadata": {}, "outputs": [], "source": [ @@ -343,7 +344,7 @@ }, { "cell_type": "markdown", - "id": "d55603d0", + "id": "21", "metadata": {}, "source": [ "Anti join: Keep rows from the left where keys do NOT appear in the right:" @@ -352,7 +353,7 @@ { "cell_type": "code", "execution_count": null, - "id": "92218fbb", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -362,7 +363,7 @@ }, { "cell_type": "markdown", - "id": "355135ce", + "id": "23", "metadata": {}, "source": [ "## Handling Duplicate Column Names\n", @@ -373,7 +374,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a042820b", + "id": "24", "metadata": {}, "outputs": [], "source": [ @@ -392,10 +393,10 @@ }, { "cell_type": "markdown", - "id": "aa6e6071", + "id": "25", "metadata": {}, "source": [ - "For more information about the available join types and options, see **polars'** [joining documentation](https://docs.pola.rs/api/python/stable/reference/index.html)." + "For more information about the available join types and options, see **polars'** [joining documentation](https://docs.pola.rs/user-guide/transformations/joins/)." ] } ], @@ -407,7 +408,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "base", "language": "python", "name": "python3" }, From 3c8e9dcb0db809024a44b7a4fea02fb56d51a35b Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 22 Jul 2026 21:58:45 +0100 Subject: [PATCH 17/30] fix: updated feedback for numbers.ipynb migration --- numbers.ipynb | 71 +++++++++++++-------------------------------------- 1 file changed, 18 insertions(+), 53 deletions(-) diff --git a/numbers.ipynb b/numbers.ipynb index e73d803..bef5b6c 100644 --- a/numbers.ipynb +++ b/numbers.ipynb @@ -13,7 +13,7 @@ "\n", "### Prerequisites\n", "\n", - "This chapter mostly uses functions from **pandas**, which you are likely to already have installed but you can install using `uv add polars` in the terminal. We'll use real examples from nycflights13, as well as toy examples made with fake data.\n", + "This chapter mostly uses functions from **polars**, which you are likely to already have installed but you can install using `uv add polars` in the terminal. We'll use real examples from nycflights13, as well as toy examples made with fake data.\n", "\n", "Let's first load up the NYC flights data\n" ] @@ -86,7 +86,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights[\"dest\"].value_counts()" + "flights[\"dest\"].value_counts(sort=True)" ] }, { @@ -94,7 +94,7 @@ "id": "cdb110f3", "metadata": {}, "source": [ - "This is automatically sorted in order of the most common category. You can perform the same computation \"by hand\" with `group_by()`, `agg()` and then using the count function. This is useful because it allows you to compute other summaries at the same time:" + "By default, `value_counts()` returns counts for each unique value. Passing `sort=True` sorts the output in order of the most common category. You can perform the same computation \"by hand\" with `group_by()`, `agg()`, and `sort()`:" ] }, { @@ -163,11 +163,11 @@ "\n", "Basic number arithmatic is achieved by `+` (addition), `-` (subtraction), `*` (multiplication), `/` (division), `**` (powers), `%` (modulo), and `@` (tensor product). Most of these functions don't need a huge amount of explanation because you'll be familiar with them already (and you can look up the others when you do need them).\n", "\n", - "When you have two numeric columns of equal length and you add or subtract them, it's pretty obvious what's going to happen. But we do need to talk about what happens when there is a variable involved that is *not* as long as the column. This is important for operations like `flights.assign(air_time = air_time / 60)` because there are 336,776 numbers on the left of `/` but only one on the right. In this case, **polars** will understand that you'd like to divide *all* values of air time by 60. This is sometimes called 'broadcasting'. Below is a digram that tries to explain what's going on:\n", + "When you have two numeric columns of equal length and you add or subtract them, it's pretty obvious what's going to happen. But we do need to talk about what happens when there is a variable involved that is *not* as long as the column. This is important for operations like `flights.with_columns(air_time = pl.col(\"air_time\") / 60)` because there are 336,776 numbers on the left of `/` but only one on the right. In this case, **polars** will understand that you'd like to divide *all* values of air time by 60. This is sometimes called 'broadcasting'. Below is a digram that tries to explain what's going on:\n", "\n", "![](https://numpy.org/doc/stable/_images/broadcasting_1.png)\n", "\n", - "You can find out much more about [broadcasting on the **numpy** documentation](https://numpy.org/doc/stable/user/basics.broadcasting.html). **polars** is built on top of **numpy** and inherits some of its functionality. \n" + "Broadcasting is a concept popularized by **numpy**. While **polars** is written in Rust and built on Apache Arrow, it adopts vectorised broadcasting behaviour for scalar operations.\n" ] }, { @@ -429,7 +429,7 @@ " pl.col(\"year\").round(),\n", " ((pl.col(\"money\") / 100).round() * 100).alias(\"money\"),\n", " ]\n", - ").head()" + ").tail()" ] }, { @@ -514,7 +514,7 @@ "id": "c22b0eda", "metadata": {}, "source": [ - "Remember that you can always apply numpy functions to **polars** data frame columns using `map_elements()`:" + "**polars** provides native expression methods like `.ceil()` and `.floor()` for rounding operations:" ] }, { @@ -524,7 +524,7 @@ "metadata": {}, "outputs": [], "source": [ - "money.with_columns(pl.col(\"money\").map_elements(np.ceil)).head()" + "money.with_columns(pl.col(\"money\").ceil()).head()" ] }, { @@ -534,7 +534,7 @@ "source": [ "### Cumulative and Rolling Aggregates\n", "\n", - "**polars** has several cumulative functions, including `.cumsum()`, `.cummax()` and `.cummin()`, and `.cumprod()`. " + "**polars** has several cumulative functions, including `.cum_sum()`, `.cum_max()`, `.cum_min()`, and `.cum_prod()`." ] }, { @@ -544,23 +544,6 @@ "metadata": {}, "outputs": [], "source": [ - "import numpy as np\n", - "import polars as pl\n", - "\n", - "# Create the money DataFrame\n", - "starting = 100\n", - "interest = 1.05\n", - "money = pl.DataFrame(\n", - " {\n", - " \"year\": 2000 + np.arange(1, 51),\n", - " \"money\": starting * interest ** np.arange(1, 51),\n", - " }\n", - ")\n", - "\n", - "# Show the data\n", - "money.head()\n", - "\n", - "# Cumulative sum\n", "money[\"money\"].cum_sum().tail()" ] }, @@ -603,7 +586,7 @@ "id": "85345760", "metadata": {}, "source": [ - "We can also do a percent rank by passing the keyword argument `method=\"average\"` and then scaling:" + "To compute a percent rank in **polars**, you can call `.rank()` and divide by the total number of rows (`df.height`):" ] }, { @@ -695,7 +678,7 @@ "id": "02c212f4", "metadata": {}, "source": [ - "Sometimes you don't just want one percentile, but a bunch of them. **polars** makes this very easy by allowing you to pass a list of quantiles:" + "Sometimes you don't just want one percentile, but a bunch of them. While **polars**' `.quantile()` function computes a single quantile level, you can calculate multiple quantiles using a list comprehension to build a DataFrame:" ] }, { @@ -705,30 +688,12 @@ "metadata": {}, "outputs": [], "source": [ - "# Create a struct with quantile values\n", - "money.select(\n", - " [\n", - " pl.struct(\n", - " [\n", - " pl.lit(0).alias(\"0%\"),\n", - " pl.lit(0.25).alias(\"25%\"),\n", - " pl.lit(0.5).alias(\"50%\"),\n", - " pl.lit(0.75).alias(\"75%\"),\n", - " ]\n", - " ).alias(\"quantiles\")\n", - " ]\n", - ")\n", - "\n", - "# Create a DataFrame with the quantile results\n", - "quantile_df = money.select(\n", - " [\n", - " pl.lit(money[\"money\"].quantile(0)).alias(\"0%\"),\n", - " pl.lit(money[\"money\"].quantile(0.25)).alias(\"25%\"),\n", - " pl.lit(money[\"money\"].quantile(0.5)).alias(\"50%\"),\n", - " pl.lit(money[\"money\"].quantile(0.75)).alias(\"75%\"),\n", - " ]\n", - ")\n", - "quantile_df" + "pl.DataFrame(\n", + " {\n", + " \"quantile\": [0.0, 0.25, 0.5, 0.75],\n", + " \"value\": [money[\"money\"].quantile(q) for q in [0.0, 0.25, 0.5, 0.75]],\n", + " }\n", + ")" ] }, { @@ -844,7 +809,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, From f2cbcbbbe83db9a12ae0153a968db67da27f2c56 Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 22 Jul 2026 23:18:03 +0100 Subject: [PATCH 18/30] fix: address review feedbacks for missing-values.ipynb --- missing-values.ipynb | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/missing-values.ipynb b/missing-values.ipynb index 37d46e8..b66a98c 100644 --- a/missing-values.ipynb +++ b/missing-values.ipynb @@ -114,7 +114,12 @@ "id": "8b5a2849", "metadata": {}, "source": [ - "Both types of missing value can be found using the `.is_null()` method, which returns a new column of boolean values that are `True` if the value is missing:" + "In **polars**, it is important to distinguish between `null` (missing data) and `NaN` (floating-point Not-a-Number, such as `np.nan`):\n", + "\n", + "- `.is_null()` checks for `null` values (returning `True` for `None` / `null`).\n", + "- `.is_nan()` checks for `NaN` values (returning `True` for `np.nan`).\n", + "\n", + "Let's see how `.is_null()` and `.is_nan()` behave on a column containing both types of missing values:" ] }, { @@ -124,7 +129,13 @@ "metadata": {}, "outputs": [], "source": [ - "fruits.select(pl.col(\"fruit\").is_null())" + "numbers.select(\n", + " [\n", + " pl.col(\"numbers\"),\n", + " pl.col(\"numbers\").is_null().alias(\"is_null\"),\n", + " pl.col(\"numbers\").is_nan().alias(\"is_nan\"),\n", + " ]\n", + ")" ] }, { @@ -132,7 +143,7 @@ "id": "0fd6764d", "metadata": {}, "source": [ - "As a convenience, there is also an .is_not_null() method:" + "For string columns, missing values are also represented by `null`. As a convenience, there is also an `.is_not_null()` method:" ] }, { @@ -268,7 +279,7 @@ "outputs": [], "source": [ "# Forward fill with limit of 1\n", - "null_df.fill_null(strategy=\"forward\", limit=2)" + "null_df.fill_null(strategy=\"forward\", limit=1)" ] }, { @@ -294,7 +305,7 @@ "id": "710bcb2c", "metadata": {}, "source": [ - "You can also drop rows where all values are null using a custom filter:" + "You can also drop rows where all values are null using `pl.all_horizontal()` in a filter:" ] }, { @@ -331,7 +342,7 @@ "id": "2b058955", "metadata": {}, "source": [ - "Notice that the only the last all null column was removed." + "Notice that the last row (where all values were `null`) was removed, while column `C` (where all values are `null`) remains because this operation filters rows, not columns." ] }, { @@ -621,7 +632,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, From 174a83396e919db3aa555f668452bc5fa3dd70b3 Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 22 Jul 2026 23:53:26 +0100 Subject: [PATCH 19/30] fix: updated to review - missing values --- missing-values.ipynb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/missing-values.ipynb b/missing-values.ipynb index b66a98c..27d4777 100644 --- a/missing-values.ipynb +++ b/missing-values.ipynb @@ -50,7 +50,7 @@ "source": [ "## Explicit Missing Values\n", "\n", - "To begin, let's explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see an `null` value.\n", + "To begin, let's explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see a `null` value.\n", "\n", "### Types of Missing Values\n", "\n", @@ -376,7 +376,7 @@ "id": "c1588f63", "metadata": {}, "source": [ - "### Adding NA values\n", + "### Adding Null values\n", "\n", "Sometimes you'll hit the opposite problem where some concrete value actually represents a missing value. This typically arises in data generated by older software that doesn't have a proper way to represent missing values, so it must instead use some special value like 99 or -999.\n", "\n", @@ -451,7 +451,7 @@ "source": [ "## Implicit Missing Values\n", "\n", - "So far we've talked about missing values that are **explicitly** missing, i.e. you can see an `NA` or similar in your data.\n", + "So far we've talked about missing values that are **explicitly** missing, i.e. you can see a `null` or similar in your data.\n", "But missing values can also be **implicitly** missing, if an entire row of data is simply absent from the data.\n", "Let's illustrate the difference with a simple data set that records the price of some stock each quarter:" ] @@ -480,7 +480,7 @@ "source": [ "This dataset has two missing observations:\n", "\n", - "- The `price` in the fourth quarter of 2020 is explicitly missing, because its value is `NA`.\n", + "- The `price` in the fourth quarter of 2020 is explicitly missing, because its value is `null`.\n", "\n", "- The `price` for the first quarter of 2021 is implicitly missing, because it simply does not appear in the dataset.\n", "\n", From 787ca880b24f515753d20b6e5058217dde58cb19 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 23 Jul 2026 01:00:59 +0100 Subject: [PATCH 20/30] fix: address review feedback for date-time.ipynb migration --- dates-and-times.ipynb | 146 ++++++++++++++++-------------------------- 1 file changed, 55 insertions(+), 91 deletions(-) diff --git a/dates-and-times.ipynb b/dates-and-times.ipynb index bc612f6..2073145 100644 --- a/dates-and-times.ipynb +++ b/dates-and-times.ipynb @@ -51,13 +51,7 @@ "source": [ "### Prerequisites\n", "\n", - "You will need to install the **seaborn** package for this chapter. This chapter uses the next generation version of **seaborn**, which can be installed by running the following on the command line (aka in the terminal): \n", - "\n", - "```bash\n", - "uv run pip install --pre seaborn\n", - "```\n", - "\n", - "We will also be using the **polars** package and numerical package **numpy**." + "This chapter focuses on working with dates and time series using **polars**. We'll also use **matplotlib** for visualising time series data. Make sure you have **polars** installed (using `uv add polars` in the terminal)." ] }, { @@ -356,7 +350,7 @@ "source": [ "## Vectorised Datetimes \n", "\n", - "Now we come to vectorised operations on datetimes using the powerful **numpy** packages (and this is what is used by **pandas**). **numpy** has its own version of datetime, called `np.datetime64`, and it's very efficient at scale. Let's see it in action:" + "Now we come to vectorised operations on datetimes using **numpy**. **numpy** has its own version of datetime, called `np.datetime64`, which is very efficient at scale. Let's see it in action:" ] }, { @@ -474,7 +468,11 @@ "metadata": {}, "outputs": [], "source": [ - "pl.datetime_range(start=\"2018-01-01\", end=\"2018-01-08\", interval=\"1d\")" + "from datetime import datetime\n", + "\n", + "pl.datetime_range(\n", + " start=datetime(2018, 1, 1), end=datetime(2018, 1, 8), interval=\"1d\", eager=True\n", + ")" ] }, { @@ -656,7 +654,7 @@ "source": [ "### Datetime Offsets\n", "\n", - "Our data, currently held in `df`, were read in as if they were from the *start* of the month but these data refer to the month that has passed and so should be for the *end* of the month. Fortunately, we can change this using date offset operations. In polars, you can add durations:" + "Our data, currently held in `df`, were read in as if they were recorded at the start of the month, e.g., 2001-05-01. But they are actually monthly averages for the given month, so snapping them to the end of the month makes more sense. We can use `.dt.month_end()` to snap dates to the last day of the month:" ] }, { @@ -666,7 +664,7 @@ "metadata": {}, "outputs": [], "source": [ - "df = df.with_columns((pl.col(\"date\") + pl.duration(days=1)).dt.offset_by(\"1mo\"))\n", + "df = df.with_columns(pl.col(\"date\").dt.month_end())\n", "df.head()" ] }, @@ -700,9 +698,9 @@ "id": "a40f982f", "metadata": {}, "source": [ - "### Creating a datetime Index and Setting the Frequency\n", + "### Sorting and Setting Up Datetime Series\n", "\n", - "For time series operations in polars, you typically don't set an index like in pandas. Instead, you keep the datetime as a column and use it in group operations. However, you can sort by the datetime column:" + "For time series operations in **polars**, you typically keep the datetime as a column and use it in group operations. First, let's ensure our datetime column is sorted:" ] }, { @@ -735,7 +733,7 @@ "df_resampled = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", " pl.col(\"Vacancies (ICT), thousands\").mean()\n", ")\n", - "df_resampled" + "df_resampled.head()" ] }, { @@ -745,7 +743,7 @@ "source": [ "## Making Quick Time Series Plots\n", "\n", - "Having managed to put your time series into a data frame, perhaps converting a column of type string into a column of type datetime in the process, you often just want to see the thing! We can achieve this by converting to pandas for plotting:\n" + "Having managed to put your time series into a **polars** DataFrame, you often want to quickly plot the results. Converting to a **pandas** DataFrame via `.to_pandas()` is a common bridging pattern for using Matplotlib's built-in time series plotting utilities:" ] }, { @@ -768,11 +766,7 @@ "source": [ "## Resampling, Rolling, and Shifting\n", "\n", - "Now our data have a *datetime index*, some common time series operations are made very easy for us.\n", - "\n", - "### Resampling\n", - "\n", - "Quite frequently, there is a situation in which one would like to change the frequency of a given time series. Polars makes this easy via the `group_by_dynamic()` function. Here's an example resampling the monthly data to annual and taking the mean:" + "Now that our data have a sorted *datetime column*, we can perform various time series operations such as resampling, rolling windows, and shifting." ] }, { @@ -782,10 +776,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", + "# Resample to annual frequency\n", + "df_annual = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", " pl.col(\"Vacancies (ICT), thousands\").mean()\n", ")\n", - "df.resample" + "df_annual" ] }, { @@ -793,7 +788,7 @@ "id": "c3b45c36", "metadata": {}, "source": [ - "Resampling can go up in frequency (up-sampling) as well as down, but we no longer need to choose an aggregation function, we must now choose how we'd like to fill in the gaps for the frequencies we didn't have in the original data." + "Resampling can go up in frequency (up-sampling) as well as down. We can use `.upsample()` to insert missing dates at a daily frequency:" ] }, { @@ -804,10 +799,8 @@ "outputs": [], "source": [ "# Upsample to daily frequency\n", - "daily = df.group_by_dynamic(\"date\", every=\"1d\").agg(\n", - " pl.col(\"Vacancies (ICT), thousands\").first()\n", - ")\n", - "daily" + "daily = df.upsample(time_column=\"date\", every=\"1d\")\n", + "daily.head()" ] }, { @@ -875,68 +868,9 @@ "metadata": {}, "outputs": [], "source": [ - "from itertools import cycle\n", - "\n", - "import matplotlib.dates as mdates\n", - "import matplotlib.pyplot as plt\n", - "\n", - "# Create a daily series\n", - "data = xf_pl.select([\"date\", \"close\"]).sort(\"date\").head(10)\n", - "\n", - "# Original observations\n", - "original = data\n", - "\n", - "# Daily frequency with missing dates\n", - "daily = data.upsample(time_column=\"date\", every=\"1d\")\n", - "\n", - "# Back-fill\n", - "bfill = daily.fill_null(strategy=\"backward\")\n", - "\n", - "# Forward-fill\n", - "ffill = daily.fill_null(strategy=\"forward\")\n", - "\n", - "fig, ax = plt.subplots()\n", - "\n", - "colour_wheel = cycle(plt.rcParams[\"axes.prop_cycle\"])\n", - "\n", - "# Original observations\n", - "ax.plot(\n", - " original[\"date\"],\n", - " original[\"close\"],\n", - " \"o\",\n", - " linestyle=\"None\",\n", - " label=\"original\",\n", - " zorder=3,\n", - ")\n", - "\n", - "# Back-fill\n", - "ax.plot(\n", - " bfill[\"date\"],\n", - " bfill[\"close\"],\n", - " \"-.o\",\n", - " color=next(colour_wheel)[\"color\"],\n", - " linewidth=1,\n", - " label=\"back-fill\",\n", - ")\n", - "\n", - "# Forward-fill\n", - "ax.plot(\n", - " ffill[\"date\"],\n", - " ffill[\"close\"],\n", - " \"--o\",\n", - " color=next(colour_wheel)[\"color\"],\n", - " linewidth=1,\n", - " label=\"forward-fill\",\n", - ")\n", - "\n", - "ax.set_ylabel(\"Close ($)\")\n", - "ax.legend()\n", - "\n", - "# Show only the day of the month on the x-axis\n", - "ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%b %d\"))\n", - "fig.autofmt_xdate(rotation=0)\n", - "\n", - "plt.show()" + "# Upsample to daily frequency\n", + "daily = df.upsample(time_column=\"date\", every=\"1d\")\n", + "daily.head()" ] }, { @@ -946,7 +880,7 @@ "source": [ "### Rolling Window Functions\n", "\n", - "Polars supports rolling window functions with .rolling():" + "Polars supports rolling window operations on expressions, including `.rolling_mean()`, `.rolling_std()`, and `.ewm_mean()`:" ] }, { @@ -962,6 +896,36 @@ ")" ] }, + { + "cell_type": "markdown", + "id": "ewm_markdown_cell", + "metadata": {}, + "source": [ + "### Exponentially Weighted and Expanding Aggregations\n", + "\n", + "Polars also supports exponentially weighted moving averages via `.ewm_mean()`, as well as expanding aggregations:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ewm_code_cell", + "metadata": {}, + "outputs": [], + "source": [ + "# Exponentially weighted moving average\n", + "df_ewm = df.with_columns(\n", + " [\n", + " pl.col(\"Vacancies (ICT), thousands\").ewm_mean(alpha=0.2).alias(\"ewm_0.2\"),\n", + " (\n", + " pl.col(\"Vacancies (ICT), thousands\").cum_sum()\n", + " / pl.int_range(1, pl.len() + 1)\n", + " ).alias(\"expanding_mean\"),\n", + " ]\n", + ")\n", + "df_ewm.head()" + ] + }, { "cell_type": "markdown", "id": "f3884a6a", @@ -1013,7 +977,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, From 3e53e28093f66111887e78613c7e4cab46da6041 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 23 Jul 2026 02:42:58 +0100 Subject: [PATCH 21/30] fix: address review feedback for boolean.ipynb migration --- boolean-data.ipynb | 49 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 13 deletions(-) diff --git a/boolean-data.ipynb b/boolean-data.ipynb index 0bd94c2..bd4c3d6 100644 --- a/boolean-data.ipynb +++ b/boolean-data.ipynb @@ -11,9 +11,13 @@ "\n", "In this chapter, we'll introduce boolean data: data that can be `True` or `False` (which can also be encoded as 1s or 0s). We'll first look at the fundamental Python true and false boolean variables before seeing how true and false work in data frames.\n", "\n", + "### Prerequisites\n", + "\n", + "This chapter focuses on working with boolean data and expressions using **polars**. Make sure you have **polars** installed (`uv add polars`). For more on Polars boolean operations, see the [Polars boolean expression documentation](https://docs.pola.rs/api/python/stable/reference/expressions/boolean.html).\n", + "\n", "## Booleans\n", "\n", - "Some of the most important operations you will perform are with `True` and `False` values, also known as boolean data types. These are fundamental Python variables, just as numbers such as `1` are. \n", + "Some of the most important operations you will perform are with `True` and `False` values, also known as boolean data types. These are fundamental Python variables, just as numbers such as `1` are.\n", "\n", "### Boolean Variables and Conditions\n", "\n", @@ -511,7 +515,7 @@ "id": "c7a0cd32", "metadata": {}, "source": [ - "We can perform operations on these just like regular **pandas** data frame columns. These accept `&` (and), `|` (or), `==` (equal), and `!=` (not equal) as operations:" + "We can perform operations on these just like regular **polars** data frame columns. These accept `&` (and), `|` (or), `==` (equal), and `!=` (not equal) as operations:" ] }, { @@ -521,7 +525,12 @@ "metadata": {}, "outputs": [], "source": [ - "df.select((pl.col(\"bool_col_1\") | pl.col(\"bool_col_2\")).alias(\"result\"))" + "df.select(\n", + " [\n", + " (pl.col(\"bool_col_1\") & pl.col(\"bool_col_2\")).alias(\"and_result\"),\n", + " (pl.col(\"bool_col_1\") | pl.col(\"bool_col_2\")).alias(\"or_result\"),\n", + " ]\n", + ")" ] }, { @@ -600,12 +609,8 @@ "metadata": {}, "outputs": [], "source": [ - "(\n", - " diamonds.with_columns((pl.col(\"price\") > 1000).alias(\"expensive\"))\n", - " # tweaked to display a mix of true and false values.\n", - " .select([\"cut\", \"carat\", \"color\", \"clarity\", \"price\", \"expensive\"])\n", - " .sample(10, shuffle=True)\n", - ")" + "diamonds = diamonds.with_columns((pl.col(\"price\") > 1000).alias(\"expensive\"))\n", + "diamonds.select([\"cut\", \"carat\", \"color\", \"clarity\", \"price\", \"expensive\"]).head()" ] }, { @@ -624,7 +629,7 @@ "outputs": [], "source": [ "diamonds.select(\n", - " pl.all(),\n", + " [\"cut\", \"carat\", \"color\", \"clarity\", \"price\"],\n", " (pl.col(\"price\") > 1000).alias(\"expensive\"),\n", ").head()" ] @@ -634,7 +639,7 @@ "id": "622ff486", "metadata": {}, "source": [ - "Another use of booleans that is quite useful when it comes to data frames is checking if a set of columns exists:" + "Another use of booleans in data frames is checking if a set of column names exists. We can use a Python list comprehension with the Python `in` operator on `diamonds.columns`:" ] }, { @@ -648,6 +653,24 @@ "[col in diamonds.columns for col in cols_to_check]" ] }, + { + "cell_type": "markdown", + "id": "polars_is_in_markdown", + "metadata": {}, + "source": [ + "Alternatively, you can check column existence natively in **polars** using `pl.Series.is_in()`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "polars_is_in_code", + "metadata": {}, + "outputs": [], + "source": [ + "pl.Series(diamonds.columns).is_in(cols_to_check)" + ] + }, { "cell_type": "markdown", "id": "6de309a4", @@ -665,7 +688,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds.select((pl.col(\"price\") > 1000).any().alias(\"any_expensive\"))" + "diamonds.select(pl.col(\"expensive\").any())" ] }, { @@ -705,7 +728,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, From 44d53c9a4c0ed6eee8f861b0567182310557b954 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 23 Jul 2026 16:58:42 +0100 Subject: [PATCH 22/30] feat: addres review feedback for quarto.qmd and prerequisites --- visualise.quarto_ipynb_1 | 136 ---------------- workflow-help.quarto_ipynb_1 | 114 -------------- ...w-packages-and-environments.quarto_ipynb_1 | 149 ------------------ 3 files changed, 399 deletions(-) delete mode 100644 visualise.quarto_ipynb_1 delete mode 100644 workflow-help.quarto_ipynb_1 delete mode 100644 workflow-packages-and-environments.quarto_ipynb_1 diff --git a/visualise.quarto_ipynb_1 b/visualise.quarto_ipynb_1 deleted file mode 100644 index 2ef6d2f..0000000 --- a/visualise.quarto_ipynb_1 +++ /dev/null @@ -1,136 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Visualisation {#sec-visualise}\n", - "\n", - "After reading the first part of the book, you understand the basics of the most important tools for doing data science. Now it’s time to start diving into the details. In this part of the book, you’ll learn about visualising data in further depth (in @sec-vis-layers), and get further stuck into the details of the different kinds of data visualisation (in @sec-exploratory-data-analysis and @sec-communicate-plots). In this short chapter, we discuss the different ways to create visualisations, and the different purposes of visualisations.\n", - "\n", - "## Philosophies of data visualisation\n", - "\n", - "There are broadly two categories of approach to using code to create data visualisations: *imperative* (build what you want from individual elements) and *declarative* (say what you want from a list of pre-existing options). Choosing which to use involves a trade-off: imperative libraries offer you flexibility but at the cost of some verbosity; declarative libraries offer you a quick way to plot your data, but only if it’s in the right format to begin with, and customisation to special chart types is more difficult.\n", - "\n", - "Python has many excellent plotting packages, including perhaps the most powerful imperative plotting package around, **matplotlib**, and an amazing declarative library that we already saw, **lets-plot**. These two libraries will get you a long way, and each could be worthy of an entire book themselves. Fortunately for us, though, we can do 95% of what we need with a small number of commands from one or the other of them. In general, to keep this book as light as possible, we've opted to use **lets-plot** wherever possible—and @sec-vis-layers is going to take you on a more in-depth tour of how to use it yourself.\n", - "\n", - "## Purposes of data visualisation\n", - "\n", - "Data visualisation has all kinds of different purposes. It can be useful to bear in mind three broad categories of visualisation that are out there:\n", - "\n", - "- exploratory\n", - "- scientific\n", - "- narrative\n", - "\n", - "Let's look at each in a bit more detail.\n", - "\n", - "### Exploratory Data Viz\n", - "\n", - "The first of the three kinds is *exploratory data visualisation*, and it's the kind that you do when you're looking and data and trying to understand it. Just plotting the data is a really good strategy for getting a feel for any issues there might be. This is perhaps most famously demonstrated by Anscombe's quartet: four different datasets with the same mean, standard deviation, and correlation but very different data distributions." - ], - "id": "f3331573" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "#| echo: false\n", - "import numpy as np\n", - "import pandas as pd\n", - "import matplotlib.pyplot as plt\n", - "import matplotlib_inline.backend_inline\n", - "\n", - "# Plot settings\n", - "plt.style.use(\"https://github.com/aeturrell/python4DS/raw/main/plot_style.txt\")\n", - "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")\n", - "\n", - "# Set max rows displayed for readability\n", - "pd.set_option(\"display.max_rows\", 6)\n", - "\n", - "x = [10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5]\n", - "y1 = [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]\n", - "y2 = [9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74]\n", - "y3 = [7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73]\n", - "x4 = [8, 8, 8, 8, 8, 8, 8, 19, 8, 8, 8]\n", - "y4 = [6.58, 5.76, 7.71, 8.84, 8.47, 7.04, 5.25, 12.50, 5.56, 7.91, 6.89]\n", - "\n", - "datasets = {\"I\": (x, y1), \"II\": (x, y2), \"III\": (x, y3), \"IV\": (x4, y4)}\n", - "\n", - "fig, axs = plt.subplots(\n", - " 2,\n", - " 2,\n", - " sharex=True,\n", - " sharey=True,\n", - " figsize=(10, 6),\n", - " gridspec_kw={\"wspace\": 0.08, \"hspace\": 0.08},\n", - ")\n", - "axs[0, 0].set(xlim=(0, 20), ylim=(2, 14))\n", - "axs[0, 0].set(xticks=(0, 10, 20), yticks=(4, 8, 12))\n", - "\n", - "for ax, (label, (x, y)) in zip(axs.flat, datasets.items()):\n", - " ax.text(0.1, 0.9, label, fontsize=20, transform=ax.transAxes, va=\"top\")\n", - " ax.tick_params(direction=\"in\", top=True, right=True)\n", - " ax.plot(x, y, \"o\")\n", - "\n", - " # linear regression\n", - " p1, p0 = np.polyfit(x, y, deg=1) # slope, intercept\n", - " ax.axline(xy1=(0, p0), slope=p1, color=\"r\", lw=2)\n", - "\n", - " # add text box for the statistics\n", - " stats = (\n", - " f\"$\\\\mu$ = {np.mean(y):.2f}\\n\"\n", - " f\"$\\\\sigma$ = {np.std(y):.2f}\\n\"\n", - " f\"$r$ = {np.corrcoef(x, y)[0][1]:.2f}\"\n", - " )\n", - " bbox = dict(boxstyle=\"round\", fc=\"blanchedalmond\", ec=\"orange\", alpha=0.5)\n", - " ax.text(\n", - " 0.95,\n", - " 0.07,\n", - " stats,\n", - " fontsize=9,\n", - " bbox=bbox,\n", - " transform=ax.transAxes,\n", - " horizontalalignment=\"right\",\n", - " )\n", - "\n", - "plt.suptitle(\"Anscombe's Quartet\")\n", - "plt.show()" - ], - "id": "64a0e7f6", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Exploratory visualisation is usually quick and dirty, and flexible too. Some exploratory data viz can be automated, and there's a whole host of packages to help with this, including [**skimpy**](https://aeturrell.github.io/skimpy/).\n", - "\n", - "Beyond you and perhaps your co-authors/collaborators, however, not many other people should be seeing your exploratory visualisation! They will typically be worked up quickly, be numerous, and be throw-away. We'll look more at this in @sec-exploratory-data-analysis.\n", - "\n", - "### Scientific Data Viz\n", - "\n", - "The second kind, scientific data visualisation, is the prime cut of your exploratory visualisation. It's the kind of plot you might include in a more technical paper, the picture that says a thousand words. I often think of the first image of a black hole @akiyama2019first as a prime example of this. You can get away with having a high density of information in a scientific plot and, in short format journals, you may need to. The journal Physical Review Letters, which has an 8 page limit, has a classic of this genre in more or less every issue. Ensuring that important values can be accurately read from the plot is especially important in these kinds of charts. But they can also be the kind of plot that presents the killer results in a study; they might not be exciting to people who don't look at charts for a living, but they might be exciting and, just as importantly, understandable by your peers.\n", - "\n", - "This type of visualisation is especially popular in the big science journals like *Nature* and *Science*, where space is at a premium. We won't cover this type of plot in this book, because it tends to be very bespoke.\n", - "\n", - "### Narrative Data Viz\n", - "\n", - "The third and final kind is narrative data visualisation. This is the one that requires the most thought in the step where you go from the first view to the end product. It's a visualisation that doesn't just show a picture, but gives an insight. These are the kind of visualisations that you might see in the *Financial Times*, *The Economist*, or on the *BBC News* website. They come with aids that help the viewer focus on the aspects that the creator wanted them to (you can think of these aids or focuses as doing for visualisation what bold font does for text). They're well worth using in your work, especially if you're trying to communicate a particular narrative, and especially if the people you're communicating with don't have deep knowledge of the topic. You might use them in a paper that you hope will have a wide readership, in a blog post summarising your work, or in a report intended for a policymaker.\n", - "\n", - "You can find more information on the topic of communicating via data visualisations in the @sec-communicate-plots chapter." - ], - "id": "30b9ff30" - } - ], - "metadata": { - "kernelspec": { - "name": "python3", - "language": "python", - "display_name": "Python 3 (ipykernel)", - "path": "/Users/omagic/Documents/GitHub/python4DSpolars/.venv/share/jupyter/kernels/python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file diff --git a/workflow-help.quarto_ipynb_1 b/workflow-help.quarto_ipynb_1 deleted file mode 100644 index d475b24..0000000 --- a/workflow-help.quarto_ipynb_1 +++ /dev/null @@ -1,114 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Postscript: Getting Further Help {#sec-workflow-help}\n", - "\n", - "This book is not an island; there is no single resource that will allow you to master Python for Data Science. As you begin to apply the techniques described in this book to your own data, you will soon find questions that we do not answer. This section describes a few tips on how to get help, and to help you keep learning.\n", - "\n", - "## Resources\n", - "\n", - "Some other resources for learning are:\n", - "\n", - "- [The Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)\n", - "- [Real Python](https://realpython.com/), which has excellent short tutorials that cover Python more broadly (not just data science)\n", - "- [freeCodeCamp's Python courses](https://www.freecodecamp.org/news/search?query=data%20science%20python), though take care to select one that's at the right level for you\n", - "- [Coding for Economists](https://aeturrell.github.io/coding-for-economists), which has similar content to this book but is more in depth and aimed at analysts (particularly in economics)\n", - "\n", - "## Google is your friend\n", - "\n", - "If you get stuck, start with Google. Typically adding \"Python\" or \"Python Data Science\" (as the Python ecosystem goes *well* beyond data science) to a query is enough to restrict it to relevant results. Google is particularly useful for error messages. If you get an error message and you have no idea what it means, try googling it! Chances are that someone else has been confused by it in the past, and there will be help somewhere on the web.\n", - "\n", - "If Google doesn't help, try [Stack Overflow](http://stackoverflow.com). Start by spending a little time searching for an existing answer, including `[Python]` to restrict your search to questions and answers that use Python.\n", - "\n", - "## In the loop\n", - "\n", - "It's also helpful to keep an eye on the latest developments in data science. There are tons of data science newsletters out there, and we recommend keeping up with the Python data science community by following the (#pydata), (#datascience), and (#python) hashtags on Twitter.\n", - "\n", - "## Making a reprex (reproducible example)\n", - "\n", - "If your googling doesn't find anything useful, it's a really good idea prepare a minimal reproducible example or **reprex**.\n", - "\n", - "A good reprex makes it easier for other people to help you, and often you'll figure out the problem yourself in the course of making it. There are two parts to creating a reprex:\n", - "\n", - "- First, you need to make your code reproducible. This means that you need to capture everything, i.e., include any packages you used and create all necessary objects. The easiest way to make sure you've done this is to use the [**watermark**](https://github.com/rasbt/watermark) package alongside whatever else you are doing:" - ], - "id": "22b3f9e0" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "import polars as pl\n", - "import numpy as np\n", - "import watermark.watermark as watermark\n", - "\n", - "print(watermark())\n", - "print(watermark(iversions=True, globals_=globals()))" - ], - "id": "a119501b", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- Second, you need to make it minimal. Strip away everything that is not directly related to your problem. This usually involves creating a much smaller and simpler Python object than the one you're facing in real life or even using built-in data.\n", - "\n", - "That sounds like a lot of work! And it can be, but it has a great payoff:\n", - "\n", - "- 80% of the time creating an excellent reprex reveals the source of your problem. It's amazing how often the process of writing up a self-contained and minimal example allows you to answer your own question.\n", - "\n", - "- The other 20% of time you will have captured the essence of your problem in a way that is easy for others to play with. This substantially improves your chances of getting help.\n", - "\n", - "There are several things you need to include to make your example reproducible: Python environment, required packages, data, and code.\n", - "\n", - "- **Python environment**--really just the Python version. This is covered by the first call to the **watermark** package.\n", - "\n", - "- **Packages** and their versions. These should be loaded at the top of the script, so it's easy to see which ones the example needs. By using **watermark** with the above configuration, you will also print the package versions. This is a good time to check that you're using the latest version of each package; it's possible you've discovered a bug that's been fixed since you installed or last updated the package.\n", - "\n", - "- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **polars**, it's quite easy to generate data from code; here's an example:" - ], - "id": "c4ac60b4" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "df = pl.DataFrame(\n", - " data=np.reshape(range(36), (6, 6)),\n", - " schema=[\"col\" + str(i) for i in range(6)],\n", - " orient=\"row\",\n", - ")\n", - "df = df.with_columns(pl.Series(\"random_normal\", np.random.normal(size=6)))\n", - "df" - ], - "id": "d1e4562c", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- **Code**: copy and paste the minimal reproducible example code (including the packages, as noted above). Make sure you've used spaces and your variable names are concise, yet informative. Use comments to indicate where your problem lies. Do your best to remove everything that is not related to the problem. Finally, the shorter your code is, the easier it is to understand, and the easier it is to fix.\n", - "\n", - "Finish by checking that you have actually made a reproducible example by starting a fresh Python session and copying and pasting your reprex in." - ], - "id": "4b75e409" - } - ], - "metadata": { - "kernelspec": { - "name": "python3", - "language": "python", - "display_name": "Python 3 (ipykernel)", - "path": "/Users/omagic/Documents/GitHub/python4DSpolars/.venv/share/jupyter/kernels/python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file diff --git a/workflow-packages-and-environments.quarto_ipynb_1 b/workflow-packages-and-environments.quarto_ipynb_1 deleted file mode 100644 index a5600ce..0000000 --- a/workflow-packages-and-environments.quarto_ipynb_1 +++ /dev/null @@ -1,149 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Workflow: Packages and Environments {#sec-workflow-packages-and-environments}\n", - "\n", - "In this chapter, you're going to learn about packages and how to install them plus virtual coding environments that keep your packages isolated and your projects reproducible.\n", - "\n", - "## Packages\n", - "\n", - "### Introduction\n", - "\n", - "Packages (also called libraries) are key to extending the functionality of Python. It won't be long before you'll need to install some. There are packages for geoscience, for building websites, for analysing genetic data, for economics—pretty much for anything you can think of. Packages are typically not written by the core maintainers of the Python language but by enthusiasts, firms, researchers, academics, all sorts! Because anyone can write packages, they vary widely in their quality and usefulness. There are some that you'll be seeing them again and again.\n", - "\n", - "

Name a more iconic trio, I'll wait. pic.twitter.com/pGaLuUxQ3r

— Vicki Boykis (\\@vboykis) August 23, 2018
\n", - "\n", - "The three Python packages **numpy**, **pandas**, and **maplotlib**, which respectively provide numerical, data analysis, and plotting functionality, are ubiquitous. So many scripts begin by importing all three of them, as in the tweet above!\n", - "\n", - "There are typically two steps to using a new Python package:\n", - "\n", - "1. *install* the package on the command line (aka the terminal), eg using `uv add pandas`\n", - "\n", - "2. *import* the package into your Python session, eg using `import pandas as pd`\n", - "\n", - "When you issue an install command for a specific package, it is automatically downloaded from the internet and installed in the appropriate place on your computer. To install extra Python packages, you issue install commands to a text-based window called the \"terminal\".\n", - "\n", - "### The Command Line in Brief\n", - "\n", - "The *terminal* or *command line* or sometimes the *command prompt* was labelled 4 in the screenshot of Visual Studio Code from the chapter on @sec-introduction. The terminal is a text-based way to issue all kinds of commands to your computer (not just Python commands) and knowing a little bit about it is really useful for coding (and more) because managing packages, environments (which we haven't yet discussed), and version control (ditto) can all be done via the terminal. We'll come to these in due course in the chapter on @sec-command-line, but for now, a little background on what the terminal is and what it does.\n", - "\n", - "::: {.callout-note}\n", - "To open up the command line within Visual Studio Code, use the ⌃ + \\` keyboard shortcut (Mac) or ctrl + \\` (Windows/Linux), or click \"View > Terminal\".\n", - "\n", - "If you want to open up the command line independently of Visual Studio Code, search for \"Terminal\" on Mac and Linux, and \"Powershell\" on Windows.\n", - ":::\n", - "\n", - "Firstly, everything you can do by clicking on icons to launch programmes on your computer, you can also do via the terminal, also known as the command line. For many programmes, a lot of their functionality can be accessed using the command line, and other programmes *only* have a command line interface (CLI), including some that are used for data science.\n", - "\n", - "::: {.callout-tip}\n", - "The command line interacts with your operating system and is used to create, activate, or change Python installations.\n", - ":::\n", - "\n", - "Use Visual Studio Code to open a terminal window by clicking Terminal -> New Terminal on the list of commands at the very top of the window. If you have installed uv on your computer, your terminal should look something like this as your 'command prompt':\n", - "\n", - "```bash\n", - "your-username@your-computer current-directory %\n", - "```\n", - "\n", - "on Mac, and the same but with '%' replaced by '$' on linux, and (using Powershell)\n", - "\n", - "```powershell\n", - "PS C:\\Windows\\System32>\n", - "```\n", - "\n", - "on Windows.\n", - "\n", - "You can check that uv has successfully installed Python in your current project's folder by running\n", - "\n", - "```bash\n", - "uv run python --version\n", - "```\n", - "\n", - "For now, to at least try out the command line, let's use something that works across all three of the major operating systems. Type `uv run python` on the command prompt that came up in your new terminal window. You should see information about your installation of Python appear, including the version, followed by a Python prompt that looks like `>>>`. This is a kind of interactive Python session, in the terminal. It's much less rich than the one available in Visual Studio Code (it can't run scripts line-by-line, for example) but you can try `print('Hello World!')` and it will run, printing your message. To exit the terminal-based Python session, type `exit()` to go back to the regular command line.\n", - "\n", - "### Installing Packages\n", - "\n", - "To install extra Python packages, the default and easiest way is to use `uv add **packagename**`. There are over 330,000 Python packages on PyPI (the Python Package Index)! You can see what packages you have installed already by running `uv pip list` into the command line.\n", - "\n", - "`uv add ...` will install packages into the special Python environment in your current folder (it sits in a subdirectory called \".venv\" which will be hidden by default on most systems.) It's really helpful and good practice to have one Python environment per project, and **uv** does this automatically for you.\n", - "\n", - "::: {.callout-tip title=\"Exercise\"}\n", - "Try installing the **matplotlib**, **pandas**, **statsmodels**, and **skimpy** packages using `uv add`.\n", - ":::\n", - "\n", - "### Using Packages\n", - "\n", - "Once you have installed a package, you need to be able to use it! This is usually done via an import statement at the top of your script or Jupyter Notebook. For example, to bring in **pandas**, it's\n", - "\n", - "```python\n", - "import pandas as pd\n", - "```\n", - "\n", - "Why does Python do this? The idea of not just loading every package is to provide clarity over what function is being called from what package. It's also not necessary to load every package for every piece of analysis, and you often actually want to know what the *minimum* set of packages is to reproduce an analysis. Making the package imports explicit helps with all of that.\n", - "\n", - "You may also wonder why one doesn't just use `import pandas as pandas`. There's actually nothing stopping you doing this except i) it's convenient to have a shorter name and ii) there does tend to be a convention around imports, ie `pd` for **pandas** and `np` for **numpy**, and your code will be clearer to yourself and others if you follow the conventions.\n", - "\n", - "## Virtual Code Environments\n", - "\n", - "Virtual code environments allow you to isolate all of the packages that you're using to do analysis for one project from the set of packages you might need for a different project. They're an important part of creating a reproducible analytical pipeline but a key benefit is that others can reproduce the environment you used and it's best practice to have an isolated environment per project.\n", - "\n", - "To be more concrete, let's say you're using Python 3.9, **statsmodels**, and **pandas** for one project, project A. And, for project B, you need to use Python 3.10 with **numpy** and **scikit-learn**. Even with the same version of Python, best practice would be to have two separate virtual Python environments: environment A, with everything needed for project A, and environment B, with everything needed for project B. For the case where you're using different versions of Python, this isn't just best practice, it's essential.\n", - "\n", - "Many programming languages now come with an option to install packages and a version of the language in isolated environments. In Python, there are multiple tools for managing different environments. And, of those, the easiest to work with is probably [**uv**](https://docs.astral.sh/uv/).\n", - "\n", - "You can see all of the packages in the environment created in your current folder by running `uv pip list` on the command line. Here's an example of looking at the installed packages within this very book, filtering them just to the ones beginning with \"s\".\n", - "\n", - "```{bash}\n", - "uv run pip list | grep ^s\n", - "```\n", - "\n", - "### The pyproject.toml file in Python Environments\n", - "\n", - "You may have noticed that a file called `pyproject.toml` has been created." - ], - "id": "8b889898" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "import toml\n", - "from rich import print_json\n", - "\n", - "print_json(data=toml.load(\"pyproject.toml\"))" - ], - "id": "688f09f1", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This lists all of the dependencies, and the version, of a **uv** Python project. There are lots of benefits to tracking what versions of packages you're using like this. One of the most important is that you can *share* projects with other people, and they can install them from these files too.\n", - "\n", - "As you install or remove packages, the `pyproject.toml` file changes in lockstep.\n", - "\n", - "Noe that Visual Studio Code shows which Python environment you are using when you open a Python script or Jupyter Notebook.\n", - "\n", - "![A typical user view in Visual Studio Code](https://github.com/aeturrell/coding-for-economists/blob/main/img/vscode_layout.png?raw=true)\n", - "\n", - "In the screenshot above, you can see the project-environment in two places: on the blue bar at the bottom of the screen, and (in 5), at the top right hand side of the interactive window. A similar top right indicator is present when you have a Jupyter Notebook open too." - ], - "id": "148595b3" - } - ], - "metadata": { - "kernelspec": { - "name": "python3", - "language": "python", - "display_name": "Python 3 (ipykernel)", - "path": "/Users/omagic/Documents/GitHub/python4DSpolars/.venv/share/jupyter/kernels/python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file From 6ef280e3cc2d3edf6380c28619b26ac1ae7b2ec9 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 23 Jul 2026 16:59:23 +0100 Subject: [PATCH 23/30] fix: addres review feedback for quarto.md and prerequisites --- .gitignore | 4 +++- data/bake_sale.xlsx | Bin 6192 -> 6192 bytes prerequisites.ipynb | 2 +- quarto.qmd | 9 ++++----- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/.gitignore b/.gitignore index af04659..3246836 100644 --- a/.gitignore +++ b/.gitignore @@ -4,7 +4,7 @@ # quarto _book/ /.quarto/ -**/*.quarto_ipynb +**/*.quarto_ipynb* _freeze/ .panache* @@ -137,3 +137,5 @@ dmypy.json # Pyre type checker .pyre/ + +**/*.quarto_ipynb diff --git a/data/bake_sale.xlsx b/data/bake_sale.xlsx index 5de6f9a30425e19aa46b807093c289a907c83332..5c410a0caa27984691f6185779001c5e4c0d01f6 100644 GIT binary patch delta 324 zcmV-K0lWUNFt9MN91;Xvm^m1eAQB&c&1%Ci5QOi23ZuI%%MNJ~wgYX+sewXWT6z=G zCK0uMkkmSU`zlW2B#={&tC{_Fc2Tw1sV?~i=cUm#B3X_S(A)^AH#K^1m+1pad|+Cz z&S;ZUhL9p#udvcuX>a&)xt|`I`-+a^X}m3>I!$ zPvYd>c{v&TzR&t{w8s1df39BM##K|P_ow23no4lW9k5_rQ{l@Z93ZuO(+i6V@wgYX+s)0gXTDlX` zBoVb9NNSz@eHFiwKvrFj?wvDt29>LWY@=VWPH0u3EJ;xWifJy?u0rq4I$om4d8#;V zwSo#A!J$=sS29DGw(x4T0qX_0D1ekB%v5Oay&)Jowg*XF5*#XsTWuxvp|m@HY$*Gt zJHUCG7FdF(oO+5+z}U=0G$J{hk>UbxF7Ats#GgKQ}LL{i?B0r&BRNb;%iF7O2;@F7ag%4%brWH{rxv0sQe8 z^c!#EB10eWC Date: Sat, 25 Jul 2026 00:42:40 +0100 Subject: [PATCH 24/30] updated convert data-tidy.ipynb to polars --- data-tidy.ipynb | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/data-tidy.ipynb b/data-tidy.ipynb index 8ce90e0..a1c90b2 100644 --- a/data-tidy.ipynb +++ b/data-tidy.ipynb @@ -11,7 +11,7 @@ "\n", "In this chapter, you will learn a consistent way to organise your data in Python using the principle known as *tidy data*. Tidy data is not appropriate for everything, but for a lot of analysis and a lot of tabular data it will be what you need. Getting your data into this format requires some work up front, but that work pays off in the long term. Once you have tidy data, you will spend much less time munging data from one representation to another, allowing you to spend more time on the data questions you care about.\n", "\n", - "In this chapter, you'll first learn the definition of tidy data and see it applied to simple toy dataset. Then we'll dive into the main tool you'll use for tidying data: melting. Melting allows you to change the form of your data, without changing any of the values. We'll finish up with a discussion of usefully untidy data, and how you can create it if needed.\n", + "In this chapter, you'll first learn the definition of tidy data and see it applied to simple toy dataset. Then we'll dive into the main tool you'll use for tidying data: unpivoting. Unpivoting allows you to change wide data into a long format without altering any underlying values. We'll finish up with a discussion of usefully untidy data, and how you can create it if needed.\n", "\n", "If you particularly enjoy this chapter and want to learn more about the underlying theory, you can learn more in the [Tidy Data](https://www.jstatsoft.org/article/view/v059i10) paper published in the Journal of Statistical Software.\n" ] @@ -127,9 +127,9 @@ " \"weight\": [130, 150],\n", " }\n", ")\n", - "print(\"\\n Unmelted: \")\n", + "print(\"\\n Wide DataFrame: \")\n", "print(df)\n", - "print(\"\\n Melted: \")\n", + "print(\"\\n Unpivoted (Long) DataFrame: \")\n", "df.unpivot(\n", " index=[\"first\", \"last\"],\n", " variable_name=\"quantity\",\n", @@ -170,7 +170,7 @@ "id": "9", "metadata": {}, "source": [ - "You can see that we have two columns for a single variable, year. Let's now melt this." + "You can see that we have two columns for a single variable, year. Let's now unpivot this." ] }, { @@ -259,7 +259,7 @@ "\n", "df_final = df_long.drop(\"variable_year\").pivot(\n", " index=[\"id\", \"X\", \"year\"],\n", - " columns=\"variable\",\n", + " on=\"variable\",\n", " values=\"value\",\n", " aggregate_function=\"first\",\n", ")\n", @@ -272,11 +272,11 @@ "id": "16", "metadata": {}, "source": [ - "### Stack and Unstack\n", + "### Reshaping Multi-Column Data\n", "\n", "Polars uses unpivot() and pivot() for reshaping. \n", "\n", - "Stack is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", + "In **polars**, reshaping between long and wide tabular formats is performed seamlessly using `unpivot()` (to go long) and `pivot()` (to go wide), without needing row indices or complex MultiIndexes.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_stack.png)\n", "\n", @@ -349,7 +349,7 @@ "# Pivot back to wider format\n", "df_unstacked = df_stacked.pivot(\n", " index=[\"first\", \"second\"],\n", - " columns=\"variable\",\n", + " on=\"variable\",\n", " values=\"value\",\n", " aggregate_function=\"first\",\n", ")\n", @@ -402,7 +402,7 @@ "source": [ "pivoted = df_tb_cp.pivot(\n", " index=[\"country\", \"year\"],\n", - " columns=\"type\",\n", + " on=\"type\",\n", " values=\"count\",\n", " aggregate_function=\"first\",\n", ").sort([\"country\", \"year\"])\n", @@ -425,15 +425,21 @@ "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", + "\n", + "dates = list(\n", + " pl.date_range(\n", + " start=pl.date(2000, 1, 31),\n", + " end=pl.date(2000, 10, 31),\n", + " interval=\"1mo\",\n", + " eager=True,\n", + " )\n", + ")\n", "\n", "data = {\n", " \"value\": np.random.randn(20),\n", " \"variable\": [\"A\"] * 10 + [\"B\"] * 10,\n", - " \"date\": (\n", - " list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", - " + list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", - " ),\n", + " \"date\": dates + dates,\n", "}\n", "df = pl.DataFrame(data)\n", "df.sample(5)" @@ -456,7 +462,7 @@ "source": [ "df_pivoted = df.pivot(\n", " index=\"date\",\n", - " columns=\"variable\",\n", + " on=\"variable\",\n", " values=\"value\",\n", " aggregate_function=\"first\",\n", ").with_columns(\n", @@ -474,12 +480,12 @@ "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", - "Why is the first entry NaN?\n", + "Why is the first entry `null`?\n", ":::\n", "\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", - "Perform a `pivot()` that applies to both the `variable` and `category` columns in the example from above where category is defined such that `df[\"category\"] = np.random.choice([\"type1\", \"type2\", \"type3\", \"type4\"], 20). (Hint: remember that you will need to pass multiple objects via a list.)\n", + "Perform a `pivot()` that applies to both the `variable` and `category` columns in the example from above where category is defined such that `df = df.with_columns(category=pl.Series(np.random.choice([\"type1\", \"type2\", \"type3\", \"type4\"], 20)))`. (Hint: pass multiple column names to the `on` parameter as a list.)\n", ":::" ] } @@ -492,7 +498,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, From 5717402f0b9b0fee31f4ad533eab90902a95ae36 Mon Sep 17 00:00:00 2001 From: Uchenna Emmanuel Ugoh <61969079+ugohuche@users.noreply.github.com> Date: Sat, 25 Jul 2026 23:39:43 +0100 Subject: [PATCH 25/30] Add database connection closure and disposal --- databases.ipynb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/databases.ipynb b/databases.ipynb index b183959..d22a460 100644 --- a/databases.ipynb +++ b/databases.ipynb @@ -233,6 +233,7 @@ "\n", "import os\n", "\n", + "con_new.close()\n", "os.remove(\"data/test_database.sqlite\")" ] }, @@ -755,6 +756,8 @@ "source": [ "# remove-cell\n", "\n", + "con.close()\n", + "engine.dispose()\n", "os.remove(\"data/hero.db\")" ] }, From 5b7de6fb849a991e8c9f9257c7587cf49defadcf Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 28 Jul 2026 00:41:12 +0100 Subject: [PATCH 26/30] Updated pandas to polars migration on Strings --- strings.ipynb | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/strings.ipynb b/strings.ipynb index bab55e9..82e0496 100644 --- a/strings.ipynb +++ b/strings.ipynb @@ -13,7 +13,11 @@ "\n", "This chapter has benefitted from the [Python String Cook Book](https://mkaz.blog/code/python-string-format-cookbook/) and Jake VanderPlas' [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html).\n", "\n", - "Note that there are more powerful methods for working with strings called *regular expressions* but these will be covered in a different chapter." + "Note that there are more powerful methods for working with strings called *regular expressions* but these will be covered in a different chapter.\n", + "\n", + "### Prerequisites\n", + "\n", + "This chapter will use standard Python for scalar string manipulation and **polars** for vectorized string operations on data frames and series." ] }, { @@ -852,21 +856,21 @@ "\n", "| Function (preceded by `.str.`) | What it does |\n", "|-----------------------------|-------------------------|\n", - "| `len()` | Length of string. |\n", - "| `to_lower()` | Put string in lower case. |\n", - "| `to_upper()` | Put string in upper case. |\n", + "| `len_chars()` | Length of string in Unicode characters. |\n", + "| `to_lowercase()` | Put string in lower case. |\n", + "| `to_uppercase()` | Put string in upper case. |\n", "| `to_titlecase()` | Put string in leading upper case. |\n", - "| `strip()` | Strip out leading and trailing whitespace |\n", - "| `rstrip()` | Strip out trailing whitespace |\n", - "| `lstrip()` | Strip out leading whitespace |\n", - "| `find()` | Return the lowest index in the data where a substring appears |\n", + "| `strip_chars()` | Strip out leading and trailing whitespace or specified characters |\n", + "| `strip_chars_end()` | Strip out trailing whitespace or specified characters |\n", + "| `strip_chars_start()` | Strip out leading whitespace or specified characters |\n", + "| `find()` | Return the lowest index in the data where a substring or pattern appears |\n", "| `split()` | Split the string using a passed substring as the delimiter |\n", - "| `startswith()` | Check whether string starts with a given sub-string |\n", + "| `starts_with()` | Check whether string starts with a given sub-string |\n", "| `ends_with()` | Check whether string ends with a given sub-string |\n", "| `contains()` | Check whether string contains a given pattern |\n", - "| `replace()` | Replace occurrences of pattern with another string |\n", + "| `replace()` | Replace first occurrence of pattern with another string |\n", "| `replace_all()` | Replace all occurrences of pattern |\n", - "| `slice()` | Extract a substring by position |\n", + "| `slice()` | Extract a substring by position and length |\n", "\n", "Regular expressions can also be scaled up with **polars**. The below table shows vectorised regular expressions.\n", "\n", @@ -943,14 +947,13 @@ "\n", "| Method | Description |\n", "|-|-|\n", - "| `slice()` | Slice each element |\n", - "| `concat()` | Concatenate strings |\n", - "| `repeat()` | Repeat values |\n", + "| `slice()` | Slice each string element by offset and length |\n", + "| `concat_str()` | Concatenate strings across columns or expressions |\n", + "| `repeat_by()` | Repeat values |\n", "| `contains()` | Check if pattern exists |\n", "\n", "\n", - "The `slice()` method gives access to substrings by position:\n", - "\n" + "The `slice()` method gives access to substrings by position:\n" ] }, { From bbf049148b83f17264cd7a277449c7efc79527a7 Mon Sep 17 00:00:00 2001 From: Nwabueze Ugoh Date: Wed, 29 Jul 2026 11:51:50 +0100 Subject: [PATCH 27/30] fix: restore last-token split example in strings chapter Show both str.slice and split().list.get(-1), and clarify fixed-field splitting versus dummy encoding. Co-authored-by: Cursor --- strings.ipynb | 203 +++++++++++++++++++++++++------------------------- 1 file changed, 103 insertions(+), 100 deletions(-) diff --git a/strings.ipynb b/strings.ipynb index 82e0496..97b44d2 100644 --- a/strings.ipynb +++ b/strings.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Strings and Text {#sec-strings}\n", @@ -22,7 +22,7 @@ }, { "cell_type": "markdown", - "id": "3fc1cd0f", + "id": "1", "metadata": {}, "source": [ "## Creating Strings\n", @@ -33,7 +33,7 @@ { "cell_type": "code", "execution_count": null, - "id": "478a847a", + "id": "2", "metadata": {}, "outputs": [], "source": [ @@ -45,7 +45,7 @@ }, { "cell_type": "markdown", - "id": "9a146322", + "id": "3", "metadata": {}, "source": [ "Strings are of type `str`:" @@ -54,7 +54,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d7f4ea2d", + "id": "4", "metadata": {}, "outputs": [], "source": [ @@ -63,7 +63,7 @@ }, { "cell_type": "markdown", - "id": "2bc26125", + "id": "5", "metadata": {}, "source": [ "Strings in Python can be indexed, so we can get certain characters out by using square brackets to say which positions we would like." @@ -72,7 +72,7 @@ { "cell_type": "code", "execution_count": null, - "id": "01379fe7", + "id": "6", "metadata": {}, "outputs": [], "source": [ @@ -82,7 +82,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d88f7928", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -91,7 +91,7 @@ }, { "cell_type": "markdown", - "id": "d76d0c0a", + "id": "8", "metadata": {}, "source": [ "The usual slicing tricks that apply to lists work for strings too, i.e. the positions you want to get can be retrieved using the `var[start:stop:step]` syntax. Here's an example of getting every other character from the string starting from the 2nd position." @@ -100,7 +100,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e03d95d1", + "id": "9", "metadata": {}, "outputs": [], "source": [ @@ -109,7 +109,7 @@ }, { "cell_type": "markdown", - "id": "ddd1bbe7", + "id": "10", "metadata": {}, "source": [ "Note that strings, like tuples such as `(1, 2, 3)` but unlike lists such as `[1, 2, 3]`, are *immutable*. This means commands like `var[0] = \"B\"` will result in an error. If you want to change a single character, you will have to replace the entire string. In this example, the command to do that would be `var = \"Banana\"`.\n", @@ -120,7 +120,7 @@ { "cell_type": "code", "execution_count": null, - "id": "83ab201b", + "id": "11", "metadata": {}, "outputs": [], "source": [ @@ -129,7 +129,7 @@ }, { "cell_type": "markdown", - "id": "7eeebc05", + "id": "12", "metadata": {}, "source": [ "You can concatenate strings using the `+` operator:" @@ -138,7 +138,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7801bd5d", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -147,7 +147,7 @@ }, { "cell_type": "markdown", - "id": "e8fb0ab8", + "id": "14", "metadata": {}, "source": [ "Note that we added extra characters so that the phrase made sense. Another way of achieving the same end that scales to many words or phrases more efficiently (if you have them in a list) is:" @@ -156,7 +156,7 @@ { "cell_type": "code", "execution_count": null, - "id": "138cef18", + "id": "15", "metadata": {}, "outputs": [], "source": [ @@ -165,7 +165,7 @@ }, { "cell_type": "markdown", - "id": "77b33c02", + "id": "16", "metadata": {}, "source": [ "Three useful functions to know about are `upper()`, `lower()`, and `title()`. Let's see what they do\n" @@ -174,7 +174,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e11896f8", + "id": "17", "metadata": {}, "outputs": [], "source": [ @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "505bec9e", + "id": "18", "metadata": {}, "source": [ "Note that there are many built-in functions for using strings in Python, you can find a comprehensive list [here](https://www.w3schools.com/python/python_ref_string.asp).\n", @@ -197,7 +197,7 @@ }, { "cell_type": "markdown", - "id": "3d70a3f6", + "id": "19", "metadata": {}, "source": [ "While we're using `print()`, it has a few tricks. If we have a list, we can print out entries with a given separator:\n" @@ -206,7 +206,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bf0aadec", + "id": "20", "metadata": {}, "outputs": [], "source": [ @@ -215,7 +215,7 @@ }, { "cell_type": "markdown", - "id": "08d9b5af", + "id": "21", "metadata": {}, "source": [ "(We'll find out more about what '\\n' does shortly.) To turn variables of other kinds into strings, use the `str()` function, for example" @@ -224,7 +224,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a96f048c", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -241,7 +241,7 @@ }, { "cell_type": "markdown", - "id": "9c79246f", + "id": "23", "metadata": {}, "source": [ "In this example two boolean variables and one integer variable were converted to strings. `str()` generally makes an intelligent guess at how you'd like to convert your non-string type variable into a string type. You can pass a variable or a literal value to `str()`." @@ -249,7 +249,7 @@ }, { "cell_type": "markdown", - "id": "e68633d9", + "id": "24", "metadata": {}, "source": [ "### f-strings\n", @@ -260,7 +260,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9dddf0da", + "id": "25", "metadata": {}, "outputs": [], "source": [ @@ -270,7 +270,7 @@ }, { "cell_type": "markdown", - "id": "027287af", + "id": "26", "metadata": {}, "source": [ "This is similar to calling `str` on variable and using `+` for concatenation but much shorter to write. You can add expressions to f-strings too:" @@ -279,7 +279,7 @@ { "cell_type": "code", "execution_count": null, - "id": "795e7c07", + "id": "27", "metadata": {}, "outputs": [], "source": [ @@ -288,7 +288,7 @@ }, { "cell_type": "markdown", - "id": "fe120dd7", + "id": "28", "metadata": {}, "source": [ "This also works with functions; after all `**2` is just a function with its own special syntax." @@ -296,7 +296,7 @@ }, { "cell_type": "markdown", - "id": "f3524d86", + "id": "29", "metadata": {}, "source": [ "In this example, the score number that came out had a lot of (probably) uninteresting decimal places. So how do we polish the printed output? You can pass more inforation to the f-string to get the output formatted just the way you want. Let's say we wanted two decimal places and a sign (although you always write `+` in the formatting, the sign comes out as + or - depending on the value):" @@ -305,7 +305,7 @@ { "cell_type": "code", "execution_count": null, - "id": "1f3d3806", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -314,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "04eb84a8", + "id": "31", "metadata": {}, "source": [ "There are a whole range of formatting options for numbers as shown in the following table:\n", @@ -340,7 +340,7 @@ }, { "cell_type": "markdown", - "id": "6ae30791", + "id": "32", "metadata": {}, "source": [ "### Special Characters and How to Escape Strings" @@ -348,7 +348,7 @@ }, { "cell_type": "markdown", - "id": "2fb21df1", + "id": "33", "metadata": {}, "source": [ "Python has a string module that comes with some useful built-in strings and characters. For example" @@ -357,7 +357,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0ccd65aa", + "id": "34", "metadata": {}, "outputs": [], "source": [ @@ -368,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "7e9646e6", + "id": "35", "metadata": {}, "source": [ "gives you all of the punctuation," @@ -377,7 +377,7 @@ { "cell_type": "code", "execution_count": null, - "id": "16205c36", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -386,7 +386,7 @@ }, { "cell_type": "markdown", - "id": "1cb1dac5", + "id": "37", "metadata": {}, "source": [ "returns all of the basic letters in the 'ASCII' encoding (with `.ascii_lowercase` and `.ascii_uppercase` variants), and" @@ -395,7 +395,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0c67f5cd", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -404,7 +404,7 @@ }, { "cell_type": "markdown", - "id": "589fff6d", + "id": "39", "metadata": {}, "source": [ "gives you the numbers from 0 to 9. Finally, though less impressive visually, `string.whitespace` gives a string containing all of the different (there is more than one!) types of whitespace." @@ -412,7 +412,7 @@ }, { "cell_type": "markdown", - "id": "794c4f7e", + "id": "40", "metadata": {}, "source": [ "There are other special characters around; in fact, we already met the most famous of them: \"\\n\" for new line. To actually print \"\\n\" we have to 'escape' the backward slash by adding another backward slash:" @@ -421,7 +421,7 @@ { "cell_type": "code", "execution_count": null, - "id": "16e9904a", + "id": "41", "metadata": {}, "outputs": [], "source": [ @@ -431,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "cf878d74", + "id": "42", "metadata": {}, "source": [ "The table below shows the most important escape commands:\n", @@ -448,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "c7dd19ac", + "id": "43", "metadata": {}, "source": [ "Here's a more complicated example:" @@ -457,7 +457,7 @@ { "cell_type": "code", "execution_count": null, - "id": "af423bd1", + "id": "44", "metadata": {}, "outputs": [], "source": [ @@ -466,7 +466,7 @@ }, { "cell_type": "markdown", - "id": "16c97e01", + "id": "45", "metadata": {}, "source": [ "### Raw Strings\n", @@ -477,7 +477,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c2b9c689", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -486,7 +486,7 @@ }, { "cell_type": "markdown", - "id": "98d896ed", + "id": "47", "metadata": {}, "source": [ "## Cleaning Text\n", @@ -501,7 +501,7 @@ { "cell_type": "code", "execution_count": null, - "id": "229ada3a", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -510,7 +510,7 @@ }, { "cell_type": "markdown", - "id": "ce0467a1", + "id": "49", "metadata": {}, "source": [ "As with any variable of a specific type (here, string), this would also work with variables:" @@ -519,7 +519,7 @@ { "cell_type": "code", "execution_count": null, - "id": "79f754dc", + "id": "50", "metadata": {}, "outputs": [], "source": [ @@ -531,7 +531,7 @@ }, { "cell_type": "markdown", - "id": "5c5ab4c1", + "id": "51", "metadata": {}, "source": [ "Note that `.replace()` performs an exact replace and so is case-sensitive." @@ -539,7 +539,7 @@ }, { "cell_type": "markdown", - "id": "85bba3f9", + "id": "52", "metadata": {}, "source": [ "### Replacing characters with translate\n", @@ -554,7 +554,7 @@ { "cell_type": "code", "execution_count": null, - "id": "99675fee", + "id": "53", "metadata": {}, "outputs": [], "source": [ @@ -566,7 +566,7 @@ }, { "cell_type": "markdown", - "id": "391d2748", + "id": "54", "metadata": {}, "source": [ "Now we turn our dictionary into a string translator and apply it to our text:\n" @@ -575,7 +575,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e48763cb", + "id": "55", "metadata": {}, "outputs": [], "source": [ @@ -585,7 +585,7 @@ }, { "cell_type": "markdown", - "id": "1c0359ed", + "id": "56", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -595,7 +595,7 @@ }, { "cell_type": "markdown", - "id": "f27549e8", + "id": "57", "metadata": {}, "source": [ "Generally, `str.translate` is very fast at replacing individual characters in strings. But you can also do it using a list comprehension and a `join()` of the resulting list, like so:" @@ -604,7 +604,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ac758b38", + "id": "58", "metadata": {}, "outputs": [], "source": [ @@ -619,7 +619,7 @@ }, { "cell_type": "markdown", - "id": "cbe854ee", + "id": "59", "metadata": {}, "source": [ "### Splitting strings\n", @@ -630,7 +630,7 @@ { "cell_type": "code", "execution_count": null, - "id": "122619bf", + "id": "60", "metadata": {}, "outputs": [], "source": [ @@ -639,7 +639,7 @@ }, { "cell_type": "markdown", - "id": "0ddc2c15", + "id": "61", "metadata": {}, "source": [ "Next up we can use the built-in `split()` function, which returns a list of places where a given sub-string occurs:\n" @@ -648,7 +648,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9fc432ed", + "id": "62", "metadata": {}, "outputs": [], "source": [ @@ -657,7 +657,7 @@ }, { "cell_type": "markdown", - "id": "992b6a79", + "id": "63", "metadata": {}, "source": [ "Note that the character used to split the string is removed from the resulting list of strings. Let's see an example with a string used for splitting instead of a single character:\n" @@ -666,7 +666,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6904e486", + "id": "64", "metadata": {}, "outputs": [], "source": [ @@ -675,7 +675,7 @@ }, { "cell_type": "markdown", - "id": "5d7beb3c", + "id": "65", "metadata": {}, "source": [ "A useful extra function to know about is `splitlines()`, which splits a string at line breaks and returns the split parts as a list." @@ -683,7 +683,7 @@ }, { "cell_type": "markdown", - "id": "752b9704", + "id": "66", "metadata": {}, "source": [ "### count and find\n", @@ -694,7 +694,7 @@ { "cell_type": "code", "execution_count": null, - "id": "22f94993", + "id": "67", "metadata": {}, "outputs": [], "source": [ @@ -705,7 +705,7 @@ }, { "cell_type": "markdown", - "id": "378dade0", + "id": "68", "metadata": {}, "source": [ "Meanwhile, `find()` returns the position where a particular word or character occurs." @@ -714,7 +714,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a351a11b", + "id": "69", "metadata": {}, "outputs": [], "source": [ @@ -723,7 +723,7 @@ }, { "cell_type": "markdown", - "id": "7fb9ae26", + "id": "70", "metadata": {}, "source": [ "We can check this using the number we get and some string indexing:" @@ -732,7 +732,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8e0a7020", + "id": "71", "metadata": {}, "outputs": [], "source": [ @@ -741,7 +741,7 @@ }, { "cell_type": "markdown", - "id": "ecc16d15", + "id": "72", "metadata": {}, "source": [ "But this isn't the only place where the word 'coffee' appears. If we want to find the last occurrence, it's" @@ -750,7 +750,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e18f64a3", + "id": "73", "metadata": {}, "outputs": [], "source": [ @@ -759,7 +759,7 @@ }, { "cell_type": "markdown", - "id": "f30ef28a", + "id": "74", "metadata": {}, "source": [ "## Working with Multiple Strings" @@ -767,7 +767,7 @@ }, { "cell_type": "markdown", - "id": "09f7c915", + "id": "75", "metadata": {}, "source": [ "We've seen how to work with individual strings. But often we want to work with a group of strings, otherwise known as a corpus, that is a collection of texts. It could be a collection of words, sentences, paragraphs, or some domain-based grouping (eg job descriptions). Just like any other Python object, you can put strings into a list (or other iterable).\n", @@ -780,7 +780,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bbc3eb7b", + "id": "76", "metadata": {}, "outputs": [], "source": [ @@ -789,7 +789,7 @@ }, { "cell_type": "markdown", - "id": "730df2c8", + "id": "77", "metadata": {}, "source": [ "A **polars** series can be used in place of a list. Let's create the series first:" @@ -798,7 +798,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c8a7f68b", + "id": "78", "metadata": {}, "outputs": [], "source": [ @@ -812,7 +812,7 @@ }, { "cell_type": "markdown", - "id": "222fbb51", + "id": "79", "metadata": {}, "source": [ "Now we use the syntax `.str.function` to change the text series:\n" @@ -821,7 +821,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7cf149b5", + "id": "80", "metadata": {}, "outputs": [], "source": [ @@ -830,7 +830,7 @@ }, { "cell_type": "markdown", - "id": "ec439fc1", + "id": "81", "metadata": {}, "source": [ "If we had a data frame and not a series, the syntax would change to refer just to the column of interest like so:" @@ -839,7 +839,7 @@ { "cell_type": "code", "execution_count": null, - "id": "26dc9a7b", + "id": "82", "metadata": {}, "outputs": [], "source": [ @@ -849,7 +849,7 @@ }, { "cell_type": "markdown", - "id": "2e430dd9", + "id": "83", "metadata": {}, "source": [ "The table below shows a non-exhaustive list of the string methods that are available in **polars**.\n", @@ -889,7 +889,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d7a29663", + "id": "84", "metadata": {}, "outputs": [], "source": [ @@ -898,7 +898,7 @@ }, { "cell_type": "markdown", - "id": "597c0d23", + "id": "85", "metadata": {}, "source": [ "It's fairly common that you want to split out strings and save the results to new columns in your data frame. You can get the columns using `list` and `to_struct()`:" @@ -907,7 +907,7 @@ { "cell_type": "code", "execution_count": null, - "id": "85a5cd2c", + "id": "86", "metadata": {}, "outputs": [], "source": [ @@ -918,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "e6bb64b2", + "id": "87", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -931,7 +931,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2e8781ba", + "id": "88", "metadata": {}, "outputs": [], "source": [ @@ -940,7 +940,7 @@ }, { "cell_type": "markdown", - "id": "7fb6d243", + "id": "89", "metadata": {}, "source": [ "There are a few more vectorised string operations that are useful.\n", @@ -953,31 +953,34 @@ "| `contains()` | Check if pattern exists |\n", "\n", "\n", - "The `slice()` method gives access to substrings by position:\n" + "The `slice()` method gives access to substrings by position. When strings contain delimited parts, you can instead combine `str.split()` with `list.get()` to select a part such as the last token:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "ba13d894", + "id": "90", "metadata": {}, "outputs": [], "source": [ - "df.with_columns(pl.col(\"names\").str.slice(0, 5))" + "df.with_columns(\n", + " pl.col(\"names\").str.slice(0, 5).alias(\"first_five_characters\"),\n", + " pl.col(\"names\").str.split(\" \").list.get(-1).alias(\"last_name\"),\n", + ")" ] }, { "cell_type": "markdown", - "id": "4ce15edb", + "id": "91", "metadata": {}, "source": [ - "If we have a column with tags split by a symbol, we can use the `str.split()` to split it out. For example, let's create a data frame with a single column that mixes subject and nationality tags:\n" + "If we have a column whose values contain a fixed set of fields separated by a symbol, we can use `str.split()` to separate those fields. This differs from dummy encoding: it creates one column per position rather than one indicator column per unique value. For example, let's create a data frame with a single column that combines nationality and subject tags:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "056147d6", + "id": "92", "metadata": {}, "outputs": [], "source": [ @@ -998,17 +1001,17 @@ }, { "cell_type": "markdown", - "id": "f44b0d84", + "id": "93", "metadata": {}, "source": [ - "If we now use `str.split()` and then list operations we can get separate columns:\n", + "Because every value contains the fields in the same order, we can use `str.split()` and list operations to create separate `country` and `subject` columns:\n", "\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "a5cbc10f", + "id": "94", "metadata": {}, "outputs": [], "source": [ @@ -1019,7 +1022,7 @@ }, { "cell_type": "markdown", - "id": "58521740", + "id": "95", "metadata": {}, "source": [ "## Reading Text In\n", From 1a032ae68470bdcee42d52598b75dc8b882cd622 Mon Sep 17 00:00:00 2001 From: Your Name Date: Mon, 13 Jul 2026 02:59:23 +0100 Subject: [PATCH 28/30] refactor: migrate exploratory data analysis examples to polars --- exploratory-data-analysis.ipynb | 374 ++++++++++++++++++-------------- 1 file changed, 209 insertions(+), 165 deletions(-) diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index 071a96b..eae03ec 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Exploratory Data Analysis {#sec-exploratory-data-analysis}\n", @@ -21,7 +21,7 @@ "\n", "### Prerequisites\n", "\n", - "For doing EDA, we'll use the **pandas**, **skimpy**, and **pandas-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", + "For doing EDA, we'll use the **polars**, **skimpy**, and **polars-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", "\n", "As ever, we begin by loading these packages that we'll use:" ] @@ -29,14 +29,13 @@ { "cell_type": "code", "execution_count": null, - "id": "a3377aa6", + "id": "1", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "from lets_plot import *\n", "from lets_plot.mapping import as_discrete\n", - "from pandas.api.types import CategoricalDtype\n", "from skimpy import skim\n", "\n", "LetsPlot.setup_html()" @@ -45,7 +44,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "2", "metadata": { "tags": [ "remove-cell" @@ -56,14 +55,13 @@ "import matplotlib.pyplot as plt\n", "import matplotlib_inline.backend_inline\n", "\n", - "# Plot settings\n", "plt.style.use(\"https://github.com/aeturrell/python4DS/raw/main/plot_style.txt\")\n", "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")" ] }, { "cell_type": "markdown", - "id": "e4ddb863", + "id": "3", "metadata": {}, "source": [ "## Questions\n", @@ -90,7 +88,7 @@ }, { "cell_type": "markdown", - "id": "6adeff41", + "id": "4", "metadata": {}, "source": [ "## Variation\n", @@ -103,27 +101,28 @@ { "cell_type": "code", "execution_count": null, - "id": "069caa7c", + "id": "5", "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", + "diamonds = pl.read_csv(\n", " \"https://github.com/mwaskom/seaborn-data/raw/master/diamonds.csv\"\n", ")\n", - "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n", - " CategoricalDtype(\n", - " categories=[\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"], ordered=True\n", - " )\n", - ")\n", - "diamonds[\"color\"] = diamonds[\"color\"].astype(\n", - " CategoricalDtype(categories=[\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"], ordered=True)\n", - ")\n", + "\n", + "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", + "diamonds_color_order = [\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"]\n", + "\n", + "diamonds = diamonds.with_columns([\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", + " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", + "])\n", + "\n", "diamonds.head()" ] }, { "cell_type": "markdown", - "id": "01f5979c", + "id": "6", "metadata": {}, "source": [ "Since `\"carat\"` is a numerical variable, we can use a histogram:" @@ -132,7 +131,7 @@ { "cell_type": "code", "execution_count": null, - "id": "97900f58", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -141,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "2307ba7c", + "id": "8", "metadata": {}, "source": [ "Now that you can visualize variation, what should you look for in your plots?\n", @@ -173,18 +172,18 @@ { "cell_type": "code", "execution_count": null, - "id": "20d75550", + "id": "9", "metadata": {}, "outputs": [], "source": [ - "smaller_diamonds = diamonds.query(\"carat < 3\").copy()\n", + "smaller_diamonds = diamonds.filter(pl.col(\"carat\") < 3)\n", "\n", "(ggplot(smaller_diamonds, aes(x=\"carat\")) + geom_histogram(binwidth=0.01))" ] }, { "cell_type": "markdown", - "id": "ba20a0e2", + "id": "10", "metadata": {}, "source": [ "This histogram suggests several interesting questions:\n", @@ -211,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "0626d35a", + "id": "11", "metadata": {}, "source": [ "### Unusual values\n", @@ -226,7 +225,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d9d7e995", + "id": "12", "metadata": {}, "outputs": [], "source": [ @@ -235,7 +234,7 @@ }, { "cell_type": "markdown", - "id": "05bcf733", + "id": "13", "metadata": {}, "source": [ "There are so many observations in the common bins that the rare bins are very short, making it very difficult to see them (although maybe if you stare intently at 0 you'll spot something).\n", @@ -245,7 +244,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ea8f8bf3", + "id": "14", "metadata": {}, "outputs": [], "source": [ @@ -258,7 +257,7 @@ }, { "cell_type": "markdown", - "id": "ba2b2c79", + "id": "15", "metadata": {}, "source": [ "`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis.\n", @@ -270,17 +269,17 @@ { "cell_type": "code", "execution_count": null, - "id": "e81ffb55", + "id": "16", "metadata": {}, "outputs": [], "source": [ - "unusual = diamonds.query(\"y < 3 or y > 20\").loc[:, [\"x\", \"y\", \"z\", \"price\"]]\n", + "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select([\"x\", \"y\", \"z\", \"price\"])\n", "unusual" ] }, { "cell_type": "markdown", - "id": "c9321f36", + "id": "17", "metadata": {}, "source": [ "The `\"y\"` variable measures one of the three dimensions of these diamonds, in mm.\n", @@ -297,7 +296,7 @@ }, { "cell_type": "markdown", - "id": "142d21d7", + "id": "18", "metadata": {}, "source": [ "### Exercises\n", @@ -321,7 +320,7 @@ }, { "cell_type": "markdown", - "id": "c51e8d6f", + "id": "19", "metadata": {}, "source": [ "## Unusual Values\n", @@ -345,18 +344,16 @@ { "cell_type": "code", "execution_count": null, - "id": "ecf345a7", + "id": "20", "metadata": {}, "outputs": [], "source": [ - "diamonds2 = diamonds.copy()\n", - "condition = (diamonds2[\"y\"] < 3) | (diamonds2[\"y\"] > 20)\n", - "diamonds2.loc[condition, \"y\"] = pd.NA" + "diamonds2 = diamonds.filter(~((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)))" ] }, { "cell_type": "markdown", - "id": "d26d922e", + "id": "21", "metadata": {}, "source": [ "It's not obvious where you should plot missing values, so **lets-plot** doesn't include them in the plot:" @@ -365,7 +362,7 @@ { "cell_type": "code", "execution_count": null, - "id": "15a43255", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -374,7 +371,7 @@ }, { "cell_type": "markdown", - "id": "e8ae854b", + "id": "23", "metadata": {}, "source": [ "Other times you want to understand what makes observations with missing values different to observations with recorded values.\n", @@ -386,28 +383,34 @@ { "cell_type": "code", "execution_count": null, - "id": "0a4ea922", + "id": "24", "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", + "import polars as pl\n", + "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights = pd.read_csv(url)\n", + "flights_pd = pd.read_csv(url)\n", + "flights = pl.from_pandas(flights_pd)\n", "flights.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "6849f4d9", + "id": "25", "metadata": {}, "outputs": [], "source": [ - "flights2 = flights.assign(\n", - " cancelled=lambda x: pd.isna(x[\"dep_time\"]),\n", - " sched_hour=lambda x: x[\"sched_dep_time\"] // 100,\n", - " sched_min=lambda x: x[\"sched_dep_time\"] % 100,\n", - " sched_dep_time=lambda x: x[\"sched_hour\"] + x[\"sched_min\"] / 60,\n", - ")\n", + "flights2 = flights.with_columns([\n", + " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", + " ((pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60).alias(\"sched_dep_time\"),\n", + "])\n", + "\n", + "from lets_plot import *\n", "\n", "(\n", " ggplot(flights2, aes(x=\"sched_dep_time\"))\n", @@ -417,7 +420,7 @@ }, { "cell_type": "markdown", - "id": "b97e453a", + "id": "26", "metadata": {}, "source": [ "However this plot isn't great because there are many more non-cancelled flights than cancelled flights.\n", @@ -436,7 +439,7 @@ }, { "cell_type": "markdown", - "id": "f63d0b9f", + "id": "27", "metadata": {}, "source": [ "## Covariation\n", @@ -453,7 +456,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e1719d8f", + "id": "28", "metadata": {}, "outputs": [], "source": [ @@ -465,7 +468,7 @@ }, { "cell_type": "markdown", - "id": "20387b96", + "id": "29", "metadata": {}, "source": [ "The default appearance of `geom_freqpoly()` is not that useful here because the height, determined by the overall count, differs so much across cuts, making it hard to see the differences in the shapes of their distributions.\n", @@ -477,7 +480,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9388e24b", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -489,7 +492,7 @@ }, { "cell_type": "markdown", - "id": "157f63dd", + "id": "31", "metadata": {}, "source": [ "There’s something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that’s because density plots are a little hard to interpret - there’s a lot going on in this plot.\n", @@ -500,7 +503,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a3f333a6", + "id": "32", "metadata": {}, "outputs": [], "source": [ @@ -509,7 +512,7 @@ }, { "cell_type": "markdown", - "id": "b9d9ef00", + "id": "33", "metadata": {}, "source": [ "We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counter-intuitive finding that better quality diamonds are typically cheaper! In the exercises, you’ll be challenged to figure out why.\n", @@ -522,21 +525,22 @@ { "cell_type": "code", "execution_count": null, - "id": "6949db81", + "id": "34", "metadata": {}, "outputs": [], "source": [ - "mpg = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n", + "mpg = pl.read_csv(\n", + " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\"\n", ")\n", - "mpg[\"class\"] = mpg[\"class\"].astype(\"category\")\n", + "\n", + "mpg = mpg.with_columns(pl.col(\"class\").cast(pl.Categorical))\n", "\n", "(ggplot(mpg, aes(x=\"class\", y=\"hwy\")) + geom_boxplot())" ] }, { "cell_type": "markdown", - "id": "871aaf1c", + "id": "35", "metadata": {}, "source": [ "To make the trend easier to see, we can reorder class based on the median value of `\"hwy\"`:" @@ -545,7 +549,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a5b1ed09", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -554,7 +558,7 @@ }, { "cell_type": "markdown", - "id": "dde59236", + "id": "37", "metadata": {}, "source": [ "If you have long variable names, geom_boxplot() will work better if you flip it 90°. You can do that by adding `coord_flip()`." @@ -563,7 +567,7 @@ { "cell_type": "code", "execution_count": null, - "id": "920a4268", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -576,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "299635df", + "id": "39", "metadata": {}, "source": [ "#### Exercises\n", @@ -598,7 +602,7 @@ }, { "cell_type": "markdown", - "id": "6e6a0b46", + "id": "40", "metadata": {}, "source": [ "### Two categorical variables\n", @@ -609,20 +613,22 @@ { "cell_type": "code", "execution_count": null, - "id": "68d330d2", + "id": "41", "metadata": {}, "outputs": [], "source": [ - "ct_cut_color = pd.melt(\n", - " pd.crosstab(diamonds[\"cut\"], diamonds[\"color\"]).reset_index(),\n", - " id_vars=[\"cut\"],\n", - " value_vars=diamonds[\"color\"].unique(),\n", - ")" + "ct_cut_color = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", + ")\n", + "\n", + "ct_cut_color_long = ct_cut_color.melt(id_vars=\"cut\", variable_name=\"color\", value_name=\"value\")" ] }, { "cell_type": "markdown", - "id": "c2ebc0c6", + "id": "42", "metadata": {}, "source": [ "Followed by visualising it with `geom_tile()`:" @@ -631,16 +637,16 @@ { "cell_type": "code", "execution_count": null, - "id": "e858cd22", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "(ggplot(ct_cut_color, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" + "(ggplot(ct_cut_color_long, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" ] }, { "cell_type": "markdown", - "id": "8ccd5c1b", + "id": "44", "metadata": {}, "source": [ "### Exercises\n", @@ -654,7 +660,7 @@ }, { "cell_type": "markdown", - "id": "b2f19afc", + "id": "45", "metadata": {}, "source": [ "### Two numerical variables\n", @@ -668,7 +674,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2afe2535", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -677,7 +683,7 @@ }, { "cell_type": "markdown", - "id": "9d8034f2", + "id": "47", "metadata": {}, "source": [ "(In this section we'll use the `smaller_diamonds` dataset to stay focused on the bulk of the diamonds that are smaller than 3 carats)\n", @@ -689,7 +695,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b55707a9", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -698,7 +704,7 @@ }, { "cell_type": "markdown", - "id": "351c22bd", + "id": "49", "metadata": {}, "source": [ "But using transparency can be challenging for very large datasets. In that case, we recommend a *binscatter*, or binned scatterplot. A binned scatterplot divides the conditioning variable, `\"carat\"` in our example, into equally sized bins or quantiles, and then plots the conditional mean of the dependent variable, `\"price\"` in our example, within each bin. Bin scatters often come with confidence intervals too. A good bin scatter package in Python is [**binsreg**](https://nppackages.github.io/binsreg/). However, bin scatters are an advanced topic, and we won't cover them here." @@ -706,7 +712,7 @@ }, { "cell_type": "markdown", - "id": "14158b48", + "id": "50", "metadata": {}, "source": [ "## **pandas** built-in tools for EDA\n", @@ -721,7 +727,7 @@ { "cell_type": "code", "execution_count": null, - "id": "13079065", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -730,7 +736,7 @@ }, { "cell_type": "markdown", - "id": "db57dbb7", + "id": "52", "metadata": {}, "source": [ "Although helpful, that sure is hard to read! We can improve this by using the `round()` method too:\n" @@ -739,17 +745,23 @@ { "cell_type": "code", "execution_count": null, - "id": "b4144440", + "id": "53", "metadata": {}, "outputs": [], "source": [ - "sum_table = diamonds.describe().round(1)\n", + "sum_table = (\n", + " diamonds.describe()\n", + " .with_columns(\n", + " pl.selectors.numeric().round(1)\n", + " )\n", + ")\n", + "\n", "sum_table" ] }, { "cell_type": "markdown", - "id": "8063b8f5", + "id": "54", "metadata": {}, "source": [ "Published summary statistics tables often list one variable per row, and if your data frame has many variables, `describe()` can quickly get too wide to read easily. You can transpose it using the `T` property (or the `transpose()` method):" @@ -758,17 +770,17 @@ { "cell_type": "code", "execution_count": null, - "id": "cd2f8772", + "id": "55", "metadata": {}, "outputs": [], "source": [ - "sum_table = sum_table.T\n", - "sum_table" + "sum_table_long = sum_table.melt(id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\")\n", + "sum_table_long" ] }, { "cell_type": "markdown", - "id": "e67104d3", + "id": "56", "metadata": {}, "source": [ "Of course, the stats provided in this pre-built table are not very customised. So what do we do to get the table that we actually want? Well, the answer is to draw on the contents of the previous data chapters, particularly the introduction to data analysis. Groupbys, merges, aggregations: use all of them to produce the EDA table that you want.\n", @@ -781,24 +793,25 @@ { "cell_type": "code", "execution_count": null, - "id": "5afcacbc", + "id": "57", "metadata": {}, "outputs": [], "source": [ - "(\n", - " diamonds.groupby([\"cut\", \"color\"])[\"price\"]\n", - " .mean()\n", - " .unstack()\n", - " .apply(lambda x: x / 1e3)\n", - " .fillna(\"-\")\n", - " .style.format(precision=2)\n", - " .set_caption(\"Sale price (thousands)\")\n", - ")" + "import polars.selectors as cs\n", + "\n", + "price_summary = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.col(\"price\").mean())\n", + " .pivot(index=\"cut\", on=\"color\", values=\"price\")\n", + " .with_columns(cs.numeric().round(2))\n", + ")\n", + "\n", + "price_summary" ] }, { "cell_type": "markdown", - "id": "d16e0fc1", + "id": "58", "metadata": {}, "source": [ "Although a neater one than we've seen, this is still a drab table of numbers. The eye is not immediately drawn to it!\n", @@ -811,16 +824,28 @@ { "cell_type": "code", "execution_count": null, - "id": "21e65189", + "id": "59", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.background_gradient(cmap=\"plasma\")" + "ct_cut_color = (\n", + " diamonds.group_by([\"color\", \"cut\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(\n", + " index=\"color\",\n", + " on=\"cut\",\n", + " values=\"count\",\n", + " aggregate_function=\"sum\",\n", + " )\n", + " .with_columns(\n", + " cs.numeric().fill_null(0)\n", + " )\n", + ")" ] }, { "cell_type": "markdown", - "id": "b7c6aa71", + "id": "60", "metadata": {}, "source": [ "By default, `background_gradient()` highlights each number relative to the others in its column; you can highlight by row using `axis=1` or relative to all table values using `axis=0`. And of course `plasma` is just one of [many available colormaps](https://matplotlib.org/stable/tutorials/colors/colormaps.html)!\n", @@ -837,38 +862,41 @@ { "cell_type": "code", "execution_count": null, - "id": "bb0162ba", + "id": "61", "metadata": {}, "outputs": [], "source": [ "(\n", - " pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"])\n", - " .style.format(precision=0)\n", + " ct_cut_color\n", + " .to_pandas()\n", + " .style\n", + " .format(precision=0)\n", " .bar(color=\"#d65f5f\")\n", ")" ] }, { "cell_type": "markdown", - "id": "e36d4795", + "id": "62", "metadata": {}, "source": [ - "Use `.hightlight_max()`, and similar commands, to show important entries:" + "Use `max_counts()`, and similar commands, to show important entries:" ] }, { "cell_type": "code", "execution_count": null, - "id": "5d19072c", + "id": "63", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.highlight_max().format(\"{:.0f}\")" + "max_counts = ct_cut_color.select(pl.exclude(\"color\").max())\n", + "max_counts" ] }, { "cell_type": "markdown", - "id": "63075da4", + "id": "64", "metadata": {}, "source": [ "You can find a full set of styling commands [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Styling)." @@ -876,10 +904,10 @@ }, { "cell_type": "markdown", - "id": "d8b6d225", + "id": "65", "metadata": {}, "source": [ - "### Exploratory Plotting with **pandas**\n", + "### Exploratory Plotting with **polars**\n", "\n", "**pandas** has some built-in plotting options to help you look at data quickly. These can be accessed via `.plot.*` or `.plot()`, depending on the context. Let's make a quick `.plot()` using a dataset on taxis." ] @@ -887,48 +915,46 @@ { "cell_type": "code", "execution_count": null, - "id": "b479d5b1", + "id": "66", "metadata": {}, "outputs": [], "source": [ - "taxis = pd.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", - "# turn the pickup time column into a datetime\n", - "taxis[\"pickup\"] = pd.to_datetime(taxis[\"pickup\"])\n", - "# set some other columns types\n", - "taxis = taxis.astype(\n", - " {\n", - " \"dropoff\": \"datetime64[ns]\",\n", - " \"pickup\": \"datetime64[ns]\",\n", - " \"color\": \"category\",\n", - " \"payment\": \"category\",\n", - " \"pickup_zone\": \"string\",\n", - " \"dropoff_zone\": \"string\",\n", - " \"pickup_borough\": \"category\",\n", - " \"dropoff_borough\": \"category\",\n", - " }\n", - ")\n", + "taxis = pl.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", + "\n", + "taxis = taxis.with_columns(pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\"))\n", + "\n", + "taxis = taxis.with_columns([\n", + " pl.col(\"color\").cast(pl.Categorical),\n", + " pl.col(\"payment\").cast(pl.Categorical),\n", + " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", + " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", + "])\n", + "\n", "taxis.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "ee971c9c", + "id": "67", "metadata": {}, "outputs": [], "source": [ - "taxis.info()" + "taxis.describe()" ] }, { "cell_type": "code", "execution_count": null, - "id": "2015b1dc", + "id": "68", "metadata": {}, "outputs": [], "source": [ + "taxis_pd = taxis.to_pandas()\n", + "taxis_pd[\"pickup\"] = pd.to_datetime(taxis_pd[\"pickup\"])\n", + "\n", "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[\"total\"]\n", " .mean()\n", " .plot(\n", @@ -936,12 +962,13 @@ " xlabel=\"\",\n", " ylabel=\"Fare (USD)\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "b9c0f990", + "id": "69", "metadata": {}, "source": [ "Again, if you can get the data in the right shape, you can plot it. The same function works with multiple lines\n" @@ -950,12 +977,12 @@ { "cell_type": "code", "execution_count": null, - "id": "51e86185", + "id": "70", "metadata": {}, "outputs": [], "source": [ "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[[\"fare\", \"tip\", \"tolls\"]]\n", " .mean()\n", " .plot(\n", @@ -964,12 +991,13 @@ " xlabel=\"\",\n", " ylabel=\"USD\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "cc227b81", + "id": "71", "metadata": {}, "source": [ "Now let's see some of the other quick `.plot.*` options." @@ -977,7 +1005,7 @@ }, { "cell_type": "markdown", - "id": "1c4278ac", + "id": "72", "metadata": {}, "source": [ "A bar chart (use `barh` for horizontal orientation; `rot` sets rotation of labels):" @@ -986,16 +1014,32 @@ { "cell_type": "code", "execution_count": null, - "id": "79ceca92", + "id": "73", "metadata": {}, "outputs": [], "source": [ - "taxis.value_counts(\"payment\").sort_index().plot.bar(title=\"Counts\", rot=0);" + "payment_counts = (\n", + " taxis\n", + " .filter(pl.col(\"payment\").is_not_null())\n", + " .group_by(\"payment\")\n", + " .agg(pl.len().alias(\"count\"))\n", + " .sort(\"payment\")\n", + ")\n", + "\n", + "payment_counts.to_pandas().plot.bar(\n", + " x=\"payment\",\n", + " y=\"count\",\n", + " legend=False,\n", + " rot=0,\n", + " title=\"Counts\",\n", + ")\n", + "\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "4e6f1cc2", + "id": "74", "metadata": {}, "source": [ "This next one, uses `.plot.hist()` to create a histogram." @@ -1004,16 +1048,17 @@ { "cell_type": "code", "execution_count": null, - "id": "5efc5817", + "id": "75", "metadata": {}, "outputs": [], "source": [ - "taxis[\"tip\"].plot.hist(bins=30, title=\"Tip\");" + "(taxis_pd[\"tip\"].plot.hist(bins=30, title=\"Tip\"))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "3f15bfa7", + "id": "76", "metadata": {}, "source": [ "Boxplot:" @@ -1022,16 +1067,17 @@ { "cell_type": "code", "execution_count": null, - "id": "0b735d15", + "id": "77", "metadata": {}, "outputs": [], "source": [ - "(taxis[[\"fare\", \"tolls\", \"tip\"]].plot.box());" + "(taxis_pd[[\"fare\", \"tolls\", \"tip\"]].plot.box())\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "67017bef", + "id": "78", "metadata": {}, "source": [ "Scatter plot:" @@ -1040,16 +1086,17 @@ { "cell_type": "code", "execution_count": null, - "id": "66adada2", + "id": "79", "metadata": {}, "outputs": [], "source": [ - "taxis.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None));" + "(taxis_pd.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None)))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "9766d2c4", + "id": "80", "metadata": {}, "source": [ "## Other tools for EDA\n", @@ -1059,7 +1106,7 @@ }, { "cell_type": "markdown", - "id": "813ec70a", + "id": "81", "metadata": {}, "source": [ "### **skimpy** for summary statistics\n", @@ -1072,7 +1119,7 @@ { "cell_type": "code", "execution_count": null, - "id": "32796b5f", + "id": "82", "metadata": {}, "outputs": [], "source": [ @@ -1081,7 +1128,7 @@ }, { "cell_type": "markdown", - "id": "f2810c9e", + "id": "83", "metadata": {}, "source": [ "## Summary\n", @@ -1095,9 +1142,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -1105,7 +1149,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, From 58f4891e258e2bebaaf5f836a1121fce0d45401b Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 14 Jul 2026 22:24:10 +0100 Subject: [PATCH 29/30] style: fix pre-commit issues in exploratory data analysis notebook --- exploratory-data-analysis.ipynb | 85 +++++++++++++++++---------------- 1 file changed, 43 insertions(+), 42 deletions(-) diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index eae03ec..5841632 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -112,10 +112,12 @@ "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", "diamonds_color_order = [\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"]\n", "\n", - "diamonds = diamonds.with_columns([\n", - " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", - " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", - "])\n", + "diamonds = diamonds.with_columns(\n", + " [\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", + " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", + " ]\n", + ")\n", "\n", "diamonds.head()" ] @@ -273,7 +275,9 @@ "metadata": {}, "outputs": [], "source": [ - "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select([\"x\", \"y\", \"z\", \"price\"])\n", + "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select(\n", + " [\"x\", \"y\", \"z\", \"price\"]\n", + ")\n", "unusual" ] }, @@ -403,15 +407,20 @@ "metadata": {}, "outputs": [], "source": [ - "flights2 = flights.with_columns([\n", - " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", - " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", - " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", - " ((pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60).alias(\"sched_dep_time\"),\n", - "])\n", - "\n", "from lets_plot import *\n", "\n", + "flights2 = flights.with_columns(\n", + " [\n", + " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", + " (\n", + " (pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60\n", + " ).alias(\"sched_dep_time\"),\n", + " ]\n", + ")\n", + "\n", + "\n", "(\n", " ggplot(flights2, aes(x=\"sched_dep_time\"))\n", " + geom_freqpoly(aes(color=\"cancelled\"), binwidth=1 / 4)\n", @@ -529,9 +538,7 @@ "metadata": {}, "outputs": [], "source": [ - "mpg = pl.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\"\n", - ")\n", + "mpg = pl.read_csv(\"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\")\n", "\n", "mpg = mpg.with_columns(pl.col(\"class\").cast(pl.Categorical))\n", "\n", @@ -623,7 +630,9 @@ " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", ")\n", "\n", - "ct_cut_color_long = ct_cut_color.melt(id_vars=\"cut\", variable_name=\"color\", value_name=\"value\")" + "ct_cut_color_long = ct_cut_color.melt(\n", + " id_vars=\"cut\", variable_name=\"color\", value_name=\"value\"\n", + ")" ] }, { @@ -749,12 +758,7 @@ "metadata": {}, "outputs": [], "source": [ - "sum_table = (\n", - " diamonds.describe()\n", - " .with_columns(\n", - " pl.selectors.numeric().round(1)\n", - " )\n", - ")\n", + "sum_table = diamonds.describe().with_columns(pl.selectors.numeric().round(1))\n", "\n", "sum_table" ] @@ -774,7 +778,9 @@ "metadata": {}, "outputs": [], "source": [ - "sum_table_long = sum_table.melt(id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\")\n", + "sum_table_long = sum_table.melt(\n", + " id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\"\n", + ")\n", "sum_table_long" ] }, @@ -837,9 +843,7 @@ " values=\"count\",\n", " aggregate_function=\"sum\",\n", " )\n", - " .with_columns(\n", - " cs.numeric().fill_null(0)\n", - " )\n", + " .with_columns(cs.numeric().fill_null(0))\n", ")" ] }, @@ -866,13 +870,7 @@ "metadata": {}, "outputs": [], "source": [ - "(\n", - " ct_cut_color\n", - " .to_pandas()\n", - " .style\n", - " .format(precision=0)\n", - " .bar(color=\"#d65f5f\")\n", - ")" + "(ct_cut_color.to_pandas().style.format(precision=0).bar(color=\"#d65f5f\"))" ] }, { @@ -921,14 +919,18 @@ "source": [ "taxis = pl.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", "\n", - "taxis = taxis.with_columns(pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\"))\n", + "taxis = taxis.with_columns(\n", + " pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\")\n", + ")\n", "\n", - "taxis = taxis.with_columns([\n", - " pl.col(\"color\").cast(pl.Categorical),\n", - " pl.col(\"payment\").cast(pl.Categorical),\n", - " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", - " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", - "])\n", + "taxis = taxis.with_columns(\n", + " [\n", + " pl.col(\"color\").cast(pl.Categorical),\n", + " pl.col(\"payment\").cast(pl.Categorical),\n", + " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", + " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", + " ]\n", + ")\n", "\n", "taxis.head()" ] @@ -1019,8 +1021,7 @@ "outputs": [], "source": [ "payment_counts = (\n", - " taxis\n", - " .filter(pl.col(\"payment\").is_not_null())\n", + " taxis.filter(pl.col(\"payment\").is_not_null())\n", " .group_by(\"payment\")\n", " .agg(pl.len().alias(\"count\"))\n", " .sort(\"payment\")\n", From 2b2145c5103abb94882a3849e524de241d0ffaeb Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 29 Jul 2026 14:47:48 +0100 Subject: [PATCH 30/30] fix: Address review issues --- exploratory-data-analysis.ipynb | 72 +++++++++++++++++---------------- 1 file changed, 38 insertions(+), 34 deletions(-) diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index 5841632..7b4dce2 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -21,7 +21,7 @@ "\n", "### Prerequisites\n", "\n", - "For doing EDA, we'll use the **polars**, **skimpy**, and **polars-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", + "For doing EDA, we'll use the **polars** and **skimpy** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", "\n", "As ever, we begin by loading these packages that we'll use:" ] @@ -265,7 +265,7 @@ "`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis.\n", "**Lets-Plot** also has `xlim()` and `ylim()` functions that work slightly differently: they throw away the data outside the limits.\n", "\n", - "This allows us to see that there are three unusual values: 0, \\~30, and \\~60. We pluck them out with **pandas**:" + "This allows us to see that there are three unusual values: 0, \\~30, and \\~60. We pluck them out with **polars**:" ] }, { @@ -288,8 +288,8 @@ "source": [ "The `\"y\"` variable measures one of the three dimensions of these diamonds, in mm.\n", "We know that diamonds can't have a width of 0mm, so these values must be incorrect.\n", - "By doing EDA, we have discovered missing data that was coded as 0, which we never would have found by simply searching for `NA`s.\n", - "Going forward we might choose to re-code these values as `NA`s in order to prevent misleading calculations.\n", + "By doing EDA, we have discovered missing data that was coded as 0, which we never would have found by simply searching for `null`s.\n", + "Going forward we might choose to re-code these values as `null`s in order to prevent misleading calculations.\n", "We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don't cost hundreds of thousands of dollars!\n", "\n", "It's good practice to repeat your analysis with and without the outliers.\n", @@ -334,15 +334,14 @@ "1. Drop the entire row with the strange values:\n", "\n", " ```python\n", - " condition = ((diamonds[\"y\"] < 3) | (diamonds[\"y\"] > 20))\n", - " diamonds2 = diamonds.loc[~condition, :]\n", + " diamonds2 = diamonds.filter(~((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)))\n", " ```\n", "\n", " We don't recommend this option because one invalid value doesn't imply that all the other values for that observation are also invalid.\n", " Additionally, if you have low quality data, by the time that you've applied this approach to every variable you might find that you don't have any data left!\n", "\n", "2. Instead, we recommend replacing the unusual values with missing values.\n", - " One way to do this, which makes a distinction between data frames that have had the unusual values replaced and the original data, is to make a copy and then set the problematic values to `pd.NA`, **pandas**'s special NA value.\n" + " One way to do this, which makes a distinction between data frames that have had the unusual values replaced and the original data, is to use `with_columns()` together with `pl.when()` to replace problematic values with `None` (`null` in Polars):\n" ] }, { @@ -352,7 +351,12 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds2 = diamonds.filter(~((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)))" + "diamonds2 = diamonds.with_columns(\n", + " pl.when((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20))\n", + " .then(None)\n", + " .otherwise(pl.col(\"y\"))\n", + " .alias(\"y\")\n", + ")\n" ] }, { @@ -381,7 +385,7 @@ "Other times you want to understand what makes observations with missing values different to observations with recorded values.\n", "For example, in the nycflights13 data, missing values in the `\"dep_time\"` variable indicate that the flight was cancelled.\n", "So you might want to compare the scheduled departure times for cancelled and non-cancelled times.\n", - "You can do this by making a new variable, using `is.na()` to check if `\"dep_time\"` is missing." + "You can do this by making a new variable, using `is_null()` to check if `\"dep_time\"` is missing." ] }, { @@ -391,13 +395,11 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", "import polars as pl\n", "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights_pd = pd.read_csv(url)\n", - "flights = pl.from_pandas(flights_pd)\n", - "flights.head()" + "flights = pl.read_csv(url, null_values=\"NA\")\n", + "flights.head()\n" ] }, { @@ -614,7 +616,7 @@ "source": [ "### Two categorical variables\n", "\n", - "To visualise the covariation between categorical variables, you'll need to count the number of observations for each combination of levels of these categorical variables. You can do this with a `pd.crosstab()` that we then melt to put it in \"tidy\" format." + "To visualise the covariation between categorical variables, you'll need to count the number of observations for each combination of levels of these categorical variables. You can do this with a Polars `group_by()` and `pivot()` that we then unpivot to put in \"tidy\" format." ] }, { @@ -627,12 +629,12 @@ "ct_cut_color = (\n", " diamonds.group_by([\"cut\", \"color\"])\n", " .agg(pl.len().alias(\"count\"))\n", - " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", + " .pivot(index=\"cut\", on=\"color\", values=\"count\", aggregate_function=\"sum\")\n", ")\n", "\n", - "ct_cut_color_long = ct_cut_color.melt(\n", - " id_vars=\"cut\", variable_name=\"color\", value_name=\"value\"\n", - ")" + "ct_cut_color_long = ct_cut_color.unpivot(\n", + " index=\"cut\", variable_name=\"color\", value_name=\"value\"\n", + ")\n" ] }, { @@ -664,7 +666,7 @@ "\n", "2. What different data insights do you get with a segmented bar chart if colour is mapped to the `x` aesthetic and cut is mapped to the fill aesthetic? Calculate the counts that fall into each of the segments.\n", "\n", - "3. Use `geom_tile()` together with **pandas** to explore how average flight departure delays vary by destination and month of year. What makes the plot difficult to read? How could you improve it?" + "3. Use `geom_tile()` together with **polars** to explore how average flight departure delays vary by destination and month of year. What makes the plot difficult to read? How could you improve it?" ] }, { @@ -724,13 +726,13 @@ "id": "50", "metadata": {}, "source": [ - "## **pandas** built-in tools for EDA\n", + "## **polars** built-in tools for EDA\n", "\n", - "**pandas** has some great options for built-in EDA; in fact we've already seen one of them, `df.info()` which, as well as reporting datatypes and memory usage, also tells us how many observations in each column are 'truthy' rather than 'falsy', ie how many have non-null values.\n", + "**polars** has convenient options for built-in EDA; for example, `.describe()` reports basic summary statistics across numerical columns.\n", "\n", "### Exploratory tables and descriptive statistics\n", "\n", - "A small step beyond `.info()` to get tables is to use `.describe()` which, if you have mixed datatypes that include floats, will report some basic summary statistics:" + "A small step to summarize tables is to use `.describe()`:\n" ] }, { @@ -778,10 +780,10 @@ "metadata": {}, "outputs": [], "source": [ - "sum_table_long = sum_table.melt(\n", - " id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\"\n", + "sum_table_long = sum_table.unpivot(\n", + " index=\"statistic\", variable_name=\"variable\", value_name=\"value\"\n", ")\n", - "sum_table_long" + "sum_table_long\n" ] }, { @@ -791,9 +793,9 @@ "source": [ "Of course, the stats provided in this pre-built table are not very customised. So what do we do to get the table that we actually want? Well, the answer is to draw on the contents of the previous data chapters, particularly the introduction to data analysis. Groupbys, merges, aggregations: use all of them to produce the EDA table that you want.\n", "\n", - "If you're exploring data, you might also want to be able to read everything clearly and see any deviations from what you'd expect quickly. **pandas** has some built-in functionality that styles data frames to help you. These styles persist when you export the data frame to, say, Excel, too.\n", + "If you're exploring data, you might also want to be able to format data frames to read everything clearly. **polars** provides groupbys, aggregations, pivoting, and selectors to shape and round summary tables easily.\n", "\n", - "Here's an example that highlights some ways of styling data frames, making use of several features such as: unstacking into a wider format (`unstack`), changing the units (`lambda` function; note that `1e3` is shorthand for `1000` on computers), fill NaNs with unobtrusive strings (`.fillna('-')`), removing numbers after the decimal place (`.style.format(precision=0)`), and adding a caption (`.style.set_caption`)." + "Here's an example that highlights how to aggregate and pivot data in Polars, grouping by `\"cut\"` and `\"color\"`, pivoting the resulting table, and rounding numbers with `cs.numeric().round(2)`:\n" ] }, { @@ -822,9 +824,9 @@ "source": [ "Although a neater one than we've seen, this is still a drab table of numbers. The eye is not immediately drawn to it!\n", "\n", - "To remedy that, let's take a look at another styling technique: the use of colour. Let's say we wanted to make a table that showed a cross-tabulation between cut and color; that is the counts of objects appearing in both of these fields according to the categories.\n", + "To remedy that, let's take a look at another technique: the use of colour. Let's say we wanted to make a table that showed a cross-tabulation between cut and color; that is the counts of objects appearing in both of these fields according to the categories.\n", "\n", - "To perform a cross-tabulation, we'll use the built-in `pd.crosstab()` but we'll ask that the values that appear in the table (counts) be lit up with a heatmap using `style.background_gradient()` too:" + "To perform a cross-tabulation in Polars, we aggregate with `group_by()` and `pivot()`. If we want to style the table, we can convert to a pandas DataFrame/Styler via `.to_pandas().style` to apply formatting such as bar charts or gradients:\n" ] }, { @@ -878,7 +880,7 @@ "id": "62", "metadata": {}, "source": [ - "Use `max_counts()`, and similar commands, to show important entries:" + "To find the maximum counts across categories in Polars, you can use `.select()` with `pl.max()`:\n" ] }, { @@ -907,7 +909,7 @@ "source": [ "### Exploratory Plotting with **polars**\n", "\n", - "**pandas** has some built-in plotting options to help you look at data quickly. These can be accessed via `.plot.*` or `.plot()`, depending on the context. Let's make a quick `.plot()` using a dataset on taxis." + "While **polars** focuses on high-performance data processing, you can seamlessly convert to pandas via `.to_pandas()` to leverage quick plotting features like `.plot.*` or `.plot()`. Let's make a quick plot using a dataset on taxis.\n" ] }, { @@ -952,6 +954,8 @@ "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", + "\n", "taxis_pd = taxis.to_pandas()\n", "taxis_pd[\"pickup\"] = pd.to_datetime(taxis_pd[\"pickup\"])\n", "\n", @@ -965,7 +969,7 @@ " ylabel=\"Fare (USD)\",\n", " )\n", ")\n", - "plt.show()" + "plt.show()\n" ] }, { @@ -1102,7 +1106,7 @@ "source": [ "## Other tools for EDA\n", "\n", - "Between **pandas** and visualisation packages, you have a lot of what you need for EDA. But there are some tools just dedicated to making EDA easier that it's worth knowing about." + "Between **polars** and visualisation packages, you have a lot of what you need for EDA. But there are some tools just dedicated to making EDA easier that it's worth knowing about." ] }, { @@ -1150,7 +1154,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" },