diff --git a/.gitignore b/.gitignore index af04659..3246836 100644 --- a/.gitignore +++ b/.gitignore @@ -4,7 +4,7 @@ # quarto _book/ /.quarto/ -**/*.quarto_ipynb +**/*.quarto_ipynb* _freeze/ .panache* @@ -137,3 +137,5 @@ dmypy.json # Pyre type checker .pyre/ + +**/*.quarto_ipynb diff --git a/boolean-data.ipynb b/boolean-data.ipynb index d389d25..bd4c3d6 100644 --- a/boolean-data.ipynb +++ b/boolean-data.ipynb @@ -11,9 +11,13 @@ "\n", "In this chapter, we'll introduce boolean data: data that can be `True` or `False` (which can also be encoded as 1s or 0s). We'll first look at the fundamental Python true and false boolean variables before seeing how true and false work in data frames.\n", "\n", + "### Prerequisites\n", + "\n", + "This chapter focuses on working with boolean data and expressions using **polars**. Make sure you have **polars** installed (`uv add polars`). For more on Polars boolean operations, see the [Polars boolean expression documentation](https://docs.pola.rs/api/python/stable/reference/expressions/boolean.html).\n", + "\n", "## Booleans\n", "\n", - "Some of the most important operations you will perform are with `True` and `False` values, also known as boolean data types. These are fundamental Python variables, just as numbers such as `1` are. \n", + "Some of the most important operations you will perform are with `True` and `False` values, also known as boolean data types. These are fundamental Python variables, just as numbers such as `1` are.\n", "\n", "### Boolean Variables and Conditions\n", "\n", @@ -481,11 +485,11 @@ "id": "8e06930b", "metadata": {}, "source": [ - "## Booleans in **pandas** data frames\n", + "## Booleans in **polars** DataFrames\n", "\n", "### Operations on booleans in data frames\n", "\n", - "Quite often, you will run into a scenario where you're working with data that have True or False values in a data frame. It is easy to create a column of booleans in a **pandas** data frame:" + "Quite often, you will run into a scenario where you're working with data that have True or False values in a data frame. It is easy to create a column of booleans in a **polars** data frame:" ] }, { @@ -495,9 +499,9 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame.from_dict(\n", + "df = pl.DataFrame(\n", " {\n", " \"bool_col_1\": [False] * 3 + [True, True],\n", " \"bool_col_2\": [True, False, True, False, True],\n", @@ -511,7 +515,7 @@ "id": "c7a0cd32", "metadata": {}, "source": [ - "We can perform operations on these just like regular **pandas** data frame columns. These accept `&` (and), `|` (or), `==` (equal), and `!=` (not equal) as operations:" + "We can perform operations on these just like regular **polars** data frame columns. These accept `&` (and), `|` (or), `==` (equal), and `!=` (not equal) as operations:" ] }, { @@ -521,7 +525,12 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"bool_col_1\"] | df[\"bool_col_2\"]" + "df.select(\n", + " [\n", + " (pl.col(\"bool_col_1\") & pl.col(\"bool_col_2\")).alias(\"and_result\"),\n", + " (pl.col(\"bool_col_1\") | pl.col(\"bool_col_2\")).alias(\"or_result\"),\n", + " ]\n", + ")" ] }, { @@ -529,7 +538,7 @@ "id": "25dede69", "metadata": {}, "source": [ - "Quite often, it's useful to have a count of the number of true values. If you take the sum of boolean columns in a **pandas** data frame, it will tot up the number of `True` values:" + "Quite often, it's useful to have a count of the number of true values. If you take the sum of boolean columns in a **polars** data frame, it will tot up the number of `True` values:" ] }, { @@ -539,7 +548,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.sum()" + "df.select(pl.col(\"*\").sum())" ] }, { @@ -557,8 +566,8 @@ "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame.from_dict({\"bool_col\": [0, 1, 0, 1, 1]})\n", - "df[\"bool_col\"].astype(bool)" + "df = pl.DataFrame({\"bool_col\": [0, 1, 0, 1, 1]})\n", + "df.with_columns(pl.col(\"bool_col\").cast(pl.Boolean))" ] }, { @@ -579,7 +588,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", + "diamonds = pl.read_csv(\n", " \"https://github.com/mwaskom/seaborn-data/raw/master/diamonds.csv\"\n", ")\n", "diamonds.head()" @@ -596,12 +605,12 @@ { "cell_type": "code", "execution_count": null, - "id": "7a27f0a0", + "id": "6b739dc5", "metadata": {}, "outputs": [], "source": [ - "diamonds[\"expensive\"] = diamonds[\"price\"] > 1000\n", - "diamonds.sample(10)" + "diamonds = diamonds.with_columns((pl.col(\"price\") > 1000).alias(\"expensive\"))\n", + "diamonds.select([\"cut\", \"carat\", \"color\", \"clarity\", \"price\", \"expensive\"]).head()" ] }, { @@ -609,7 +618,7 @@ "id": "8383cab5", "metadata": {}, "source": [ - "Of course, this could also have been achieved in a call to assign:" + "Of course, this could also have been achieved with `select()` or `with_columns()`:" ] }, { @@ -619,7 +628,10 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds.assign(expensive=lambda x: x[\"price\"] > 1000).head()" + "diamonds.select(\n", + " [\"cut\", \"carat\", \"color\", \"clarity\", \"price\"],\n", + " (pl.col(\"price\") > 1000).alias(\"expensive\"),\n", + ").head()" ] }, { @@ -627,7 +639,7 @@ "id": "622ff486", "metadata": {}, "source": [ - "Another use of booleans that is quite useful when it comes to data frames is the `.isin()` function. For example, if you just want some True or False values for whether a set of columns is in a data frame:" + "Another use of booleans in data frames is checking if a set of column names exists. We can use a Python list comprehension with the Python `in` operator on `diamonds.columns`:" ] }, { @@ -637,7 +649,26 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds.columns.isin([\"x\", \"y\", \"z\"])" + "cols_to_check = [\"x\", \"y\", \"z\"]\n", + "[col in diamonds.columns for col in cols_to_check]" + ] + }, + { + "cell_type": "markdown", + "id": "polars_is_in_markdown", + "metadata": {}, + "source": [ + "Alternatively, you can check column existence natively in **polars** using `pl.Series.is_in()`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "polars_is_in_code", + "metadata": {}, + "outputs": [], + "source": [ + "pl.Series(diamonds.columns).is_in(cols_to_check)" ] }, { @@ -647,7 +678,7 @@ "source": [ "### any() and all() in data frames\n", "\n", - "A **pandas** column of booleans behaves a lot like a list of booleans, and we can apply the same logic to it via **pandas** built-in `.any()` and `.all()` methods. We expect some entries for `\"expensive\"` to be true, so `any()` should return true:" + "A **polars** column of booleans behaves a lot like a list of booleans, and we can apply the same logic to it via **polars** built-in `.any()` and `.all()` methods. We expect some entries for `\"expensive\"` to be true, so `any()` should return true:" ] }, { @@ -657,7 +688,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds[\"expensive\"].any()" + "diamonds.select(pl.col(\"expensive\").any())" ] }, { @@ -677,7 +708,7 @@ "metadata": {}, "outputs": [], "source": [ - "diamonds[diamonds[\"x\"] > diamonds[\"y\"]]" + "diamonds.filter(pl.col(\"x\") > pl.col(\"y\"))" ] }, { @@ -685,14 +716,11 @@ "id": "9b5906ba", "metadata": {}, "source": [ - "The expression `diamonds[\"x\"] > diamonds[\"y\"]` creates a column of booleans that is used to filter to just the rows where the condition is true." + "The expression `pl.col(\"x\") > pl.col(\"y\")` creates a column of booleans that is used to filter to just the rows where the condition is true." ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", diff --git a/categorical-data.ipynb b/categorical-data.ipynb index ab6ba30..123f03d 100644 --- a/categorical-data.ipynb +++ b/categorical-data.ipynb @@ -2,25 +2,21 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "cell_000", "metadata": {}, "source": [ "# Categorical Data {#sec-categorical-data}\n", "\n", "## Introduction\n", "\n", - "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **pandas** [documentation](https://pandas.pydata.org/).\n" + "In this chapter, we'll introduce how to work with categorical variables—that is, variables that have a fixed and known set of possible values. This chapter is enormously indebted to the **polars** [documentation](https://docs.pola.rs/api/python/stable/reference/index.html).\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "51a55374", - "metadata": { - "tags": [ - "remove-cell" - ] - }, + "id": "cell_001", + "metadata": {}, "outputs": [], "source": [ "# remove cell\n", @@ -34,248 +30,273 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "cell_002", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package.\n" ] }, { "cell_type": "markdown", - "id": "8757f369", + "id": "cell_003", "metadata": {}, "source": [ "## The Category Datatype\n", "\n", - "Everything in Python has a type, even the data in **pandas** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical`. There are some benefits to using categorical variables (where appropriate):\n", + "Everything in Python has a type, even the data in **polars** data frame columns. While you may be more familiar with numbers and even strings, there is also a special data type for categorical data called `Categorical` and `Enum`. There are some benefits to using categorical variables (where appropriate):\n", "\n", "- they can keep track even when elements of the category isn't present, which can sometimes be as interesting as when they are (imagine you find no-one from a particular school goes to university)\n", "- they can use vastly less of your computer's memory than encoding the same information in other ways\n", "- they can be used efficiently with modelling packages, where they will be recognised as potential 'dummy variables', or with plotting packages, which will treat them as discrete values\n", - "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\")\n", + "- you can order them (for example, \"neutral\", \"agree\", \"strongly agree\") using pl.Enum\n", + "\n", + "All values of categorical data for a **polars** column are either in the given categories or take the value `null`.\n", + "\n", + "Polars has two categorical types:\n", "\n", - "All values of categorical data for a **pandas** column are either in the given categories or take the value `np.nan`. \n", + "- pl.Categorical: Unordered categories (like pandas category)\n", + "\n", + "- pl.Enum: Ordered categories with a fixed set of values (similar to pandas ordered categorical)\n", "\n", "## Creating Categorical Data\n", "\n", - "Let's create a categorical column of data:" + "Let's create a categorical column of data:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "535ef959", + "id": "cell_004", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", + "df = pl.DataFrame({\"A\": [\"a\", \"b\", \"c\", \"a\"]})\n", "\n", - "df[\"A\"] = df[\"A\"].astype(\"category\")\n", + "df = df.with_columns(pl.col(\"A\").cast(pl.Categorical))\n", "df[\"A\"]" ] }, { "cell_type": "markdown", - "id": "62db4dec", + "id": "cell_005", "metadata": {}, "source": [ - "Notice that we get some additional information at the bottom of the shown series: we get told that not only is this a categorical column type, but it has three values 'a', 'b', and 'c'.\n", + "Notice that polars tracks the categorical type, though the display is more compact than pandas.\n", "\n", - "You can also use special functions, such as `pd.cut()`, to groups data into discrete bins. Here's an example where specify the labels for the categories directly:\n" + "You can also use special expression methods, such as pl.col().cut(), to group data into discrete bins. Here's an example where we specify the labels for each category directly.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "358c83bb", + "id": "cell_006", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", + "df = pl.DataFrame({\"value\": np.random.randint(0, 100, 20)})\n", "labels = [f\"{i} - {i+9}\" for i in range(0, 100, 10)]\n", - "df[\"group\"] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)\n", + "df = df.with_columns(\n", + " pl.col(\"value\")\n", + " .cut(breaks=list(range(10, 100, 10)), labels=labels, left_closed=True)\n", + " .alias(\"group\")\n", + ")\n", "df.head()" ] }, { "cell_type": "markdown", - "id": "87bc56b6", + "id": "cell_007", "metadata": {}, "source": [ "In the example above, the `group` column is of categorical type.\n", "\n", - "Another way to create a categorical variable is directly using the `pd.Categorical()` function:" + "Another way to create a categorical variable is directly using the `pl.Categorical` or `pl.Enum` types:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "fb389105", + "id": "cell_008", "metadata": {}, "outputs": [], "source": [ - "raw_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"], categories=[\"b\", \"c\", \"d\"]\n", + "raw_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"b\", \"c\", \"d\"]), strict=False\n", ")\n", "raw_cat" ] }, { "cell_type": "markdown", - "id": "ebfdf815", + "id": "cell_009", + "metadata": {}, + "source": [ + "`pl.Enum` is preferred when the specific categories are known upfront. `pl.Categorical` infers categories as it reads the dataframe column.\n" + ] + }, + { + "cell_type": "markdown", + "id": "5decf775", "metadata": {}, "source": [ - "We can then enter this into a data frame:" + "We can then enter this into a data frame:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "0497fc16", + "id": "cell_010", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(raw_cat, columns=[\"cat_type\"])\n", + "df = pl.DataFrame({\"cat_type\": raw_cat})\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "76dc9dd6", + "id": "cell_011", "metadata": {}, "source": [ - "Note that NaNs appear for any value that *isn't* in the categories we specified—you can find more on this in @sec-missing-values." + "Note that `null` appears for any value that _isn't_ in the categories we specified—you can find more on this in @sec-missing-values.\n" ] }, { "cell_type": "markdown", - "id": "aa76a795", + "id": "cell_012", "metadata": {}, "source": [ - "You can also create ordered categories:" + "You can also create ordered categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f7520d3d", + "id": "cell_013", "metadata": {}, "outputs": [], "source": [ - "ordered_cat = pd.Categorical(\n", - " [\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"],\n", - " categories=[\"a\", \"b\", \"c\", \"d\"],\n", - " ordered=True,\n", + "ordered_cat = pl.Series([\"a\", \"b\", \"c\", \"a\", \"d\", \"a\", \"c\"]).cast(\n", + " pl.Enum([\"a\", \"b\", \"c\", \"d\"])\n", ")\n", "ordered_cat" ] }, { "cell_type": "markdown", - "id": "258cc6f5", + "id": "cell_014", "metadata": {}, "source": [ "## Working with Categories\n", "\n", - "Categorical data has a `categories` and a `ordered` property; these list the possible values and whether the ordering matters or not respectively. These properties are exposed as `.cat.categories` and `.cat.ordered`. If you don’t manually specify categories and ordering, they are inferred from the passed arguments.\n", + "Categorical data in polars has categories and ordering properties. For `pl.Enum`, the categories are fixed and ordered. For `pl.Categorical`, categories are not ordered and can be more flexible.\n", "\n", - "Let's see some examples:" + "Let's see some examples:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "2caba354", + "id": "cell_015", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.categories" + "df[\"cat_type\"].dtype.categories" ] }, { "cell_type": "code", "execution_count": null, - "id": "5f1fe093", + "id": "cell_016", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].cat.ordered" + "isinstance(df[\"cat_type\"].dtype, pl.Enum)" ] }, { "cell_type": "markdown", - "id": "694c07b0", + "id": "cell_017", "metadata": {}, "source": [ - "If categorical data is ordered (ie `.cat.ordered == True`), then the order of the categories has a meaning and certain operations are possible: you can sort values (with `.sort_values`), and apply `.min` and `.max`." + "If categorical data is ordered (ie `pl.Enum`), then the order of the categories has a meaning and certain operations are possible: you can sort values, and apply `.min()` and `.max()`.\n" ] }, { "cell_type": "markdown", - "id": "0a6d7a90", + "id": "cell_018", "metadata": {}, "source": [ "### Renaming Categories\n", "\n", - "Renaming categories is done via the `rename_categories()` method (which works with a list or a dictionary)." + "Renaming categories in polars requires re-casting with new categories:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "097171b8", + "id": "cell_019", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.rename_categories([\"alpha\", \"beta\", \"gamma\"])" + "df = df.with_columns(\n", + " pl.col(\"cat_type\")\n", + " .cast(pl.String)\n", + " .replace({\"b\": \"alpha\", \"c\": \"beta\", \"d\": \"gamma\"})\n", + " .cast(pl.Enum([\"alpha\", \"beta\", \"gamma\"]))\n", + ")" ] }, { "cell_type": "markdown", - "id": "3de7fd23", + "id": "cell_020", "metadata": {}, "source": [ - "Quite often, you'll run into a situation where you want to add a category. You can do this with `.add_categories()`:" + "To add categories, you need to re-cast with an extended Enum:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4ae6df38", + "id": "cell_021", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"] = df[\"cat_type\"].cat.add_categories([\"delta\"])\n", + "df = df.with_columns(\n", + " pl.col(\"cat_type\").cast(pl.Enum([\"alpha\", \"beta\", \"gamma\", \"delta\"]))\n", + ")\n", "df[\"cat_type\"]" ] }, { "cell_type": "markdown", - "id": "ba525491", + "id": "cell_022", "metadata": {}, "source": [ - "Similarly, there is a `.remove_categories()` function and a `.remove_unused_categories()` function. `.set_categories` adds and removes categories in one fell swoop. One of the nice properties of set categories is that Remember that you need to do `df[\"columnname\"].cat` before calling any cat(egory) functions though." + "Because Enum categories are fixed in Polars, you cannot remove categories directly. Instead, ensure that the data contains only the desired values, then re-cast the column to a new Enum containing just those categories.\n" ] }, { "cell_type": "markdown", - "id": "76b38a2a", + "id": "cell_023", "metadata": {}, "source": [ "## Operations on Categories\n", "\n", - "As noted, ordered categories will already undergo some operations. But there are some that work on any set of categories. Perhaps the most useful is `value_counts()`" + "`Enum` columns behave like any other Polars column, but they also retain the complete set of allowed categories. This means Polars remembers categories even if no rows currently contain them.\n", + "\n", + "For example, suppose we define an `Enum` with five possible categories, but only use four of them:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "19f5bdda", + "id": "cell_024", "metadata": {}, "outputs": [], "source": [ @@ -284,42 +305,112 @@ }, { "cell_type": "markdown", - "id": "a36db155", + "id": "cell_025", "metadata": {}, "source": [ - "Note that even though 'delta' doesn't appear at all, it gets a count (of zero). This tracking of missing values can be quite handy.\n", + "Notice that \"delta\" is not shown because value_counts() only reports values that actually occur in the data.\n", "\n", - "`mode()` is another one:" + "However, the Enum still knows that \"delta\" is a valid category.\n", + "\n", + "You can inspect the complete set of categories stored in the Enum:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "f52d5d0d", + "id": "cell_026", "metadata": {}, "outputs": [], "source": [ - "df[\"cat_type\"].mode()" + "df[\"cat_type\"].dtype.categories" ] }, { "cell_type": "markdown", - "id": "d28f34d8", + "id": "cell_027", "metadata": {}, "source": [ - "And if your categorical column happens to consist of *elements* that can undergo operations, those same operations will still work. For example," + "If you want a frequency table that includes unused categories, combine the stored category list with the observed counts:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "4d43e94d", + "id": "cell_028", "metadata": {}, "outputs": [], "source": [ - "time_df = pd.DataFrame(\n", - " pd.Series(pd.date_range(\"2015/05/01\", periods=5, freq=\"M\"), dtype=\"category\"),\n", - " columns=[\"datetime\"],\n", + "all_categories = pl.DataFrame(\n", + " {\"cat_type\": df[\"cat_type\"].dtype.categories.cast(df[\"cat_type\"].dtype)}\n", + ")\n", + "all_categories.join(df[\"cat_type\"].value_counts(), on=\"cat_type\", how=\"left\").fill_null(\n", + " 0\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "cell_029", + "metadata": {}, + "source": [ + "This demonstrates one advantage of `Enum`: the complete set of allowed categories is retained, even when some categories are not present in the data.\n", + "\n", + "You can also inspect the datatype of the column:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_030", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"cat_type\"].dtype" + ] + }, + { + "cell_type": "markdown", + "id": "cell_031", + "metadata": {}, + "source": [ + "You can compute the most frequent category:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_032", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"cat_type\"].drop_nulls().mode()" + ] + }, + { + "cell_type": "markdown", + "id": "cell_033", + "metadata": {}, + "source": [ + "And if your categorical column happens to consist of _elements_ that can undergo operations, those same operations will still work. For example:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cell_034", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime\n", + "\n", + "time_df = pl.DataFrame(\n", + " {\n", + " \"datetime\": pl.date_range(\n", + " datetime.date(2015, 5, 31), datetime.date(2015, 9, 30), \"1mo\", eager=True\n", + " )\n", + " .cast(pl.String)\n", + " .cast(pl.Categorical)\n", + " }\n", ")\n", "time_df" ] @@ -327,36 +418,33 @@ { "cell_type": "code", "execution_count": null, - "id": "db697f86", + "id": "cell_035", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].dt.month" + "time_df[\"datetime\"].cast(pl.String).str.to_date().dt.month()" ] }, { "cell_type": "markdown", - "id": "c202f0d1", + "id": "cell_036", "metadata": {}, "source": [ - "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.cat.codes` to get unique codes for each value." + "Finally, if you ever need to translate your actual data types in your categorical column into a code, you can use `.to_physical()` to get unique codes for each value.\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "13e7bc66", + "id": "cell_037", "metadata": {}, "outputs": [], "source": [ - "time_df[\"datetime\"].cat.codes" + "time_df[\"datetime\"].to_physical()" ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -364,7 +452,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "python4ds (3.12.12.final.0)", "language": "python", "name": "python3" }, @@ -378,7 +466,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.13" + "version": "3.12.12" }, "toc-showtags": true }, diff --git a/data-tidy.ipynb b/data-tidy.ipynb index 157081f..a1c90b2 100644 --- a/data-tidy.ipynb +++ b/data-tidy.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Tidy Data {#sec-data-tidy}\n", @@ -11,7 +11,7 @@ "\n", "In this chapter, you will learn a consistent way to organise your data in Python using the principle known as *tidy data*. Tidy data is not appropriate for everything, but for a lot of analysis and a lot of tabular data it will be what you need. Getting your data into this format requires some work up front, but that work pays off in the long term. Once you have tidy data, you will spend much less time munging data from one representation to another, allowing you to spend more time on the data questions you care about.\n", "\n", - "In this chapter, you'll first learn the definition of tidy data and see it applied to simple toy dataset. Then we'll dive into the main tool you'll use for tidying data: melting. Melting allows you to change the form of your data, without changing any of the values. We'll finish up with a discussion of usefully untidy data, and how you can create it if needed.\n", + "In this chapter, you'll first learn the definition of tidy data and see it applied to simple toy dataset. Then we'll dive into the main tool you'll use for tidying data: unpivoting. Unpivoting allows you to change wide data into a long format without altering any underlying values. We'll finish up with a discussion of usefully untidy data, and how you can create it if needed.\n", "\n", "If you particularly enjoy this chapter and want to learn more about the underlying theory, you can learn more in the [Tidy Data](https://www.jstatsoft.org/article/view/v059i10) paper published in the Journal of Statistical Software.\n" ] @@ -19,7 +19,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "1", "metadata": { "tags": [ "remove-cell" @@ -38,17 +38,17 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package." ] }, { "cell_type": "markdown", - "id": "8b5f8255", + "id": "3", "metadata": {}, "source": [ "## Tidy Data\n", @@ -68,12 +68,12 @@ "1. There's a general advantage to picking one consistent way of storing data.\n", " If you have a consistent data structure, it's easier to learn the tools that work with it because they have an underlying uniformity. Some tools, for example data visualisation package **seaborn**, are designed with tidy data in mind.\n", "\n", - "2. There's a specific advantage to placing variables in columns because it allows you to take advantage of **pandas**' vectorised operations (operations that are more efficient).\n", + "2. There's a specific advantage to placing variables in columns because it allows you to take advantage of **polars**' vectorised operations (operations that are more efficient).\n", "\n", "\n", "Tidy data aren't going to be appropriate *every* time and in every case, but they're a really, really good default for tabular data. Once you use it as your default, it's easier to think about how to perform subsequent operations.\n", "\n", - "Having said that tidy data are great, they are, but one of **pandas**' advantages relative to other data analysis libraries is that it isn't *too* tied to tidy data and can navigate awkward non-tidy data manipulation tasks happily too.\n", + "Having said that tidy data are great, they are, but one of **polars**' advantages relative to other data analysis libraries is that it isn't *too* tied to tidy data and can navigate awkward non-tidy data manipulation tasks happily too.\n", "\n", "There are two common problems you find in data that are ingested that make them not tidy:\n", "\n", @@ -89,20 +89,20 @@ }, { "cell_type": "markdown", - "id": "deb8cf13", + "id": "4", "metadata": {}, "source": [ - "## Tools to Make Data Tidy with **pandas**" + "## Tools to Make Data Tidy with **polars**" ] }, { "cell_type": "markdown", - "id": "bd134b04", + "id": "5", "metadata": {}, "source": [ - "### Melt\n", + "### Unpivot\n", "\n", - "`melt()` can help you go from \"wider\" data to \"longer\" data, and is a *really* good one to remember.\n", + "`unpivot()` can help you go from \"wider\" data to \"longer\" data, and is a *really* good one to remember.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_melt.png)\n", "\n", @@ -112,13 +112,13 @@ { "cell_type": "code", "execution_count": null, - "id": "0f9fbf5a", + "id": "6", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", " \"first\": [\"John\", \"Mary\"],\n", " \"last\": [\"Doe\", \"Bo\"],\n", @@ -127,19 +127,24 @@ " \"weight\": [130, 150],\n", " }\n", ")\n", - "print(\"\\n Unmelted: \")\n", + "print(\"\\n Wide DataFrame: \")\n", "print(df)\n", - "print(\"\\n Melted: \")\n", - "df.melt(id_vars=[\"first\", \"last\"], var_name=\"quantity\", value_vars=[\"height\", \"weight\"])" + "print(\"\\n Unpivoted (Long) DataFrame: \")\n", + "df.unpivot(\n", + " index=[\"first\", \"last\"],\n", + " variable_name=\"quantity\",\n", + " value_name=\"value\",\n", + " on=[\"height\", \"weight\"],\n", + ")" ] }, { "cell_type": "markdown", - "id": "e62978fd", + "id": "7", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", - "Perform a `melt()` that uses `job` as the id instead of `first` and `last`.\n", + "Perform a `unpivot()` that uses `job` as the id instead of `first` and `last`.\n", ":::\n", "\n", "How does this relate to tidy data? Sometimes you'll have a variable spread over multiple columns that you want to turn tidy. Let's look at this example that uses cases of [tuburculosis from the World Health Organisation](https://www.who.int/teams/global-tuberculosis-programme/data).\n", @@ -150,11 +155,11 @@ { "cell_type": "code", "execution_count": null, - "id": "bfa121cf", + "id": "8", "metadata": {}, "outputs": [], "source": [ - "df_tb = pd.read_parquet(\n", + "df_tb = pl.read_parquet(\n", " \"https://github.com/aeturrell/python4DS/raw/refs/heads/main/data/who_tb_cases.parquet\"\n", ")\n", "df_tb.head()" @@ -162,30 +167,30 @@ }, { "cell_type": "markdown", - "id": "583e419d", + "id": "9", "metadata": {}, "source": [ - "You can see that we have two columns for a single variable, year. Let's now melt this." + "You can see that we have two columns for a single variable, year. Let's now unpivot this." ] }, { "cell_type": "code", "execution_count": null, - "id": "dc03ccd9", + "id": "10", "metadata": {}, "outputs": [], "source": [ - "df_tb.melt(\n", - " id_vars=[\"country\"],\n", - " var_name=\"year\",\n", - " value_vars=[\"1999\", \"2000\"],\n", + "df_tb.unpivot(\n", + " index=[\"country\"],\n", + " variable_name=\"year\",\n", " value_name=\"cases\",\n", + " on=[\"1999\", \"2000\"],\n", ")" ] }, { "cell_type": "markdown", - "id": "74d81b30", + "id": "11", "metadata": {}, "source": [ "We now have one observation per row, and one variable per column: tidy!" @@ -193,31 +198,31 @@ }, { "cell_type": "markdown", - "id": "488f5f10", + "id": "12", "metadata": {}, "source": [ - "### A simpler wide to long\n", + "### Reshaping Wide Data with Multiple Variables\n", "\n", - "If you don't want the headscratching of `melt()`, there's also `wide_to_long()`, which is really useful for typical data cleaning cases where you have data like this:" + "Sometimes you have multiple variables spread across columns in a wide format. Here's how to handle that in polars:" ] }, { "cell_type": "code", "execution_count": null, - "id": "293768c1", + "id": "13", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", - " \"A1970\": {0: \"a\", 1: \"b\", 2: \"c\"},\n", - " \"A1980\": {0: \"d\", 1: \"e\", 2: \"f\"},\n", - " \"B1970\": {0: 2.5, 1: 1.2, 2: 0.7},\n", - " \"B1980\": {0: 3.2, 1: 1.3, 2: 0.1},\n", - " \"X\": dict(zip(range(3), np.random.randn(3))),\n", - " \"id\": dict(zip(range(3), range(3))),\n", + " \"A1970\": [\"a\", \"b\", \"c\"],\n", + " \"A1980\": [\"d\", \"e\", \"f\"],\n", + " \"B1970\": [2.5, 1.2, 0.7],\n", + " \"B1980\": [3.2, 1.3, 0.1],\n", + " \"X\": np.random.randn(3),\n", + " \"id\": [0, 1, 2],\n", " }\n", ")\n", "df" @@ -225,147 +230,154 @@ }, { "cell_type": "markdown", - "id": "fb6c09e6", + "id": "14", "metadata": {}, "source": [ - "i.e. data where there are different variables and time periods across the columns. Wide to long is going to let us give info on what the stubnames are ('A', 'B'), the name of the variable that's always across columns (here, a year), any values (X here), and an id column." + "To reshape this data, we first unpivot, then extract the year and variable:" ] }, { "cell_type": "code", "execution_count": null, - "id": "a9ca2fa8", + "id": "15", "metadata": {}, "outputs": [], "source": [ - "pd.wide_to_long(df, stubnames=[\"A\", \"B\"], i=\"id\", j=\"year\")" + "df_long = df.unpivot(\n", + " index=[\"id\", \"X\"],\n", + " variable_name=\"variable_year\",\n", + " value_name=\"value\",\n", + " on=[\"A1970\", \"A1980\", \"B1970\", \"B1980\"],\n", + ")\n", + "\n", + "df_long = df_long.with_columns(\n", + " [\n", + " pl.col(\"variable_year\").str.slice(0, 1).alias(\"variable\"),\n", + " pl.col(\"variable_year\").str.slice(1, 4).cast(pl.Int64).alias(\"year\"),\n", + " ]\n", + ")\n", + "\n", + "df_final = df_long.drop(\"variable_year\").pivot(\n", + " index=[\"id\", \"X\", \"year\"],\n", + " on=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ")\n", + "\n", + "df_final.sort([\"id\", \"year\"])" ] }, { "cell_type": "markdown", - "id": "1a6bf17c", + "id": "16", "metadata": {}, "source": [ - "### Stack and Unstack\n", + "### Reshaping Multi-Column Data\n", + "\n", + "Polars uses unpivot() and pivot() for reshaping. \n", "\n", - "Stack, `stack()` is a shortcut for taking a single type of wide data variable from columns and turning it into a long form dataset, but with an extra index.\n", + "In **polars**, reshaping between long and wide tabular formats is performed seamlessly using `unpivot()` (to go long) and `pivot()` (to go wide), without needing row indices or complex MultiIndexes.\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_stack.png)\n", "\n", - "Unstack, `unstack()` unsurprisingly does the same operation, but in reverse.\n", + "To go back to wider format (unstack)\n", "\n", "![](https://pandas.pydata.org/docs/_images/reshaping_unstack.png)\n", "\n", - "Let's define a multi-index data frame to demonstrate this:" + "Here's how to work with hierarchical data:" ] }, { "cell_type": "code", "execution_count": null, - "id": "2b791dd1", + "id": "17", "metadata": {}, "outputs": [], "source": [ - "tuples = list(\n", - " zip(\n", - " *[\n", - " [\"bar\", \"bar\", \"baz\", \"baz\", \"foo\", \"foo\", \"qux\", \"qux\"],\n", - " [\"one\", \"two\", \"one\", \"two\", \"one\", \"two\", \"one\", \"two\"],\n", - " ]\n", - " )\n", + "# Create a DataFrame with multiple columns\n", + "df = pl.DataFrame(\n", + " {\n", + " \"first\": [\"bar\", \"bar\", \"baz\", \"baz\", \"foo\", \"foo\", \"qux\", \"qux\"],\n", + " \"second\": [\"one\", \"two\", \"one\", \"two\", \"one\", \"two\", \"one\", \"two\"],\n", + " \"A\": np.random.randn(8),\n", + " \"B\": np.random.randn(8),\n", + " }\n", ")\n", - "index = pd.MultiIndex.from_tuples(tuples, names=[\"first\", \"second\"])\n", - "df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=[\"A\", \"B\"])\n", "df" ] }, { "cell_type": "markdown", - "id": "38e7c349", + "id": "18", "metadata": {}, "source": [ - "Let's stack this to create a tidy dataset:" + "To go to a longer format (stack):" ] }, { "cell_type": "code", "execution_count": null, - "id": "d25eb012", + "id": "19", "metadata": {}, "outputs": [], "source": [ - "df = df.stack()\n", - "df" - ] - }, - { - "cell_type": "markdown", - "id": "edeef123", - "metadata": {}, - "source": [ - "This has automatically created a multi-layered index but that can be reverted to a numbered index using `df.reset_index()`" + "# Unpivot to stack the A and B columns\n", + "df_stacked = df.unpivot(\n", + " index=[\"first\", \"second\"],\n", + " variable_name=\"variable\",\n", + " value_name=\"value\",\n", + " on=[\"A\", \"B\"],\n", + ")\n", + "df_stacked" ] }, { "cell_type": "markdown", - "id": "6b1e4e03", + "id": "20", "metadata": {}, "source": [ - "Now let's see unstack but, instead of unstacking the 'A', 'B' variables we began with, let's unstack the 'first' column by passing `level=0` (the default is to unstack the innermost index). This diagram shows what's going on:\n", - "\n", - "![](https://pandas.pydata.org/docs/_images/reshaping_unstack_0.png)\n", - "\n", - "And here's the code:" + "then to wider format (unstack)" ] }, { "cell_type": "code", "execution_count": null, - "id": "b6742a54", + "id": "21", "metadata": {}, "outputs": [], "source": [ - "df.unstack(level=0)" - ] - }, - { - "cell_type": "markdown", - "id": "185eae1f", - "metadata": {}, - "source": [ - "::: {.callout-tip title=\"Exercise\"}\n", - "What happens if you unstack to `level=1` instead? What about applying `unstack()` twice?\n", - ":::" + "# Pivot back to wider format\n", + "df_unstacked = df_stacked.pivot(\n", + " index=[\"first\", \"second\"],\n", + " on=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ")\n", + "df_unstacked" ] }, { "cell_type": "markdown", - "id": "39a210bb", + "id": "22", "metadata": {}, "source": [ - "### Pivoting data from long to wide\n", + "### Pivoting from Long to Wide\n", "\n", - "`pivot()` and `pivot_table()` help you to sort out data in which a single observation is scattered over multiple rows.\n", + "pivot() helps you to sort out data in which a single observation is scattered over multiple rows.\n", + "\n", + "![](https://pandas.pydata.org/docs/_images/reshaping_pivot.png)\n", "\n", - "![](https://pandas.pydata.org/docs/_images/reshaping_pivot.png)\n" - ] - }, - { - "cell_type": "markdown", - "id": "8acbcedf", - "metadata": {}, - "source": [ "Here's an example dataframe where observations are spread over multiple rows:" ] }, { "cell_type": "code", "execution_count": null, - "id": "fa612456", + "id": "23", "metadata": {}, "outputs": [], "source": [ - "df_tb_cp = pd.read_parquet(\n", + "df_tb_cp = pl.read_parquet(\n", " \"https://github.com/aeturrell/python4DS/raw/refs/heads/main/data/who_tb_case_and_pop.parquet\"\n", ")\n", "df_tb_cp.head()" @@ -373,92 +385,107 @@ }, { "cell_type": "markdown", - "id": "0d4a4077", - "metadata": {}, - "source": [ - "You see that we have, for each year-country, \"case\" and \"population\" in different rows." - ] - }, - { - "cell_type": "markdown", - "id": "e7c9ed1b", + "id": "24", "metadata": {}, "source": [ + "You see that we have, for each year-country, \"case\" and \"population\" in different rows.\n", + "\n", "Now let's pivot this to see the difference:" ] }, { "cell_type": "code", "execution_count": null, - "id": "e584cf37", + "id": "25", "metadata": {}, "outputs": [], "source": [ "pivoted = df_tb_cp.pivot(\n", - " index=[\"country\", \"year\"], columns=[\"type\"], values=\"count\"\n", - ").reset_index()\n", + " index=[\"country\", \"year\"],\n", + " on=\"type\",\n", + " values=\"count\",\n", + " aggregate_function=\"first\",\n", + ").sort([\"country\", \"year\"])\n", "pivoted" ] }, { "cell_type": "markdown", - "id": "d82307c0", + "id": "26", "metadata": {}, "source": [ - "Pivots are especially useful for time series data, where operations like `shift()` or `diff()` are typically applied assuming that an entry in one row follows (in time) from the one above. When we do `shift()` we often want to shift a single variable in time, but if a single observation (in this case a date) is over multiple rows, the timing is going go awry. Let's see an example." + "Pivots are especially useful for time series data, where operations like `shift()` are typically applied assuming that an entry in one row follows (in time) from the one above. When we do `shift()` we often want to shift a single variable in time, but if a single observation (in this case a date) is over multiple rows, the timing goes awry. Let's see an example." ] }, { "cell_type": "code", "execution_count": null, - "id": "97c6d139", + "id": "27", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", + "import polars as pl\n", + "\n", + "dates = list(\n", + " pl.date_range(\n", + " start=pl.date(2000, 1, 31),\n", + " end=pl.date(2000, 10, 31),\n", + " interval=\"1mo\",\n", + " eager=True,\n", + " )\n", + ")\n", "\n", "data = {\n", " \"value\": np.random.randn(20),\n", " \"variable\": [\"A\"] * 10 + [\"B\"] * 10,\n", - " \"date\": (\n", - " list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", - " + list(pd.date_range(\"1/1/2000\", periods=10, freq=\"ME\"))\n", - " ),\n", + " \"date\": dates + dates,\n", "}\n", - "df = pd.DataFrame(data, columns=[\"date\", \"variable\", \"value\"])\n", + "df = pl.DataFrame(data)\n", "df.sample(5)" ] }, { "cell_type": "markdown", - "id": "90a5b930", + "id": "28", "metadata": {}, "source": [ - "If we just run `shift()` on the above, it's going to shift variable B's and A's together even though these overlap in time and are different variables. So we pivot to a wider format (and then we can shift in time safely)." + "If we just run `shift()` on the above, it's going to shift variable B's and A's together even though they overlap in time and are different variables. So we pivot to a wider format to shift safely:" ] }, { "cell_type": "code", "execution_count": null, - "id": "04f2bd28", + "id": "29", "metadata": {}, "outputs": [], "source": [ - "df.pivot(index=\"date\", columns=\"variable\", values=\"value\").shift(1)" + "df_pivoted = df.pivot(\n", + " index=\"date\",\n", + " on=\"variable\",\n", + " values=\"value\",\n", + " aggregate_function=\"first\",\n", + ").with_columns(\n", + " [\n", + " pl.col(\"A\").shift(1),\n", + " pl.col(\"B\").shift(1),\n", + " ]\n", + ")\n", + "df_pivoted" ] }, { "cell_type": "markdown", - "id": "841c4821", + "id": "30", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", - "Why is the first entry NaN?\n", + "Why is the first entry `null`?\n", ":::\n", "\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", - "Perform a `pivot()` that applies to both the `variable` and `category` columns in the example from above where category is defined such that `df[\"category\"] = np.random.choice([\"type1\", \"type2\", \"type3\", \"type4\"], 20). (Hint: remember that you will need to pass multiple objects via a list.)\n", + "Perform a `pivot()` that applies to both the `variable` and `category` columns in the example from above where category is defined such that `df = df.with_columns(category=pl.Series(np.random.choice([\"type1\", \"type2\", \"type3\", \"type4\"], 20)))`. (Hint: pass multiple column names to the `on` parameter as a list.)\n", ":::" ] } diff --git a/data-transform.ipynb b/data-transform.ipynb index 8dfc2c1..ccc75f0 100644 --- a/data-transform.ipynb +++ b/data-transform.ipynb @@ -164,7 +164,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights.with_columns(pl.col(\"time_hour\").str.to_datetime())" + "flights.with_columns(pl.col(\"time_hour\").str.to_datetime(time_zone=\"UTC\"))" ] }, { diff --git a/data/bake_sale.xlsx b/data/bake_sale.xlsx index 5de6f9a..5c410a0 100644 Binary files a/data/bake_sale.xlsx and b/data/bake_sale.xlsx differ diff --git a/databases.ipynb b/databases.ipynb index b183959..d22a460 100644 --- a/databases.ipynb +++ b/databases.ipynb @@ -233,6 +233,7 @@ "\n", "import os\n", "\n", + "con_new.close()\n", "os.remove(\"data/test_database.sqlite\")" ] }, @@ -755,6 +756,8 @@ "source": [ "# remove-cell\n", "\n", + "con.close()\n", + "engine.dispose()\n", "os.remove(\"data/hero.db\")" ] }, diff --git a/dates-and-times.ipynb b/dates-and-times.ipynb index 47b53ad..2073145 100644 --- a/dates-and-times.ipynb +++ b/dates-and-times.ipynb @@ -51,13 +51,7 @@ "source": [ "### Prerequisites\n", "\n", - "You will need to install the **seaborn** package for this chapter. This chapter uses the next generation version of **seaborn**, which can be installed by running the following on the command line (aka in the terminal): \n", - "\n", - "```bash\n", - "uv run pip install --pre seaborn\n", - "```\n", - "\n", - "We will also be using the **pandas** package and numerical package **numpy**." + "This chapter focuses on working with dates and time series using **polars**. We'll also use **matplotlib** for visualising time series data. Make sure you have **polars** installed (using `uv add polars` in the terminal)." ] }, { @@ -69,7 +63,7 @@ "\n", "A point in time as represented in data science is composed of a clock time and a date. These two elements are brought together as a *datetime*.\n", "\n", - "The datetime object is the fundamental time object in Python. It’s useful to know about these before moving on to datetime operations using **pandas** (which you’re far more likely to use in practice). Python's *datetime* objects capture the year, month, day, hour, second, and microsecond. Let’s import the class that deals with datetimes (whose objects are of type datetime.datetime) and take a look at it." + "The datetime object is the fundamental time object in Python. It’s useful to know about these before moving on to datetime operations using **polars** (which you’re far more likely to use in practice). Python's *datetime* objects capture the year, month, day, hour, second, and microsecond. Let’s import the class that deals with datetimes (whose objects are of type datetime.datetime) and take a look at it." ] }, { @@ -356,7 +350,7 @@ "source": [ "## Vectorised Datetimes \n", "\n", - "Now we come to vectorised operations on datetimes using the powerful **numpy** packages (and this is what is used by **pandas**). **numpy** has its own version of datetime, called `np.datetime64`, and it's very efficient at scale. Let's see it in action:" + "Now we come to vectorised operations on datetimes using **numpy**. **numpy** has its own version of datetime, called `np.datetime64`, which is very efficient at scale. Let's see it in action:" ] }, { @@ -415,7 +409,7 @@ "id": "c0a974aa", "metadata": {}, "source": [ - "One word of warning with **numpy** and datetimes though: the more precise you go, and you can go down to femtoseconds ($10^{-15}$ seconds), the more precise you go the smaller the range of dates you can hit. A popular choice of precision is `datetime64[ns]`, which can encode times from 1678 AD to 2262 AD. Working with seconds gets you 2.9$\\times 10^9$ BC to 2.9$\\times 10^9$ AD." + "One word of warning with **numpy** and datetimes though: the more precise you go, and you can go down to femtoseconds ($10^{-15}$ seconds), the smaller the range of dates you can hit. A popular choice of precision is `datetime64[ns]`, which can encode times from 1678 AD to 2262 AD. Working with seconds gets you 2.9$\\times 10^9$ BC to 2.9$\\times 10^9$ AD." ] }, { @@ -431,7 +425,7 @@ "id": "82a4a3f3", "metadata": {}, "source": [ - "[**pandas**](https://pandas.pydata.org/) is the workhorse of time series analysis in Python. The basic object is a *timestamp*. The `pd.to_datetime()` function creates timestamps from strings that could reasonably represent datetimes. Let's see an example of using `pd.to_datetime()` to create a timestamp and then take a look at it." + "[**polars**](https://docs.pola.rs/api/python/stable/reference/dataframe/index.html) is the workhorse of time series analysis in Python. The basic object is a *Datetime* type. Let's create a datetime from a string: " ] }, { @@ -441,9 +435,9 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "date = pd.to_datetime(\"16th of February, 2020\")\n", + "date = pl.Series([\"2020-02-16\"]).str.strptime(pl.Datetime, \"%Y-%m-%d\")\n", "date" ] }, @@ -452,7 +446,7 @@ "id": "8b083619", "metadata": {}, "source": [ - "This is of type `Timestamp` and you can see that it has many of the same properties as the built-in Python `datetime.datetime` class from the previous chapter. As with that, the default setting for `tz` (timezone) and `tzinfo` is `None`. There are some extra properties, though, such as `freq` for frequency, which will be very useful when it comes to manipulating time *series* as opposed to just one or two datetimes." + "This creates a column of type `Datetime`. As with Python's datetime, the default timezone is `None`." ] }, { @@ -462,9 +456,9 @@ "source": [ "### Creating and Using Time Series\n", "\n", - "There are two main scenarios in which you might be creating time series using **pandas**: i) creating one from scratch or ii) reading in data from a file. Let's look at a few ways to do i) first. \n", + "There are two main scenarios in which you might be creating time series using **polars**: i) creating one from scratch or ii) reading in data from a file. Let's look at a few ways to do i) first. \n", "\n", - "You can create a time series with **pandas** by taking a date as created above and extending it using **pandas** timedelta function:" + "You can create a time series with **polars** using `pl.datetime_range()`:" ] }, { @@ -474,7 +468,11 @@ "metadata": {}, "outputs": [], "source": [ - "date + pd.to_timedelta(np.arange(12), \"D\")" + "from datetime import datetime\n", + "\n", + "pl.datetime_range(\n", + " start=datetime(2018, 1, 1), end=datetime(2018, 1, 8), interval=\"1d\", eager=True\n", + ")" ] }, { @@ -482,9 +480,7 @@ "id": "3ba587ff", "metadata": {}, "source": [ - "This has created a datetime index of type `datetime65[ns]` (remember, an index is a special type of **pandas** column), where \"ns\" stands for nano-second resolution.\n", - "\n", - "Another method is to create a range of dates (pass a frequency using the `freq=` keyword argument):" + "Another method is to specify the number of periods and the frequency:" ] }, { @@ -494,7 +490,14 @@ "metadata": {}, "outputs": [], "source": [ - "pd.date_range(start=\"2018/1/1\", end=\"2018/1/8\")" + "from datetime import datetime\n", + "\n", + "pl.datetime_range(\n", + " start=datetime(2018, 1, 1, 0, 0),\n", + " end=datetime(2018, 1, 1, 2, 0),\n", + " interval=\"1h\",\n", + " eager=True,\n", + ")" ] }, { @@ -502,7 +505,7 @@ "id": "3f6c8a4f", "metadata": {}, "source": [ - "Another way to create ranges is to specify the number of periods and the frequency:" + "Polars also supports timezone operations:" ] }, { @@ -512,26 +515,16 @@ "metadata": {}, "outputs": [], "source": [ - "pd.date_range(\"2018-01-01\", periods=3, freq=\"h\")" - ] - }, - { - "cell_type": "markdown", - "id": "ce8c06d8", - "metadata": {}, - "source": [ - "Following the discussion of the previous chapter on timezones, you can also localise timezones directly in **pandas** data frames:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "6703682c", - "metadata": {}, - "outputs": [], - "source": [ - "dti = pd.date_range(\"2018-01-01\", periods=3, freq=\"h\").tz_localize(\"UTC\")\n", - "dti.tz_convert(\"US/Pacific\")" + "from datetime import datetime\n", + "\n", + "dti = pl.datetime_range(\n", + " start=datetime(2018, 1, 1, 0, 0),\n", + " end=datetime(2018, 1, 1, 2, 0),\n", + " interval=\"1h\",\n", + " eager=True,\n", + ").dt.replace_time_zone(\"UTC\")\n", + "\n", + "dti.dt.convert_time_zone(\"US/Pacific\")" ] }, { @@ -539,7 +532,7 @@ "id": "4d3e7668", "metadata": {}, "source": [ - "Now let's see how to turn data that has been read in with a non-datetime type into a vector of datetimes. This happens *all the time* in practice. We'll read in some data on job vacancies for information and communication jobs, ONS code UNEM-JP9P, and then try to wrangle the given \"date\" column into a **pandas** datetime column." + "Now let's see how to turn data that has been read in with a non-datetime type into a vector of datetimes. This happens *all the time* in practice. We'll read in some data on job vacancies for information and communication jobs, ONS code UNEM-JP9P, and then try to wrangle the given \"date\" column into a **polars** datetime column." ] }, { @@ -553,12 +546,15 @@ "\n", "url = \"https://api.beta.ons.gov.uk/v1/data?uri=/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/timeseries/jp9z/lms/previous/v108\"\n", "\n", - "# Get the data from the ONS API:\n", + "# Get the data from the ONS API\n", "json_data = requests.get(url).json()\n", - "df = pd.DataFrame(pd.json_normalize(json_data[\"months\"]))\n", - "df[\"value\"] = pd.to_numeric(df[\"value\"])\n", - "df = df[[\"date\", \"value\"]]\n", - "df = df.rename(columns={\"value\": \"Vacancies (ICT), thousands\"})\n", + "df = pl.DataFrame(json_data[\"months\"])\n", + "df = df.select(\n", + " [\n", + " pl.col(\"date\"),\n", + " pl.col(\"value\").cast(pl.Float64).alias(\"Vacancies (ICT), thousands\"),\n", + " ]\n", + ")\n", "df.head()" ] }, @@ -567,7 +563,7 @@ "id": "1be10bf8", "metadata": {}, "source": [ - "We have the data in. Let's look at the column types that arrived." + "We have the data in. Let's look at the column types:" ] }, { @@ -577,7 +573,17 @@ "metadata": {}, "outputs": [], "source": [ - "df.info()" + "df.schema" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "924ce7e4", + "metadata": {}, + "outputs": [], + "source": [ + "df[\"date\"].head(10)" ] }, { @@ -585,7 +591,7 @@ "id": "71bbbb02", "metadata": {}, "source": [ - "This is the default 'object' type, but we want the date column to have `datetime64[ns]`, which is a datetime type. Again, we use `pd.to_datetime()`:" + "We want the date column to have `Datetime` type. We can use `str.strptime()`:" ] }, { @@ -595,7 +601,10 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"date\"] = pd.to_datetime(df[\"date\"])\n", + "df = df.with_columns(\n", + " pl.col(\"date\").str.to_titlecase().str.strptime(pl.Date, format=\"%Y %b\")\n", + ")\n", + "\n", "df[\"date\"].head()" ] }, @@ -604,7 +613,7 @@ "id": "ca0b1727", "metadata": {}, "source": [ - "In this case, the conversion from the format of data that was put in of \"2001 MAY\" to datetime worked out-of-the-box. `pd.to_datetime` will always take an educated guess as to the format, but it won't always work out.\n", + "In this case, the conversion from the format of data that was put in of \"2001 MAY\" to datetime worked out-of-the-box.\n", "\n", "What happens if we have a more tricky-to-read-in datetime column? This frequently occurs in practice so it's well worth exploring an example. Let's create some random data with dates in an unusual format with month first, then year, then day, eg \"1, '19, 29\" and so on." ] @@ -616,7 +625,7 @@ "metadata": {}, "outputs": [], "source": [ - "small_df = pd.DataFrame({\"date\": [\"1, '19, 22\", \"1, '19, 23\"], \"values\": [\"1\", \"2\"]})\n", + "small_df = pl.DataFrame({\"date\": [\"1, '19, 22\", \"1, '19, 23\"], \"values\": [\"1\", \"2\"]})\n", "small_df[\"date\"]" ] }, @@ -625,7 +634,7 @@ "id": "b1e6f80c", "metadata": {}, "source": [ - "Now, if we were to run this via `pd.to_datetime` with no further input, it would misinterpret, for example, the first date as `2022-01-19`. So we must pass a bit more info to `pd.to_datetime` to help it out. We can pass a `format=` keyword argument with the format that the datetime takes. Here, we'll use `%m` for month in number format, `%y` for year in 2-digit format, and `%d` for 2-digit day. We can also add in the other characters such as `'` and `,`. You can find a list of datetime format identifiers above or over at [https://strftime.org/](https://strftime.org/)." + "Now, if we were to run this via `str.strptime()` with no further input, it would misinterpret the format. So we must pass a `format=` keyword argument with the format that the datetime takes. Here, we'll use `%m` for month in number format, `%y` for year in 2-digit format, and `%d` for 2-digit day. We can also add in the other characters such as `'` and `,`." ] }, { @@ -635,7 +644,7 @@ "metadata": {}, "outputs": [], "source": [ - "pd.to_datetime(small_df[\"date\"], format=\"%m, '%y, %d\")" + "small_df.with_columns(pl.col(\"date\").str.strptime(pl.Datetime, format=\"%m, '%y, %d\"))" ] }, { @@ -645,7 +654,7 @@ "source": [ "### Datetime Offsets\n", "\n", - "Our data, currently held in `df`, were read in as if they were from the *start* of the month but these data refer to the month that has passed and so should be for the *end* of the month. Fortunately, we can change this using a time offset." + "Our data, currently held in `df`, were read in as if they were recorded at the start of the month, e.g., 2001-05-01. But they are actually monthly averages for the given month, so snapping them to the end of the month makes more sense. We can use `.dt.month_end()` to snap dates to the last day of the month:" ] }, { @@ -655,18 +664,10 @@ "metadata": {}, "outputs": [], "source": [ - "df[\"date\"] = df[\"date\"] + pd.offsets.MonthEnd()\n", + "df = df.with_columns(pl.col(\"date\").dt.month_end())\n", "df.head()" ] }, - { - "cell_type": "markdown", - "id": "61ef45b4", - "metadata": {}, - "source": [ - "While we used the `MonthEnd` offset here, there are many different offsets available. You can find a [full table of date offsets here](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects)." - ] - }, { "cell_type": "markdown", "id": "5afd54be", @@ -674,7 +675,7 @@ "source": [ "### The `.dt` accessor\n", "\n", - "When you have a datetime column, you can use the `.dt` accessor to grab lots of useful information from it such as the `minute`, `month`, and so on. Some that are functions, rather than just accessors of underlying properties, are followed by brackets, `()`, because they are functions. Here are a few useful examples:" + "When you have a datetime column, you can use the `.dt` accessor to grab lots of useful information from it such as the `minute`, `month`, and so on. Here are a few useful examples:" ] }, { @@ -684,12 +685,12 @@ "metadata": {}, "outputs": [], "source": [ - "print(\"Using `dt.day_name()`\")\n", - "print(df[\"date\"].dt.day_name().head())\n", - "print(\"Using `dt.isocalendar()`\")\n", - "print(df[\"date\"].dt.isocalendar().head())\n", - "print(\"Using `dt.month`\")\n", - "print(df[\"date\"].dt.month.head())" + "print(\"Using `dt.to_string()`\")\n", + "print(df[\"date\"].dt.to_string(\"%A\").head())\n", + "print(\"Using `dt.weekday()`\")\n", + "print(df[\"date\"].dt.weekday().head())\n", + "print(\"Using `dt.month()`\")\n", + "print(df[\"date\"].dt.month().head())" ] }, { @@ -697,9 +698,9 @@ "id": "a40f982f", "metadata": {}, "source": [ - "### Creating a datetime Index and Setting the Frequency\n", + "### Sorting and Setting Up Datetime Series\n", "\n", - "For the subsequent parts, we'll set the datetime column to be the index of the data frame. *This is the standard setup you will likely want to use when dealing with time series.*" + "For time series operations in **polars**, you typically keep the datetime as a column and use it in group operations. First, let's ensure our datetime column is sorted:" ] }, { @@ -709,7 +710,7 @@ "metadata": {}, "outputs": [], "source": [ - "df = df.set_index(\"date\")\n", + "df = df.sort(\"date\")\n", "df.head()" ] }, @@ -718,7 +719,7 @@ "id": "8d5017ed", "metadata": {}, "source": [ - "Now, if we look at the first few entries of the index of data frame (a datetime index) using `head` as above, we'll see that the `freq=` parameter is set as `None`." + "Polars supports resampling operations using `group_by_dynamic()`:" ] }, { @@ -728,58 +729,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.index[:5]" - ] - }, - { - "cell_type": "markdown", - "id": "3bd0fe54", - "metadata": {}, - "source": [ - "This can be set for the whole data frame using the `asfreq()` function:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9146c99d", - "metadata": {}, - "outputs": [], - "source": [ - "df = df.asfreq(\"M\")\n", - "df.index[:5]" - ] - }, - { - "cell_type": "markdown", - "id": "8f024684", - "metadata": {}, - "source": [ - "Although most of the time it doesn't matter about the fact that `freq=None`, some aggregation operations need to know the frequency of the time series in order to work and it's good practice to set it if your data *are* regular. You can use `asfreq` to go from a higher frequency to a lower frequency too: the last entry from the higher frequency that aligns with the lower frequency will be taken, for example in going from months to years, December's value would be used.\n", - "\n", - "Note that trying to set the frequency when your datetime index doesn't match up to a particular frequency will cause errors or problems.\n", - "\n", - "A few useful frequencies to know about are in the table below; all of these can be used with `pd.to_datetime()` too.\n", - "\n", - "| Code | Represents |\n", - "|-------|---------------------------------------------------------------------|\n", - "| D | Calendar day |\n", - "| W | Weekly |\n", - "| M | Month end |\n", - "| Q | Quarter end |\n", - "| A | Year end |\n", - "| H | Hours |\n", - "| T | Minutes |\n", - "| S | Seconds |\n", - "| B | Business day |\n", - "| BM | Business month end |\n", - "| BQ | Business quarter end |\n", - "| BA | Business year end |\n", - "| BH | Business hours |\n", - "| MS | Month start |\n", - "| QS | Quarter start |\n", - "| W-SUN | Weeks beginning with Sunday (similar for other days) |\n", - "| 2M | Every 2 months (works with other combinations of numbers and codes) |" + "# Resample to annual frequency with mean\n", + "df_resampled = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", + " pl.col(\"Vacancies (ICT), thousands\").mean()\n", + ")\n", + "df_resampled.head()" ] }, { @@ -789,7 +743,7 @@ "source": [ "## Making Quick Time Series Plots\n", "\n", - "Having managed to put your time series into a data frame, perhaps converting a column of type string into a colume of type datetime in the process, you often just want to see the thing! We can achieve this using the `plot()` command, as long as we have a datetime index.\n" + "Having managed to put your time series into a **polars** DataFrame, you often want to quickly plot the results. Converting to a **pandas** DataFrame via `.to_pandas()` is a common bridging pattern for using Matplotlib's built-in time series plotting utilities:" ] }, { @@ -799,7 +753,10 @@ "metadata": {}, "outputs": [], "source": [ - "df.plot();" + "# Convert to pandas for plotting\n", + "df_pd = df.to_pandas()\n", + "df_pd.set_index(\"date\").plot()\n", + "plt.show()" ] }, { @@ -809,11 +766,7 @@ "source": [ "## Resampling, Rolling, and Shifting\n", "\n", - "Now our data have a *datetime index*, some common time series operations are made very easy for us.\n", - "\n", - "### Resampling\n", - "\n", - "Quite frequently, there is a situation in which one would like to change the frequency of a given time series. A time index-based data frame makes this easy via the `resample()` function. `resample()` must be told *how* you'd like to resample the data, for example via the mean or median. Here's an example resampling the monthly data to annual and taking the mean:" + "Now that our data have a sorted *datetime column*, we can perform various time series operations such as resampling, rolling windows, and shifting." ] }, { @@ -823,25 +776,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample(\"A\").mean()" - ] - }, - { - "cell_type": "markdown", - "id": "d320a934", - "metadata": {}, - "source": [ - "As resample is just a special type of aggregation, it can work with all of the usual functions that aggregations do, including in-built functions or user-defined functions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fbbbcdff", - "metadata": {}, - "outputs": [], - "source": [ - "df.resample(\"5YE\").agg([\"mean\", \"std\"]).head()" + "# Resample to annual frequency\n", + "df_annual = df.group_by_dynamic(\"date\", every=\"1y\").agg(\n", + " pl.col(\"Vacancies (ICT), thousands\").mean()\n", + ")\n", + "df_annual" ] }, { @@ -849,7 +788,7 @@ "id": "c3b45c36", "metadata": {}, "source": [ - "Resampling can go up in frequency (up-sampling) as well as down, but we no longer need to choose an aggregation function, we must now choose how we'd like to fill in the gaps for the frequencies we didn't have in the original data. In the example below, they are just left as NaNs." + "Resampling can go up in frequency (up-sampling) as well as down. We can use `.upsample()` to insert missing dates at a daily frequency:" ] }, { @@ -859,7 +798,9 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample(\"D\").asfreq()" + "# Upsample to daily frequency\n", + "daily = df.upsample(time_column=\"date\", every=\"1d\")\n", + "daily.head()" ] }, { @@ -867,7 +808,7 @@ "id": "c08e1dad", "metadata": {}, "source": [ - "Options to fill in missing time series data include using `bfill` or `ffill` to fill in the blanks based on the next or last available value, respectively, or `interpolate()` (note how only the first 3 NaNs are replaced using the `limit` keyword argument):" + "Options to fill in missing time series data include using forward or backward fill:" ] }, { @@ -877,7 +818,11 @@ "metadata": {}, "outputs": [], "source": [ - "df.resample(\"D\").interpolate(method=\"linear\", limit_direction=\"forward\", limit=3)[:6]" + "# Forward fill missing values\n", + "daily_filled = daily.with_columns(\n", + " pl.col(\"Vacancies (ICT), thousands\").fill_null(strategy=\"forward\")\n", + ")\n", + "daily_filled.head()" ] }, { @@ -895,22 +840,25 @@ "metadata": {}, "outputs": [], "source": [ - "# Get stock market data\n", "import yfinance as yf\n", "\n", - "xf = yf.download(\"AAPL\", start=\"2017-01-01\", end=\"2019-06-01\")\n", - "xf = xf.sort_index()\n", - "xf.columns = xf.columns.droplevel(1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ab1efae3", - "metadata": {}, - "outputs": [], - "source": [ - "plt.rcParams[\"axes.prop_cycle\"]" + "xf = yf.download(\n", + " \"AAPL\",\n", + " start=\"2017-01-01\",\n", + " end=\"2019-06-01\",\n", + " multi_level_index=False,\n", + ")\n", + "\n", + "xf = xf.reset_index()\n", + "\n", + "xf_pl = pl.from_pandas(xf).rename(\n", + " {\n", + " \"Date\": \"date\",\n", + " \"Close\": \"close\",\n", + " }\n", + ")\n", + "\n", + "xf_pl.head()" ] }, { @@ -920,21 +868,9 @@ "metadata": {}, "outputs": [], "source": [ - "from itertools import cycle\n", - "\n", - "fig, ax = plt.subplots()\n", - "data = xf.iloc[:10, 3]\n", - "colour_wheel = cycle(plt.rcParams[\"axes.prop_cycle\"])\n", - "\n", - "data.asfreq(\"D\").plot(ax=ax, marker=\"o\", linestyle=\"None\", zorder=3)\n", - "data.asfreq(\"D\", method=\"bfill\").plot(\n", - " ax=ax, style=\"-.o\", lw=1, color=next(colour_wheel)[\"color\"]\n", - ")\n", - "data.asfreq(\"D\", method=\"ffill\").plot(\n", - " ax=ax, style=\"--o\", lw=1, color=next(colour_wheel)[\"color\"]\n", - ")\n", - "ax.set_ylabel(\"Close ($)\")\n", - "ax.legend([\"original\", \"back-fill\", \"forward-fill\"]);" + "# Upsample to daily frequency\n", + "daily = df.upsample(time_column=\"date\", every=\"1d\")\n", + "daily.head()" ] }, { @@ -944,15 +880,7 @@ "source": [ "### Rolling Window Functions\n", "\n", - "The `rolling()` and `ewm()` methods are both rolling window functions. The first includes functions of the sequence\n", - "\n", - "$$\n", - "y_t = f(\\{x_{t-i} \\}_{i=0}^{i=R-1})\n", - "$$\n", - "\n", - "where $R$ is the number of periods to use for the rolling window. For example, if the function is the mean, then $f$ takes the form $\\frac{1}{R}\\displaystyle\\sum_{i=0}^{i=R-1} x_{t-i}$.\n", - "\n", - "The example below is a 2-period rolling mean:" + "Polars supports rolling window operations on expressions, including `.rolling_mean()`, `.rolling_std()`, and `.ewm_mean()`:" ] }, { @@ -962,87 +890,40 @@ "metadata": {}, "outputs": [], "source": [ - "df.rolling(2).mean()" + "# 2-period rolling mean\n", + "df.with_columns(\n", + " pl.col(\"Vacancies (ICT), thousands\").rolling_mean(window_size=2).alias(\"rolling_2\")\n", + ")" ] }, { "cell_type": "markdown", - "id": "488f3c7c", + "id": "ewm_markdown_cell", "metadata": {}, "source": [ - "The `ewm()` includes the class of functions where data point $x_{t-i}$ has a weight $w_i = (1-\\alpha)^i$. As $0 < \\alpha < 1$, points further back in time are given less weight. For example, an exponentially moving average is given by\n", - "\n", - "$$\n", - "y_t = \\frac{x_t + (1 - \\alpha)x_{t-1} + (1 - \\alpha)^2 x_{t-2} + ...\n", - "+ (1 - \\alpha)^t x_{0}}{1 + (1 - \\alpha) + (1 - \\alpha)^2 + ...\n", - "+ (1 - \\alpha)^t}\n", - "$$\n", + "### Exponentially Weighted and Expanding Aggregations\n", "\n", - "The example below shows the code for the exponentially weighted moving average:" + "Polars also supports exponentially weighted moving averages via `.ewm_mean()`, as well as expanding aggregations:" ] }, { "cell_type": "code", "execution_count": null, - "id": "0ea9c8ce", + "id": "ewm_code_cell", "metadata": {}, "outputs": [], "source": [ - "df.ewm(alpha=0.2).mean()" - ] - }, - { - "cell_type": "markdown", - "id": "8d9eb44c", - "metadata": {}, - "source": [ - "Let's see these methods together on the stock market data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "0af7b5e6", - "metadata": {}, - "outputs": [], - "source": [ - "fig, ax = plt.subplots()\n", - "roll_num = 28\n", - "alpha = 0.03\n", - "xf[\"Close\"].plot(label=\"Raw\", alpha=0.5)\n", - "xf[\"Close\"].expanding().mean().plot(label=\"Expanding Average\", style=\":\")\n", - "xf[\"Close\"].ewm(alpha=alpha).mean().plot(\n", - " label=f\"EWMA ($\\\\alpha=${alpha:.2f})\", style=\"--\"\n", + "# Exponentially weighted moving average\n", + "df_ewm = df.with_columns(\n", + " [\n", + " pl.col(\"Vacancies (ICT), thousands\").ewm_mean(alpha=0.2).alias(\"ewm_0.2\"),\n", + " (\n", + " pl.col(\"Vacancies (ICT), thousands\").cum_sum()\n", + " / pl.int_range(1, pl.len() + 1)\n", + " ).alias(\"expanding_mean\"),\n", + " ]\n", ")\n", - "xf[\"Close\"].rolling(roll_num).mean().plot(label=f\"{roll_num}D MA\", style=\"-.\")\n", - "ax.legend()\n", - "ax.set_ylabel(\"Close ($)\");" - ] - }, - { - "cell_type": "markdown", - "id": "6f795baa", - "metadata": {}, - "source": [ - "For more tools to analyse stocks, see the [**Pandas TA**](https://twopirllc.github.io/pandas-ta/) package.\n", - "\n", - "We can also use `rolling()` as an intermediate step in creating more than one type of aggregation:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "134199ae", - "metadata": {}, - "outputs": [], - "source": [ - "roll = xf[\"Close\"].rolling(50, center=True)\n", - "\n", - "fig, ax = plt.subplots()\n", - "m = roll.agg([\"mean\", \"std\"])\n", - "m[\"mean\"].plot(ax=ax)\n", - "ax.fill_between(m.index, m[\"mean\"] - m[\"std\"], m[\"mean\"] + m[\"std\"], alpha=0.2)\n", - "ax.set_ylabel(\"Close ($)\");" + "df_ewm.head()" ] }, { @@ -1052,7 +933,7 @@ "source": [ "### Shifting\n", "\n", - "Shifting can move series around in time; it's what we need to create leads and lags of time series. Let's create a lead and a lag in the data. Remember that a lead is going to shift the pattern in the data to the left (ie earlier in time), while the lag is going to shift patterns later in time (ie to the right)." + "Shifting can move series around in time; it's what we need to create leads and lags of time series. Let's create a lead and a lag in the data. Remember that a lead is going to shift the pattern in the data to the left (i.e. earlier in time), while the lag is going to shift patterns later in time (i.e. to the right)." ] }, { @@ -1064,10 +945,14 @@ "source": [ "lead = 12\n", "lag = 3\n", - "orig_series_name = df.columns[0]\n", - "df[f\"lead ({lead} months)\"] = df[orig_series_name].shift(-lead)\n", - "df[f\"lag ({lag} months)\"] = df[orig_series_name].shift(lag)\n", - "df.head()" + "orig_series = \"Vacancies (ICT), thousands\"\n", + "df_shifted = df.with_columns(\n", + " [\n", + " pl.col(orig_series).shift(-lead).alias(f\"lead ({lead} months)\"),\n", + " pl.col(orig_series).shift(lag).alias(f\"lag ({lag} months)\"),\n", + " ]\n", + ")\n", + "df_shifted.head()" ] }, { @@ -1077,7 +962,10 @@ "metadata": {}, "outputs": [], "source": [ - "df.iloc[100:300, :].plot();" + "# Plot the shifted series\n", + "df_shifted_pd = df_shifted.to_pandas()\n", + "df_shifted_pd.set_index(\"date\").iloc[100:300].plot()\n", + "plt.show()" ] } ], diff --git a/exploratory-data-analysis.ipynb b/exploratory-data-analysis.ipynb index 071a96b..5841632 100644 --- a/exploratory-data-analysis.ipynb +++ b/exploratory-data-analysis.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Exploratory Data Analysis {#sec-exploratory-data-analysis}\n", @@ -21,7 +21,7 @@ "\n", "### Prerequisites\n", "\n", - "For doing EDA, we'll use the **pandas**, **skimpy**, and **pandas-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", + "For doing EDA, we'll use the **polars**, **skimpy**, and **polars-profiling** packages. We'll also need **lets-plot** for data visualisation. All of these can be installed via `uv add `.\n", "\n", "As ever, we begin by loading these packages that we'll use:" ] @@ -29,14 +29,13 @@ { "cell_type": "code", "execution_count": null, - "id": "a3377aa6", + "id": "1", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "from lets_plot import *\n", "from lets_plot.mapping import as_discrete\n", - "from pandas.api.types import CategoricalDtype\n", "from skimpy import skim\n", "\n", "LetsPlot.setup_html()" @@ -45,7 +44,7 @@ { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "2", "metadata": { "tags": [ "remove-cell" @@ -56,14 +55,13 @@ "import matplotlib.pyplot as plt\n", "import matplotlib_inline.backend_inline\n", "\n", - "# Plot settings\n", "plt.style.use(\"https://github.com/aeturrell/python4DS/raw/main/plot_style.txt\")\n", "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")" ] }, { "cell_type": "markdown", - "id": "e4ddb863", + "id": "3", "metadata": {}, "source": [ "## Questions\n", @@ -90,7 +88,7 @@ }, { "cell_type": "markdown", - "id": "6adeff41", + "id": "4", "metadata": {}, "source": [ "## Variation\n", @@ -103,27 +101,30 @@ { "cell_type": "code", "execution_count": null, - "id": "069caa7c", + "id": "5", "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", + "diamonds = pl.read_csv(\n", " \"https://github.com/mwaskom/seaborn-data/raw/master/diamonds.csv\"\n", ")\n", - "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n", - " CategoricalDtype(\n", - " categories=[\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"], ordered=True\n", - " )\n", - ")\n", - "diamonds[\"color\"] = diamonds[\"color\"].astype(\n", - " CategoricalDtype(categories=[\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"], ordered=True)\n", + "\n", + "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", + "diamonds_color_order = [\"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\"]\n", + "\n", + "diamonds = diamonds.with_columns(\n", + " [\n", + " pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)),\n", + " pl.col(\"color\").cast(pl.Enum(diamonds_color_order)),\n", + " ]\n", ")\n", + "\n", "diamonds.head()" ] }, { "cell_type": "markdown", - "id": "01f5979c", + "id": "6", "metadata": {}, "source": [ "Since `\"carat\"` is a numerical variable, we can use a histogram:" @@ -132,7 +133,7 @@ { "cell_type": "code", "execution_count": null, - "id": "97900f58", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -141,7 +142,7 @@ }, { "cell_type": "markdown", - "id": "2307ba7c", + "id": "8", "metadata": {}, "source": [ "Now that you can visualize variation, what should you look for in your plots?\n", @@ -173,18 +174,18 @@ { "cell_type": "code", "execution_count": null, - "id": "20d75550", + "id": "9", "metadata": {}, "outputs": [], "source": [ - "smaller_diamonds = diamonds.query(\"carat < 3\").copy()\n", + "smaller_diamonds = diamonds.filter(pl.col(\"carat\") < 3)\n", "\n", "(ggplot(smaller_diamonds, aes(x=\"carat\")) + geom_histogram(binwidth=0.01))" ] }, { "cell_type": "markdown", - "id": "ba20a0e2", + "id": "10", "metadata": {}, "source": [ "This histogram suggests several interesting questions:\n", @@ -211,7 +212,7 @@ }, { "cell_type": "markdown", - "id": "0626d35a", + "id": "11", "metadata": {}, "source": [ "### Unusual values\n", @@ -226,7 +227,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d9d7e995", + "id": "12", "metadata": {}, "outputs": [], "source": [ @@ -235,7 +236,7 @@ }, { "cell_type": "markdown", - "id": "05bcf733", + "id": "13", "metadata": {}, "source": [ "There are so many observations in the common bins that the rare bins are very short, making it very difficult to see them (although maybe if you stare intently at 0 you'll spot something).\n", @@ -245,7 +246,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ea8f8bf3", + "id": "14", "metadata": {}, "outputs": [], "source": [ @@ -258,7 +259,7 @@ }, { "cell_type": "markdown", - "id": "ba2b2c79", + "id": "15", "metadata": {}, "source": [ "`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis.\n", @@ -270,17 +271,19 @@ { "cell_type": "code", "execution_count": null, - "id": "e81ffb55", + "id": "16", "metadata": {}, "outputs": [], "source": [ - "unusual = diamonds.query(\"y < 3 or y > 20\").loc[:, [\"x\", \"y\", \"z\", \"price\"]]\n", + "unusual = diamonds.filter((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)).select(\n", + " [\"x\", \"y\", \"z\", \"price\"]\n", + ")\n", "unusual" ] }, { "cell_type": "markdown", - "id": "c9321f36", + "id": "17", "metadata": {}, "source": [ "The `\"y\"` variable measures one of the three dimensions of these diamonds, in mm.\n", @@ -297,7 +300,7 @@ }, { "cell_type": "markdown", - "id": "142d21d7", + "id": "18", "metadata": {}, "source": [ "### Exercises\n", @@ -321,7 +324,7 @@ }, { "cell_type": "markdown", - "id": "c51e8d6f", + "id": "19", "metadata": {}, "source": [ "## Unusual Values\n", @@ -345,18 +348,16 @@ { "cell_type": "code", "execution_count": null, - "id": "ecf345a7", + "id": "20", "metadata": {}, "outputs": [], "source": [ - "diamonds2 = diamonds.copy()\n", - "condition = (diamonds2[\"y\"] < 3) | (diamonds2[\"y\"] > 20)\n", - "diamonds2.loc[condition, \"y\"] = pd.NA" + "diamonds2 = diamonds.filter(~((pl.col(\"y\") < 3) | (pl.col(\"y\") > 20)))" ] }, { "cell_type": "markdown", - "id": "d26d922e", + "id": "21", "metadata": {}, "source": [ "It's not obvious where you should plot missing values, so **lets-plot** doesn't include them in the plot:" @@ -365,7 +366,7 @@ { "cell_type": "code", "execution_count": null, - "id": "15a43255", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -374,7 +375,7 @@ }, { "cell_type": "markdown", - "id": "e8ae854b", + "id": "23", "metadata": {}, "source": [ "Other times you want to understand what makes observations with missing values different to observations with recorded values.\n", @@ -386,29 +387,40 @@ { "cell_type": "code", "execution_count": null, - "id": "0a4ea922", + "id": "24", "metadata": {}, "outputs": [], "source": [ + "import pandas as pd\n", + "import polars as pl\n", + "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights = pd.read_csv(url)\n", + "flights_pd = pd.read_csv(url)\n", + "flights = pl.from_pandas(flights_pd)\n", "flights.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "6849f4d9", + "id": "25", "metadata": {}, "outputs": [], "source": [ - "flights2 = flights.assign(\n", - " cancelled=lambda x: pd.isna(x[\"dep_time\"]),\n", - " sched_hour=lambda x: x[\"sched_dep_time\"] // 100,\n", - " sched_min=lambda x: x[\"sched_dep_time\"] % 100,\n", - " sched_dep_time=lambda x: x[\"sched_hour\"] + x[\"sched_min\"] / 60,\n", + "from lets_plot import *\n", + "\n", + "flights2 = flights.with_columns(\n", + " [\n", + " pl.col(\"dep_time\").is_null().alias(\"cancelled\"),\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"sched_hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"sched_min\"),\n", + " (\n", + " (pl.col(\"sched_dep_time\") // 100) + (pl.col(\"sched_dep_time\") % 100) / 60\n", + " ).alias(\"sched_dep_time\"),\n", + " ]\n", ")\n", "\n", + "\n", "(\n", " ggplot(flights2, aes(x=\"sched_dep_time\"))\n", " + geom_freqpoly(aes(color=\"cancelled\"), binwidth=1 / 4)\n", @@ -417,7 +429,7 @@ }, { "cell_type": "markdown", - "id": "b97e453a", + "id": "26", "metadata": {}, "source": [ "However this plot isn't great because there are many more non-cancelled flights than cancelled flights.\n", @@ -436,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "f63d0b9f", + "id": "27", "metadata": {}, "source": [ "## Covariation\n", @@ -453,7 +465,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e1719d8f", + "id": "28", "metadata": {}, "outputs": [], "source": [ @@ -465,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "20387b96", + "id": "29", "metadata": {}, "source": [ "The default appearance of `geom_freqpoly()` is not that useful here because the height, determined by the overall count, differs so much across cuts, making it hard to see the differences in the shapes of their distributions.\n", @@ -477,7 +489,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9388e24b", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -489,7 +501,7 @@ }, { "cell_type": "markdown", - "id": "157f63dd", + "id": "31", "metadata": {}, "source": [ "There’s something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that’s because density plots are a little hard to interpret - there’s a lot going on in this plot.\n", @@ -500,7 +512,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a3f333a6", + "id": "32", "metadata": {}, "outputs": [], "source": [ @@ -509,7 +521,7 @@ }, { "cell_type": "markdown", - "id": "b9d9ef00", + "id": "33", "metadata": {}, "source": [ "We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counter-intuitive finding that better quality diamonds are typically cheaper! In the exercises, you’ll be challenged to figure out why.\n", @@ -522,21 +534,20 @@ { "cell_type": "code", "execution_count": null, - "id": "6949db81", + "id": "34", "metadata": {}, "outputs": [], "source": [ - "mpg = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n", - ")\n", - "mpg[\"class\"] = mpg[\"class\"].astype(\"category\")\n", + "mpg = pl.read_csv(\"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\")\n", + "\n", + "mpg = mpg.with_columns(pl.col(\"class\").cast(pl.Categorical))\n", "\n", "(ggplot(mpg, aes(x=\"class\", y=\"hwy\")) + geom_boxplot())" ] }, { "cell_type": "markdown", - "id": "871aaf1c", + "id": "35", "metadata": {}, "source": [ "To make the trend easier to see, we can reorder class based on the median value of `\"hwy\"`:" @@ -545,7 +556,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a5b1ed09", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -554,7 +565,7 @@ }, { "cell_type": "markdown", - "id": "dde59236", + "id": "37", "metadata": {}, "source": [ "If you have long variable names, geom_boxplot() will work better if you flip it 90°. You can do that by adding `coord_flip()`." @@ -563,7 +574,7 @@ { "cell_type": "code", "execution_count": null, - "id": "920a4268", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -576,7 +587,7 @@ }, { "cell_type": "markdown", - "id": "299635df", + "id": "39", "metadata": {}, "source": [ "#### Exercises\n", @@ -598,7 +609,7 @@ }, { "cell_type": "markdown", - "id": "6e6a0b46", + "id": "40", "metadata": {}, "source": [ "### Two categorical variables\n", @@ -609,20 +620,24 @@ { "cell_type": "code", "execution_count": null, - "id": "68d330d2", + "id": "41", "metadata": {}, "outputs": [], "source": [ - "ct_cut_color = pd.melt(\n", - " pd.crosstab(diamonds[\"cut\"], diamonds[\"color\"]).reset_index(),\n", - " id_vars=[\"cut\"],\n", - " value_vars=diamonds[\"color\"].unique(),\n", + "ct_cut_color = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(index=\"cut\", columns=\"color\", values=\"count\", aggregate_function=\"sum\")\n", + ")\n", + "\n", + "ct_cut_color_long = ct_cut_color.melt(\n", + " id_vars=\"cut\", variable_name=\"color\", value_name=\"value\"\n", ")" ] }, { "cell_type": "markdown", - "id": "c2ebc0c6", + "id": "42", "metadata": {}, "source": [ "Followed by visualising it with `geom_tile()`:" @@ -631,16 +646,16 @@ { "cell_type": "code", "execution_count": null, - "id": "e858cd22", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "(ggplot(ct_cut_color, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" + "(ggplot(ct_cut_color_long, aes(x=\"color\", y=\"cut\")) + geom_tile(aes(fill=\"value\")))" ] }, { "cell_type": "markdown", - "id": "8ccd5c1b", + "id": "44", "metadata": {}, "source": [ "### Exercises\n", @@ -654,7 +669,7 @@ }, { "cell_type": "markdown", - "id": "b2f19afc", + "id": "45", "metadata": {}, "source": [ "### Two numerical variables\n", @@ -668,7 +683,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2afe2535", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -677,7 +692,7 @@ }, { "cell_type": "markdown", - "id": "9d8034f2", + "id": "47", "metadata": {}, "source": [ "(In this section we'll use the `smaller_diamonds` dataset to stay focused on the bulk of the diamonds that are smaller than 3 carats)\n", @@ -689,7 +704,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b55707a9", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -698,7 +713,7 @@ }, { "cell_type": "markdown", - "id": "351c22bd", + "id": "49", "metadata": {}, "source": [ "But using transparency can be challenging for very large datasets. In that case, we recommend a *binscatter*, or binned scatterplot. A binned scatterplot divides the conditioning variable, `\"carat\"` in our example, into equally sized bins or quantiles, and then plots the conditional mean of the dependent variable, `\"price\"` in our example, within each bin. Bin scatters often come with confidence intervals too. A good bin scatter package in Python is [**binsreg**](https://nppackages.github.io/binsreg/). However, bin scatters are an advanced topic, and we won't cover them here." @@ -706,7 +721,7 @@ }, { "cell_type": "markdown", - "id": "14158b48", + "id": "50", "metadata": {}, "source": [ "## **pandas** built-in tools for EDA\n", @@ -721,7 +736,7 @@ { "cell_type": "code", "execution_count": null, - "id": "13079065", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -730,7 +745,7 @@ }, { "cell_type": "markdown", - "id": "db57dbb7", + "id": "52", "metadata": {}, "source": [ "Although helpful, that sure is hard to read! We can improve this by using the `round()` method too:\n" @@ -739,17 +754,18 @@ { "cell_type": "code", "execution_count": null, - "id": "b4144440", + "id": "53", "metadata": {}, "outputs": [], "source": [ - "sum_table = diamonds.describe().round(1)\n", + "sum_table = diamonds.describe().with_columns(pl.selectors.numeric().round(1))\n", + "\n", "sum_table" ] }, { "cell_type": "markdown", - "id": "8063b8f5", + "id": "54", "metadata": {}, "source": [ "Published summary statistics tables often list one variable per row, and if your data frame has many variables, `describe()` can quickly get too wide to read easily. You can transpose it using the `T` property (or the `transpose()` method):" @@ -758,17 +774,19 @@ { "cell_type": "code", "execution_count": null, - "id": "cd2f8772", + "id": "55", "metadata": {}, "outputs": [], "source": [ - "sum_table = sum_table.T\n", - "sum_table" + "sum_table_long = sum_table.melt(\n", + " id_vars=\"statistic\", variable_name=\"variable\", value_name=\"value\"\n", + ")\n", + "sum_table_long" ] }, { "cell_type": "markdown", - "id": "e67104d3", + "id": "56", "metadata": {}, "source": [ "Of course, the stats provided in this pre-built table are not very customised. So what do we do to get the table that we actually want? Well, the answer is to draw on the contents of the previous data chapters, particularly the introduction to data analysis. Groupbys, merges, aggregations: use all of them to produce the EDA table that you want.\n", @@ -781,24 +799,25 @@ { "cell_type": "code", "execution_count": null, - "id": "5afcacbc", + "id": "57", "metadata": {}, "outputs": [], "source": [ - "(\n", - " diamonds.groupby([\"cut\", \"color\"])[\"price\"]\n", - " .mean()\n", - " .unstack()\n", - " .apply(lambda x: x / 1e3)\n", - " .fillna(\"-\")\n", - " .style.format(precision=2)\n", - " .set_caption(\"Sale price (thousands)\")\n", - ")" + "import polars.selectors as cs\n", + "\n", + "price_summary = (\n", + " diamonds.group_by([\"cut\", \"color\"])\n", + " .agg(pl.col(\"price\").mean())\n", + " .pivot(index=\"cut\", on=\"color\", values=\"price\")\n", + " .with_columns(cs.numeric().round(2))\n", + ")\n", + "\n", + "price_summary" ] }, { "cell_type": "markdown", - "id": "d16e0fc1", + "id": "58", "metadata": {}, "source": [ "Although a neater one than we've seen, this is still a drab table of numbers. The eye is not immediately drawn to it!\n", @@ -811,16 +830,26 @@ { "cell_type": "code", "execution_count": null, - "id": "21e65189", + "id": "59", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.background_gradient(cmap=\"plasma\")" + "ct_cut_color = (\n", + " diamonds.group_by([\"color\", \"cut\"])\n", + " .agg(pl.len().alias(\"count\"))\n", + " .pivot(\n", + " index=\"color\",\n", + " on=\"cut\",\n", + " values=\"count\",\n", + " aggregate_function=\"sum\",\n", + " )\n", + " .with_columns(cs.numeric().fill_null(0))\n", + ")" ] }, { "cell_type": "markdown", - "id": "b7c6aa71", + "id": "60", "metadata": {}, "source": [ "By default, `background_gradient()` highlights each number relative to the others in its column; you can highlight by row using `axis=1` or relative to all table values using `axis=0`. And of course `plasma` is just one of [many available colormaps](https://matplotlib.org/stable/tutorials/colors/colormaps.html)!\n", @@ -837,38 +866,35 @@ { "cell_type": "code", "execution_count": null, - "id": "bb0162ba", + "id": "61", "metadata": {}, "outputs": [], "source": [ - "(\n", - " pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"])\n", - " .style.format(precision=0)\n", - " .bar(color=\"#d65f5f\")\n", - ")" + "(ct_cut_color.to_pandas().style.format(precision=0).bar(color=\"#d65f5f\"))" ] }, { "cell_type": "markdown", - "id": "e36d4795", + "id": "62", "metadata": {}, "source": [ - "Use `.hightlight_max()`, and similar commands, to show important entries:" + "Use `max_counts()`, and similar commands, to show important entries:" ] }, { "cell_type": "code", "execution_count": null, - "id": "5d19072c", + "id": "63", "metadata": {}, "outputs": [], "source": [ - "pd.crosstab(diamonds[\"color\"], diamonds[\"cut\"]).style.highlight_max().format(\"{:.0f}\")" + "max_counts = ct_cut_color.select(pl.exclude(\"color\").max())\n", + "max_counts" ] }, { "cell_type": "markdown", - "id": "63075da4", + "id": "64", "metadata": {}, "source": [ "You can find a full set of styling commands [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Styling)." @@ -876,10 +902,10 @@ }, { "cell_type": "markdown", - "id": "d8b6d225", + "id": "65", "metadata": {}, "source": [ - "### Exploratory Plotting with **pandas**\n", + "### Exploratory Plotting with **polars**\n", "\n", "**pandas** has some built-in plotting options to help you look at data quickly. These can be accessed via `.plot.*` or `.plot()`, depending on the context. Let's make a quick `.plot()` using a dataset on taxis." ] @@ -887,48 +913,50 @@ { "cell_type": "code", "execution_count": null, - "id": "b479d5b1", + "id": "66", "metadata": {}, "outputs": [], "source": [ - "taxis = pd.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", - "# turn the pickup time column into a datetime\n", - "taxis[\"pickup\"] = pd.to_datetime(taxis[\"pickup\"])\n", - "# set some other columns types\n", - "taxis = taxis.astype(\n", - " {\n", - " \"dropoff\": \"datetime64[ns]\",\n", - " \"pickup\": \"datetime64[ns]\",\n", - " \"color\": \"category\",\n", - " \"payment\": \"category\",\n", - " \"pickup_zone\": \"string\",\n", - " \"dropoff_zone\": \"string\",\n", - " \"pickup_borough\": \"category\",\n", - " \"dropoff_borough\": \"category\",\n", - " }\n", + "taxis = pl.read_csv(\"https://github.com/mwaskom/seaborn-data/raw/master/taxis.csv\")\n", + "\n", + "taxis = taxis.with_columns(\n", + " pl.col(\"pickup\").str.strptime(pl.Datetime, format=\"%Y-%m-%d %H:%M:%S\")\n", + ")\n", + "\n", + "taxis = taxis.with_columns(\n", + " [\n", + " pl.col(\"color\").cast(pl.Categorical),\n", + " pl.col(\"payment\").cast(pl.Categorical),\n", + " pl.col(\"pickup_borough\").cast(pl.Categorical),\n", + " pl.col(\"dropoff_borough\").cast(pl.Categorical),\n", + " ]\n", ")\n", + "\n", "taxis.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "ee971c9c", + "id": "67", "metadata": {}, "outputs": [], "source": [ - "taxis.info()" + "taxis.describe()" ] }, { "cell_type": "code", "execution_count": null, - "id": "2015b1dc", + "id": "68", "metadata": {}, "outputs": [], "source": [ + "taxis_pd = taxis.to_pandas()\n", + "taxis_pd[\"pickup\"] = pd.to_datetime(taxis_pd[\"pickup\"])\n", + "\n", "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[\"total\"]\n", " .mean()\n", " .plot(\n", @@ -936,12 +964,13 @@ " xlabel=\"\",\n", " ylabel=\"Fare (USD)\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "b9c0f990", + "id": "69", "metadata": {}, "source": [ "Again, if you can get the data in the right shape, you can plot it. The same function works with multiple lines\n" @@ -950,12 +979,12 @@ { "cell_type": "code", "execution_count": null, - "id": "51e86185", + "id": "70", "metadata": {}, "outputs": [], "source": [ "(\n", - " taxis.set_index(\"pickup\")\n", + " taxis_pd.set_index(\"pickup\")\n", " .groupby(pd.Grouper(freq=\"D\"))[[\"fare\", \"tip\", \"tolls\"]]\n", " .mean()\n", " .plot(\n", @@ -964,12 +993,13 @@ " xlabel=\"\",\n", " ylabel=\"USD\",\n", " )\n", - ");" + ")\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "cc227b81", + "id": "71", "metadata": {}, "source": [ "Now let's see some of the other quick `.plot.*` options." @@ -977,7 +1007,7 @@ }, { "cell_type": "markdown", - "id": "1c4278ac", + "id": "72", "metadata": {}, "source": [ "A bar chart (use `barh` for horizontal orientation; `rot` sets rotation of labels):" @@ -986,16 +1016,31 @@ { "cell_type": "code", "execution_count": null, - "id": "79ceca92", + "id": "73", "metadata": {}, "outputs": [], "source": [ - "taxis.value_counts(\"payment\").sort_index().plot.bar(title=\"Counts\", rot=0);" + "payment_counts = (\n", + " taxis.filter(pl.col(\"payment\").is_not_null())\n", + " .group_by(\"payment\")\n", + " .agg(pl.len().alias(\"count\"))\n", + " .sort(\"payment\")\n", + ")\n", + "\n", + "payment_counts.to_pandas().plot.bar(\n", + " x=\"payment\",\n", + " y=\"count\",\n", + " legend=False,\n", + " rot=0,\n", + " title=\"Counts\",\n", + ")\n", + "\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "4e6f1cc2", + "id": "74", "metadata": {}, "source": [ "This next one, uses `.plot.hist()` to create a histogram." @@ -1004,16 +1049,17 @@ { "cell_type": "code", "execution_count": null, - "id": "5efc5817", + "id": "75", "metadata": {}, "outputs": [], "source": [ - "taxis[\"tip\"].plot.hist(bins=30, title=\"Tip\");" + "(taxis_pd[\"tip\"].plot.hist(bins=30, title=\"Tip\"))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "3f15bfa7", + "id": "76", "metadata": {}, "source": [ "Boxplot:" @@ -1022,16 +1068,17 @@ { "cell_type": "code", "execution_count": null, - "id": "0b735d15", + "id": "77", "metadata": {}, "outputs": [], "source": [ - "(taxis[[\"fare\", \"tolls\", \"tip\"]].plot.box());" + "(taxis_pd[[\"fare\", \"tolls\", \"tip\"]].plot.box())\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "67017bef", + "id": "78", "metadata": {}, "source": [ "Scatter plot:" @@ -1040,16 +1087,17 @@ { "cell_type": "code", "execution_count": null, - "id": "66adada2", + "id": "79", "metadata": {}, "outputs": [], "source": [ - "taxis.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None));" + "(taxis_pd.plot.scatter(x=\"fare\", y=\"tip\", alpha=0.7, ylim=(0, None)))\n", + "plt.show()" ] }, { "cell_type": "markdown", - "id": "9766d2c4", + "id": "80", "metadata": {}, "source": [ "## Other tools for EDA\n", @@ -1059,7 +1107,7 @@ }, { "cell_type": "markdown", - "id": "813ec70a", + "id": "81", "metadata": {}, "source": [ "### **skimpy** for summary statistics\n", @@ -1072,7 +1120,7 @@ { "cell_type": "code", "execution_count": null, - "id": "32796b5f", + "id": "82", "metadata": {}, "outputs": [], "source": [ @@ -1081,7 +1129,7 @@ }, { "cell_type": "markdown", - "id": "f2810c9e", + "id": "83", "metadata": {}, "source": [ "## Summary\n", @@ -1095,9 +1143,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -1105,7 +1150,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, diff --git a/joins.ipynb b/joins.ipynb index d494d84..e6cdeb0 100644 --- a/joins.ipynb +++ b/joins.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Joins {#sec-joins}\n", @@ -11,13 +11,13 @@ "\n", "It's rare that a data analysis involves only a single data frame. Typically you have many data frames, and you must *join* them together to answer the questions that you're interested in.\n", "\n", - "**pandas** has a really rich set of options for combining one or more data frames, with the two most important being concatenate and merge. Some of the examples in this chapter show you how to join a pair of data frames. Fortunately this is enough, since you can combine three data frames by combining two pairs." + "**polars** has a really rich set of options for combining one or more data frames, with the two most important being concatenate and joins. Some of the examples in this chapter show you how to join a pair of data frames. Fortunately this is enough, since you can combine three data frames by combining two pairs." ] }, { "cell_type": "code", "execution_count": null, - "id": "51a55374", + "id": "1", "metadata": { "tags": [ "remove-cell" @@ -36,40 +36,39 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package. For reading Stata (.dta) files, we'll use pandas as a bridge (pd.read_stata()) and then convert to polars with pl.from_pandas()." ] }, { "cell_type": "markdown", - "id": "80710c6e", + "id": "3", "metadata": {}, "source": [ "## Concatenate\n", "\n", - "If you have two or more data frames with the same index or the same columns, you can glue them together into a single data frame using `pd.concat()`.\n", + "If you have two or more data frames with the same columns, you can glue them together into a single data frame using pl.concat().\n", "\n", "![](https://pandas.pydata.org/docs/_images/08_concat_row.svg)\n", "\n", - "For the same columns, pass `axis=0` to glue the index together; for the same index, pass `axis=1` to glue the columns together. The concatenate function will typically be used on a list of data frames.\n", + "For the same columns, use pl.concat() to stack rows vertically. Polars doesn't have a row index like pandas; `pl.concat(how=\"horizontal\")` joins DataFrames by position, aligning rows in order.\n", "\n", - "If you want to track where the original data came from in the final data frame, use the `keys` keyword.\n", - "\n", - "Here's an example using data on two different states' populations that also makes uses of the `keys` option:" + "Here's an example using data on two different states' populations:" ] }, { "cell_type": "code", "execution_count": null, - "id": "f5ef4f37", + "id": "4", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", + "import polars as pl\n", "\n", "base_url = (\n", " \"https://github.com/aeturrell/coding-for-economists/raw/refs/heads/main/data/\"\n", @@ -77,32 +76,64 @@ "state_codes = [\"ca\", \"il\"]\n", "end_url = \"pop.dta\"\n", "\n", - "# This grabs the two dataframes, one for each state\n", - "list_of_state_dfs = [pd.read_stata(base_url + state + end_url) for state in state_codes]\n", - "# Show example of first entry in list of dataframes\n", + "# Read the Stata files and convert to polars\n", + "list_of_state_dfs = []\n", + "for state in state_codes:\n", + " temp_df = pd.read_stata(base_url + state + end_url)\n", + " list_of_state_dfs.append(pl.from_pandas(temp_df))\n", + "\n", + "# shows example of the 2 dataframes\n", "print(list_of_state_dfs[0])\n", + "print(list_of_state_dfs[1])\n", "\n", - "# Concatenate the list of dataframes\n", - "df = pd.concat(list_of_state_dfs, keys=state_codes, axis=0)\n", + "# concatenate the list of dataframes\n", + "df = pl.concat(list_of_state_dfs)\n", "df" ] }, { "cell_type": "markdown", - "id": "e0c343b0", + "id": "5", + "metadata": {}, + "source": [ + "Note that when concatenating, you can track the origin of rows by adding a column before concatenation:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", "metadata": {}, + "outputs": [], "source": [ - "Note that the `keys` argument is optional, but is useful for keeping track of origin data frames within the merged data frame.\n", + "# Add a state identifier column to each dataframe\n", + "list_with_state = []\n", + "for state, df_state in zip(state_codes, list_of_state_dfs):\n", + " df_state = df_state.with_columns(pl.lit(state).alias(\"state\"))\n", + " list_with_state.append(df_state)\n", "\n", + "# Concatenate with state identifiers\n", + "df_with_state = pl.concat(list_with_state)\n", + "df_with_state" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ "::: {.callout-tip title=\"Exercise\"}\n", "Concatenate the follow two data frames:\n", "\n", "```python\n", - "df1 = pd.DataFrame([['a', 1], ['b', 2]],\n", - " columns=['letter', 'number'])\n", + "df1 = pl.DataFrame({\n", + " 'letter': ['a', 'b'],\n", + " 'number': [1, 2]\n", + "})\n", "\n", - "df2 = pd.DataFrame([['c', 3], ['d', 4]],\n", - " columns=['letter', 'number'])\n", + "df2 = pl.DataFrame({\n", + " 'letter': ['c', 'd'],\n", + " 'number': [3, 4]\n", "```\n", "\n", ":::" @@ -110,20 +141,28 @@ }, { "cell_type": "markdown", - "id": "f6e91777", + "id": "8", "metadata": {}, "source": [ - "## Merge\n", + "## Joins\n", "\n", - "There are so many options for merging data frames using `pd.merge(left, right, on=..., how=...` that we won't be able to cover them all here. The most important features are: the two data frames to be merged, what variables (aka keys) to merge on (and these can be indexes) via `on=`, and *how* to do the merge (eg left, right, outer, inner) via `how=`. This diagram shows an example of a merge using keys from the left-hand data frame:\n", + "There are many options for joining data frames using `pl.DataFrame.join()`. The most important features are: the two data frames to be joined, what variables (aka keys) to join on via `on=`, and *how* to do the join (eg left, right, outer, inner) via `how=`. This diagram shows an example of a left join:\n", "\n", "![](https://pandas.pydata.org/docs/_images/08_merge_left.svg)\n", "\n", "The `how=` keyword works in the following ways:\n", - "- `how='left'` uses keys from the left data frame only to merge.\n", - "- `how='right'` uses keys from the right data frame only to merge.\n", - "- `how='inner'` uses keys that appear in both data frames to merge.\n", - "- `how='outer'` uses the cartesian product of keys in both data frames to merge on.\n", + "\n", + "- how=\"left\" uses keys from the left data frame only to join.\n", + "\n", + "- how=\"right\" uses keys from the right data frame only to join.\n", + "\n", + "- how=\"inner\" uses keys that appear in both data frames to join.\n", + "\n", + "- how=\"full\" uses all keys from both data frames to join on (full outter join).\n", + "\n", + "- how=\"semi\" returns rows from the left where keys appear in the right.\n", + "\n", + "- how=\"anti\" returns rows from the left where keys do NOT appear in the right.\n", "\n", "Let's see examples of some of these:" ] @@ -131,11 +170,11 @@ { "cell_type": "code", "execution_count": null, - "id": "53c66d5d", + "id": "9", "metadata": {}, "outputs": [], "source": [ - "left = pd.DataFrame(\n", + "left = pl.DataFrame(\n", " {\n", " \"key1\": [\"K0\", \"K0\", \"K1\", \"K2\"],\n", " \"key2\": [\"K0\", \"K1\", \"K0\", \"K1\"],\n", @@ -143,97 +182,225 @@ " \"B\": [\"B0\", \"B1\", \"B2\", \"B3\"],\n", " }\n", ")\n", - "right = pd.DataFrame(\n", + "right = pl.DataFrame(\n", " {\n", " \"key1\": [\"K0\", \"K1\", \"K1\", \"K2\"],\n", " \"key2\": [\"K0\", \"K0\", \"K0\", \"K0\"],\n", " \"C\": [\"C0\", \"C1\", \"C2\", \"C3\"],\n", " \"D\": [\"D0\", \"D1\", \"D2\", \"D3\"],\n", " }\n", - ")\n", - "# Right merge\n", - "pd.merge(left, right, on=[\"key1\", \"key2\"], how=\"right\")" + ")" ] }, { "cell_type": "markdown", - "id": "8af0241e", + "id": "10", "metadata": {}, "source": [ - "Note that the key combination of K2 and K0 did not exist in the left-hand data frame, and so its entries in the final data frame are NaNs. But it *does* have entries because we chose the keys from the right-hand data frame.\n", + "Right join:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11", + "metadata": {}, + "outputs": [], + "source": [ + "left.join(right, on=[\"key1\", \"key2\"], how=\"right\")" + ] + }, + { + "cell_type": "markdown", + "id": "12", + "metadata": {}, + "source": [ + "Note that the key combination of K2 and K0 did not exist in the left-hand data frame, and so its entries in the final data frame are null values. But it does have entries because we chose the keys from the right-hand data frame.\n", "\n", - "What about an inner merge?" + "What about an inner join?" ] }, { "cell_type": "code", "execution_count": null, - "id": "5e73608f", + "id": "13", "metadata": {}, "outputs": [], "source": [ - "pd.merge(left, right, on=[\"key1\", \"key2\"], how=\"inner\")" + "left.join(right, on=[\"key1\", \"key2\"], how=\"inner\")" ] }, { "cell_type": "markdown", - "id": "67ac74c0", + "id": "14", "metadata": {}, "source": [ "Now we see that the combination K2 and K0 are excluded because they didn't exist in the overlap of keys in both data frames.\n", "\n", - "Finally, let's take a look at an outer merge that comes with some extra info via the `indicator` keyword:" + "Finally, let's take a look at an outer join:" ] }, { "cell_type": "code", "execution_count": null, - "id": "5d209fb9", + "id": "15", "metadata": {}, "outputs": [], "source": [ - "pd.merge(left, right, on=[\"key1\", \"key2\"], how=\"outer\", indicator=True)" + "left.join(right, on=[\"key1\", \"key2\"], how=\"full\")" ] }, { "cell_type": "markdown", - "id": "67ca901b", + "id": "16", "metadata": {}, "source": [ - "Now we can see that the products of all key combinations are here. The `indicator=True` option has caused an extra column to be added, called '_merge', that tells us which data frame the keys on that row came from.\n", + "This performs a full outer join, keeping all rows from both data frames.\n", "\n", + "Polars doesn't include a built-in indicator parameter to track which side contributed each row in a join. On a full join, polars keeps the join keys from both sides with suffixes. However, you can achieve the same effect by comparing the two DataFrames after the join:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "17", + "metadata": {}, + "outputs": [], + "source": [ + "# Perform an outer join\n", + "joined = left.join(right, on=[\"key1\", \"key2\"], how=\"full\")\n", + "\n", + "# Add indicator column manually\n", + "# Rows with nulls in B only came from right\n", + "# Rows with nulls in D only came from left\n", + "# Rows with no nulls came from both\n", + "joined = joined.with_columns(\n", + " pl.when(pl.col(\"B\").is_null() & pl.col(\"D\").is_not_null())\n", + " .then(pl.lit(\"right_only\"))\n", + " .when(pl.col(\"B\").is_not_null() & pl.col(\"D\").is_null())\n", + " .then(pl.lit(\"left_only\"))\n", + " .otherwise(pl.lit(\"both\"))\n", + " .alias(\"_merge\")\n", + ")\n", + "joined" + ] + }, + { + "cell_type": "markdown", + "id": "18", + "metadata": {}, + "source": [ "::: {.callout-tip title=\"Exercise\"}\n", "\n", - "Merge the following two data frames using the `left_on` and `right_on` keyword arguments to specify a join on `lkey` and `rkey` respectively:\n", + "Join the following two data frames using the `left_on` and `right_on` keyword arguments to specify a join on lkey and rkey respectively:\n", "\n", "```python\n", - "df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],\n", - " 'value': [1, 2, 3, 5]})\n", - "df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],\n", - " 'value': [5, 6, 7, 8]})\n", + "df1 = pl.DataFrame({\n", + " 'lkey': ['foo', 'bar', 'baz', 'foo'],\n", + " 'value': [1, 2, 3, 5]\n", + "})\n", + "df2 = pl.DataFrame({\n", + " 'rkey': ['foo', 'bar', 'baz', 'foo'],\n", + " 'value': [5, 6, 7, 8]\n", + "})\n", "```\n", ":::\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", "\n", - "Merge the following two data frames on `\"a\"` using `how=\"left\"` as a keyword argument:\n", + "Join the following two data frames on `\"a\"` using `how=\"left\"`:\n", "\n", "```python\n", - "df1 = pd.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]})\n", - "df2 = pd.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]})\n", + "df1 = pl.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]})\n", + "df2 = pl.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]})\n", "```\n", "\n", - "What do you notice about the position `.loc[1, \"c\"]` in the merged data frame? \n", - ":::\n", + "What do you notice about the resulting row for `\"baz\"`?\n", + ":::\n" + ] + }, + { + "cell_type": "markdown", + "id": "19", + "metadata": {}, + "source": [ + "## Semi and Anti Joins\n", + "\n", + "Polars also supports semi and anti joins, which are useful for filtering:\n", + "\n", + "Semi join: Keep rows from the left where keys appear in the right:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "20", + "metadata": {}, + "outputs": [], + "source": [ + "# Semi join - only keeps left rows that have matches in right\n", + "left.join(right, on=[\"key1\", \"key2\"], how=\"semi\")" + ] + }, + { + "cell_type": "markdown", + "id": "21", + "metadata": {}, + "source": [ + "Anti join: Keep rows from the left where keys do NOT appear in the right:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "22", + "metadata": {}, + "outputs": [], + "source": [ + "# Anti join - only keeps left rows without matches in right\n", + "left.join(right, on=[\"key1\", \"key2\"], how=\"anti\")" + ] + }, + { + "cell_type": "markdown", + "id": "23", + "metadata": {}, + "source": [ + "## Handling Duplicate Column Names\n", "\n", - "For more on the options for merging, see **pandas**' comprehensive [merging documentation](https://pandas.pydata.org/docs/user_guide/merging.html#database-style-dataframe-or-named-series-joining-merging)." + "When joining, if both DataFrames have columns with the same name (other than the join keys), polars will add suffixes to differentiate them. By default it adds `_right`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "24", + "metadata": {}, + "outputs": [], + "source": [ + "# Create DataFrames with overlapping column names\n", + "left2 = pl.DataFrame(\n", + " {\"key\": [\"K0\", \"K1\", \"K2\"], \"value\": [1, 2, 3], \"shared\": [\"L0\", \"L1\", \"L2\"]}\n", + ")\n", + "\n", + "right2 = pl.DataFrame(\n", + " {\"key\": [\"K0\", \"K1\", \"K2\"], \"value\": [4, 5, 6], \"shared\": [\"R0\", \"R1\", \"R2\"]}\n", + ")\n", + "\n", + "# Join with default suffix\n", + "left2.join(right2, on=\"key\", suffix=\"_right\")" + ] + }, + { + "cell_type": "markdown", + "id": "25", + "metadata": {}, + "source": [ + "For more information about the available join types and options, see **polars'** [joining documentation](https://docs.pola.rs/user-guide/transformations/joins/)." ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -241,7 +408,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "base", "language": "python", "name": "python3" }, diff --git a/missing-values.ipynb b/missing-values.ipynb index 67e95ef..27d4777 100644 --- a/missing-values.ipynb +++ b/missing-values.ipynb @@ -9,7 +9,7 @@ "\n", "## Introduction\n", "\n", - "In this chapter, we'll look at the tools and tricks for dealing with missing values. We'll start by discussing some general tools for working with missing values recorded as `NA`s. We'll then explore the idea of implicitly missing values, values are that are simply absent from your data, and show some tools you can use to make them explicit.\n", + "In this chapter, we'll look at the tools and tricks for dealing with missing values. We'll start by discussing some general tools for working with missing values recorded as `null` values. We'll then explore the idea of implicitly missing values, values are that are simply absent from your data, and show some tools you can use to make them explicit.\n", "We'll finish off with a related discussion of empty groups, caused by categories that don't appear in the data." ] }, @@ -40,7 +40,7 @@ "source": [ "### Prerequisites\n", "\n", - "This chapter will use the **pandas** data analysis package." + "This chapter will use the **polars** data analysis package." ] }, { @@ -50,13 +50,11 @@ "source": [ "## Explicit Missing Values\n", "\n", - "To begin, let's explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see an `NA` or `nan`.\n", + "To begin, let's explore a few handy tools for creating or eliminating missing explicit values, i.e. cells where you see a `null` value.\n", "\n", "### Types of Missing Values\n", "\n", - "One thing to note about missing values in **pandas** is that they are not all created alike!\n", - "\n", - "For example, real numbers in **pandas** (such as the `float64` dtype) uses a 'nan' (aka, Not a Number):" + "In polars, missing values are represented by `null`:" ] }, { @@ -67,9 +65,9 @@ "outputs": [], "source": [ "import numpy as np\n", - "import pandas as pd\n", + "import polars as pl\n", "\n", - "df = pd.DataFrame([5, 27.3, np.nan, -16], columns=[\"numbers\"])\n", + "df = pl.DataFrame({\"numbers\": [5.0, 27.3, None, -16.0]})\n", "df" ] }, @@ -78,7 +76,7 @@ "id": "4b1c9ea7", "metadata": {}, "source": [ - "But this isn't the only way! We can also use Python's built-in `None` value (which, here, gets converted to a NaN because the valid values are all floating point numbers) and **pandas**' `pd.NA`:" + "Polars also handles Python's built-in None values seamlessly:" ] }, { @@ -88,7 +86,7 @@ "metadata": {}, "outputs": [], "source": [ - "numbers = pd.DataFrame([pd.NA, 27.3, np.nan, -16, None], columns=[\"numbers\"])\n", + "numbers = pl.DataFrame({\"numbers\": [None, 27.3, np.nan, -16.0, None]})\n", "numbers" ] }, @@ -97,7 +95,7 @@ "id": "b2dda456", "metadata": {}, "source": [ - "However, with the object data type (the default for strings), the types can co-exist:" + "For string columns, missing values are also represented by `null`:" ] }, { @@ -107,9 +105,7 @@ "metadata": {}, "outputs": [], "source": [ - "fruits = pd.DataFrame(\n", - " [\"orange\", np.nan, \"apple\", None, \"banana\", pd.NA], columns=[\"fruit\"]\n", - ")\n", + "fruits = pl.DataFrame({\"fruit\": [\"orange\", None, \"apple\", None, \"banana\", None]})\n", "fruits" ] }, @@ -118,7 +114,12 @@ "id": "8b5a2849", "metadata": {}, "source": [ - "Both of these types of missing value can be found using **pandas** `.isna()` function. This returns a new column of boolean values that are `True` if the value is any kind of missing value." + "In **polars**, it is important to distinguish between `null` (missing data) and `NaN` (floating-point Not-a-Number, such as `np.nan`):\n", + "\n", + "- `.is_null()` checks for `null` values (returning `True` for `None` / `null`).\n", + "- `.is_nan()` checks for `NaN` values (returning `True` for `np.nan`).\n", + "\n", + "Let's see how `.is_null()` and `.is_nan()` behave on a column containing both types of missing values:" ] }, { @@ -128,7 +129,13 @@ "metadata": {}, "outputs": [], "source": [ - "fruits.isna()" + "numbers.select(\n", + " [\n", + " pl.col(\"numbers\"),\n", + " pl.col(\"numbers\").is_null().alias(\"is_null\"),\n", + " pl.col(\"numbers\").is_nan().alias(\"is_nan\"),\n", + " ]\n", + ")" ] }, { @@ -136,7 +143,17 @@ "id": "0fd6764d", "metadata": {}, "source": [ - "As a convenience, there is also a `notna()` function." + "For string columns, missing values are also represented by `null`. As a convenience, there is also an `.is_not_null()` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3fb8082", + "metadata": {}, + "outputs": [], + "source": [ + "fruits.select(pl.col(\"fruit\").is_not_null())" ] }, { @@ -146,7 +163,7 @@ "source": [ "### Dealing with Explicit Missing Values\n", "\n", - "There are various options for dealing with missing values. The `fillna()` function achieves this. Let's take a look at it with some test data:" + "There are various options for dealing with missing values. The `fill_null()` method achieves this. Let's take a look at it with some test data:" ] }, { @@ -156,17 +173,15 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df = pd.DataFrame(\n", - " [\n", - " [np.nan, 2, None, 0],\n", - " [3, 4, np.nan, 1],\n", - " [5, np.nan, np.nan, pd.NA],\n", - " [np.nan, 3, np.nan, 4],\n", - " ],\n", - " columns=list(\"ABCD\"),\n", + "null_df = pl.DataFrame(\n", + " {\n", + " \"A\": [None, 3, 5, None],\n", + " \"B\": [2, 4, None, 3],\n", + " \"C\": [None, None, None, None],\n", + " \"D\": [0, 1, None, 4],\n", + " }\n", ")\n", - "\n", - "nan_df" + "null_df" ] }, { @@ -184,7 +199,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(0)" + "null_df.fill_null(0)" ] }, { @@ -192,7 +207,7 @@ "id": "af8e2cb7", "metadata": {}, "source": [ - "This can be done on a by-column basis; here we replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1, 2, and 3 respectively." + "This can be done on a by-column basis by passing a dictionary:" ] }, { @@ -202,7 +217,14 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(value={\"A\": 0, \"B\": 1, \"C\": 2, \"D\": 3})" + "null_df.with_columns(\n", + " [\n", + " pl.col(\"A\").fill_null(0),\n", + " pl.col(\"B\").fill_null(1),\n", + " pl.col(\"C\").fill_null(2),\n", + " pl.col(\"D\").fill_null(3),\n", + " ]\n", + ")" ] }, { @@ -210,7 +232,7 @@ "id": "2544fe5a", "metadata": {}, "source": [ - "We can also propagate non-null values forward or backward (relative to the index)" + "We can also propagate non-null values forward or backward:" ] }, { @@ -220,7 +242,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(method=\"ffill\")" + "null_df.fill_null(strategy=\"forward\")" ] }, { @@ -230,7 +252,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(method=\"bfill\")" + "null_df.fill_null(strategy=\"backward\")" ] }, { @@ -246,7 +268,7 @@ "id": "ca7e4ca9", "metadata": {}, "source": [ - "Another feature of all of these functions is that you can limit the number of NaNs that get replaced using the `limit=` keyword argument." + "Another feature of all of these functions is that you can limit the number of nulls that get replaced using the `limit=` keyword argument." ] }, { @@ -256,7 +278,8 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.fillna(value={\"A\": 0, \"B\": 1, \"C\": 2, \"D\": 3}, limit=1)" + "# Forward fill with limit of 1\n", + "null_df.fill_null(strategy=\"forward\", limit=1)" ] }, { @@ -264,7 +287,7 @@ "id": "99b94a01", "metadata": {}, "source": [ - "Of course, another option might be just to filter out the missing values altogether. There are a couple of ways to do this depending if you want to remove entire rows (`axis=0`) or columns (`axis=1`—but in this case, as there is at least one NaN in each column though, there will be no data left!)" + "Of course, another option might be just to filter out the missing values altogether. The `.drop_nulls()` method removes rows with any missing values:" ] }, { @@ -274,7 +297,15 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df[\"A\"].dropna(axis=0) # on a single column" + "null_df.drop_nulls()" + ] + }, + { + "cell_type": "markdown", + "id": "710bcb2c", + "metadata": {}, + "source": [ + "You can also drop rows where all values are null using `pl.all_horizontal()` in a filter:" ] }, { @@ -284,25 +315,34 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df.dropna(axis=1)" + "null_exp_df = pl.DataFrame(\n", + " {\n", + " \"A\": [None, 3, 5, None, None],\n", + " \"B\": [2, 4, None, 3, None],\n", + " \"C\": [None, None, None, None, None],\n", + " \"D\": [0, 1, None, 4, None],\n", + " }\n", + ")\n", + "\n", + "null_exp_df" ] }, { - "cell_type": "markdown", - "id": "9bc57652", + "cell_type": "code", + "execution_count": null, + "id": "f7b38107", "metadata": {}, + "outputs": [], "source": [ - "`dropna()` takes some keyword arguments too; for example `how=\"all\"` only drops a column or row if *all* of the values are NA." + "null_exp_df.filter(~pl.all_horizontal(pl.all().is_null()))" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "3296ea35", + "cell_type": "markdown", + "id": "2b058955", "metadata": {}, - "outputs": [], "source": [ - "nan_df.dropna(how=\"all\")" + "Notice that the last row (where all values were `null`) was removed, while column `C` (where all values are `null`) remains because this operation filters rows, not columns." ] }, { @@ -318,7 +358,7 @@ "id": "095f53a5", "metadata": {}, "source": [ - "Another way to filter out nans is to use the same filtering methods you would use normally, via boolean columns, in combination with the `.notna()` function. In the below example, we see all columns for the rows for which A is not NA." + "Another way to filter out nulls is to use the same filtering methods you would use normally, via boolean columns, in combination with the `.is_not_null()` method. In the below example, we see the rows for which column A is not null:" ] }, { @@ -328,7 +368,7 @@ "metadata": {}, "outputs": [], "source": [ - "nan_df[nan_df[\"A\"].notna()]" + "null_df.filter(pl.col(\"A\").is_not_null())" ] }, { @@ -336,11 +376,11 @@ "id": "c1588f63", "metadata": {}, "source": [ - "### Adding NA values\n", + "### Adding Null values\n", "\n", "Sometimes you'll hit the opposite problem where some concrete value actually represents a missing value. This typically arises in data generated by older software that doesn't have a proper way to represent missing values, so it must instead use some special value like 99 or -999.\n", "\n", - "If possible, handle this when reading in the data, for example, by using the `na_values=` keyword argument when calling `pd.read_csv()`. If you discover the problem later, or your data source doesn't provide a way to handle it on reading the file, you can use a range of options to replace the given data:" + "If possible, handle this when reading in the data, for example, by using the `null_values=` keyword argument when calling `pl.read_csv()`. If you discover the problem later, or your data source doesn't provide a way to handle it on reading the file, you can use a range of options to replace the given data:" ] }, { @@ -350,8 +390,13 @@ "metadata": {}, "outputs": [], "source": [ - "stata_df = pd.DataFrame([[3, 4, 5], [-7, 4, -99], [-99, 6, 5]], columns=list(\"ABC\"))\n", - "\n", + "stata_df = pl.DataFrame(\n", + " {\n", + " \"A\": [3, -7, -99],\n", + " \"B\": [4, 4, 6],\n", + " \"C\": [5, -99, 5],\n", + " }\n", + ")\n", "stata_df" ] }, @@ -360,7 +405,7 @@ "id": "81685807", "metadata": {}, "source": [ - "The easiest option is probably `.replace()`:" + "The easiest option is probably the expression-level `replace()` method:" ] }, { @@ -370,7 +415,7 @@ "metadata": {}, "outputs": [], "source": [ - "stata_df.replace({-99: pd.NA})" + "stata_df.with_columns(pl.all().replace(-99, None))" ] }, { @@ -378,7 +423,7 @@ "id": "9d4d3d3d", "metadata": {}, "source": [ - "Because `.replace()` accepts a dictionary, it's possible to replace several values at once:" + "Because `replace()` accepts a dictionary, it's possible to replace several values at once:" ] }, { @@ -388,7 +433,7 @@ "metadata": {}, "outputs": [], "source": [ - "stata_df.replace({-99: pd.NA, -7: pd.NA})" + "stata_df.with_columns(pl.all().replace({-99: None, -7: None}))" ] }, { @@ -406,7 +451,7 @@ "source": [ "## Implicit Missing Values\n", "\n", - "So far we've talked about missing values that are **explicitly** missing, i.e. you can see an `NA` or similar in your data.\n", + "So far we've talked about missing values that are **explicitly** missing, i.e. you can see a `null` or similar in your data.\n", "But missing values can also be **implicitly** missing, if an entire row of data is simply absent from the data.\n", "Let's illustrate the difference with a simple data set that records the price of some stock each quarter:" ] @@ -418,11 +463,11 @@ "metadata": {}, "outputs": [], "source": [ - "stocks = pd.DataFrame(\n", + "stocks = pl.DataFrame(\n", " {\n", " \"year\": [2020, 2020, 2020, 2020, 2021, 2021, 2021],\n", " \"qtr\": [1, 2, 3, 4, 2, 3, 4],\n", - " \"price\": [1.88, 0.59, 0.35, np.nan, 0.92, 0.17, 2.66],\n", + " \"price\": [1.88, 0.59, 0.35, None, 0.92, 0.17, 2.66],\n", " }\n", ")\n", "stocks" @@ -435,7 +480,7 @@ "source": [ "This dataset has two missing observations:\n", "\n", - "- The `price` in the fourth quarter of 2020 is explicitly missing, because its value is `NA`.\n", + "- The `price` in the fourth quarter of 2020 is explicitly missing, because its value is `null`.\n", "\n", "- The `price` for the first quarter of 2021 is implicitly missing, because it simply does not appear in the dataset.\n", "\n", @@ -467,7 +512,7 @@ "metadata": {}, "outputs": [], "source": [ - "stocks.pivot(columns=\"qtr\", values=\"price\", index=\"year\")" + "stocks.pivot(index=\"year\", on=\"qtr\", values=\"price\", aggregate_function=\"first\")" ] }, { @@ -497,14 +542,16 @@ "metadata": {}, "outputs": [], "source": [ - "health = pd.DataFrame(\n", + "health = pl.DataFrame(\n", " {\n", " \"name\": [\"Ikaia\", \"Oletta\", \"Leriah\", \"Dashay\", \"Tresaun\"],\n", " \"smoker\": [\"no\", \"no\", \"previously\", \"no\", \"yes\"],\n", " \"age\": [34, 88, 75, 47, 56],\n", " }\n", ")\n", - "health[\"smoker\"] = health[\"smoker\"].astype(\"category\")" + "health = health.with_columns(\n", + " pl.col(\"smoker\").cast(pl.Enum([\"no\", \"previously\", \"yes\"]))\n", + ")" ] }, { @@ -522,7 +569,7 @@ "metadata": {}, "outputs": [], "source": [ - "health_cut = health.iloc[:-1, :]\n", + "health_cut = health[:-1]\n", "health_cut" ] }, @@ -531,7 +578,7 @@ "id": "7ae95b5f", "metadata": {}, "source": [ - "The value 'yes' for smoker now doesn't (seem to) appear anywhere in our data frame. But if we run `value_counts()` to get a count of the number of each type of category, you'll see that the data frame 'remembers' that there's a 'yes' category that isn't currently present:" + "The value 'yes' for smoker now doesn't (seem to) appear anywhere in our data frame. Because we used an `Enum` type, the column still remembers that 'yes' is a valid category. We can inspect the categories of the column's datatype:" ] }, { @@ -541,7 +588,7 @@ "metadata": {}, "outputs": [], "source": [ - "health_cut[\"smoker\"].value_counts()" + "health_cut[\"smoker\"].dtype.categories" ] }, { @@ -549,7 +596,7 @@ "id": "bde48398", "metadata": {}, "source": [ - "You'll see the same thing happen with a `groupby()` operation:" + "If we perform aggregation operations, Polars will only return results for groups present in the data by default. If we need to include all categories (even those with no observations), we can join the categories list back to the aggregated result:" ] }, { @@ -559,7 +606,13 @@ "metadata": {}, "outputs": [], "source": [ - "health_cut.groupby(\"smoker\")[\"age\"].mean()" + "all_smokers = pl.DataFrame({\"smoker\": health_cut[\"smoker\"].dtype.categories})\n", + "means = (\n", + " health_cut.group_by(\"smoker\")\n", + " .agg(pl.col(\"age\").mean())\n", + " .with_columns(pl.col(\"smoker\").cast(pl.Utf8))\n", + ")\n", + "all_smokers.join(means, on=\"smoker\", how=\"left\")" ] }, { @@ -567,14 +620,11 @@ "id": "5a2af19e", "metadata": {}, "source": [ - "You can see here that, because we took the mean of a number that doesn't exist, we got a NaN in place of a real value for the yes row (but there is a 'yes' row)." + "You can see here that, because we took the mean of a number that doesn't exist, we got a null in place of a real value for the yes row (but there is a 'yes' row)." ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", diff --git a/numbers.ipynb b/numbers.ipynb index 0b96fbb..bef5b6c 100644 --- a/numbers.ipynb +++ b/numbers.ipynb @@ -13,7 +13,7 @@ "\n", "### Prerequisites\n", "\n", - "This chapter mostly uses functions from **pandas**, which you are likely to already have installed but you can install using `uv add pandas` in the terminal. We'll use real examples from nycflights13, as well as toy examples made with fake data.\n", + "This chapter mostly uses functions from **polars**, which you are likely to already have installed but you can install using `uv add polars` in the terminal. We'll use real examples from nycflights13, as well as toy examples made with fake data.\n", "\n", "Let's first load up the NYC flights data\n" ] @@ -44,10 +44,11 @@ "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", "url = \"https://raw.githubusercontent.com/byuidatascience/data4python4ds/master/data-raw/flights/flights.csv\"\n", - "flights = pd.read_csv(url)" + "\n", + "flights = pl.read_csv(url, use_pyarrow=True)" ] }, { @@ -57,7 +58,7 @@ "source": [ "### Counts\n", "\n", - "It's surprising how much data science you can do with just counts and a little basic arithmetic, so **pandas** strives to make counting as easy as possible with `.count()` and `.value_counts()`. The former just provides a straight count of all the non NA items:" + "It's surprising how much data science you can do with just counts and a little basic arithmetic, so **polars** strives to make counting as easy as possible with `.count()` and `.value_counts()`. The former provides a straight count of all the non-null items:" ] }, { @@ -85,7 +86,7 @@ "metadata": {}, "outputs": [], "source": [ - "flights[\"dest\"].value_counts()" + "flights[\"dest\"].value_counts(sort=True)" ] }, { @@ -93,7 +94,7 @@ "id": "cdb110f3", "metadata": {}, "source": [ - "This is automatically sorted in order of the most common category. You can perform the same computation \"by hand\" with `group_by()`, `agg()` and then using the count function. This is useful because it allows you to compute other summaries at the same time:" + "By default, `value_counts()` returns counts for each unique value. Passing `sort=True` sorts the output in order of the most common category. You can perform the same computation \"by hand\" with `group_by()`, `agg()`, and `sort()`:" ] }, { @@ -104,12 +105,14 @@ "outputs": [], "source": [ "(\n", - " flights.groupby([\"dest\"])\n", + " flights.group_by(\"dest\")\n", " .agg(\n", - " mean_delay=(\"dep_delay\", \"mean\"),\n", - " count_flights=(\"dest\", \"count\"),\n", + " [\n", + " pl.col(\"dep_delay\").mean().alias(\"mean_delay\"),\n", + " pl.col(\"dest\").count().alias(\"count_flights\"),\n", + " ]\n", " )\n", - " .sort_values(by=\"count_flights\", ascending=False)\n", + " .sort(\"count_flights\", descending=True)\n", ")" ] }, @@ -128,7 +131,7 @@ "metadata": {}, "outputs": [], "source": [ - "(flights.groupby(\"tailnum\").agg(miles=(\"distance\", \"sum\")))" + "flights.group_by(\"tailnum\").agg(pl.col(\"distance\").sum().alias(\"miles\"))" ] }, { @@ -136,7 +139,7 @@ "id": "1b489765", "metadata": {}, "source": [ - "You can count missing values by combining `sum()` and `isnull()`. In the flights dataset this represents flights that are cancelled. Note that because there isn't a simple string name for applying `.isnull()` followed by `.sum()` (unlike just running `sum()`, which would be given by the string \"sum\"), we need to use a lambda function in the below:" + "You can count missing values by combining sum() and is_null(). In the flights dataset this represents flights that are cancelled:" ] }, { @@ -146,7 +149,7 @@ "metadata": {}, "outputs": [], "source": [ - "(flights.groupby(\"dest\").agg(n_cancelled=(\"dep_time\", lambda x: x.isnull().sum())))" + "flights.group_by(\"dest\").agg(pl.col(\"dep_time\").is_null().sum().alias(\"n_cancelled\"))" ] }, { @@ -160,11 +163,11 @@ "\n", "Basic number arithmatic is achieved by `+` (addition), `-` (subtraction), `*` (multiplication), `/` (division), `**` (powers), `%` (modulo), and `@` (tensor product). Most of these functions don't need a huge amount of explanation because you'll be familiar with them already (and you can look up the others when you do need them).\n", "\n", - "When you have two numeric columns of equal length and you add or subtract them, it's pretty obvious what's going to happen. But we do need to talk about what happens when there is a variable involved that is *not* as long as the column. This is important for operations like `flights.assign(air_time = air_time / 60)` because there are 336,776 numbers on the left of `/` but only one on the right. In this case, **pandas** will understand that you'd like to divide *all* values of air time by 60. This is sometimes called 'broadcasting'. Below is a digram that tries to explain what's going on:\n", + "When you have two numeric columns of equal length and you add or subtract them, it's pretty obvious what's going to happen. But we do need to talk about what happens when there is a variable involved that is *not* as long as the column. This is important for operations like `flights.with_columns(air_time = pl.col(\"air_time\") / 60)` because there are 336,776 numbers on the left of `/` but only one on the right. In this case, **polars** will understand that you'd like to divide *all* values of air time by 60. This is sometimes called 'broadcasting'. Below is a digram that tries to explain what's going on:\n", "\n", "![](https://numpy.org/doc/stable/_images/broadcasting_1.png)\n", "\n", - "You can find out much more about [broadcasting on the **numpy** documentation](https://numpy.org/doc/stable/user/basics.broadcasting.html). **pandas** is built on top of **numpy** and inherits some of its functionality. \n" + "Broadcasting is a concept popularized by **numpy**. While **polars** is written in Rust and built on Apache Arrow, it adopts vectorised broadcasting behaviour for scalar operations.\n" ] }, { @@ -172,7 +175,7 @@ "id": "adb10890", "metadata": {}, "source": [ - "When operating on two columns, **pandas** compares their shapes element-wise. Two columns are compatible when they are equal, or one of them is a scalar. If these conditions are not met, you will get an error.\n", + "When operating on two columns, **polars** compares their shapes element-wise. Two columns are compatible when they are equal, or one of them is a scalar. If these conditions are not met, you will get an error.\n", "\n" ] }, @@ -201,7 +204,7 @@ "id": "dd24faa5", "metadata": {}, "source": [ - "Sometimes, you'd like to look at the maximum or minimum value across rows *or* columns. As often is the case with **pandas**, you can specify rows or columns to apply functions to by passing `axis=0` (index) or `axis=1` (columns) to that function. The axis designation can be confusing: remember that you are asking which dimension you wish to aggregate over, leaving you with the other dimension. So if we wish to find the minimum in each row, we aggregate / collapse columns, so we need to pass `axis=1`." + "Sometimes, you'd like to look at the maximum or minimum value across rows or columns. To find the minimum in each row, you can use the horizontal functions:" ] }, { @@ -211,7 +214,7 @@ "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame({\"x\": [1, 5, 7], \"y\": [3, 2, pd.NA]})\n", + "df = pl.DataFrame({\"x\": [1, 5, 7], \"y\": [3, 2, None]})\n", "df" ] }, @@ -230,7 +233,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.min(axis=1)" + "df.select(pl.min_horizontal(\"x\", \"y\"))" ] }, { @@ -273,9 +276,11 @@ "metadata": {}, "outputs": [], "source": [ - "flights.assign(\n", - " hour=lambda x: x[\"sched_dep_time\"] // 100,\n", - " minute=lambda x: x[\"sched_dep_time\"] % 100,\n", + "flights.with_columns(\n", + " [\n", + " (pl.col(\"sched_dep_time\") // 100).alias(\"hour\"),\n", + " (pl.col(\"sched_dep_time\") % 100).alias(\"minute\"),\n", + " ]\n", ")" ] }, @@ -300,8 +305,11 @@ "\n", "starting = 100\n", "interest = 1.05\n", - "money = pd.DataFrame(\n", - " {\"year\": 2000 + np.arange(1, 51), \"money\": starting * interest ** np.arange(1, 51)}\n", + "money = pl.DataFrame(\n", + " {\n", + " \"year\": 2000 + np.arange(1, 51),\n", + " \"money\": starting * interest ** np.arange(1, 51),\n", + " }\n", ")\n", "money.head()" ] @@ -321,7 +329,10 @@ "metadata": {}, "outputs": [], "source": [ - "money.plot(x=\"year\", y=\"money\");" + "# Convert to pandas for plotting\n", + "money_pd = money.to_pandas()\n", + "money_pd.plot(x=\"year\", y=\"money\")\n", + "plt.show()" ] }, { @@ -339,7 +350,8 @@ "metadata": {}, "outputs": [], "source": [ - "money.plot(x=\"year\", y=\"money\", logy=True);" + "money_pd.plot(x=\"year\", y=\"money\", logy=True)\n", + "plt.show()" ] }, { @@ -371,7 +383,7 @@ "metadata": {}, "outputs": [], "source": [ - "money.head().round(2)" + "money.select(pl.all().round(2)).head()" ] }, { @@ -379,7 +391,7 @@ "id": "2b26e633", "metadata": {}, "source": [ - "This can also be applied to individual columns or differentially to columns via a dictionary:" + "This can also be applied to individual columns:" ] }, { @@ -389,7 +401,12 @@ "metadata": {}, "outputs": [], "source": [ - "money.head().round({\"year\": 0, \"money\": 1})" + "money.with_columns(\n", + " [\n", + " pl.col(\"year\").round(0),\n", + " pl.col(\"money\").round(1),\n", + " ]\n", + ").head()" ] }, { @@ -407,7 +424,12 @@ "metadata": {}, "outputs": [], "source": [ - "money.tail().round({\"year\": 0, \"money\": -2})" + "money.with_columns(\n", + " [\n", + " pl.col(\"year\").round(),\n", + " ((pl.col(\"money\") / 100).round() * 100).alias(\"money\"),\n", + " ]\n", + ").tail()" ] }, { @@ -415,7 +437,7 @@ "id": "6544b94e", "metadata": {}, "source": [ - "Sometimes, you'll want to round according to significant figures rather than decimal places. There's not a really easy way to do this but you can define a custom function to do it. Here's an example of rounding to 2 significant figures (change the `2` in the below to round to a different number of significant figures):" + "Sometimes, you'll want to round according to significant figures rather than decimal places. There's not a really easy way to do this but you can define a custom function to do it. Here's an example of rounding to 2 significant figures (change the `2` in the code block below to round to a different number of significant figures):" ] }, { @@ -425,7 +447,9 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].head().apply(lambda x: float(f'{float(f\"{x:.2g}\"):g}'))" + "money.with_columns(\n", + " pl.col(\"money\").map_elements(lambda x: float(f'{float(f\"{x:.2g}\"):g}'))\n", + ")" ] }, { @@ -490,7 +514,7 @@ "id": "c22b0eda", "metadata": {}, "source": [ - "Remember that you can always apply **numpy** functions to **pandas** data frame columns like so:" + "**polars** provides native expression methods like `.ceil()` and `.floor()` for rounding operations:" ] }, { @@ -500,7 +524,7 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].head().apply(np.ceil)" + "money.with_columns(pl.col(\"money\").ceil()).head()" ] }, { @@ -510,7 +534,7 @@ "source": [ "### Cumulative and Rolling Aggregates\n", "\n", - "**pandas** has several cumulative functions, including `.cumsum()`, `.cummax()` and `.cummin()`, and `.cumprod()`. " + "**polars** has several cumulative functions, including `.cum_sum()`, `.cum_max()`, `.cum_min()`, and `.cum_prod()`." ] }, { @@ -520,15 +544,7 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].tail().cumsum()" - ] - }, - { - "cell_type": "markdown", - "id": "c4b569cb", - "metadata": {}, - "source": [ - "As ever, this can be applied across rows too by passing `axis=1`." + "money[\"money\"].cum_sum().tail()" ] }, { @@ -542,7 +558,7 @@ "\n", "### Ranking\n", "\n", - "**pandas**' rank function is `.rank()`. Let's look back at the data we made earlier and rank it:" + "**polars**' rank function is `.rank()`. Let's look back at the data we made earlier and rank it:" ] }, { @@ -562,7 +578,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.rank()" + "df.select(pl.all().rank())" ] }, { @@ -570,7 +586,7 @@ "id": "85345760", "metadata": {}, "source": [ - "Of course, there's no change here because the items were already ranked! We can also do a pct rank by passing the keyword argument `pct=True`." + "To compute a percent rank in **polars**, you can call `.rank()` and divide by the total number of rows (`df.height`):" ] }, { @@ -580,7 +596,7 @@ "metadata": {}, "outputs": [], "source": [ - "df.rank(pct=True)" + "df.select(pl.all().rank(method=\"average\") / df.height)" ] }, { @@ -602,9 +618,15 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money_lag_5\"] = money[\"money\"].shift(5)\n", - "money[\"money_lead_10\"] = money[\"money\"].shift(-10)\n", - "money.set_index(\"year\").plot();" + "money_with_shifts = money.with_columns(\n", + " [\n", + " pl.col(\"money\").shift(5).alias(\"money_lag_5\"),\n", + " pl.col(\"money\").shift(-10).alias(\"money_lead_10\"),\n", + " ]\n", + ")\n", + "money_with_shifts_pd = money_with_shifts.to_pandas()\n", + "money_with_shifts_pd.set_index(\"year\").plot()\n", + "plt.show()" ] }, { @@ -636,7 +658,7 @@ "source": [ "## More Useful Summary Statistics\n", "\n", - "We've seen how useful `.mean()`, `.count()`, and `.value_counts()` can be for analysis. **pandas** has a great many more built-in summary statistics functions, however. These include `.median()` (you may find it interesting to compare the mean vs the median when looking at the hourly departure delay in the flights data), `.mode()`, `.min()`, and `.max()`.\n", + "We've seen how useful `.mean()`, `.count()`, and `.value_counts()` can be for analysis. **polars** has a great many more built-in summary statistics functions, however. These include `.median()`, `.mode()`, `.min()`, and `.max()`.\n", "\n", "A class of useful summary statistics are provided by the `.quantile` function, which is the same as `median` for `.quantile(0.5)`. The quantile at x% is the value that x% of values are below. (Note that under this definition, `.quantile(1)` will be the same as `.max()`.) Let's see an example with the 25th percentile." ] @@ -656,7 +678,7 @@ "id": "02c212f4", "metadata": {}, "source": [ - "Sometimes you don't just want one percentile, but a bunch of them. **pandas** makes this very easy by allowing you to pass a list of quantiles:" + "Sometimes you don't just want one percentile, but a bunch of them. While **polars**' `.quantile()` function computes a single quantile level, you can calculate multiple quantiles using a list comprehension to build a DataFrame:" ] }, { @@ -666,7 +688,12 @@ "metadata": {}, "outputs": [], "source": [ - "money[\"money\"].quantile([0, 0.25, 0.5, 0.75])" + "pl.DataFrame(\n", + " {\n", + " \"quantile\": [0.0, 0.25, 0.5, 0.75],\n", + " \"value\": [money[\"money\"].quantile(q) for q in [0.0, 0.25, 0.5, 0.75]],\n", + " }\n", + ")" ] }, { @@ -690,12 +717,16 @@ "outputs": [], "source": [ "(\n", - " flights.groupby([\"origin\", \"dest\"])\n", + " flights.group_by([\"origin\", \"dest\"])\n", " .agg(\n", - " distance_sd=(\"distance\", lambda x: x.quantile(0.75) - x.quantile(0.25)),\n", - " count=(\"distance\", \"count\"),\n", + " [\n", + " (\n", + " pl.col(\"distance\").quantile(0.75) - pl.col(\"distance\").quantile(0.25)\n", + " ).alias(\"distance_sd\"),\n", + " pl.col(\"distance\").count().alias(\"count\"),\n", + " ]\n", " )\n", - " .query(\"distance_sd > 0\")\n", + " .filter(pl.col(\"distance_sd\") > 0)\n", ")" ] }, @@ -718,7 +749,9 @@ "metadata": {}, "outputs": [], "source": [ - "flights[\"dep_delay\"].plot.hist(bins=50, title=\" Distribution: length of delay\");" + "flights_pd = flights.to_pandas()\n", + "flights_pd[\"dep_delay\"].plot.hist(bins=50, title=\"Distribution: length of delay\")\n", + "plt.show()" ] }, { @@ -728,9 +761,10 @@ "metadata": {}, "outputs": [], "source": [ - "flights.query(\"dep_delay <= 120\")[\"dep_delay\"].plot.hist(\n", - " bins=50, title=\" Distribution: length of delay\"\n", - ");" + "flights_pd.query(\"dep_delay <= 120\")[\"dep_delay\"].plot.hist(\n", + " bins=50, title=\"Distribution: length of delay\"\n", + ")\n", + "plt.show()" ] }, { @@ -768,9 +802,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", diff --git a/prerequisites.ipynb b/prerequisites.ipynb index 07119a8..1e38207 100644 --- a/prerequisites.ipynb +++ b/prerequisites.ipynb @@ -44,7 +44,7 @@ "\n", "### Packages\n", "\n", - "A Python **package** is a collection of functions, data, and documentation that extends the capabilities of an installed version of Python. Using packages is key to most data science because most of the functionality we'll need comes from extra packages. You'll see statetments like `import numpy as np` at the start of many Python code scripts—these are instructions to use an installed package (here one called `numpy`) and to give it a shortened name (`np`, for convenience) in the rest of the script. The functions in the `numpy` package are then accessed through syntax like `np.`; for example, you can take logs with `np.log(x)` where `x` is a variable containing a number. You need only install packages once.\n", + "A Python **package** is a collection of functions, data, and documentation that extends the capabilities of an installed version of Python. Using packages is key to most data science because most of the functionality we'll need comes from extra packages. You'll see statetments like `import polars as pl` at the start of many Python code scripts—these are instructions to use an installed package (here one called `polars`) and to give it a shortened name (`pl`, for convenience) in the rest of the script. The functions in the `polars` package are then accessed through syntax like `pl.`; for example, you can read a CSV with `pl.read_csv('file.csv')`. You need only install packages once.\n", "\n", "### Typical workflow\n", "\n", @@ -75,8 +75,6 @@ "\n", "### Installing Python\n", "\n", - "### Installing Python\n", - "\n", "To download and install Python, we'll use the uv \"distribution\" of Python, which is available on all major operating systems. To install it, follow the instructions at [this website](https://docs.astral.sh/uv/getting-started/installation/#installation-methods). Unlike installing normal programmes, we're going to use the *command line* to install Python. Linux, Mac, and Windows all have built-in command lines: search for *Terminal* on Mac or Linux, and *Powershell* on Windows. These apps will bring up boxes that you can type commands in. As of the time of writing, the commands are:\n", "\n", "```bash\n", @@ -257,7 +255,7 @@ "uv add packagename\n", "```\n", "\n", - "and hit return. In the above, `packagename` might be `pandas`, for example. If you have problems installing, make sure that you are connected to the internet, and that [PyPI](https://pypi.org/) (the Python package index) isn't blocked by your firewall or proxy. You could try installing the data analysispackage **polars** this way, by running `uv add polars`. We'll see how to use **polars** in later chapters, but if you want to know if it installed correctly, just look for the message saying \"Successfully installed polars\" followed by the version number.\n", + "and hit return. In the above, `packagename` might be `polars`, for example. If you have problems installing, make sure that you are connected to the internet, and that [PyPI](https://pypi.org/) (the Python package index) isn't blocked by your firewall or proxy. You could try installing the data analysispackage **polars** this way, by running `uv add polars`. We'll see how to use **polars** in later chapters, but if you want to know if it installed correctly, just look for the message saying \"Successfully installed polars\" followed by the version number.\n", "\n", "You can see what packages you have previously installed by entering `uv pip list` into the command line.\n", "\n", diff --git a/quarto.qmd b/quarto.qmd index 7d2561d..233872f 100644 --- a/quarto.qmd +++ b/quarto.qmd @@ -144,11 +144,12 @@ For an example of a code output where the input is not shown, the code below wil ```` ```{{python}} #| echo: false -import pandas as pd -import seaborn as sns +import polars as pl -df = sns.load_dataset("penguins") -pd.crosstab(df["species"], [df["island"], df["sex"]], margins=True) +df = pl.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv").drop_nulls(subset=["species", "island", "sex"]) +ct = df.group_by(["species", "island", "sex"]).agg(pl.len().alias("count")) +ct = ct.with_columns(pl.concat_str(["island", "sex"], separator="_").alias("island_sex")) +ct.pivot(index="species", on="island_sex", values="count", aggregate_function="sum") ``` ```` @@ -186,17 +187,16 @@ For an example of a code output where the input is not shown, the code below wil ```{{python}} #| echo: false from IPython.display import display, Markdown -import pandas as pd -import seaborn as sns +import polars as pl -df = sns.load_dataset("penguins") +df = pl.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv") big_pen = df["body_mass_g"].max() number = len(df) ``` ### The Heaviest Penguin -We find that the heaviest penguin, out of a total of `python f"{number}"` penguins, has a mass of `python f"{big_pen:.2f}"` kilograms. +We find that the heaviest penguin, out of a total of `python f"{number}"` penguins, has a mass of `python f"{big_pen:.2f}"` grams. ```` diff --git a/strings.ipynb b/strings.ipynb index b1a00fe..97b44d2 100644 --- a/strings.ipynb +++ b/strings.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Strings and Text {#sec-strings}\n", @@ -13,12 +13,16 @@ "\n", "This chapter has benefitted from the [Python String Cook Book](https://mkaz.blog/code/python-string-format-cookbook/) and Jake VanderPlas' [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/03.10-working-with-strings.html).\n", "\n", - "Note that there are more powerful methods for working with strings called *regular expressions* but these will be covered in a different chapter." + "Note that there are more powerful methods for working with strings called *regular expressions* but these will be covered in a different chapter.\n", + "\n", + "### Prerequisites\n", + "\n", + "This chapter will use standard Python for scalar string manipulation and **polars** for vectorized string operations on data frames and series." ] }, { "cell_type": "markdown", - "id": "3fc1cd0f", + "id": "1", "metadata": {}, "source": [ "## Creating Strings\n", @@ -29,7 +33,7 @@ { "cell_type": "code", "execution_count": null, - "id": "478a847a", + "id": "2", "metadata": {}, "outputs": [], "source": [ @@ -41,7 +45,7 @@ }, { "cell_type": "markdown", - "id": "9a146322", + "id": "3", "metadata": {}, "source": [ "Strings are of type `str`:" @@ -50,7 +54,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d7f4ea2d", + "id": "4", "metadata": {}, "outputs": [], "source": [ @@ -59,7 +63,7 @@ }, { "cell_type": "markdown", - "id": "2bc26125", + "id": "5", "metadata": {}, "source": [ "Strings in Python can be indexed, so we can get certain characters out by using square brackets to say which positions we would like." @@ -68,7 +72,7 @@ { "cell_type": "code", "execution_count": null, - "id": "01379fe7", + "id": "6", "metadata": {}, "outputs": [], "source": [ @@ -78,7 +82,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d88f7928", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -87,7 +91,7 @@ }, { "cell_type": "markdown", - "id": "d76d0c0a", + "id": "8", "metadata": {}, "source": [ "The usual slicing tricks that apply to lists work for strings too, i.e. the positions you want to get can be retrieved using the `var[start:stop:step]` syntax. Here's an example of getting every other character from the string starting from the 2nd position." @@ -96,7 +100,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e03d95d1", + "id": "9", "metadata": {}, "outputs": [], "source": [ @@ -105,7 +109,7 @@ }, { "cell_type": "markdown", - "id": "ddd1bbe7", + "id": "10", "metadata": {}, "source": [ "Note that strings, like tuples such as `(1, 2, 3)` but unlike lists such as `[1, 2, 3]`, are *immutable*. This means commands like `var[0] = \"B\"` will result in an error. If you want to change a single character, you will have to replace the entire string. In this example, the command to do that would be `var = \"Banana\"`.\n", @@ -116,7 +120,7 @@ { "cell_type": "code", "execution_count": null, - "id": "83ab201b", + "id": "11", "metadata": {}, "outputs": [], "source": [ @@ -125,7 +129,7 @@ }, { "cell_type": "markdown", - "id": "7eeebc05", + "id": "12", "metadata": {}, "source": [ "You can concatenate strings using the `+` operator:" @@ -134,7 +138,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7801bd5d", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -143,7 +147,7 @@ }, { "cell_type": "markdown", - "id": "e8fb0ab8", + "id": "14", "metadata": {}, "source": [ "Note that we added extra characters so that the phrase made sense. Another way of achieving the same end that scales to many words or phrases more efficiently (if you have them in a list) is:" @@ -152,7 +156,7 @@ { "cell_type": "code", "execution_count": null, - "id": "138cef18", + "id": "15", "metadata": {}, "outputs": [], "source": [ @@ -161,7 +165,7 @@ }, { "cell_type": "markdown", - "id": "77b33c02", + "id": "16", "metadata": {}, "source": [ "Three useful functions to know about are `upper()`, `lower()`, and `title()`. Let's see what they do\n" @@ -170,7 +174,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e11896f8", + "id": "17", "metadata": {}, "outputs": [], "source": [ @@ -181,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "505bec9e", + "id": "18", "metadata": {}, "source": [ "Note that there are many built-in functions for using strings in Python, you can find a comprehensive list [here](https://www.w3schools.com/python/python_ref_string.asp).\n", @@ -193,7 +197,7 @@ }, { "cell_type": "markdown", - "id": "3d70a3f6", + "id": "19", "metadata": {}, "source": [ "While we're using `print()`, it has a few tricks. If we have a list, we can print out entries with a given separator:\n" @@ -202,7 +206,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bf0aadec", + "id": "20", "metadata": {}, "outputs": [], "source": [ @@ -211,7 +215,7 @@ }, { "cell_type": "markdown", - "id": "08d9b5af", + "id": "21", "metadata": {}, "source": [ "(We'll find out more about what '\\n' does shortly.) To turn variables of other kinds into strings, use the `str()` function, for example" @@ -220,7 +224,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a96f048c", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -237,7 +241,7 @@ }, { "cell_type": "markdown", - "id": "9c79246f", + "id": "23", "metadata": {}, "source": [ "In this example two boolean variables and one integer variable were converted to strings. `str()` generally makes an intelligent guess at how you'd like to convert your non-string type variable into a string type. You can pass a variable or a literal value to `str()`." @@ -245,7 +249,7 @@ }, { "cell_type": "markdown", - "id": "e68633d9", + "id": "24", "metadata": {}, "source": [ "### f-strings\n", @@ -256,7 +260,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9dddf0da", + "id": "25", "metadata": {}, "outputs": [], "source": [ @@ -266,7 +270,7 @@ }, { "cell_type": "markdown", - "id": "027287af", + "id": "26", "metadata": {}, "source": [ "This is similar to calling `str` on variable and using `+` for concatenation but much shorter to write. You can add expressions to f-strings too:" @@ -275,7 +279,7 @@ { "cell_type": "code", "execution_count": null, - "id": "795e7c07", + "id": "27", "metadata": {}, "outputs": [], "source": [ @@ -284,7 +288,7 @@ }, { "cell_type": "markdown", - "id": "fe120dd7", + "id": "28", "metadata": {}, "source": [ "This also works with functions; after all `**2` is just a function with its own special syntax." @@ -292,7 +296,7 @@ }, { "cell_type": "markdown", - "id": "f3524d86", + "id": "29", "metadata": {}, "source": [ "In this example, the score number that came out had a lot of (probably) uninteresting decimal places. So how do we polish the printed output? You can pass more inforation to the f-string to get the output formatted just the way you want. Let's say we wanted two decimal places and a sign (although you always write `+` in the formatting, the sign comes out as + or - depending on the value):" @@ -301,7 +305,7 @@ { "cell_type": "code", "execution_count": null, - "id": "1f3d3806", + "id": "30", "metadata": {}, "outputs": [], "source": [ @@ -310,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "04eb84a8", + "id": "31", "metadata": {}, "source": [ "There are a whole range of formatting options for numbers as shown in the following table:\n", @@ -336,7 +340,7 @@ }, { "cell_type": "markdown", - "id": "6ae30791", + "id": "32", "metadata": {}, "source": [ "### Special Characters and How to Escape Strings" @@ -344,7 +348,7 @@ }, { "cell_type": "markdown", - "id": "2fb21df1", + "id": "33", "metadata": {}, "source": [ "Python has a string module that comes with some useful built-in strings and characters. For example" @@ -353,7 +357,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0ccd65aa", + "id": "34", "metadata": {}, "outputs": [], "source": [ @@ -364,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "7e9646e6", + "id": "35", "metadata": {}, "source": [ "gives you all of the punctuation," @@ -373,7 +377,7 @@ { "cell_type": "code", "execution_count": null, - "id": "16205c36", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -382,7 +386,7 @@ }, { "cell_type": "markdown", - "id": "1cb1dac5", + "id": "37", "metadata": {}, "source": [ "returns all of the basic letters in the 'ASCII' encoding (with `.ascii_lowercase` and `.ascii_uppercase` variants), and" @@ -391,7 +395,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0c67f5cd", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -400,7 +404,7 @@ }, { "cell_type": "markdown", - "id": "589fff6d", + "id": "39", "metadata": {}, "source": [ "gives you the numbers from 0 to 9. Finally, though less impressive visually, `string.whitespace` gives a string containing all of the different (there is more than one!) types of whitespace." @@ -408,7 +412,7 @@ }, { "cell_type": "markdown", - "id": "794c4f7e", + "id": "40", "metadata": {}, "source": [ "There are other special characters around; in fact, we already met the most famous of them: \"\\n\" for new line. To actually print \"\\n\" we have to 'escape' the backward slash by adding another backward slash:" @@ -417,7 +421,7 @@ { "cell_type": "code", "execution_count": null, - "id": "16e9904a", + "id": "41", "metadata": {}, "outputs": [], "source": [ @@ -427,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "cf878d74", + "id": "42", "metadata": {}, "source": [ "The table below shows the most important escape commands:\n", @@ -444,7 +448,7 @@ }, { "cell_type": "markdown", - "id": "c7dd19ac", + "id": "43", "metadata": {}, "source": [ "Here's a more complicated example:" @@ -453,7 +457,7 @@ { "cell_type": "code", "execution_count": null, - "id": "af423bd1", + "id": "44", "metadata": {}, "outputs": [], "source": [ @@ -462,7 +466,7 @@ }, { "cell_type": "markdown", - "id": "16c97e01", + "id": "45", "metadata": {}, "source": [ "### Raw Strings\n", @@ -473,7 +477,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c2b9c689", + "id": "46", "metadata": {}, "outputs": [], "source": [ @@ -482,7 +486,7 @@ }, { "cell_type": "markdown", - "id": "98d896ed", + "id": "47", "metadata": {}, "source": [ "## Cleaning Text\n", @@ -497,7 +501,7 @@ { "cell_type": "code", "execution_count": null, - "id": "229ada3a", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -506,7 +510,7 @@ }, { "cell_type": "markdown", - "id": "ce0467a1", + "id": "49", "metadata": {}, "source": [ "As with any variable of a specific type (here, string), this would also work with variables:" @@ -515,7 +519,7 @@ { "cell_type": "code", "execution_count": null, - "id": "79f754dc", + "id": "50", "metadata": {}, "outputs": [], "source": [ @@ -527,7 +531,7 @@ }, { "cell_type": "markdown", - "id": "5c5ab4c1", + "id": "51", "metadata": {}, "source": [ "Note that `.replace()` performs an exact replace and so is case-sensitive." @@ -535,7 +539,7 @@ }, { "cell_type": "markdown", - "id": "85bba3f9", + "id": "52", "metadata": {}, "source": [ "### Replacing characters with translate\n", @@ -550,7 +554,7 @@ { "cell_type": "code", "execution_count": null, - "id": "99675fee", + "id": "53", "metadata": {}, "outputs": [], "source": [ @@ -562,7 +566,7 @@ }, { "cell_type": "markdown", - "id": "391d2748", + "id": "54", "metadata": {}, "source": [ "Now we turn our dictionary into a string translator and apply it to our text:\n" @@ -571,7 +575,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e48763cb", + "id": "55", "metadata": {}, "outputs": [], "source": [ @@ -581,7 +585,7 @@ }, { "cell_type": "markdown", - "id": "1c0359ed", + "id": "56", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", @@ -591,7 +595,7 @@ }, { "cell_type": "markdown", - "id": "f27549e8", + "id": "57", "metadata": {}, "source": [ "Generally, `str.translate` is very fast at replacing individual characters in strings. But you can also do it using a list comprehension and a `join()` of the resulting list, like so:" @@ -600,7 +604,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ac758b38", + "id": "58", "metadata": {}, "outputs": [], "source": [ @@ -615,7 +619,7 @@ }, { "cell_type": "markdown", - "id": "cbe854ee", + "id": "59", "metadata": {}, "source": [ "### Splitting strings\n", @@ -626,7 +630,7 @@ { "cell_type": "code", "execution_count": null, - "id": "122619bf", + "id": "60", "metadata": {}, "outputs": [], "source": [ @@ -635,7 +639,7 @@ }, { "cell_type": "markdown", - "id": "0ddc2c15", + "id": "61", "metadata": {}, "source": [ "Next up we can use the built-in `split()` function, which returns a list of places where a given sub-string occurs:\n" @@ -644,7 +648,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9fc432ed", + "id": "62", "metadata": {}, "outputs": [], "source": [ @@ -653,7 +657,7 @@ }, { "cell_type": "markdown", - "id": "992b6a79", + "id": "63", "metadata": {}, "source": [ "Note that the character used to split the string is removed from the resulting list of strings. Let's see an example with a string used for splitting instead of a single character:\n" @@ -662,7 +666,7 @@ { "cell_type": "code", "execution_count": null, - "id": "6904e486", + "id": "64", "metadata": {}, "outputs": [], "source": [ @@ -671,7 +675,7 @@ }, { "cell_type": "markdown", - "id": "5d7beb3c", + "id": "65", "metadata": {}, "source": [ "A useful extra function to know about is `splitlines()`, which splits a string at line breaks and returns the split parts as a list." @@ -679,7 +683,7 @@ }, { "cell_type": "markdown", - "id": "752b9704", + "id": "66", "metadata": {}, "source": [ "### count and find\n", @@ -690,7 +694,7 @@ { "cell_type": "code", "execution_count": null, - "id": "22f94993", + "id": "67", "metadata": {}, "outputs": [], "source": [ @@ -701,7 +705,7 @@ }, { "cell_type": "markdown", - "id": "378dade0", + "id": "68", "metadata": {}, "source": [ "Meanwhile, `find()` returns the position where a particular word or character occurs." @@ -710,7 +714,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a351a11b", + "id": "69", "metadata": {}, "outputs": [], "source": [ @@ -719,7 +723,7 @@ }, { "cell_type": "markdown", - "id": "7fb9ae26", + "id": "70", "metadata": {}, "source": [ "We can check this using the number we get and some string indexing:" @@ -728,7 +732,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8e0a7020", + "id": "71", "metadata": {}, "outputs": [], "source": [ @@ -737,7 +741,7 @@ }, { "cell_type": "markdown", - "id": "ecc16d15", + "id": "72", "metadata": {}, "source": [ "But this isn't the only place where the word 'coffee' appears. If we want to find the last occurrence, it's" @@ -746,7 +750,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e18f64a3", + "id": "73", "metadata": {}, "outputs": [], "source": [ @@ -755,7 +759,7 @@ }, { "cell_type": "markdown", - "id": "f30ef28a", + "id": "74", "metadata": {}, "source": [ "## Working with Multiple Strings" @@ -763,12 +767,12 @@ }, { "cell_type": "markdown", - "id": "09f7c915", + "id": "75", "metadata": {}, "source": [ "We've seen how to work with individual strings. But often we want to work with a group of strings, otherwise known as a corpus, that is a collection of texts. It could be a collection of words, sentences, paragraphs, or some domain-based grouping (eg job descriptions). Just like any other Python object, you can put strings into a list (or other iterable).\n", "\n", - "And, fortunately, many of the methods that we have seen deployed on a single string can be straightforwardly scaled up to hundreds, thousands, or millions of strings using **pandas** or other tools. This scaling up is achieved via *vectorisation*, in analogy with going from a single value (a scalar) to multiple values in a list (a vector).\n", + "And, fortunately, many of the methods that we have seen deployed on a single string can be straightforwardly scaled up to hundreds, thousands, or millions of strings using **polars** or other tools. This scaling up is achieved via *vectorisation*, in analogy with going from a single value (a scalar) to multiple values in a list (a vector).\n", "\n", "As a very minimal example, here is capitalisation of names vectorised using a list comprehension:\n" ] @@ -776,7 +780,7 @@ { "cell_type": "code", "execution_count": null, - "id": "bbc3eb7b", + "id": "76", "metadata": {}, "outputs": [], "source": [ @@ -785,49 +789,48 @@ }, { "cell_type": "markdown", - "id": "730df2c8", + "id": "77", "metadata": {}, "source": [ - "A **pandas** series can be used in place of a list. Let's create the series first:" + "A **polars** series can be used in place of a list. Let's create the series first:" ] }, { "cell_type": "code", "execution_count": null, - "id": "c8a7f68b", + "id": "78", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "\n", - "dfs = pd.Series(\n", + "dfs = pl.Series(\n", " [\"ada lovelace\", \"adam smith\", \"elinor ostrom\", \"grace hopper\", \"jean bartik\"],\n", - " dtype=\"string\",\n", ")\n", "dfs" ] }, { "cell_type": "markdown", - "id": "222fbb51", + "id": "79", "metadata": {}, "source": [ - "Now we use the syntax series.str.function to change the text series:\n" + "Now we use the syntax `.str.function` to change the text series:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "7cf149b5", + "id": "80", "metadata": {}, "outputs": [], "source": [ - "dfs.str.title()" + "dfs.str.to_titlecase()" ] }, { "cell_type": "markdown", - "id": "ec439fc1", + "id": "81", "metadata": {}, "source": [ "If we had a data frame and not a series, the syntax would change to refer just to the column of interest like so:" @@ -836,55 +839,48 @@ { "cell_type": "code", "execution_count": null, - "id": "26dc9a7b", + "id": "82", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(dfs, columns=[\"names\"])\n", - "df[\"names\"].str.title()" + "df = pl.DataFrame({\"names\": dfs})\n", + "df.with_columns(pl.col(\"names\").str.to_titlecase())" ] }, { "cell_type": "markdown", - "id": "2e430dd9", + "id": "83", "metadata": {}, "source": [ - "The table below shows a non-exhaustive list of the string methods that are available in **pandas**.\n", + "The table below shows a non-exhaustive list of the string methods that are available in **polars**.\n", "\n", "| Function (preceded by `.str.`) | What it does |\n", "|-----------------------------|-------------------------|\n", - "| `len()` | Length of string. |\n", - "| `lower()` | Put string in lower case. |\n", - "| `upper()` | Put string in upper case. |\n", - "| `capitalize()` | Put string in leading upper case. |\n", - "| `swapcase()` | Swap cases in a string. |\n", - "| `translate()` | Returns a copy of the string in which each character has been mapped through a given translation table. |\n", - "| `ljust()` | Left pad a string (default is to pad with spaces) |\n", - "| `rjust()` | Right pad a string (default is to pad with spaces) |\n", - "| `center()` | Pad such that string appears in centre (default is to pad with spaces) |\n", - "| `zfill()` | Pad with zeros |\n", - "| `strip()` | Strip out leading and trailing whitespace |\n", - "| `rstrip()` | Strip out trailing whitespace |\n", - "| `lstrip()` | Strip out leading whitespace |\n", - "| `find()` | Return the lowest index in the data where a substring appears |\n", + "| `len_chars()` | Length of string in Unicode characters. |\n", + "| `to_lowercase()` | Put string in lower case. |\n", + "| `to_uppercase()` | Put string in upper case. |\n", + "| `to_titlecase()` | Put string in leading upper case. |\n", + "| `strip_chars()` | Strip out leading and trailing whitespace or specified characters |\n", + "| `strip_chars_end()` | Strip out trailing whitespace or specified characters |\n", + "| `strip_chars_start()` | Strip out leading whitespace or specified characters |\n", + "| `find()` | Return the lowest index in the data where a substring or pattern appears |\n", "| `split()` | Split the string using a passed substring as the delimiter |\n", - "| `isupper()` | Check whether string is upper case |\n", - "| `isdigit()` | Check whether string is composed of digits |\n", - "| `islower()` | Check whether string is lower case |\n", - "| `startswith()` | Check whether string starts with a given sub-string |\n", + "| `starts_with()` | Check whether string starts with a given sub-string |\n", + "| `ends_with()` | Check whether string ends with a given sub-string |\n", + "| `contains()` | Check whether string contains a given pattern |\n", + "| `replace()` | Replace first occurrence of pattern with another string |\n", + "| `replace_all()` | Replace all occurrences of pattern |\n", + "| `slice()` | Extract a substring by position and length |\n", "\n", - "Regular expressions can also be scaled up with **pandas**. The below table shows vectorised regular expressions.\n", + "Regular expressions can also be scaled up with **polars**. The below table shows vectorised regular expressions.\n", "\n", "| Function | What it does |\n", "|-|----------------------------------|\n", - "| `match()` | Call `re.match()` on each element, returning a boolean. |\n", - "| `extract()` | Call `re.match()` on each element, returning matched groups as strings. |\n", - "| `findall()` | Call `re.findall()` on each element |\n", + "| `contains()` | Check if pattern exists, returning a boolean |\n", "| `replace()` | Replace occurrences of pattern with some other string |\n", - "| `contains()` | Call `re.search()` on each element, returning a boolean |\n", - "| `count()` | Count occurrences of pattern |\n", - "| `split()` | Equivalent to `str.split()`, but accepts regexes |\n", - "| `rsplit()` | Equivalent to `str.rsplit()`, but accepts regexes |\n", + "| `replace_all()` | Replace all occurrences of pattern |\n", + "| `extract()` | Extract matched groups as strings |\n", + "| `extract_all()` | Extract all matched groups |\n", "\n", "\n", "Let's see a couple of these in action. First, splitting on a given sub-string:" @@ -893,104 +889,102 @@ { "cell_type": "code", "execution_count": null, - "id": "d7a29663", + "id": "84", "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.split(\" \")" + "df.with_columns(pl.col(\"names\").str.split(\" \"))" ] }, { "cell_type": "markdown", - "id": "597c0d23", + "id": "85", "metadata": {}, "source": [ - "It's fairly common that you want to split out strings and save the results to new columns in your data frame. You can specify a (max) number of splits via the `n=` kwarg and you can get the columns using `expand`" + "It's fairly common that you want to split out strings and save the results to new columns in your data frame. You can get the columns using `list` and `to_struct()`:" ] }, { "cell_type": "code", "execution_count": null, - "id": "85a5cd2c", + "id": "86", "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.split(\" \", n=2, expand=True)" + "df.with_columns(\n", + " pl.col(\"names\").str.split(\" \").list.to_struct(fields=[\"first\", \"last\"])\n", + ").unnest(\"names\")" ] }, { "cell_type": "markdown", - "id": "e6bb64b2", + "id": "87", "metadata": {}, "source": [ "::: {.callout-tip title=\"Exercise\"}\n", "Using vectorised operations, create a new column with the index position where the first vowel occurs for each row in the `names` column.\n", ":::\n", "\n", - "Here's an example of using a regex function with **pandas**:" + "Here's an example of using a regex function with **polars**:" ] }, { "cell_type": "code", "execution_count": null, - "id": "2e8781ba", + "id": "88", "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.extract(\"(\\w+)\", expand=False)" + "df.with_columns(pl.col(\"names\").str.extract(r\"(\\w+)\", 1))" ] }, { "cell_type": "markdown", - "id": "7fb6d243", + "id": "89", "metadata": {}, "source": [ "There are a few more vectorised string operations that are useful.\n", "\n", "| Method | Description |\n", "|-|-|\n", - "| `get()` | Index each element |\n", - "| `slice()` | Slice each element |\n", - "| `slice_replace()` | Replace slice in each element with passed value |\n", - "| `cat()` | Concatenate strings |\n", - "| `repeat()` | Repeat values |\n", - "| `normalize()` | Return Unicode form of string |\n", - "| `pad()` | Add whitespace to left, right, or both sides of strings |\n", - "| `wrap()` | Split long strings into lines with length less than a given width |\n", - "| `join()` | Join strings in each element of the Series with passed separator |\n", - "| `get_dummies()` | extract dummy variables as a data frame |\n", + "| `slice()` | Slice each string element by offset and length |\n", + "| `concat_str()` | Concatenate strings across columns or expressions |\n", + "| `repeat_by()` | Repeat values |\n", + "| `contains()` | Check if pattern exists |\n", "\n", "\n", - "The `get()` and `slice()` methods give access to elements of the lists returned by `split()`. Here's an example that combines `split()` and `get()`:\n", - "\n" + "The `slice()` method gives access to substrings by position. When strings contain delimited parts, you can instead combine `str.split()` with `list.get()` to select a part such as the last token:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "ba13d894", + "id": "90", "metadata": {}, "outputs": [], "source": [ - "df[\"names\"].str.split().str.get(-1)" + "df.with_columns(\n", + " pl.col(\"names\").str.slice(0, 5).alias(\"first_five_characters\"),\n", + " pl.col(\"names\").str.split(\" \").list.get(-1).alias(\"last_name\"),\n", + ")" ] }, { "cell_type": "markdown", - "id": "4ce15edb", + "id": "91", "metadata": {}, "source": [ - "If we have a column with tags split by a symbol, we can use the `get_dummies()` function to split it out. For example, let's create a data frame with a single column that mixes subject and nationality tags:\n" + "If we have a column whose values contain a fixed set of fields separated by a symbol, we can use `str.split()` to separate those fields. This differs from dummy encoding: it creates one column per position rather than one indicator column per unique value. For example, let's create a data frame with a single column that combines nationality and subject tags:\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "056147d6", + "id": "92", "metadata": {}, "outputs": [], "source": [ - "df = pd.DataFrame(\n", + "df = pl.DataFrame(\n", " {\n", " \"names\": [\n", " \"ada lovelace\",\n", @@ -1007,25 +1001,28 @@ }, { "cell_type": "markdown", - "id": "f44b0d84", + "id": "93", "metadata": {}, "source": [ - "If we now use `str.get_dummies()` and split on `;` we can get a data frame of dummies." + "Because every value contains the fields in the same order, we can use `str.split()` and list operations to create separate `country` and `subject` columns:\n", + "\n" ] }, { "cell_type": "code", "execution_count": null, - "id": "a5cbc10f", + "id": "94", "metadata": {}, "outputs": [], "source": [ - "df[\"tags\"].str.get_dummies(\";\")" + "df.with_columns(\n", + " pl.col(\"tags\").str.split(\"; \").list.to_struct(fields=[\"country\", \"subject\"])\n", + ").unnest(\"tags\")" ] }, { "cell_type": "markdown", - "id": "58521740", + "id": "95", "metadata": {}, "source": [ "## Reading Text In\n", @@ -1040,34 +1037,29 @@ " text_of_book = f.read()\n", "```\n", "\n", - "You can also read a text file directly into a **pandas** data frame using \n", + "You can also read a text file directly into a **polars** data frame using \n", "\n", "```python\n", - "df = pd.read_csv('book.txt', delimiter = \"\\n\")\n", + "df = pl.read_csv('book.txt', separator=\"\\n\", has_header=False)\n", "```\n", "\n", "In the above, the delimiter for different rows of the data frame is set as \"\\n\", which means new line, but you could use whatever delimiter you prefer.\n", "\n", "::: {.callout-tip title=\"Exercise\"}\n", - "Download the file 'smith_won.txt' using this [link](https://github.com/aeturrell/coding-for-economists/blob/main/data/smith_won.txt) (use right-click and save as). Then read the text in using **pandas**.\n", + "Download the file 'smith_won.txt' using this [link](https://github.com/aeturrell/coding-for-economists/blob/main/data/smith_won.txt) (use right-click and save as). Then read the text in using **polars**.\n", ":::\n", "\n", "### CSV file\n", "\n", - "CSV files are already split into rows. By far the easiest way to read in csv files is using **pandas**,\n", + "CSV files are already split into rows. By far the easiest way to read in csv files is using **polars**,\n", "\n", "```python\n", - "df = pd.read_csv('book.csv')\n", - "```\n", - "\n", - "Remember that **pandas** can read many other file types too." + "df = pl.read_csv('book.csv')\n", + "```" ] } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -1075,7 +1067,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, diff --git a/vis-layers.ipynb b/vis-layers.ipynb index 6a92e84..cf6a92d 100644 --- a/vis-layers.ipynb +++ b/vis-layers.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Layers {#sec-vis-layers}\n" @@ -10,7 +10,7 @@ }, { "cell_type": "markdown", - "id": "9fdd3b8a", + "id": "1", "metadata": {}, "source": [ "## Introduction\n", @@ -30,12 +30,12 @@ }, { "cell_type": "markdown", - "id": "17575f3a", + "id": "2", "metadata": {}, "source": [ "### Prerequisites\n", "\n", - "You will need to install the **letsplot** package for this chapter, as well as **pandas**.\n", + "You will need to install the **letsplot** package for this chapter, as well as **polars**.\n", "\n", "In your Python session, import the libraries we'll be using:" ] @@ -43,11 +43,11 @@ { "cell_type": "code", "execution_count": null, - "id": "a86fb211", + "id": "3", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd\n", + "import polars as pl\n", "from lets_plot import *\n", "\n", "LetsPlot.setup_html()" @@ -55,7 +55,7 @@ }, { "cell_type": "markdown", - "id": "55b00fde", + "id": "4", "metadata": {}, "source": [ "## Aesthetic mappings\n", @@ -68,35 +68,29 @@ { "cell_type": "code", "execution_count": null, - "id": "39a6d993", + "id": "5", "metadata": {}, "outputs": [], "source": [ - "mpg = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\", index_col=0\n", + "mpg = pl.read_csv(\"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/mpg.csv\")\n", + "\n", + "mpg = mpg.with_columns(\n", + " [\n", + " pl.col(\"manufacturer\").cast(pl.Categorical),\n", + " pl.col(\"model\").cast(pl.Categorical),\n", + " pl.col(\"trans\").cast(pl.Categorical),\n", + " pl.col(\"drv\").cast(pl.Categorical),\n", + " pl.col(\"fl\").cast(pl.Categorical),\n", + " pl.col(\"class\").cast(pl.Categorical),\n", + " ]\n", ")\n", "\n", - "mpg = mpg.astype(\n", - " {\n", - " \"manufacturer\": \"category\",\n", - " \"model\": \"category\",\n", - " \"displ\": \"double\",\n", - " \"year\": \"int64\",\n", - " \"cyl\": \"int64\",\n", - " \"trans\": \"category\",\n", - " \"drv\": \"category\",\n", - " \"cty\": \"double\",\n", - " \"hwy\": \"double\",\n", - " \"fl\": \"category\",\n", - " \"class\": \"category\",\n", - " }\n", - ")\n", "mpg.head()" ] }, { "cell_type": "markdown", - "id": "6d6f1307", + "id": "6", "metadata": {}, "source": [ "Among the variables in `mpg` are:\n", @@ -118,7 +112,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fe77349a", + "id": "7", "metadata": {}, "outputs": [], "source": [ @@ -128,7 +122,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e77b5640", + "id": "8", "metadata": {}, "outputs": [], "source": [ @@ -137,7 +131,7 @@ }, { "cell_type": "markdown", - "id": "53e51510", + "id": "9", "metadata": {}, "source": [ "Similarly, we can map `class` to `size` or `alpha` aesthetics as well, which control the shape and the transparency of the points, respectively." @@ -146,7 +140,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ef221330", + "id": "10", "metadata": {}, "outputs": [], "source": [ @@ -156,7 +150,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d042255e", + "id": "11", "metadata": {}, "outputs": [], "source": [ @@ -165,7 +159,7 @@ }, { "cell_type": "markdown", - "id": "ec07349f", + "id": "12", "metadata": {}, "source": [ "While we are able to do it, mapping an unordered discrete (categorical) variable (`class`) to an ordered aesthetic variable (`size` or `alpha`) is generally not a good idea because it implies a ranking that does not in fact exist.\n", @@ -182,7 +176,7 @@ { "cell_type": "code", "execution_count": null, - "id": "618edcb4", + "id": "13", "metadata": {}, "outputs": [], "source": [ @@ -191,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "52611640", + "id": "14", "metadata": {}, "source": [ "Here, the colour doesn't convey information about a variable, but only changes the appearance of the plot.\n", @@ -206,7 +200,7 @@ }, { "cell_type": "markdown", - "id": "37d49ff8", + "id": "15", "metadata": {}, "source": [ "So far we have discussed aesthetics that we can map or set in a scatterplot, when using a point geom.\n", @@ -217,7 +211,7 @@ }, { "cell_type": "markdown", - "id": "3f1da019", + "id": "16", "metadata": {}, "source": [ "1. Create a scatterplot of `hwy` vs. `displ` where the points are pink filled in triangles.\n", @@ -240,7 +234,7 @@ }, { "cell_type": "markdown", - "id": "83aa98f0", + "id": "17", "metadata": {}, "source": [ "## Geometric objects\n", @@ -251,7 +245,7 @@ { "cell_type": "code", "execution_count": null, - "id": "277a4c0f", + "id": "18", "metadata": {}, "outputs": [], "source": [ @@ -261,7 +255,7 @@ { "cell_type": "code", "execution_count": null, - "id": "07247ba9", + "id": "19", "metadata": {}, "outputs": [], "source": [ @@ -270,7 +264,7 @@ }, { "cell_type": "markdown", - "id": "58824a10", + "id": "20", "metadata": {}, "source": [ "Both plots contain the same x variable, the same y variable, and both describe the same data.\n", @@ -283,7 +277,7 @@ }, { "cell_type": "markdown", - "id": "0db26c6c", + "id": "21", "metadata": {}, "source": [ "Every geom function in **lets-plot** takes a `mapping` argument, either defined locally in the geom layer or globally in the `ggplot()` layer.\n", @@ -299,7 +293,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4b20c825", + "id": "22", "metadata": {}, "outputs": [], "source": [ @@ -309,7 +303,7 @@ { "cell_type": "code", "execution_count": null, - "id": "84df3e78", + "id": "23", "metadata": {}, "outputs": [], "source": [ @@ -318,7 +312,7 @@ }, { "cell_type": "markdown", - "id": "7b114911", + "id": "24", "metadata": {}, "source": [ "Here, `geom_smooth()` separates the cars into three lines based on their `drv` value, which describes a car's drive train.\n", @@ -331,7 +325,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c9e8d92f", + "id": "25", "metadata": {}, "outputs": [], "source": [ @@ -344,7 +338,7 @@ }, { "cell_type": "markdown", - "id": "b6392da1", + "id": "26", "metadata": {}, "source": [ "Notice that this plot contains two geoms in the same graph.\n", @@ -363,7 +357,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b3916558", + "id": "27", "metadata": {}, "outputs": [], "source": [ @@ -372,7 +366,7 @@ }, { "cell_type": "markdown", - "id": "88708546", + "id": "28", "metadata": {}, "source": [ "You can use the same idea to specify different data for each layer.\n", @@ -383,23 +377,23 @@ { "cell_type": "code", "execution_count": null, - "id": "38870eb5", + "id": "29", "metadata": {}, "outputs": [], "source": [ "(\n", " ggplot(mpg, aes(x=\"displ\", y=\"hwy\"))\n", " + geom_point()\n", - " + geom_point(data=mpg.loc[mpg[\"class\"] == \"2seater\", :], color=\"red\", size=2)\n", + " + geom_point(data=mpg.filter(pl.col(\"class\") == \"2seater\"), color=\"red\", size=2)\n", " + geom_point(\n", - " data=mpg.loc[mpg[\"class\"] == \"2seater\", :], shape=1, size=3, color=\"red\"\n", + " data=mpg.filter(pl.col(\"class\") == \"2seater\"), shape=1, size=3, color=\"red\"\n", " )\n", ")" ] }, { "cell_type": "markdown", - "id": "35a3d017", + "id": "30", "metadata": {}, "source": [ "Geoms are the fundamental building blocks of **lets-plot**.\n", @@ -415,7 +409,7 @@ }, { "cell_type": "markdown", - "id": "39a12f36", + "id": "31", "metadata": {}, "source": [ "### Exercises\n", @@ -443,7 +437,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ae75c5c1", + "id": "32", "metadata": { "tags": [ "remove-cell" @@ -460,7 +454,7 @@ }, { "cell_type": "markdown", - "id": "4c1d45ab", + "id": "33", "metadata": {}, "source": [ "## Facets\n", @@ -471,7 +465,7 @@ { "cell_type": "code", "execution_count": null, - "id": "cb651300", + "id": "34", "metadata": {}, "outputs": [], "source": [ @@ -480,7 +474,7 @@ }, { "cell_type": "markdown", - "id": "0cd9ef67", + "id": "35", "metadata": {}, "source": [ "To facet your plot with the combination of two variables, switch from `facet_wrap()` to `facet_grid()`." @@ -489,7 +483,7 @@ { "cell_type": "code", "execution_count": null, - "id": "61481052", + "id": "36", "metadata": {}, "outputs": [], "source": [ @@ -498,7 +492,7 @@ }, { "cell_type": "markdown", - "id": "4f502141", + "id": "37", "metadata": {}, "source": [ "By default each of the facets share the same scale and range for x and y axes.\n", @@ -509,7 +503,7 @@ { "cell_type": "code", "execution_count": null, - "id": "adcd9079", + "id": "38", "metadata": {}, "outputs": [], "source": [ @@ -523,7 +517,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ceb2a354", + "id": "39", "metadata": {}, "outputs": [], "source": [ @@ -532,7 +526,7 @@ }, { "cell_type": "markdown", - "id": "5e3a949f", + "id": "40", "metadata": {}, "source": [ "### Exercises\n", @@ -602,7 +596,7 @@ }, { "cell_type": "markdown", - "id": "cacf1fb5", + "id": "41", "metadata": {}, "source": [ "## Statistical transformations\n", @@ -616,34 +610,34 @@ { "cell_type": "code", "execution_count": null, - "id": "f379e31b", + "id": "42", "metadata": {}, "outputs": [], "source": [ - "diamonds = pd.read_csv(\n", - " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv\",\n", - " index_col=0,\n", + "diamonds = pl.read_csv(\n", + " \"https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv\"\n", ")\n", "diamonds_cut_order = [\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]\n", - "diamonds[\"cut\"] = diamonds[\"cut\"].astype(\n", - " pd.CategoricalDtype(categories=diamonds_cut_order, ordered=True)\n", - ")\n", + "diamonds = diamonds.with_columns(pl.col(\"cut\").cast(pl.Enum(diamonds_cut_order)))\n", + "\n", "diamonds.head()" ] }, { "cell_type": "code", "execution_count": null, - "id": "d8faf1ab", + "id": "43", "metadata": {}, "outputs": [], "source": [ - "(ggplot(diamonds, aes(x=\"cut\")) + geom_bar())" + "diamonds_sorted = diamonds.sort(\"cut\")\n", + "\n", + "(ggplot(diamonds_sorted, aes(x=\"cut\")) + geom_bar())" ] }, { "cell_type": "markdown", - "id": "a27666cf", + "id": "44", "metadata": {}, "source": [ "On the x-axis, the chart displays `cut`, a variable from `diamonds`.\n", @@ -666,7 +660,7 @@ }, { "cell_type": "markdown", - "id": "62519f73", + "id": "45", "metadata": {}, "source": [ "You can learn which stat a geom uses by inspecting the default value for the `stat` argument.\n", @@ -680,22 +674,18 @@ { "cell_type": "code", "execution_count": null, - "id": "ca772dd5", + "id": "46", "metadata": {}, "outputs": [], "source": [ - "(\n", - " ggplot(\n", - " diamonds.value_counts(\"cut\").reset_index(name=\"counts\"),\n", - " aes(x=\"cut\", y=\"counts\"),\n", - " )\n", - " + geom_bar(stat=\"identity\")\n", - ")" + "diamonds_counts = diamonds.group_by(\"cut\").agg(pl.len().alias(\"counts\")).sort(\"cut\")\n", + "\n", + "(ggplot(diamonds_counts, aes(x=\"cut\", y=\"counts\")) + geom_bar(stat=\"identity\"))" ] }, { "cell_type": "markdown", - "id": "e365aaaf", + "id": "47", "metadata": {}, "source": [ "## Position adjustments\n", @@ -707,7 +697,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f8da7d91", + "id": "48", "metadata": {}, "outputs": [], "source": [ @@ -717,7 +707,7 @@ { "cell_type": "code", "execution_count": null, - "id": "088e7550", + "id": "49", "metadata": {}, "outputs": [], "source": [ @@ -726,7 +716,7 @@ }, { "cell_type": "markdown", - "id": "c3f7a0a3", + "id": "50", "metadata": {}, "source": [ "Note what happens if you map the fill aesthetic to another variable, like `class`: the bars are automatically stacked.\n", @@ -736,7 +726,7 @@ { "cell_type": "code", "execution_count": null, - "id": "181c70d2", + "id": "51", "metadata": {}, "outputs": [], "source": [ @@ -745,7 +735,7 @@ }, { "cell_type": "markdown", - "id": "b3f74621", + "id": "52", "metadata": {}, "source": [ "The stacking is performed automatically using the **position adjustment** specified by the `position` argument.\n", @@ -759,7 +749,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a8e9c378", + "id": "53", "metadata": {}, "outputs": [], "source": [ @@ -768,7 +758,7 @@ }, { "cell_type": "markdown", - "id": "2aeffbf7", + "id": "54", "metadata": {}, "source": [ "The identity position adjustment is more useful for 2d geoms, like points, where it is the default.\n", @@ -780,7 +770,7 @@ { "cell_type": "code", "execution_count": null, - "id": "14205000", + "id": "55", "metadata": {}, "outputs": [], "source": [ @@ -789,7 +779,7 @@ }, { "cell_type": "markdown", - "id": "3a8e72bd", + "id": "56", "metadata": {}, "source": [ "- `position = \"dodge\"` places overlapping objects directly *beside* one another.\n", @@ -799,7 +789,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c33c4a03", + "id": "57", "metadata": {}, "outputs": [], "source": [ @@ -808,7 +798,7 @@ }, { "cell_type": "markdown", - "id": "cd46f22b", + "id": "58", "metadata": {}, "source": [ "There's one other type of adjustment that's not useful for bar charts, but can be very useful for scatterplots.\n", @@ -819,7 +809,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ba4161de", + "id": "59", "metadata": {}, "outputs": [], "source": [ @@ -828,7 +818,7 @@ }, { "cell_type": "markdown", - "id": "28a3bfc7", + "id": "60", "metadata": {}, "source": [ "The underlying values of `hwy` and `displ` are rounded so the points appear on a grid and many points overlap each other.\n", @@ -844,7 +834,7 @@ { "cell_type": "code", "execution_count": null, - "id": "414ce7af", + "id": "61", "metadata": {}, "outputs": [], "source": [ @@ -853,7 +843,7 @@ }, { "cell_type": "markdown", - "id": "96277926", + "id": "62", "metadata": {}, "source": [ "Adding randomness seems like a strange way to improve your plot, but while it makes your graph less accurate at small scales, it makes your graph *more* revealing at large scales.\n", @@ -867,7 +857,7 @@ }, { "cell_type": "markdown", - "id": "90d09736", + "id": "63", "metadata": {}, "source": [ "### Exercises\n", @@ -879,7 +869,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9bc38aef", + "id": "64", "metadata": {}, "outputs": [], "source": [ @@ -888,7 +878,7 @@ }, { "cell_type": "markdown", - "id": "57b6523f", + "id": "65", "metadata": {}, "source": [ "2. What, if anything, is the difference between the two plots?\n", @@ -908,7 +898,7 @@ }, { "cell_type": "markdown", - "id": "a3138f75", + "id": "66", "metadata": {}, "source": [ "3. What parameters to `geom_jitter()` control the amount of jittering?\n", @@ -919,7 +909,7 @@ }, { "cell_type": "markdown", - "id": "7d302c31", + "id": "67", "metadata": {}, "source": [ "## The layered grammar of graphics\n", @@ -954,7 +944,7 @@ }, { "cell_type": "markdown", - "id": "c0f7a7ff", + "id": "68", "metadata": {}, "source": [ "At this point, you would have a complete graph, but you could further adjust the positions of the geoms within the coordinate system (a position adjustment) or split the graph into subplots (faceting).\n", @@ -971,9 +961,6 @@ } ], "metadata": { - "interpreter": { - "hash": "9d7534ecd9fbc7d385378f8400cf4d6cb9c6175408a574f1c99c5269f08771cc" - }, "jupytext": { "cell_metadata_filter": "-all", "encoding": "# -*- coding: utf-8 -*-", @@ -981,7 +968,7 @@ "main_language": "python" }, "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": ".venv", "language": "python", "name": "python3" }, diff --git a/visualise.quarto_ipynb_1 b/visualise.quarto_ipynb_1 deleted file mode 100644 index 2ef6d2f..0000000 --- a/visualise.quarto_ipynb_1 +++ /dev/null @@ -1,136 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Visualisation {#sec-visualise}\n", - "\n", - "After reading the first part of the book, you understand the basics of the most important tools for doing data science. Now it’s time to start diving into the details. In this part of the book, you’ll learn about visualising data in further depth (in @sec-vis-layers), and get further stuck into the details of the different kinds of data visualisation (in @sec-exploratory-data-analysis and @sec-communicate-plots). In this short chapter, we discuss the different ways to create visualisations, and the different purposes of visualisations.\n", - "\n", - "## Philosophies of data visualisation\n", - "\n", - "There are broadly two categories of approach to using code to create data visualisations: *imperative* (build what you want from individual elements) and *declarative* (say what you want from a list of pre-existing options). Choosing which to use involves a trade-off: imperative libraries offer you flexibility but at the cost of some verbosity; declarative libraries offer you a quick way to plot your data, but only if it’s in the right format to begin with, and customisation to special chart types is more difficult.\n", - "\n", - "Python has many excellent plotting packages, including perhaps the most powerful imperative plotting package around, **matplotlib**, and an amazing declarative library that we already saw, **lets-plot**. These two libraries will get you a long way, and each could be worthy of an entire book themselves. Fortunately for us, though, we can do 95% of what we need with a small number of commands from one or the other of them. In general, to keep this book as light as possible, we've opted to use **lets-plot** wherever possible—and @sec-vis-layers is going to take you on a more in-depth tour of how to use it yourself.\n", - "\n", - "## Purposes of data visualisation\n", - "\n", - "Data visualisation has all kinds of different purposes. It can be useful to bear in mind three broad categories of visualisation that are out there:\n", - "\n", - "- exploratory\n", - "- scientific\n", - "- narrative\n", - "\n", - "Let's look at each in a bit more detail.\n", - "\n", - "### Exploratory Data Viz\n", - "\n", - "The first of the three kinds is *exploratory data visualisation*, and it's the kind that you do when you're looking and data and trying to understand it. Just plotting the data is a really good strategy for getting a feel for any issues there might be. This is perhaps most famously demonstrated by Anscombe's quartet: four different datasets with the same mean, standard deviation, and correlation but very different data distributions." - ], - "id": "f3331573" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "#| echo: false\n", - "import numpy as np\n", - "import pandas as pd\n", - "import matplotlib.pyplot as plt\n", - "import matplotlib_inline.backend_inline\n", - "\n", - "# Plot settings\n", - "plt.style.use(\"https://github.com/aeturrell/python4DS/raw/main/plot_style.txt\")\n", - "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")\n", - "\n", - "# Set max rows displayed for readability\n", - "pd.set_option(\"display.max_rows\", 6)\n", - "\n", - "x = [10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5]\n", - "y1 = [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]\n", - "y2 = [9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74]\n", - "y3 = [7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73]\n", - "x4 = [8, 8, 8, 8, 8, 8, 8, 19, 8, 8, 8]\n", - "y4 = [6.58, 5.76, 7.71, 8.84, 8.47, 7.04, 5.25, 12.50, 5.56, 7.91, 6.89]\n", - "\n", - "datasets = {\"I\": (x, y1), \"II\": (x, y2), \"III\": (x, y3), \"IV\": (x4, y4)}\n", - "\n", - "fig, axs = plt.subplots(\n", - " 2,\n", - " 2,\n", - " sharex=True,\n", - " sharey=True,\n", - " figsize=(10, 6),\n", - " gridspec_kw={\"wspace\": 0.08, \"hspace\": 0.08},\n", - ")\n", - "axs[0, 0].set(xlim=(0, 20), ylim=(2, 14))\n", - "axs[0, 0].set(xticks=(0, 10, 20), yticks=(4, 8, 12))\n", - "\n", - "for ax, (label, (x, y)) in zip(axs.flat, datasets.items()):\n", - " ax.text(0.1, 0.9, label, fontsize=20, transform=ax.transAxes, va=\"top\")\n", - " ax.tick_params(direction=\"in\", top=True, right=True)\n", - " ax.plot(x, y, \"o\")\n", - "\n", - " # linear regression\n", - " p1, p0 = np.polyfit(x, y, deg=1) # slope, intercept\n", - " ax.axline(xy1=(0, p0), slope=p1, color=\"r\", lw=2)\n", - "\n", - " # add text box for the statistics\n", - " stats = (\n", - " f\"$\\\\mu$ = {np.mean(y):.2f}\\n\"\n", - " f\"$\\\\sigma$ = {np.std(y):.2f}\\n\"\n", - " f\"$r$ = {np.corrcoef(x, y)[0][1]:.2f}\"\n", - " )\n", - " bbox = dict(boxstyle=\"round\", fc=\"blanchedalmond\", ec=\"orange\", alpha=0.5)\n", - " ax.text(\n", - " 0.95,\n", - " 0.07,\n", - " stats,\n", - " fontsize=9,\n", - " bbox=bbox,\n", - " transform=ax.transAxes,\n", - " horizontalalignment=\"right\",\n", - " )\n", - "\n", - "plt.suptitle(\"Anscombe's Quartet\")\n", - "plt.show()" - ], - "id": "64a0e7f6", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Exploratory visualisation is usually quick and dirty, and flexible too. Some exploratory data viz can be automated, and there's a whole host of packages to help with this, including [**skimpy**](https://aeturrell.github.io/skimpy/).\n", - "\n", - "Beyond you and perhaps your co-authors/collaborators, however, not many other people should be seeing your exploratory visualisation! They will typically be worked up quickly, be numerous, and be throw-away. We'll look more at this in @sec-exploratory-data-analysis.\n", - "\n", - "### Scientific Data Viz\n", - "\n", - "The second kind, scientific data visualisation, is the prime cut of your exploratory visualisation. It's the kind of plot you might include in a more technical paper, the picture that says a thousand words. I often think of the first image of a black hole @akiyama2019first as a prime example of this. You can get away with having a high density of information in a scientific plot and, in short format journals, you may need to. The journal Physical Review Letters, which has an 8 page limit, has a classic of this genre in more or less every issue. Ensuring that important values can be accurately read from the plot is especially important in these kinds of charts. But they can also be the kind of plot that presents the killer results in a study; they might not be exciting to people who don't look at charts for a living, but they might be exciting and, just as importantly, understandable by your peers.\n", - "\n", - "This type of visualisation is especially popular in the big science journals like *Nature* and *Science*, where space is at a premium. We won't cover this type of plot in this book, because it tends to be very bespoke.\n", - "\n", - "### Narrative Data Viz\n", - "\n", - "The third and final kind is narrative data visualisation. This is the one that requires the most thought in the step where you go from the first view to the end product. It's a visualisation that doesn't just show a picture, but gives an insight. These are the kind of visualisations that you might see in the *Financial Times*, *The Economist*, or on the *BBC News* website. They come with aids that help the viewer focus on the aspects that the creator wanted them to (you can think of these aids or focuses as doing for visualisation what bold font does for text). They're well worth using in your work, especially if you're trying to communicate a particular narrative, and especially if the people you're communicating with don't have deep knowledge of the topic. You might use them in a paper that you hope will have a wide readership, in a blog post summarising your work, or in a report intended for a policymaker.\n", - "\n", - "You can find more information on the topic of communicating via data visualisations in the @sec-communicate-plots chapter." - ], - "id": "30b9ff30" - } - ], - "metadata": { - "kernelspec": { - "name": "python3", - "language": "python", - "display_name": "Python 3 (ipykernel)", - "path": "/Users/omagic/Documents/GitHub/python4DSpolars/.venv/share/jupyter/kernels/python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file diff --git a/workflow-help.qmd b/workflow-help.qmd index 443bc32..d96321d 100644 --- a/workflow-help.qmd +++ b/workflow-help.qmd @@ -30,7 +30,7 @@ A good reprex makes it easier for other people to help you, and often you'll fig - First, you need to make your code reproducible. This means that you need to capture everything, i.e., include any packages you used and create all necessary objects. The easiest way to make sure you've done this is to use the [**watermark**](https://github.com/rasbt/watermark) package alongside whatever else you are doing: ```{python} -import pandas as pd +import polars as pl import numpy as np import watermark.watermark as watermark @@ -52,16 +52,15 @@ There are several things you need to include to make your example reproducible: - **Packages** and their versions. These should be loaded at the top of the script, so it's easy to see which ones the example needs. By using **watermark** with the above configuration, you will also print the package versions. This is a good time to check that you're using the latest version of each package; it's possible you've discovered a bug that's been fixed since you installed or last updated the package. -- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **pandas**, it's quite easy to generate data from code; here's an example: +- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **polars**, it's quite easy to generate data from code; here's an example: ```{python} -df = pd.DataFrame( +df = pl.DataFrame( data=np.reshape(range(36), (6, 6)), - index=["a", "b", "c", "d", "e", "f"], - columns=["col" + str(i) for i in range(6)], - dtype=float, + schema=["col" + str(i) for i in range(6)], + orient="row", ) -df["random_normal"] = np.random.normal(size=6) +df = df.with_columns(pl.Series("random_normal", np.random.normal(size=6))) df ``` diff --git a/workflow-help.quarto_ipynb_1 b/workflow-help.quarto_ipynb_1 deleted file mode 100644 index e7bebf8..0000000 --- a/workflow-help.quarto_ipynb_1 +++ /dev/null @@ -1,115 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Postscript: Getting Further Help {#sec-workflow-help}\n", - "\n", - "This book is not an island; there is no single resource that will allow you to master Python for Data Science. As you begin to apply the techniques described in this book to your own data, you will soon find questions that we do not answer. This section describes a few tips on how to get help, and to help you keep learning.\n", - "\n", - "## Resources\n", - "\n", - "Some other resources for learning are:\n", - "\n", - "- [The Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)\n", - "- [Real Python](https://realpython.com/), which has excellent short tutorials that cover Python more broadly (not just data science)\n", - "- [freeCodeCamp's Python courses](https://www.freecodecamp.org/news/search?query=data%20science%20python), though take care to select one that's at the right level for you\n", - "- [Coding for Economists](https://aeturrell.github.io/coding-for-economists), which has similar content to this book but is more in depth and aimed at analysts (particularly in economics)\n", - "\n", - "## Google is your friend\n", - "\n", - "If you get stuck, start with Google. Typically adding \"Python\" or \"Python Data Science\" (as the Python ecosystem goes *well* beyond data science) to a query is enough to restrict it to relevant results. Google is particularly useful for error messages. If you get an error message and you have no idea what it means, try googling it! Chances are that someone else has been confused by it in the past, and there will be help somewhere on the web.\n", - "\n", - "If Google doesn't help, try [Stack Overflow](http://stackoverflow.com). Start by spending a little time searching for an existing answer, including `[Python]` to restrict your search to questions and answers that use Python.\n", - "\n", - "## In the loop\n", - "\n", - "It's also helpful to keep an eye on the latest developments in data science. There are tons of data science newsletters out there, and we recommend keeping up with the Python data science community by following the (#pydata), (#datascience), and (#python) hashtags on Twitter.\n", - "\n", - "## Making a reprex (reproducible example)\n", - "\n", - "If your googling doesn't find anything useful, it's a really good idea prepare a minimal reproducible example or **reprex**.\n", - "\n", - "A good reprex makes it easier for other people to help you, and often you'll figure out the problem yourself in the course of making it. There are two parts to creating a reprex:\n", - "\n", - "- First, you need to make your code reproducible. This means that you need to capture everything, i.e., include any packages you used and create all necessary objects. The easiest way to make sure you've done this is to use the [**watermark**](https://github.com/rasbt/watermark) package alongside whatever else you are doing:" - ], - "id": "22b3f9e0" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "import pandas as pd\n", - "import numpy as np\n", - "import watermark.watermark as watermark\n", - "\n", - "print(watermark())\n", - "print(watermark(iversions=True, globals_=globals()))" - ], - "id": "a119501b", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- Second, you need to make it minimal. Strip away everything that is not directly related to your problem. This usually involves creating a much smaller and simpler Python object than the one you're facing in real life or even using built-in data.\n", - "\n", - "That sounds like a lot of work! And it can be, but it has a great payoff:\n", - "\n", - "- 80% of the time creating an excellent reprex reveals the source of your problem. It's amazing how often the process of writing up a self-contained and minimal example allows you to answer your own question.\n", - "\n", - "- The other 20% of time you will have captured the essence of your problem in a way that is easy for others to play with. This substantially improves your chances of getting help.\n", - "\n", - "There are several things you need to include to make your example reproducible: Python environment, required packages, data, and code.\n", - "\n", - "- **Python environment**--really just the Python version. This is covered by the first call to the **watermark** package.\n", - "\n", - "- **Packages** and their versions. These should be loaded at the top of the script, so it's easy to see which ones the example needs. By using **watermark** with the above configuration, you will also print the package versions. This is a good time to check that you're using the latest version of each package; it's possible you've discovered a bug that's been fixed since you installed or last updated the package.\n", - "\n", - "- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **pandas**, it's quite easy to generate data from code; here's an example:" - ], - "id": "c4ac60b4" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "df = pd.DataFrame(\n", - " data=np.reshape(range(36), (6, 6)),\n", - " index=[\"a\", \"b\", \"c\", \"d\", \"e\", \"f\"],\n", - " columns=[\"col\" + str(i) for i in range(6)],\n", - " dtype=float,\n", - ")\n", - "df[\"random_normal\"] = np.random.normal(size=6)\n", - "df" - ], - "id": "d1e4562c", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- **Code**: copy and paste the minimal reproducible example code (including the packages, as noted above). Make sure you've used spaces and your variable names are concise, yet informative. Use comments to indicate where your problem lies. Do your best to remove everything that is not related to the problem. Finally, the shorter your code is, the easier it is to understand, and the easier it is to fix.\n", - "\n", - "Finish by checking that you have actually made a reproducible example by starting a fresh Python session and copying and pasting your reprex in." - ], - "id": "4b75e409" - } - ], - "metadata": { - "kernelspec": { - "name": "python3", - "language": "python", - "display_name": "Python 3 (ipykernel)", - "path": "/Users/omagic/Documents/GitHub/python4DSpolars/.venv/share/jupyter/kernels/python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file diff --git a/workflow-packages-and-environments.quarto_ipynb_1 b/workflow-packages-and-environments.quarto_ipynb_1 deleted file mode 100644 index a5600ce..0000000 --- a/workflow-packages-and-environments.quarto_ipynb_1 +++ /dev/null @@ -1,149 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Workflow: Packages and Environments {#sec-workflow-packages-and-environments}\n", - "\n", - "In this chapter, you're going to learn about packages and how to install them plus virtual coding environments that keep your packages isolated and your projects reproducible.\n", - "\n", - "## Packages\n", - "\n", - "### Introduction\n", - "\n", - "Packages (also called libraries) are key to extending the functionality of Python. It won't be long before you'll need to install some. There are packages for geoscience, for building websites, for analysing genetic data, for economics—pretty much for anything you can think of. Packages are typically not written by the core maintainers of the Python language but by enthusiasts, firms, researchers, academics, all sorts! Because anyone can write packages, they vary widely in their quality and usefulness. There are some that you'll be seeing them again and again.\n", - "\n", - "

Name a more iconic trio, I'll wait. pic.twitter.com/pGaLuUxQ3r

— Vicki Boykis (\\@vboykis) August 23, 2018
\n", - "\n", - "The three Python packages **numpy**, **pandas**, and **maplotlib**, which respectively provide numerical, data analysis, and plotting functionality, are ubiquitous. So many scripts begin by importing all three of them, as in the tweet above!\n", - "\n", - "There are typically two steps to using a new Python package:\n", - "\n", - "1. *install* the package on the command line (aka the terminal), eg using `uv add pandas`\n", - "\n", - "2. *import* the package into your Python session, eg using `import pandas as pd`\n", - "\n", - "When you issue an install command for a specific package, it is automatically downloaded from the internet and installed in the appropriate place on your computer. To install extra Python packages, you issue install commands to a text-based window called the \"terminal\".\n", - "\n", - "### The Command Line in Brief\n", - "\n", - "The *terminal* or *command line* or sometimes the *command prompt* was labelled 4 in the screenshot of Visual Studio Code from the chapter on @sec-introduction. The terminal is a text-based way to issue all kinds of commands to your computer (not just Python commands) and knowing a little bit about it is really useful for coding (and more) because managing packages, environments (which we haven't yet discussed), and version control (ditto) can all be done via the terminal. We'll come to these in due course in the chapter on @sec-command-line, but for now, a little background on what the terminal is and what it does.\n", - "\n", - "::: {.callout-note}\n", - "To open up the command line within Visual Studio Code, use the + \\` keyboard shortcut (Mac) or ctrl + \\` (Windows/Linux), or click \"View > Terminal\".\n", - "\n", - "If you want to open up the command line independently of Visual Studio Code, search for \"Terminal\" on Mac and Linux, and \"Powershell\" on Windows.\n", - ":::\n", - "\n", - "Firstly, everything you can do by clicking on icons to launch programmes on your computer, you can also do via the terminal, also known as the command line. For many programmes, a lot of their functionality can be accessed using the command line, and other programmes *only* have a command line interface (CLI), including some that are used for data science.\n", - "\n", - "::: {.callout-tip}\n", - "The command line interacts with your operating system and is used to create, activate, or change Python installations.\n", - ":::\n", - "\n", - "Use Visual Studio Code to open a terminal window by clicking Terminal -> New Terminal on the list of commands at the very top of the window. If you have installed uv on your computer, your terminal should look something like this as your 'command prompt':\n", - "\n", - "```bash\n", - "your-username@your-computer current-directory %\n", - "```\n", - "\n", - "on Mac, and the same but with '%' replaced by '$' on linux, and (using Powershell)\n", - "\n", - "```powershell\n", - "PS C:\\Windows\\System32>\n", - "```\n", - "\n", - "on Windows.\n", - "\n", - "You can check that uv has successfully installed Python in your current project's folder by running\n", - "\n", - "```bash\n", - "uv run python --version\n", - "```\n", - "\n", - "For now, to at least try out the command line, let's use something that works across all three of the major operating systems. Type `uv run python` on the command prompt that came up in your new terminal window. You should see information about your installation of Python appear, including the version, followed by a Python prompt that looks like `>>>`. This is a kind of interactive Python session, in the terminal. It's much less rich than the one available in Visual Studio Code (it can't run scripts line-by-line, for example) but you can try `print('Hello World!')` and it will run, printing your message. To exit the terminal-based Python session, type `exit()` to go back to the regular command line.\n", - "\n", - "### Installing Packages\n", - "\n", - "To install extra Python packages, the default and easiest way is to use `uv add **packagename**`. There are over 330,000 Python packages on PyPI (the Python Package Index)! You can see what packages you have installed already by running `uv pip list` into the command line.\n", - "\n", - "`uv add ...` will install packages into the special Python environment in your current folder (it sits in a subdirectory called \".venv\" which will be hidden by default on most systems.) It's really helpful and good practice to have one Python environment per project, and **uv** does this automatically for you.\n", - "\n", - "::: {.callout-tip title=\"Exercise\"}\n", - "Try installing the **matplotlib**, **pandas**, **statsmodels**, and **skimpy** packages using `uv add`.\n", - ":::\n", - "\n", - "### Using Packages\n", - "\n", - "Once you have installed a package, you need to be able to use it! This is usually done via an import statement at the top of your script or Jupyter Notebook. For example, to bring in **pandas**, it's\n", - "\n", - "```python\n", - "import pandas as pd\n", - "```\n", - "\n", - "Why does Python do this? The idea of not just loading every package is to provide clarity over what function is being called from what package. It's also not necessary to load every package for every piece of analysis, and you often actually want to know what the *minimum* set of packages is to reproduce an analysis. Making the package imports explicit helps with all of that.\n", - "\n", - "You may also wonder why one doesn't just use `import pandas as pandas`. There's actually nothing stopping you doing this except i) it's convenient to have a shorter name and ii) there does tend to be a convention around imports, ie `pd` for **pandas** and `np` for **numpy**, and your code will be clearer to yourself and others if you follow the conventions.\n", - "\n", - "## Virtual Code Environments\n", - "\n", - "Virtual code environments allow you to isolate all of the packages that you're using to do analysis for one project from the set of packages you might need for a different project. They're an important part of creating a reproducible analytical pipeline but a key benefit is that others can reproduce the environment you used and it's best practice to have an isolated environment per project.\n", - "\n", - "To be more concrete, let's say you're using Python 3.9, **statsmodels**, and **pandas** for one project, project A. And, for project B, you need to use Python 3.10 with **numpy** and **scikit-learn**. Even with the same version of Python, best practice would be to have two separate virtual Python environments: environment A, with everything needed for project A, and environment B, with everything needed for project B. For the case where you're using different versions of Python, this isn't just best practice, it's essential.\n", - "\n", - "Many programming languages now come with an option to install packages and a version of the language in isolated environments. In Python, there are multiple tools for managing different environments. And, of those, the easiest to work with is probably [**uv**](https://docs.astral.sh/uv/).\n", - "\n", - "You can see all of the packages in the environment created in your current folder by running `uv pip list` on the command line. Here's an example of looking at the installed packages within this very book, filtering them just to the ones beginning with \"s\".\n", - "\n", - "```{bash}\n", - "uv run pip list | grep ^s\n", - "```\n", - "\n", - "### The pyproject.toml file in Python Environments\n", - "\n", - "You may have noticed that a file called `pyproject.toml` has been created." - ], - "id": "8b889898" - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "import toml\n", - "from rich import print_json\n", - "\n", - "print_json(data=toml.load(\"pyproject.toml\"))" - ], - "id": "688f09f1", - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This lists all of the dependencies, and the version, of a **uv** Python project. There are lots of benefits to tracking what versions of packages you're using like this. One of the most important is that you can *share* projects with other people, and they can install them from these files too.\n", - "\n", - "As you install or remove packages, the `pyproject.toml` file changes in lockstep.\n", - "\n", - "Noe that Visual Studio Code shows which Python environment you are using when you open a Python script or Jupyter Notebook.\n", - "\n", - "![A typical user view in Visual Studio Code](https://github.com/aeturrell/coding-for-economists/blob/main/img/vscode_layout.png?raw=true)\n", - "\n", - "In the screenshot above, you can see the project-environment in two places: on the blue bar at the bottom of the screen, and (in 5), at the top right hand side of the interactive window. A similar top right indicator is present when you have a Jupyter Notebook open too." - ], - "id": "148595b3" - } - ], - "metadata": { - "kernelspec": { - "name": "python3", - "language": "python", - "display_name": "Python 3 (ipykernel)", - "path": "/Users/omagic/Documents/GitHub/python4DSpolars/.venv/share/jupyter/kernels/python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} \ No newline at end of file diff --git a/workflow-style.ipynb b/workflow-style.ipynb index 3de114b..98b5469 100644 --- a/workflow-style.ipynb +++ b/workflow-style.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "95f0a171", + "id": "0", "metadata": {}, "source": [ "# Workflow: Style {#sec-workflow-style}\n", @@ -18,7 +18,7 @@ }, { "cell_type": "markdown", - "id": "c8111c37", + "id": "1", "metadata": {}, "source": [ "## Names\n", @@ -68,7 +68,7 @@ }, { "cell_type": "markdown", - "id": "b552c344", + "id": "2", "metadata": {}, "source": [ "## Whitespace\n", @@ -96,7 +96,7 @@ }, { "cell_type": "markdown", - "id": "54048af1", + "id": "3", "metadata": {}, "source": [ "## Code Comments\n", @@ -126,7 +126,7 @@ }, { "cell_type": "markdown", - "id": "861f3102", + "id": "4", "metadata": {}, "source": [ "## Line width and line continuation\n", @@ -149,7 +149,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f0f5bb37", + "id": "5", "metadata": {}, "outputs": [], "source": [ @@ -177,7 +177,7 @@ }, { "cell_type": "markdown", - "id": "1d6f3bf8", + "id": "6", "metadata": {}, "source": [ "And this is what _not_ to do:\n", @@ -191,7 +191,7 @@ }, { "cell_type": "markdown", - "id": "d016d530", + "id": "7", "metadata": {}, "source": [ "## Principles of Clean Code\n",