{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Working with NumPy Arrays\n", "=========================\n", "\n", "``````{admonition} Overview\n", ":class: overview\n", "\n", "Questions:\n", "\n", "* When should I use NumPy arrays instead of Pandas dataframes?\n", "\n", "* What does the shape of a NumPy array mean?\n", "\n", "* How can I reshape arrays?\n", "\n", "Objectives:\n", "\n", "* Convert data from a Pandas dataframe to a NumPy array.\n", "\n", "* Use the `reshape` function to reshape a NumPy array.\n", "\n", "``````" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pandas dataframes are built on top of a data structure known as the NumPy Array. If you completed the first MolSSI Python scripting workshop, you are already familiar with some properties of NumPy arrays.\n", "\n", "In general, you should use pandas dataframe when working with data which is:\n", " - Two dimensional (rows and columns).\n", " - Labeled.\n", " - Mixed type.\n", " - Something for which you would like to be able to easily get statistics.\n", " \n", "You should work with NumPy arrays when:\n", " - You have higher dimensional data (collection of two dimensional arrays).\n", " - You need to perform advanced mathematics like linear algebra.\n", " - You are using a library which requires NumPy arrays (scikitlearn)." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "For our discussion of NumPy, we are first going to load our data into a pandas dataframe then convert this data into a NumPy array." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Compound IDESOL predicted log solubility in mols per litreMinimum DegreeMolecular WeightNumber of H-Bond DonorsNumber of RingsNumber of Rotatable BondsPolar Surface Areameasured log solubility in mols per litresmiles
0Amigdalin-0.9741457.432737202.32-0.77OCC3OC(OCC2OC(OC(C#N)c1ccccc1)C(O)C(O)C2O)C(O)...
1Fenfuram-2.8851201.22512242.24-3.30Cc1occc1C(=O)Nc2ccccc2
2citral-2.5791152.23700417.07-2.06CC(C)=CCCC(C)=CC(=O)
3Picene-6.6182278.3540500.00-7.87c1ccc2c(c1)ccc3c2ccc4c5ccccc5ccc43
4Thiophene-2.232284.1430100.00-1.33c1ccsc1
\n", "
" ], "text/plain": [ " Compound ID ESOL predicted log solubility in mols per litre \\\n", "0 Amigdalin -0.974 \n", "1 Fenfuram -2.885 \n", "2 citral -2.579 \n", "3 Picene -6.618 \n", "4 Thiophene -2.232 \n", "\n", " Minimum Degree Molecular Weight Number of H-Bond Donors Number of Rings \\\n", "0 1 457.432 7 3 \n", "1 1 201.225 1 2 \n", "2 1 152.237 0 0 \n", "3 2 278.354 0 5 \n", "4 2 84.143 0 1 \n", "\n", " Number of Rotatable Bonds Polar Surface Area \\\n", "0 7 202.32 \n", "1 2 42.24 \n", "2 4 17.07 \n", "3 0 0.00 \n", "4 0 0.00 \n", "\n", " measured log solubility in mols per litre \\\n", "0 -0.77 \n", "1 -3.30 \n", "2 -2.06 \n", "3 -7.87 \n", "4 -1.33 \n", "\n", " smiles \n", "0 OCC3OC(OCC2OC(OC(C#N)c1ccccc1)C(O)C(O)C2O)C(O)... \n", "1 Cc1occc1C(=O)Nc2ccccc2 \n", "2 CC(C)=CCCC(C)=CC(=O) \n", "3 c1ccc2c(c1)ccc3c2ccc4c5ccccc5ccc43 \n", "4 c1ccsc1 " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_csv(\"data/delaney-processed.csv\")\n", "df.head()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "We will rename the columns again for convenience in referencing." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "df.rename( columns = {\n", " \"ESOL predicted log solubility in mols per litre\": \"ESOL solubility (mol/L)\",\n", " \"measured log solubility in mols per litre\" : \"measured solubility (mol/L)\"\n", "}, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Converting a Pandas DataFrame to a NumPy Array\n", "\n", "To convert a dataframe to a numpy array, use the function `.to_numpy()`. You will notice that both the numpy array and the pandas dataframe have the same shape after conversion." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[['Amigdalin' -0.974 1 ... 202.32 -0.77\n", " 'OCC3OC(OCC2OC(OC(C#N)c1ccccc1)C(O)C(O)C2O)C(O)C(O)C3O ']\n", " ['Fenfuram' -2.885 1 ... 42.24 -3.3 'Cc1occc1C(=O)Nc2ccccc2']\n", " ['citral' -2.579 1 ... 17.07 -2.06 'CC(C)=CCCC(C)=CC(=O)']\n", " ...\n", " ['Thiometon' -3.323 1 ... 18.46 -3.091 'CCSCCSP(=S)(OC)OC']\n", " ['2-Methylbutane' -2.245 1 ... 0.0 -3.18 'CCC(C)C']\n", " ['Stirofos' -4.32 1 ... 44.760000000000005 -4.522\n", " 'COP(=O)(OC)OC(=CCl)c1cc(Cl)c(Cl)cc1Cl']]\n" ] } ], "source": [ "np_array = df.to_numpy()\n", "\n", "print(np_array)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1128, 10)" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.shape" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1128, 10)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np_array.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At the end of the numpy array, you will see `dtype=object`. When we converted the dataframe to a NumPy array, pandas attempted to give the dataframe the simplest type possible. Since we had a mix of strings and numbers, the assigned type of the numpy array is object. \n", "\n", "Note that we have both strings and numbers in the array." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dtype('O')" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np_array.dtype" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Amigdalin\n", "\n", "-0.974\n", "\n" ] } ], "source": [ "print(np_array[0, 0])\n", "print(type(np_array[0, 0]))\n", "\n", "print(np_array[0, 1])\n", "print(type(np_array[0, 1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reshaping Arrays\n", "\n", "NumPy arrays have shapes just like Pandas DataFrames. One major difference is that they can have more dimensions than two. NumPy arrays are n-dimensional, meaning they can have any number of dimensions. Unlike pandas dataframes, NumPy arrays can be reshaped.\n", "\n", "When we fit our data using scikit-learn later in this workshop, we will need to make sure our arrays are the proper shape. Unlike pandas dataframes, NumPy arrays can have more than two dimensions. Let's see how we might reshape an array." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "numbers = df[[\"ESOL solubility (mol/L)\", \"measured solubility (mol/L)\"]].to_numpy()" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1128, 2)" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "numbers.shape" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dtype('float64')" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "numbers.dtype" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To change the shape of an array, use the function `.reshape`. Inside the reshape function, you list the size you would like for each dimension of the array to be. We could shape the `numbers` array we just created to be three dimensional rather than two dimensional. We might want rows with one column. For the third dimension you can think of it as being two columns stacked on one another.\n", "\n", "If you do not care what the shape of one of the dimensions is when doing a reshape, you can specify a `-1` for that dimension (or `unknown`). It essentially means to make that dimension whatever size it needs to be to fit the other specified dimensions." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(1128, 1, 2)\n" ] } ], "source": [ "# reshape so each \"category\" is its own two dimensional array\n", "numbers_reshaped = numbers.reshape(-1, 1, 2)\n", "print(numbers_reshaped.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can slice into our reshaped array." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[-0.974],\n", " [-2.885],\n", " [-2.579],\n", " ...,\n", " [-3.323],\n", " [-2.245],\n", " [-4.32 ]])" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get ESOL values\n", "numbers_reshaped[:, :, 0]" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([-0.974, -2.885, -2.579, ..., -3.323, -2.245, -4.32 ])" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "numbers[:, 0]" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[-0.77 ],\n", " [-3.3 ],\n", " [-2.06 ],\n", " ...,\n", " [-3.091],\n", " [-3.18 ],\n", " [-4.522]])" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get measured values\n", "numbers_reshaped[:, :, 1]" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['Amigdalin' 'Fenfuram' 'citral' ... 'Thiometon' '2-Methylbutane'\n", " 'Stirofos']\n", "[['Amigdalin']\n", " ['Fenfuram']\n", " ['citral']\n", " ...\n", " ['Thiometon']\n", " ['2-Methylbutane']\n", " ['Stirofos']]\n" ] } ], "source": [ "# By default pandas Series are one-dimensional. We will have to reshape the arrays sometimes.\n", "names = df[\"Compound ID\"].to_numpy()\n", "print(names)\n", "names = names.reshape(-1, 1)\n", "print(names)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "``````{admonition} Key Points\n", ":class: key\n", "\n", "* Pandas dataframes are built on top of NumPy arrays.\n", "\n", "* NumPy arrays have all the same data type.\n", "\n", "* NumPy is used for numerical applications.\n", "\n", "``````" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "molssi-training", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:40) [GCC 10.4.0]" }, "vscode": { "interpreter": { "hash": "16d4a7bb199d969b1271ebe46f77414b0d9cd01b3c3983c2b2742fc6cd4503d3" } } }, "nbformat": 4, "nbformat_minor": 4 }