EDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "389d40d6-22b0-4fc1-bd41-8ff074e724bc",
   "metadata": {},
   "source": [
    "# Exploratory data analisys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74080397-b020-4a5a-83f2-a92d7bdf0747",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3114b604-2950-4ebb-8c12-d59cfd3a1c97",
   "metadata": {},
   "source": [
    "## H&M recommendations"
   ]
  },
  {
   "cell_type": "raw",
   "id": "91fe17ce-b145-4a64-b7f9-f05b584f6fc2",
   "metadata": {},
   "source": [
    "Article file contains detailed metadata for each article_id available for purchase such as a type of product, a color, a product group and other features.\n",
    "\n",
    "Article data description:\n",
    "    article_id : identifier of an article\n",
    "    colour_group_code, colour_group_name : group of color and its name\n",
    "    department_no, department_name: identifier of a department and its name\n",
    "    detail_desc: details\n",
    "    garment_group_no, garment_group_name: identifier of every garment and its name\n",
    "    graphical_appearance_no, graphical_appearance_name : group of item appearance and its name\n",
    "    index_code, index_name identifier of an index and its name\n",
    "    index_group_no, index_group_name: group of indeces and its name\n",
    "    product_code, prod_name : identifier of every product and its name\n",
    "    product_type, product_type_name : group of product_code and its name\n",
    "    perceived_colour_value_id, perceived_colour_value_name, perceived_colour_master_id, perceived_colour_master_name : additional     \n",
    "                                                                                                                       color info\n",
    "    section_no, section_name: identifier of every section and its name\n",
    " \n",
    "Customers data description:\n",
    "    Active : 1 or missed\n",
    "    age : age of a customer\n",
    "    customer_id: identifier of a customer\n",
    "    club_member_status : status in club\n",
    "    fashion_news_frequency : how often H&M may send news to customer\n",
    "    FN : 1 or missed\n",
    "    postal_code : postal code of a customer\n",
    "\n",
    "Transactions data description:\n",
    "    article_id : identifier of every article (in articles table)\n",
    "    customer_id : identifier of every customer (in customers table)\n",
    "    price : price of purchase\n",
    "    sales_channel_id : 1 or 2\n",
    "    t_dat : identifier of every customer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bd812b5-22a9-4f07-a7f8-942da25650be",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = pd.read_csv('articles.csv.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aa0a0ca-ae31-4ef8-8101-abf60d2df708",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9499c9af-88e1-48b1-a3cd-01c71302c9c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles.index_name.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42895878-40eb-4128-ba79-015e33dd1a73",
   "metadata": {},
   "outputs": [],
   "source": [
    "_, ax = plt.subplots(figsize=(15, 7))\n",
    "ax = sns.histplot(data=articles, y='index_name', color='green')\n",
    "ax.set_xlabel('Count by index name')\n",
    "ax.set_ylabel('Index name')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4340c227-22fc-4024-8ed6-13e5da1bcc08",
   "metadata": {},
   "source": [
    "**To identify indexes accounting for a largest and smallest shares of all items.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbd707b3-11a9-4284-bc46-f49409b51a7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "_, ax = plt.subplots(figsize=(15, 7))\n",
    "ax = sns.histplot(data=articles, y='garment_group_name', color='orange', \n",
    "                  hue='index_group_name', multiple='stack')\n",
    "ax.set_xlabel('Count by garment group')\n",
    "ax.set_ylabel('Garment group')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d19d6fe-9b21-4040-bee1-2d6533605df3",
   "metadata": {},
   "source": [
    "**To identify most popular garments within index.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c6053e6-bebf-44a5-814e-e6986d8387c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = articles.groupby(['product_group_name'])['product_type_name'].nunique()\n",
    "df = pd.DataFrame({'Product group': temp.index,\n",
    "                   'Product types number': temp.values\n",
    "                  })\n",
    "df = df.sort_values(['Product types number'], ascending=False)\n",
    "plt.figure(figsize = (8,6))\n",
    "plt.title('Number of Product types per each Product group')\n",
    "s = sns.barplot(x = 'Product group', y='Product types number', data=df)\n",
    "s.set_xticklabels(s.get_xticklabels(),rotation=90)\n",
    "locs, labels = plt.xticks()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ee5167d-e11a-4911-8322-b8fd06f01816",
   "metadata": {},
   "source": [
    "**To identify number of Product types per each Product group** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "987ae983-21bb-4f02-aabb-f903f7c33258",
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = articles.groupby(['perceived_colour_master_name'])['article_id'].nunique()\n",
    "df = pd.DataFrame({'Perceived colour master name': temp.index,\n",
    "                   'Num of articles': temp.values\n",
    "                  })\n",
    "df = df.sort_values(['Num of articles'], ascending=False)\n",
    "plt.figure(figsize = (12,6))\n",
    "plt.title(f'Number of Articles per each Perceived colour master Name')\n",
    "s = sns.barplot(x = 'Perceived colour master name', y='Num of articles', data=df)\n",
    "s.set_xticklabels(s.get_xticklabels(),rotation=90)\n",
    "locs, labels = plt.xticks()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e1a7d2c-da03-490e-bcf0-81f831d5ad76",
   "metadata": {},
   "source": [
    "**To identify the most popular colors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b00b6a80-2466-4671-8f82-b9f0b649046c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud, STOPWORDS\n",
    "stopwords = set(STOPWORDS)\n",
    "\n",
    "def show_wordcloud(data, title = None):\n",
    "    wordcloud = WordCloud(\n",
    "        background_color='white',\n",
    "        stopwords=stopwords,\n",
    "        max_words=200,\n",
    "        max_font_size=40, \n",
    "        scale=5,\n",
    "        random_state=1\n",
    "    ).generate(str(data))\n",
    "\n",
    "    fig = plt.figure(1, figsize=(10,10))\n",
    "    plt.axis('off')\n",
    "    if title: \n",
    "        fig.suptitle(title, fontsize=14)\n",
    "        fig.subplots_adjust(top=2.3)\n",
    "\n",
    "    plt.imshow(wordcloud)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b6d71f9-df27-442d-9516-752797a42f00",
   "metadata": {},
   "outputs": [],
   "source": [
    "show_wordcloud(articles['prod_name'], 'Wordcloud from product name')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abe20aa6-7d86-4609-8cc2-d73d770e3340",
   "metadata": {},
   "source": [
    "**To identify frequent words vissually.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bdf239c-a564-45de-a28e-ac490741ba2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "show_wordcloud(articles['detail_desc'], 'Wordcloud from detailed description of items')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4589d29-a6d4-4ed8-bfd8-066cc57e7cea",
   "metadata": {},
   "outputs": [],
   "source": [
    "customers = pd.read_csv('customers.csv.zip')\n",
    "customers[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6adf755-5476-48cd-95cf-d6fe823e1ac0",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_style('darkgrid')\n",
    "\n",
    "_, ax = plt.subplots(figsize=(10,5))\n",
    "ax = sns.histplot(data=customers, x='age', bins=50, color='green')\n",
    "ax.set_xlabel('Distribution of the customers age')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31e0b178-b5a0-498c-b03c-673324129b1f",
   "metadata": {},
   "source": [
    "**To identify the most common age.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13c20667-1cd5-4a0a-8f5b-b12569207d17",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions = pd.read_csv('transactions_train.csv.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "000fe3ed-f700-437f-973a-c43d3b4ab111",
   "metadata": {},
   "outputs": [],
   "source": [
    "transactions[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b2059cc-d764-48ab-9c1e-a4139b6d53d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "merged = transactions[['customer_id', 'article_id', \n",
    "                                   'price', 't_dat']].merge(articles[['article_id', 'prod_name', \n",
    "                                                                      'product_type_name', 'product_group_name', \n",
    "                                                                      'index_name']], on='article_id', how='left')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ade8e206-e05f-4805-a4d1-343b6040ad65",
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_index = merged[['index_name', 'price']].groupby('index_name').mean()\n",
    "sns.set_style('darkgrid')\n",
    "_, ax = plt.subplots(figsize=(10,5))\n",
    "ax = sns.barplot(x=articles_index.price, y=articles_index.index, color='green', alpha=0.8)\n",
    "ax.set_xlabel('Price by index')\n",
    "ax.set_ylabel('Index')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ca1e9d4-7165-471c-b6a3-534f66dbe494",
   "metadata": {},
   "source": [
    "**To identify indexes with the highest and lowest mean prices.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e5c0eb6-a3e1-4c10-ac43-bdcdad0810ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "grouped = transactions.sample(200000).groupby(['t_dat', 'sales_channel_id'])['article_id'].count().reset_index()\n",
    "grouped['t_dat'] = grouped['t_dat'].apply(lambda x: datetime.strptime(x, '%Y-%m-%d'))\n",
    "\n",
    "grouped.columns = ['Date', 'Sales Channel Id', \"Transactions\"]\n",
    "\n",
    "_, ax = plt.subplots(1, 1, figsize=(16,6))\n",
    "g1 = ax.plot(grouped.loc[grouped['Sales Channel Id']==1, 'Date'], \n",
    "             grouped.loc[grouped['Sales Channel Id']==1, 'Transactions'], label='Sales Channel 1')\n",
    "\n",
    "g2 = ax.plot(grouped.loc[grouped[\"Sales Channel Id\"]==2, 'Date'], \n",
    "             grouped.loc[grouped[\"Sales Channel Id\"]==2, 'Transactions'], label='Sales Channel 2')\n",
    "\n",
    "plt.xlabel('Date')\n",
    "plt.ylabel('Num of transactions')\n",
    "ax.legend()\n",
    "plt.title(f'Number of transactions per day, grouped by Sales channel (200k sample)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a94e0106-bc6d-4bdf-af8f-8580d7d6d7f8",
   "metadata": {},
   "source": [
    "## AMEX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d1e4dbb-c9a7-46b9-a1a1-41a2245e9fc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_time_series(prefix='D', cols=None, display_ct=32):\n",
    "    \n",
    "    # DETERMINE WHICH COLUMNS TO PLOT\n",
    "    if cols is not None and len(cols)==0: cols = None\n",
    "    if cols is None:\n",
    "        COLS = df.columns[2:-1]\n",
    "        COLS = np.sort( [int(x[2:]) for x in COLS if x[0]==prefix] )\n",
    "        COLS = [f'{prefix}_{x}' for x in COLS]\n",
    "        print('#'*25)\n",
    "        print(f'Plotting all {len(COLS)} columns with prefix {prefix}')\n",
    "        print('#'*25)\n",
    "    else:\n",
    "        COLS = [f'{prefix}_{x}' for x in cols]\n",
    "        print('#'*25)\n",
    "        print(f'Plotting {len(COLS)} columns with prefix {prefix}')\n",
    "        print('#'*25)\n",
    "\n",
    "    # ITERATE COLUMNS\n",
    "    for c in COLS:\n",
    "\n",
    "        # CONVERT DATAFRAME INTO SERIES WITH COLUMN\n",
    "        tmp = df[['customer_ID','S_2',c,'target']].copy()\n",
    "        tmp2 = tmp.groupby(['customer_ID','target'])[['S_2',c]].agg(list).reset_index()\n",
    "        tmp3 = tmp2.loc[tmp2.target==1]\n",
    "        tmp4 = tmp2.loc[tmp2.target==0]\n",
    "\n",
    "        # FORMAT PLOT\n",
    "        spec = gridspec.GridSpec(ncols=2, nrows=1,\n",
    "                             width_ratios=[3, 1], wspace=0.1,\n",
    "                             hspace=0.5, height_ratios=[1])\n",
    "        fig = plt.figure(figsize=(20,10))\n",
    "        ax0 = fig.add_subplot(spec[0])\n",
    "\n",
    "        # PLOT 32 DEFAULT CUSTOMERS AND 32 NON-DEFAULT CUSTOMERS\n",
    "        t0 = []; t1 = []\n",
    "        for k in range(display_ct):\n",
    "            try:\n",
    "                # PLOT DEFAULTING CUSTOMERS\n",
    "                row = tmp3.iloc[k]\n",
    "                ax0.plot(row.S_2,row[c],'-o',color='blue')\n",
    "                t1 += row[c]\n",
    "                # PLOT NON-DEFAULT CUSTOMERS\n",
    "                row = tmp4.iloc[k]\n",
    "                ax0.plot(row.S_2,row[c],'-o',color='orange')\n",
    "                t0 += row[c]\n",
    "            except:\n",
    "                pass\n",
    "        plt.title(f'Feature {c} (Key: BLUE=DEFAULT, orange=no default)',size=18)\n",
    "\n",
    "        # PLOT HISTOGRAMS\n",
    "        ax1 = fig.add_subplot(spec[1])\n",
    "        try:\n",
    "            # COMPUTE BINS\n",
    "            t = t0+t1; mn = np.nanmin(t); mx = np.nanmax(t)\n",
    "            if mx==mn:\n",
    "                mx += 0.01; mn -= 0.01\n",
    "            bins = np.arange(mn,mx+(mx-mn)/20,(mx-mn)/20 )\n",
    "            # PLOT HISTOGRAMS\n",
    "            if np.sum(np.isnan(t1))!=len(t1):\n",
    "                ax1.hist(t1,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='blue')\n",
    "            if np.sum(np.isnan(t0))!=len(t0):\n",
    "                ax1.hist(t0,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='orange')\n",
    "        except:\n",
    "            pass\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06105d69-3310-41a1-8190-798b5233ff13",
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import gridspec\n",
    "\n",
    "def plot_customers(data, col, date_col, display_ct=32):\n",
    "    \n",
    "        tmp = data.copy()\n",
    "        tmp2 = tmp.groupby(['customer_ID','target'])[[date_col, col]].agg(list).reset_index()\n",
    "        tmp3 = tmp2.loc[tmp2.target==1]\n",
    "        tmp4 = tmp2.loc[tmp2.target==0]\n",
    "\n",
    "        # FORMAT PLOT\n",
    "        spec = gridspec.GridSpec(ncols=2, nrows=1,\n",
    "                             width_ratios=[3, 1], wspace=0.1,\n",
    "                             hspace=0.5, height_ratios=[1])\n",
    "        fig = plt.figure(figsize=(20,10))\n",
    "        ax0 = fig.add_subplot(spec[0])\n",
    "\n",
    "        # PLOT 32 DEFAULT CUSTOMERS AND 32 NON-DEFAULT CUSTOMERS\n",
    "        t0 = []\n",
    "        t1 = []\n",
    "        for k in range(display_ct):\n",
    "            try:\n",
    "                # PLOT DEFAULTING CUSTOMERS\n",
    "                row = tmp3.iloc[k]\n",
    "                ax0.plot(row[date_col], row[col], '-o', color='blue')\n",
    "                t1 += row[col]\n",
    "                # PLOT NON-DEFAULT CUSTOMERS\n",
    "                row = tmp4.iloc[k]\n",
    "                ax0.plot(row[date_col], row[col],'-o', color='orange')\n",
    "                t0 += row[col]\n",
    "            except:\n",
    "                pass\n",
    "        plt.title(f'Feature {col} (Key: blue=default, orange=non default)',size=18)\n",
    "\n",
    "        # PLOT HISTOGRAMS\n",
    "        ax1 = fig.add_subplot(spec[1])\n",
    "        try:\n",
    "            # COMPUTE BINS\n",
    "            t = t0 + t1\n",
    "            mn = np.nanmin(t)\n",
    "            mx = np.nanmax(t)\n",
    "            if mx == mn:\n",
    "                mx += 0.01\n",
    "                mn -= 0.01\n",
    "            bins = np.arange(mn, mx+(mx-mn)/20, (mx-mn)/20 )\n",
    "            # PLOT HISTOGRAMS\n",
    "            if np.sum(np.isnan(t1))!=len(t1):\n",
    "                ax1.hist(t1,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='blue')\n",
    "            if np.sum(np.isnan(t0))!=len(t0):\n",
    "                ax1.hist(t0,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='orange')\n",
    "        except:\n",
    "            pass\n",
    "        \n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64e9e658-691e-4c2c-a7d4-6926b982d99b",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.read_csv('train_data.csv.zip', nrows=100000)\n",
    "\n",
    "train_data.S_2 = pd.to_datetime(train_data.S_2)\n",
    "\n",
    "train_labels = pd.read_csv('train_labels.csv.zip')\n",
    "\n",
    "train_data = train_data.merge(train_labels,on='customer_ID', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3779b80b-33d2-46f4-99fb-deb4846d780a",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_customers(train_data, 'D_41', 'S_2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c585cb1c-6cad-443a-a46c-1e11494e15c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_customers(train_data, 'P_2', 'S_2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc303edf-29e2-4495-87f2-8a1074779f63",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plot_customers(train_data, 'D_96', 'S_2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "381ad4c2-4c79-4ac8-ab08-704fea6d77ff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ce247f-3f72-4cfa-a19e-63f2c65adfdb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "raw",
   "id": "41be335d-93d8-4e24-9b32-74bd64228fd5",
   "metadata": {},
   "source": [
    "Materials:\n",
    "https://www.kaggle.com/code/vanguarde/h-m-eda-first-look/notebook\n",
    "https://www.kaggle.com/code/cdeotte/time-series-eda/notebook#Load-Train-Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5f1ce4b-009d-4014-bd4c-a594cd6dc50f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}