2025-plovdiv-data/exercises/tabular_join/tabular_join.ipynb
2025-09-24 13:09:53 +03:00

63 KiB
Raw Blame History

Exercise on Joins and anti-joins: add information from other tables

In [2]:
import pandas as pd

Load data from clinical trial

Data comes in two different files. The file predimed_records.csv file contains the clinical data for each patient, except which diet group they were assigned. The file predimed_mapping.csv contain the information of which patient was assigned to which diet group.

In [3]:
df = pd.read_csv('../../data/predimed_records.csv')
df.head()
Out[3]:
patient-id location-id sex age smoke bmi waist wth htn diab hyperchol famhist hormo p14 toevent event
0 436 4 Male 58 Former 33.53 122 0.753086 No No Yes No No 10 5.374401 Yes
1 1130 4 Male 77 Current 31.05 119 0.730061 Yes Yes No No No 10 6.097194 No
2 1131 4 Female 72 Former 30.86 106 0.654321 No Yes No Yes No 8 5.946612 No
3 1132 4 Male 71 Former 27.68 118 0.694118 Yes No Yes No No 8 2.907598 Yes
4 1111 2 Female 79 Never 35.94 129 0.806250 Yes No Yes No No 9 4.761123 No
In [4]:
info = pd.read_csv('../../data/predimed_mapping.csv')
info.head()
Out[4]:
location-id patient-id group
0 2 885 MedDiet + VOO
1 1 182 MedDiet + Nuts
2 1 971 MedDiet + Nuts
3 2 691 MedDiet + Nuts
4 2 632 Control

There were 5 different locations where the study was conducted, each one gave an identification number patient-id to each participant.

In [4]:
info['location-id'].unique()
Out[4]:
array([2, 1, 3, 4, 5])

1. Add diet information to the patients' records

  • For how many patients do we have clinical information? (i.e., rows in df)
In [7]:
## your code here

print(df.shape)

identifier = ['location-id', 'patient-id']
(6324, 16)
  • For how many patients do we have diet information? (i.e., rows in info)
In [8]:
## your code here

info.shape
Out[8]:
(6287, 3)

Perform the merge, keeping in mind that it only make sense to analyze patients with the diet information.

  • Which type of merge would you do?
  • For how many patients do we have full information (records and which diet they followed?
In [10]:
pd.merge?
Signature:
pd.merge(
    left: 'DataFrame | Series',
    right: 'DataFrame | Series',
    how: 'MergeHow' = 'inner',
    on: 'IndexLabel | AnyArrayLike | None' = None,
    left_on: 'IndexLabel | AnyArrayLike | None' = None,
    right_on: 'IndexLabel | AnyArrayLike | None' = None,
    left_index: 'bool' = False,
    right_index: 'bool' = False,
    sort: 'bool' = False,
    suffixes: 'Suffixes' = ('_x', '_y'),
    copy: 'bool | None' = None,
    indicator: 'str | bool' = False,
    validate: 'str | None' = None,
) -> 'DataFrame'
Docstring:
Merge DataFrame or named Series objects with a database-style join.

A named Series object is treated as a DataFrame with a single named column.

The join is done on columns or indexes. If joining columns on
columns, the DataFrame indexes *will be ignored*. Otherwise if joining indexes
on indexes or indexes on a column or columns, the index will be passed on.
When performing a cross merge, no column specifications to merge on are
allowed.

.. warning::

    If both key columns contain rows where the key is a null value, those
    rows will be matched against each other. This is different from usual SQL
    join behaviour and can lead to unexpected results.

Parameters
----------
left : DataFrame or named Series
right : DataFrame or named Series
    Object to merge with.
how : {'left', 'right', 'outer', 'inner', 'cross'}, default 'inner'
    Type of merge to be performed.

    * left: use only keys from left frame, similar to a SQL left outer join;
      preserve key order.
    * right: use only keys from right frame, similar to a SQL right outer join;
      preserve key order.
    * outer: use union of keys from both frames, similar to a SQL full outer
      join; sort keys lexicographically.
    * inner: use intersection of keys from both frames, similar to a SQL inner
      join; preserve the order of the left keys.
    * cross: creates the cartesian product from both frames, preserves the order
      of the left keys.
on : label or list
    Column or index level names to join on. These must be found in both
    DataFrames. If `on` is None and not merging on indexes then this defaults
    to the intersection of the columns in both DataFrames.
left_on : label or list, or array-like
    Column or index level names to join on in the left DataFrame. Can also
    be an array or list of arrays of the length of the left DataFrame.
    These arrays are treated as if they are columns.
right_on : label or list, or array-like
    Column or index level names to join on in the right DataFrame. Can also
    be an array or list of arrays of the length of the right DataFrame.
    These arrays are treated as if they are columns.
left_index : bool, default False
    Use the index from the left DataFrame as the join key(s). If it is a
    MultiIndex, the number of keys in the other DataFrame (either the index
    or a number of columns) must match the number of levels.
right_index : bool, default False
    Use the index from the right DataFrame as the join key. Same caveats as
    left_index.
sort : bool, default False
    Sort the join keys lexicographically in the result DataFrame. If False,
    the order of the join keys depends on the join type (how keyword).
suffixes : list-like, default is ("_x", "_y")
    A length-2 sequence where each element is optionally a string
    indicating the suffix to add to overlapping column names in
    `left` and `right` respectively. Pass a value of `None` instead
    of a string to indicate that the column name from `left` or
    `right` should be left as-is, with no suffix. At least one of the
    values must not be None.
copy : bool, default True
    If False, avoid copy if possible.

    .. note::
        The `copy` keyword will change behavior in pandas 3.0.
        `Copy-on-Write
        <https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html>`__
        will be enabled by default, which means that all methods with a
        `copy` keyword will use a lazy copy mechanism to defer the copy and
        ignore the `copy` keyword. The `copy` keyword will be removed in a
        future version of pandas.

        You can already get the future behavior and improvements through
        enabling copy on write ``pd.options.mode.copy_on_write = True``
indicator : bool or str, default False
    If True, adds a column to the output DataFrame called "_merge" with
    information on the source of each row. The column can be given a different
    name by providing a string argument. The column will have a Categorical
    type with the value of "left_only" for observations whose merge key only
    appears in the left DataFrame, "right_only" for observations
    whose merge key only appears in the right DataFrame, and "both"
    if the observation's merge key is found in both DataFrames.

validate : str, optional
    If specified, checks if merge is of specified type.

    * "one_to_one" or "1:1": check if merge keys are unique in both
      left and right datasets.
    * "one_to_many" or "1:m": check if merge keys are unique in left
      dataset.
    * "many_to_one" or "m:1": check if merge keys are unique in right
      dataset.
    * "many_to_many" or "m:m": allowed, but does not result in checks.

Returns
-------
DataFrame
    A DataFrame of the two merged objects.

See Also
--------
merge_ordered : Merge with optional filling/interpolation.
merge_asof : Merge on nearest keys.
DataFrame.join : Similar method using indices.

Examples
--------
>>> df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],
...                     'value': [1, 2, 3, 5]})
>>> df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],
...                     'value': [5, 6, 7, 8]})
>>> df1
    lkey value
0   foo      1
1   bar      2
2   baz      3
3   foo      5
>>> df2
    rkey value
0   foo      5
1   bar      6
2   baz      7
3   foo      8

Merge df1 and df2 on the lkey and rkey columns. The value columns have
the default suffixes, _x and _y, appended.

>>> df1.merge(df2, left_on='lkey', right_on='rkey')
  lkey  value_x rkey  value_y
0  foo        1  foo        5
1  foo        1  foo        8
2  bar        2  bar        6
3  baz        3  baz        7
4  foo        5  foo        5
5  foo        5  foo        8

Merge DataFrames df1 and df2 with specified left and right suffixes
appended to any overlapping columns.

>>> df1.merge(df2, left_on='lkey', right_on='rkey',
...           suffixes=('_left', '_right'))
  lkey  value_left rkey  value_right
0  foo           1  foo            5
1  foo           1  foo            8
2  bar           2  bar            6
3  baz           3  baz            7
4  foo           5  foo            5
5  foo           5  foo            8

Merge DataFrames df1 and df2, but raise an exception if the DataFrames have
any overlapping columns.

>>> df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False))
Traceback (most recent call last):
...
ValueError: columns overlap but no suffix specified:
    Index(['value'], dtype='object')

>>> df1 = pd.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]})
>>> df2 = pd.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]})
>>> df1
      a  b
0   foo  1
1   bar  2
>>> df2
      a  c
0   foo  3
1   baz  4

>>> df1.merge(df2, how='inner', on='a')
      a  b  c
0   foo  1  3

>>> df1.merge(df2, how='left', on='a')
      a  b  c
0   foo  1  3.0
1   bar  2  NaN

>>> df1 = pd.DataFrame({'left': ['foo', 'bar']})
>>> df2 = pd.DataFrame({'right': [7, 8]})
>>> df1
    left
0   foo
1   bar
>>> df2
    right
0   7
1   8

>>> df1.merge(df2, how='cross')
   left  right
0   foo      7
1   foo      8
2   bar      7
3   bar      8
File:      /usr/lib64/python3.13/site-packages/pandas/core/reshape/merge.py
Type:      function
In [12]:
## your code here


df_merged = df.merge(info, how = "left", on = identifier)
df_merged
Out[12]:
patient-id location-id sex age smoke bmi waist wth htn diab hyperchol famhist hormo p14 toevent event group
0 436 4 Male 58 Former 33.53 122 0.753086 No No Yes No No 10 5.374401 Yes Control
1 1130 4 Male 77 Current 31.05 119 0.730061 Yes Yes No No No 10 6.097194 No Control
2 1131 4 Female 72 Former 30.86 106 0.654321 No Yes No Yes No 8 5.946612 No MedDiet + VOO
3 1132 4 Male 71 Former 27.68 118 0.694118 Yes No Yes No No 8 2.907598 Yes MedDiet + Nuts
4 1111 2 Female 79 Never 35.94 129 0.806250 Yes No Yes No No 9 4.761123 No MedDiet + VOO
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
6319 120 5 Female 66 Never 28.51 104 0.645963 Yes No Yes Yes No 8 3.550992 No Control
6320 118 5 Male 80 Never 23.81 109 0.589189 Yes Yes Yes Yes No 8 2.743326 No Control
6321 351 3 Male 57 Former 25.24 100 0.571429 Yes No Yes No NaN 7 0.479124 No MedDiet + Nuts
6322 499 5 Female 71 Never 32.04 98 0.653333 Yes No Yes Yes No 6 2.587269 No MedDiet + VOO
6323 1257 5 Male 58 Former 24.43 93 0.547059 Yes Yes Yes No No 9 2.590007 No MedDiet + Nuts

6324 rows × 17 columns

2. Add location information to the patients' records

There were five locations where the study was conducted. Here is a DataFrame containing the information of each location.

  • Add a new column to the dataset that contains the city where each patient was recorded.
In [14]:
locations = pd.DataFrame.from_dict({'location-id': [1, 2, 3, 4, 5], 
                                    'City': ['Madrid', 'Valencia', 'Barcelona', 'Bilbao','Malaga']})
locations
Out[14]:
location-id City
0 1 Madrid
1 2 Valencia
2 3 Barcelona
3 4 Bilbao
4 5 Malaga
In [16]:
## your code here:

df_location = df_merged.merge(locations, on = "location-id")
df_location
Out[16]:
patient-id location-id sex age smoke bmi waist wth htn diab hyperchol famhist hormo p14 toevent event group City
0 436 4 Male 58 Former 33.53 122 0.753086 No No Yes No No 10 5.374401 Yes Control Bilbao
1 1130 4 Male 77 Current 31.05 119 0.730061 Yes Yes No No No 10 6.097194 No Control Bilbao
2 1131 4 Female 72 Former 30.86 106 0.654321 No Yes No Yes No 8 5.946612 No MedDiet + VOO Bilbao
3 1132 4 Male 71 Former 27.68 118 0.694118 Yes No Yes No No 8 2.907598 Yes MedDiet + Nuts Bilbao
4 1111 2 Female 79 Never 35.94 129 0.806250 Yes No Yes No No 9 4.761123 No MedDiet + VOO Valencia
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
6319 120 5 Female 66 Never 28.51 104 0.645963 Yes No Yes Yes No 8 3.550992 No Control Malaga
6320 118 5 Male 80 Never 23.81 109 0.589189 Yes Yes Yes Yes No 8 2.743326 No Control Malaga
6321 351 3 Male 57 Former 25.24 100 0.571429 Yes No Yes No NaN 7 0.479124 No MedDiet + Nuts Barcelona
6322 499 5 Female 71 Never 32.04 98 0.653333 Yes No Yes Yes No 6 2.587269 No MedDiet + VOO Malaga
6323 1257 5 Male 58 Former 24.43 93 0.547059 Yes Yes Yes No No 9 2.590007 No MedDiet + Nuts Malaga

6324 rows × 18 columns

3. Remove drops from table

Some patients drop from the study early on and they should be removed from our analysis. Their IDS are stored in file dropped.csv.

  1. Load the list of patients who droped, from dropped.csv
  2. Use an anti-join to remove them from the table
  3. How many patients (rows) are left in the data?
In [17]:
dropped = pd.read_csv('dropped.csv')
In [18]:
dropped.shape
Out[18]:
(42, 2)
In [19]:
dropped.head()
Out[19]:
location-id patient-id
0 1 217
1 1 1147
2 1 1170
3 1 627
4 4 541
In [31]:
# your code here

df_removed = df_location.merge(dropped, how = "outer", on = identifier, indicator = True)

df_removed = df_removed.loc[df_removed["_merge"] == "left_only",].drop(["_merge"], axis = 1)
df_removed
Out[31]:
patient-id location-id sex age smoke bmi waist wth htn diab hyperchol famhist hormo p14 toevent event group City
0 1 1 Female 77 Never 25.92 94 0.657343 Yes No Yes Yes No 9 5.538672 No MedDiet + VOO Madrid
1 2 1 Female 68 Never 34.85 150 0.949367 Yes No Yes Yes NaN 10 3.063655 No MedDiet + Nuts Madrid
2 3 1 Female 66 Never 37.50 120 0.750000 Yes Yes No No No 6 5.590691 No MedDiet + Nuts Madrid
3 4 1 Female 77 Never 29.26 93 0.628378 Yes Yes No No No 6 5.456537 No MedDiet + VOO Madrid
4 5 1 Female 60 Never 30.02 104 0.662420 Yes No Yes No No 9 2.746064 No Control Madrid
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
6319 1253 5 Male 79 Never 25.28 105 0.640244 Yes No Yes No No 8 5.828884 No MedDiet + VOO Malaga
6320 1254 5 Male 62 Former 27.10 104 0.594286 Yes No Yes Yes No 9 5.067762 No MedDiet + Nuts Malaga
6321 1255 5 Female 65 Never 35.02 103 0.686667 Yes No Yes No No 10 1.993155 No MedDiet + VOO Malaga
6322 1256 5 Male 61 Never 28.42 94 0.576687 Yes Yes No No No 9 2.039699 No MedDiet + Nuts Malaga
6323 1257 5 Male 58 Former 24.43 93 0.547059 Yes Yes Yes No No 9 2.590007 No MedDiet + Nuts Malaga

6282 rows × 18 columns

4. Save final result in processed_data_predimed.csv

  1. Using the .to_csv method of Pandas DataFrames
In [32]:
fname = 'processed_data_predimed.csv'

#  your code here

df_removed.to_csv(fname)