Data handling

Converting dataset coordinates from flat list to grid

By default measurement control returnds data in a sparse format. The resulting dataset has a single dimension, and all settables and gettables share it, for example:

from pathlib import Path

import numpy as np
from qcodes import ManualParameter, Parameter, validators

from quantify_core.data.handling import set_datadir, to_gridded_dataset
from quantify_core.measurement import MeasurementControl

set_datadir(Path.home() / "quantify-data")

time_a = ManualParameter(
    name="time_a",
    label="Time A",
    unit="s",
    vals=validators.Numbers(),
    initial_value=1,
)
time_b = ManualParameter(
    name="time_b",
    label="Time B",
    unit="s",
    vals=validators.Numbers(),
    initial_value=1,
)
signal = Parameter(
    name="sig_a",
    label="Signal A",
    unit="V",
    get_cmd=lambda: np.exp(time_a()) + 0.5 * np.exp(time_b()),
)

meas_ctrl = MeasurementControl("meas_ctrl")
meas_ctrl.settables([time_a, time_b])
meas_ctrl.gettables(signal)
meas_ctrl.setpoints_grid([np.linspace(0, 5, 10), np.linspace(5, 0, 12)])
dset = meas_ctrl.run("2D-single-float-valued-settable-gettable")
dset

Starting iterative measurement...

<xarray.Dataset>
Dimensions:  (dim_0: 120)
Coordinates:
    x0       (dim_0) float64 0.0 0.5556 1.111 1.667 ... 3.333 3.889 4.444 5.0
    x1       (dim_0) float64 5.0 5.0 5.0 5.0 5.0 5.0 ... 0.0 0.0 0.0 0.0 0.0 0.0
Dimensions without coordinates: dim_0
Data variables:
    y0       (dim_0) float64 75.21 75.95 77.24 79.5 ... 28.53 49.36 85.65 148.9
Attributes:
    tuid:                             20231215-162554-320-4a92f9
    name:                             2D-single-float-valued-settable-gettable
    grid_2d:                          True
    grid_2d_uniformly_spaced:         True
    1d_2_settables_uniformly_spaced:  False
    xlen:                             10
    ylen:                             12

xarray.Dataset

• Dimensions:
  • dim_0: 120
• Coordinates: (2)
  • x0
    (dim_0)
    float64
    0.0 0.5556 1.111 ... 4.444 5.0

    name :

    time_a

    long_name :

    Time A

    units :

    s

    batched :

    False

    array([0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ,
           0.        , 0.55555556, 1.11111111, 1.66666667, 2.22222222,
           2.77777778, 3.33333333, 3.88888889, 4.44444444, 5.        ])

  • x1
    (dim_0)
    float64
    5.0 5.0 5.0 5.0 ... 0.0 0.0 0.0 0.0

    name :

    time_b

    long_name :

    Time B

    units :

    s

    batched :

    False

    array([5.        , 5.        , 5.        , 5.        , 5.        ,
           5.        , 5.        , 5.        , 5.        , 5.        ,
           4.54545455, 4.54545455, 4.54545455, 4.54545455, 4.54545455,
           4.54545455, 4.54545455, 4.54545455, 4.54545455, 4.54545455,
           4.09090909, 4.09090909, 4.09090909, 4.09090909, 4.09090909,
           4.09090909, 4.09090909, 4.09090909, 4.09090909, 4.09090909,
           3.63636364, 3.63636364, 3.63636364, 3.63636364, 3.63636364,
           3.63636364, 3.63636364, 3.63636364, 3.63636364, 3.63636364,
           3.18181818, 3.18181818, 3.18181818, 3.18181818, 3.18181818,
           3.18181818, 3.18181818, 3.18181818, 3.18181818, 3.18181818,
           2.72727273, 2.72727273, 2.72727273, 2.72727273, 2.72727273,
           2.72727273, 2.72727273, 2.72727273, 2.72727273, 2.72727273,
           2.27272727, 2.27272727, 2.27272727, 2.27272727, 2.27272727,
           2.27272727, 2.27272727, 2.27272727, 2.27272727, 2.27272727,
           1.81818182, 1.81818182, 1.81818182, 1.81818182, 1.81818182,
           1.81818182, 1.81818182, 1.81818182, 1.81818182, 1.81818182,
           1.36363636, 1.36363636, 1.36363636, 1.36363636, 1.36363636,
           1.36363636, 1.36363636, 1.36363636, 1.36363636, 1.36363636,
           0.90909091, 0.90909091, 0.90909091, 0.90909091, 0.90909091,
           0.90909091, 0.90909091, 0.90909091, 0.90909091, 0.90909091,
           0.45454545, 0.45454545, 0.45454545, 0.45454545, 0.45454545,
           0.45454545, 0.45454545, 0.45454545, 0.45454545, 0.45454545,
           0.        , 0.        , 0.        , 0.        , 0.        ,
           0.        , 0.        , 0.        , 0.        , 0.        ])

• Data variables: (1)
  • y0
    (dim_0)
    float64
    75.21 75.95 77.24 ... 85.65 148.9

    name :

    sig_a

    long_name :

    Signal A

    units :

    V

    batched :

    False

    array([ 75.20657955,  75.94948855,  77.24431133,  79.5010696 ,
            83.4343939 ,  90.28982022, 102.23820445, 123.06315083,
           159.35913727, 222.61973865,  48.10161857,  48.84452756,
            50.13935034,  52.39610862,  56.32943292,  63.18485924,
            75.13324346,  95.95818984, 132.25417628, 195.51477767,
            30.89711269,  31.64002169,  32.93484447,  35.19160275,
            39.12492705,  45.98035337,  57.92873759,  78.75368397,
           115.04967041, 178.3102718 ,  19.97678625,  20.71969525,
            22.01451803,  24.2712763 ,  28.2046006 ,  35.06002692,
            47.00841114,  67.83335752, 104.12934397, 167.38994535,
            13.04525735,  13.78816635,  15.08298912,  17.3397474 ,
            21.2730717 ,  28.12849802,  40.07688224,  60.90182862,
            97.19781506, 160.45841645,   8.64556351,   9.38847251,
            10.68329529,  12.94005356,  16.87337787,  23.72880419,
            35.67718841,  56.50213479,  92.79812123, 156.05872262,
             5.85291761,   6.5958266 ,   7.89064938,  10.14740766,
            14.08073196,  20.93615828,  32.8845425 ,  53.70948888,
            90.00547532, 153.26607671,   4.08032354,   4.82323254,
             6.11805532,   8.37481359,  12.30813789,  19.16356421,
            31.11194844,  51.93689482,  88.23288126, 151.49348264,
             2.95519353,   3.69810253,   4.99292531,   7.24968358,
            11.18300789,  18.03843421,  29.98681843,  50.81176481,
            87.10775125, 150.36835264,   2.24103254,   2.98394154,
             4.27876432,   6.53552259,  10.46884689,  17.32427321,
            29.27265744,  50.09760382,  86.39359026, 149.65419164,
             1.78772855,   2.53063755,   3.82546033,   6.0822186 ,
            10.0155429 ,  16.87096922,  28.81935345,  49.64429983,
            85.94028627, 149.20088765,   1.5       ,   2.242909  ,
             3.53773178,   5.79449005,   9.72781435,  16.58324067,
            28.53162489,  49.35657127,  85.65255772, 148.9131591 ])

• Indexes: (0)
• Attributes: (7)

  tuid :

  20231215-162554-320-4a92f9

  name :

  2D-single-float-valued-settable-gettable

  grid_2d :

  True

  grid_2d_uniformly_spaced :

  True

  1d_2_settables_uniformly_spaced :

  False

  xlen :

  10

  ylen :

  12

This format is very close to COO sparse matrix, except that coordinates are not integer.

If the initial data is gridded, it is more convenient to resotre this structure in the dataset for processing. There is a utility function quantify_core.data.handling.to_gridded_dataset() for that:

dset_grid = to_gridded_dataset(dset)
dset_grid

<xarray.Dataset>
Dimensions:  (x0: 10, x1: 12)
Coordinates:
  * x0       (x0) float64 0.0 0.5556 1.111 1.667 2.222 ... 3.333 3.889 4.444 5.0
  * x1       (x1) float64 0.0 0.4545 0.9091 1.364 ... 3.636 4.091 4.545 5.0
Data variables:
    y0       (x0, x1) float64 1.5 1.788 2.241 2.955 ... 167.4 178.3 195.5 222.6
Attributes:
    tuid:                             20231215-162554-320-4a92f9
    name:                             2D-single-float-valued-settable-gettable
    grid_2d:                          False
    grid_2d_uniformly_spaced:         True
    1d_2_settables_uniformly_spaced:  False
    xlen:                             10
    ylen:                             12

xarray.Dataset

• Dimensions:
  • x0: 10
  • x1: 12
• Coordinates: (2)
  • x0
    (x0)
    float64
    0.0 0.5556 1.111 ... 4.444 5.0

    name :

    time_a

    long_name :

    Time A

    units :

    s

    batched :

    False

    array([0.      , 0.555556, 1.111111, 1.666667, 2.222222, 2.777778, 3.333333,
           3.888889, 4.444444, 5.      ])

  • x1
    (x1)
    float64
    0.0 0.4545 0.9091 ... 4.545 5.0

    name :

    time_b

    long_name :

    Time B

    units :

    s

    batched :

    False

    array([0.      , 0.454545, 0.909091, 1.363636, 1.818182, 2.272727, 2.727273,
           3.181818, 3.636364, 4.090909, 4.545455, 5.      ])

• Data variables: (1)
  • y0
    (x0, x1)
    float64
    1.5 1.788 2.241 ... 195.5 222.6

    name :

    sig_a

    long_name :

    Signal A

    units :

    V

    batched :

    False

    array([[  1.5       ,   1.78772855,   2.24103254,   2.95519353,
              4.08032354,   5.85291761,   8.64556351,  13.04525735,
             19.97678625,  30.89711269,  48.10161857,  75.20657955],
           [  2.242909  ,   2.53063755,   2.98394154,   3.69810253,
              4.82323254,   6.5958266 ,   9.38847251,  13.78816635,
             20.71969525,  31.64002169,  48.84452756,  75.94948855],
           [  3.53773178,   3.82546033,   4.27876432,   4.99292531,
              6.11805532,   7.89064938,  10.68329529,  15.08298912,
             22.01451803,  32.93484447,  50.13935034,  77.24431133],
           [  5.79449005,   6.0822186 ,   6.53552259,   7.24968358,
              8.37481359,  10.14740766,  12.94005356,  17.3397474 ,
             24.2712763 ,  35.19160275,  52.39610862,  79.5010696 ],
           [  9.72781435,  10.0155429 ,  10.46884689,  11.18300789,
             12.30813789,  14.08073196,  16.87337787,  21.2730717 ,
             28.2046006 ,  39.12492705,  56.32943292,  83.4343939 ],
           [ 16.58324067,  16.87096922,  17.32427321,  18.03843421,
             19.16356421,  20.93615828,  23.72880419,  28.12849802,
             35.06002692,  45.98035337,  63.18485924,  90.28982022],
           [ 28.53162489,  28.81935345,  29.27265744,  29.98681843,
             31.11194844,  32.8845425 ,  35.67718841,  40.07688224,
             47.00841114,  57.92873759,  75.13324346, 102.23820445],
           [ 49.35657127,  49.64429983,  50.09760382,  50.81176481,
             51.93689482,  53.70948888,  56.50213479,  60.90182862,
             67.83335752,  78.75368397,  95.95818984, 123.06315083],
           [ 85.65255772,  85.94028627,  86.39359026,  87.10775125,
             88.23288126,  90.00547532,  92.79812123,  97.19781506,
            104.12934397, 115.04967041, 132.25417628, 159.35913727],
           [148.9131591 , 149.20088765, 149.65419164, 150.36835264,
            151.49348264, 153.26607671, 156.05872262, 160.45841645,
            167.38994535, 178.3102718 , 195.51477767, 222.61973865]])

• Indexes: (2)
  • x0
    PandasIndex

    PandasIndex(Index([               0.0, 0.5555555555555556, 1.1111111111111112,
           1.6666666666666667, 2.2222222222222223, 2.7777777777777777,
           3.3333333333333335, 3.8888888888888893,  4.444444444444445,
                          5.0],
          dtype='float64', name='x0'))

  • x1
    PandasIndex

    PandasIndex(Index([                0.0, 0.45454545454545503,  0.9090909090909092,
            1.3636363636363638,  1.8181818181818183,   2.272727272727273,
            2.7272727272727275,  3.1818181818181817,  3.6363636363636367,
             4.090909090909091,   4.545454545454546,                 5.0],
          dtype='float64', name='x1'))

• Attributes: (7)

  tuid :

  20231215-162554-320-4a92f9

  name :

  2D-single-float-valued-settable-gettable

  grid_2d :

  False

  grid_2d_uniformly_spaced :

  True

  1d_2_settables_uniformly_spaced :

  False

  xlen :

  10

  ylen :

  12

We see that now dataset has two dimensions (x0 and x1), that represent the initial grid for settables time_a and time_b. Frecuently this simplifies data processing and analysis a lot, for example, we can display the data straight away:

dset_grid.y0.plot(cmap="viridis")

<matplotlib.collections.QuadMesh at 0x7f61c4aed400>