{ "cells": [ { "cell_type": "markdown", "id": "3b8ae811", "metadata": {}, "source": [ "# Profiling notebook" ] }, { "cell_type": "markdown", "id": "1097bd1e", "metadata": {}, "source": [ "This notebook collects and compares the run time for several notebooks. These notebooks are specified in the `metrics.py`, in the `EXPERIMENT_NOTEBOOKS` variable.\n", "Simply run the whole notebook, and the results will be displayed in tables at the end.\n", "\n", "Each notebook listed in the `EXPERIMENT_NOTEBOOKS` (see `metrics.py`) table must have the `run_experiment()` defined. Optionally, you can define the `close_experiment()` function.\n", "\n", "This profiler will profile the `run_experiment` function, and after that's done, call the `close_experiment` (if it exists). The `close_experiment` function is not mandatory, but if there are any resources that need to be closed, you can implement that here. The profiler measures the times listed in the `METHODS` variable (see `metrics.py`), and the total time." ] }, { "cell_type": "markdown", "id": "13ae83c1", "metadata": {}, "source": [ "After the profiling is done, the notebook generates a file in this directory for each notebook. This file contains the detailed profiling report. For the notebook `.ipynb` it generates `.ipynb.prof` file, which can be opened with snakeviz (`pip install snakeviz`): `snakeviz .ipynb.prof`." ] }, { "cell_type": "markdown", "id": "04139070", "metadata": {}, "source": [ "## Configuration" ] }, { "cell_type": "code", "execution_count": 1, "id": "e2521e52", "metadata": {}, "outputs": [], "source": [ "# `benchmark_mode` sets whether we run the schedules in benchmark mode.\n", "# If it's benchmark mode, we override the reference measurements file\n", "# with the current timing values, and that will be those will be the new reference values.\n", "benchmark_mode = False\n", "profiling_reference_filename = \"profiling_reference_values.pickle\"" ] }, { "cell_type": "code", "execution_count": 2, "id": "90faba6d", "metadata": {}, "outputs": [], "source": [ "# The end result table will display each cell in different colors.\n", "# Each value's \"sigma\" is practically it's measurement error,\n", "# and if the current time is above/below\n", "# the `reference value±sigma*sigma_multiplier_threshold`\n", "# the cell will be displayed in different colors.\n", "sigma_multiplier_threshold = 2.0 # 2.0 is a reasonable value." ] }, { "cell_type": "markdown", "id": "cf4f5061", "metadata": {}, "source": [ "## Loading reference data" ] }, { "cell_type": "code", "execution_count": 3, "id": "758b1d4d", "metadata": {}, "outputs": [], "source": [ "# Reference values for profiling.\n", "# Each notebook has a reference timing value.\n", "import pickle\n", "from os.path import exists\n", "\n", "if not benchmark_mode:\n", " if not exists(profiling_reference_filename):\n", " raise RuntimeError(\n", " f\"Reference file '{profiling_reference_filename}' does not exist! \"\n", " f\"Make sure this file is created by first running the profiling with 'benchmark_mode=True'!\"\n", " )\n", " with open(profiling_reference_filename, \"rb\") as f:\n", " reference = pickle.load(f)" ] }, { "cell_type": "markdown", "id": "5e840374", "metadata": {}, "source": [ "## Running the profiling" ] }, { "cell_type": "code", "execution_count": null, "id": "109b43ac", "metadata": { "scrolled": false }, "outputs": [], "source": [ "import metrics\n", "\n", "measured_data = metrics.measure_experiment_runtimes()" ] }, { "cell_type": "code", "execution_count": 5, "id": "34fd0713", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('simple_binned_acquisition',\n", " [(0.9946758512999999, 0.060520291938758214),\n", " (0.1054618794, 0.0023019861591870687),\n", " (0.2950480362000001, 0.07708127271427773),\n", " (0.0283041775, 0.0005857374014951724),\n", " (0.0103337558, 0.001243391736535975)],\n", " (1.5076610159000001, 0.0752881679789585)),\n", " ('resonator_spectroscopy',\n", " [(0.24496450730000002, 0.08676685139754789),\n", " (0.15548121180000002, 0.0021772059857387196),\n", " (0.062539442, 0.0009306177699387383),\n", " (0.0027010211000000005, 0.00014065694190436276),\n", " (0.012418523100000002, 0.000398350933049799)],\n", " (0.6421398809, 0.09432553767909004)),\n", " ('random_gates',\n", " [(1.030944426, 0.07697153933708932),\n", " (0.11840048560000001, 0.0015075617320314957),\n", " (0.33237856450000003, 0.007154834526667149),\n", " (0.00043242370000000005, 5.157527486897152e-05),\n", " (0.0032713507000000004, 0.00022519136410841211)],\n", " (1.4963750570999994, 0.07880670434003852))]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "measured_data" ] }, { "cell_type": "code", "execution_count": 6, "id": "a02eac5c", "metadata": {}, "outputs": [], "source": [ "if benchmark_mode:\n", " with open(profiling_reference_filename, \"wb\") as f:\n", " pickle.dump(measured_data, f)\n", " reference = measured_data" ] }, { "cell_type": "markdown", "id": "ebf6126f", "metadata": {}, "source": [ "## Displaying the results" ] }, { "cell_type": "code", "execution_count": 7, "id": "dc64e1bd", "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/plain": [ "[('simple_binned_acquisition',\n", " [(0.9926163084, 0.06192060210975775),\n", " (0.10498222319999999, 0.0015965531111733849),\n", " (0.29654308549999997, 0.07541338532140536),\n", " (0.028360908400000006, 0.0005025226272929342),\n", " (0.010576865600000002, 0.0013565539147959512)],\n", " (1.5076823723000008, 0.073541431604032)),\n", " ('resonator_spectroscopy',\n", " [(0.24136320129999994, 0.08250571058252025),\n", " (0.15536156240000001, 0.0037741058575561878),\n", " (0.061933728300000004, 0.0006777892286995014),\n", " (0.0026206999, 4.7912470390180906e-05),\n", " (0.012418401700000001, 0.00048759705280013274)],\n", " (0.6349079799999999, 0.09074046088950546)),\n", " ('random_gates',\n", " [(1.0291132077, 0.07413907143808215),\n", " (0.1209051369, 0.0023142911883413317),\n", " (0.3420565029, 0.004800341060046611),\n", " (0.0004728105, 0.00010186780396597362),\n", " (0.0033847724000000004, 0.00022984585643086612)],\n", " (1.5066835084, 0.07132325504794586))]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "reference" ] }, { "cell_type": "code", "execution_count": 8, "id": "2c06a609", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import metrics" ] }, { "cell_type": "code", "execution_count": 9, "id": "ea1af4bb", "metadata": {}, "outputs": [], "source": [ "table = []\n", "header = []\n", "table_diff = []\n", "header_diff = []\n", "\n", "header.append(\"\")\n", "header_diff.append(\"\")\n", "for method in metrics.METHODS:\n", " header.append(method[0])\n", " header_diff.append(method[0])\n", "header.append(\"total\")\n", "header_diff.append(\"total\")\n", "\n", "for row_id, (experiment_notebook, times, total_time) in enumerate(measured_data):\n", " row = []\n", " row_diff = []\n", " row.append(experiment_notebook)\n", " row_diff.append(experiment_notebook)\n", " for column_id, time in enumerate(times):\n", " expected_value = time[0]\n", " sigma = time[1]\n", " row.append(f\"{expected_value:.2g} ± {sigma:.2g} s\")\n", "\n", " time_diff = expected_value - reference[row_id][1][column_id][0]\n", " row_diff.append(f\"{time_diff:.2g} ± {sigma:.2g} s\")\n", "\n", " row.append(f\"{total_time[0]:.2g} ± {total_time[1]:.2g} s\")\n", "\n", " total_time_diff = total_time[0] - reference[row_id][2][0]\n", " row_diff.append(f\"{total_time_diff:.2g} ± {total_time[1]:.2g} s\")\n", "\n", " table.append(row)\n", " table_diff.append(row_diff)" ] }, { "cell_type": "code", "execution_count": 10, "id": "fe75788c", "metadata": {}, "outputs": [], "source": [ "def diff_to_style(current, ref):\n", " green = \"#d0ffd0\"\n", " red = \"#ffd0d0\"\n", " val, sigma = current[0], current[1]\n", " ref_val, ref_sigma = ref[0], ref[1]\n", " if (val - sigma * sigma_multiplier_threshold) > (\n", " ref_val + ref_sigma * sigma_multiplier_threshold\n", " ):\n", " return f\"background-color: {red}\"\n", " if (val + sigma * sigma_multiplier_threshold) < (\n", " ref_val - ref_sigma * sigma_multiplier_threshold\n", " ):\n", " return f\"background-color: {green}\"\n", " return \"\"" ] }, { "cell_type": "code", "execution_count": 11, "id": "a6b824c2", "metadata": {}, "outputs": [], "source": [ "style_table = []\n", "\n", "for row_id, (experiment_notebook, times, total_time) in enumerate(measured_data):\n", " row = []\n", " row.append(\"\")\n", " for column_id, time in enumerate(times):\n", " if row_id < len(reference) and column_id < len(reference[row_id][1]):\n", " row.append(diff_to_style(time, reference[row_id][1][column_id]))\n", " else:\n", " row.append(\"\")\n", " if row_id < len(reference):\n", " row.append(diff_to_style(total_time, reference[row_id][2]))\n", " else:\n", " row.append(\"\")\n", " style_table.append(row)" ] }, { "cell_type": "code", "execution_count": 12, "id": "2271fdaf", "metadata": {}, "outputs": [], "source": [ "style_table = np.array(style_table)\n", "style_properties = {\"border\": \"1px solid gray\"}\n", "styles = [\n", " dict(\n", " selector=\"caption\",\n", " props=[(\"text-align\", \"center\"), (\"font-size\", \"200%\"), (\"color\", \"black\")],\n", " )\n", "]" ] }, { "cell_type": "code", "execution_count": 13, "id": "d74afb03", "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame(table, columns=header)\n", "df = df.style.set_properties(**style_properties).apply(lambda _: style_table, axis=None)\n", "df = df.set_caption(\"Measured times\").set_table_styles(styles)" ] }, { "cell_type": "code", "execution_count": 14, "id": "bb38048d", "metadata": {}, "outputs": [], "source": [ "df_diff = pd.DataFrame(table_diff, columns=header)\n", "df_diff = df_diff.style.set_properties(**style_properties).apply(\n", " lambda _: style_table, axis=None\n", ")\n", "df_diff = df_diff.set_caption(\"Measured diffs to reference\").set_table_styles(styles)" ] }, { "cell_type": "code", "execution_count": 15, "id": "822cb265", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Measured times
 compileprepareschedulerunprocesstotal
0simple_binned_acquisition0.99 ± 0.061 s0.11 ± 0.0023 s0.3 ± 0.077 s0.028 ± 0.00059 s0.01 ± 0.0012 s1.5 ± 0.075 s
1resonator_spectroscopy0.24 ± 0.087 s0.16 ± 0.0022 s0.063 ± 0.00093 s0.0027 ± 0.00014 s0.012 ± 0.0004 s0.64 ± 0.094 s
2random_gates1 ± 0.077 s0.12 ± 0.0015 s0.33 ± 0.0072 s0.00043 ± 5.2e-05 s0.0033 ± 0.00023 s1.5 ± 0.079 s
\n" ], "text/plain": [ "" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# If the cell is green (or red), the current time\n", "# is significantly less (or more) than the reference time.\n", "df" ] }, { "cell_type": "code", "execution_count": 16, "id": "f25477aa", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Measured diffs to reference
 compileprepareschedulerunprocesstotal
0simple_binned_acquisition0.0021 ± 0.061 s0.00048 ± 0.0023 s-0.0015 ± 0.077 s-5.7e-05 ± 0.00059 s-0.00024 ± 0.0012 s-2.1e-05 ± 0.075 s
1resonator_spectroscopy0.0036 ± 0.087 s0.00012 ± 0.0022 s0.00061 ± 0.00093 s8e-05 ± 0.00014 s1.2e-07 ± 0.0004 s0.0072 ± 0.094 s
2random_gates0.0018 ± 0.077 s-0.0025 ± 0.0015 s-0.0097 ± 0.0072 s-4e-05 ± 5.2e-05 s-0.00011 ± 0.00023 s-0.01 ± 0.079 s
\n" ], "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# All data is (current_time - reference_time).\n", "# If the cell is green (or red), the current time\n", "# is significantly less (or more) than the reference time.\n", "df_diff" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 5 }