Benchmark

Benchmark#

Whenever we make changes to the code we will rerun a simple benchmark in order to keep track of how the simulation results might change over time. It is in general hard to verify that the equations are implemented correctly because there exist no analytic solutions to the equations. However, by running the same benchmark over time we can see if the changes we make will impact the simulation results. Here the different biomarkers are taken from traces evaluated a the center of at slab that is simulated for 1000 ms (see the benchmark in the library for more details)

import json
import numpy as np
from collections import defaultdict
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots
import requests

pd.options.display.max_columns = 5

headers = {
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
}
print("Get existing data from gist")
response = requests.get(
    "https://api.github.com/gists/73fa6531f28da2b3633a7ddaca38a7cd",
    headers=headers,
)
data = json.loads(response.json()["files"]["convergence_test.json"]["content"])

Get existing data from gist

You can checkout a specific commit by going to the link

ComputationalPhysiology/simcardems

where you swap out GIT_COMMIT_HASH with the specific hash, i.e

for sha in data.keys():
    print(f"https://github.com/ComputationalPhysiology/simcardems/commit/{sha}")

https://github.com/ComputationalPhysiology/simcardems/commit/0aba8258f0bce7101726b115b43564c5b9bb4e2f
https://github.com/ComputationalPhysiology/simcardems/commit/4ea18e8b30ce71b94e8a9a90242abd8dfc2829b0
https://github.com/ComputationalPhysiology/simcardems/commit/1c922747d0bc449c13ffd53458e0ea2cc8578b29
https://github.com/ComputationalPhysiology/simcardems/commit/18054cabc1aeb6db715acecb9cc58a8544992b05
https://github.com/ComputationalPhysiology/simcardems/commit/4f723fe75a3f63fd90e36be73ac615f505e3bdde
https://github.com/ComputationalPhysiology/simcardems/commit/49bef8e75111ff21e4a7d1931e923fa107fc06c7
https://github.com/ComputationalPhysiology/simcardems/commit/74e331ebb96d3717a3bad42861ef2b3cf088332e
https://github.com/ComputationalPhysiology/simcardems/commit/bf7468a3d188295e082d6795c0097eda65e4ae70
https://github.com/ComputationalPhysiology/simcardems/commit/a3ce91b26d63e91ce5d7701333ff3da13de38cc3
https://github.com/ComputationalPhysiology/simcardems/commit/19f9c461fbaf4206d48775c151116cff6d661369
https://github.com/ComputationalPhysiology/simcardems/commit/00dae0a1bd0b7919dee6ba4d96a4cf63f8949b0c
https://github.com/ComputationalPhysiology/simcardems/commit/7c99dbc59c2c38c7c192be00eac500546471bed0
https://github.com/ComputationalPhysiology/simcardems/commit/47edd23
https://github.com/ComputationalPhysiology/simcardems/commit/d44db58
https://github.com/ComputationalPhysiology/simcardems/commit/37fb7d1
https://github.com/ComputationalPhysiology/simcardems/commit/172470c
https://github.com/ComputationalPhysiology/simcardems/commit/5cc1a44
https://github.com/ComputationalPhysiology/simcardems/commit/7d2252c
https://github.com/ComputationalPhysiology/simcardems/commit/3b1d61b
https://github.com/ComputationalPhysiology/simcardems/commit/5992724
https://github.com/ComputationalPhysiology/simcardems/commit/5282d4f
https://github.com/ComputationalPhysiology/simcardems/commit/ce433ea
https://github.com/ComputationalPhysiology/simcardems/commit/3ed0bc0
https://github.com/ComputationalPhysiology/simcardems/commit/0e0091e
https://github.com/ComputationalPhysiology/simcardems/commit/5a86d28
https://github.com/ComputationalPhysiology/simcardems/commit/e616f0a
https://github.com/ComputationalPhysiology/simcardems/commit/5580bef
https://github.com/ComputationalPhysiology/simcardems/commit/0d80766
https://github.com/ComputationalPhysiology/simcardems/commit/7ffac90
https://github.com/ComputationalPhysiology/simcardems/commit/999fcfa
https://github.com/ComputationalPhysiology/simcardems/commit/97e958d
https://github.com/ComputationalPhysiology/simcardems/commit/4cc3e7a
https://github.com/ComputationalPhysiology/simcardems/commit/5edda13
https://github.com/ComputationalPhysiology/simcardems/commit/37f4c61
https://github.com/ComputationalPhysiology/simcardems/commit/c7df02b
https://github.com/ComputationalPhysiology/simcardems/commit/8b11e8d
https://github.com/ComputationalPhysiology/simcardems/commit/0221cc5
https://github.com/ComputationalPhysiology/simcardems/commit/6dde0c0
https://github.com/ComputationalPhysiology/simcardems/commit/a4ea024
https://github.com/ComputationalPhysiology/simcardems/commit/1f7912f
https://github.com/ComputationalPhysiology/simcardems/commit/947b107
https://github.com/ComputationalPhysiology/simcardems/commit/3d43780
https://github.com/ComputationalPhysiology/simcardems/commit/e99e183
https://github.com/ComputationalPhysiology/simcardems/commit/f4b9b88
https://github.com/ComputationalPhysiology/simcardems/commit/3a87bf1
https://github.com/ComputationalPhysiology/simcardems/commit/d5245d4
https://github.com/ComputationalPhysiology/simcardems/commit/813ba9c
https://github.com/ComputationalPhysiology/simcardems/commit/c242e33
https://github.com/ComputationalPhysiology/simcardems/commit/1ed82dc
https://github.com/ComputationalPhysiology/simcardems/commit/f5511ff
https://github.com/ComputationalPhysiology/simcardems/commit/68ae7bb
https://github.com/ComputationalPhysiology/simcardems/commit/739dc1c
https://github.com/ComputationalPhysiology/simcardems/commit/0c195ff
https://github.com/ComputationalPhysiology/simcardems/commit/71a7b0e
https://github.com/ComputationalPhysiology/simcardems/commit/9d5c1b1
https://github.com/ComputationalPhysiology/simcardems/commit/38ad52c
https://github.com/ComputationalPhysiology/simcardems/commit/9f2fd38
https://github.com/ComputationalPhysiology/simcardems/commit/c9b3528
https://github.com/ComputationalPhysiology/simcardems/commit/1353445
https://github.com/ComputationalPhysiology/simcardems/commit/df9ad53
https://github.com/ComputationalPhysiology/simcardems/commit/fbfcf55
https://github.com/ComputationalPhysiology/simcardems/commit/90b65d3
https://github.com/ComputationalPhysiology/simcardems/commit/bc47c96
https://github.com/ComputationalPhysiology/simcardems/commit/322191d
https://github.com/ComputationalPhysiology/simcardems/commit/68926df
https://github.com/ComputationalPhysiology/simcardems/commit/cbc3038
https://github.com/ComputationalPhysiology/simcardems/commit/32e43ff
https://github.com/ComputationalPhysiology/simcardems/commit/4d3a653
https://github.com/ComputationalPhysiology/simcardems/commit/09a7e5b
https://github.com/ComputationalPhysiology/simcardems/commit/9b246fe
https://github.com/ComputationalPhysiology/simcardems/commit/695cd7b
https://github.com/ComputationalPhysiology/simcardems/commit/edc973c
https://github.com/ComputationalPhysiology/simcardems/commit/f38605c
https://github.com/ComputationalPhysiology/simcardems/commit/3398fd2
https://github.com/ComputationalPhysiology/simcardems/commit/f63afc9
https://github.com/ComputationalPhysiology/simcardems/commit/8629d78
https://github.com/ComputationalPhysiology/simcardems/commit/f702955
https://github.com/ComputationalPhysiology/simcardems/commit/376167c
https://github.com/ComputationalPhysiology/simcardems/commit/69503d0
https://github.com/ComputationalPhysiology/simcardems/commit/67aba10
https://github.com/ComputationalPhysiology/simcardems/commit/cb33e06
https://github.com/ComputationalPhysiology/simcardems/commit/12ad5f2
https://github.com/ComputationalPhysiology/simcardems/commit/42363b0
https://github.com/ComputationalPhysiology/simcardems/commit/de8a7ff
https://github.com/ComputationalPhysiology/simcardems/commit/0648726
https://github.com/ComputationalPhysiology/simcardems/commit/f73b25e
https://github.com/ComputationalPhysiology/simcardems/commit/5d5bd73
https://github.com/ComputationalPhysiology/simcardems/commit/2fe1309
https://github.com/ComputationalPhysiology/simcardems/commit/04a9790
https://github.com/ComputationalPhysiology/simcardems/commit/7f51a50
https://github.com/ComputationalPhysiology/simcardems/commit/93c349e
https://github.com/ComputationalPhysiology/simcardems/commit/e8e6d35
https://github.com/ComputationalPhysiology/simcardems/commit/630ed09
https://github.com/ComputationalPhysiology/simcardems/commit/6e55904
https://github.com/ComputationalPhysiology/simcardems/commit/fde1483
https://github.com/ComputationalPhysiology/simcardems/commit/de72206
https://github.com/ComputationalPhysiology/simcardems/commit/e3254b0
https://github.com/ComputationalPhysiology/simcardems/commit/dbbec69
https://github.com/ComputationalPhysiology/simcardems/commit/f214f80
https://github.com/ComputationalPhysiology/simcardems/commit/159c6bc
https://github.com/ComputationalPhysiology/simcardems/commit/a3a23b2
https://github.com/ComputationalPhysiology/simcardems/commit/60e69df
https://github.com/ComputationalPhysiology/simcardems/commit/cf7cc9d
https://github.com/ComputationalPhysiology/simcardems/commit/10592e9
https://github.com/ComputationalPhysiology/simcardems/commit/9d72060
https://github.com/ComputationalPhysiology/simcardems/commit/cc84512
https://github.com/ComputationalPhysiology/simcardems/commit/e36932a

len(data)

107

excluded_keys = ["dt", "dx", "sha", "num_cells_mechanics", "num_cells_ep", "num_vertices_mechanics", "num_vertices_ep"]
all_results = defaultdict(list)
for git_hash, d in data.items():
    all_results["git_hash"].append(git_hash)
    for key, value in d.get("dx0.2_dt0.05").items():
        if key in excluded_keys:
            continue
        all_results[key].append(value)

results = {}
for k, v in all_results.items():
    print(k, len(v))
    if len(v) == len(data):
        results[k] = v

git_hash 107
import_time 107
timestamp 107
simcardems_version 107
coupling_type 6
create_runner_time 107
solve_time 107
APD40 107
APD50 107
APD90 107
triangulation 107
Vpeak 107
Vmin 107
dvdt_max 6
maxCa 107
ampCa 107
CaTD50 107
CaTD80 107
maxTa 107
ampTa 107
ttp_Ta 107
rt50_Ta 107
rt95_Ta 107
maxlmbda 107
minlmbda 107
ttplmbda 107
lmbdaD50 107
lmbdaD80 107
lmbdaD90 107
rt50_lmbda 107
rt95_lmbda 107
max_displacement_norm 107
min_displacement_norm 6
time_to_max_displacement_norm 107
time_to_min_displacement_norm 6
max_displacement_x 107
min_displacement_x 6
time_to_max_displacement_x 107
time_to_min_displacement_x 6
max_displacement_y 107
min_displacement_y 6
time_to_max_displacement_y 107
time_to_min_displacement_y 6
max_displacement_z 107
min_displacement_z 6
time_to_max_displacement_z 107
time_to_min_displacement_z 6
dvdt 101
CaTD90 101
rel_max_displacement_norm 101
max_displacement_perc_norm 101
rel_max_displacement_perc_norm 101
rel_max_displacement_x 101
max_displacement_perc_x 101
rel_max_displacement_perc_x 101
rel_max_displacement_y 101
max_displacement_perc_y 101
rel_max_displacement_perc_y 101
rel_max_displacement_z 101
max_displacement_perc_z 101
rel_max_displacement_perc_z 101

df = pd.DataFrame(results)
df["timestamp"] = pd.to_datetime(df["timestamp"])
df
git_hash import_time ... max_displacement_z time_to_max_displacement_z
0 0aba8258f0bce7101726b115b43564c5b9bb4e2f 7.600000e-06 ... 0.104241 137.05
1 4ea18e8b30ce71b94e8a9a90242abd8dfc2829b0 9.000000e-07 ... 0.096905 139.05
2 1c922747d0bc449c13ffd53458e0ea2cc8578b29 9.000000e-07 ... 0.096905 139.05
3 18054cabc1aeb6db715acecb9cc58a8544992b05 1.100000e-06 ... 0.096905 139.05
4 4f723fe75a3f63fd90e36be73ac615f505e3bdde 1.600000e-06 ... 0.096905 139.05
... ... ... ... ... ...
102 cf7cc9d 9.000000e-07 ... 0.000000 138.05
103 10592e9 9.000000e-07 ... 0.096905 0.05
104 9d72060 1.000000e-06 ... 0.096910 0.05
105 cc84512 8.000000e-07 ... 0.096910 0.05
106 e36932a 1.000000e-06 ... 0.096910 0.05

107 rows × 37 columns

df = df.sort_values(by="timestamp")
df
git_hash import_time ... max_displacement_z time_to_max_displacement_z
74 f63afc9 1.001000e-06 ... 0.000000 138.05
92 630ed09 1.000000e-06 ... 0.000000 138.05
67 4d3a653 1.100000e-06 ... 0.000000 138.05
52 0c195ff 1.000000e-06 ... 0.000000 138.05
34 c7df02b 9.000000e-07 ... 0.000000 138.05
... ... ... ... ... ...
4 4f723fe75a3f63fd90e36be73ac615f505e3bdde 1.600000e-06 ... 0.096905 139.05
3 18054cabc1aeb6db715acecb9cc58a8544992b05 1.100000e-06 ... 0.096905 139.05
2 1c922747d0bc449c13ffd53458e0ea2cc8578b29 9.000000e-07 ... 0.096905 139.05
1 4ea18e8b30ce71b94e8a9a90242abd8dfc2829b0 9.000000e-07 ... 0.096905 139.05
0 0aba8258f0bce7101726b115b43564c5b9bb4e2f 7.600000e-06 ... 0.104241 137.05

107 rows × 37 columns

df[["simcardems_version", "timestamp", "git_hash"]]
simcardems_version timestamp git_hash
74 2022.3.0 2022-10-06 16:25:06.830650 f63afc9
92 2022.3.0 2022-10-06 16:55:04.711879 630ed09
67 2022.3.0 2022-10-06 20:09:07.230901 4d3a653
52 2022.3.0 2022-10-06 20:19:12.227464 0c195ff
34 2022.3.0 2022-10-06 20:28:25.015389 c7df02b
... ... ... ...
4 2023.5.5 2023-06-05 10:35:21.889433 4f723fe75a3f63fd90e36be73ac615f505e3bdde
3 2023.6.0 2023-06-27 12:42:43.474076 18054cabc1aeb6db715acecb9cc58a8544992b05
2 2023.6.1 2023-09-11 12:49:33.033569 1c922747d0bc449c13ffd53458e0ea2cc8578b29
1 2023.7.0 2023-09-24 20:54:02.914676 4ea18e8b30ce71b94e8a9a90242abd8dfc2829b0
0 2023.7.1 2023-10-26 09:55:49.558780 0aba8258f0bce7101726b115b43564c5b9bb4e2f

107 rows × 3 columns

def get_ylim(values):
    if np.isclose(values, 0).all():
        return (-0.01, 0.01)
    y_mean = np.mean(values)
    y_max = max(np.max(values), y_mean + 0.01 * abs(y_mean))
    y_min = min(np.min(values), y_mean - 0.01 * abs(y_mean))
    return (y_min, y_max)

columns = [c for c in df.columns if c not in ["timestamp", "simcardems_version", "git_hash"]]
git_hash = df["git_hash"]
versions = df["simcardems_version"]
dates = [t.date() for t in df["timestamp"]]

text = []
for h, v, t in zip(git_hash, versions, dates):
    text.append(
        "\n".join(
            [
                f"<br>Git Hash: {h}</br>",
                f"<br>Version: {v}</br>",
                f"<br>Timestamp {t}</br>",
            ]
        )
    )

fig = make_subplots(rows=len(columns), cols=1, subplot_titles=columns, shared_xaxes=True)

yranges = []
for i, c in enumerate(columns):
    row = i + 1
    col = 1

    fig.add_trace(go.Scatter(x=df["timestamp"], y=df[c], text=text, hovertemplate="%{text}"), row=row, col=col)
    fig.update_yaxes(range=get_ylim(df[c]), row=row, col=col)

fig.update_layout(height=5000, showlegend=False)
fig.show()