Performance test results

These tests compare prefab_classes with other modules that, as at least part of their feature set, handle writing class boilerplate. Some of these modules may have more features or do extra work which could make them more useful in certain situations.

The main modules being compared are prefab_classes, dataclasses, attrs and pydantic. Some tests are also run against namedtuple and NamedTuple along with handwritten classes.

Rough specs:

2023 Windows 10 Desktop: Intel i5-13600KF, Crucial P3 1TB PCIe M.2 2280 SSD

Versions

Software Package

Version Number

Python

3.12.2

prefab_classes

0.12.0

attrs

23.2.0

cattrs

23.2.3

pydantic

2.6.4

pydantic_core

2.16.3

Hyperfine tests

These tests are run using Hyperfine

The shell scripts to run these tests can be generated by running perf/hyperfine_testmaker.py.

Import Time

This just tests the overall time to launch python and import the module to be used for constructing classes. python -c "pass" used as a baseline.

Command

Mean [ms]

Min [ms]

Max [ms]

Relative

python -c "pass"

22.7 ± 0.8

21.4

24.9

1.00

python -c "from prefab_classes import prefab"

23.9 ± 1.0

23.0

27.1

1.06 ± 0.06

python -c "from collections import namedtuple"

23.6 ± 0.5

22.9

24.9

1.04 ± 0.04

python -c "from typing import NamedTuple"

31.3 ± 0.4

30.7

32.7

1.38 ± 0.05

python -c "from dataclasses import dataclass"

38.0 ± 0.5

36.9

38.9

1.68 ± 0.06

python -c "from attrs import define"

52.1 ± 0.8

50.7

54.0

2.30 ± 0.09

python -c "from pydantic import BaseModel"

70.0 ± 3.7

65.6

79.3

3.09 ± 0.20

Class Contruction

This is a series of tests of the time it takes to launch python and generate 100 classes each with 5 attributes.

The code for each class looks roughly like this:

@dataclass
class C0:
    a: int
    b: int
    c: int
    d: int
    e: int

In some cases the __init__, __repr__ and __eq__ functions are also looked up in order to force them to be generated (prefab_eval).

For a baseline comparison, python -c "pass" is included just to show the overhead from the interpreter.

As prefab_classes has multiple ways of generating classes it is here multiple times.

prefab_classes_timer generates the classes dynamically, but leaves the methods unused so they are not yet evaluated. This is how they would usually be on import.

prefab_eval_timer generates the classes dynamically, and also generates their methods as they would be after first use.

Command

Mean [ms]

Min [ms]

Max [ms]

Relative

python -c "pass"

22.3 ± 0.5

21.4

23.5

1.00

python hyperfine_importers/native_classes_timer.py

23.1 ± 0.6

22.5

25.7

1.04 ± 0.04

python hyperfine_importers/prefab_classes_timer.py

25.7 ± 1.0

24.7

30.0

1.15 ± 0.05

python hyperfine_importers/prefab_slots_timer.py

25.9 ± 0.4

25.0

27.2

1.16 ± 0.03

python hyperfine_importers/prefab_eval_timer.py

36.4 ± 0.6

35.2

37.6

1.63 ± 0.05

python hyperfine_importers/namedtuples_timer.py

27.6 ± 0.6

26.8

29.2

1.24 ± 0.04

python hyperfine_importers/typed_namedtuples_timer.py

37.4 ± 0.7

36.4

39.4

1.68 ± 0.05

python hyperfine_importers/dataclasses_timer.py

59.1 ± 1.9

57.0

67.9

2.65 ± 0.10

python hyperfine_importers/attrs_noslots_timer.py

87.4 ± 2.8

84.2

94.9

3.92 ± 0.16

python hyperfine_importers/attrs_slots_timer.py

93.1 ± 3.5

87.1

102.8

4.18 ± 0.19

python hyperfine_importers/pydantic_timer.py

167.3 ± 5.2

159.0

180.9

7.51 ± 0.30

json tests

Based on the ORJSON dataclasses performance test (but without using orjson).

With recent updates to other serialization methods, the prefab_classes to_json method is no longer faster than the naive dataclasses as_dict by such a margin and two of the other packages perform serialization faster.

Pydantic’s conversion just uses its .json() method with no arguments. This has had a drastic improvement since the previous test and is now the fastest method. In older versions of Pydantic this was by far the slowest method of json conversion.

cattrs conversion is also faster by making use of type hints to generate more direct serialization functions.

Result from perf/serialization/json_conversion_speed.py

Python Version: 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)]
Prefab Classes version: v0.11.0
Platform: Windows-10-10.0.19045-SP0

Method

Time /s

prefab_to_json

0.45

dataclasses_asdict

0.94

attrs_asdict

1.12

cattrs

0.29

pydantic

0.16

While improvements to match at least cattrs performance would be nice, the current focus is still on improving import and class generation time.

perf_profile.py

This is NOT an overall performance test. This is just intended to check - very roughly - how quickly a module with a large number of generated classes will import.

prefab_attributes uses attribute() instead of type hints for class generation. This test is included to make sure there haven’t been regressions in either form of definition.

Python Version: 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)]
Prefab Classes version: v0.11.0
Platform: Windows-10-10.0.19045-SP0
Time for 100 imports of 100 classes defined with 5 basic attributes

Method

Total Time (seconds)

standard classes

0.07

namedtuple

0.30

NamedTuple

0.46

dataclasses

1.87

attrs 23.2.0

3.49

pydantic 2.6.4

4.06

dabeaz/cluegen

0.09

dabeaz/cluegen_eval

0.86

dabeaz/dataklasses

0.09

dabeaz/dataklasses_eval

0.10

prefab v0.12.0

0.14

prefab_attributes v0.12.0

0.14

prefab_eval v0.12.0

1.09