We are happy to announce the release of Pynguin version 0.26.0.
Changelog for 0.26.0
Allow the calculation of coverage values regardless of optimisation.
Allows to yield, for example, the resulting line coverage of the test
suite, while the optimisation during test generation was done for
branch coverage. Add the required coverage values to your
—output-variables list to activate this feature.
Note: when doing this, the Coverage output variable will contain
the average value of the incorporated coverage values!
Provide a Cobertura-like coverage report
Pynguin already provides an HTML report that can be activated by
setting —create-coverage-report True. This report is nice for
human users but not very usable if one wants to automatically reason
about the achieved coverage using tools. We thus also emit an XML
report in the style of the
Cobertura tool that can be
used for further automated tools.
Fix typo on test_parameterizedstatements.py
Fix typo on testcase.py
Improve mutation-based assertion generation
The assertion generation now does not more compare the recorded
assertion traces but actually executes the assertions to make the
whole process more reliable. Besides, we do not check for is not
None as a fallback for object checks any more, but use an
isinstance check on the object’s type to have a more precise assertion.
Make statistics tests debuggable
Some accidental circular import made it impossible to run a debugger
on tests in pynguin.utils. We resolved this by moving tests to
another package.
Partial rework of internal type system
We added an abstraction layer over the existing type hints from
a module to make handling and reasoning with types easier. This
abstraction is based on the one used by
mypy, however, we only cover a small part of
what PEP-484 actually defines.
Make assertion generation more strict
posted 2022–09–19 10:00
We are happy to announce the release of Pynguin version 0.25.0.
Changelog for 0.25.0
- Fix further issues with test-case isolation during execution.
- Fix some bugs regarding type information.
- Use astroid instead of Python’s
ast module for our module analysis
due to its enhanced features.
posted 2022–07–05 09:38
We are happy to announce the release of Pynguin version 0.24.0.
Changelog for 0.24.0
Fix killing mutant reporting
Use black to format the generated test cases.
Pynguin now requires black as a run-time dependency to use its code formatting.
posted 2022–06–29 12:44
We are happy to announce the release of Pynguin version 0.23.0.
Changelog for 0.23.0
Provide a naive inheritance graph to improve input generation.
Improve killing of long-running test-case executions
Add computation of mutation scores for MUTATION_ANALYSIS assertion generation.
The output variables NumberOfCreatedMutants, NumberOfKilledMutants,
NumberOfTimedOutMutants, and MutationScore allow to export those values.
Do not enable typing.TYPE_CHECKING for SUT analysis as this may cause circular imports.
Improve the black list of modules that shall not be incorporated into the test cluster.
Annotate failing tests with @pytest.mark.xfail(strict=True).
Improve log output of mutation-based assertion generation.
Add instrumentation to mutated modules to easier kill them.
This change is relevant only to the MUTATION_ANALYSIS assertion-generation strategy.
Write errors in execution threads to the log instead of STDERR to avoid cluttering
log output.
Add limits for amount and size of constants in the constant pool.
The configuration options max_dynamic_length and max_dynamic_pool_size allow
to set sizes for the maximum length of strings/bytes that shall be stored in the
dynamic constant pool and the maximum numbers of constants of a type, respectively.
This prevents the constant pool from growing unreasonably large.
Improve handling of type annotations.
Fix computation of cyclomatic complexity.
Computing cyclomatic complexity does not work for functions that are not present
in the AST, e.g., default constructors. We now omit those from the computation of
the cyclomatic-complexity output variables.
posted 2022–06–24 12:19
We are happy to announce the release of Pynguin version 0.22.0.
Changelog for 0.22.0
Fix selection of type-inference strategy.
Fix a bug in the type inference regarding cases where not type information is present.
Add a PyLint checker for calls to print().
Extend the blacklist of modules that shall not be analysed.
Raise RuntimeError from tracer when called from another thread.
Provide better exception messages for critical failures.
Apply a further limit to the execution time of a single generated test case to at
most 10 seconds.
Exclude empty enum classes from test cluster to fix test generation.
Parsing included modules raised an issue when the enum module is used: the test
cluster then had a reference to the enum.Enum class, which obviously does not
contain any fields. In the following, generating tests failed, as soon as this
class was selected to fulfil parameter values because there was no field to select
from, e.g., MyEnum.MY_FIELD. We now exclude empty enums from the test cluster.
posted 2022–06–08 08:23
We are happy to announce the release of Pynguin version 0.21.0.
Changelog for 0.21.0
Fix a bug in the module analysis regarding nested functions.
Nested functions/closures caused Pynguin’s module analysis to crash with a failing assertion.
Improve the branch-distance computation for bool values.
Allow for more statistics variables regarding number of lines and cyclomatic complexity.
posted 2022–05–25 14:44
We are happy to announce the release of Pynguin version 0.20.0 and 0.20.1.
Changelog for 0.20.1
Fix a bug terminating Pynguin due to threading
Changelog for 0.20.0
Breaking Changes
Remove splitting into passing and failing test suite.
Previously, we consider a test case passing if it did not raise any exception
during its execution; it was considered failing otherwise. Pynguin did a split of
the test cases into two test suites before exporting them. This was mainly an
artefact from implementing the random algorithm in the very beginning of the
project. Due to the improved assertion export for exception assertions we can now
get rid of the split and export only one test module containing all generated test cases.
Remove the option to use a log file (--log_file or --log-file).
Pynguin writes its output to STDOUT/STDERR now, if requested by the -v/-vv
switch. This output is formatted by
@willmcgugan‘s amazing
rich library. A user can disable the output
formatting by setting the --no-rich flag. Of course, because we believe that
rich is such an awesome library, we also provide an alias for this flag, called
--poor 😉
Further Changes
Distinguish between expected and unexpected exceptions.
We consider an exception to be expected if it is explicitly raised in the code
under test or is documented in the code’s docstring. For those exceptions we
build an with pytest.raises block around the exception-raising statement. All
other exceptions are considered to be unexpected. We decorate the test method
with the @pytest.mark.xfail decorator and let the exception happen. It is up to
the user to decide whether such an exception is expected. An exception here is
the AssertionError: it is considered to be expected as soon as there is an
assert statement in the code under test.
Improve installation description to explicitly point to using a virtual environment
(see GitHub issue #23, thanks to
@tuckcodes).
Improve variable names and exception assertions
The assertion generation got an improved handling for asserting on exceptions,
which creates more meaningful and (hopefully) better understandable assertions for exceptions.
Enhance the module analysis
This is basically a rewrite of our previously existing test cluster, which keeps
track of all the callables from the subject under test as well as the subject’s
dependencies. It also incorporates an analysis of the subject’s AST (if present)
and allows for more and more precise information about the subject which can then
improve the quality of the generated tests.
To distinguish bytecode instructions during instrumentation we add an
ArtificialInstr for our own added instructions.
Fix a bug in the tracing of runtime types.
During assertion generation Pynguin tracks the variable types to decide for which
values it actually is able to generate assertions. Creating an assertion on a
generator function does not work, as the type is not exposed by Python but only
present during runtime—thus generating an object of this type always fail. We
mitigate this by ignoring objects of type builtins.generator from the assertion generation.
Improve documentation regarding coverage measurement and the coverage report
posted 2022–05–24 16:15
We are happy to announce the release of Pynguin version 0.18.0 and 0.19.0.
Changelog for 0.19.0
Breaking Change
One can now use multiple stopping conditions at a time.
This breaks the CLI in two ways:
The parameter --stopping-condition has been removed.
The parameter --budget was renamed to --maximum-search-time.
Users have to change their run configurations accordingly!
To specify stopping conditions, add one or many from --maximum-search-time,
--maximum-test-executions, --maximum-statement-executions, and
--maximum-iterations to your command line.
Further Changes
- Clarify log output for search phases
- Pynguin now uses the ast.unparse function from Python’s AST library instead of
the third-party astor library to generate the source code of the test cases.
Changelog for 0.18.0
Breaking Change
We drop the support for Python 3.8 and Python 3.9 with this version!
You need Python 3.10 to run Pynguin! We recommend using our Docker container,
which is already based on Python 3.10, to run Pynguin.
Further Changes
- Add line coverage visualisation to the coverage report.
- Add a citation reference to our freshly accepted ICSE‘22 tool demo paper “Pynguin:
Automated Unit Test Generation for Python”.
- Unify the modules for the analysis of the module under test.
posted 2022–03–16 11:02
I am happy to announce that our paper “Pynguin: Automated Unit Test Generation for
Python” was accepted in this year’s ICSE Demo Track.
posted 2022–02–11 07:47
We are happy to announce the release of Pynguin version 0.17.0.
Changelog
Add line coverage as another coverage type (thanks to
Simon Labrenz).
The user can now choose between using either line or branch coverage or both by
setting the parameter --coverage-metrics to LINE, BRANCH, or
LINE,BRANCH. Default value is BRANCH.
posted 2022–02–04 12:40
