Relintai/regression-test-project
1
0
Fork 0
mirror of https://github.com/Relintai/regression-test-project.git synced 2024-12-23 13:57:21 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Rafał Mikrut 2021-04-16 18:05:00 +02:00
parent 45d4c3ad3e
commit 9a4b3def33
1 changed files with 114 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

154
README.md
View File

@ -1,64 +1,138 @@
## Godot regression test project
This repository contains Godot project which is tested in offical Godot repository(3.2 branch)
# Godot regression test project
This repository contains project which is used to find regressions in Godot.
It aims to be complex project, which will allow to find crashes, leaks and invalid memory writes before PR is merged.
It aims to check as much as possible functions and states, be easy in maintain and provide reproducible results.
Sadly it can't find automatically any logic errors.
## Basic Informations
This project contains a few different scenes and `Start.tscn`(default one) which opens every other scene.
## Reproduce bugs
If CI find bug, you can easily without much effort check which scene cause problems(stacktrace, address or leak sanitizer log will be available).
List of this scenes are available in `Autoload.gd` file.
All main scenes are independent of each other(only `Autoload.gd` is required), so it is easy to disable some for testing.
![B](https://user-images.githubusercontent.com/41945903/104442905-060e7c00-5596-11eb-9000-f9bb338ece79.png)
To create minimal test scene:
- Look at the console output - there is printed info about current used scene
- If you run `Start.tscn` scene(which run other scenes in queue) then you can modify list of scenes which needs to be run in `Autoload.gd` by commenting records in `alone_steps` or `all_in_one` array(just like in picture)
![Zrzut ekranu z 2021-01-13 11-52-41](https://user-images.githubusercontent.com/41945903/104442905-060e7c00-5596-11eb-9000-f9bb338ece79.png)
It is possible manually set how long project need to be executed from command line, just by adding at the end number of seconds - `godot 40`.
## How it works?
At first `all_in_one` scenes opens, execute `_ready` function and exit.
The available time(default 25s) is divided equally between each scene in `alone_steps`.
Then, one by one, each scene is created and after a certain amount of time it is deleted and another scene from the `alone_steps` takes its place.
This project is running in CI with a version of Godot compiled with Address and Undefined sanitizers(`scons use_asan=yes use_ubsan=yes`) so without these options it won't always be possible to detect the bug or get so detailed backtrace.
## Searching for a malfunctioning scene
### Logs
When searching through the logs for the error you may come across something like this
```
Changed scene to res://Rendering/Lights2D/Lights2D.tscn
Test is running now 35 seconds
Test is running now 40 seconds
Changed scene to res://Rendering/Lights3D/Lights3D.tscn
##### CRASH #####
Godot crashed with signal ...
```
This log indicates 2 potentially broken scenes
- `Lights3D.tscn` - crash occured because scene started work
- `Lights2D.tscn` - crash occured because scene was removed
### Autoload
There is only one scene which is loaded when project starts - "Autoload.gd".
It handle exiting project after selected number of seconds
You can freely comment out selected lines in `alone_steps` or `all_in_one` to choose which scenes will run `Start.tscn`.
You can also use `time_to_show` variable to set time how long the project will run.
When opening any scene, automatically time to exit is set.
![Autoloads](https://user-images.githubusercontent.com/41945903/115044743-16582f00-9ed6-11eb-889f-4f07ad6c7d13.png)
If running projet with e.g. this parameters
### File System
The last way is to manually run each scene from the file system to find the one malfunctioning.
![File](https://user-images.githubusercontent.com/41945903/115044733-13f5d500-9ed6-11eb-9364-43a1a142a6f1.png)
## "Safe" fuzzer
A scene that will probably give people a hard time quite often is `FunctionExecutor.tscn`.
This is a fuzzer, but with removed ability to use random argument values (the arguments are identical every time it is run).
When the engine crashes, in logs usually will be something like this:
```
godot 20
#################### SkeletonModification2DPhysicalBones ####################
SkeletonModification2DPhysicalBones.set_physical_bone_node
Parameters [100, ]
ERROR: Joint index out of range!
at: set_physical_bone_node (scene/resources/skeleton_modification_2d_physicalbones.cpp:259)
SkeletonModification2DPhysicalBones.get_physical_bone_node
Parameters [100]
ERROR: Joint index out of range!
at: get_physical_bone_node (scene/resources/skeleton_modification_2d_physicalbones.cpp:265)
SkeletonModification2DPhysicalBones.fetch_physical_bones
Parameters []
scene/resources/skeleton_modification_2d_physicalbones.cpp:186:2: runtime error: member access within null pointer of type 'struct SkeletonModificationStack2D'
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on godotengine/godot/issues
[1] bin/godot.linuxbsd.tools.64s() [0x1e697d8] (/home/runner/work/godot/godot/platform/linuxbsd/crash_handler_linuxbsd.cpp:54)
[2] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fd1ca5b0210] (??:0)
```
Then time is set to 20 seconds so it means that if scenes is 10 (EACH in `alone_steps` array + one for ALL scenes in `all_in_one` array), then each scene will be show for 2 seconds
### Checking All scenes at once
There are two scenes which opens all scenes:
- All.tscn - opens all scenes at once
- Start.tscn - opens each scene one by one
### "Safe" Fuzzer
Available inside `AutomaticBugs` directory, check all methods with specific arguments in allowed classes.
During calculations, to output should be written informations about current classes and executed functions with arguments e.g.
There are some interesting things to discuss here.
This line shows what class we are testing now
```
############### CLASS ############### - PopupMenu
add_icon_radio_check_shortcut
Parameters [[InputEventMouseMotion:12379], [SoftBody:12380], -109, True]
#################### SkeletonModification2DPhysicalBones ####################
```
which method
```
SkeletonModification2DPhysicalBones.set_physical_bone_node
```
and which parameters (here are two arguments - `100`, \` \`(empty string))
```
Parameters [100, ]
```
To make testing easier, each object is created from scratch and by looking at the class definition in Godot - `set_physical_bone_node(int, String)` we can create an expression that is executed on this line:
```
SkeletonModification2DPhysicalBones.new().set_physical_bone_node(100,"")
```
Then you can see errors caused by invalid arguments, which you can ignore if they don't cause other crashes/leaks etc.
```
ERROR: Joint index out of range!
at: get_physical_bone_node (scene/resources/skeleton_modification_2d_physicalbones.cpp:265)
```
At the end we can see Godot's crash log with additional information that tried to use null pointer incorrectly:
```
scene/resources/skeleton_modification_2d_physicalbones.cpp:186:2: runtime error: member access within null pointer of type 'struct SkeletonModificationStack2D'
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on godotengine/godot/issues
[1] bin/godot.linuxbsd.tools.64s() [0x1e697d8] (/home/runner/work/godot/godot/platform/linuxbsd/crash_handler_linuxbsd.cpp:54)
[2] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fd1ca5b0210] (??:0)
```
we can assume that this is caused by calling the function immediately before the crash:
```
SkeletonModification2DPhysicalBones.fetch_physical_bones
Parameters []
```
We can test this by running a function in Godot that looks like this(just add `.new()` after class name and fill function arguments from `Parameters`):
```
SkeletonModification2DPhysicalBones.new().fetch_physical_bones()
```
This fuzzer should in most situations be safe to use, because always use same set of arguments in functions, and objects are cleared before executing next function.
This tool is developed in external repository - https://github.com/qarmin/Qarminer.
## Nodes
Scene simply adds and removes at once all available nodes.
It is used to catch early very obvious and easy to reproduce bugs.
### AIO
This are scenes which only opens once, because there is no need to open it more times - used to check loading of specific types of nodes or executing its `_ready` functions.
## ReparentingDeleting
This is more advanced variation of Nodes scene.
In random order adds, remove and move in scene tree nodes. It may not sound spectacular, but it sometimes allows you to find bugs that are hard to detect.
### MainScenes
Collections of all nodes, which are after some time simply removed and added to scene.
## Others
Scenes like `Physics2D.tscn` or `Lights3D.tscn` are normal scenes with specific types of nodes. They are only used to manually check visual differences between different Godot versions.
### ReparentingDeleting
This scene randomly reparent, add or delete nodes inside it, to check correctness this operations.
![Physics](https://user-images.githubusercontent.com/41945903/115050994-9da8a100-9edc-11eb-99f6-9375ef917be1.png)
### Other Scenes
Each other scenes checks specific types of nodes like lights, rendering, physics, text.
## TODO
- Add physics test - currently blocked by several crashes - https://github.com/godotengine/godot/issues/47440
## 4.0 version limitations
ReparentingDeleting is in 4.0 only Reparenting - bug https://github.com/godotengine/godot/issues/45471
Some scenes available in 3.x branch, but due freezes and long loading times are disabled.
## Epilepsy Warning
Due using by project a lot of functions from each type of Node, screen may flicker, images and objects may change randomly color and size which may lead some users to health problems.
## Problems with project
The project should not cause too many problems in CI when adding and removing features in Godot, since it don't uses too much functions but for example removing a base type e.g. `TYPE_INT` or changes in GDScript(e.g. changing `instance` to `instantiate`) can mess it up.
If you have problem with this project e.g. in CI, just ping me -> @qarmin <- and after that I will try help to fix issues which you have with it or add exception to project.