Improvements to automated testing

[lastmjs]

Improvements

• Test self-referential relations once they are enabled...basically I believe we'll allow multiple relation directives per field
• Order tests do not seem to work going past the second level of relations on relation many
• The disconnect tests (the connect part of the tests) does not seem to be checking for the non-null relation one error
• Relation many arbitrary should create a variable number of relations
• Should we also test multiple relations, updating multiple times? Right now we always start with an empty multiple relation, and then we simply add one item into it
• Test self-referencing relations
• Clear the database between tests?
• Really ensure shrinking is working perfectly
• do we also want to test that not putting a value in doesn't change the value?
  • for example, we want to make sure that not putting a value in the input doesn't set it to null for some reason
• Test multiple mutations or queries at a time
• Do a better test of the append functionality
• Test many to many updates, ensure that removing from one side removes you from the other side...you know, this might already be covered...
• Allow any kind of field omitting, not just nullable types. Everything but the id can be omitted possibly
• So far three major types of tests seem appropriate...general updates, append for blob, and disconnect on relation one and relation many
• I am starting to think the append on bytes should be its own test
• The create and update tests might want to do explicit queries after the mutations to ensure that the state changes were effectual...for example, perhaps the update call reverts for some reason, we might still get the correct result saying that the change has occured, but because we did the mutation and the read within the same update call, we aren't entirely sure that the state has changed. This would happen if you accidentally did a mutation through a query call for example
• proptest::collection::vec should be used for the create and update tests instead of the silly way I am doing it now, the way I am currently doing it is much more complicated
• Do I actually check deletion behavior when a relation many nullable is actually set to null? (I am pretty sure this is being tested, if the relation many is null on the deleted object side, then the object would be deleted successfully and there would be nothing to check. If the object deleted has a relation and the other side is a nullable relation many...
• Read tests with very high levels of relations could be interesting...the read test failed with 5 levels on horses, it created one of the largest GraphQL queries I have ever seen. I am not sure why it failed, but it seemed unreasonably large
• Clean up all code
• Use idiomatic Rust
• Get shrinking to work
• Make tests orders of magnitude faster (best chance is to probably run the tests inside of the emulator)
• Create combination test for limit, offset, order, and search
• Create read test for blob
• Add cross-relational search tests
• the offset and limit tests are so similar that they should really use generics and closures to reuse most of their code
• Abstract out the limit functionality to be used for offset, order, and search. I think the functionality is substantially similar and we should only have to implement a few select pieces, I think mostly in the expected value area
• Get rid of all 'static references if possible, I might be able to just add a 'static generic parameter to the GraphQL parser types, instead of making them 'static references
• There should be a very simple command line or cargo command to run the sudograph tests against your own schema...that would be so cool. You just hit test and it will just start running the tests. In this way each project can have their own assurances about their own schemas, in addition to the random or specific schemas that the Sudograph project will be creating itself. These need to run fast so that people will want to run the tests on their own
• This error seems to creep up very rarely in the order tests: thread 'test_order' panicked at 'called Option::unwrap() on a None value', src/arbitraries/order/order_read.rs:234:70
• For ultimate robustness, we probably want to have many more individual tests...each capability should have an individual test, and then we should have combined tests on top
  • For example, for create we should test creating records with each individual type at a time, just test String in isolation, Boolean in isolation, etc. On top of that we would have the generalized test that we already have...it would be a lot of work, but I think it might be worth it to have the granularity and assurance. But key to all of this is being able to run the tests extremely quickly. They are very slow right now
• Read search tests improvements
  • Having specific and generalized tests here might be really important
    • Test one field at a time and each search operation at a time
    • Test one field at a time and multiple search operations at a time
    • Test multiple fields at a time and multiple search operations at a time
    • This one could be broken down even more, I guess a line has to be drawn somewhere
• limit, offset, order, search
• I think it is very important to have the combination test as well, problems that might not manifest individually could manifest when combined. I might be running into one of these issues with the basic read tests
• It would be amazing to combine all of the tests above into one test as well, which will generate queries with random combinations of limit, offset, order, search at arbitrary levels of the selection set. This could be pretty difficult (think about the complexity of the create and update tests vs the update disconnect and delete tests). The individual tests shouldn't actually be too hard to create, but combining them all could prove difficult. I am not sure if I should just start with the crazy combination test or start with the individual tests. Perhaps I should start with the individual tests and once I have two built, consider how to combine them and work from there. It will be good to have the individual tests so that we can easily pinpoint issues
• Don't forget to test bytes in the selection set
• This is possibly its own test...though I already have basic tests for this functionality in the update tests, and Blob is kind of an extra scalar that I wasn't originally going to include, so perhaps I will skip these tests for now
• Actually I don't think I have any tests for this functionality, I have append and replace tests only, but I don't think I have limit or offset for bytes in the selection set, seems this should just be its own test
• change and/or to use an array in the read search tests, and allow duplicate field values. Right now we are only testing the object syntax, which is weird
• Test search or with array, we are only testing objects
• enums should be added to limit, offset, and order I believe: I have not been testing enums with any of these tests. I don't know how long it has been since I have thought about enums, but we need to add them in
• Panic if any graphql query or mutation returns an error in the order, limit, offset, search tests...well maybe just the mutation create mutation. The problem is if that fails, then the read objects will be empty and the tests will just pass
• Manually test ors and ands within each other, especially empty ors within empty ors
• Order create has some issues with not testing 0 objects to be created...an edge case that we should probably handle but would take some refactoring to figure out