Respect returned synapse (and its status code)

[mjurbanski-reef]

This continues from #1803 by adding fix for bittensor ignoring synapse objects returned by forward_fn. Most painful effect of that was that while user may have been convinced synapse.axon.status_code will be propagated to the client, it in fact, was not.

I have noticed that @ifrit98 was also hit by this bug at some point:
https://github.com/ifrit98/storage-subnet/blob/b92052e0c33aea0d3d04ce707e183964e442c1b5/neurons/miner.py#L776-L780

Fixes&improvements

• forward_fn return synapse is no longer ignored - before, if we for example did synapse = synapse.copy() at the beginning of that function and modified only a copy, you could spent hours no end without seeing why your axon doesn't actually return that to the client (dendrite); don't ask me how I know :)
• synapse.axon.processing_time is now also sent to the end client (before it was only visible on the server)
• added some tests testing the axon-fastapi interface proving that the proposed change does work (they use fastapi mock server)
• SynapseException risen from forward_fn allow to set custom error and status codes
• other internal exception are no longer shown to the remote client as they could disclose confidential information (same thing as

Overall this should improve debugging experience a lot. It sure did for me.

[gus-opentensor]

Thanks @mjurbanski-reef ! we will take a look

[mjurbanski-reef]

@gus-opentensor I have rebased to current staging. IMO it would be good to include in v7.0 since it is a substantial change, also it hangs here for 2 weeks already and it would be a shame to let it waste away ;)

Since I had some time to play with it, I additionally, fixed the exception handling, so now if SynapseException is risen from forward_fn it will respect its value as well, but perhaps more importantly, it no longer will return exception as "raw" to the client.
This was a potential security issue as for example

yaml.safe_load("{password=SuperConfidentialPassword")

if triggered under forward_fn would happily return to the remote client the 'while parsing a flow mapping\n in "<unicode string>", line 1, column 1:\n {password=SuperConfidentialPassword\n ^\nexpected \',\' or \'}\', but got \'<stream end>\'\n in "<unicode string>", line 1, column 36:\n {password=SuperConfidentialPassword\n ^' error message containing the password.

[mjurbanski-reef]

Seems on dendrite side the error message was ignored as well, so fixed that as well and added some more tests

[mjurbanski-reef]

another cool application possible after these changes is using BackgroundTasks from fastapi https://fastapi.tiangolo.com/tutorial/background-tasks/ in forward_fn

[mjurbanski-reef]

@gus-opentensor so this PR is 3 weeks old, any hope for it to be included in v7?

[mjurbanski-reef]

@gus-opentensor thank you!