Thinking, thinking, thinking. Trying to work towards an example.
Spent more time than I meant making sure I understand how gradients act, their covariant nature under coordinate changes. Wrote out a "simple" example, just a 2D transformation of the gradient of a simple function. But it still took awhile. I made an arithmetic, an algebra, and a calculus mistake the first time through; am I ever rusty!
If you are looking for an example of covariance of a tensor or vector, try it:
Now to make neuron-like structures do the math for me ...