A manuscript has been published for this project!
Figure: Summary of the intra-trial (3506 wingstrokes) and inter-trial (4914 wingstrokes) prediction errors for each muscle. Known values of precision for each muscle are also given for comparison with spike time prediction errors.
In this project, we built a visual encoder and model of a sensorimotor system based on a recurrent neural network (RNN) that utilizes spike timing encoding during smooth pursuit target tracking. We used this to predict a nearly complete, spike-resolved motor program of a hawkmoth that requires coordinated millisecond precision across 10 major flight motor units. Each motor unit innervates one muscle and utilizes both rate and timing encoding. Our model includes a motion detection mechanism inspired by the hawkmoth’s compound eye, a convolutional encoder that compresses the sensory input into a latent space, and a simple RNN that is sufficient to sequentially predict wingstroke-to-wingstroke modulation in millisecond-precise spike timings. The two-layer output architecture of the RNN separately predicts the occurrence and timing of each spike in the motor program. Intra-trial and same-subject inter-trial predictions on the test data show that nearly every spike can be predicted within 2 ms of its known spike timing precision values. Whereas, spike occurrence prediction accuracy is about 90%. Overall, our model can predict the precise spike timing of a nearly complete motor program for hawkmoth flight with a precision comparable to that seen in agile flying insects. In the neuromorphic sensing and control pipeline, the output of the model directly links with a reduced kernel Hilbert space (RKHS) decoder that transforms precisely timed motor spike trains into aerodynamic forces and torques generated by the hawkmoth.