1115

In this paper, we investigate the performance of generalized optical multiple-input multiple-output (MIMO) systems using a deep learning-enabled joint detection scheme. In the generalized optical MIMO system applying both generalized spatial modulation (GSM) and generalized spatial multiplexing (GSMP), a fully connected deep neural network (DNN) is employed for the joint detection of spatial and constellation information. To efficiently train the DDN detector, the received signal after zero-forcing (ZF) equalization is taken as the input while the corresponding transmitted binary bits are used as the output. Our simulation shows that, in a 4 x 4 generalized optical MIMO system with two activated light-emitting diode (LED) transmitters, the ZF-DNN detector can achieve comparable bit error rate (BER) performance as the high-complexity joint maximum-likelihood (ML) detector in the high signal-to-noise ratio (SNR) region for both GSM and GSMP. Moreover, the ZF-DNN detector achieves substantially improved BER performance than the conventional ZF-based maximum-likelihood (ML) detector. Due to the ability to eliminate error propagation, the performance gain of GSMP over GSM is greatly improved by using the ZF-DNN detector in comparison to the ZF-ML detector.