The representation of the flow of information between neurons in the brain based on their activity is termed the causal functional connectome. Such representation incorporates the dynamic nature of neuronal activity and causal interactions between them. In contrast to connectome, the causal functional connectome is not directly observed and needs to be inferred from neural time series. A popular statistical framework for inferring causal connectivity from observations is the directed probabilistic graphical modeling. Its common formulation is not suitable for neural time series since was developed for variables with independent and identically distributed static samples. In this work, we propose to model and estimate the causal functional connectivity from neural time series using a novel approach that adapts directed probabilistic graphical modeling to the time series scenario. In particular, we develop the Time-Aware PC (TPC) algorithm for estimating the causal functional connectivity, which adapts the PC algorithm a state-of-the-art method for statistical causal inference. We show that the model outcome of TPC has the properties of reflecting causality of neural interactions such as being non-parametric, exhibits the directed Markov property in a time-series setting, and is predictive of the consequence of counterfactual interventions on the time series. We demonstrate the utility of the methodology to obtain the causal functional connectome for several datasets including simulations, benchmark datasets, and recent multi-array electro-physiological recordings from the mouse visual cortex.