Purpose: Time-of-flight (TOF) been successfully implemented in whole body PET, significantly improving clinical performance. However, TOF has not been a priority in development of dedicated brain PET systems due the relatively small size of the human head, where coincidence timing resolution (CTR) below 200 ps is necessary to arrive at substantial performance improvements. The Brain PET (BET) consortium is developing a PET detector block with ultrafast CTR, high sensitivity and high spatial resolution (X, Y, depth of interaction, DOI) that provides a pathway to significantly improved brain PET. Methods: We have implemented analytical and Monte Carlo models of scintillation photons transport in scintillator segments with the trans-axial cross-section equal or smaller than 3x3 mm2 . Results: The signal amplitude and timing of W mm x W mm x L mm scintillators (1 mm<W<3 mm, 5 mm <L< 30 mm) are strongly influenced by sidewall surface polish and external reflector. Highly polished surfaces provide nearly perfect total internal reflection (TIR), enabling the ultrafast timing performance to be relatively independent of scintillator crosssection. The signal amplitude in such a configuration does not depend on DOI. However, the differential signal from top and bottom SiPM in the dual-ended readout can be used to determine DOI. Using TIR alone, the average of the photon detection times at the top and bottom SiPMs provides a good estimation of the gamma ray absorption time. Averaging ~10 photons starting from 3rd photon produces the shortest CTR for SPTR=50 ps. Conclusions: We established that the advanced silicon photomultiplier designs with high single photon detection efficiency (QE=60%) and high single photon timing resolution (SPTR =50 ps) are critical for achieving ultrafast TOF-PET performance with CTR ~50 ps and ~4 mm DOI resolution.