In this work, we present a gaze prediction model for Virtual Reality task-oriented environments. Unlike past work which focuses on gaze prediction for specific tasks, we investigate the role and potential of temporal continuity in enabling accurate predictions in diverse task categories. The model reduces input complexity while maintaining high prediction accuracy. Evaluated on the OpenNEEDS dataset, it significantly outperforms baseline methods. The model demonstrates strong potential for integration into gaze-based VR interactions and foveated rendering pipelines. Future work will focus on runtime optimization and expanding evaluation across diverse VR scenarios.