The first question you must ask is ‘What is Time?’
As a concept of our experience at least, Time is dependent on Change, without Change, the passage of Time is a meaningless idea.
From a conventional materialist perspective, Change is dependent on physical matter, because for change to occur, there must be something that changes.
From a conventional scientific perspective, the laws of causality bind matter. Our current conventional understanding of causality, despite some strange findings on a quantum level, still maintains that effect necessarily follows cause.
Neurolanis is along the right track, in the sense that it has been shown that the closer you get to the speed of light, the slower Time travels (or to put it perhaps less romantically, the slower change occurs) as Einstein's general relativity theory predicted.
If we want to keep the common sense view that Time is meaningless without change, that the laws of causality are real, and that they apply across the whole of reality, we must accept that time travel is impossible. However, these kind of assumptions do not go unchallenged:
To take the first assumption, that Time is meaningless without change:
The first line of attack is to say that Time is not dependent on matter, and therefore not dependent on change.
There are two ways of looking at time and space, either as empty containers within which matter resides, or as ways in which matter exists relative to itself. Isaac Newton believed that even if the universe was ‘empty’, time would continue pass, and there would still be ‘empty space’. Scientifically and metaphysically, this view has fallen out of favour. Scientifically, because of relativity theory, and metaphysically, because when analysed, such ‘empty’ space is an incomprehensible idea. This establishes Time’s dependent on physical matter.
There are arguments that show weaknesses in the idea of causality, and in applying causality to ALL matter, but I won’t go in to them here.
The other line of attack is to say that the idea of Change itself is incomprehensible, as a number of Greek philosophers believed, but that I shall not go in to here either.
The subject expands exponentially, this is as brief a digest as I could devise, I would recommend reading: