Physicist: This is a classic philosophical question!
For a while, it was generally assumed that without matter distance and time would still exist. After all: why not? Time is time and space is space, and the material things in them are just “actors on a stage”. If you remove all the actors, the stage is still there.
That’s not a proof one way or the other, but it does present a salient point: “what’s the damn problem?”
However, that stance took a hit when relativity came along. First, Galilean relativity, that states that the physics describing how things move around works whether you yourself are moving or not. His example was someone dropping a stone from the mast of a ship: from the perspective of someone on the ship the stone falls in a straight line (along the mast), and from the perspective of someone on the shore it falls along a curved path. Both of those people perceive and describe the world exactly as though their “stage” were stationary around them. Point is; there’s no way to tell whether or not you’re the “correct one” just by chucking rocks about.
The take-away from Galilean relativity is that, for any pair of events that happen at different times, perspectives moving with respect to each other (boat vs. shore) will see them happen at different relative locations. Say you’re floating in a void and two firecrackers go off, one after the other. If you’re moving in the right direction, at the right speed you can “catch” them both and see them both go off right in from of you (it’s important to have protective eye wear in an infinite void). From most perspectives the firecrackers went off in different places and you had to move to catch them, but from yours, you were stationary and they moved so that they would explode in the same place.
Still, there’s no problem with there being some particular “stage” with everything in the universe moving around on it. Heck, the Earth is moving all over the place, but you’d never notice. The important thing is that a given object will take up a given amount of space (have a set size) that everything can agree on, and will experience the same amount of time as everything else. In Galilean relativity, the size of objects, the distance between two events that happen at the same time, and time itself are all fixed. So the dude on the boat and the dude on the shore will always agree on what time it is, as well as how far apart they are at any moment.
The only problem so far is that you can’t tell what’s moving and what’s not. But there’s nothing to say that somebody out there is standing still.
The issues really start rolling in when you switch over to Einstein’s relativity (also known as “relativity”). Both Galilean and Einsteinian relativity state that all of physics works the same independent of movement, but Einstein’s relativity includes the added rule “the speed of light is the same to everything, regardless of how that thing is moving”.
That extra rule has a lot of weird consequences that you have to be some kind of Einstein to figure out.
Just like in Galilean relativity, different observers can disagree about where an event can happen, but unlike Galilean relativity, in Einstein’s relativity they can also disagree on when. Worse than that, the physical size of objects is different, and the duration of events is different, depending on how you’re moving. You genuinely cannot talk meaningfully about much time or space things “actually” take up.
What you can do, is establish a particular “reference frame”. Essentially, you measure time and distance from the perspective of some object (one might say “relative” to the object). In a single reference frame (from the perspective of anything moving at the same speed as the particular object) everything has a size and a position and a time. The unfortunate thing is that as soon as you look at the same things from a different reference frame, sizes, positions, times, and durations all change.
So finally, to the point. If there’s nothing at all in the universe to base a reference frame on, then how do you define distance and time? Keep in mind that you can’t point at a location and say “here” because there’s no you to do the pointing either. You can easily talk about the size and age of an object in the universe, but the “stage and clock” of the universe itself, on its own, has no particular size or age without something to reference.
Or maybe it does!
Impossible to tell for sure.
The clockmaker image was stolen, without pitty, from http://www.post-gazette.com/pg/10034/1032914-325.stm.
The “cloudthing” is from Hertzfeldt’s opus: Rejected.