In my experience most people, including most developers but perhaps excluding many HN readers, don't understand what the timezone setting does. The time on all contemporary computers is always Unix Epoch Time, which is close to (but not exactly) UTC. Setting your timezone doesn't actually change anything about how time works on the computer. It just changes how time is printed. That's it.
So the second problem is that our text-based server world, for all its benefits, doesn't cope well with per-user parameters. I mean ... timezone support on Unix is basically buggy. It doesn't matter what TZ I set in my .profile, nevertheless the timezone of logs is in whatever TZ the server is running in. That's pretty stupid. And even simple calculations like "how many seconds elapsed between this log entry and this one" - which is really common - are very difficult, regardless of the TZ setting.
In my own personal perfect world, all logs would be printed in the first column of log files using the epoch time in seconds, and we'd have a command which would convert from the epoch time to the local timezone. So we could do something like:
...in which case I would be able to choose the timezone and format of the timestamps to meet the needs of the job i was doing.
So for me, this whole argument over UTC is dumb. It doesn't matter what timezone the system is set to; any zone can be "default". The real problem is that different users have different time formatting requirements, and to date we haven't had tools which deal with that properly.
(even logging all timestamps consistently in ISO format would be a big step up...)
> In my own personal perfect world, all logs would be printed in the first column of log files using the epoch time in seconds, and we'd have a command which would convert from the epoch time to the local timezone.
For a small number of input formats `ts` from Joey Hess' moreutils¹ can do this. Its `-r` option pulls dates out of its input and reformats them, either as a relative timestamp by default(33m57s ago) or a user-specified format if given.
Your example becomes `tail | grep | ts -r %FT%T` or something like that.
It supports syslog format, a subset of ISO-8601, and a few others. It doesn't support epoch in seconds though, but Joey is nice to interact with if you were to offer a patch.
---
This, among other reasons, is why I like all my logging going through systemd's journal nowadays. With journalctl you have a good choice of output options for date formatting or dumping as JSON if you want more.
There are major pitfalls with using local time, unfortunately. The big one is printing timestamps without the time zone which gives you ambiguous timestamps around the daylight savings time switch. But it's absolutely fine to record timestamps in local time as long as you record the time zone too. You can even randomly switch your time zone throughout the day if you want. Doesn't matter. Your theoretical tool would still work as long as a standard format like ISO8601 is used (also unambiguous wrt leap seconds, unlike Unix).
I think the insistence on UTC is a case of "a little knowledge". What happens is some developer somewhere has to learn the hard way about time zones. Now he could either learn about the problem in depth or he finds the shortcut solution: just use UTC. He's now unlikely to ever get bitten again, but he never really learnt what the problem was. He now tells other developers to use UTC and many will blindly follow.
In that way it's a bit like the whole insistence on spaces over tabs. Another instance of "a little knowledge" and losing the power of the tab character as a result.
For me the biggest plus of UTC is that the values do not jump back in time in spring during the switch to day light saving times. Surely there are tools that allow to compare/sort time properly as long as there is TZ in the date, but it is easy to forget to use them especially under the pressure. Then those quick-and-dirty scripts ends up as a permanent utility in production, then on the first of April (or the first of March depending on the country) one spends a lot of time trying to understand why the log merge script fails for the previous months. I really prefer to avoid a chance of that in the first place.
Now, as Europe got some sanity and will abandon time shifts the next year, local time as long as as it comes with time zone will be OK. But out of habit I guess I will continue to configure servers to use UTC.
The thing is, time itself never jumps back. If you correctly encode your timestamps then you will never experience a discontinuity even if you use a local time zone. In the UK on the 29th March 2020 the time was 00:59:59 GMT and one second later it was 02:00:00 BST. There is no discontinuity there. 02:00:00 BST is 01:00:00 GMT. The mistake is to think the time zone is not part of the value. A "naïve" datetime (as the Python standard library calls it) is simply an incorrect way to store a timestamp.
Unfortunately it is Unix time that has a real discontinuity. This happens around leap seconds when Unix time will jump backwards or forwards by a whole number of seconds. This is actually the fault of UTC, though. Leap seconds are an abomination. Unix had the right idea all along which is to just count a number of seconds since an epoch and not worry about the movements of the solar system.
Time is so often done wrong because we're using a bare number to represent time, and then just assuming what that number actually means. It's like the bad old days of text, when you'd get a "string", but you'd have no idea of what encoding it used, so everyone would just assume something. We solved that by settling on UTF-8 for everything - which is fine - but we can't do the same for time, because time zones aren't going away.
It's gotten so bad that I've made binary [1] and text [2] formats that handle all cases and have sane defaults (yes, even ISO8601 gets it wrong).
A time value without accompanying time zone data is just a bug waiting to happen. Once you add unambiguous timezone information to the time structure, getting correct time information every time is just a matter of making conversion libraries for language X. No more guesswork, no more bugs.
> We solved that by settling on UTF-8 for everything - which is fine - but we can't do the same for time, because time zones aren't going away.
What the GP is suggesting is exactly doing the same thing for time. If the internal format is always UTC, that doesn't mean timezones are going away for the user, just that they shouldn't show up in the data layer.
Which is fine for past events, but doesn't solve the issue of future or repeating events. There are actually three main kinds of time: absolute time, fixed time, and floating time [1]. Storing everything in UTC only solves for situations involving absolute time.
Time zone designators can only be represented as value offsets from UTC, which means that it can't refer to an actual location. Time zones change for political reasons all the time, so using ISO8601 for future events becomes buggy. If Germany changes their policy on Daylight Savings, ISO8601 times such as 2021-08-02T09:00:00+0200 will have a 50% chance of suddenly becoming wrong (because they no longer refer to Berlin time).
Another problem with ISO8601 is that omitting the time zone info implies floating time rather than UTC time, and this causes no end of bugs in all implementations.
When I finally gave up on it, I came up with the following alternative:
Time zone data omitted (implies UTC):
2019-01-23/14:08:51.941245 = January 23, 2019, at 14:08:51 and 941245 microseconds, UTC
Area/location based timezone:
1985-10-26/01:20:01.105/America/Los_Angeles = October 26, 1985, at 1:20:01 and 105 milliseconds, Los Angeles time
Latitude/longitude based timezone:
5192-11-01/03:00:00/48.86/2.36 = November 1st, 5192, at 3:00:00, at whatever is in the place of Paris at that time
> The time on all contemporary computers is always Unix Epoch Time, which is close to (but not exactly) UTC. Setting your timezone doesn't actually change anything about how time works on the computer. It just changes how time is printed. That's it.
Sure, but the only thing users normally see is that printed representation. What's the actual problem you see here?
> I mean ... timezone support on Unix is basically buggy. It doesn't matter what TZ I set in my .profile, nevertheless the timezone of logs is in whatever TZ the server is running in.
I don't see what's buggy there. If you set a TZ in your .profile, logs produced by processes within the context of that .profile will be in your desired timezone. Processes outside that context look at the system timezone, configured using /etc/localtime.
The UTC system time crowd basically just wants all logs (that don't explicitly specify further) to be in a standard timezone that does not move, so that it can be reasoned about without a lot of context. Your perfect world with an epoch time in the first column does exactly that, with the downside of not meaning anything to a human without the timestamp-conversion command.
I agree we need better tools to deal with timestamps after the fact.
The "bug" -- it's a strawman bug, actually im gonna go with strawbug :) -- is that TZ is supposed to specify what timezone I want to see times written in, and yet when I look at logs the time is not in the timezone I've chosen.
So my ideal would be to be able to set TZ and see logs in TZ time.
My ideal, to use epoch time instead of local time in logs, is because in a large number of cases I'm more interested in the relative time of log entries than the absolute time. Computing relative time (seconds intervals) between log entries is error prone.
I suppose in a sense I'm in both the UTC and non-UTC camp. On reflection, I agree that server logs should be in UTC. I just don't think I should have to read them in UTC or personally operate in UTC.
The ts tool mentioned above sounds perfect, and I'm gonna start playing with it...
Last time I checked three years ago, the UEFI spec states that the hardware clock (rtc) is set to "current local time". UEFI runtime service's GetTime() includes TimeZone value in minutes offset from UTC, and a Daylight bitmask.
Windows, Linux, macOS, all set the timezone value to 0x07FF which is defined as EFI_UNSPECIFIED_TIMEZONE which is to be treated as local time (but without a timezone).
In effect, it's the same mess as BIOS. It shows how specs can just be thwarted.
There must be. I dual-boot between Ubuntu and Windows 10, and whenever I boot up Windows, the clock is behind one or two hours (DST-depending), until it re-syncs itself from the Internet after a while. Ubuntu seems immune, though. So either Ubuntu is smarter about reading the hardware clock, or Windows side writes to the clock differently than it reads from it.
I haven't played with Windows since the 90s and just assumed that would have been fixed by now! It seems really silly to set the RTC to local time. Especially given that most devices are mobile these days. Put it to sleep in Melbourne and it wakes up in Singapore...
>In my experience most people, including most developers but perhaps excluding many HN readers, don't understand what the timezone setting does. The time on all contemporary computers is always Unix Epoch Time, which is close to (but not exactly) UTC. Setting your timezone doesn't actually change anything about how time works on the computer. It just changes how time is printed. That's it.
This applies to e.g. Linux.
But the RTC in your computer might store localtime. I do recommend using UTC there no matter what. If dual-booting windows, this needs a little extra work[0].
I use vim to browse most logs and have a cheap workaround: a one-liner function with keyboard binding to print the word under cursor converted to local time.
It usually starts like this " but we're all in <tz> so we should just use that so no one needs to do any maths."
Then later when operations spread to elsewhere it becomes "but most stuff is still done in <original tz> so we will still use that and everyone in other places will just need to deal with it and use our 'real' tz."
Then when you are operating in 3 or 4 or 5 or more locations it is too late to change anything and you are screwed because now you are in some provincial timezone for 50% of your systems - complete with clock changes - and the rest are a mess of either in UTC or some other TZ local to that server with their own different clock changes to the original TZ. Chaos.
And you can guarantee that none of the log files have TZ in their timestamps because of that original "we're only in one TZ" mindset so why bother saying what TZ it is... From then on every time you look at a log or a database or a web UI or whatever you need to double guess the timezone.
For those that come after you, please please please if you can't use UTC then at the very least always show the timezone in your log files/UIs/etc even of it seems redundant at the time - in the future someone handling an incident or trying to fix a P1 bug on a Friday afternoon will thank you.
The reason I would argue for always using UTC is that local time in a time that has timezones isn't clearly unambiguous. In fall, when you set the clock back an hour, suddenly the time 02-03 in the morning happens twice. So if you have an event at 02:13, it could have happened at either of the two 02:13's that happened during that night, and since you are using server local time, you won't necessarily be able to know which one afterwards.
Whether being able to figure this out afterwards is important depends on your use case, of course.
UTC also has this problem, to a lesser degree. The length of its second is determined atomically, but it’s also guaranteed to stay within 0.9 seconds of solar time. Individual seconds are either skipped or repeated to maintain this guarantee in the face of solar clock drift.
One issue with having your servers not in UTC will be that you're always wondering whether other software e.g. logging software is using UTC or local time. Also, in your database you should always use UTC if you have daylight saving time or else you might have surprising issues where records are out of order by date during time changes. Also, what happens when you start a collaboration with a university in another country and suddenly have users in another time zone. As someone who's worked on many projects with timezones I've seen many people make arguments for not using UTC. Something always comes along that makes them wish they'd used UTC. If you use UTC for everything then you're always sure it's UTC, and it's easy to make the conversion. Otherwise you're always unsure and the costs of uncertainty can be very high indeed.
Unix does the sensible thing and records timestamps in seconds since the epoch. The epoch is always 1970 January 1 00:00:00 UTC. Timestamps in logs and sent over the wire in that format then don't have a timezone encoded because the "timezone" only gets applied when converting to a format that encodes time of day. When used as a wire format this also allows every device to display the time in its own timezone.
> Unix does the sensible thing and records timestamps in seconds since the epoch.
If only, but unfortunately that's not how UNIX time works. Instead its more like number of days since epoch * 86400 + seconds since midnight. Which is 27 seconds different than simple seconds since epoch count. This is why I would advise against using UNIX timestamps for anything critical because they are fundamentally ambiguous
> Instead its more like number of days since epoch * 86400 + seconds since midnight. Which is 27 seconds different than simple seconds since epoch count.
Yeah, the people who decided to do that were Wrong.
Fortunately it only really matters during leap seconds, and nothing stops you from doing it the right way and having your timestamps in actual seconds since epoch and then converting to "Unix time" or anything else you need. See e.g. libtai:
Time stamp + TZ is a useful feature if you want to store TZ along with the time stamp. But I think that’s nearly always a bad approach. The only part of your system that needs the time stamp converted into a TZ is your human user. Performing that conversion on the presentation layer will almost always lead to less complexity.
The most common situation I know of where you store the time with its original TZ is when dealing with recurring times (opening/closing items, appointments), as you need to follow the original DST/summertime the user was in.
Which also requires you to store the TZ location (ie city) and not just the name or offset of the timezone itself.
Yeah, I meant to put a “usually” qualifier in that previous comment. Rostering has the same problem. Event scheduling can also run into similar problems. The best solution really depends on a lot of context specific details, but most of the better solutions I’ve personally seen have put TZ in a seperate field, rather than using TZ-typed timestamps, and then use the TZ when required instead of as the default storage format.
For recording when something happend, that makes some sense. Though if you want to report the day of an event, it is still neccesary to include a timezone.
For planned events, it becomes really hard to determine what to do when changing locale.
Not sure about all different databases but Postgres, for example, always stores datetimes in UTC and converts them back to a display timezone when outputting the values[0].
> For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time, traditionally known as Greenwich Mean Time, GMT). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone.
You're assuming that the "timestamp with time zone" type is always used. I've worked on an application where it was assumed/required that UTC was configured on the server level, so regular timestamps were used. Didn't agree with that at all, but it happens.
I have my doubts, having witnessed the repeated confusions between americans (who mindlessly spew out times in like EST, hey, it's called "standard") and non-americans who then try to convert using google which was not always correct, esp. when they forget about am/pm.
A lot of systems let you choose what timezone you want to use in their UI and you can choose to use your local timezone or UTC. We log everything in UTC but I can still filter or even display logs with my local timezone without any issues or major complications.
Having clocks around for local and UTC and CET and EST/PST and JST means you internalize the offsets really quickly and get a good feel for the ball spinning in space. Make it so that whenever you see your local clock, you see all the clocks.
The best reason for running a system in UTC is that you do want to present dates & times in a user preferred timezone. When you have users in more than one timezone as you do with any internet service, then you substantially reduce your chances of errors if you only convert once and not between two non-UTC timezones.
Edit: The other thing about examining logs is that I don't care what local time it occurred. Once an incident is identified all I care about is the time of the incident which can all be in UTC, my mind is at those hours while reading logs from every source.
The only sensible alternative to UTC for the system time is TAI, but I don't know of an OS that lets you use TAI for everything. Perhaps some specialist OS used for satellites? Being able to take the difference of two numerical values to determine the elapsed time, without having to worry about leap seconds, would be so much simpler.
You could convert to UTC for display purposes, when possible. It's not always possible for times in the future, but that's a concept people are already used to: you can't reliably convert a future time stamp between local time zones, either.
You should always set it to UTC regardless. User interfaces can use local time zone. Even if you're a small mom-n-pop shop, good engineering is long term planning. Let's say you get some 3rd party managed appliance and you work with their techs to troubleshoot an issue, or host something in the cloud but all the stackdriver/cloudtrail is not on your local time. Just use UTC until you have a specific reason not to,even then do it at the user/setting level not a system wide default.
I’d argue that the only time you shouldn’t use UTC is when you live in UTC. The point is mainly to ensure that your code and institutional processes handle time zones correctly by ensuring nothing accidentally works correctly if they don’t.
Whatever you do pick, you definitely want to avoid Daylight Savings Time because it’s not just twice a year - it’s a different twice a year depending on which years (laws enacted it, and have moved it multiple times).
True. People I correspond with in the UK have the tendency to just say "1PM GMT" all year round, and I have to infer they're actually referring to "whatever the local time is in the UK at the moment", not specifically UTC+0 in summer.
The UK had "Double Summer Time" during WW2, and I believe again in the 1970s. Granted that was before computers but still.
When we were implementing an ERP solution to a manufacturer based in PST and factories situated in China, we faced period close issues since our servers were in PST timezone and factories in China had to wait for around half a day to have the next period open. Switching to UTC has actually made the different timezones overlap and reduced overtime efforts. Anyone here faced similar issues?
Unfortunately, it seems most people don't understand UTC. For a brief period, when talking to customers and other employees who I knew were in another timezone, I would always refer to times in UTC, e.g. 14:30 UTC. A lot of them didn't understand and either got the maths wrong or thought I was referring to their timezone, which is why I stopped doing it. They would rather use their timezones or mine.
One issue I’ve had with having my local machine on UTC is that people in the same time zone won’t always specify the time zone and things like mail clients will guess and automatically convert times.
I gave up after missing meetings a couple times this way.
Many universities these days have students scattered around the world in various distance education programs, or have students/staff/faculty at various other locations doing research in collaboration with other labs, facilities, etc.
Then there are their research partners, which again may be scattered around the world, etc.
So, no not customers in the traditional sense, but they depend on University services, and many of them are paying for the privilege.
A trick I learned a very long time ago — I set all my servers to Arizona time. They don’t have daylight savings time.
Since I’m on the west coast, 7.5 months of the year we’re in the same time zone and for the other 4.5 it’s not hard to subtract 1. Crons don’t really need adjustment with the time change in pacific because off peak is still off peak.
In my experience most people, including most developers but perhaps excluding many HN readers, don't understand what the timezone setting does. The time on all contemporary computers is always Unix Epoch Time, which is close to (but not exactly) UTC. Setting your timezone doesn't actually change anything about how time works on the computer. It just changes how time is printed. That's it.
So the second problem is that our text-based server world, for all its benefits, doesn't cope well with per-user parameters. I mean ... timezone support on Unix is basically buggy. It doesn't matter what TZ I set in my .profile, nevertheless the timezone of logs is in whatever TZ the server is running in. That's pretty stupid. And even simple calculations like "how many seconds elapsed between this log entry and this one" - which is really common - are very difficult, regardless of the TZ setting.
In my own personal perfect world, all logs would be printed in the first column of log files using the epoch time in seconds, and we'd have a command which would convert from the epoch time to the local timezone. So we could do something like:
...in which case I would be able to choose the timezone and format of the timestamps to meet the needs of the job i was doing.So for me, this whole argument over UTC is dumb. It doesn't matter what timezone the system is set to; any zone can be "default". The real problem is that different users have different time formatting requirements, and to date we haven't had tools which deal with that properly.
(even logging all timestamps consistently in ISO format would be a big step up...)