Many of us navigate life each day with a smartphone. And on that smartphone is an atomic clock of sorts. No it isn't a "real" atomic clock but given a smartphone is typically connected to cell towers and/or a wi-fi network the time and day on a smartphone is generally correct. I'd be willing to bet many of you use your smartphone as your daily watch. We've pretty much come to expect we will always have the correct time (within a few seconds) available at a quick glance.
Servers should be no different. During the past four years I've been responsible for leading the technology team at Dataium. We have a fair number of servers which make up our instrastructure, from basic CentOS virtual machines to clusters of database servers running open and closed source database packages. One thing we figured out early on was just how important it was to ensure our various machines have the proper day, time, and time zone configured. There are so many different reasons why this is important and we've been bitten several times when we deployed a new machine and forgot to ensure time synching was set up.
In this short post I cover how to install NTP (Network Time Protocol) a daemon which runs on a Linux box and keeps the system clock up to date. I also cover a few things I've run into over time in installing and using NTP. This post, like so many of my others, is geared specifically to CentOS but the principles can be applied to other Linux distributions such as Ubuntu.
Installing NTP via yum
First things first, you may need to install NTP on your server. NTP runs as a daemon called ntpd which, depending on how your server was initially created, may or may not be installed. To check, run the following to see if a status is returned for the daemon.
If you receive a message about ntpd running or in stopped status you already have ntpd installed and can begin using it. The rest of this post may still be valuable to you so feel free to read on.
If ntpd isn't installed, run the following yum installation command.
Once installed, starting and stopping ntpd is just as easy as reporting its status.
service ntpd stop
service ntpd restart (stop/start all in one)
service ntpd status
Configuring ntpd to start on system boot
If you had to manually install ntpd it is likely the service is not configured to start on system boot. While ntpd may be running right now it may not start up automatically if you need to reboot your server. To check this, run the following "check config" command to list all the services configured to start on boot at the various runlevels.
Look through the list to find the row for ntpd or, run the following command to filter the list to show only ntpd. When you grep the chkconfig list command you will likely see two results, one for the ntpd daemon and one for the standlone ntpdate utility. More on this utility later in the post.
If you see ntp listed and configured to start at some combination of runlevels 2-5 you are good. The row for ntpd might look something like this: "0:off 1:off 2:off 3:on 4:on 5:on 6:off." If you don't see something like this you'll need to configure ntpd to start on system boot. I recommend setting it to start at runlevels 3, 4, and 5.
If you make a mistake or decide to change the configuration in the future you can turn off specific runlevels like this.
At this point, your server should be set up properly and ntpd should be running. Again, you can verify ntpd is running by issuing the service ntpd status command. If you want to check your server's system date you can run the following command.
If you notice your server has an incorrect date or time just wait about 30 minutes and check it again. Chances are ntpd will synchronize your system clock and make the necessary adjustments. Without manual intervention I've seen this take as few as 5 minutes or as many as 30 on a system where ntpd was just installed and configured.
Changing your NTP server pool
When you install NTP on CentOS a configuration file is placed at /etc/ntp.conf. You can inspect this file and take a look at its contents. Toward the middle of the configuration file will be a list of servers your machine will check when synchronizing the clock. For the CentOS distribution the servers are a sub-domain of the public ntp.org website. You should see several rows which look like the following.
There's nothing wrong with using these synchronization endpoints but sometimes I choose the non-CentOS variants. If you want to do this you can change the above lines to look like this.
If you edit /etc/ntp.conf make sure you cycle the ntpd service with service ntpd restart in order for your changes to take affect.
iptables and UDP port 123
ntpd synchronizes the system clock with the remote NTP pools listed in your /etc/ntp.conf file over UDP port 123. While it's fairly rare I have had some issues with ntpd synching properly over port 123. When its happened the symptoms were the server having an incorrect date/time and the value not correcting itself over time via the ntpd daemon.
If this happens to you there are some things you can do to correct the issue. Again, it's doubtful this is something you'll run into.
In a few cases I've had to make changes to iptables, a built-in firewall on Linux. If you aren't running iptables this change wouldn't be necessary. To see if iptables is running execute the following command.
If iptables is running and you continue to have issues with ntpd correcting your clock you can open up UDP port 123 by editing the configuration of iptables and restarting the service.
Now, add the following ACCEPT line after the lowest existing UDP line.
Finally, restart iptables for your changes to take affect.
To see whether your server is listening on port 123 you can use netstat.
You can now wait another 5-30 minutes to see if ntpd is able to correct your clock.
Manually correcting your system clock with ntpdate
Another thing you might want to do is manually correct your system clock using ntpdate. This utility should've been installed when you installed ntpd via yum. To manually synch your clock you'll need to stop ntpd and then run ntpdate.
Inspecting NTP log messages
Lastly, there are times where you want to check your system logs to see what ntpd is doing. Clock synchronizations and resets are logged to the /var/log/messages log. Inspecting this log file in various ways will allow you to determine whether ntpd is doing anything to your clock. If nothing prints from these commands, ntpd hasn't synchronized the system clock for some reason.
cat /var/log/messages | grep 'time reset'
cat /var/log/messages | grep ntpd
With ntpd running all the time your system clock should never drift more than a second or two and when it does it'll be corrected periodically.
About this post:
This entry was posted by Aaron West on March 29, 2014 at 12:03 PM. It was filed in the following categories: Linux. It has been viewed 13128 times and has 0 comments.