Many of us navigate life each day with a smartphone. And on that smartphone is an atomic clock of sorts. No it isn’t a “real” atomic clock but given a smartphone is typically connected to cell towers and/or a wi-fi network the time and day on a smartphone is generally correct. I’d be willing to bet many of you use your smartphone as your daily watch. We’ve pretty much come to expect we will always have the correct time (within a few seconds) available at a quick glance.

Servers should be no different. During the past four years I’ve been responsible for leading the technology team at Dataium. We have a fair number of servers which make up our instrastructure, from basic CentOS virtual machines to clusters of database servers running open and closed source database packages. One thing we figured out early on was just how important it was to ensure our various machines have the proper day, time, and time zone configured. There are so many different reasons why this is important and we’ve been bitten several times when we deployed a new machine and forgot to ensure time synching was set up.

In this short post I cover how to install NTP (Network Time Protocol) a daemon which runs on a Linux box and keeps the system clock up to date. I also cover a few things I’ve run into over time in installing and using NTP. This post, like so many of my others, is geared specifically to CentOS but the principles can be applied to other Linux distributions such as Ubuntu.

Installing NTP via yum

First things first, you may need to install NTP on your server. NTP runs as a daemon called ntpd which, depending on how your server was initially created, may or may not be installed. To check, run the following to see if a status is returned for the daemon.

service ntpd status

If you receive a message about ntpd running or in stopped status you already have ntpd installed and can begin using it. The rest of this post may still be valuable to you so feel free to read on.

If ntpd isn’t installed, run the following yum installation command.

yum install ntp

Once installed, starting and stopping ntpd is just as easy as reporting its status.

service ntpd start
service ntpd stop
service ntpd restart (stop/start all in one)
service ntpd status

Configuring ntpd to start on system boot

If you had to manually install ntpd it is likely the service is not configured to start on system boot. While ntpd may be running right now it may not start up automatically if you need to reboot your server. To check this, run the following “check config” command to list all the services configured to start on boot at the various runlevels.

chkconfig --list

Look through the list to find the row for ntpd or, run the following command to filter the list to show only ntpd. When you grep the chkconfig list command you will likely see two results, one for the ntpd daemon and one for the standlone ntpdate utility. More on this utility later in the post.

chkconfig --list | grep ntpd

If you see ntp listed and configured to start at some combination of runlevels 2-5 you are good. The row for ntpd might look something like this: “0:off 1:off 2:off 3:on 4:on 5:on 6:off.” If you don’t see something like this you’ll need to configure ntpd to start on system boot. I recommend setting it to start at runlevels 3, 4, and 5.

chkconfig --level 345 ntpd on

If you make a mistake or decide to change the configuration in the future you can turn off specific runlevels like this.

chkconfig --level 2 ntpd off

At this point, your server should be set up properly and ntpd should be running. Again, you can verify ntpd is running by issuing the service ntpd status command. If you want to check your server’s system date you can run the following command.

date

If you notice your server has an incorrect date or time just wait about 30 minutes and check it again. Chances are ntpd will synchronize your system clock and make the necessary adjustments. Without manual intervention I’ve seen this take as few as 5 minutes or as many as 30 on a system where ntpd was just installed and configured.

Extra stuff

Changing your NTP server pool

When you install NTP on CentOS a configuration file is placed at /etc/ntp.conf. You can inspect this file and take a look at its contents. Toward the middle of the configuration file will be a list of servers your machine will check when synchronizing the clock. For the CentOS distribution the servers are a sub-domain of the public ntp.org website. You should see several rows which look like the following.

server 0.centos.pool.ntp.org
server 1.centos.pool.ntp.org
server 2.centos.pool.ntp.org

There’s nothing wrong with using these synchronization endpoints but sometimes I choose the non-CentOS variants. If you want to do this you can change the above lines to look like this.

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

If you edit /etc/ntp.conf make sure you cycle the ntpd service with service ntpd restart in order for your changes to take affect.

iptables and UDP port 123

ntpd synchronizes the system clock with the remote NTP pools listed in your /etc/ntp.conf file over UDP port 123. While it’s fairly rare I have had some issues with ntpd synching properly over port 123. When its happened the symptoms were the server having an incorrect date/time and the value not correcting itself over time via the ntpd daemon.

If this happens to you there are some things you can do to correct the issue. Again, it’s doubtful this is something you’ll run into.

In a few cases I’ve had to make changes to iptables, a built-in firewall on Linux. If you aren’t running iptables this change wouldn’t be necessary. To see if iptables is running execute the following command.

service iptables status

If iptables is running and you continue to have issues with ntpd correcting your clock you can open up UDP port 123 by editing the configuration of iptables and restarting the service.

vim /etc/sysconfig/iptables

Now, add the following ACCEPT line after the lowest existing UDP line.

-A RH-Firewall-1-INPUT -p udp -m udp --dport 123 -j ACCEPT

Finally, restart iptables for your changes to take affect.

service iptables restart

To see whether your server is listening on port 123 you can use netstat.

netstat -an | grep 123

You can now wait another 5-30 minutes to see if ntpd is able to correct your clock.

Manually correcting your system clock with ntpdate

Another thing you might want to do is manually correct your system clock using ntpdate. This utility should’ve been installed when you installed ntpd via yum. To manually synch your clock you’ll need to stop ntpd and then run ntpdate.

ntpdate pool.ntp.org

Inspecting NTP log messages

Lastly, there are times where you want to check your system logs to see what ntpd is doing. Clock synchronizations and resets are logged to the /var/log/messages log. Inspecting this log file in various ways will allow you to determine whether ntpd is doing anything to your clock. If nothing prints from these commands, ntpd hasn’t synchronized the system clock for some reason.

# This command will only show time resets.
cat /var/log/messages | grep 'time reset'

# This command will show all ntpd log entries.
cat /var/log/messages | grep ntpd

With ntpd running all the time your system clock should never drift more than a second or two and when it does it’ll be corrected periodically.