Troubleshooting

Common troubleshooting and log inspection

Check the Logs

If things are not behaving as expected, a good first step is always to check the logs. There may be messages logged by Guacamole or guacd which describe exactly what is failing, or which will at least point you in the right direction toward resolving a problem.

Advanced Linux Install method

Both Guacamole (running beneath Tomcat) and guacd log to the systemd journal, which is viewed using the journalctl command. By default, simply running journalctl will allow you to page through all messages logged to the systemd journal, including those from the numerous other services running on a typical server. Options can be provided to narrow these logs to messages from Tomcat or guacd.

In general:

• The Guacamole logs will be useful when debugging issues with web application authentication (database, LDAP, Active Directory, etc.). Depending on the nature of the issue, the Guacamole logs may be useful when debugging connectivity issues, particularly between the browser and Tomcat, however only guacd will be directly aware of connectivity issues related to the underlying remote desktop connection.

• The guacd logs will be useful when debugging issues with remote desktop behavior, issues related to authentication with the remote desktop (not authentication with the web application), or issues related to guacd accessing the file system (session recording, file transfer).

Accessing the Guacamole logs (kcm-guacamole-standalone)

When installed using the kcm package, the Guacamole web application will log its messages to syslog. To view the log messages from Guacamole specifically, specify the "-t guacamole" option to restrict journalctl to showing only log messages from the "guacamole" syslog identifier:

$ journalctl -t guacamole

Once open, the arrow keys, page up/down, etc. can be used to scroll through the Guacamole logs. You may need to scroll to the right to see the details of lengthy messages. Alternatively, the logs can also be sent to a file using the standard output redirection provided by the shell:

$ journalctl -t guacamole > my-guacamole-logs.txt

Accessing the Tomcat logs (manual deployment of kcm-guacamole)

The Guacamole web application will log its messages through Tomcat. To view the Tomcat logs specifically, specify the "-u tomcat" option to restrict journalctl to showing only log messages from Tomcat's systemd unit:

$ journalctl -u tomcat

Once open, the arrow keys, page up/down, etc. can be used to scroll through the Tomcat logs. You may need to scroll to the right to see the details of lengthy messages. Alternatively, the logs can also be sent to a file using the standard output redirection provided by the shell:

$ journalctl -u tomcat > my-tomcat-logs.txt

Accessing the guacd logs

The guacd service uses syslog to log its messages. To view the log messages from guacd specifically, specify the "-t guacd" option to restrict journalctl to showing only log messages from the "guacd" syslog identifier:

$ journalctl -t guacd

Once open, the arrow keys, page up/down, etc. can be used to scroll through the guacd logs. You may need to scroll to the right to see the details of lengthy messages. Alternatively, the logs can also be sent to a file using the standard output redirection provided by the shell:

$ journalctl -t guacd > my-guacd-logs.txt

Connection Issues

If Keeper Connection Manager is hanging during startup, or unable to connect to any hosts, one thing to quickly check is if the Linux machine is capable of generating enough entropy for random number generation. To resolve this, we recommend installing the haveged package.

These packages can be installed using the commands below (CentOS / RHEL):

sudo yum install epel-release
sudo yum install haveged
sudo systemctl start haveged
sudo systemctl enable haveged

Installation Issues

Packages cannot be downloaded from the Keeper Connection Manager repository

Verify that your server can access the internet

Depending on the configuration of your network and server, internet access may be unavailable. If this is the case, the site hosting the repository will not be reachable and the packages will not be able to be downloaded. You will need to adjust the configuration of your network/server to ensure the repository can be accessed. Alternatively, if your organization maintains its own internal set of repository mirrors, you can configure your repository to mirror ours and point your server at your mirror instead.

Dependency errors block installation of the packages

Verify that you are using a supported Linux distribution

Keeper Connection Manager supports CentOS and RHEL for the Keeper Connection Manager product. The packages should not be installed under other Linux distributions, regardless of whether those Linux distributions also use RPM. For more information, please see the overall list of supported distributions in our scope of support. If you are unable to deploy a CentOS or RHEL server, you can still install Keeper Connection Manager by leveraging our Docker images.

Verify that your version of CentOS or RHEL is supported.

Keeper Connection Manager supports specific versions of CentOS and RHEL. It is our intent to support all non-EOL versions of CentOS and RHEL, however this may not be the case for relatively new releases of these distributions, particularly of those releases introduce major differences in the handling of packaging, repositories, or in the availability of dependencies. Please see the list of specific CentOS and RHEL versions supported in our scope of support.

Verify that the repository you are using matches your version of CentOS or RHEL

It is not possible to use the EL6 repository on CentOS / RHEL 7, nor is it possible to use the EL7 repository on CentOS / RHEL 6. The version of the repository must match the version of CentOS or RHEL for the dependencies of the packages, the versions of the APIs used by the packages, etc. to be correct.

Check whether EPEL is required for the package in question

Older versions of CentOS and RHEL, particularly version 6, require the EPEL repository for fundamental dependencies of the Apache Guacamole stack like libwebp. On CentOS / RHEL 6, the EPEL repository must be installed before the packages that make up Keeper Connection Manager can be installed. Newer versions of CentOS and RHEL do not strictly require EPEL. For CentOS / RHEL 7, you will only need EPEL if you are installing support for telnet.

Web application reachability and behavior

Cannot connect to http://SERVER:8080/ after installation

Verify that the firewall is not blocking access

CentOS and RHEL come with the "firewalld" firewall by default, typically allowing inbound connections only to port 22 for SSH. If firewalld is enabled on your system, and you wish to access to port 8080, you will need to either temporarily disable the firewall (if you are only testing whether things work as expected) or allow direct access to port 8080 (if another server on your network will be providing SSL termination, and this server is not publicly visible).

To disable the firewall temporarily:

$ sudo systemctl stop firewalld

To allow direct access to port 8080:

$ sudo firewall-cmd --zone=public --add-port=8080/tcp --permanent
$ sudo firewall-cmd --reload

Verify that the server in question is reachable

If your server is on a private network, it may be the case that the machine you are testing from is simply unable to reach that network. You may need to move to a machine within the same network to perform your tests, or establish a temporary tunnel or bridge between the networks to cause the server to become reachable.

Verify that Tomcat is indeed running

The Apache Guacamole web application is run using Apache Tomcat. Installing Tomcat is part of the Keeper Connection Manager install procedures, and the Tomcat service must be running for the Guacamole web application installed as part of Keeper Connection Manager to become reachable. If unsure whether Tomcat is running, the easiest way to check is using systemctl:

$ sudo systemctl status tomcat

If Tomcat is not running, it will need to be started for the web application to be accessible via port 8080. The service should also be set to automatically start on boot, such that the service will come back online after a server restart.

To start Tomcat manually:

$ sudo systemctl start tomcat

To configure Tomcat to start automatically when the server next boots:

$ sudo systemctl enable tomcat

Verify that the web application (guacamole.war) is deployed

While we only strictly support Apache Tomcat, the packages provided by Keeper Connection Manager do not assume that you will be using Apache Tomcat. The file containing the web application, guacamole.war, therefore must be manually deployed after Tomcat is installed. If the web application is not deployed, Tomcat will accept connections to port 8080 but will report errors when you attempt to access Guacamole.

The web application is deployed by creating a symbolic link to /opt/keeper/share/guacamole/guacamole.war within /var/lib/tomcat/webapps, as documented by the installation instructions. If deployed correctly, a directory listing of /var/lib/tomcat/webapps should show a symbolic link to guacamole.war, even if the name of the link differs from the file it points to. For example, if deployed as "guacamole":

$ ls -l /var/lib/tomcat/webapps
lrwxrwxrwx. 1 root root 34 Mar 29 20:20 guacamole.war -> /opt/keeper/share/guacamole/guacamole.war
$

Or, if deployed to the root path:

$ ls -l /var/lib/tomcat/webapps
lrwxrwxrwx. 1 root root 34 Mar 29 20:20 ROOT.war -> /opt/keeper/share/guacamole/guacamole.war
$

Cannot connect to https://SERVER/ after setting up SSL termination

Verify that the firewall is not blocking access

CentOS and RHEL come with the "firewalld" firewall by default, typically allowing inbound connections only to port 22 for SSH. If firewalld is enabled on your system, and you are using the same system to provide SSL termination, you will need allow direct access to the standard HTTPS port (443). To allow direct access to port 443:

$ sudo firewall-cmd --zone=public --add-service=https --permanent
$ sudo firewall-cmd --reload

Note that there is no need to expose direct access to port 8080 if the server will be serving as SSL termination for your deployment. Port 8080 would need to be available only locally, to be used strictly internally by the proxy.

Check whether SELinux is denying proxy connections to Tomcat

By default, the SELinux system will block both the Apache HTTP server and Nginx from establishing outbound network connections, including TCP connections to an internal instance of Apache Tomcat. If this is the case, you may see AVC denials within the SELinux audit logs (/var/log/audit/audit.log), and SELinux must be configured to allow these connections by setting the httpd_can_network_connect boolean to "1":

$ sudo setsebool -P httpd_can_network_connect 1

Verify that the reverse proxy is running

Following packaging best practices, neither the packages providing the Apache HTTP server nor the packages providing Nginx will start their corresponding services by default. They must be manually started and manually configured to start automatically on boot. If this has not been done, there will be no service listening on port 443, and connections will either timeout or be refused entirely.

To start the Apache HTTP server and configure it to start automatically on boot:

$ sudo systemctl start httpd
$ sudo systemctl enable httpd

To start NGINX and configure it to start automatically on boot:

$ sudo systemctl start nginx
$ sudo systemctl enable nginx

Login page does not display

Verify that the "tomcat" user Is part of the "guacamole" group

The Keeper Connection Manager packages create a "guacamole" group for controlling access to files that should be readable only by the Apache Guacamole web application. Configuration files specific to Guacamole, like /etc/guacamole/guacamole.properties, have their permissions set such that only root and members of the "guacamole" group have read access. As it is Apache Tomcat which runs the Guacamole web application, and the Tomcat service runs as the "tomcat" user, the "tomcat" user must be a member of the "guacamole" group:

$ sudo usermod -aG guacamole tomcat

If this is not done, Guacamole will not be able to read its own configuration files, and web application startup will fail.

Check whether SELinux is denying access to the database

By default, the SELinux system will block Apache Tomcat from establishing outbound network connections to databases, including connections to local database instances. If this is the case, you may see AVC denials within the SELinux audit logs (/var/log/audit/audit.log), and SELinux must be configured to allow these connections by setting the tomcat_can_network_connect_db boolean to "1":

$ sudo setsebool -P tomcat_can_network_connect_db 1

The database initialization scripts may not have not been run

Apache Guacamole's database must be manually initialized using the SQL scripts provided. If these scripts have not been run, the tables required by Guacamole will not exist, and its attempts to query these tables will fail. If using MySQL, verify that you have indeed run the MySQL initialization scripts as documented. If using PostgreSQL, verify that you have run the PostgreSQL initialization scripts as documented, and that you did so prior to granting the required database permissions.

(If using PostgreSQL) Check whether PostgreSQL is configured to expect "Ident" authentication

If PostgreSQL has been installed locally, on the same server as Apache Guacamole, the default configuration of PostgreSQL will prevent Guacamole from authenticating with a username and password. The mechanisms used by PostgreSQL for authentication depend on the source of the database connection. By default, PostgreSQL is configured to expect "Ident" authentication rather than a username and password, which will result in Guacamole being unable to connect to PostgreSQL.

Check the contents of /var/lib/pgsql/data/pg_hba.conf, looking for one or more lines which associate IPv4 or IPv6 loopback addresses with "Ident":

host    all     all     127.0.0.1/32    ident
host    all     all     ::1/128         ident

If found, the keyword ident should be changed to md5 to allow username/password authentication from these addresses:

host    all     all     127.0.0.1/32    md5
host    all     all     ::1/128         md5

PostgreSQL will then need to be restarted to apply these changes:

$ sudo systemctl restart postgresql

(If using MySQL) Check whether MySQL is configured with default anonymous users

If MySQL or MariaDB are installed locally, on the same server as Apache Guacamole, the default configuration may prevent Guacamole from authenticating due to the way that MySQL and MariaDB handle authentication. If an anonymous user is defined for the same hostname/address, authentication using a non-anonymous user and password from the same hostname/address will fail.

This can be checked by querying the MySQL user table directly:

SELECT Host, User FROM mysql.user;

Any users with empty usernames in the results of the above query are anonymous users which may block authentication from succeeding:

+---------------------+----------------+
| Host                | User           |
+---------------------+----------------+
| %                   | guacamole_user |
| 127.0.0.1           | root           |
| ::1                 | root           |
| the.server.hostname |                |
| the.server.hostname | root           |
| localhost           |                |
| localhost           | root           |
+---------------------+----------------+

Dropping those users should allow non-anonymous authentication from those same hosts to succeed:

DROP USER ''@'localhost';
DROP USER ''@'the.server.hostname';

Verify the necessary database permissions have been granted to the database user

Apache Guacamole requires SELECT, INSERT, UPDATE, and DELETE privileges on all tables within its database. For PostgreSQL, it additionally requires SELECT and USAGE privileges on all sequences in its database.

• If using MySQL, verify that you have indeed run the GRANT statement and FLUSH PRIVILEGES as documented. It is important to remember that MySQL caches database privileges and will not apply changes to privileges unless FLUSH PRIVILEGES is run.

• If using PostgreSQL, verify that you have run both GRANT statements as documented, and that you did so prior to granting the required database permissions. It is important to remember that PostgreSQL can only grant permissions related to tables and sequences that exist. If GRANT statements are run before Guacamole's database tables exist, they will not have any effect.

Check for trailing whitespace after database-related properties

While the Java .properties files do not assign meaning to whitespace that occurs before a property value, everything after the first non-whitespace character of a property value is part of the property value, including further whitespace. If you are confident that all database-related properties within /etc/guacamole/guacamole.properties look correct, check that you have not accidentally inserted whitespace at the end of what otherwise appears to be a correct value.

Administration

LDAP users are not listed when integrating LDAP with a database

Check that your (administrative) Guacamole user account has permission to list LDAP users

Apache Guacamole's LDAP support will not bypass the permissions enforced by your LDAP directory. This means that your Guacamole user will only be able to see LDAP users within Guacamole's admin interface if your corresponding LDAP user has this permission to list these users, regardless of whether Guacamole's "search" LDAP user has permission and regardless of whether your database user has general permission to manage users.

If unsure whether your user has permission, a good way to check is to execute an LDAP query against your directory, binding using the DN and password associated with your LDAP account. The "ldapsearch" utility is a standard tool which can perform such a search:

$ ldapsearch -x -LLL -H ldap://YOUR_LDAP_SERVER -D "YOUR_LDAP_DN" -W -b USER_BASE_DN "(objectClass=*)"

where YOUR_LDAP_SERVER is the hostname or IP address of your LDAP server, YOUR_LDAP_DN is the full DN of your account within LDAP, and USER_BASE_DN is the base DN of all relevant users within your LDAP directory (as specified with the ldap-user-base-dn property in /etc/guacamole/guacamole.properties).

Check that you are using _LDAP** credentials to sign into Guacamole, not **database_** credentials**

If your user account within the database has an associated password, and you specify that password instead of the password associated with your LDAP account, you will successfully authenticate against the database only. This is a valid authentication result, but means that your session will lack access to LDAP objects. To be able to access objects within LDAP, including users, you must authenticate with credentials that your LDAP directory will accept.

Check for trailing whitespace after LDAP-related properties

While the Java .properties files do not assign meaning to whitespace that occurs before a property value, everything after the first non-whitespace character of a property value is part of the property value, including further whitespace. If you are confident that all LDAP-related properties within /etc/guacamole/guacamole.properties look correct, check that you have not accidentally inserted whitespace at the end of what otherwise appears to be a correct value. The presence of whitespace at the end of LDAP-related properties may result in queries failing and may cause Guacamole to incorrectly interpret the results of queries that appear to succeed.

The password for "guacadmin" (or the current user) cannot be changed

Use the "Preferences" interface for changing your own password, not the administrative user editor

Apache Guacamole has two mechanisms available for changing a user's password: the user editor, for changing the passwords of other users (without having to know their current passwords), and the "Preferences" screen, for changing your own password after verifying that you know your current password. As only the "Preferences" screen verifies that you know your current password, this is the only mechanism which Guacamole will allow if you are making changes to your own user. If you attempt to change your own password using the user editor, permission will be denied. For administrators, the "Preferences" screen can be accessed by going to "Settings" and then selecting "Preferences".

Check that the user in question has permission to change their own password

User accounts defined within the Apache Guacamole database do not necessarily have permission to change their own passwords. Permission to update your own password must be explicitly granted by the administrator. If you are an administrator and you want a newly-created user to be able to change their own password, check the box next to the "Change own password" permission within the user editor.

Remote desktop reachability and behavior

Cannot connect to any remote desktops

Verify that guacd is running

The Apache Guacamole web application relies upon a service called "guacd" to handle low-level communication with remote desktops, translation from native remote desktop protocols to the Guacamole protocol, and optimization. The guacd service must be running for the Guacamole web application installed as part of Keeper Connection Manager to be able to connect to remote desktops. If unsure whether guacd is running, the easiest way to check is using systemctl:

$ sudo systemctl status guacd

If guacd is not running, it will need to be started. The service should also be set to automatically start on boot, such that the service will come back online after a server restart.

To start guacd manually:

$ sudo systemctl start guacd

To configure guacd to start automatically when the server next boots:

$ sudo systemctl enable guacd

Verify that the remote desktops in question are indeed reachable

Apache Guacamole is a remote desktop gateway. As it is Guacamole (and, more specifically, guacd) which connects to remote desktops on each user's behalf, the Guacamole server itself must be able to reach these remote desktops. The user attempting to connect does not need direct network access to their remote desktops, but the Guacamole server must be able to reach them.

If connections are unexpectedly failing, check that the failing remote desktop(s) are reachable from the Guacamole server's network and that no firewall rules on the remote desktop(s) themselves are blocking this access.

Cannot connect to Windows using RDP

Try explicitly specifying the security method

Recent versions of Windows require NLA (Network Level Authentication) by default, an RDP-specific security method which uses higher-grade encryption and which enforces authentication before allocating desktop session resources. If connections to your Windows RDP servers are failing, try selecting the "NLA" or "TLS" security mode within the failing connection(s) parameters. If you do not also specify the username, password, and (if applicable) domain, Guacamole will prompt you for these details upon connecting.

Similarly, older versions of Windows may not support NLA or TLS by default, and Guacamole's initial security negotiation may fail unless the older "RDP" security method is explicitly selected. In older versions of Guacamole, the "RDP" security method was the default.

Try disabling validation of the RDP server certificate

Both the NLA (Network Level Authentication) and TLS security modes involve client verification of the RDP server's certificate. In the case of most Windows servers, this certificate will be self-signed and cannot be verified. Lacking the ability to verify the certificate against a known certificate authority, Guacamole will refuse to complete the RDP connection. Individual RDP connections can be configured to bypass this verification by checking the "Ignore server certificate" option within the connection parameters.

Check whether Windows firewall rules are blocking RDP connections from guacd

The Windows firewall may be configured to block inbound RDP connections, either locally or through GPOs. If the Windows machine in question has historically been accessed using RDP, it may still be the case that RDP access is blocked for all but a specific subnet in which the Guacamole server does not reside. If this is the case, the Windows firewall should be reconfigured to allow the Guacamole server (or its subnet) to connect via RDP. The Windows server does not need to be publicly accessible.

File transfer using RDP drive redirection is not working

Check that the drive path points to a directory that is writable by the "guacd" user or group

The Keeper Connection Manager packages create a "guacd" user and group for controlling access to files that should be accessible only by guacd. If you are configuring RDP drive redirection, and this user (or group) is not given write access to the directory chosen as the RDP drive path, guacd will not be able to read/write files within that directory as it emulates the redirected drive. Appropriate permissions and ownership should be assigned to any directory intended for use as an RDP connection's drive path:

$ sudo chown -R root:guacd /path/to/directory
$ sudo chmod 770 /path/to/directory

File Permission Errors

When the packages are installed, the folder /var/lib/guacamole will be created by default for each service.

The "guacd" service is run as the guacd user and the guacd user is the only user that needs read/write permission on the subfolders within the /var/lib/guacamole subfolders, such as the "recordings" and "drives" folder.

The "guacamole" process runs as "guacamole" user, and needs read access to the "recordings" subfolder and files.