Troubleshoot Windows logon issues

This article describes the logs and error messages Windows provides when a user logs on using certificates and/or smart cards. These logs provide information you can use to troubleshoot authentication failures.

Certificates and public key infrastructure

Windows Active Directory maintains several certificate stores that manage certificates for users logging on.

  • NTAuth certificate store: To authenticate to Windows, the certificate authority immediately issuing user certificates (that is, no chaining is supported) must be placed in the NTAuth store. To see these certificates, from the certutil program, enter: certutil –viewstore –enterprise NTAuth.
  • Root and intermediate certificate stores: Usually, certificate logon systems can provide only a single certificate, so if a chain is in use, the intermediate certificate store on all machines must include these certificates. The root certificate must be in the Trusted Root Store, and the penultimate certificate must be in the NTAuth store.
  • Logon certificate extensions and Group Policy: Windows can be configured to enforce verification of EKUs and other certificate policies. See the Microsoft documentation: https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/ff404287(v=ws.10).
Registry policy Description
AllowCertificatesWithNoEKU When disabled, certificates must include the smart card logon Extended Key Usage (EKU).
AllowSignatureOnlyKeys By default, Windows filters out certificates private keys that do not allow RSA decryption. This option overrides that filter.
AllowTimeInvalidCertificates By default, Windows filters out expired certificates. This option overrides that filter.
EnumerateECCCerts Enables elliptic curve authentication.
X509HintsNeeded If a certificate does not contain a unique User Principal Name (UPN), or it could be ambiguous, this option allows users to manually specify their Windows logon account.
UseCachedCRLOnlyAnd, IgnoreRevocationUnknownErrors Disables revocation checking (usually set on the domain controller).
  • Domain controller certificates: To authenticate Kerberos connections, all servers must have appropriate “Domain Controller” certificates. These can be requested using the “Local Computer Certificate Personal Store” MMC snap-in menu.

UPN name and certificate mapping

It is recommended that user certificates include a unique User Principal Name (UPN) in the Subject Alternate Name extension.

UPN names in Active Directory

By default, every user in Active Directory has an implicit UPN based on the pattern <samUsername>@<domainNetBios> and <samUsername>@<domainFQDN>. The available domains and FQDNs are included in the RootDSE entry for the forest. Note that a single domain can have multiple FQDN addresses registered in the RootDSE.

Additionally, every user in Active Directory has an explicit UPN and altUserPrincipalNames. These are LDAP entries that specify the UPN for the user.

When searching for users by UPN, Windows looks first in the current domain (based on the identity of the process looking up the UPN) for explicit UPNs, then alterative UPNs. If there are no matches, it looks up the implicit UPN, which may resolve to different domains in the forest.

Certificate Mapping Service

If a certificate does not include an explicit UPN, Active Directory has the option to store an exact public certificate for each use in an “x509certificate” attribute. To resolve such a certificate to a user, a computer can query for this attribute directly (by default, in a single domain).

An option is provided for the user to specify a user account that speeds up this search, and also allows this feature to be used in a cross-domain environment.

If there are multiple domains in the forest, and the user does not explicitly specify a domain, the Active Directory rootDSE specifies the location of the Certificate Mapping Service. This is usually located on a global catalog machine, and has a cached view of all x509certificate attributes in the forest. This computer can be used to efficiently find a user account in any domain, based on only the certificate.

Control logon domain controller selection

When an environment contains multiple domain controllers, it is useful to see and restrict which domain controller is used for authentication, so that logs can be enabled and retrieved.

Control domain controller selection

To force Windows to use a particular Windows domain controller for logon, you can explicitly set the list of domain controllers that a Windows machine uses by configuring the lmhosts file: \Windows\System32\drivers\etc\lmhosts.

There is usually a sample file named “lmhosts.sam” in that location. Simply include a line:

1.2.3.4 dcnetbiosname #PRE #DOM:mydomai

Where “1.2.3.4” is the IP address of the domain controller named “dcnetbiosname” in the “mydomain” domain.

After a restart, the Windows machine uses that information to log on to mydomain. Note that this configuration must be reverted when debugging is complete.

Identify the domain controller in use

At logon, Windows sets an MSDOS environment variable with the domain controller that logged the user on. To see this, start the command prompt with the command: echo %LOGONSERVER%.

Logs relating to authentication are stored on the computer returned by this command.

Enable account audit events

By default, Windows domain controllers do not enable full account audit logs. This can be controlled through audit policies in the security settings in the Group Policy editor. After they are enabled, the domain controller produces extra event log information in the security log file.

localized image

Certificate validation logs

Check certificate validity

If a smartcard certificate is exported as a DER certificate (no private key required), you can validate it with the command: certutil –verify user.cer

Enable CAPI logging

On the domain controller and users machine, open the event viewer and enable logging for Microsoft/Windows/CAPI2/Operational Logs.

You can control CAPI logging with the registry keys at: CurrentControlSet\Services\crypt32.

Value Description
DiagLevel (DWORD) Verbosity level (0 to 5)
DiagMatchAnyMask (QUADWORD) Event filter (use 0xffffff for all)
DiagProcessName (MULTI_SZ) Filter by process name (for example, LSASS.exe)

CAPI logs

Message Description
Build Chain LSA called CertGetCertificateChain (includes result)
Verify Revocation LSA called CertVerifyRevocation (includes result)
X509 Objects In verbose mode, certificates and Certificate Revocation Lists (CRLs) are dumped to AppData\LocalLow\Microsoft\X509Objects
Verify Chain Policy LSA called CertVerifyChainPolicy (includes parameters)

Error messages

Error code Description
Certificate not trusted The smart card certificate could not be built using certificates in the computer’s intermediate and trusted root certificate stores.
Certificate revocation check error The CRL for the smart card could not be downloaded from the address specified by the certificate CRL distribution point. If revocation checking is mandated, this prevents logon from succeeding. See the Certificates and public key infrastructure section.
Certificate Usage errors The certificate is not suitable for logon. For example, it might be a server certificate or a signing certificate.

Kerberos logs

To enable Kerberos logging, on the domain controller and the end user machine, create the following registry values:

Hive Value name Value [DWORD]
CurrentControlSet\Control\Lsa\Kerberos\Parameters LogLevel 0x1
CurrentControlSet\Control\Lsa\Kerberos\Parameters KerbDebuglevel 0xffffffff
CurrentControlSet\Services\Kdc KdcDebugLevel 0x1
CurrentControlSet\Services\Kdc KdcExtraLogLevel 0x1f

Kerberos logging is output to the System event log.

  • Messages such as “untrusted certificate” should be easy to diagnose.
  • Two error codes are informational, and can be safely ignored:
    • KDC_ERR_PREAUTH_REQUIRED (used for backward compatibility with older domain controllers)
    • Unknown error 0x4b

Domain controller and workstation logs

This section describes the expected log entries on the domain controller and workstation when the user logs on with a certificate.

  • Domain controller CAPI2 log
  • Domain controller security logs
  • Virtual Delivery Agent (VDA) security log
  • VDA CAPI log
  • VDA system log

Domain controller CAPI2 log

During a logon, the domain controller validates the caller’s certificate, producing a sequence of log entries in the following form.

localized image

The final event log message shows lsass.exe on the domain controller constructing a chain based on the certificate provided by the VDA, and verifying it for validity (including revocation). The result is returned as “ERROR_SUCCESS”.

localized image

Domain controller security log

The domain controller shows a sequence of logon events, the key event being 4768, where the certificate is used to issue the Kerberos Ticket Granting Ticket (krbtgt).

The messages before this show the machine account of the server authenticating to the domain controller. The messages following this show the user account belonging to the new krbtgt being used to authenticate to the domain controller.

localized image

VDA security log

The VDA security audit log corresponding to the logon event is the entry with event ID 4648, originating from winlogon.exe.

localized image

VDA CAPI log

This example VDA CAPI log shows a single chain build and verification sequence from lsass.exe, validating the domain controller certificate (dc.citrixtest.net).

localized image

localized image

VDA system log

When Kerberos logging is enabled, the system log shows the error KDC_ERR_PREAUTH_REQUIRED (which can be ignored), and an entry from Winlogon showing that the Kerberos logon was successful.

localized image

Monitoring FAS using Windows event log

All FAS events are written to the Windows Application event log. You can use products such as System Centre Operations Manager (SCOM) to monitor the health of your FAS service using the processes and events described here.

Is the FAS service running?

To determine if the FAS service is running, monitor the process Citrix.Authentication.FederatedAuthenticationService.exe.

Only the most important events for monitoring the FAS service are described in this section. For the full list of FAS event codes, see FAS event logs.

FAS health events

The following events show whether your FAS service is healthy.

The event source is Citrix.Authentication.FederatedAuthenticationService.

Event Event text Explanation Notes
[S003] Administrator [{0}] setting Maintenance Mode to [{1}] The FAS service was put into, or taken out of, maintenance mode. While in maintenance mode, the FAS server is not usable for single sign-on.
[S022] Administrator [{0}] setting Maintenance Mode to Off The FAS service was taken out of maintenance mode. Available from FAS 10.7 / CVAD 2109.
[S023] Administrator [{0}] setting Maintenance Mode to On The FAS service was put into maintenance mode. Available from FAS 10.7 / CVAD 2109.
[S123] Failed to issue a certificate for [upn: {0} role: {1}] [exception: {2}] This event happens after [S124] if none of the CAs FAS is configured with successfully issued a user certificate. Single sign-on will fail for that user. This event may indicate that all configured CAs are not working. If FAS is configured to use an HSM, it may also indicate that the HSM is not working.
[S124] Failed to issue a certificate for [upn: {0} role: {1}] at [certifcate authority: {2}] [exception: {3}] A failure occurred when FAS attempted to request a user certificate from the given CA. If FAS is configured with more than one CA, FAS will try the request at another CA. This event may indicate that the CA is not working, or is not contactable. If FAS is configured to use an HSM, it may also indicate that the HSM is not working. The exception can be used to help identify the cause of the problem.

Cloud-connected FAS events

If you are using FAS in conjunction with Citrix Cloud, the following events show whether your FAS service is healthy.

The event source is Citrix.Fas.Cloud.

Event Event text Explanation Notes
[S012] The FAS service is available for single sign-on from Citrix Cloud This event indicates that single sign-on from Workspace (i.e. Citrix Cloud) should be working. Before issuing this event, FAS checks (1) that it is configured, (2) is not in maintenance mode, and (3) is connected to Citrix Cloud.
[S013] The FAS service is not available for single sign-on from Citrix Cloud. [{0}] Further details can be found in the admin console. This event indicates that FAS is not able to provide single sign-on from Workspace (i.e. Citrix Cloud). The message includes the reason why single sign-on is not working. FAS maintains a persistent connection to Citrix Cloud. From time-to-time, this connection may terminate for a variety of reasons (such as a network glitch, or a connection lifetime policy on a proxy server). When this happens, the event text contains “Service is not connected to the cloud”. This is normal behaviour, and FAS immediately attempts to re-establish a connection to Citrix Cloud.

Security events

The following events indicate that an unauthorized entity attempted to use FAS.

The event source is Citrix.Authentication.FederatedAuthenticationService.

Event Event text Explanation
[S001] ACCESS DENIED: User [{0}] is not a member of Administrators group An attempt was made to view or change te configuration of FAS, but the caller was not a FAS administrator.
[S002] ACCESS DENIED: User [{0}] is not an Administrator of Role [{1}] An attempt was made to view or change the configuration of a FAS rule, but the caller was not a FAS administrator.
[S101] Server [{0}] is not authorized to assert identities in role [{1}] An attempt was made to assert user identities, but the caller is not permitted to do so. Only StoreFront servers which have been permitted in the FAS rule configuration (and Workspace if applicable) are allowed to assert user identities.
[S104] Server [{0}] failed to assert UPN [{1}] (UPN not allowed by role [{2}]) An attempt was made to assert a user identity, but the user’s account is not permitted according to the FAS rule configuration.
[S205] Relying party access denied - the calling account [{0}] is not a permitted relying party of the rule [{1}] A VDA attempted to perform single sign-on with FAS, but the VDA is not permitted according to the FAS rule configuration.

FAS event logs

The following tables list the event log entries generated by FAS.

Administration events [Federated Authentication Service]

[Event Source: Citrix.Authentication.FederatedAuthenticationService]

These events are logged in response to a configuration change in the FAS server.

Log codes
[S001] ACCESS DENIED: User [{0}] is not a member of Administrators group
[S002] ACCESS DENIED: User [{0}] is not an Administrator of Role [{1}]
[S003] Administrator [{0}] setting Maintenance Mode to [{1}]
[S004] Administrator [{0}] requesting authorization certificate from CA [{1}] using templates [{2} and {3}]
[S005] Administrator [{0}] de-authorizing CA [{1}]
[S006] Administrator [{0}] creating new Certificate Definition [{1}]
[S007] Administrator [{0}] updating Certificate Definition [{1}]
[S008] Administrator [{0}] deleting Certificate Definition [{1}]
[S009] Administrator [{0}] creating new Rule [{1}]
[S010] Administrator [{0}] updating Rule [{1}]
[S011] Administrator [{0}] deleting Rule [{1}]
[S012] Administrator [{0}] creating certificate [upn: {1} sid: {2} rule: {3}][Certificate Definition: {4}][Security Context: {5}]
[S013] Administrator [{0}] deleting certificates [upn: {1} role: {2} Certificate Definition: {3} Security Context: {4}]
[S015] Administrator [{0}] creating certificate request [TPM: {1}]
[S016] Administrator [{0}] importing Authorization certificate [Reference: {1}]
[S022] Administrator [{0}] setting Maintenance Mode to Off
[S023] Administrator [{0}] setting Maintenance Mode to On
[S024] Administrator [{0}] setting system health monitor
[S025] Administrator [{0}] setting system health monitor
[S050] Administrator [{0}] creating new cloud configuration: [{1}]
[S051] Administrator [{0}] updating cloud configuration: [{1}]
[S052] Administrator [{0}] removing cloud configuration
Log Codes
[S401] Performing configuration upgrade – [From version {0}][to version {1}]
[S402] ERROR: The Citrix Federated Authentication Service must be run as Network Service [currently running as: {0}]
[S404] Forcefully erasing the Citrix Federated Authentication Service database
[S405] An error occured while migrating data from the registry to the database: [{0}]
[S406] Migration of data from registry to database is complete (note: user certificates are not migrated)
[S407] Registry-based data was not migrated to a database since a database already existed
[S408] Cannot downgrade the configuration – [From version {0}][to version {1}]
[S409] ThreadPool configuration succeeded - MinThreads adjusted from [workers: {0} completion: {1}] to: [workers: {2} completion: {3}]
[S410] ThreadPool configuration failed - failed to adjust MinThreads from [workers: {0} completion: {1}] to: [workers: {2} completion: {3}]; this may impact the scalablilty of the FAS server
[S411] Error starting the FAS service: [{0}]
[S412] Configuration upgrade complete – [From version {0}][to version {1}]

Creating identity assertions [Federated Authentication Service]

[Event Source: Citrix.Authentication.FederatedAuthenticationService]

These events are logged at runtime on the FAS server when a trusted server asserts a user logon.

Log Codes
[S101] Server [{0}] is not authorized to assert identities in role [{1}]
[S102] Server [{0}] failed to assert UPN [{1}] (Exception: {2}{3})
[S103] Server [{0}] requested UPN [{1}] SID {2}, but lookup returned SID {3}
[S104] Server [{0}] failed to assert UPN [{1}] (UPN not allowed by role [{2}])
[S105] Server [{0}] issued identity assertion [upn: {1}, role {2}, Security Context: [{3}]]
[S120] Issuing certificate to [upn: {0} role: {1} Security Context: [{2}]]
[S121] Certificate issued to [upn: {0} role: {1}] by [certifcate authority: {2}]
[S122] Warning: Server is overloaded [upn: {0} role: {1}][Requests per minute {2}].
[S123] Failed to issue a certificate for [upn: {0} role: {1}] [exception: {2}]
[S124] Failed to issue a certificate for [upn: {0} role: {1}] at [certifcate authority: {2}] [exception: {3}]

Acting as a relying party [Federated Authentication Service]

[Event Source: Citrix.Authentication.FederatedAuthenticationService]

These events are logged at runtime on the FAS server when a VDA logs on a user.

Log Codes
[S201] Relying party [{0}] does not have access to a password.
[S202] Relying party [{0}] does not have access to a certificate.
[S203] Relying party [{0}] does not have access to the Logon CSP
[S204] Relying party [{0}] accessing the Logon CSP for [upn: {1}] in role: [{2}] [Operation: {3}] as authorized by [{4}]
[S205] Relying party access denied - the calling account [{0}] is not a permitted relying party of the rule [{1}]
[S206] Calling account [{0}] is not a relying party
[S208] Private Key operation failed [Operation: {0}][upn: {1} role: {2} certificateDefinition {3}][Error {4} {5}].

In-session certificate server [Federated Authentication Service]

[Event Source: Citrix.Authentication.FederatedAuthenticationService]

These events are logged on the FAS server when a user uses an in-session certificate.

Log Codes
[S301] Access Denied: User [{0}] does not have access to a Virtual Smart Card
[S302] User [{0}] requested unknown Virtual Smart Card [thumbprint: {1}]
[S303] Access Denied: User [{0}] does not match Virtual Smart Card [upn: {1}]
[S304] User [{0}] running program [{1}] on computer [{2}] using Virtual Smart Card [upn: {3} role: {4} thumbprint: {5}] for private key operation [{6}]
[S305] Private Key operation failed [Operation: {0}][upn: {1} role: {2} containerName {3}][Error {4} {5}].

FAS assertion plugin [Federated Authentication Service]

[Event Source: Citrix.Authentication.FederatedAuthenticationService]

These events are logged by the FAS assertion plugin.

Log Codes
[S500] No FAS assertion plugin is configured
[S501] The configured FAS assertion plugin could not be loaded [exception:{0}]
[S502] FAS assertion plugin loaded [pluginId={0}] [assembly={1}] [location={2}]
[S503] Server [{0}] failed to assert UPN [{1}] (logon evidence was supplied but the plugin [{2}] does not support it)
[S504] Server [{0}] failed to assert UPN [{1}] (logon evidence was supplied but there is no configured FAS plugin)
[S505] Server [{0}] failed to assert UPN [{1}] (the plugin [{2}] rejected the logon evidence with status [{3}] and message [{4}])
[S506] The plugin [{0}] accepted logon evidence from server [{1}] for UPN [{2}] with message [{3}]
[S507] Server [{0}] failed to assert UPN [{1}] (the plugin [{2}] threw exception [{3}] during method [{4}])
[S507] Server [{0}] failed to assert UPN [{1}] (the plugin [{2}] threw exception [{3}])
[S508] Server [{0}] failed to assert UPN [{1}] (access disposition was supplied but the plugin [{2}] does not support it)
[S509] Server [{0}] failed to assert UPN [{1}] (access disposition was supplied but there is no configured FAS plugin)
[S510] Server [{0}] failed to assert UPN [{1}] (the access disposition was deemed invalid by plugin [{2}]

Workspace-enabled FAS [Federated Authentication Service]

[Event Source: Citrix.Fas.Cloud]

These events are logged when FAS is used in conjunction with Workspace.

Log Codes
[S001] Rotating Citrix Cloud service keys [fas id={0}]
[S002] The cloud support module is starting. FasHub cloud service URL: {0}
[S003] FAS registered with the cloud [fas id: {0}] [transaction id: {1}]
[S004] FAS failed to register with the cloud [fas id: {0}] [transaction id: {1}] [exception: {2}]
[S005] FAS sent its current configuration to the cloud [fas id: {0}] [transaction id: {1}]
[S006] FAS failed to send its current configuration to the cloud [fas id: {0}] [transaction id: {1}] [exception: {2}]
[S007] FAS unregisted from the cloud [fas id: {0}] [transaction id: {1}]
[S009] FAS failed to unregister from the cloud [fas id: {0}] [transaction id: {1}] [exception: {2}]
[S010] The FAS service is connected to the cloud messaging URL: {0}
[S011] The FAS service is not connected to the cloud
[S012] The FAS service is available for single sign-on from Citrix Cloud
[S013] The FAS service is not available for single sign-on from Citrix Cloud. [{0}] Further details can be found in the admin console
[S014] A call to the cloud service <service name> failed [fas id: {0}] [transaction id: {1}] [exception: {2}]
[S015] A message from Citrix Cloud was blocked because the caller is not permitted [message ID {0}] [transaction ID {1}] [caller {2}]
[S016] A call to the cloud service <service name> succeeded [fas id: {0}] [transaction id: {1}]
[S019] FAS downloaded its configuration from the cloud [fas id: {0}] [transaction id: {1}]
[S020] FAS failed to download its configuration from the cloud [fas id: {0}] [transaction id: {1}] [exception: {2}]
[S021] The cloud support module failed to start. Exception: {0}
[S022] The cloud support module is stopping
[S202] The health monitor encountered an unexpected exception and may have stopped working: exception: {0}

Log on [VDA]

[Event Source: Citrix.Authentication.IdentityAssertion]

These events are logged on the VDA during the logon stage.

Log Codes
[S101] Identity Assertion Logon failed. Unrecognised Federated Authentication Service [id: {0}]
[S102] Identity Assertion Logon failed. Could not lookup SID for {0} [Exception: {1}{2}]
[S103] Identity Assertion Logon failed. User {0} has SID {1}, expected SID {2}
[S104] Identity Assertion Logon failed. Failed to connect to Federated Authentication Service: {0} [Error: {1} {2}]
[S105] Identity Assertion Logon. Logging in [Username: {0}][Domain: {1}]
[S106] Identity Assertion Logon.\n\nFederated Authentication Service: {0}\n\nLogging in [Certificate: {1}]
[S107] Identity Assertion Logon failed. [Exception: {0}{1}]
[S108] Identity Assertion Subsystem. ACCESS_DENIED [Caller: {0}]

In-session certificates [VDA]

[Event Source: Citrix.Authentication.IdentityAssertion]

These events are logged on the VDA when a user attempts to use an in-session certificate.

Log Codes
[S201] Virtual smart card access authorized by [{0}] for [PID: {1} Program Name: {2}][Certificate thumbprint: {3}]
[S203] Virtual Smart Card Subsystem. Access Denied [caller: {0}, session {1}]
[S204] Virtual Smart Card Subsystem. Smart card support disabled

Certificate request and key pair generation [Federated Authentication Service]

[Event Source: Citrix.Fas.PkiCore]

These events are logged when the FAS server performs low-level cryptographic operations.

Log Codes
[S001] TrustArea::TrustArea: Installed certificate [TrustArea: {0}] [Certificate {1}][TrustAreaJoinParameters{2}
[S014] Pkcs10Request::Create: Created PKCS10 request [Distinguished Name {0}]
[S016] PrivateKey::Create [Identifier {0}][MachineWide: {1}][Provider: {2}][ProviderType: {3}][EllipticCurve: {4}][KeyLength: {5}][isExportable: {6}]
[S017] PrivateKey::Delete [CspName: {0}, Identifier {1}]
Log Codes
[S104] MicrosoftCertificateAuthority::GetCredentials: Authorized to use {0}
[S105] MicrosoftCertificateAuthority::SubmitCertificateRequest Error submit response [{0}]
[S106] MicrosoftCertificateAuthority::SubmitCertificateRequest Issued certificate [{0}]
[S112] MicrosoftCertificateAuthority::SubmitCertificateRequest - Waiting for approval [CR_DISP_UNDER_SUBMISSION] [Reference: {0}]

End user error messages

This section lists common error messages displayed to a user on the Windows logon page.

Error message displayed Description and reference
Invalid Username or Password The computer believes that you have a valid certificate and private key, but the Kerberos domain controller has rejected the connection. See the Kerberos logs section of this article.
The system could not log you on. Your credentials could not be verified. / The request is not supported The domain controller cannot be contacted, or the domain controller has not been configured with a certificate to support Smart Card authentication. Enroll the domain controller for a “Kerberos Authentication”, “Domain Controller Authentication”, or “Domain Controller” certificate. This is usually worth trying, even when the existing certificate appears to be valid.
The system could not log you on. The smartcard certificate used for authentication was not trusted. The intermediate and root certificates are not installed on the local computer. See Certificates and public key infrastructure.
Bad Request This usually indicates that the extensions on the certificate are not set correctly, or the RSA key is too short (<2048 bits).