DNS Troubleshooting.pdf
- 2. Introduction to BIGIP DNS
- 3. DNS Hierarchy Sample of a Zone File $TTL 86400 ; 24 hours could have been written as 24h or 1d ; $TTL used for all RRs without explicit TTL value $ORIGIN example.com. @ 1D IN SOA ns1.example.com. hostmaster.example.com. ( 2002022401 ; serial 3H ; refresh 15 ; retry 1w ; expire 3h ; nxdomain ttl ) IN NS ns1.example.com. ; in the domain IN NS ns2.smokeyjoe.com. ; external to domain IN MX 10 mail.another.com. ; external mail provider ; server host definitions ns1 IN A 192.168.0.1 ;name server definition www IN A 192.168.0.2 ;web server definition ftp IN CNAME www.example.com. ;ftp server definition ; non server domain hosts bill IN A 192.168.0.3 fred IN A 192.168.0.4
- 4. BIG-IP DNS (formerly Global Traffic Manager or GTM) It’s a load balancer for DNS queries (caching, traffic management), decision making to load balance between datacenters. Terms • Wide IP - Maps FQDN to one or more pools of virtual servers that host content of the domain. It responds to listener requests. This will auto create a zone that matches the Wide IP • Server object - Server defined in BIG-IP DNS is either a BIG-IP or other 3rd party system responsible for owning one or more virtual server service. i. BIGIP devices (LTM/ASM/APM/etc) – Standalone/Redudant-Pair ii. Generic LB/Host (3rd party system) – Citrix LB, Cisco CSS, Centos machine • Listener - BIG-IP uses TCP/UDP listeners to respond to DNS queries. • Pool - In BIG-IP DNS a pool contains one or more virtual servers.
- 6. Listener
- 7. Wide IP - Maps FQDN to set of Virtual Server that host the domain content - Uses Pool to organize Virtual Server
- 8. iQuery communication and troubleshooting
- 9. Purpose Establishing communication between GTM and other system to be in a same Sync Group Requirement 1) DNS members must be running on same version (source: K13703) i. BIG-IP DNS synchronization group communication ii. Monitored BIG-IP systems must run the same or newer big3d version as the DNS / GTM that are monitoring them 2) Sync parameter must defined properly 3) NTP in sync 4) Port lockdown allowing 4353 and 443 5) Compatible big3d version iQuery
- 10. Virtual Server/Link Autodiscovery (K13994) 1) Virtual server and link auto-discovery feature allows: 2) BIG-IP DNS and BIG-IP Link Controller systems to automatically discover virtual servers and links that are associated with defined BIG-IP systems. 3) Uses iQuery protocol to automatically discover objects on the remote BIG-IP system if enabled. • The BIG-IP DNS configuration contains one or more BIG-IP server objects • TCP port 4353 is allowed between the BIG-IP DNS system and target BIG-IP systems • The target BIG-IP system's virtual server addresses must not employ network address translation Important Note: K9138: The BIG-IP GTM system disables virtual server auto-discovery for BIG-IP systems that use translated virtual server addresses K14106: Troubleshooting virtual server and link auto-discovery (11.x - 13.x) - telnet <remote_bigip_selfip> 4353 - iqdump
- 11. K13312: Overview of the BIG-IP GTM big3d_install, bigip_add, and gtm_add utilities (11.x - 13.x) Requirement of iQuery - TCP port 4353 - SSH port 22 (for initial certificate transfer/copy) bigip_add - Exchanges iQuery SSL certificate with remove BIGIP - Append Local GTM system’s certificate to remote BIGIP authorized cert (stored in /config/big3d/client.crt) - Append remote iQuery cert to local GTM list of authenticate iQuery (stored in .config/gtm/server.crt) bigip_install (K13703) - Similar to bigip_add but install the big3d version if its older than the local GTM F5 system - To check: run: # /usr/sbin/big3d -v (default big3d agent) #/shared/bin/big3d –v (executable file) gtm_add - Integrate new GTM system into existing sync group - Replace current config (bigip_gtm.conf, named.conf and the name zone files)
- 12. Troubleshoot iQuery 1. Config Utility - Check the status of the server object (Global Traffic -> Server -> Server List - iQuery Stat (Statistic -> Global Traffic -> Statistic Type -> iQuery) - Summary Stat (Statistic -> Global Traffic) 2. TMSH - Server (tmsh show /gtm server all) - iQuery (tmsh show /gtm iquery all) - GTM (tmsh show /gtm) 3. /var/log/gtm 4. Verify the big3d version # /usr/sbin/big3d -v (default big3d agent) #/shared/bin/big3d –v (executable file) 5. Check the iQuery processes # netstat –nap | grep 4353
- 13. Cont* 6. Iqdump utility (run from the GTM) iqdump 10.10.10.20 <sync_group_name> • If the iQuery channel is not established, it will prompt error 46947856243768:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1168: • If the iQuery channel is established, iqdump returns XML similar to the following example: <!-- Local hostname: lc1.example.com --> <!-- Connected to big3d at: ::ffff:10.10.10.10:4353 --> <!-- Subscribing to syncgroup: default --> <!-- Tue May 6 09:55:43 2014 --> <xml_connection> <version>11.5.1</version> <big3d>big3d Version 11.5.1.0.0.110</big3d> 7. Verify device Certificate openssl x509 -noout -text -in /config/httpd/conf/ssl.crt/server.crt - Verify the certificate validity date and confirm whether the certificate is expired. - If necessary, renew the certificate. To do so, refer to K6353: Updating an SSL device certificate on a BIG-IP system.
- 14. Prober pool Collection of device that perform monitor probes of servers to gather data about the health and performance of the resources on the servers By default, the members of a GTM sync group dynamically determine the best BIG-IP device within the sync group configuration to use as the prober for the non-BIG-IP device server objects. Devices defined within the same data center as the server object to be probed are preferred. If no local prober is available, a remote prober is used.
- 15. Debugging To enable debugging tmsh modify /sys db log.gtm.level value debug tmsh modify /sys db log.big3d.level value debug tmsh modify /sys db gtm.debugprobelogging value enable To disable debugging tmsh modify /sys db log.gtm.level value debug tmsh modify /sys db log.big3d.level value debug tmsh modify /sys db gtm.debugprobelogging value enable Collect qkview and full tar ball for review. #qkview –s0 #tar -czvpf /var/tmp/$HOSTNAME-logs.tgz /var/log
- 16. Configure Decision Logging https://devcentral.f5.com/articles/configuring-decision-logging-for-the-f5-big-ip-global-traffic- manager • Modify WIP(s) to enable LB Decision Logging • Log Publisher • DNS Logging Profile • Custom DNS Profile and attach the logging profile • Apply the DNS Profile to the Listener
- 17. DNS Express
- 18. DNS Express Alows the BIG-IP to perform zone transfers from multiple primary DNS servers that are responsible for different zones, perform a zone transfer from the local BIND server on the BIG-IP, and serve DNS records faster than the primary DNS servers and the local BIND server. • Perform zone transfers from multiple primary DNS servers that are responsible for different zones. • Perform a zone transfer from the local BIND server on the BIG-IP system. • Serve DNS records faster than the primary DNS servers
- 19. K15298: Overview of the dnsxdump utility You can use the dnsxdump utility to view the DNS Express database information, which includes zone information and statistics. • The DB Dump section of the dnsxdump utility output displays the zone information for all configured DNS Express zones. • The DB Stats section of the dnsxdump utility output displays a cumulative count of records for all configured DNS Express zones. dnsxdump > /var/tmp/my_zones.txt
- 20. Zonerunner
- 21. Managing the BIG-IP BIND configuration file ZoneRunner utility is used to manage both DNS zone files and the BIND configuration file on the BIG-IP GTM system • Import and transfer DNS zone files • Manage zone resource records • Manage views • Manage a local nameserver and the associated configuration file, named.conf • Transfer zone files to a nameserver • Import only primary zone files from a nameserver
- 22. Cont* By default, BIG-IP GTM is configured to secure BIND to not allow zone transfers except from the localhost. Modify the allow-transfer statement to include the IP address of the GTM. You can modify the following allow-transfer statement to use the IP address of the GTM. DNS > Zones > ZoneRunner > named Configuration. allow-transfer { localhost; 192.168.10.105; } To verify zone transfers are working properly # dig @<IP address> es.net. axfr Directory of the zone file stored # cd /var/named/config/namedb/ Check the named configuration
- 23. K7032: Freezing zone files to allow manual update to ZoneRunner- managed zone files All changes made to a zone using dynamic update are written to the zone's journal file. When the BIG-IP DNS system restarts after a shutdown, the system replays the journal file to incorporate any updates that took place after the last zone file update into the zone. Dynamic update periodically flushes the complete contents of the updated zone to its zone file and automatically deletes the journal file. i. cd /var/named/config/namedb ii. cp <zone_filename> <zone_filename>.original iii. bigstart stop zrd iv. rndc freeze <zone name> <class> <view> v. Manually edit the zone for any changes vi. rndc sync -clean vii. Run the named-checkzone command to check the file for any syntax errors • named-checkzone askf5.net db.external.askf5.net • named-checkconf -t /var/named -z -j /config/named.conf viii. rndc thaw <zone name> <class> <view> ix. bigstart start zrd .
- 24. Behaviour of zrd - When a new dns express zone is added , it writes the data to zxfrd.bin - It then copies from zxfrd.bin to zxfrd-tmp.bin (15sec timer) - Rename the zxfrd-tmp.bin to tmmdns.bin (database) TMM then reload the database from tmmdns.bin - For VIPRION, csyncd monitors tmmdns.bin for any changes - Csyncd trigger tmm to reload on primary blades then populate to other blades # bigstart stop # rm -rf /var/db/{tmmdns.bin,zxfrd.bin} # bigstart start tmsh modify sys db log.zxfrd.level value debug Collect qkview and full tar ball for review. #qkview –s0 #tar -czvpf /var/tmp/$HOSTNAME-logs.tgz /var/log