![Digital Video Surveillance Basic Troubleshooting Guide
Compliments of Seneca < http://www.senecadata.com >
Once Task Manager (TM) is opened, select the ‘Performance’ tab and also select ‘Resource Monitor’
which will open a new window of information as shown in Figure 1.
Take away from this window: A quick view of the CPU and Memory being used up shown by the yellow
square.
Any value greater than 70% should raise a flag.
Resource Monitor
The Resource Monitor window looks like this:
‘Click’ on the yellow highlighted items to expand each section.
Take away from this window:
This adds in information about the Storage (Hard Disks) and Network resources being used up.
Once again, you do not want to see a number higher than 70% for the Network.
For the Storage, under the ‘Disk’ tab, you are verifying that the video is being written to the correct
array [‘File’ column] and that the ‘Queue Length’ [red square] is not very large (single digits…preferably
less than one) as shown in Figure 3.](https://image.slidesharecdn.com/5d986c2a-b3e0-4741-8c4c-9dead924a870-150327160548-conversion-gate01/85/VMS-Troubleshooting-Guide-6-320.jpg)
VMS Troubleshooting Guide
- 1. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > Contents VMS Troubleshooting Questions ..............................................................................................................2 Typical DSS system view .......................................................................................................................2 A) Basic ..............................................................................................................................................2 B) System Topology Information.......................................................................................................3 C) Questions if intermittent camera connectivity problems exist....................................................3 D) Things to look out for that might cause issues. ............................................................................3 E) Local IT/Engineering gets involved for the following troubleshooting.........................................3 TOOLS........................................................................................................................................................4 Basic Windows Performance Tools Detailed Usage .............................................................................4 Task Manager/Resource Monitor.........................................................................................................5 Resource Monitor .................................................................................................................................6 PERFMON: Dynamic and long term resource viewing and collecting................................................7 RealTemp …. Looking for hot systems ................................................................................................9 GPUZ a look at your video subsystem...................................................................................................9
- 2. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > VMS Troubleshooting Questions Typical DSS system view: A) Basic 1. What is the problem environment...i.e....what is the server/client being asked to do and what is it not doing? "Are you using the appliance for recording, viewing or both?" o Note that large scale viewing on a recorder is not recommended. "What are you having trouble with…. recording, viewing or both?" Physical Environment (looking for Temperature, Humidity, Vibration, Dust) (RealTemp log) Windows Software/Hardware environment (Task Manager and Resource View…. CPU, Memory, HDD, Network…anything either pegged or zero?) If a RAID array is being used…is it in Writeback Always mode? 2. If you are having problems viewing video on a separate client / workstation How many simultaneous streams on the system? Live or Playback (Playback stresses server HDD whereas Live does not) Resolution & FPS total being attempted
- 3. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > One Client machine or All Client machines (All suggests network issue or underspec’d system) 3. If application is crashing, Look at GPU utilization and memory utilization (GPUZ is a great tool for monitoring this) *note some Client application use hardware resources differently Where is the application memory at the time of the crash (use Task Manager-Resource view) B) System Topology Information 1. How many NICs are being utilized 2. NIC bandwidth (BW) =? (Resource Monitor) 3. Network Topology =? (Subnet usage important) i.e. the recorder is on separate subnet from the client viewing application, are they sharing BW with Corporate data? 4. Storage BW =? (Only for a Server/Storage appliance) (Resource Monitor) 5. What VMS are you running? (VMSs do use resources differently) 6. What version are you on? 7. System hardware details = (BOM) C) Questions if intermittent camera connectivity problems exist 1. Is one camera affected, some or all? 2. If all cameras, are they the same brand, make, model, and firmware version? 3. If ‘some’ then are they connected to a common device? 4. Using TCP or UDP for your RTSP stream? UDP will drop packets and frames, causing loss. D) Things to look out for that might cause issues. 1) Is there other network traffic or is it closed for security only? 2) Are there windows updates and other utilities such as defrag, virus scan etc running on the storage discs? 3) Is windows firewall on / off if their network policy allows for it to be off please turn it off E) Local IT/Engineering gets involved for the following troubleshooting. 1. If some cameras, what's common, the switch? The network? The subnet? Good managed switches will give you statistics on dropped packets and other issues. I would do that first before setting up NAGIOS or MRTG from scratch to troubleshoot. Besides, those packages require switches to have that basic functionality anyway (sFlow or SNMP) so look at the switch first! 2. Network cabling good? A bad cable causes issues. 3. If you rule out the network, then look at your storage and your hosts.
- 4. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > TOOLS Here are a few tools that are easily used on a Windows system to aid in debugging. A) Windows has Performance Monitor. a. "Setup metrics on CPU, RAM, page faults, network statistics and I/O, disk stats and I/O, disk queue depth, etc. b. Monitor all statistics against known good numbers. " (see below) B) Windows Event Logs. a. Use the filtering option on the right to look for warnings/errors and critical events C) Windows Task Manager which is a high level view of everything. D) Storage logs. a. Controller and disk failures can cause I/O to go crazy on those path's as rewrite operations are sent out. This may bring down entire production critical systems on top- tier storage arrays. b. LSI RAID controllers have a ‘lsigetwin’ tool available for automatic dump of info. E) File system integrity. How does SCANDISK come back? Bad records and bad allocation tables can do bad things. Basic Windows Performance Tools Detailed Usage Here are details basic Windows tools that will help you determine how well your Windows PC is performing.
- 5. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > Task Manager/Resource Monitor Figure 1 What to open within Task Manager How to get this: Right Mouse Button on the Task Bar as shown below then pick Task Manager. An alternate method is to use the Alt-Ctrl-Del combo and select it. Figure 2 Using RMB to open Task Manager
- 6. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > Once Task Manager (TM) is opened, select the ‘Performance’ tab and also select ‘Resource Monitor’ which will open a new window of information as shown in Figure 1. Take away from this window: A quick view of the CPU and Memory being used up shown by the yellow square. Any value greater than 70% should raise a flag. Resource Monitor The Resource Monitor window looks like this: ‘Click’ on the yellow highlighted items to expand each section. Take away from this window: This adds in information about the Storage (Hard Disks) and Network resources being used up. Once again, you do not want to see a number higher than 70% for the Network. For the Storage, under the ‘Disk’ tab, you are verifying that the video is being written to the correct array [‘File’ column] and that the ‘Queue Length’ [red square] is not very large (single digits…preferably less than one) as shown in Figure 3.
- 7. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > Figure 3 Storage Resource tab If the resource usage of all these elements is less than 70%, then it is possible to give the system more work to do. PERFMON: Dynamic and long term resource viewing and collecting. Create a User Defined data collector set as follows: 1) Start typing ‘performance’ in the Win7 command line and select Performance Monitor (perfmon on other versions)
- 8. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > 2) To collect the information in real time, add the counters under Performance->Monitoring Tools- >Performance Monitor. Save this resulting setup file in the documents dir for easy recall later (RMB…. Save Settings). View of ‘add counters’ and the order of use pic: (a) Step 3 is scroll to desired counter (b) Step 4 is select counter (c) Step 5 is expand counter list (d) Step 6 is select desired counters (e) Step 7 adds them to the list Add these counters : b) From each archiving LogicalDisk i) Disk Read Bytes/sec for the drive where the live/archived video are stored ii) Disk Write Bytes/sec for the drive where the live/archived video are stored iii) Avg. Disk Queue Length for the drive where the live/archived video are stored (IMPORTANT) iv) Current Disk Queue length for the drive where the live/archived video are stored (IMPORTANT) v) Disk Writes/sec for the drive where the live/archived video are stored c) From Network Interface <select the NICs you are using> i) Bytes Received/sec ii) Bytes Sent/sec iii) Bytes Total/sec iv) Packets Received Errors v) Packets Received Discarded d) From Processor i) %Processor Time for the _Total and all logical processor instances e) From Memory i) Pages/sec ii) Available Mbytes (lets you discover memory leaks) 3) On Performance Monitor (1 in pic above) do RMB then create Data Collector <<<< do this to record a log file to review later or run over a long time >>>>
- 9. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com > 4) Type a name such as XYZ Server Storage Certification 5) Browse for the path where you want to save the logs, then Finish RealTemp …. Looking for hot systems Activate the log file as shown…. The log file is RealTempLog.txt and stored where RealTemp is installed GPUZ a look at your video subsystem This tool tells us about the Video Card and will log the usage. Activate the logging. Note that on systems with multiple Video sources (ie mobo and card), you need to select the one you are using. This can be done dynamically.
- 10. Digital Video Surveillance Basic Troubleshooting Guide Compliments of Seneca < http://www.senecadata.com >