IDS Survey on Entropy
1 like1,316 views
The document discusses various research papers related to intrusion detection systems that utilize entropy-based methods for identifying and mitigating DDoS (Distributed Denial of Service) attacks. It highlights the effectiveness of these methods in reducing false positives, minimizing computational time, and improving detection accuracy compared to traditional approaches. Several innovative algorithms and metrics developed for detecting and profiling network traffic, as well as strategies for future enhancements, are also summarized.
IDS Survey on Entropy
- 1. Survey on Intrusion Detection System based on Entropy MEthods IEEE Papers Raj Kamal IIT Guwahati June 8, 2012
- 2. Table 1: Entropy Based IEEE Papers Tittle Author Year Abstract Theme An Efficient and Giseop No† 2009 In this paper, we pro- Uses fast entrpoy and Reliable DDoS and Ilkyeun pose a fast entropy scheme moving average to cal- Attack Detection Ra. De- that can overcome the is- cualte entropy.If network Using a Fast En- partment of sue of false negatives and traffic changes from nor- tropy Computation Computer will not increase the com- mal to abnormal status Method Science and putational time. Our sim- such as when the DDoS Engineering. ulation shows that the attacker sends a bulk of University fast entropy computing packets with the same of Colorado method not only reduced port number to saturate a Denver USA. computational time by certain port, the entropy more than 90 % compared of this port number will be to conventional entropy, decreased. By contrast, but also increased the under normal conditions, detection accuracy com- the entropy of the port pared to conventional and number will be increased. compression entropy ap- This phenomenon can be proaches. applied to various net- work information such as source IP address, desti- nation IP address, source port, destination port, to- tal number of packets, and even in the data cluster- ing schemes. our Fast Entropy scheme reduced computational time by 90 /of conventional entropy scheme while maintaining detection accuracy. Fast Entropy is even faster than compression entropy scheme in computing en- tropy values with same or better detection accu- racy. For our future work, we have been developing an adaptive fast entropy algorithm that will fur- ther reduce the false posi- tives as well as false nega- tives without adding over- head by introducing dy- namic moving average and detection threshold value with respect to behavior of attacks. 1
- 3. Table 2: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Effective Discovery Chan- 2009 This IDS is based on the We implemented the of Attacks using Kyu Han notion of packet dynam- proposed algorithm using Entropy of Packet Hyoung- ics, rather than packet perl and ran it on real Dynamics Kee Choi content, as a way to traffic traces available on Sungkyunkwan cope with the increasing the Internet. We used University complexity of attacks. four traces containing five We employ a concept of malicious attacks: they entropy to measure time- are Code Red Worm, variant packet dynamics Witty Worm, Slammer and, further, to extrapo- Worm, DoS and DDOS late this entropy to detect attacks.Here thermody- network attacks. The namic approach is used entropy of network traffic with moving average . It should vary abruptly once further uses ROC curve the distinct patterns of to find out thershold. packet dynamics embed- ded in attacks appear. The proposed classifier is evaluated by comparing independent statistics de- rived from five well-known attacks. Our classifier detects those five attacks with high accuracy1 and does so in a timely man- ner For instance, a Denial of Service (DoS) attack and flash crowds cause destination hosts to con- centrate the distribution of traffic on the victim. Network scanning has a dispersed distribution for destination hosts and a bottleneck distribution for destination services. This bottleneck distribution is concentrated on the vulnerable ports. Con- centration and dispersion are, respectively, two pat- terns of packet dynamics frequently perceived in a DoS attack and network scanning. The key idea is that once abnormal traffic contaminates long-term behavior, the entropy value of the system should 2 immediately reflect this contamination.This de- tection method takes advantage of fluctua- tions in the entropy values of flow-related metrics.Bogus requests do not generate immediate
- 4. Table 3: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Entropy-Based Tsern-Huei 2009 we present an entropy- In this paper, we pro- Profiling of Net- Lee , Jyun- based network traffic posed a novel, two-stage work Traffic for De He profiling scheme for de- approach for detecting Detection of Secu- Department tecting security attacks. network attacks. In rity Attack of Com- The proposed scheme the first stage, normal munication consists of two stages. behavior profiles are Engineering The purpose of the first constructed based on National stage is to systematically Relative Uncertainty. In Chiao Tung construct the probability the second stage, the Chi- University distribution of Relative Square Goodness-of-Fit ,Taiwan Uncertainty for normal Test is performed for the network traffic behavior. distributions obtained In the second stage, from behavior profiling we use the Chi-Square and network activities Goodness-of-Fit Test, a collected online. We calculation that measures demonstrated the effec- the level of difference of tiveness of our proposed two probability distribu- scheme with the KDD tions, to detect abnormal 1999 dataset for DoS at- network activities. The tacks. Simulation results probability distribution of show that our proposed the Relative Uncertainty scheme achieves lower for short-term network complexity and higher behavior is compared accuracy than previous with that of the long- schemes. Based on the term profile constructed experimental results, we in the first stage. We believe that the proposed demonstrate the perfor- scheme could be a good mance of our proposed choice for network behav- scheme for DoS attacks ior profiling and attack with the dataset derived detection. from KDD CUP 1999. Experimental results show that our proposed scheme achieves high accuracy if the features are selected appropriately. The top six features ranked by the accuracy are srcbytes,dstbytes, srvdiffhos- trate,dsthostcount,dsthostsamesrcportrate and dsthostsrvdiffhos- trate.These features can be used to detect DoS attacks effectively. 3
- 5. Table 4: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Entropy-Based Shui Yu 2008 A community network we focus on detection of Collaborative De- and Wanlei often operates with the DDoS attacks in commu- tection of DDOS Zhou School same Internet Service nity networks. Our mo- Attacks on Com- of Engi- Provider domain or the tivation comes from dis- munity Networks neering and virtual network of dif- criminate the DDoS at- Information ferent entities who are tacks from surge legiti- Technol- cooperating with each mate accessing, and iden- ogy Deakin other. In such a federated tify attacks at the early University, network environment, stage, even before the at- Burwood, routers can work closely tack packages reaching the VIC 3125, to raise early warning target server. The en- Australia of DDoS attacks to void tropy of flows at a router, catastrophic damages. router entropy, is calcu- However, the attackers lated, if the router entropy simulate the normal is less than a given thresh- network behaviors, e.g. old, then a attack alarm pumping the attack is raised; the routers on packages as Poisson the path of the suspected distribution, to disable flow will calculate the en- detection algorithms. tropy rate of the suspected We noticed that the flow. If the entropy rates attackers use the same are the same or the differ- mathematical functions ence is less than a given to control the speed of value, then we can confirm attack package pumping that it is an attack, other- to the victim. Based wise, it is a surge of legit- on this observation, the imate accessing. different attack flows of a DDoS attack share the same regularities, which is different from the real surging accessing in a short time period. We apply information theory parameter, entropy rate, to discriminate the DDoS attack from the surge legitimate accessing. We proved the effectiveness of our method in theory, Here number of packets to different destinations are used. 4
- 6. Table 5: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Low-Rate DDoS Yang Xiang, 2011 A low-rate distributed de- we propose two new and Attacks Detection Member, nial of service (DDoS) at- effective information met- and Traceback by IEEE, Ke Li, tack has significant ability rics for low-rate DDoS at- Using New Infor- and Wanlei of concealing its traffic be- tacks detection: general- mation Metrics Zhou, Senior cause it is very much like ized en- tropy and in- Member, normal traffic. An infor- formation distance met- IEEE mation metric can quan- ric. The experimental re- tify the differences of net- sults show that these met- work traffic with various rics work effectively and probability distributions. stably. They out- per- In this paper, we innova- form the traditional Shan- tively propose using two non entropy and Kull- new information metrics back–Leibler distance ap- such as the generalized en- proaches, respectively, in tropy metric and the in- detecting anomaly traffic. formation distance metric In particular, these met- to detect low-rate DDoS rics can improve (or match attacks by measuring the the various re- quirements difference between legit- of) the systems’ detection imate traffic and attack sensitivity by effectively traffic. The proposed adjusting the value of or- generalized entropy met- der of the generalized en- ric can detect attacks sev- tropy and information dis- eral hops earlier than the tance metrics. As the traditional Shannon met- proposed metrics can in- ric. The proposed in- crease the information dis- formation distance met- tance (gap) between at- ric outperforms the pop- tack traffic and legitimate ular Kullback–Leibler di- traffic, they can effectively vergence approach as it detect low-rate DDoS at- can clearly enlarge the tacks early and reduce the adjudication distance and false positive rate clearly. then obtain the op- timal The pro- posed informa- detection sensitivity. The tion distance metric over- experimental results show comes the properties of that the proposed infor- asymmetric of both Kull- mation metrics can ef- back–Leibler and informa- fectively detect low-rate tion diver- gences. Fur- DDoS attacks and clearly thermore, the proposed IP reduce the false positive traceback scheme based rate. Furthermore, the on information metrics proposed IP traceback al- can effectively trace all gorithm can find all at- attacks until their own tacks as well as at- tackers LANs (zombies). In from their own local area conclusion, our proposed networks (LANs) and dis- infor- mation metrics can card attack packet substantially improve the performance of low-rate DDoS attacks detection 5 and IP traceback over the tra- ditional approaches.
- 7. Table 6: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Joint Entropy Hamza 2009 Network traffic charac- In this paper, we have Analysis Model Rahmani, terization with behaviour proposed statistical ap- for DDoS Attack Nabil Sahli, modelling could be a proach for DDoS attacks Detection Farouk good indication of attack detection. Our experi- Kammoun detection witch can be ences were made on a real CRISTAL performed via abnormal traffic flow issued from Lab., Na- behaviour identification. a “CAIDA data collec- tional School Moreover, it is hard to tion” collected in 2007. for Com- distinguish the difference Our proposed approach is puter Sci- of an unusual high volume based on the evaluation ences of of traffic which is caused of the degree of coherence Tunis Uni- by the attack or occurs between the received traf- versity when a huge number of fic volume and the num- campus users occasionally ac- ber of connections per Manouba cess the target machine time interval with the aim Manouba, at the same time. We of thresholding calculated Tunisia observe that the time distances between a cur- series of IP-flow number rent observation window and aggregate traffic size and a given reference. The are strongly statistically main contribution of this dependant. The occur- paper is that our proposal rence of attack affects this model allows us to identify dependence and causes DDoS attacks regardless a rupture in time series of the traffic volume size. of joint entropy values. A legitimate augmenta- Experiment results show tion at large scale will not that this method could be detected through this lead to more accurate method which minimising and effective DDoS de- false alarms. In addition, tection.We propose a our proposal does need to measurement method inspect few fields for each which focuses on quan- packet. This makes it sim- tifying the information pler and more practical for expressed by the joint real-time implementation. system of two random variables in traffic-based network. By measuring the degree of coherence between the number of packets and the number of IP-flow first obtained in regular traffic, then in traffics presenting a large variety of anoma- lies including mainly legitimate anomalies, we can differentiate traffic changes caused by flash crowd (FC) or by DoS 6 attack. This method allows reducing signifi- cantly the false positives alarms. To study the network characteristics by generating the histogram of the size of IP-flow during a timeinterval T.
- 8. Table 7: Entropy Based IEEE Papers Tittle Author Year Abstract Theme A Network Ya-ling 2009 A new network anomaly The RETAD sets up Anomaly De- Zhang, detection method has SVLNM by training the tection Method Zhao-guo been proposed in this normal network traffic. Based on Relative Han, Jiao- paper. The main idea of The network anomaly Entropy Theory xia Ren the method is network detection system based School of traffic is analyzed and es- on RET is achieved by Computer timated by using Relative comparing SVLD with Science and Entropy Theory (RET), SVLNM. The test results Engineering and a network anomaly show that the detection Xi’an Uni- detection model based on rate of RETAD is higher versity of RET is designed as well. than the EMERALD, Technology The numerical value of PHAD, ALAD, NETAD Xi’an, China relative entropy is used and FAD. The RETAD to alleviate the inherent has three advantages. contradictions between Firstly, algorithm compu- improving detection rate tation is so easy that it and reducing false alarm can be used to the high rate, which is more pre- speed network. Secondly, cise and can effectively the method has a strong reduce the error of es- detection capability, es- timation. On the 1999 pecially for the detection DARPA/Lincoln Labo- of intermittent anomalies. ratory IDS evaluation In addition, the RETAD data set, the detection has a good adaptability. results showed that the Based on RET, the packet method can reach a length has been chose higher detection rate at as measures to detect the premise of low false anomaly. Furthermore, alarm rate.These mea- the detection models sures have three features: using other measures need compose a full-probability to be further studied. event and cover all gath- ered information;be able to comprehensively reflect a variety of abnormal that cause the abnor- mal network traffic;does not contain sensitive information, such as IP address, port number or packet content informa- tion. Packet Lengths are taken into account to calculate relative entropy and drawing conclusions. 7
- 9. Table 8: Entropy Based IEEE Papers Tittle Author Year Abstract Theme An Approach on Zhiwen 2011 In this paper we propose The test data set with Detecting Network Wang, Qin an approach on detecting more alerts is used to eval- Attack Based on Xia De- network attack based on uate our method. There Entropy partment of entropy from millions of are 166,326 alerts in the Computer alerts. Shannon entropy test data. 9.83them are Science and is developed firstly to ana- generated by 86 network Technol- lyze the distribution char- attack occurs within 430 ogy Xi’an acteristics of alert with seconds. We successfully Jiaotong five key attributes includ- detect all the attacks with University ing source IP address, 2 false detections.In this Xi’an, China destination IP address, paper, we proposed a new source threat, destina- network attack detection tion threat and datagram method base on entropy. length. Then, the Renyi Five features of IDS alerts cross entropy is employed are selected from tens of to fuse the Shannon en- Snort alert attributions. tropy vector and detect The Shannon entropy is the anomalies. The IDS used to analyze the alerts used in our experiment is to measure the regularity Snort, and the experimen- of current network status. tal results based on actual The Renyi cross entropy network data show that is employed to detect net- our approach can detect work attack. The Renyi network attack quickly cross entropy value is near and accurately. In this 0 when the network runs paper, Snort is used to in normal, otherwise the monitor the network and value will change abruptly five statistical features of when attack occurs. The the Snort alert are se- experimental results un- lected: source IP address, der actual data show that destination IP address, the framework in our work source threat, destina- can detect network attack tion threat and datagram quickly and accurately. In length. The Shannon en- next step, more alerts tropy is used to analyze from different time seg- the distribution character- ments will be collected to istics of alert that reflect test our method and an at- the regularity of network tack classification method status. When the moni- will be considered. tored network runs in nor- mal way, the entropy val- ues are relatively smooth. Otherwise, the entropy value of one or more fea- tures would change. The Renyi cross entropy of these features is calculated to measure the network status and detect network 8 attacks. Time series is cal- culated based on shannon entropy and which is used to calculate renny entropy and compared with previ- ous and alarm is generated based on thereshod.
- 10. Table 9: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Detecting DDoS Yun Liu 2010 After analyzing the The results demonstrate Attacks Using Con- ,Jieren characteristics of DDoS that TFCE is more ro- ditional Entropy Cheng,Jianping attacks and the existing bust of the interference of Yin,Boyun approaches to detect background traffic. The Zhang DDoS attacks, a novel reason lies in the fact School of detection method based that the corresponding re- Computer, on conditional entropy lations between traffic fea- National is proposed in this pa- tures are considered here. University per. First, a group of TFCE compute the rele- of Defense statistical features based tive distribution between Technology on conditional entropy is traffic features and include Changsha, defined, which is named the information of joint China Traffic Feature Condi- probilities of traffic fea- tional Entropy (TFCE), tures, so has stronger abil- to depict the basic charac- ity to uncover the differ- teristics of DDoS attacks, ence of attack traffic and such as high traffic vol- normal traffic. ume and Multiple-to-one relationships. Then, a trained support vector machine (SVM) classifier is applied to identify the DDoS attacks. We experiment with the MIT Data Set in order to evaluate our approach. The results show that the proposed method not only can distinguish between attack traffic and normal traffic accurately, but also is more robustness to resist disturbance of back- ground traffic compared with its counterparts. Sr- cIP,DestIP,DestPort are taken into account.Then use three conditional entropy and sip sip dip H( ), H( )H( ) dip dport dport to characterize three kinds of multiple-to-one rela- tion in DDoS attacks, namely, called Traffic Fea- ture Conditional Entropy (TFCE).This measure the 9 diversity of sip to dip,sip to dport, dport to dip,or their uncertainity. After we include SVM into pic- ture ,train it with same set of factors and used it to detect real time anamoly.
- 11. Table 10: Entropy Based IEEE Papers Tittle Author Year Abstract Theme A New Relative Jin 2010 Distributed Denial of Ser- This paper analyzes the Entropy Based Wang,Xiaolong vice (abbreviated DDoS) application layer DDoS App-DDoS Detec- Yang Keping attack is a serious problem and proposes a new rel- tion Method Long Re- to the network services. ative entropy based app- search This paper analyzed some DDoS detection method. Center for solutions to the appli- We validate our method Optical cation layer DDoS (ab- by simulation, and the Internet breviated app-DDoS) at- results suggest that our Mobile In- tack, and proposed a rel- method can be used to fonnation ative entropy based app- detect app-DDoS attacks. Network, DDoS detection method. This paper validates the University Our scheme includes two usefulness of the relative of Electronic stages: learning stage and entropy based app-DDoS Science detection stage. Firstly at detection method. Our Technology the learning stage, it ex- future work will focus on of China, tracts main click features how to handle false detec- Chengdu of web objects with the tion. Sichuan cluster methods. Then 610056,China. at the detection stages, it Network computes the relative en- Center of tropy for each session ac- Chengdu cording to the learning re- University, sult. The greater the ses- Chengdu sion’s relative entropy, the Sichuan more suspicious the ses- 610106, sion is. At last, simula- China tion results suggest that this method can differenti- ate the attack session with high detection rate and low false alarm. 10
- 12. Table 11: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Entropy-based Suratose 2010 The most common type of In summary, an entropy- Input-Output Tritilanunt, DoS attack occurs when based technique provides Traffic Mode De- Suphannee adversaries flood a large more accurately denial-of- tection Scheme for Sivakorn, amount of bogus data service detection than a DoS/DDoS Attacks Choochern to interfere or disrupt volume-based technique. Juengjin- the service on the server. Moreover, the detecting charoen, Au- By using a volume- time to discover both sanee Siri- based scheme ,packe- long- term and short- pornpisan trate,bandwidth,packetsize term denial-of-service Computer to detect such attacks, attacks in our scheme Engineering this technique would not is another key strength Department, be able to inspect short- over a feature-based de- Faculty of term denial-of- service tection approach. These Engineering, attacks, as well as cannot two major advantages Mahidol distinguish between heavy are supported by the University, load from legitimate users experimental results as Thailand and huge number of bogus demonstrated in this sec- 25/25, messages from attackers. tion.Short term and long Salaya, As a result, this paper term attacks are detected. Phutta- provides a detection monthol, mechanism based on a Nakorn- technique of entropy- pathom, based input-output traffic Thailand, mode detection scheme. 73170 The experimental re- sults demonstrate that our approach is able to detect several kinds of denial-of-service attacks, even small spike of such attacks. This paper uses entropy of packet size to detect attacks. 11
- 13. Table 12: Entropy Based IEEE Papers Tittle Author Year Abstract Theme Entropy Based Laleh Ar- 2011 In this paper we present a The point is that as SYN Flooding shadi Amir novel approach for detect- the arrival rate decreases Detection Hossein ing SYN flooding attacks the packets become less Jahangir by investigating the en- dependent and the en- Computer tropy of SYN packet inter- tropy increases as a re- Engineering arrival times as a mea- sult whereas an increase Department sure of randomness. We in the arrival rate re- Sharif Uni- argue that normal SYN sults in more dependency versity of packets are almost inde- between the packets and Iran Tehran, pendent leading to higher a decrease in the en- Iran values of entropy while tropy consequently. There SYN flooding attacks con- are two major challenges sist of a high volume of faced by the anomaly de- related SYN packets and tection techniques. First so the entropy of their is the problem of defin- inter-arrival times would ing a general rule for be less than normal. We the distinction of normal apply this entropy-based and anomalous traffic and method on different data the second is the high sets of network traffic both volume of the processing in off-line and real-time data. We see that our modes. In this paper we entropy based detection examine the changes in technique can easily over- the entropy of inter-arrival come both challenges by times of TCP SYN pack- investigating the random- ets to detect SYN flood- ness of TCP SYN packets’ ing attacks. Our ex- inter-arrival times. While periments are based upon deriving the SYN pack- this argument that nor- ets, extracting their inter- mal SYN packets are al- arrival times and comput- most independent leading ing the entropy is not com- to higher values of en- putationally intensive and tropy while SYN flooding can easily be performed attacks consist of many in real-time As for fu- related SYN packets sent ture work it may be use- from either the same ori- ful to observe the entropy gin to various destinations of other flow inter-arrival or from multiple sources times, e.g. TCP-SYN- to a single destination and ACK, TCP- ACK, TCP- consequently the entropy RST, UDP or ICMP pack- of their inter-arrival times ets. In case the entropy would be less than normal. changes as an anomaly oc- curs, it would be possible to identify the anomalous portions of the traffic in the same way we detect the SYN flooding attacks 12