AURORA: Traffic analysis and visualization

AURORA screenshot

Overview

AURORA is an IBM Research project and the name of a traffic analysis and visualization system. The research project is targeted at flow-based network traffic analysis and visualization for very large networks. We run several AURORA sub-projects on anomaly and virus detection/mitigation, network forensics, distributed flow processing, BGP/OSPF/RIP monitoring, traffic network maps and indexing of very large flow repositories. The base AURORA system is now commercially available as Tivoli Netcool Performance Flow Analyzer (TNPFA). The research system includes special pre-product components and features.

The project investigates new techniques for collecting, storing and analyzing flow-based network traffic information. The techniques help to optimize and protect business-critical networked infrastructures through tight control of resource usage. With server relationship and dependency discovery components, we support sequencing of server relocations and the understanding of what has been deployed to support a business-level process or value chain. We are furthermore able to identify server consolidation opportunities based on load and utilization tracking.

The AURORA / TNPFA system operates passively by generating detailed network traffic reports from NetFlow / IPFIX. Traffic reports show detailed host and application communication patterns including protocol and server usage trends. AURORA supports network planning as well as operation, for instance through identification of network congestion causes. The system can also be used to estimate traffic impact with consolidation and application roll-outs.

Publications

  1. F. Fusco, M. Vlachos, M.P. Stoecklin
    "Real-time creation of bitmap indexes on streaming network data"
    The VLDB Journal, Online First, 29 July 2011.
  2. L. Deri, S. Mainardi, F. Fusco
    "tsdb: A Compressed Database for Time Series"
    Proc. 4th Int'l Workshop TMA 2012, LNCS 7189, Springer 2012.
  3. A. Cardigliano, L. Deri, J. Gasparakis, F. Fusco
    "vPF_RING: towards wire-speed network monitoring using virtual machines"
    Proc. 2011 ACM SIGCOMM conference IMC '11 on Internet measurement conference
  4. F. Fusco, X. Dimitropoulos, M. Vlachos, L. Deri
    "pcapIndex: An Index for Network Packet Traces with Legacy Compatibility"
    Computer Communication Review (CCR), Vol. 42, No. 1, Jan 2012.
  5. T. Locher
    "Finding Heavy Distinct Hitters in Data Streams"
    Proc. 23rd ACM Symp. on Parallelism in Algorithms and Architectures "SPAA '11," San Jose, CA (ACM, June 2011) 299-308.
  6. F. Fusco, M. Stoecklin, M. Vlachos
    "NET-FLi: On-the-Fly Compression, Archiving and Indexing of Streaming Network Traffic"
    Proc. 36th International Conference on Very Large Data Bases "VLDB 2010," Singapore, vol. 3(2), (PVLDB On-line, September 2010) 1382-1393.
  7. P. Siska, M. Stoecklin, A. Kind
    "A Flow Trace Generator using Graph-based Traffic Classification Techniques"
    Proc. 6th Int'l Wireless Communications and Mobile Computing Conf.,1st Int'l Workshop on Traffic Analysis and Classification "TRAC," Caen, France, (ACM, June 2010) 457-462.
  8. F. Fusco, L. Deri
    "High Speed Network Traffic Analysis with Commodity Multi-Core Systems"
    Proc. 10th Annual Conf. on Internet measurement "IMC '10," Melbourne, Australia (ACM, November 2010) 218-242.
  9. F. Fusco, L. Deri, J. Gasparakis
    "Towards Monitoring Programmability in Future Internet: Challenges and Solutions"
    Proc. 21st Tyrrhenian Workshop on Digital Communications: Trustworthy Internet "ITWDC," Ponza, Italy, September 6-8, 2010 (Springer, in press).
  10. L. Deri, J. Gasparakis, P. Waskiewicz, F. Fusco
    "Wire-Speed Hardware-Assisted Traffic Filtering with Mainstream Network Adapters"
    Proc. 1st IEEE/IFIP Int'l Workshop on Network Embedded Management & Applications "NEMA 2010," Niagra Falls, Canada, October 28, 2010 (Springer, in press).
  11. M. Stoecklin, J.-Y. Le Boudec, A. Kind
    "A Two Layered Anomaly Detection Technique Based on Multi-Modal Flow Behavior Models"
    in "Passive and Active Network Measurement," Proc. PAM '08, Lecture Notes in Computer Science, vol. 4979, edited by M. Claypool and S. Uhlig (Springer Verlag, 2008), pp. 212-221.
  12. Andreas Kind, Marc Stoecklin, Xenofontas Dimitropoulos
    "Histogram-based Traffic Anomaly Detection,"
    IEEE Transactions on Network and Service Management, 2009 (to appear).
  13. Xenofontas Dimitropoulos, Paul Hurley, Andreas Kind, Marc Stoecklin
    "On the 95-percentile billing method,"
    In Springer Proceedings of Passive and Active Measurements (PAM) Conference 2009.
  14. Xenofontas Dimitropoulos, Paul Hurley, and Andreas Kind
    "Probabilistic Lossy Counting: An Efficient Algorithm for Finding Heavy Hitters,"
    ACM SIGCOMM Computer Communication Review, Jan. 2008.
  15. Xenofontas Dimitropoulos, Marc Stoecklin, Paul Hurley, and Andreas Kind
    "The Eternal Sunshine of the Sketch Data Structure,"
    Elsevier Computer Networks, 2008.
  16. Alexandru Caracas¸ Dimitrios Dechouniotis, Stefan Fussenegger, Dieter Gantenbein, Andreas Kind
    "Mining Semantic Relations using NetFlow,"
    Third IEEE/IFIP International Workshop on Business-driven IT Management (BDIM 2008), in conjunction with IEEE Network Operations and Management Symposium (NOMS 2008), Salvador, Bahia, Brazil, April 7, 2008.
  17. Dimitrios Dechouniotis, Xenofontas Dimitropoulos, Andreas Kind, and Spyros Denazis
    "Dependency Detection Using a Fuzzy Engine,"
    18th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM '07), October 29-31, San Jose, California, USA, 2007.
  18. Andreas Kind, Dieter Gantenbein, and Hiroaki Etoh
    "Relationship Discovery with NetFlow to Enable Business-Driven IT Management,"
    In Proceedings of IEEE/IFIP International Workshop on Business-Driven IT Management (BDIM), 2006.
  19. Marc Stoecklin and Andreas Kind
    "Dynamic Adaptation of Flow Information Granularity for Incident Analysis,"
    In Proceedings of CERT FloCon 2008.
  20. Xenofontas Dimitropoulos and Andreas Kind
    "Automating the Configuration of Flow Monitoring Probes,"
    In Proceedings of CERT FloCon 2008.
  21. Marc Stoecklin, Jean-Yves Le Boudec, and Andreas Kind
    "A Two-Layered Anomaly Detection Technique based on Multi-modal Flow Behavior Models,"
    In Springer Proceedings of Passive and Active Measurements Conference 2008.
  22. Marc Stoecklin
    "Anomaly Detection by Finding Feature Distribution Outliers,"
    In Proceedings of CoNEXT Conference 2006 (poster).