AURORA: Traffic analysis and visualization

AURORA is an IBM Research project and the name of a traffic analysis and visualization system. The research project is targeted at flow-based network traffic analysis and visualization for very large networks. We run several AURORA sub-projects on anomaly and virus detection/mitigation, network forensics, distributed flow processing, BGP/OSPF/RIP monitoring, traffic network maps and indexing of very large flow repositories. The base AURORA system is now commercially available as Tivoli Netcool Performance Flow Analyzer (TNPFA). The research system includes special pre-product components and features.

The project investigates new techniques for collecting, storing and analyzing flow-based network traffic information. The techniques help to optimize and protect business-critical networked infrastructures through tight control of resource usage. With server relationship and dependency discovery components, we support sequencing of server relocations and the understanding of what has been deployed to support a business-level process or value chain. We are furthermore able to identify server consolidation opportunities based on load and utilization tracking.

The AURORA / TNPFA system operates passively by generating detailed network traffic reports from NetFlow / IPFIX. Traffic reports show detailed host and application communication patterns including protocol and server usage trends. AURORA supports network planning as well as operation, for instance through identification of network congestion causes. The system can also be used to estimate traffic impact with consolidation and application roll-outs.

1. Andreas Kind, Marc Stoecklin, Xenofontas Dimitropoulos
   "Histogram-based Traffic Anomaly Detection,"
   IEEE Transactions on Network and Service Management, 2009 (to appear).
2. Xenofontas Dimitropoulos, Paul Hurley, Andreas Kind, Marc Stoecklin
   "On the 95-percentile billing method,"
   In Springer Proceedings of Passive and Active Measurements (PAM) Conference 2009.
3. Xenofontas Dimitropoulos, Paul Hurley, and Andreas Kind
   "Probabilistic Lossy Counting: An Efficient Algorithm for Finding Heavy Hitters,"
   ACM SIGCOMM Computer Communication Review, Jan. 2008.
4. Xenofontas Dimitropoulos, Marc Stoecklin, Paul Hurley, and Andreas Kind
   "The Eternal Sunshine of the Sketch Data Structure,"
   Elsevier Computer Networks, 2008.
5. Alexandru Caracas¸ Dimitrios Dechouniotis, Stefan Fussenegger, Dieter Gantenbein, Andreas Kind
   "Mining Semantic Relations using NetFlow,"
   Third IEEE/IFIP International Workshop on Business-driven IT Management (BDIM 2008), in conjunction with IEEE Network Operations and Management Symposium (NOMS 2008), Salvador, Bahia, Brazil, April 7, 2008.
6. Dimitrios Dechouniotis, Xenofontas Dimitropoulos, Andreas Kind, and Spyros Denazis
   "Dependency Detection Using a Fuzzy Engine,"
   18th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM '07), October 29-31, San Jose, California, USA, 2007.
7. Andreas Kind, Dieter Gantenbein, and Hiroaki Etoh
   "Relationship Discovery with NetFlow to Enable Business-Driven IT Management,"
   In Proceedings of IEEE/IFIP International Workshop on Business-Driven IT Management (BDIM), 2006.
8. Marc Stoecklin and Andreas Kind
   "Dynamic Adaptation of Flow Information Granularity for Incident Analysis,"
   In Proceedings of CERT FloCon 2008.
9. Xenofontas Dimitropoulos and Andreas Kind
   "Automating the Configuration of Flow Monitoring Probes,"
   In Proceedings of CERT FloCon 2008.
10. Marc Stoecklin, Jean-Yves Le Boudec, and Andreas Kind
    "A Two-Layered Anomaly Detection Technique based on Multi-modal Flow Behavior Models,"
    In Springer Proceedings of Passive and Active Measurements Conference 2008.
11. Marc Stoecklin
    "Anomaly Detection by Finding Feature Distribution Outliers,"
    In Proceedings of CoNEXT Conference 2006 (poster).