Traffic analysis and visualization

Analysis and visualization of network traffic is important for optimizing and protecting the operation of networked IT infrastructures.

The flow-based network profiling system is designed to gain tight control over end-to-end resource usage for hosts, servers, services, applications, protocols, domains, autonomous systems, quality-of-service classes, interfaces and user-defined combinations of these aspects.

The system operates passively by generating detailed network traffic reports from flow-information streams, such as NetFlow, IPFIX, jFlow, cflowd and NetStream. Traffic reports provide detailed asset usage information ranging from seconds to years. The profiling system supports network planning as well as network operation, for instance through identification of network congestion causes. It can also be used to estimate traffic impact with server consolidation and new application roll-outs (piloting).

Traffic usage reports are provided for bit, packet and flow rates in interactive tables, pie charts and graphs.

Total traffic volumes by direction (sent/received) for all configured aspects. Reports are automatically generated for selected periods in HTML, PDF, XML, JSON and as textual output. In addition, ad-hoc zoom reporting can be performed with filters for specific traffic aspects and periods.

The flow-based network profiling system supports domain accounting including 95th-percentile computation per direction.

User-defined combinations of traffic aspects provide better understanding of traffic flows. An example of a user-defined aspect is a quality-of-service breakdown by application and source / destination autonomous system.

Traffic flow profiling requires advanced database technology to handle high flow volumes in large enterprise and service provider infrastructures. The flow-based network profiling system uses an aggregation database (ADB) that was specifically designed for low memory and storage footprints.

ADB provides a mechanism for efficient incremental storage of primary traffic data values associated with time intervals. The database stores data values in groups of circular arrays of decreasing resolution and is, therefore, able to handle large flow data sets with short access times and limited storage. ADB automatically ensures that the array resolution of older data values is lower than the resolution of newer data values. Additionally, the design of ADB reduces memory to disk synchronization and accelerates data import and export. Users can quickly change their viewpoint when looking at traffic flows. Because data is represented in ADB for multiple viewpoints, flow files do not have to be reanalyzed or newly indexed.

Array grouping in ADB is efficient for obtaining a sorted view of related parameters. This feature is of great importance for efficiently displaying sorted lists of top protocols, top hosts, top flows, etc. ADB supports period durations of hour, day, week, month quarter and year. The resolution of every period can be adjusted.

Flow-based Network Profiling System is designed for high performance. The system makes use of parallelism on multi-core architectures and uses a fast in-memory aggregation database (ADB). A single installation can accommodate flow-information records exported from many routers, switches and interfaces. On a typical dual-core server configuration, a processing speed of 50 K flows per second can be achieved. Higher flow rates require more resources or a distributed setup.

The user interface of Flow-based Network Profiling System is web-based (over Secure Socket Layer). Built-in user management provides user roles with password-protected access via local or LDAP authentication.

The user interface supports multiple languages (UTF enabled) and skins. Each end-user can choose a language and skin individually. Configuration of user roles, aspect composition and item grouping (e.g., domain definition) is supported in the user interface.

Tables, pie charts and graphs are linked and enabled for interactive drill down based on AJAX/SVG. The interactive functions comprise:

Flow-based Network Profiling System offers an application programming interface (API) to its backend. All configuration, control and database access functions are supported by the API. The API is also used for scripted output (e.g., CSV and PDF). Advanced users can write custom scripts to export periodically into a desired output format. The scripting language can, furthermore, be used for event notification. Events can be signaled, for instance, as syslog messages.

Flow-based Network Profiling System is integrated with IBM Tivoli Netcool/Proviso.