Yann Dirson ydirson@altern.org This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License, version 2, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. 2005 Yann Dirson

Graph-includes creates a graph of dependencies between source-files and/or groups of source-files, with an emphasis on getting readable and usable graphs even for large projects. Usability of the dependency graphs are currently improved by: customizable grouping of several source files into a single node transitive reduction of the graph It currently supports graphing the C/C++ #include relationship, using graphviz.

This tool has evolved from a 50-line script written for a particular project (Battle for Wesnoth). Although it has been generalized much, there are still somewhat ad-hoc heuristics harcoded here and there, especially in the default project class (see class descriptions below). Although work is under way to make this tool as generic as possible, work still has to be done at all levels. It is still under development, and may not suit your needs (at least, not yet).

Be sure you have a recent version of Perl installed. At least List::Util is missing from versions earlier than 5.8. You can also just fetch this additional package from CPAN if you cannot upgrade. If you notice that another package is missing from your installation, please report it, so it can be listed here. Install it like standard perl packages. $ perl Makefile.PL prefix=/usr/local $ make $ su # make install Be sure that the directory in which the library modules got installed is in your perl library path. Eg, if "graph-includes --version" does not give the expected result, try setting the PERL5LIB environment variable to (in the above example) /usr/local/share/perl/5.8.4/. New versions can be found at http://ydirson.free.fr/soft/graph-includes/. A darcs repository is available at http://ydirson.free.fr/soft/graph-includes/darcs/. To be able to format the produced graphs, you will need one of graphviz and tulip

Graph-includes is only a supporting tool for a refactoring effort. It can be useful in helping a developper to see where he should put its efforts in order to get cleaner and saner dependencies in a project. In this respect, it is quite similar to a microscope: if you don't look at the right place, you won't see anything interesting. But if you start with a small magnifying factor, you can locate regions of interest, and then zoom on those to get to the interesting stuff.

When developping a project of medium size (we'll talk mostly C/C++ here, but that will apply to most languages), expecially with many people writing code, it is quite easy to get to a point where each file (out of several tens of hundreds of files) depends on too many other files. The most obvious relation is the #include one. The more #includes a file has, the more time it takes to build - especially when those included files #include themselves a bunch of other files. For a project of about 100 files, just producing a graph of all those files, with arrows representing the #include dependencies, will usually give an unreadable graph, and will show very little about possible improvements. This is why this tool has been written: to make it possible to get to the useful information hidden in this unusable dependency graph.

A less obvious relation appears more clearly when you consider not files by themselves, but the set of files made of an interface and the matching implementation. Let's consider two such sets, made of the files a.h, a.c, b.h, b.c. a.c includes b.h, and b.c includes a.h, and each implementation, following good practice, includes its own interface. A simple dependency graph as described above would show such a graph: a.c -> b.h \ /| \/ / / \| b.c -> a.h If OTOH we represent those sets of files instead of the files themselves, we now have something like: a <--> b This shows much more clearly that those two modules are intrinsicately related. In many cases, this will express that whenever you use the a.o file resulting from the build of a.c, you'll need to link b.o as well, and vice versa. This will be the case when each file uses the headers to get function prototypes. Then hunting for abusive dependencies will allow, for example, to select with finer grain which of those modules of code will need to go into which executable, thus producing lighter executables. Note that such a reciprocal dependency may not be pathological. Many projects tend to split a large module into several files for clarity, even when those files are inter-dependant. It is much often in cycles of unidirectional dependencies that we find dependencies that should not be there. In other cases, headers would just have been used to access a type definition from b.h, and the associated b.o would not be needed. In such cases, you may want to consider splitting such "low-level" declarations into their own headers. Not only this would simplify the graph, allowing you to get a better grasp on your source code, but it can also lead to faster compilations, since each file will be able include less unrelated definitions.

Your first run will surely looks somewhat like: graph-includes -o project.ps src/ lib/ You will take care of specifying all directories or individual source files that make up your project. In addition to an initial graph in the project.ps file, which is quite likely to be incomplete by far, you will find a file named project.ps.graph-includes.report. It is a text file, which will help us to finetune ou command-line. Its first section will look something like: General statistics: ------------------- 412 files, 353 nodes (14% dropped) 245 dependencies, 137 edges (44% dropped) 225 leaf node(s) 280 dependencies not found 0 dependencies identified as system headers As you can see, many dependencies are declared as "not found". What happens is quite similar to running a C compiler without any -I flags: most header files are not located. We have in graph-includes two different flags to specify paths where to look for the dependencies. -I (aka -Include) specifies directories that are part of the project, and will allow to find all of our include-style dependencies. OTOH, -sysI (aka -sysInclude) specifies system directories; included files found in such a directory will of course not result in an intra-project dependency, and will add no edge to our graph, but will stop being displayed as part of the "dependencies not found" count. Thus, they will help us to see how far we are from specifying all the -I flags. Now you will most likely require several iterations of adding -I/-sysI flags and checking the results. But that alone may not be sufficient to reach the ultimate "0 dependencies not found": multi-platform source often have conditional #include directives, and eg. win32 headers will probably not be located on a Un*x box. some generated files will require the source tree to be configured in some way, or even to be partly or completely built (eg. config.h generated by a "configure" script, or Qt source generated by the meta-object compiler) When you are confident that those remaining missing dependencies are system headers for other platforms, you can go on and look at the graph.

Keeping in mind that we are essentially looking for dependency loops, we expect to obtain in then end a graph that will be wihout cycles, that is, with all (or, at least, most of) arrows pointing from left to right in our graph. Then we will look for those arrows pointing backwards, as a sure sign for a cycle. Remember that if the cycle is not a long one, it may be legitimate; only if you judge that some of the modules in this cycle are really unrelated, should your consider it pathological. Those backward arrows are not necessarily directly pointing to the abusive dependency, but they can surely be used to locate the culprit: by finding the various cycles of which our backward arrows are part of, and checking one by one all the dependencies in those cycles, you can bet at least one that, with some work, could be cleared. Then, the way to modifications to do are really dependant on your code. Some possibilities include: removing an #include which is not necessary, perhaps remaining from a revious code reorganization splitting a file in two parts, when you can easily split the components of the file into distinct sets. One productive distinction to look for is to find a couple of really high-level parts, that not all the parts depending on this module would need. This will most probably be related to the dependency that you found abusive when looking at the cycle in the graph.

Graph-includes was initially developped with only a handful of ideas, and then started to grow as I noticed where useful things were missing. That initial phase was useful for me to get a grasp on the domain of dependency graphing, and provided the ground for a (hopefully) decent design, which still has to be completely implemented. Together with blocking issues marked + in the TODO list, the implementation of this design shall be the goal for a 1.0 release. The planned design is architectured as successive layers, all of which should be pluggable to allow a high degree of customization. source locator | v dependency extractor | v graph transformations | v styling | v layout engine | v rendering We will then be able to consider the graph-includes project as being made of a number of parts: core classes and glue, implementing the above design standard classes, doing the real work command-line and gui tools to allow easy use of the whole This will hopefully make it easy for anyone to plug their own work at any place in the architecture.

Currently, only the extractor and the renderer are properly customizable. The source locator takes parameters from the per-language extractors to find files. Project classes allow some customisation of the grouping transformation, and of styling, but this has to be split and generalized, as the design plans imply. Graph transformations are mostly limited to a set of hardcoded operations (grouping, transitive reduction, edge labelling), not all of which can be individually switched off; only the "special edge" mechanism is customizable to some degree. There is no distinction (yet ?) between the layout engine and the renderer. In fact, it may not be easy to do this, since most layout engines are tied to a particular renderer.

See "graph-includes --help".

The default output is a .dot file on standard output, suitable for formatting by dot (from the graphviz toolkit), or interactive editing by dotty (also from graphviz). Alternatively, a graph file for the Tulip graph visualizer can be generated instead using "--renderer=tulip". You can ask graph-includes to do the formatting for you, eg. using "--output=<file>.<suffix>". It will run "dot -T<suffix>", so that "--output=mydeps.ps" or "--output=mydeps.jpg" will have the expected behaviour. If your suffix is not known to dot, it will complain itself, so asking for --output=foo.bar will cause a message like: Warning: language bar not recognized, use one of: canon cmap cmapx dia dot fig gd gd2 gif hpgl imap ismap jpeg jpg mif mp pcl pic plain plain-ext png ps ps2 svg svgz vrml vtx wbmp xdot If you intend to print the result on paper, the default layout will likely be too large. You can use --paper=a4 to select parameters that will produce a smaller graph and spilt it into pages. This flag also changes the default output format to postscript. Be warned that dot may not honor the page-splitting parameter for all output formats. Since the transitive reduction can take time, you may like the --verbose switch, which will show a progress bar.

The files to be analyzed are given as non-option arguments, and can be explicitely specified, or found by recursing in directories. Eg, to analyse foo.c in current directory, as well as all C/C++ files in the src/ directory, use: $ graph-includes foo.c src/ When an directory argument is specified, it is searched for files whose name matches a specific regexp pattern, whose default value depends on the specified language (see --language below). This pattern can be overriden using the --fileregexp option. Eg, to match in addition to .c and .h files, those with an additional .tmpl suffix, you could write: $ graph-includes -I src -fileregexp '\.[ch](\.tmpl)?$' src/ How dependencies get extracted from the source files depend on the language used in those files. You can specify it with the --language flag. Default value is C (which should also be used for other languages based on the C preprocessor, like C++). There is also some partial support for perl - see comments in lib/graphincludes/extractor/perl.pm for more details. In order to tell the #include resolver where to look for included files, you can use the cpp-like -I (aka. --Include) flag. Eg: $ graph-includes -I src src/ Dependencies not found in the project (ie. files appearing in #include but not given on command-line) are listed as "not found" in the graph-includes.report file for diagnostics purposes, unless they are found in a system directory. System directories are declared in a similar fashion, with the --sysInclude option. Eg: $ graph-includes -I src -sysI /opt/foo/include src/ Language extractor have some knowledge about default system include dirs: the C extractor knows about /usr/include, and the Perl extractor asks perl itself. To avoid having useless information on the graph, --prefixstrip=<prefix> can be used to avoid repeating a given prefix in all node labels. Typically: $ graph-includes --prefixstrip=src/ src/

Files will be grouped in a hierarchy of groups, level 0 groups typically containing just one file. Groups are defined by the selected project class, selected by the --class=<class> option. See below for descriptions of the project classes available by default, and for instructions to write customized project classes. The range of group levels to be drawn is selected with --group=<min>-<max>, which defaults to 1-1. Eg, for class "default", whose group levels are defined as: 0 one file per group 1 what/ever.* go into a what/ever group (usually interface + implementation) 2 what/* go into a what group, supposing top-level directories denote modules of some sort Group levels below "min" or above "max" are not displayed as nodes. Groups of level "min" are drawn as nodes of the graph. If "max" is strictly greater than "min", then groups of levels "min+1" through "max" are drawn as box clusters containing lower-level groups. Since such a way of grouping nodes will not improve the readability in projects where the inter-groups dependencies have not been cleaned up yet, higher-level groups can instead be colored, using a class-defined color scheme, possibly modified by --color <n>:<label>=<color>[,<label>=<color>…] options, where <n> is the group level in which the group name <label> will receive a background of the specified color, which can be defined either by a named X11 color (like "blue" or "palegreen"), or by a RGB color using the standard X11 "#RRGGBB" syntax. The number of grouping levels to be colored is limited by the renderer to be used. As of 0.11, the dot renderer only supports coloring 2 group levels. Groups of a lower level than the minimal level requested to --group cannot be colored, for obvious reasons. For those wanting to see what edges the transitive reduction dropped, the --showdropped will add them to the graph in a different color. Be prepared for your computer room to get a noticeable temperature increase for anything else than a small set of files with only few dependencies. OTOH, --focus=node-label will do the same, but only for the dependencies of a specified node. That should prevent the nasty effects described above, and will be useful for various purposes, including debugging the transitive reducer. The node-label refers to a node in the lowest group-level drawn, ie. the "min" argument to --group. People still getting cold may also like to circumvent the transitive-reduction engine completely, using --alldeps. The author assumes no responsibility for losses of mental health induced by trying to make any serious use of the resulting graph.

As implied by its name, it is the one which will be used unless you use the --class option. Although it is the default one, it may still be quite rough at the moment, still using some ad-hoc heuristics, and will be improved in the near future. Here are its main characteristics: looks at C-style #include lines creates level-1 groups for all files sharing the same path and (disregarding the suffix) filename. Eg, files "foo/bar.c" and "foo/bar.h" would be grouped in a "foo/bar" level-1 group. In clear, it won't connect include files if they are all located in an include/ directory. creates by-directory level-2 groups. Eg. in the above example, a group "foo" would exist at level-2.

Built on top of the default class, it is meant for projects where file names are kept unique across all directories. If the ad-hoc #include processing of the default class does not suit your project, it is the only out-of-the-box alternative available today. Here are its main characteristics: provides a single grouping level based on filenames, disregarding all the directory hierarchy. Note that it is not meant for general use, as: it will group any files with the same name in the same level-0 group, possibly causing confusion. it does not make any directory name appear in the node names

$ ./graph-includes -lang perl -I lib -prefixstrip lib/ -o deps.ps graph-includes lib/ graph-includes does not know in advance which classes it will use

a rather clean dependency graph Maelstrom-3.0.6$ graph-includes -v -sysI /usr/include/SDL -I . -I ./netlogic -I ./maclib -I ./screenlib --prefixstrip ./ -o deps.ps . more work has to be put in the wesnoth example class: wesnoth-0.9.1$ graph-includes -v --class wesnoth --group 1-1 -sysI /usr/include/c++/3.3 -sysI /usr/include/SDL --prefixstrip src/ -I src -o deps.ps src/

needs supporting features for multi-arch source trees: qemu-0.7.0$ graph-includes -v -sysI /usr/include/SDL $(find -name CVS -prune -o -type d -printf -I %p\n) -o deps.ps . needs proper file-grouping: mesag-6.2.1$ graph-includes -o -I ./include -I ./include/GL -I ./src/mesa -I ./src/mesa/main -I ./src/glu/sgi/include -I ./src/glu/sgi/libnurbs/internals -I ./src/mesa/glapi -o deps.ps .

See graphincludes::project::wesnoth in the examples/ dir as an example of a custom project class. Keep in mind that the API is not frozen yet, and will probably be overhauled more than once before an official API gets blessed.

this script only handles explicitely-declared dependencies, it won't detect it if eg. a prototype was cut'n'pasted instead of using the correct #include, but you shouldn't do that anyway :)

I finally found a couple of tools out there, from which I may borrow ideas some day. I'd be happy to hear about more of them. cinclude2dot, originally from Darxus (http://www.chaosreigns.com/code/cinclude2dot/), then taken over by F. Flourish (http://www.flourish.org/cinclude2dot/) is a GPL C/C++-only tool, which apparently has support for grouping, but not for transitive reduction. Should I have searched better, and found it a couple of months ago, maybe graph-includes would have never been developped :) http://www.tarind.com/depgraph.html has a dependency grapher for python, without transitive reduction as well. It does however allow customisation of project classes, somewhat similar to graph-includes. OptimalAdvisor (http://javacentral.compuware.com/pasta/) is a refactoring tool, which goes far beyond simple dependency analysis, but is non-free/libre/open-source (also they have a functionally-limited free/gratis edition) and seems to support only java. codeproject.com has some VisualStudio(tm) plugins targetting C++, which I cannot test, but appear to scale badly for large projects (http://www.codeproject.com/csharp/DependencyGraph.asp).

consider using Set::Object instead of hashes for sets continue merging the verbose/debug behaviour into the global report file change case of class names when the API gets stabilized finalize filename portability support, using File::Spec volume information

allow to associate attributes to files (eg. an ARCH attribute for multi-architecture trees, like kernels, development tools and emulators) modularization (finish the restructuring into a cleaner and more modular design) rework the recording of edges to make them apply to files, not to graph nodes, since more advanced features will need more flexibility allow passing options to modules (-O param=value ?) separate styling from project classes allow to define several views in a project-class, several of which can be generated by default. find out whether we can declare protocols/pure-virtual-classes in some way, to cleanup the class graph generalize --prefix-strip give consistent access to all commonly-needed features through command-line and class customization generalize the special_edge() mechanism (use a hash of edge attributes ?) Maybe allow to use as nodes other objects than files (eg. URI objects ?), for ultimate generalization.

allow to run from a build directory --class does not allow to find the project file (need to set PERL5LIB) caller must prepend path to source tree to --prefixstrip and all relative -I flags find the accessory classes as easily as possible (like bugzilla ?) better robustness to incorrect arguments (eg. --group 1:2) automate --help production (see Pod::Usage ?) multi-sheet paper support may be broken use an existing source of paper formats (libpaper, LC_PAPER, whatever) maybe use graphviz' tred(1) to check our transitive reductions. some autodetection of the language to use based on filenames ? provide an initial list of system directories to avoid repeating them (ask compiler)

allow -I syntax for programs using eg. -I. from source subdirectory behave as expected wrt leading "./", use File::Spec for more portability consider using Cwd::realpath or so, for correct "../" handling write other extractors (java, python, …) some support for CPP symbol conditionals (mostly #ifdef), perhaps coupling this with attributes extract more fine-grained dependency (depending on a header does not necessarily imply depending on code) handle (warn about) the case where the declarations for a given implementation file are scattered in more than one header detect undeclared dependencies (eg. manually inserted prototypes) check necessity of declared includes remove arbitrary limitations report use/require lines we could not completely parse do some invariant analysis when importing a module using a variable name. Investigate drawing 1->n links in this case.

proper way to define include paths in project class make default project-class consider multiple levels of directories as group levels, but only if they (consistently ?) have multiple subgroups ? write a linux-kernel class and others as examples :) provide a simple hash-based filelabel implementation

Abstract the grouping process into GroupSet objects, computed independently from other processes, and customizable Provide a regexp-based grouper, useful from command-line, and as a base for current level-1 grouping, and possibly to automatic per-directory grouping provide tools for automatic grouping (eg. using cycles, or selected external deps, or from leaves) Give more weight to intra-group edges Improve transitive reductions implying multiple edges into a single cycle, to prefer an edge into the lowest-level common group

allow styling through font color, node shape (dot/tulip), number of peripheries (dot) allow to draw non-consecutive group levels (eg. --groups 1,3) allow different node shapes when mixing high-level nodes with lower-level ones through the default singleton groups (special_node mechanism similar to the special_edge one ?) optionally show labels (using attributes ?) or count for files (subnodes) in a node and color arcs according to them optionally show external deps (deps on files not on command-line) limit graph to one or more given group(s) of files (specified by <level>:<label>) draw cycles in a given color draw a specific path provide automatic coloring schemes color intra-group edges with the same color as nodes (post-processing ?) allow to request drawing of who in a high-level node points to another node (ie. violates some constraint) propagate excuses in some way when they are dropped by the transitive reducer investigate candidate tools for hyperbolic layout ? allow to show the count of deps in a given edge using line width instead of labels

write more documentation

write a testsuite. ensure that all provided non-abstract classes are self-contained

The standard GUI should be able to navigate the project definition, visualizing the hierarchy of groups, (un)folding groups and edges, displaying single culprits from a group, and anything you can think of. For an engine, maybe with graphviz' lefty, or write a specialized tulip gui ? A self-customizable GUI like entity may be a good idea. Since entity has support for OpenGL areas, maybe it can be made to embed tulip graphs.

volume names in paths are not handled yet (eg. on windows) on windows, backslash path separator in -prefixstrip argument interferes with regexp ("Trailing \ in regex m/ ... at wesnoth.pm line 21") traversal counts on edges are unreasonably high on cycles (real graph traversal issue, but unlikely to get fixed before we get another visual way to spot cycles ;) the colored style of a node of level < min is not shown when that node is displayed because it is not part of any node > min and < max. --showdropped mode draws too many edges as dropped (ie. does not consider marked edges as dropped when deciding whether to consider subsequent edges as dropped) transitive reduction may not be complete, some more edges could possibly be dropped - wesnoth tree at 2005-03-25 exhibits the problem with the "display -> builder -> animated -> image" path