ROSE Compiler Framework/PreprocessingInfo

Overivew
In addition to nodes and edges, ROSE AST may have attributes in addition to nodes and edges that are attached for preprocessing information like #include or #if .. #else. They are attached before, after, or within a nearby AST node (only the one with source location information.)

An example translator will traverse the input code's AST and dump information which may include preprocessing information.
 * https://github.com/rose-compiler/rose-develop/blob/master/exampleTranslators/defaultTranslator/preprocessingInfoDumper.C

For example

exampleTranslators/defaultTranslator/preprocessingInfoDumper -c main.cxx --- Found an IR node with preprocessing Info attached: (memory address: 0x2b7e1852c7d0 Sage type: SgFunctionDeclaration) in file /export/tmp.liao6/workspace/userSupport/main.cxx (line 3 column 1) -PreprocessingInfo #0 --- : classification = CpreprocessorIncludeDeclaration: String format = #include "all_headers.h"

relative position is = before

Source: http://www.rosecompiler.org/ROSE_Tutorial/ROSE-Tutorial.pdf (Chapter 29 - Handling Comments, Preprocessor Directives, And Adding Arbitrary Text to Generated Code)

What is PreprocessingInfo
It is a type of ROSE AST meta information to store source code comments.

They do not exist independently, but are attached to located sgnode.
 * before: before a code segment
 * after: after a code segment
 * inside: inside an empty scope

PreprocessingInfo::RelativePositionType position=PreprocessingInfo::before

typedef Rose_STL_Container AttachedPreprocessingInfoType;


 * source comments: c style, c++ style
 * preprocessor info: include, define, ifdef, line declaration
 * and more

SageInterface::dumpPreprocInfo
void 	dumpPreprocInfo (SgLocatedNode *locatedNode)

Dumps a located node's preprocessing information.

(gdb) p SageInterface::dumpPreprocInfo(scope) ---  Found an IR node (at 0x7ffff7ec2230 of type: SgGlobal) in file rose_test_20_2019_lib.cpp with attached preprocessingInfo numbering #0 :- classification= CplusplusStyleComment: String format:// tool_G -roseoutline:use_dlopen -rose:outline:copy_orig_file -rose:unparseHeaderFiles -c test_20.cpp

relative position is: after with attached preprocessingInfo numbering #1 :- classification= CplusplusStyleComment: String format://

relative position is: after with attached preprocessingInfo numbering #2 :- classification= CplusplusStyleComment: String format:// This causes the reprocessing of generated files : generating rose_rose_xxx_lib_lib.cp

relative position is: after with attached preprocessingInfo numbering #3 :- classification= CplusplusStyleComment: String format://

relative position is: after with attached preprocessingInfo numbering #4 :- classification= CpreprocessorIncludeDeclaration: String format:#include "test_20_2019.h"

relative position is: after with attached preprocessingInfo numbering #5 :- classification= CplusplusStyleComment: String format:// other comments here

relative position is: after $4 = void

the builtin preprocessingInfoDumper
This tool is not built by default. You have to build it manaully


 * cd buildDebug/exampleTranslators/defaultTranslator
 * make install

ROSE_SRC/exampleTranslators/defaultTranslator/preprocessingInfoDumper.C

ROSE_INSTALL/bin/preprocessingInfoDumper

! cat test2020_comment_1.f90 subroutine suba implicit none write(13,fmt='( & &3x,"Damping work: ")') ! good comment write(13,fmt='( & &3x,"Friction work: ")') ! BAD 1 This comment comes one line later end subroutine suba

./preprocessingInfoDumper -c test2020_comment_1.f90

--- Found an IR node with preprocessing Info attached: (memory address: 0x2ce0700 Sage type: SgWriteStatement) in file

test2020_comment_1.f90 (line 5 column 4) -PreprocessingInfo #0 --- : classification = FortranStyleComment: String format = ! good comment relative position is = before --- Found an IR node with preprocessing Info attached: (memory address: 0x2b3e640 Sage type: SgInitializedName) in file

test2020_comment_1.f90 (line 0 column 0) -PreprocessingInfo #0 --- : classification = FortranStyleComment: String format = ! BAD 1 This comment comes one line later relative position is = after

see also ROSEAttributesList::display from rose_attributes_list.C for dumping it

call stack
Two phases: collect them first, then attach them into matching nodes

main
 * frontend
 * frontend
 * SgProject
 * SgProject::parse
 * SgProject::parse
 * determineFileType
 * SgSourceFile::callFrontEnd
 * SgFile::callFrontEnd
 * attachPreprocessingInfo(sourceFile); attachPreprocessingInfo.C

class AttachPreprocessingInfoTreeTrav // attach_all_info.C/h
 * AttachAllPreprocessingInfoTreeTrav::evaluateSynthesizedAttribute
 * AttachAllPreprocessingInfoTreeTrav::evaluateInheritedAttribute

Fortran code main frontend frontend SgProject::SgProject SgProject::parse SgProject::parse SgFile::secondaryPassOverSourceFile src/frontend/SageIII/sageSupport.C:5049 attachPreprocessingInfo src/frontend/SageIII/attachPreprocessingInfo.C:636 AstTopDownBottomUpProcessing SgTreeTraversal AttachPreprocessingInfoTreeTrav::evaluateInheritedAttribute src/frontend/SageIII/attachPreprocessingInfoTraversal.C:750 AttachPreprocessingInfoTreeTrav::getListOfAttributes AttachPreprocessingInfoTreeTrav::buildCommentAndCppDirectiveList src/frontend/SageIII/attachPreprocessingInfoTraversal.C:514 ROSEAttributesList::collectPreprocessorDirectivesAndCommentsForAST src/frontend/SageIII/rose_attributes_list.C:1973 Calling attachPreprocessingInfo to attach them
 * C/C++ SgFile::callFrontEnd
 * Fortran: SgSourceFile::build_Fortran_AST

SgLocatedNode::addToAttachedPreprocessingInfo(PreprocessingInfo *prepInfoPtr, PreprocessingInfo::RelativePositionType locationInList )

The locationInList abuses the PreprocessingInfo::RelativePositionType, only before and after positions are meaningful here, which is used to specify the head or the rear of the info list.

PreprocessingInfo::before

gdb breakpoint:

> break PreprocessingInfo::PreprocessingInfo(PreprocessingInfo::DirectiveType, std::string const&, std::string const&, int, int, int, PreprocessingInfo::RelativePositionType)

algorithm
src/frontend/SageIII/attachPreprocessingInfoTraversal.C

The SgFile (always) constructor calls the function invokes a tree traversal in order to attach the preprocessor directives (i.e., the preprocessingInfo objects) to located nodes in the AST.
 * void attachPreprocessingInfo(SgFile *sageFilePtr); which in turn calls
 * getPreprocessorDirectives (see above) and then

For this purpose, a data member attachedPreprocessingInfoType* attachedPreprocessingInfoPtr; is available to the SgLocatedNode class. This is done in ROSETTA/src/node.C.

Furthermore, There are corresponding access functions: to the SgLocatedNode class. This is done in ROSETTA/Grammar/LocatedNode.code.
 * void addToAttachedPreprocessingInfo(preprocessingInfo *prepInfoPtr);
 * attachedPreprocessingInfoType* getAttachedPreprocessingInfo(void);

The tree traversal works as follows:
 * whenever it hits a located node
 * it checks if there is preprocessing info the line number of which is less or equal than the line number of the current located node (currently: of the current statement).
 * If this is the case, the corresponding preprocessing info is attached to the current located node (currently: before the current statement), unparse flag: "before". All this is done in the evaluateInheritedAttribute member function of the derived tree traversal class.
 * The evaluateSynthesizedAttribute member function deletes the list of preprocessingInfo objects as soon as the traversal returns to a SgFile object and attaches trailing preprocessing information to the last located node (currently to the last statement) that has been visited in the file (unparse flag: "after").

Node that the preprocessingInfo objects are always attached to AST nodes. By switching the USE_OLD_MECHANISM_OF_HANDLING_PREPROCESSING_INFO flag, you only change the mechanism which the unparser is based on! If USE_OLD_MECHANISM_OF_HANDLING_PREPROCESSING_INFO is set to 1, then the unparser simply ignores the preprocessingInfo objects that have been attached to the AST nodes.

Problems with the handling of preprocessing information can be found in the directory ROSE/TESTS/KnownBugs/AttachPreprocessingInfo.

attaching function
// frontend/SageIII/attachPreprocessingInfoTraversal.C

void AttachPreprocessingInfoTreeTrav::iterateOverListAndInsertPreviouslyUninsertedElementsAppearingBeforeLineNumber ( SgLocatedNode* locatedNode, int lineNumber, PreprocessingInfo::RelativePositionType location, bool reset_start_index, ROSEAttributesList *currentListOfAttributes)

different call sites of iterateOverListAndInsertPreviouslyUninsertedElementsAppearingBeforeLineNumber

how to decide on different locations
 * before: AttachPreprocessingInfoTreeTrav::evaluateInheritedAttribute ( SgNode *n,AttachPreprocessingInfoTreeTraversalInheritedAttrribute inheritedAttribute)
 * attach to the current statement's before position if traversal is in evaluateInheritedAttribute
 * calling iterateOverListAndInsertPreviouslyUninsertedElementsAppearingBeforeLineNumber(currentLocNodePtr,line,PreprocessingInfo::before, reset_start_index, currentListOfAttributes );
 * inside: AttachPreprocessingInfoTreeTrav::evaluateSynthesizedAttribute( SgNode *n, AttachPreprocessingInfoTreeTraversalInheritedAttrribute inheritedAttribute, SubTreeSynthesizedAttributes synthiziedAttributeList)
 * attach to the inside position of a multiline located node when leaving the node during evaluateSynthesizedAttribute
 * after: AttachPreprocessingInfoTreeTrav::evaluateSynthesizedAttribute( SgNode *n, AttachPreprocessingInfoTreeTraversalInheritedAttrribute inheritedAttribute, SubTreeSynthesizedAttributes synthiziedAttributeList)
 * e.g. attach to the previous located node 's after position if the traversal is leaving V_SgSourceFile during evaluateSynthesizedAttribute

comments
Not all Located nodes can have comments!!

arbitrary text
More versatile than comments:

deletion
Cannot remove elements while iterating the container. Save a subset of elements into another container and remove them later on

Move comments to new statements
Source comments and #if directives should be preserved when we remove or replace statements in the AST

A possible call stack is
 * SageInterface::removeStatement from src/frontend/SageIII/sageInterface/sageInterface.C:10086
 * SageInterface::moveCommentsToNewStatement from src/frontend/SageIII/sageInterface/sageInterface.C:10193

How to debug
Debug the insertion action: SgLocatedNode::addToAttachedPreprocessingInfo in buildTree 's Cxx_Grammar.C

b SgLocatedNode::addToAttachedPreprocessingInfo(PreprocessingInfo*, PreprocessingInfo::RelativePositionType)

C/C++ example debug output
b Cxx_Grammar.C:54114

Breakpoint 1, SgLocatedNode::addToAttachedPreprocessingInfo (this=0x2aadaee62010, prepInfoPtr=0x2aadb00c0190, locationInList=after) at Cxx_Grammar.C:54114 54114       ROSE_ASSERT(prepInfoPtr != NULL); (gdb) bt   locatedNode=0x2aadaee62010, lineNumber=1000000000, location=after, reset_start_index=true, currentListOfAttributes=0x2aadb00bf840)    at ../../../../sourcetree/src/frontend/SageIII/attachPreprocessingInfoTraversal.C:313    synthiziedAttributeList=...) at ../../../../sourcetree/src/frontend/SageIII/attachPreprocessingInfoTraversal.C:1081 treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:1297   treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:879 treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:841
 * 1) 0 SgLocatedNode::addToAttachedPreprocessingInfo (this=0x2aadaee62010, prepInfoPtr=0x2aadb00c0190, locationInList=after) at Cxx_Grammar.C:54114
 * 2) 1 0x00002aaaabb6afc3 in AttachPreprocessingInfoTreeTrav::iterateOverListAndInsertPreviouslyUninsertedElementsAppearingBeforeLineNumber (this=0x7fffffffd820,
 * 1) 2 0x00002aaaabb6d8a9 in AttachPreprocessingInfoTreeTrav::evaluateSynthesizedAttribute (this=0x7fffffffd820, n=0x2aaab05eb010, inheritedAttribute=...,
 * 1) 3 0x00002aaaabb19e60 in SgTreeTraversal::performTraversal(SgNode *, AttachPreprocessingInfoTreeTraversalInheritedAttrribute, ) (this=0x7fffffffd820, node=0x2aaab05eb010, inheritedValue=...,
 * 1) 4 0x00002aaaabb13730 in SgTreeTraversal::traverse(SgNode *, AttachPreprocessingInfoTreeTraversalInheritedAttrribute, ) (this=0x7fffffffd820, node=0x2aaab05eb010, inheritedValue=...,
 * 1) 5 0x00002aaaabb1381e in SgTreeTraversal::traverseWithinFile(SgNode *, AttachPreprocessingInfoTreeTraversalInheritedAttrribute, ) (this=0x7fffffffd820, node=0x2aaab05eb010, inheritedValue=...,
 * 1) 6 0x00002aaaabb0cf65 in AstTopDownBottomUpProcessing::traverseWithinFile (this=0x7fffffffd820, node=0x2aaab05eb010, inheritedValue=...) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:659
 * 2) 7 0x00002aaaabb0268d in attachPreprocessingInfo (sageFilePtr=0x2aaab05eb010) at ../../../../sourcetree/src/frontend/SageIII/attachPreprocessingInfo.C:636
 * 3) 8 0x00002aaaabb979f5 in SgFile::secondaryPassOverSourceFile (this=0x2aaab05eb010) at ../../../../sourcetree/src/frontend/SageIII/sageSupport.C:5049
 * 4) 9 0x00002aaaabb95360 in SgProject::parse (this=0x2aaab057d010) at ../../../../sourcetree/src/frontend/SageIII/sageSupport.C:4104
 * 5) 10 0x00002aaaabb94668 in SgProject::parse (this=0x2aaab057d010, argv=...) at ../../../../sourcetree/src/frontend/SageIII/sageSupport.C:3828
 * 6) 11 0x00002aaaabc2704a in SgProject::SgProject (this=0x2aaab057d010, argv=...) at Cxx_Grammar.C:21076
 * 7) 12 0x00002aaaad5af1d4 in frontend (argv=...) at ../../../sourcetree/src/roseSupport/utility_functions.C:138
 * 8) 13 0x00002aaaad5af070 in frontend (argc=2, argv=0x7fffffffdde8) at ../../../sourcetree/src/roseSupport/utility_functions.C:118
 * 9) 14 0x0000000000403e4d in main (argc=2, argv=0x7fffffffdde8) at ../../../sourcetree/exampleTranslators/defaultTranslator/preprocessingInfoDumper.C:54

Fortran debugging output
Breakpoint 1, SgLocatedNode::addToAttachedPreprocessingInfo (this=0x20b0f50, prepInfoPtr=0xb0ca50, locationInList=PreprocessingInfo::after) at Cxx_Grammar.C:89431 (gdb) info breakpoints Num    Type           Disp Enb Address            What 1      breakpoint     keep y   0x00007ffff458a702 in SgLocatedNode::addToAttachedPreprocessingInfo(PreprocessingInfo*, PreprocessingInfo::RelativePositionType) at Cxx_Grammar.C:89431 breakpoint already hit 1 time (gdb) p this->class_name $1 = "SgWriteStatement" (gdb) bt gInfo::before, reset_start_index=false, currentListOfAttributes=0xb02bc0) at ../../../../sourcetree/src/frontend/SageIII/attachPreprocessingInfoTraversal.C:539 aversal.C:2837 ritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:774 ritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:828 ritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:828 inheritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:828 inheritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:828 inheritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:828 edValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:742 0, inheritedValue=..., treeTraversalOrder=preandpostorder) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:702 0x7fffeb8de010, inheritedValue=...) at ../../../../sourcetree/src/midend/astProcessing/AstProcessing.h:502
 * 1) 0 SgLocatedNode::addToAttachedPreprocessingInfo (this=0x20b0f50, prepInfoPtr=0xb0ca50, locationInList=PreprocessingInfo::after) at Cxx_Grammar.C:89431
 * 2) 1 0x00007ffff440b302 in AttachPreprocessingInfoTreeTrav::iterateOverListAndInsertPreviouslyUninsertedElementsAppearingBeforeLineNumber (this=0x7fffffff9840, locatedNode=0x20b0f50, lineNumber=5, location=Preprocessin
 * 1) 2 0x00007ffff440e3b6 in AttachPreprocessingInfoTreeTrav::evaluateInheritedAttribute (this=0x7fffffff9840, n=0x20b0f50, inheritedAttribute=...) at ../../../../sourcetree/src/frontend/SageIII/attachPreprocessingInfoTr
 * 1) 3 0x00007ffff43c4fcf in SgTreeTraversal::performTraversal (this=0x7fffffff9840, node=0x20b0f50, inhe
 * 1) 4 0x00007ffff43c5168 in SgTreeTraversal::performTraversal (this=0x7fffffff9840, node=0x1c53ae0, inhe
 * 1) 5 0x00007ffff43c5168 in SgTreeTraversal::performTraversal (this=0x7fffffff9840, node=0x1cf7bf0, inhe
 * 1) 6 0x00007ffff43c5168 in SgTreeTraversal::performTraversal (this=0x7fffffff9840, node=0x7fff39b31010,
 * 1) 7 0x00007ffff43c5168 in SgTreeTraversal::performTraversal (this=0x7fffffff9840, node=0x7ffff7ee1120,
 * 1) 8 0x00007ffff43c5168 in SgTreeTraversal::performTraversal (this=0x7fffffff9840, node=0x7fffeb8de010,
 * 1) 9 0x00007ffff43bfb13 in SgTreeTraversal::traverse (this=0x7fffffff9840, node=0x7fffeb8de010, inherit
 * 1) 10 0x00007ffff43bfcb3 in SgTreeTraversal::traverseWithinFile (this=0x7fffffff9840, node=0x7fffeb8de01
 * 1) 11 0x00007ffff43bbd0f in AstTopDownBottomUpProcessing::traverseWithinFile (this=0x7fffffff9840, node=
 * 1) 12 0x00007ffff43b3740 in attachPreprocessingInfo (sageFilePtr=0x7fffeb8de010) at ../../../../sourcetree/src/frontend/SageIII/attachPreprocessingInfo.C:797
 * 2) 13 0x00007ffff444a14c in SgFile::secondaryPassOverSourceFile (this=0x7fffeb8de010) at ../../../../sourcetree/src/frontend/SageIII/sage_support/sage_support.cpp:3597
 * 3) 14 0x00007ffff4447a5d in SgProject::parse (this=0x7fffeba2b010) at ../../../../sourcetree/src/frontend/SageIII/sage_support/sage_support.cpp:2593
 * 4) 15 0x00007ffff444673a in SgProject::parse (this=0x7fffeba2b010, argv=std::vector of length 9, capacity 9 = {...}) at ../../../../sourcetree/src/frontend/SageIII/sage_support/sage_support.cpp:1965
 * 5) 16 0x00007ffff4525653 in SgProject::SgProject (this=0x7fffeba2b010, argv=std::vector of length 9, capacity 9 = {...}, frontendConstantFolding=false) at Cxx_Grammar.C:29216
 * 6) 17 0x00007ffff674644d in frontend (argv=std::vector of length 9, capacity 9 = {...}, frontendConstantFolding=false) at ../../../sourcetree/src/roseSupport/utility_functions.C:482
 * 7) 18 0x00007ffff6746300 in frontend (argc=9, argv=0x7fffffffa738, frontendConstantFolding=false) at ../../../sourcetree/src/roseSupport/utility_functions.C:444
 * 8) 19 0x000000000040a820 in main (argc=9, argv=0x7fffffffa738) at ../../../../sourcetree/tests/nonsmoke/functional/testTranslator.C:59