Working with AST matcher without ClangTool compilation database

When building AST Matchers for Clang, many beginners face problems running the documentation example described at LibASTMatchers Tutorial encountering errors such as:

LLVM ERROR: Could not auto-detect compilation database

and

json-compilation-database: Error while opening JSON database: The system cannot find the file specified

The tutorial described in the documentation is very straight forward and simple. It requires creating a ClangTool variable and running it with the AST Matcher, in just few lines of code as shown below:

int main(int argc, const char **argv) {
  CommonOptionsParser OptionsParser(argc, argv, MyToolCategory);
  ClangTool Tool(OptionsParser.getCompilations(),
                 OptionsParser.getSourcePathList());

  LoopPrinter Printer;
  MatchFinder Finder;
  Finder.addMatcher(LoopMatcher, &Printer);

  return Tool.run(newFrontendActionFactory(&Finder).get());
}

However, when built and run, the code crashes with asserts right on the first two lines, complaining about the absence of a file named compile_commands.json.

As indicated at the Compilation Database documentation the ClangTool requires a JSON compilation database to work properly and could be easily generated with the CMAKE_EXPORT_COMPILE_COMMANDS option using CMake on Unix/Linux systems. However, with Visual Studio many have complained that the said option does not work.

Now, you can take time to understand the structure of the said compilation database and create one yourself, put it in the required directory and run the examples. Or you could find alternate ways of telling the Clang compiler what it wants to know about the compilation properties without having to maintain such build database JSON files on the disk. For those who would like to take the second approach of directly configuring the Clang compiler instance at runtime dynamically without having to maintain a compilation database file on the disk, one good way is to avoid using the ClangTool altogether and work directly with Clang CompilerInstance (which ClangTool uses anyway, behind the scenes).

In this method, you would create a clang CompilerInstance object using the regular method and issue the ParseAST as usual, except that you want to use the MatchFinder‘s specialized AST consumer to take care of visiting all the AST matches. But how do you get an ASTConsumer from a MatchFinder? Thats pretty straight-forward:

std::unique_ptr pAstConsumer (matchFinder.newASTConsumer());

clang::ast_matchers::MatchFinder has a newASTConsumer() method that creates the appropriate consumer object that knows how to visit all the required matches you have specified. Complete code looks like below (add appropriate headers and link against the right libraries).

////////////////////////////////////////////////////////
/// (c) 2014. My-Classes.com
/// Dealing with AST matcher without ClangTool compilation database.
///
/// Read More at:
/// http://my-classes.com/2014/07/01/working-with-ast-matcher-without-clangtool-compilation-database/
///
///////////////////////////////////////////////////////
 
class ClangParser
{
  clang::CompilerInstance m_CompilerInstance;
 
  void prepareCompilerforFile(const char* szSourceCodeFilePath)
  {
    // To reuse the source manager we have to create these tables.
    m_CompilerInstance.getSourceManager().clearIDTables();
 
    // supply the file to the source manager and set it as main-file
    const clang::FileEntry * file = m_CompilerInstance.getFileManager().getFile(szSourceCodeFilePath);
    clang::FileID fileID = m_CompilerInstance.getSourceManager().createFileID(file, clang::SourceLocation(), clang::SrcMgr::CharacteristicKind::C_User);
    m_CompilerInstance.getSourceManager().setMainFileID(fileID);
 
    // CodeGenAction needs this
    clang::FrontendOptions& feOptions = m_CompilerInstance.getFrontendOpts();
    feOptions.Inputs.clear();
    feOptions.Inputs.push_back(clang::FrontendInputFile(szSourceCodeFilePath, 
            clang::FrontendOptions::getInputKindForExtension(clang::StringRef(szSourceCodeFilePath).rsplit('.').second), false));
  }
public:
  ClangParser()
  {
    // Usually all examples try to build the CompilerInstance Object from CompilerInvocation object.
    // However, CompilterInstance already has an in-built CompilerInvocation object. So we use it.
    // Only problem is: target options are not set correctly for the in-built object. We do it manually.
    // This below line is just to assert that the object exist.
    clang::CompilerInvocation& invocation = m_CompilerInstance.getInvocation();
 
    // Diagnostics is needed - set it up
    m_CompilerInstance.createDiagnostics();
 
    // set the include directories path - you can configure these at run-time from command-line options
    clang::HeaderSearchOptions & headerSearchOptions = m_CompilerInstance.getHeaderSearchOpts();
    headerSearchOptions.AddPath("C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\include", clang::frontend::IncludeDirGroup::Angled, false, false);
    // headerSearchOptions.Verbose = true;
 
    // set few options
    clang::LangOptions& langOptions = m_CompilerInstance.getLangOpts();
    langOptions.CPlusPlus = 1;
    langOptions.Bool = 1;
    langOptions.RTTI = 0;
#if defined(_MSC_VER)
    langOptions.MicrosoftExt = 1;
    langOptions.MSVCCompat = 1;
    langOptions.MSBitfields = 1;
    langOptions.DelayedTemplateParsing = 1; // MSVC parses templates at the time of actual use
    m_CompilerInstance.getDiagnosticOpts().setFormat(clang::TextDiagnosticFormat::Msvc);
    //clInstance.getTargetOpts().ABI = "microsoft";
#endif
 
    // Need to set the source manager before AST
    m_CompilerInstance.createFileManager();
    m_CompilerInstance.createSourceManager(m_CompilerInstance.getFileManager());
 
    // Need to set the target before AST. Adjust the default target options and create a target
    m_CompilerInstance.getTargetOpts().Triple = llvm::sys::getProcessTriple();
    m_CompilerInstance.setTarget(clang::TargetInfo::CreateTargetInfo(m_CompilerInstance.getDiagnostics(), &m_CompilerInstance.getTargetOpts()));
 
    // Create pre-processor and AST Context
    m_CompilerInstance.createPreprocessor(clang::TranslationUnitKind::TU_Module);
    m_CompilerInstance.createASTContext();
    if (m_CompilerInstance.hasPreprocessor())
    {
      clang::Preprocessor & preprocessor = m_CompilerInstance.getPreprocessor();
      preprocessor.getBuiltinInfo().InitializeBuiltins(preprocessor.getIdentifierTable(), preprocessor.getLangOpts());
    }
  }
 
  bool parseAST(const char* szSourceCodeFilePath, clang::ast_matchers::MatchFinder finder)
  {
    // create the compiler instance setup for this file as main file
    prepareCompilerforFile(szSourceCodeFilePath);
 
    std::unique_ptr pAstConsumer(finder.newASTConsumer());
 
    clang::DiagnosticConsumer& diagConsumer = m_CompilerInstance.getDiagnosticClient();
    diagConsumer.BeginSourceFile(m_CompilerInstance.getLangOpts(), &m_CompilerInstance.getPreprocessor());
    clang::ParseAST(m_CompilerInstance.getPreprocessor(), pAstConsumer.get(), m_CompilerInstance.getASTContext());
    diagConsumer.EndSourceFile();
 
    return diagConsumer.getNumErrors() != 0;
  }
 
  // Executes CodeGenAction and returns the pointer (caller should own and delete it)
  // Returns NULL on failure (in case of any compiler errors etc.)
  clang::CodeGenAction* emitLLVM(const char* szSourceCodeFilePath)
  {
    // create the compiler instance setup for this file as the main file
    prepareCompilerforFile(szSourceCodeFilePath);
 
    // Create an action and make the compiler instance carry it out
    clang::CodeGenAction *Act = new clang::EmitLLVMOnlyAction();
    if (!m_CompilerInstance.ExecuteAction(*Act))
      return NULL;
 
    return Act;
  }
 
  clang::ASTContext& GetASTContext() { return m_CompilerInstance.getASTContext(); }
};
 
class RecordDeclPrinter : public MatchFinder::MatchCallback
{
public:
  virtual void run(const MatchFinder::MatchResult &Result);
};
 
// walk on all struct/class/unions
DeclarationMatcher recordDeclMatcher = recordDecl(has(fieldDecl())).bind("recordDecl");
 
void RecordDeclPrinter::run(const MatchFinder::MatchResult &Result)
{
  const clang::RecordDecl *st = Result.Nodes.getNodeAs("recordDecl");
 
  std::ofstream out(stdout);
  out << st->getName();
 
  for (clang::RecordDecl::field_iterator it = st->field_begin(); it != st->field_end(); it++)
  {
    if (!(*it)) continue;
 
    if ((*it)->getType()->isBuiltinType())
      out << getType().getAsString());
    else if ((*it)->getType()->isPointerType())
      out << std::string("pointer");
    else
      out << std::string("nothing");
  }
}
 
int main(int argc, const char **argv)
{
  clang::ast_matchers::MatchFinder Finder;
 
  RecordDeclPrinter allRecords;
  Finder.addMatcher(recordDeclMatcher, &allRecords);
 
  ClangParser parser;
  parser.parseAST("Input_File.h", Finder, clang::tooling::newFrontendActionFactory(&Finder).get());
  return 0;
}

For more AST matcher examples, you may find the project Coding Assistance with Clang useful. The struct finder AST matcher shown in the above code snippet was taken from the MatchStruct.cpp in that particular project.

Interested in learning more? Join our Advanced C++ / Clang classes or Write to us.

Analytics Training Videos

Comments