Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1297

datanode sending block reports to namenode once every second

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.13.0
    • None
    • None

    Description

      The namenode is requesting a block to be deleted. The datanode tries this operation and encounters an error because the block is not in the blockMap. The processCommand() method raises an exception. The code is such that the variable lastBlockReport is not set if processCommand() raises an exception. This means that the datanode immediately send another block report to the namenode. The eats up quite a bit of CPU on namenode.

      In short, the above condition causes the datanode to send blockReports almost once every second!

      I propose that we do the following:

      1. in Datanode.offerService, replace the following piece of code

      DatanodeCommand cmd = namenode.blockReport(dnRegistration,
      data.getBlockReport());
      processCommand(cmd);
      lastBlockReport = now;

      with

      DatanodeCommand cmd = namenode.blockReport(dnRegistration,
      data.getBlockReport());
      lastBlockReport = now;
      processCommand(cmd);

      2. In FSDataSet.invalidate:
      a) continue to process all blocks in invalidBlks[] even if one in the middle encounters a problem.
      b) if getFile() returns null, still invoke volumeMap.get() and print whether we found the block in
      volumes or not. The volumeMap is used to generate the blockReport and this might help in debugging.
      [

      Attachments

        1. datanodeDeleteBlocks-0.12.3.patch
          4 kB
          Dhruba Borthakur
        2. datanodeDeleteBlocks2.patch
          4 kB
          Dhruba Borthakur

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dhruba Dhruba Borthakur
            dhruba Dhruba Borthakur
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment