Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

Quick Launch

Average Rating:

facebook Twitter
Email
Print Bookmark Alert me when this article is updated

Feedback

ERROR: "HBase Table 'XYZ' was not found, got: ABCDEF" when  Axon scanner fails in a Kerberos enabled Catalog Service
Problem Description
Axon scanner fails with the following exception in a Kerberos enabled Catalog Service.

           Table 'hbase:acl' was not found, got: SYSTEM:STATS.
Cause
This happens due to the incomplete table creation in HBase (HBase bug). Read the More Information section to know how we isolated and fixed this.
Solution
​Get help from Informatica Global Customer support to fix this.
More Information
The following stack trace is from the HBase master server (master passes the exception stack trace to the client when an RPC fails). In HBase, the master is the server that manages the metadata of the tables, so we need to understand why the scanner was communicating with the master. 

Wed Apr 08 06:11:49 CDT 2020, RpcRetryingCaller{globalStartTime=1586343646566, pause=100, retries=40}, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: Table 'hbase:acl' was not found, got: SYSTEM:STATS.: 1 time, servers with issues: null,
                at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:212)
                at org.apache.hadoop.hbase.security.access.AccessController.postModifyTable(AccessController.java:1129)
                at org.apache.hadoop.hbase.master.MasterCoprocessorHost$24.call(MasterCoprocessorHost.java:352)
                at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1192)
                at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postModifyTable(MasterCoprocessorHost.java:348)
                at org.apache.hadoop.hbase.master.HMaster$11.run(HMaster.java:2222)
                at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
                at org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2209)
                at org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1188)
                at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58547)
                at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
                at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
                at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
                at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
            Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: Table 'hbase:acl' was not found, got: SYSTEM:STATS.: 1 time, servers with issues: null,
                at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:260)
                at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2400(AsyncProcess.java:240)
                at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1889)
                at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:248)
                at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:197)
                at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1461)
                at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1017)
                at org.apache.hadoop.hbase.client.HTableWrapper.put(HTableWrapper.java:149)
                at org.apache.hadoop.hbase.security.access.AccessControlLists.addUserPermission(AccessControlLists.java:174)
                at org.apache.hadoop.hbase.security.access.AccessController$5.run(AccessController.java:1134)
                at org.apache.hadoop.hbase.security.access.AccessController$5.run(AccessController.java:1129)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:422)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
                at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
                at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
                at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
                at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:210)
                ... 13 more

From the scanner logs, it was clear that it was trying to modify a table (green frame in the stack trace) scanner log also tells the same

Scanner log snippet

2020-04-08 06:11:49,741 [ExecuteThread-{Task_axon_AxonScanner_CORE}] ERROR com.infa.products.ldm.framework.scanner.execution.impl.ScannerTask- {RunId=bbc0efbb-3416-4dfe-b468-fb10654f77b4}{Resource=axon}{ScannerId=AxonScanner} Error occured in scanner init Error creating bean with name 'metadataSimilarityDAO' defined in com.infa.products.ldm.beans.simprof.similarity.SimilarityDAOConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.infa.products.ldm.simprof.similarity.persistence.api.MetadataSimilarityDAO]: Factory method 'metadataSimilarityDAO' threw exception; nested exception is org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [ALTER TABLE "DPRT_N" ADD IF NOT EXISTS F1.OTP VARCHAR]; Failed after attempts=40, exceptions:


Ok, we are altering simns:DPRT_N, however, why hbase:acl is being accessed here?

When we enable Kerberos for the HBase, the access control gets enabled and this HBase table (hbase:acl) is created to store the ACLs and every time a DDL query is invoked the access control coprocessor running in master intercepts this DDL RPC and does some action needed for ACL and it depends on this table.

Alright, why the table is not available, and what is this error message?

   Table 'hbase:acl' was not found, got: SYSTEM:STATS.

Answer lies in the way a table is accessed..

1. Client finds out the location of the hbase:meta table (META table) from the Zookeeper.

          a. META table contains info about which user table region is hosted in which Region Server. The table is sorted on the key and structure is as follows

         
Key (Region name)​ 1 (column family)​ ​ ​ ​

regioninfo​
seqNumDuringOpen​ server
serverstartcode​
​.... ​... ​... ​... ​...
SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008. {ENCODED => ed12b64bce9bcfd49105cb5fd9808008, NAME => 'SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008.', STARTKEY => '', ENDKEY => ''} \x00\x00\x00\x00\x00\x00\x02X abc.com:14485 1586395811360
hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a. {ENCODED => 5301422f2f8a237be8237a5e2400698a, NAME => 'hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a.', STARTKEY => '', ENDKEY => ''} \x00\x00\x00\x00\x00\x00\x00\x02 abc.com:14485 1586395811360
​.... ​... ​... ​... ​...
 
2. It figures out the address of the RegionServer where the interested region exists (region of hbase:meta table containing the info about the hbase:acl, In this case region containing row 2)
3. Client connects to the Region Server where the region of the META table is hosted. (in our case it's abc.com:14485)
4. Invokes a reverse scan with start row = hbase:aclXXXXXXXX
5. If the row for this region doesn't exist in META, the entry for the previous region is returned (remember it's a reverse scan).
6. In EDC hbase lexicographically SYSTEM:STATS comes before hbase:acl so the entry for this table (SYSTEM:STATS) is returned (if hbase:acl is not in META).
7. Client sees that it was searching hbase:acl but found SYSTEM:STATS and throws the above exception (The message is fixed in HBASE-18022)

So why there is no record for the hbase:acl in the META. From the master logs we can see that, master couldn't complete the table creation for some unknown reason. We should collect heapdump for investigating this.

Master log snippets

      2020-04-05 21:35:56,004 INFO  [abc.com:32597.activeMasterManager] master.HMaster: Client=null/null create 'hbase:acl', {NAME => 'l', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}
Once the table is created the Master logs a message similar to (which was missing in the HMaster logs)
      2020-04-09 14:23:26,008 INFO  [RegionOpenAndInitThread-hbase:acl-1] regionserver.HRegion: creating HRegion hbase:acl HTD == 'hbase:acl', {NAME => 'l', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://abc.com:8020/Informatica/LDM/infa1040/hbase/.tmp Table name == hbase:acl


Solution

1. We had to create the hbase:acl table
2. The DDL in hbase shell failed to create the table this was because HMaster could see a entry for hbase:acl in ZK/Informatica/LDM/informatica/hbase/table
3. Deleting this znode and recreating the table with the DDL (got it from the HMaster logs) went successful


Listing of hbase:meta
         > scan "hbase:meta"
         .....
         SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008.    column=info:regioninfo, timestamp=1586395829734, value={ENCODED => ed12b64bce9bcfd49105cb5fd9808008, NAME => 'SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008.', STARTKEY => '', ENDKEY => ''}
         SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008.    column=info:seqnumDuringOpen, timestamp=1586395829734, value=\x00\x00\x00\x00\x00\x00\x02X
         SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008.    column=info:server, timestamp=1586395829734, value=abc.com:14485
         SYSTEM:STATS,,1585917212164.ed12b64bce9bcfd49105cb5fd9808008.    column=info:serverstartcode, timestamp=1586395829734, value=1586395811360
         hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a.        column=info:regioninfo, timestamp=1586438875567, value={ENCODED => 5301422f2f8a237be8237a5e2400698a, NAME => 'hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a.', STARTKEY => '', ENDKEY => ''}
         hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a.        column=info:seqnumDuringOpen, timestamp=1586438875567, value=\x00\x00\x00\x00\x00\x00\x00\x02
         hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a.        column=info:server, timestamp=1586438875567, value=abc.com:14485
         hbase:acl,,1586438875015.5301422f2f8a237be8237a5e2400698a.        column=info:serverstartcode, timestamp=1586438875567, value=1586395811360

         ....
Applies To
Product: Enterprise Data Catalog
Problem Type: Crash/Hang
User Type: Administrator
Project Phase: Implement; Onboard; Optimize; Configure
Product Version:
Database:
Operating System:
Other Software:

Reference
Attachments
Last Modified Date:7/30/2020 3:08 AMID:621148
People who viewed this also viewed

Feedback

Did this KB document help you?



What can we do to improve this information (2000 or fewer characters)