EM13c job failed with ORA-20233: Unable to determine OS for :. Error: ORA-01403: no data found

I recently came across this error in our EM13c environment after pushing a couple of ODA hardware plugin to the monitoring agents.

Even though configuration for the Plugin looks okay to my knowledge

After this plugin was push, all the EM13 jobs (backup, Enterprise -> Job) are all failing with following error messages

 ORA-20233: Unable to determine OS for <Target Name>:<Target Type>. Error: ORA-01403: no data found 
 ORA-06512: at "SYSMAN.EM_JOB_UTIL", line 2149 
 ORA-06512: at "SYSMAN.EM_JOB_UTIL", line 1505 
 ORA-06512: at "SYSMAN.EM_JOB_UTIL", line 1563 
 ORA-06512: at line 1  

I found the culprit under /u01/app/oms132/gc_inst/em/EMGC_OMS1/sysman/log/emoms_pbs.trc after a few hours of head scratching sessions.

2019-06-18 10:51:46,630 [Job Step 583201 Job Long-System Pool:JobWorker] ERROR target.EntityUtil logp.251 - PromoteTarget for target: zsc-odaem.zeusinc.com:host on agent: https://zsc-odaem.zeusinc.com:3872/emd/main/ got transient InvalidTargetException : The zsc-odaem.zeusinc.com target "host" has an invalid attribute.
 oracle.sysman.emSDK.agent.client.exception.InvalidTargetException: The zsc-odaem.zeusinc.com target "host" has an invalid attribute.
...
Caused by: oracle.sysman.gcagent.cxl.ProxyException: EMA-00003: A new definition of an existing target host.zsc-odaem.zeusinc.com is saved to the agent with a different IDENTIFIER

Then i came across this table/view that has some data abnormality

select * from sysman.EM_MANAGEABLE_ENTITIES where ENTITY_TYPE = ‘host’;

Data abnormality in sysman own table

HOST_NAME columns for the host that has “no data found” error
zsc-oda0-0.NAME:
zsc-odaem.NAME:
zsc-oda1-1.NAME:

HOST_NAME columns for the host that doesn’t have the above issue
zsc-oradev-01.FQDN
…
etc

Take a backup of sysman.EM_MANAGEABLE_ENTITIES_E table

Change the invalid attribute “HOST_NAME” from “.NAME:” to “FQDN”

Rerun the EM database backup jobs or EM jobs, without any issues.