using dtrace

i tracked what was doing it with DTRACE and it's this process that is (ultimately) firing off the iscsiadm commands that are failing


horatio:/ # ps -wwax|fgrep 10371
10371 ? R 155:11 java -Xmx150m -Xss192k -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -client -Dxvmserver=true -Dfile.encoding=utf-8
-Djava.endorsed.dirs=/usr/lib/cacao/lib/endorsed -classpath
/usr/share/lib/jdmk/jdmkrt.jar:/usr/share/lib/jdmk/jmxremote_optional.jar:/usr/lib/cacao/lib/cacao_cacao.jar:/usr/lib/cacao/lib/cacao_j5core.jar:/usr/li
b/cacao/lib/bcprov-jdk14.jar -Djavax.management.builder.initial=com.sun.jdmk.JdmkMBeanServerBuilder -Dcacao.print.status=true
-Dcacao.config.dir=/etc/cacao/instances/scn-agent -Dcacao.monitoring.mode=smf
-Dcom.sun.cacao.ssl.keystore.password.file=/etc/cacao/instances/scn-agent/security/password com.sun.cacao.container.impl.ContainerPrivate

i ran this dtrace script to log the executable binary's name and PID and PPID of every process that gets run


dtrace -qn 'syscall::exec*:return { printf("%Y (pid=%d) (ppid=%d) %s\n",walltimestamp,pid,ppid,curpsinfo->pr_psargs); }' > OUT


and piped the output to a file and then did the rigmorale of reverse tracing it until i found the real process that was still running


horatio:/home/sysmwd 2230 # fgrep " iscsiadm " OUT
2013 Sep 30 14:38:50 (pid=2079) (ppid=2078) iscsiadm list initiator-node
2013 Sep 30 14:41:50 (pid=3059) (ppid=3058) iscsiadm list initiator-node


and eventually if you follow the PPID chain, it turns out to be that cacao java thing