I recently had to enable the normal unicast Coherence cluster required by the Oracle SOA suite on a set of Windows servers which was completely separated by local firewalls. It took a good bit of trial and error to get it to work, so I thought I would sum up what I learned during the endevaour.
[oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster......
[2014-10-08T13:22:14.221+02:00] [soa_server02] [WARNING] [] [Coherence] [tid: Logger@1932191207 3.7.1.1] [userId: <anonymous>] [ecid: 0000KZjU_CZEKQG5Mzt1ie1KDHoj000002,0] [APP: soa-infra] 2014-10-08 13:22:14.221/375.070 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMontor failed to verify the reachability of senior Member(Id=1, Timestamp=2014-10-08 13:18:54.344, Address=10.0.0.1:8088, MachineId=3060, Location=site:,machine:exampleerver1,process:4864, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7
<08-10-2014 15:48:30 CEST> <Error> <Coherence> <BEA-000000> <2014-10-08 15:48:30.468/517.608 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}}>
<08-10-2014 15:48:30 CEST> <Error> <Coherence> <BEA-000000> <2014-10-08 15:48:30.499/517.639 Oracle Coherence GE 3.7.1.1 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump
<08-10-2014 15:48:56 CEST> <Error> <oracle.integration.platform.blocks.deploy.coordinator> <BEA-000000> <Cluster communication initialization failed. If you are using multicast, please make sure multicast is enabled on your network and that there is no interference on the address in use. Please see the documentation for more details.
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(.......
It took a bit of research to figure all of this out, so I hope it helps others having problems with Coherence clustering.
Error Messages
Initially we saw a lot of deployment and Coherence errors with timeouts and connection issues like these:[oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster......
[2014-10-08T13:22:14.221+02:00] [soa_server02] [WARNING] [] [Coherence] [tid: Logger@1932191207 3.7.1.1] [userId: <anonymous>] [ecid: 0000KZjU_CZEKQG5Mzt1ie1KDHoj000002,0] [APP: soa-infra] 2014-10-08 13:22:14.221/375.070 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMontor failed to verify the reachability of senior Member(Id=1, Timestamp=2014-10-08 13:18:54.344, Address=10.0.0.1:8088, MachineId=3060, Location=site:,machine:exampleerver1,process:4864, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7
<08-10-2014 15:48:30 CEST> <Error> <Coherence> <BEA-000000> <2014-10-08 15:48:30.468/517.608 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}}>
<08-10-2014 15:48:30 CEST> <Error> <Coherence> <BEA-000000> <2014-10-08 15:48:30.499/517.639 Oracle Coherence GE 3.7.1.1 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump
<08-10-2014 15:48:56 CEST> <Error> <oracle.integration.platform.blocks.deploy.coordinator> <BEA-000000> <Cluster communication initialization failed. If you are using multicast, please make sure multicast is enabled on your network and that there is no interference on the address in use. Please see the documentation for more details.
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(.......
Needed Ports
During the normal configuration of Coherence you define, among other things, what port it should run on. What isn't apparent is that Coherence actually uses two ports. The one you configure + The one above. So if you configure it to use 8088, it also needs 8089. Ensure that both ports are open on TCP and UDP both. Besides that, the firewall needs to allow communication on port 7 (Echo) and to allow ICMP ping packages. This is used to keep the Coherence cluster alive.Echo Service
Even though the Coherence documentation states that port 7 just needs to be open. I actually had to have a proper echo service respond. In windows this is done by adding the feature "Simple TCP/IP Services".It took a bit of research to figure all of this out, so I hope it helps others having problems with Coherence clustering.