If you learned something
today, please

SSH NodeManager Headache Solved

March 26th, 2015
No Gravatar

For the longest time I could not get the ssh NodeManager to work on my VirtualBox VM I use for proving out approaches. It was not a big problem to use the command line start, so I did, until recently I needed to precisely replicate the start up process used in production.

After much thumping of forehead on desk (and steering wheel, and pillow, and…) I found this nugget at http://www.pythian.com/blog/why-was-sshd-refusing-my-key/:

…In our environment, we are not using SELinux so we disabled it by setting SELINUX=disabled
in /etc/selinux/config and rebooted the system….


Customizing WebCenter Logging with WLST

January 17th, 2015
No Gravatar

Like many enterprises, the one I’m currently working with only provides developers with the Monitor role in production. This requires scripting tasks that are well-documented for manually performing but take some real thinking and testing to get right as a script.

One of my team members frequently says that code should be self-documenting, so rather a long blog entry I thought I would simply post the script along with some minimal comments and see if everyone finds that useful:

def configureAdfLogHandlers():
#The full script has multiple methods, so I print the currently running method to follow along
	print 'Start configureAdfLogHandlers()'
	for managedServer in managedServers:
		path = '/Servers/' + sname
		lh = listLogHandlers(returnMap='true')
		for l in lh:
			lname = l.get('name')
#Even with this script, the odl logs are quite verbose and we run an automated arciving process. 
#To simplify the process, we just put the logs in the archive path since the odl log handlers don't provide an archive location option
			odlfile = '${domain.home}/logs/LOGS/${weblogic.Name}-' + lname + '.log'
			print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
			print '   Setting log handler configuration for ' + lname
			print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
#The custom code rarely uses any levels between debug and warning, and the notice level for Oracle code is too verbose for most trouble-shooting needs
		print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
		print '   Setting logger configuration for ' + sname
		print '%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%'
#The custom code used by the application sets of warnings in these packages that are not issues for the application, so we set the level to ERROR here



Speeding Up WebLogic Server Console First Access on Linux

October 20th, 2014
No Gravatar

It has been posted many times, but since I ran across it again and it made a significant improvement, I’ll repeat it for others.

Add the following to your WLS start up arguements:


If you are interested in details, see https://bugs.openjdk.java.net/browse/JDK-6202721

Fixing Silent Domain Builds with WebCenter Portal

October 12th, 2014
No Gravatar

Automating domain creation is important for enterprises to minimize maintenance cost and be prepared for efficient disaster recovery. One enterprise I worked with was struggling with the domains they were building through scripts as the Enterprise Manager was not working properly in the completed domains and the managed servers would come up in Admin mode.

After comparing the scripted domain to a manually created domain that worked correctly it was found that the paths for the libraries did not match. The paths in config.xml were missing the beginning of the path where the libraries were actually located. While the installation paths chosen were not the default paths, this did not entirely explain the issue, and internal processes made the custom paths highly desirable.

The correction for this issue was to add a function to the offline WLST script run on completion of the domain creation to correct the library paths:

def fixApplicationSourcePaths():
  appPathPrefix = os.getenv('DOMAIN_HOME')+'/applications/'
  deployed_apps = ls(returnMap='true')
  for app in deployed_apps:
    apppath = get('SourcePath')
    if apppath.find('/app/')!=0:
      apppath = apppath.split('/')
      apppath = apppath[-1]
  deployed_libs = ls(returnMap='true')
  for lib in deployed_libs:
    apppath = get('SourcePath')
    if apppath.find('/app/')!=0:

With this change, domains were created consistently as desired.

Correcting WLS Console Hang with SSH Node Manager Environment

October 4th, 2014
No Gravatar

I was having a frustrating situation yesterday where the WLS console would just hang after clicking the Servers link. Naturally I go to the Unix console to tail the logs and see what’s what, and these prompts for a password keep popping up. Entering the admin password is no help.

So I went to the next natural step of rebuilding the domain (the build is scripted, and yes, I backed up the domain first) and the problem is still there!

An email to a colleague more steeped in the dark arts of server administration than I results in the suggestion that this is due to a problem with SSH (the domain uses SSH Node Manager) and tells me test it with:

ssh username@hostname

to confirm connectivity and permissions.  What I get back is:




Permissions 0755 for '/app/home/username/.ssh/id_rsa' are too open.

It is required that your private key files are NOT accessible by others.

This private key will be ignored.

bad permissions: ignore key: /app/home/username/.ssh/id_rsa


With this new detail in mind, I went to an environment that was working, found it had the file permission set at 600, and preforming

chmod 600 /app/home/username/.ssh/id_rsa

Got me up and running again

For Those That Don’t Get Git

September 15th, 2014
No Gravatar

I do appreciate why developers like Git, with the low overhead and distributed repository.

As someone who is tasked with code reviewing in addition to writing code and managing builds and releases, I find it difficult to get behind Git because of the lack of tools for performing these tasks. Until I work on a project where it can be taken for granted that everyone wrote perfect code, I need something to help locate newly checked in files and allows for manual merging.

Until then, I have found Git Extensions is at least a good start. The diff and blame views make for easier code reviewing. It still lacks an easy way of locating updates by date for an entire repository, but it is a start to getting by with Git.

If anyone has found better tools for reviewing code in a Git repository, please post your comments!

The Art of Stubbing

August 5th, 2014
No Gravatar

(Originally published as Stubbing For Fun, Profit, and Survival )

Anyone reading or watching the news these days know that IT project budgets will be tightening further than you may already be experiencing. While proponents of some languages and platforms claim to provide lower development costs, many others firmly believe that it is the habits of the developers who prefer those environments (more than the environment itself) that leads to rapid results. One of these habits is the proper use of method stubs. By proper, let’s define that as stubs that facilitate the development process first, and checking off “done” on a project plan second.

Stubbing Defined

Half the time someone asks me if I am the person who wrote a particular article it is a manager or analyst rather than a developer. Since the next person who asks you what you mean by “stubbing” may be using this article as their only reference, I thought I would add something that makes it easy to clarify this technique, and some fodder that will allow you to either argue or support my particular view on the topic…

Wikipedia defines a stub as “a piece of code used to stand in for some other programming functionality”. Sadly, the use of stubs in “general software development and testing” is thrown in as an after-thought. They give much more emphasis to its use in distributed computing, which is logical in the sense that the project failure rate is much higher if stubs aren’t used in distributed application development, and some techniques, such as SOAP-based web services require them as part of the API. In my mind, the only reason not to stub is when you can write the final code from end to end as a single developer in a single day. Any task more complex requires some type of stubbing, the specifics of which will vary between teams.

Benefits of Stubbing

The following are just a few of the advantages of developing methods stubs that return a meaningful value. I’ve left out some as obvious (despite the fact that I often say the obvious bears repeating in development) and others that are a bit inflammatory (having worked many long hours fixing the issues of not stubbing properly I find it hard to be objective).

The Ability to Demonstrate Progress on Demand

Many developers will only stub their methods if they are working in a large team. One value of stubbing methods even when working alone is that not only will the code compile, the code will run and return meaningful values. While this may appear as a minimal value to code quality and speed of development, it can be an enormous advantage to demonstrate to non-technical user (i.e., those that determine your budget) exactly how much progress you have made. While the code may be in the same stage of completion with or without meaningful return values, it is far more acceptable to clients and management to explain that the demonstration is not ready for production than to have to constantly excuse test cases that fail because of a null value.

Facilitating a Test First Development Approach

Test first development is a very useful Agile method. By writing test cases first that check for valid values and then implementing code that moves the test cases from “fail” to “pass”, you will know that your methods are written correctly at the earliest point possible in the development process. By running your test cases automatically at each build you can then pinpoint many bugs immediately after they are written rather once your code is integrated into the full project. Integrated code takes longer to debug for many reasons, one of which is that everyone assumes it is someone else’s code, thus costing the time of multiple developers to locate the issue rather than the one developer who wrote it.

Parallel Development

While it can be argued that it is possible to develop in parallel without providing stubs, it is rare that the eventual integration of layers developed in parallel goes smoothly if stubs with meaningful return values are not used to test the contracts between teams early and often.

Weekends Off

Ok, this one won’t apply to everyone, but I bring it up because during the holiday weekend I wrote this article I was also debugging integration issues for the very reason that the points covered in this article were not followed by experienced developers. I understand that it was not their intent to cause more work, which is why the lessons here are important for both the newbies and seasoned professionals.

Misconceptions about Stubbed Methods

Stubbing is Only Necessary at the Model Layer

As mentioned under Benefits of Stubbing, one advantage to stubs is the ability to demonstrate on demand. The HTML of a view object that is blank or displays “null” is far less likely to help your budget case than seeing “John Doe”, which is obviously not real data but is meaningful data.

When working in parallel development it may be tempting to point at a project plan that has dependencies being delivered in perfect synchronization as consider stubbing a controller with null because the model should be ready before the view is integrated. In the event that everyone does not hit their milestone exactly (gee, how often does that happen?) development can fall behind waiting for someone else to catch up. Alternatively, if you get ahead of your own schedule you will not be able to progress until your dependencies are at the same plan stage as you. If all developers stub their code with meaningful value at every step, they are not dependant on others until the actual point of integration, and then only in the event of bugs, which are far fewer if stubbing was occurring at each layer and each stage.

Null is Not a Stub

There are two phrases that will raise the blood pressure of many team leads and development managers. The first is “it works on my machine”, and the second is “it compiled for me, so I checked it into source control”. It isn’t the mere fact that a developer’s machine will not be the one used in production that is the issue here (though, it is one of the obvious things that still need to be pointed out often), it is that these phrases are only used when the code in question has broken the application as a whole. The most common cause of integration builds failing is a null value that should not be null. It may even (and probably will) compile within the integrated application. The failure point is when the application is run, a point that is far more costly and the a compilation failure because it can impact other developers (who have to trace their code to find the cause from someone else’s code), QA engineering teams (who have to reschedule testing because the application isn’t available), and likelihood of future team projects if this disaster occurs during a client demonstration.

The effort involved in adding and later removing:

//FIXME: Remove when data available
if(value==null) {return “John Doe”;}
else return value;

should total about 15 minutes, versus the average 1.5 man hours it takes to detect the issues caused after deployment. And 15 minutes is on the long side, as it took me 25 seconds to write the above example and should take no more than 1 minute to remove it.

“Foo” Is Only Missing an “L”

Our example above returns a meaningful value. Even in the case where full JavaDoc comments are omitted, any developer seeing “John Doe” returned has a pretty good idea how they can use the method.  A return value of “foo” would give no such indication. Even if the method has full JavaDoc notations, the value “foo” is very unlikely to be of use to the client code. It may be helpful to consider that code examples that use “foo” generally use “bar” as the second value. “foo bar” is a homonym for FUBAR, which is an acronym for “F***** Up Beyond All Recovery”.

I recently worked on a project where stubs returned both null and “foo”. These values, coupled with generated JavaDoc comments rather than informative descriptions, led to delays in integration testing and the need to re-write 70% of the consumer code once integration began due to the assumed values not matching the actual values. A valid value returned by stubs for all methods would have prevented this and saved a substantial amount money (and one of my weekends).

Meaningful Stubs May End Up In Production

Some people believe that cleaning up stubbed code takes too much of their time, or that stubs can make their way into production by being missed. Looking at that second argument, it is only valid if comments are not used properly. All modern IDEs will provide a listing of comments that start with FIXME and TODO. Stubs that are commented with FIXME and cultivating the habit of reviewing these lists will always prevent stubs from making it to production and lead to cleaner code. Using these notations also allows for being able to quickly review code where stubs have been left and remove them when they are no longer needed.

Figure 1: Task View in Eclipse

Figure 1: Task View in Eclipse


Using a Stub Harness

If you don’t know how long it will be until you have data, consider writing a stub harness. A stub harness is a simple check to see if the class should use stubbed values before returning them. Whether to use the stubs or not can be set in a property file at a class or as a node in your deployment descriptor. This value can be set at a method level:


at the application level:


or both:

if(property.useStubs && property.thisMethod.useStubs)

The key to this approach is to make sure your build process will never allow these values to be set in either pre-production or production (or other applicable environments, depending on how your organization operates). Providing your processes include a regression tests in pre-prod, methods that are still stubbed will fail prior to going to production. It also allows you to switch off the stubs at any stage of development to run integration tests.


There are two inspirations for my article topics. Things that are cool, and things that are painful. In both cases my goal is to share the solution in the hopes of saving others the long, tedious discovery process that I and others have gone through. In the case of this particular topic, the tedious process isn’t the discovery but the realization that an approach that may seem at first glance to be time consuming is in fact time saving.

Fix for InvalidRegistrationException with a WLP Producer and WebCenter Portal Consumer

July 15th, 2014
No Gravatar

I have come across a case where I have deployed a WebCenter Portal EAR to a newly created domain that was copied in from another domain. When attempting to update the WLP producer registration with the new environment, I received a com.bea.wsrp.faults.InvalidRegistrationException.

I had done this sort of new domain configuration many times, with the only difference here being that the EAR I was deploying was copied from a running domain. While I don’t know the actual root cause of why it had a problem this time, I suspect that OEM is writing back to the EAR for some reason. Anyways…

After much head-banging, and successfully registering other domains to the same producer in a vain attempt to prove I was not insane, I re-read the error for the umpteenth time and finally realized what was causing the exception. Do you see it?:

<BEA-000000> <<Jun 16, 2014 3:08:28 PM PDT> <Error> <oracle.portlet.client.connection.wsrp.HTTPClientTransport> <WCS-40152> <A request to the producer URL "http://myhost5:8001/myAccount/producer/wsrp-2.0/markup" resulted in a status 500 response with fault string "The given registrationHandle [22001] is invalid.". The fault code given was "{urn:oasis:names:tc:wsrp:v2:types}InvalidRegistration". The producer generated a timestamp of 2014-06-16T15:08:28.584-07:00 and associated the following stack trace with the fault message: com.bea.wsrp.faults.InvalidRegistrationException: The given registrationHandle [22001] is invalid.

So, since it was trying to register for the first time with a handle that had not been created yet, it made sense that it would fail. The WLP schema is a bit too complex to allow just creating the one row, so instead I went back to JDeveloper, updated the registration handle there and built a new EAR. The new EAR had a new value for MDS so that when I tried for the last time to update the producer it decided to request a handle rather than insist it already had one.

Adding Problem 3 for “23.2.1 Portlet Displays a Portlet Consumer Error”

June 20th, 2014
No Gravatar

The Troubleshooting Oracle WebCenter Portlets documentation lists two problems under the Portlet Displays a Remote Portlet Communication Error section. Both seem legitimate causes and solutions, and I have run across both of them in the past. I hit a new one this week. Number three is running out of available threads in the Managed Server that is handling the request. This one was fun to find, because OOM errors that do not cause the server to fall over eventually clear up, and with an adequately sized cluster (the memory errors were caused by a dependent service failing) the issue is very random.

The symptom to look for is the OOM error in the log, which looks like this:

oracle.portlet.client.container.PortletException: java.lang.OutOfMemoryError: 
unable to create new native thread
	at oracle.portlet.client.tech.common.PortletFutureUtils.get(Portlet
	at oracle.portlet.client.tech.common.PortletSessionManager$InitSession
	at oracle.portlet.client.service.pipeline.FutureConditionTask.done
	at java.util.concurrent.FutureTask$Sync.innerSetException(Future
	at java.util.concurrent.FutureTask.setException(FutureTask.java:125)

Restarting the servers will give you temporary relief, though for the long-run you need to find the root cause. In this case, it was an undersized WebCenter Content cluster running on the same physical box sucking all the resources down. A larger cluster and load balancing corrected the issue for good (knock wood).

Leveling the Playing Field

May 11th, 2014
No Gravatar

(semi-continuation of A Tale of Two Migrations)

Before I get into the trials and tribulations of dealing with the inevitable custom implementations that are part of every WebLogic Portal application that is older than a year and has more than 5 pages (and there are newer, smaller apps with more customizations, trust me on that), it is best to look at some basics. In the path taken by this blog series we are using WSRP with the retiring WebLogic Portal (WLP) application as the producer and WebCenter Portal (WCP) as the migration target and WSRP consumer. It is definitely of value to review the standard documentation about how to configure this standards-based integration, which can be found for 10.3.6 at http://docs.oracle.com/cd/E41236_01/wlp.1036/e14235/chap_webcenter_interop.htm/ .

With all of the useful points provided in the documentation, the one point that I would like to add is to first start with upgrading your WebLogic Portal application and environment to the latest version. I am a strong proponent of not upgrading unless there is a clear value in doing so, and in this case there are several. One is that a migration from WLP to WCP is almost definitely going to require at least one (and most likely several) Oracle Support cases to complete. You will hear every time you open a case “please upgrade to the latest version”. I’m not a big fan of this approach, but since the goal of the project is to move everything to the WebCenter and all of the functionality will eventually need to be updated to the latest version as part of that process, it is worthwhile to get that out of the way now.

Another value of upgrading your WebLogic Portal before embarking on a staged migration is that with one of the release of WebLogic Portal 10.3.x from .2 or later (I didn’t know the specific version, am too lazy to go look it up, and don’t feel bad about that because it is moot at this point in time) and the corresponding WebCenter Portal release (see previous parenthetical comment) the WSRP engine was moved from the portal framework to the application server (WebLogic Server). This makes a great deal of sense since WebLogic Server has had a WSRP engine for quite some time (again, not doing the historical research for the curious). Since the underlying WSRP support is coming from the application server, it makes good sense to have the same engine on both sides of the tractor-pull known as WSRP. While it may not be entirely necessary between some versions (I strongly suspect –but have no proof- that this is the case between 10.3.5 and 10.3.6), it eliminates one point in the long chain that will eventually be examined in detail for the source of some unexpected result or other, so why not get it out of the way now?

Finally, WebLogic Server is still the gold standard of J2EE application servers and does continuously improve with each release (disclaimer: I have not worked with the 12c versions and am just hoping that they have not proven me wrong). So why not start off with the best foundation possible before dealing with the known risk of every migration which is that there will be unknown and unforeseen issues that arise.

In my case I pushed for the WebLogic Portal and Server upgrade to 10.3.6 because WebCenter had just been released with its capability of including the MAR within the EAR, a major time and hassle saver. WebCenter was release with no support for WebLogic Server 10.3.5, which explains the “coincidence” of WebLogic Portal 10.3.6 coming out two weeks before hand. Interestingly enough, development on WCP is done with JDeveloper which is only configured to work with WebLogic Server 10.3.5. Ah, if Fox Mulder were only a software engineer instead of an FBI agent ;)