Plan <attributes/> and <property/> usage in nested plans

[dsklyut]

Hi,

I wonder if there is a way to specify properties for bundles in one plan (nested.plan) and attributes for that property in plan (outer.plan) that will import "nested.plan"?

nested.plan

...
<artifact type="bundle" name="webapp">
        <property name="header:Web-ContextPath" value="${host.context.path}"/>
</artifact>
...

outer.plan

<attributes>
        <attribute name="host.context.path" value="/host-outer"/>
    </attributes>
<artifact type="bundle" name="core"/>
<artifact type="bundle" name="application"/>
<artifact type="plan" name="nested.plan"/>

From my testing it looks like attributes specified in the outer plan and not propagated into nested plan. If nested plan does not specify attribute for a placeholder value, generation of ArtifactDescriptor fails. If both nested.plan and outer.plan specify attribute value, the one in nested.plan is used.

Failed to read plan descriptor
	at com.springsource.kernel.artifact.plan.PlanBridge.generateArtifactDescriptor(PlanBridge.java:84)
	at com.springsource.repository.internal.LocalRepository.createArtifactDescriptor(LocalRepository.java:70)
	at com.springsource.repository.internal.watched.WatchedStorageRepository.access$2(WatchedStorageRepository.java:1)
	at com.springsource.repository.internal.watched.WatchedStorageRepository$DirectoryWatcher$1.onChange(WatchedStorageRepository.java:153)
....

Caused by: java.lang.RuntimeException: Failed to read plan descriptor
	at com.springsource.kernel.artifact.plan.PlanReader.read(PlanReader.java:95)
	at com.springsource.kernel.artifact.plan.PlanBridge.generateArtifactDescriptor(PlanBridge.java:82)
	... 58 common frames omitted
Caused by: java.lang.RuntimeException: No value found for placeholder 'host.context.path'
	at com.springsource.util.common.PropertyPlaceholderResolver.resolve(PropertyPlaceholderResolver.java:122)
	at com.springsource.util.common.PropertyPlaceholderResolver.resolve(PropertyPlaceholderResolver.java:134)
	at com.springsource.util.common.PropertyPlaceholderResolver.resolveProperty(PropertyPlaceholderResolver.java:114)
	at com.springsource.util.common.PropertyPlaceholderResolver.resolve(PropertyPlaceholderResolver.java:107)
	at com.springsource.util.common.PropertyPlaceholderResolver.resolve(PropertyPlaceholderResolver.java:92)
	at com.springsource.kernel.artifact.plan.PlanReader.replacePlaceholders(PlanReader.java:170)
	at com.springsource.kernel.artifact.plan.PlanReader.parseArtifactProperties(PlanReader.java:163)
	at com.springsource.kernel.artifact.plan.PlanReader.parseArtifactElements(PlanReader.java:149)
	at com.springsource.kernel.artifact.plan.PlanReader.parsePlanElement(PlanReader.java:122)
	at com.springsource.kernel.artifact.plan.PlanReader.read(PlanReader.java:93)
	... 59 common frames omitted

I don't mind modifying dm-server code or decorating it in some way. Our deployment model is to use nested plans right now. Outer plan file is basically a "scope" aggregator of related "un-scoped" components to form an "application". Use of <attributes/> and <properties/> has a potential for providing an application configuration option that is not really available from config admin stand point - because (correct me here) config admin is not "scoped" to application. Another point of doing this is to be able to deploy multiple applications side by side (i.e. dev and qa) and specify context url for web-apps at the outer most level.

[dsklyut]

A little follow up:

I spent some time trolling through code base of deployer and I think I can achieve some of that by introducing a new artifact type. Something like "plan-fragment" in line with bundle fragments. This will get me over the ArtifactDescriptor creation issue.
But I would still need to either:
1. modify PlanResolver to push outer.plan attributes on the included plan-fragments.
2. create another transformer that runs prior(?) or right after (which is better I think as the tree would be created) PlanResolver and push outer plan attributes on child plan-fragments

I haven't scoped the amount of work that this involve. Am I even close to the "golden path" with this analysis?

[Glyn Normington]

Sounds like there is missing function here. I don't like the idea of plan fragments though as they may have all the downsides of bundle fragments (yes, I know they can be incredibly useful, but they have far outgrown their intended I18N use case).

A basic but clean approach would be to propagate (i.e. copy when not overriddn) attributes down the tree either during plan resolution or, preferably from an encapsulation/code structure/unit testing POV, soon after. So it sounds like a new Transformer needs adding with an appropriate service ranking.

[dsklyut]

One problem with that is eager resolution of property placeholder during initial provisioning. At that time there is no knowledge of who is the parent of the placeholder holding plan. That information is only available when parent plan is deployed.

Thought:
It would be nice to have another attribute on the plan to mark it "incomplete" vs. creating a brand new artifact type. This attribute can be used at an argument to PlanReader.PropertyPlaceholderResolver to skip unresolved placeholders. Later when deployment happens are we have a full tree - plans with "incomplete" flag are re-resolved for placeholder values and deployment fails in that case if there are any unresolved placeholders left.

In any case there must be a new transformer that pushes parent plan attributes onto child plans during/after PlanResolver phase. That same transformer could be in charge of re-resolving placeholders or create one more transformer i.e. SecondPassPlanAttributeResolver to do resolution.

This is interesting stuff. I really dig the ability to extend the server with fairly minimal pain.

[Glyn Normington]

Please could you say what you mean by "initial provisioning" as, from my perspective, the plan resolver recursively provisions the tree of plans and so as each node of type PlanInstallArtifact is created, we know who its parent is.

(It's a good job we put the restriction of at most one parent into 2.0! That is, install artifacts form a forest rather than a general DAG. There will be lots of headaches if and when we remove that restriction, but let's discuss that on another occasion.)

[dsklyut]

By "initial provisioning" I mean this chain of calls:

WatchedStorageRepository -> LocalRepository.createArtifactDescriptor -> PlanBridge.generateArtifactDescriptor

This happens when repositories are scanned prior to anything in pickup being deployed. Basically the stack trace that I provided in the initial post.

This might be a bad choice of words, because there is no "provisioning" that happens just examination of available artifacts in the repositories. Sorry for confusion.

I really would like to find some solution to this. I need to be able to switch context path for web applications without modifying multiple plan definitions. Need to re-examine our deployment strategy I guess.

[Glyn Normington]

I see. That chain of events is the watched directory adding the plan artifact to the repository. But this isn't necessarily happening in the server in which the plan will be deployed so there is no way we can transform the content of the plan or do property placeholder processing etc.

For example, the dm Server instance in which the repository is defined could have a very different configuration to the dm Server instance that will eventually deploy the plan.

Or, as another example, the plan could be deployed into more than onedm Server instance each with differing configuration (but sharing a common repository).

Please note I would prefer to reserve the term "provisioning" for when we pull artifacts out of a repository in response to a deployment operation.

So how does this affect your analysis?

[dsklyut]

Do you see a problem with delaying placeholder replacement till "deployment" time?

To summarize:

Without modifications to dm-server code base there is no way to have parametarized plans that can be configured with a parent plan attribute values.

Because we were heading down the path of using nested plans to deploy multiple configurations of application side by side in the same dm-server instance (qa, dev, demo, etc), we would need to re-examine this approach and maybe go with a single plan that includes all artifacts, attributes, and properties to allow for side by side deployment.

Currently attributes/properties in a single plan seems to be the only supported way to modify web application context path.

I am not saying that this is a bad thing, just trying to figure out my options.

[Glyn Normington]

Hmmm - I thought it was sounding plausible for a Transformer to propagate the necessary plan configuration from parent plans to child plans. I must be missing something. Referring back to your example at the start of this thread, the Transformer would convert the child plan from:

...
<artifact type="bundle" name="webapp">
        <property name="header:Web-ContextPath" value="${host.context.path}"/>
</artifact>
...

to:

...
<attributes>
        <attribute name="host.context.path" value="/host-outer"/>
</attributes>
<artifact type="bundle" name="webapp">
        <property name="header:Web-ContextPath" value="${host.context.path}"/>
</artifact>
...

[dsklyut]

Yes, but before we get to a transformer phase, there is a little thing of a artifact repository. Because nested.plan fails to be added to repository, outer.plan can't find nested.plan to transform.

Chicken and egg problem

[Glyn Normington]

Thanks for your patience: at last I understand! We need to shoot the chicken or break the egg. :-)

What I mean is that property resolution should not be happening under the PlanBridge, which only needs to extract the name and version from a plan file. Therefore, in retrospect (since properties were, I think, added relatively late in the day), reusing PlanReader was a mistake.

A common helper class, called something like PlanIdentityParser, should be factored out of PlanReader. PlanIdentityParser should be responsible for extracting the name and version from a plan file. PlanReader and PlanBridge can use PlanIdentityParser. Then PlanBridge won't try to resolve property placeholders.

[dsklyut]

That would do the trick. Should I create an issue for this?

[Glyn Normington]

If you like, but we are planning on doing a big cull of the issues in order to clear the air ready for Virgo, so why not add this to a list of proposed improvements and float it on the Virgo forum until such time as we have a bug tracker for Virgo?

[dsklyut]

Will do.

Thanks for your help.