within a story, {after} block executed not after all scenarios, but right after {before} block #9

sergey-lysenko · 2014-01-30T15:21:01Z

I tried to upgrade to easyb-core 1.5 (from easyb 0.9.7_1) and stumbled upon very strange behavior described in title of this issue. Is this a known issue, or misconfiguration, or just needs to be investigated and fixed?

rvowles · 2014-01-30T18:15:32Z

Could you provide your scenario?
On 31 Jan 2014 04:21, "Sergey Lysenko" [email protected] wrote:

I tried to upgrade to easyb-core 1.5 (from easyb 0.9.7_1) and stumbled
upon very strange behavior described in title of this issue. Is this a
known issue, or misconfiguration, or just needs to be investigated and
fixed?

Reply to this email directly or view it on GitHubhttps://github.com//issues/9
.

JuergenKindler · 2014-01-31T12:13:28Z

As far as I know, the scenario is the one below.
Basically it is the same as the issue103.specification , but in a story ….
I am looking into some logs and see that the order of the execution is not
before, scenario, after but before, after, scenario ….

Instead of parsing first and then executing, it looks like easyb is executing in parsing phase …
Note, however: even if I move the after block behind the scenario, it still gets executed before the scenario :-(

import org.easyb.exception.VerificationException

Logger log = Logger.getLogger("Issue103.story")

before "setup string", {

var = "set in before"

log.info("In before block '$var'")

}

after "tear down", {

var = "set in after"

log.info("In after block '$var'")

}

scenario "Validate setup and teardown", {

then "the variable should be set with value from before block" , {

    log.info("In then block '$var'")

    var.shouldBe "set in before"

}

}

if (var != "set in after") {

throw new VerificationException("after closure doesn't appear to be working!")

}

Cheers

Jürgen

Jürgen Kindler

Von: Richard Vowles <[email protected]mailto:[email protected]>
Antworten an: easyb/easyb-core <[email protected]mailto:[email protected]>
Datum: Thursday, January 30, 2014 19:15
An: easyb/easyb-core <[email protected]mailto:[email protected]>
Betreff: Re: [easyb-core] within a story, {after} block executed not after all scenarios, but right after {before} block (#9)

Could you provide your scenario?
On 31 Jan 2014 04:21, "Sergey Lysenko" <[email protected]mailto:[email protected]> wrote:

I tried to upgrade to easyb-core 1.5 (from easyb 0.9.7_1) and stumbled
upon very strange behavior described in title of this issue. Is this a
known issue, or misconfiguration, or just needs to be investigated and
fixed?

Reply to this email directly or view it on GitHubhttps://github.com//issues/9
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-33715898.

sergey-lysenko · 2014-01-31T18:46:41Z

Another example:

before "...",{
    ...
}

scenario "....", {
...
}

scenario "....", {
...
}

scenario "....", {
...
}

scenario "....", {
...
}

after "...",{
    ...
}

Order of execution is

before
after
scenario
scenario
scenario
scenario

rvowles · 2014-01-31T18:55:11Z

It should parse these and collect the closure blocks before execution of
the closure blocks. It won't matter where the blocks are.

This was a documented behaviour change from the 0.x version to support the
extended syntax.
On 1 Feb 2014 07:46, "Sergey Lysenko" [email protected] wrote:

Another example:

before "...",{
...
}

scenario "....", {
...
}

scenario "....", {
...
}

scenario "....", {
...
}

scenario "....", {
...
}

after "...",{
...
}

Order of execution is

before
after
scenario
scenario
scenario
scenario

Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-33830181
.

JuergenKindler · 2014-02-03T07:27:46Z

Hi Richard,

I am not sure what you mean with "It should parse these and collect the closure blocks before execution of the closure blocks" ?
If I look at org.easyb.StoryKeyword.parseScenario I do not see where the before / before_each / after /after_each blocks are collected.
Instead is looks like, they will be executed right away. The confusing thing is that the closure execution has the comment " // now parse the scenario".

I played a bit with this method and added conditions for before[_each] and after[_each] to store the closures in the StoryContext variables

BehaviorStep beforeScenarios

BehaviorStep afterScenarios

BehaviorStep beforeEach

BehaviorStep afterEach

But some of the test cases – e.g. in org.easyb.bdd.prototype (OneTimeFixtureFeature) require the direct execution of some blocks (as otherwise variables that get defined inside closures in these test cases would otherwise be undefined when the last script statement gets executed).

If I modify my example a little bit I (and use the checked-in version of the logic) get a failure that shows what happens by the content of the variable var. When I execute:
import org.easyb.exception.VerificationException
import java.util.logging.Logger

Logger log = Logger.getLogger("Issue103.story")
def setInBefore = "Set in before"
def addInGiven = " + in given"
def addInAfter = " + in after"

before "setup string", {
var = setInBefore
log.info("In before block '$var'")
}

after "tear down", {
var = var += addInAfter
log.info("In after block '$var'")
}

scenario "Validate setup and teardown", {
given "var is changed", {
var += addInGiven
}
then "the variable should be set with value from before block" , {
log.info("In then block '$var'")
var.shouldBe "set in before" + addInGiven
}
}

println "var = $var"
// This would be my intuitive expectation about the order of execution: before, given, after
if (var != "${setInBefore}${addInGiven}${addInAfter}") {
throw new VerificationException("after closure doesn't appear to be working!")
}

I get the following output (contains a little more than the expected lines as I added a bit of logging also in StoryProcessing and StoryKeywords:
[easyb] Running issue103 story (/home/jkindler/workspaces/easyb/easyb-core/src/test/groovy/org/easyb/bdd/issues/Issue103.story)
[easyb] AST processing Issue103 (/home/jkindler/workspaces/easyb/easyb-core/src/test/groovy/org/easyb/bdd/issues/Issue103.story)
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.reflection.CachedMethod invoke
[easyb] Information: Parse scenario setup string
[easyb] Type: BEFORE
[easyb] Source: before "setup string", {
[easyb] var = setInBefore
[easyb] log.info("In before block '$var'")
[easyb] }
[easyb] Line: 9 -> org.easyb.StoryContext@654efa45
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap invoke
[easyb] Information: In before block 'Set in before'
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.reflection.CachedMethod invoke
[easyb] Information: Parse scenario tear down
[easyb] Type: AFTER
[easyb] Source: after "tear down", {
[easyb] var = var += addInAfter
[easyb] log.info("In after block '$var'")
[easyb] }
[easyb] Line: 19 -> org.easyb.StoryContext@654efa45
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap invoke
[easyb] Information: In after block 'Set in before + in after'
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.reflection.CachedMethod invoke
[easyb] Information: In scenario Validate setup and teardown
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.reflection.CachedMethod invoke
[easyb] Information: Parse scenario Validate setup and teardown
[easyb] Type: SCENARIO
[easyb] Source: scenario "Validate setup and teardown", {
[easyb] given "var is changed", {
[easyb] var += addInGiven
[easyb] }
[easyb] then "the variable should be set with value from before block" , {
[easyb] log.info("In then block '$var'")
[easyb] var.shouldBe "set in before" + addInGiven
[easyb] }
[easyb] }
[easyb] Line: 29 -> org.easyb.StoryContext@654efa45
[easyb] Feb 03, 2014 7:46:04 AM org.codehaus.groovy.reflection.CachedMethod invoke
[easyb] Information: After scenario Validate setup and teardown
[easyb] var = Set in before + in after
[easyb] There was an error running your easyb story or specification
[easyb] source is null
[easyb] VerificationException: after closure doesn't appear to be working!:
[easyb] at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
[easyb] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[easyb] at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
[easyb] at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
[easyb] at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
[easyb] at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)
[easyb] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
[easyb] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
[easyb] at Issue103.run(Issue103.story:41)
[easyb] at groovy.lang.GroovyShell.evaluate(GroovyShell.java:518)
[easyb] at groovy.lang.GroovyShell.evaluate(GroovyShell.java:556)
[easyb] at groovy.lang.GroovyShell.evaluate(GroovyShell.java:537)
[easyb] at org.easyb.domain.Story.execute(Story.java:56)
[easyb] at org.easyb.BehaviorRunner.executeBehaviors(BehaviorRunner.java:131)
[easyb] at org.easyb.BehaviorRunner.runBehaviors(BehaviorRunner.java:83)
[easyb] at org.easyb.BehaviorRunner.main(BehaviorRunner.java:60)
[easyb] easyb execution FAILED

I was not involved in the design & implementation of the execution model, so this triggers a couple of questions in my head:

is it intentional that variables defined in before[_each] / after[_each] closures do not have to be defined on script level?
Where would I find the code that would need to collect and coordinate the execution of those blocks during evaluation of the script and ensures that those blocks are being executed possibly multiple times (for the *_each blocks) ?
Is that something that would first have to happen on the level of the AST to transform the script source into a tree that (esp. In case of _each blocks) ensures that those blocks are added in the correct spots so that the groovy interpreter (which does not know about the semantics of the easyb DSL) can call then in the right moment?
What is the expected order of execution in case we have before_each and before or after and after_each blocks?

I'm sure you know more about that – for myself I have to admit that I still lack understanding the code :-(

Cheers
Jürgen

Jürgen Kindler

Von: Richard Vowles <[email protected]mailto:[email protected]>
Antworten an: easyb/easyb-core <[email protected]mailto:[email protected]>
Datum: Friday, January 31, 2014 19:55
An: easyb/easyb-core <[email protected]mailto:[email protected]>
Cc: Jürgen Kindler <[email protected]mailto:[email protected]>
Betreff: Re: [easyb-core] within a story, {after} block executed not after all scenarios, but right after {before} block (#9)

It should parse these and collect the closure blocks before execution of
the closure blocks. It won't matter where the blocks are.

This was a documented behaviour change from the 0.x version to support the
extended syntax.
On 1 Feb 2014 07:46, "Sergey Lysenko" <[email protected]mailto:[email protected]> wrote:

Another example:

before "...",{
...
}

scenario "....", {
...
}

scenario "....", {
...
}

scenario "....", {
...
}

scenario "....", {
...
}

after "...",{
...
}

Order of execution is

before
after
scenario
scenario
scenario
scenario

Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-33830181
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-33830997.

rvowles · 2014-02-03T21:01:45Z

On Mon, Feb 3, 2014 at 8:27 PM, Jürgen Kindler [email protected]:

Hi Richard,

I was not involved in the design & implementation of the execution model,
so this triggers a couple of questions in my head:

is it intentional that variables defined in before[_each] / after[_each]
closures do not have to be defined on script level?

Thats just Groovy scripts, so normally you don't use def anywhere in your
scripts, so yes :-) If you don't use def, they get defined on the binding
itself and can be passed around more easily.

Where would I find the code that would need to collect and coordinate
the execution of those blocks during evaluation of the script and ensures
that those blocks are being executed possibly multiple times (for the
*_each blocks) ?

Yes, I was working on my own at that point and I really think I missed
something - I had intended to capture the
after/before/before_each/after_each just like I capture the
given/when/then/where/example in the scenario blocks, but I appear not to
have done so. I'm not sure why and I would need to look more closely - I
need an excuse to use easyb again it seems - I havent' really looked at it
for almost 3 years.

Is that something that would first have to happen on the level of the
AST to transform the script source into a tree that (esp. In case of _each
blocks) ensures that those blocks are added in the correct spots so that
the groovy interpreter (which does not know about the semantics of the
easyb DSL) can call then in the right moment?

The groovy interpretor is just executing them sequentially - the closure
blocks are passed as parameters, so the code itself is deciding when to
execute them. For some reason, the before/after/etc are being executed
immediately - scenario should be, but not the others, unless there is some
extra syntax I am missing.

What is the expected order of execution in case we have before_each and
before or after and after_each blocks?

It should go: before, before_each/scenario/after_each, after - as you would
sort of expect :-)

I'm sure you know more about that - for myself I have to admit that I
still lack understanding the code :-(

It took me ages to figure it all out. But this bug is clearly a bug as it
is executing the after immediately, but the scenarios are just being
parsed.

Richard Vowles,
Groovy, Java, Javascript, AngularJS

ph: +64275467747, web: www.google.com/+RichardVowles

JuergenKindler · 2014-02-04T11:41:22Z

Thanks, Richard!
I'll try to dig in to this as well – seems to be a good spot to gain a deeper understanding this …

Cheers

Jürgen

Jürgen Kindler

Von: Richard Vowles <[email protected]mailto:[email protected]>
Antworten an: easyb/easyb-core <[email protected]mailto:[email protected]>
Datum: Monday, February 3, 2014 22:01
An: easyb/easyb-core <[email protected]mailto:[email protected]>
Cc: Jürgen Kindler <[email protected]mailto:[email protected]>
Betreff: Re: [easyb-core] within a story, {after} block executed not after all scenarios, but right after {before} block (#9)

On Mon, Feb 3, 2014 at 8:27 PM, Jürgen Kindler <[email protected]mailto:[email protected]>wrote:

Hi Richard,

I was not involved in the design & implementation of the execution model,
so this triggers a couple of questions in my head:

is it intentional that variables defined in before[_each] / after[_each]
closures do not have to be defined on script level?

Thats just Groovy scripts, so normally you don't use def anywhere in your
scripts, so yes :-) If you don't use def, they get defined on the binding
itself and can be passed around more easily.

Where would I find the code that would need to collect and coordinate
the execution of those blocks during evaluation of the script and ensures
that those blocks are being executed possibly multiple times (for the
*_each blocks) ?

Yes, I was working on my own at that point and I really think I missed
something - I had intended to capture the
after/before/before_each/after_each just like I capture the
given/when/then/where/example in the scenario blocks, but I appear not to
have done so. I'm not sure why and I would need to look more closely - I
need an excuse to use easyb again it seems - I havent' really looked at it
for almost 3 years.

Is that something that would first have to happen on the level of the
AST to transform the script source into a tree that (esp. In case of _each
blocks) ensures that those blocks are added in the correct spots so that
the groovy interpreter (which does not know about the semantics of the
easyb DSL) can call then in the right moment?

The groovy interpretor is just executing them sequentially - the closure
blocks are passed as parameters, so the code itself is deciding when to
execute them. For some reason, the before/after/etc are being executed
immediately - scenario should be, but not the others, unless there is some
extra syntax I am missing.

What is the expected order of execution in case we have before_each and
before or after and after_each blocks?

It should go: before, before_each/scenario/after_each, after - as you would
sort of expect :-)

I'm sure you know more about that - for myself I have to admit that I
still lack understanding the code :-(

It took me ages to figure it all out. But this bug is clearly a bug as it
is executing the after immediately, but the scenarios are just being
parsed.

Richard Vowles,
Groovy, Java, Javascript, AngularJS

ph: +64275467747, web: www.google.com/+RichardVowles

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-33999444.

JuergenKindler · 2014-02-21T09:07:22Z

Hi Richard,

I have gotten closer to solving this issue, but I feel the solution is only partially.
The point I am currently stuck with is that it seems that what is called "parsing" in StoryKeywords.parseScenario is actually executing the closures of the real test script.
This is all fine if closures like before[_each] and after[_each] are containing other easyb keywords. If not the closure will really execute code and thus we get a strange situation then: those directly executable code fragments will get executed twice. Note that in the issue I define the variable var at the top and then only update it:

import org.easyb.exception.VerificationException

import java.util.logging.Logger

Logger log = Logger.getLogger("Issue103.story")

def var = ""

def addInBefore = "In before"

def addInBeforeEach = " + in each before"

def addInGiven = " + in given"

def addInAfter = " + in after"

def addInAfterEach = " + in each after"

before "[Before block", {

var += addInBefore

log.info("In before block '$var'")

}

before_each "[Before_each block", {

var += addInBeforeEach

log.info("In before_each block '$var'")

}

after "After block]", {

var += addInAfter

log.info("In after block '$var'")

}

after_each "After each block]", {

var += addInAfterEach

log.info("In after_each block '$var'")

}

scenario "Validate setup and teardown", {

given "var is changed", {

    var += addInGiven

}

then "the variable should be set with value from before block" , {

    log.info("In then block '$var'")

    var.shouldBe addInBefore + addInBeforeEach + addInGiven

}

}

The result is unfortunately:
[easyb] FAILURE Scenarios run: 1, Failures: 1, Pending: 0, Time elapsed: 1.894 sec
[easyb]
[easyb] scenario "Validate setup and teardown"
[easyb] step THEN "the variable should be set with value from before block" -- expected In before + in each before + in given but was In before + in each before + in after + in each afterIn before + in each before + in given
[easyb] 1 total behavior ran with 1 failure

So the green parts in the actual value is what would be expected from execution and the red ones are related to "parsing". So at least they get executed also when they should be executed, but unfortunately not exclusively then.
That can have quite some side effects when handling external resources like database connection & initialization in such blocks. :-(
I would assume that there will be no problems if we have blocks like given/then/when in there, but directly executable closures have this side effect.
Gets me wondering if traversing the AST and using that information would help here …
I'm not using directly executable closures, but others might …

Cheers
Jürgen

Jürgen Kindler

JuergenKindler · 2015-05-11T06:39:05Z

With version 1.6 that problem should be fixed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

within a story, {after} block executed not after all scenarios, but right after {before} block #9

within a story, {after} block executed not after all scenarios, but right after {before} block #9

sergey-lysenko commented Jan 30, 2014

rvowles commented Jan 30, 2014

JuergenKindler commented Jan 31, 2014

sergey-lysenko commented Jan 31, 2014

rvowles commented Jan 31, 2014

JuergenKindler commented Feb 3, 2014

rvowles commented Feb 3, 2014

JuergenKindler commented Feb 4, 2014

JuergenKindler commented Feb 21, 2014

JuergenKindler commented May 11, 2015

within a story, {after} block executed not after all scenarios, but right after {before} block #9

within a story, {after} block executed not after all scenarios, but right after {before} block #9

Comments

sergey-lysenko commented Jan 30, 2014

rvowles commented Jan 30, 2014

JuergenKindler commented Jan 31, 2014

Jürgen

sergey-lysenko commented Jan 31, 2014

rvowles commented Jan 31, 2014

JuergenKindler commented Feb 3, 2014

rvowles commented Feb 3, 2014

JuergenKindler commented Feb 4, 2014

Jürgen

JuergenKindler commented Feb 21, 2014

JuergenKindler commented May 11, 2015