Obfuscation & unit testing

[Michael Reppy]

Given the relatively easy decompiling of .Net assemblies, we have
some interest in applying obfustication to our commercially released
assemblies using a product like Dotfuscator. I'm just poking around
at the issues and whatnot involved, but it seems like obfuscation and
unit testing are somewhat orthogonal to each other. I was just
wondering what others in the community had done or if there was more
info. Sourceforge wouldn't let me search the archives of the mailing
list and I couldn't find much of anything on Google.

Thanks for any advice, comments or links!

-Mikey
Mikey Reppy,
***@abacalab.com


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[Cory Foy]

Post by Michael Reppy
Given the relatively easy decompiling of .Net assemblies, we have
some interest in applying obfustication to our commercially released
assemblies using a product like Dotfuscator. I'm just poking around
at the issues and whatnot involved, but it seems like obfuscation and
unit testing are somewhat orthogonal to each other. I was just
wondering what others in the community had done or if there was more
info. Sourceforge wouldn't let me search the archives of the mailing
list and I couldn't find much of anything on Google.

Hi Mikey,

Typically obsfucation is done just before the product releases, after
the unit testing has been done. Personally, my job involved flying
onsite to customers when systems go wrong, and debugging third-party
systems, and obsfucation ends up just delaying us getting the systems
online. I curse at SharePoint everytime we have to dig into it (yes,
aa() is calling into b() and it's call to zhfrtt() is taking 38 seconds).

I guess the question is, are you looking to run your unit tests after
obsfucation, or use obsfucated binaries?
--
Cory Foy
http://www.cornetdesign.com


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[Michael Reppy]

Post by Cory Foy
Typically obsfucation is done just before the product releases, after
the unit testing has been done. Personally, my job involved flying
onsite to customers when systems go wrong, and debugging third-party
systems, and obsfucation ends up just delaying us getting the systems
online. I curse at SharePoint everytime we have to dig into it (yes,
aa() is calling into b() and it's call to zhfrtt() is taking 38 seconds).
I guess the question is, are you looking to run your unit tests after
obsfucation, or use obsfucated binaries?

I appreciate the support problems with obfuscation, and I guess I'm just
going to have to live with them as it is mostly not my call there. I
had already raised that issue with management and agreed to lose on it :-)

I want to be able to run the unit tests with and without obfuscation. I
am pretty leery of the idea of taking nicely tested code and then going
and futzing with it right before packaging a release! But if the unit
test run is clean, it's probably not more broken than what I wrote in
the first place.

What I envision is a TDD process without obfuscation most of the time.
When an external release is made, the normal tests will be run, the
application dlls will be obfuscated and the tests will be rerun to
verify that there were no issues introduced by obfuscation. I then want
to package those very same dlls as the release items, so the release
itself is tested.

Our application is complicated by the fact that we have a compact
framework application which is 80% of the code and a small desktop
component that is the remainder, so we've got two different platforms to
target.

It seems if I rearrange how we are handling strong naming issues and set
up the solution to deploy the application dlls and test dlls to one
folder I can obfuscate the application dlls but not the test dlls and
still have them linked and the tests continue to have their highly
descriptive names but the stack traces show that they are testing the
obfuscted dlls.

That all seemed to work fine. The only potentially fatal flaw I found
is with the strong naming issue, something I don't entirely understand
in .net, particularly the differences between reflection on the desktop
and handheld. The help & support from the Dotfuscator people seemed to
have a process for applying strong names after the fact, which I can
follow after I spend some time figuring out what's going on.

Anyway, that's what I've figured out so far. Any thoughts, advice &
experience appreciated!

-Mikey Reppy
***@abacalab.com


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

Hi Michael,

Post by Michael Reppy
I appreciate the support problems with obfuscation, and I
guess I'm just going to have to live with them as it is
mostly not my call there. I had already raised that issue
with management and agreed to lose on it :-)
I want to be able to run the unit tests with and without
obfuscation. I am pretty leery of the idea of taking nicely
tested code and then going and futzing with it right before
packaging a release! But if the unit test run is clean, it's
probably not more broken than what I wrote in the first place.
What I envision is a TDD process without obfuscation most of
the time.
When an external release is made, the normal tests will be
run, the application dlls will be obfuscated and the tests
will be rerun to verify that there were no issues introduced
by obfuscation. I then want to package those very same dlls
as the release items, so the release itself is tested.

Now that you mention it, that makes sense. I'd also want to have
a higher-level, acceptance-style test, which could be run on
the shipping release.

This is actually a new argument for something I have talked
about before: don't do acceptance tests in NUnit.

Post by Michael Reppy
Our application is complicated by the fact that we have a
compact framework application which is 80% of the code and a
small desktop component that is the remainder, so we've got
two different platforms to target.

Off topic... keep an eye peeled for coming integration between
NUnit and NUnitLite for testing compact framework apps.

Post by Michael Reppy
It seems if I rearrange how we are handling strong naming
issues and set up the solution to deploy the application dlls
and test dlls to one folder I can obfuscate the application
dlls but not the test dlls and still have them linked and the
tests continue to have their highly descriptive names but the
stack traces show that they are testing the obfuscted dlls.

Interesting. I wouldn't have thought you could do that.

If that works, it makes sense: no reason to obfuscate the
tests.

Charlie

Post by Michael Reppy
That all seemed to work fine. The only potentially fatal
flaw I found is with the strong naming issue, something I
don't entirely understand in .net, particularly the
differences between reflection on the desktop and handheld.
The help & support from the Dotfuscator people seemed to have
a process for applying strong names after the fact, which I
can follow after I spend some time figuring out what's going on.
Anyway, that's what I've figured out so far. Any thoughts,
advice & experience appreciated!
-Mikey Reppy
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Nunit-users mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-users

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Michael Reppy]

Post by Charlie Poole
Now that you mention it, that makes sense. I'd also want to have
a higher-level, acceptance-style test, which could be run on
the shipping release.

We have two levels of acceptance tests: (1) Scripts that a human has
to follow, which verify that various things are working on the device
side as well as device to desktop communication, and (2) long
complicated NUnit tests, often using NUNit Forms to drive the
application. These are at least automated. I'd be running all the
automated tests before each commit and before and after the obfuscation
step. I'd just be doing the manual tests (which are a real PIA...) on
the packaged release.

Post by Charlie Poole
This is actually a new argument for something I have talked
about before: don't do acceptance tests in NUnit.

I would use something else, but I haven't found it. Probably because I
haven't looked yet, but NUnit with NUnitForms support has been the best
I've seen so far. There's nothing to say that the acceptance test DLL
has to be part of the main solution, but it is for us. Sometimes we've
found that to reproduce a bug, we have to write a long test first,
narrow down the problem and replicate the component issue with a unit test.

What are options for acceptance testing? I realize this is beyond the
scope of the NUnit community, but I'm still curious what I should be
looking at.

Post by Charlie Poole

Post by Michael Reppy
Our application is complicated by the fact that we have a
compact framework application which is 80% of the code and a
small desktop component that is the remainder, so we've got
two different platforms to target.

Off topic... keep an eye peeled for coming integration between
NUnit and NUnitLite for testing compact framework apps.

Yep, yep, yep, gonna be checking out NUnitLite RSN... :-)

-Mikey Reppy
***@abacalab.com



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

Hi Michael,

Post by Michael Reppy

This is actually a new argument for something I have talked about
before: don't do acceptance tests in NUnit.

I would use something else, but I haven't found it. Probably
because I haven't looked yet, but NUnit with NUnitForms
support has been the best I've seen so far. There's nothing
to say that the acceptance test DLL has to be part of the
main solution, but it is for us. Sometimes we've found that
to reproduce a bug, we have to write a long test first,
narrow down the problem and replicate the component issue
with a unit test.
What are options for acceptance testing? I realize this is
beyond the scope of the NUnit community, but I'm still
curious what I should be looking at.

I lean towards FIT when it's my choice, but I've also used
Exactor with some success. Fit, of course, uses a table
metaphor. Exactor uses a script. I find it's easy to
emulate a script with a table, but not so easy to emulate
a table using a linear script. :-)

Charlie

Post by Michael Reppy

Post by Michael Reppy
Our application is complicated by the fact that we have a compact
framework application which is 80% of the code and a small desktop
component that is the remainder, so we've got two

different platforms

Post by Michael Reppy
to target.

Off topic... keep an eye peeled for coming integration

between NUnit

and NUnitLite for testing compact framework apps.

Yep, yep, yep, gonna be checking out NUnitLite RSN... :-)
-Mikey Reppy
--------------------------------------------------------------
-----------
This SF.net email is sponsored by: Microsoft Defy all
challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Nunit-users mailing list
https://lists.sourceforge.net/lists/listinfo/nunit-users

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Brad Stiles]

This is actually a new argument for something I have talked about
before: don't do acceptance tests in NUnit.

I'm just starting to set up a FitNesse server now, so maybe that will
lead me to agree with you, but I'm curious, what leads you to say this?
I've used it for that task, with some success.

Brad


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

Hi Brad,

Post by Brad Stiles

This is actually a new argument for something I have talked about
before: don't do acceptance tests in NUnit.

I'm just starting to set up a FitNesse server now, so maybe
that will lead me to agree with you, but I'm curious, what
leads you to say this?
I've used it for that task, with some success.

I've also done it when I had to, with "some" success. It's
just not as good as it could be, IMO, for several reasons.

1) Acceptance/Customer tests are supposed to be understandable
by the customer. NUnit displays a tree of class and method
names, marked with a green (we hope) icon. No matter how clearly
we name the tests, its still just a report of names. For some
shops, I've created html reports of the test results, but again,
it's still just a list of names.

2) The exercise of sitting with a customer to define and
refine a FIT test is so valuable that it the interaction
suddenly became unnecessary due to advances in technology,
we would have to invent another reason to do it.

3) In some domains, tests need to be transparent, in the sense
that given enough time one could take manually verify that the
inputs match the outputs. Even where this is not required, I
have found that customers tend to like this approach. FIT
allows me to set up a display in that fashion.

4) Changing to a different tool helps programmers to change
their mindset - which they should do when they are working
on customer tests.

5) Most acceptance-level tests require execution of a continuous
script, in which each step depends on the previous. By design,
NUnit does not support this, although some people have
created extensions to enable it.

6) Because low-level programmer testing requires a special
environment, platforms like NUnit provide it. There are
many aspects of that environment which cause the code
being tested to work differently than it does in the
actual app.(*) The more sophisticated a framework
like NUnit becomes, the more differences there will be.
Acceptance tests require an environment identical to that
in which the tests will be run.

Charlie

* I know you'll ask me for an example. :-) Here's one:
There are APIs in .NET which only work in the primary
AppDomain. NUnit runs tests in a secondary AppDomain.
Q.E.D.



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Brad Stiles]

Post by Charlie Poole
Hi Brad,

Post by Brad Stiles

This is actually a new argument for something I have talked about
before: don't do acceptance tests in NUnit.

I'm just starting to set up a FitNesse server now, so maybe
that will lead me to agree with you, but I'm curious, what
leads you to say this? I've used it for that task, with some
success.

I've also done it when I had to, with "some" success. It's
just not as good as it could be, IMO, for several reasons.

Well, you know what they say about only having a hammer...

Post by Charlie Poole
1) Acceptance/Customer tests are supposed to be understandable
by the customer.

Well, I haven't yet gotten into really involved user tests; our codebase
won't support those anyway, it's so convoluted. The ones I've done so
far are pretty simple, being applied to some new code that I can get
away with doing TDD on. :) More involved ones, which would likely kick
the "too complicated for users" filter, will hopefully be imlemented as
FIT tests (or whatever might be appropriate; FIT/FitNesse was one I'd
heard and read good things about, so it gets to be first).

Post by Charlie Poole
2) The exercise of sitting with a customer to define and
refine a FIT test is so valuable that it the interaction
suddenly became unnecessary due to advances in technology,
we would have to invent another reason to do it.
4) Changing to a different tool helps programmers to change
their mindset - which they should do when they are working
on customer tests.

I find these potentially some of the most compelling reasons in your
list.

Post by Charlie Poole
3) In some domains, tests need to be transparent, in the sense
that given enough time one could take manually verify that the
inputs match the outputs. Even where this is not required, I
have found that customers tend to like this approach. FIT
allows me to set up a display in that fashion.

I like the idea that if the user finds some edge case in an existing
test, they can test it themselves without bothering me unless it doesn't
work. :) And although the FitNesse documentation has been somewhat
difficult for me to understand, I've found enough guides and examples
(the one from Gojko Adzic being especially helpful), that clarify things
for me that I think I can teach people who want to add their own test
scenarios.

I think one of the biggest advantages for me is that something like
FIT/FitNesse is going to let me test the seams between my different
layers without resorting to extraordinary measures trying to mock out
things like databases and such.

Post by Charlie Poole
There are APIs in .NET which only work in the primary
AppDomain. NUnit runs tests in a secondary AppDomain.
Q.E.D.

Actually, I was thinking of examples as I read your note. :)

Thanks for the response,
Brad


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Brad Stiles]

Post by Charlie Poole
5) Most acceptance-level tests require execution of a continuous
script, in which each step depends on the previous. By design,
NUnit does not support this, although some people have
created extensions to enable it.

It struck me after sending the previous one that this type of thing
could be made *much* easier with your proposed add-in architecture for
NUnit 3.0.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Charlie Poole]

Hi Brad,

Post by Charlie Poole
5) Most acceptance-level tests require execution of a continuous
script, in which each step depends on the previous. By

design, NUnit

Post by Charlie Poole
does not support this, although some people have created

extensions to

Post by Charlie Poole
enable it.

It struck me after sending the previous one that this type of
thing could be made *much* easier with your proposed add-in
architecture for NUnit 3.0.

Yes, several people have worked on doing this with the existing
addin architeture. It's not impossible, but there are issues, some
of which led me to think we needed a more robust architecture.

Charlie



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[Cory Foy]

Post by Michael Reppy
It seems if I rearrange how we are handling strong naming issues and set
up the solution to deploy the application dlls and test dlls to one
folder I can obfuscate the application dlls but not the test dlls and
still have them linked and the tests continue to have their highly
descriptive names but the stack traces show that they are testing the
obfuscted dlls.

That's what I was going to recommend. It will rename the calls to the
methods in the CUT, but not your test names. I suppose as long as if you
run into problems you only make changes to the unobsfucated code, then
you'll be good.

Post by Michael Reppy
That all seemed to work fine. The only potentially fatal flaw I found
is with the strong naming issue, something I don't entirely understand
in .net, particularly the differences between reflection on the desktop
and handheld. The help & support from the Dotfuscator people seemed to
have a process for applying strong names after the fact, which I can
follow after I spend some time figuring out what's going on.

What's the problem you are running into? It sounds like you have a
couple of things:

- Reflection /is/ different on the CF. I'm not sure what you need it
for, other than the tests, which Charlie has already mentioned the
upcoming NUnitLite stuff.

- Strong naming is needed to sign the code, and you can use various
tactics (like delay signing) depending on what your needs are. Of
course, you can't sign the code until it is obsfucated (AFAIK), but I'm
not sure what problems you are running into.

I can email some of the internal dev lists at my work to see how people
get around the problems you are running into if I can get a clear
definition of what they are.
--
Cory Foy
http://www.cornetdesign.com



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/