Integrate smoke tests for post deployment
While we can run smoke tests by using the PostDeploy.ps1, such as running nunit.exe or something similar (frisby.js, etc), it would be great to have a way to integrate back into the dashboard (through the rest api?).
Related links:
http://help.octopusdeploy.com/discussions/questions/214-running-smoke-tests-after-deploy
http://help.octopusdeploy.com/discussions/problems/53646
http://jimblogdog.blogspot.co.uk/2012/08/walk-thru-using-wolfpack-to.html
Thanks for the suggestion and for voting everyone. I’d like to get feedback on where we should go with this.
There are a few things in Octopus that already help with this – first, it’s possible to package your NUnit tests and run them as part of a package test (we do this ourselves), or to write a PowerShell step that pings your website to see if it is online. So the actual process of performing the tests is pretty easy. Do we need to write more blog posts/documentation to show how it can be done?
The next part is viewing the test results – in Octopus 2.0 you can “attach” files to a deployment via PowerShell which are then uploaded and available on the deployment page.
Viewing NUnit results XML isn’t that nice though, so I guess it would be nice to be able to render those NUnit test results in the browser, or to add ‘badges’ to a deployment (such as the number of tests passed). Is that all that is missing to mark this feature as “done”, or is there more we would need to do?
Thanks again for the suggestion!
-
Anonymous commented
great post
-
Matt Frear commented
I can't believe here we are in 2018 and this hasn't been done yet
-
Paul Farry commented
My thoughts based on the previous comment : 2017-09-30 below, I'd like to see some kind of new component of the system. Something like Project Steps that can be attached to a project to "Verify" either at the project level or the Step level.
* Check 200 for PayLoad
* Post {data} to Address confirm ReponseCode
* Run query .. expect N Records (GIGO principal on the query)
* Check for FileThen having the same kind of thing as step templates when I can make my own and have them either part of our Octopus server for others to consume or Community templates.
-
Anonymous commented
Having the ability to integrate testing results directly into the Octopus UI would be powerful and make octopus a truly enterprise grade deployment tool. Writing powershell or C# testing scripts is not ideal and adds complexity to the whole release process.
-
Daniel Powell commented
Any movement on this?
-
Paul Shmakov commented
I'm considering using Octopus Deploy as an integration testing tool. The pipeline is the following: a build is taken from TeamCity, Octopus deploys it on a number of VMs, the integration tests are executed, depending on the execution results, the build is tagged in TeamCity as tested or not.
As I see it now, the really missing piece of the puzzle is showing test results in the deployment log. We can attach a file, but something similar to test results view in TeamCity would make OD very integration/smoke tests friendly.
-
Simon Wears commented
Smoke tests don't really sit within the scope of Octopus - however I've got this working in conjunction with TeamCity. TeamCity is responsible for taking the source code, running unit tests, packaging and sending build artifacts to Octopus. Octopus then deploys the website, and after deployment runs a powershell script that triggers another TeamCity build. This new TC build is responsible for running the test script (in my case it's selenium based functional testing), and returns a non-zero code if the tests fail. This means that the deployment is 'paused' while tests are executed. In the future I'm going to look into TeamCity generating a useful test report which can then be automatically uploaded as an Octopus deployment artifact.
Any build specific stuff (such as changesets, etc) are done in TeamCity just before packaging, when TC will generate a PostDeploy.ps1 script that echos octopus variables I can access during the deployment.
It's not a perfect solution but it does work, and means fully automated tests and deployments.
Powershell Script:
https://gist.github.com/munkyjunky/44fba7ab7b36c4dbeeab95dec2c125a3 -
Matthew Colter commented
There's already some pretty cool scripting capabilities utilizing powershell and C#. A build step could be defined to run the NUnit xml results through third party tools like NUnit Orange, so the transformation of test result output is probably something beyond the scope of Octopus. Though, the ability to view those results in the dashboard would be very efficient. Maybe there's already a way to add a clickable link to a build step, that would point to wherever the test results were hosted? Another option would be to allow the test results to be uploaded and add some navigation to the uploaded page to the dashboard, but this introduces a security issue where one might generate html pages to hijack the system admin's account. Yet another option would be to allow for custom results parsers like Jenkins or Hudson, the community could build parsing solutions and Octopus could host them. This last one would be the most complicated because of licensing issues and development time, but it would also be the most awesome... We just create a step to run the test and output the results where Octopus can find them, then define the parser to transform results into some beautiful html page that the task results can point to and see it right there in the dashboard.
The solution I'm using now is to create my own executable that runs the NUnit tests, passes the results through NUnit Orange, and puts both files out to a directory being hosted by a simple IIS site. This is cool and all, but it's not integrated with the dashboard. People have to see the deploy fail, then navigate to the internal site and find the results themselves. They seem pretty excited about the idea of being able to just click a link and find the results right in front of them instead. So I'm here. :D
-
Anonymous commented
A blog detailing how to run Nunit and incorporate the results in the dasboard would be great.
Also failing the pipeline if the test dont run, so a promotion is not possible. -
Anonymous commented
I agree, i would appreciate to see an example of it.
-
Gary Ewan Park commented
I would love to see more blog posts on how this can be achieved as this is something that I would like to do as well, but I have not been able to put the pieces together yet. Is it possible to share the configuration, process steps that you use internally somewhere so we can see how to do it. Does anyone have any links that they can share?
-
Yahya commented
Hi - has there been any more official blog posts/documentation added to show how integration testing is properly done using octopus deploy ? We are using OD v2.4.7.85
-
Raffaele Garofalo commented
In our company we use Lab Manager (MTM).
We have a CI build that creates a Nuget package and then call Octopus to create a release in our "Staging" environment.
At the end of the build, MTM runs the tests on top of the release and then we update the build output with the test results. Quite easy to achieve with Octopus + TFS + MTM -
Robert Gibbens commented
1. More blog posts/documentation is ALWAYS welcome. :)
2. Attaching the test results would be fine in my opinion
3. Badges, or some UI indicator of Pass/Fail is really the key ingredient here. Maybe in the dashboard, color the deployment as yellow to indicate that it was deployed, but tests failed? -
Wayne Brantley commented
We need an easy way to get the 'badges' showing success/failure from the result of PS scripts or nunit, etc. Also, once implemented, a good blog post would be helpful.
-
Rudi commented
Yeah exactly, I think a means to integrate the xml with the browser would be ideal. Reading the xml is kind of a drag, as is using a viewer to check out the results. Your last paragraph basically hit it dead on the head, for each deployment having a list, whether that's badges or something else, of what tests passed (and which failed) is what I was seeking when I initially voted on the thread. Thanks for the response!
-
Wayne Brantley commented
This would be great. I think some sort of basic support for websites should be included. Just give OD a url and it hits it looks for a 200 result. That would be a place to start. Full UI testing would be more like what you specified above, but I would like just a built in 'sanity check'.
-
Matt C commented
We run our selenium ui tests using mstest.exe as part of deploy.ps1, we then fail the deployment if the tests fail. Finaly we hoover up the log files so you can see them in the octopus ui under output.