Neoload 7 0 2

1. Neoload 7 0 20

Usage: neoload workspaces OPTIONS ls use NAMEORID neoload workspaces use 'Default Workspace' Since Neoload Web 2.5 (August 2020), assets are scoped to workspaces. The CLI allows you to choose your workspace at login or with the 'use' sub-command, otherwise the 'Default Workspace' is used. The zones are shared between workspaces. Name Email Dev Id Roles Organization; Benoit Derouet: benoit.derouetneotys.com: Neotys.

This plugin allows you to monitor load tests performed by NeoLoad from Jenkins. NeoLoad test results are combined with the other integration jobs in the Jenkins dashboard. Performance regression issues are raised quickly to make Jenkins integration projects more relevant. For regular jobs or pipeline jobs.

Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:

• the HTML summary of the NeoLoad test result per build available with the Performance Result command
• the JUnit details of every passed or failed test available with the Test Result command---also used in the test result trend graph

See https://www.neotys.com/documents/doc/neoload/latest/en/html/#5769.htm for detailed documentation.

Prerequisites

The NeoLoad plugin, from version 2.1.0 requires:

• Jenkins 1.609.1 or later,
• Java 7 or later.

Collaboration Configuration

The plugin allows to configure a collaboration server like Neotys Team Server (NTS) with the Manage Jenkins > Configure System command.

Build Step

The plugin adds a new build step: Execute a NeoLoad Scenario.

The following steps are required:

• Type the path of the NeoLoad executable.
• Select Local Project or Shared Project and configure it.
• Type the name of the NeoLoad scenario in the Scenario Name field.
• Select Default Report File Names or Custom Report File Names and configure it.

• Select Existing License or Shared License and configure it.

• Select Display Trend Graph: Average Response Time (all pages) to include the Avg. Resp. Time trend graph in Jenkins.
• Select Display Trend Graph: Error Rate % to include the Error Rate trend graph in Jenkins.
• (Recommended) Add as many user-defined graphs as wanted with several curves on each graph.
  

Post-Build Action

The plugin needs the Archive the artifacts post-build action. The regeneration of trends could be triggered. Please archive the artifact before Refresh trends.

Example Trend Graphs

Pipelines Steps

The 'neoloadRun' step in the Jenkins Snippet Generator makes it possible to run a NeoLoad scenario from Jenkins. It also archives the reports and refreshes the graphs.

• Warning: To use the Snippet Generator, the Jenkins project including the job to configure must be compliant with Pipeline as code. For more information, see Pipeline as code.

Once the Jenkins project is selected, the Snippet Generator is accessible with a click on the Pipeline Syntax link.

This plugin provides two steps:

• neoloadRun: to run NeoLoad scenario, archive report and refresh the trends.

• neoloadRefreshTrends: to refresh or change the trends only.

FAQ

Why don't I see any trend graphs?

In order to see trend graphs, please verify:

• The Archive the artifacts post-build action has been added.
• Either Default Report File Names or Custom Report File Names is selected and an xml report is defined.
• At least two executions were run.
• Date and time is synchronized between the Jenkins machine and the build machine.

Known Issues

1. The NeoLoad report file (via artifacts and the Performance Result link) displays a blank page. This affects versions released before NeoLoad 5.2.

Workaround

Use the Jenkins Script Console to disable the sandboxing security by executing the following script. The Script Console is under Jenkins -> Manage Jenkins -> Script Console.

Clear the cache afterwards (hold shift and reload the page).

See https://wiki.jenkins-ci.org/display/JENKINS/Configuring+Content+Security+Policy for more information.

2. The NeoLoad Graphs aren't displayed in the main page of my job.

Workaround

Make sure you used a 'Freestyle project' for your job. If you use (for example) the Maven Plugin for your job, create a 'Freestyle project' then add Maven configuration build step.

The following steps are required:

• Type the path of the NeoLoad executable.
• Select Local Project or Shared Project and configure it.
• Type the name of the NeoLoad scenario in the Scenario Name field.
• Select Default Report File Names or Custom Report File Names and configure it.

• Select Existing License or Shared License and configure it.

• Select Display Trend Graph: Average Response Time (all pages) to include the Avg. Resp. Time trend graph in Jenkins.
• Select Display Trend Graph: Error Rate % to include the Error Rate trend graph in Jenkins.
• (Recommended) Add as many user-defined graphs as wanted with several curves on each graph.
  

Post-Build Action

The plugin needs the Archive the artifacts post-build action. The regeneration of trends could be triggered. Please archive the artifact before Refresh trends.

Example Trend Graphs

Pipelines Steps

The 'neoloadRun' step in the Jenkins Snippet Generator makes it possible to run a NeoLoad scenario from Jenkins. It also archives the reports and refreshes the graphs.

• Warning: To use the Snippet Generator, the Jenkins project including the job to configure must be compliant with Pipeline as code. For more information, see Pipeline as code.

Once the Jenkins project is selected, the Snippet Generator is accessible with a click on the Pipeline Syntax link.

This plugin provides two steps:

• neoloadRun: to run NeoLoad scenario, archive report and refresh the trends.

• neoloadRefreshTrends: to refresh or change the trends only.

FAQ

Why don't I see any trend graphs?

In order to see trend graphs, please verify:

• The Archive the artifacts post-build action has been added.
• Either Default Report File Names or Custom Report File Names is selected and an xml report is defined.
• At least two executions were run.
• Date and time is synchronized between the Jenkins machine and the build machine.

Known Issues

1. The NeoLoad report file (via artifacts and the Performance Result link) displays a blank page. This affects versions released before NeoLoad 5.2.

Workaround

Use the Jenkins Script Console to disable the sandboxing security by executing the following script. The Script Console is under Jenkins -> Manage Jenkins -> Script Console.

Clear the cache afterwards (hold shift and reload the page).

See https://wiki.jenkins-ci.org/display/JENKINS/Configuring+Content+Security+Policy for more information.

2. The NeoLoad Graphs aren't displayed in the main page of my job.

Workaround

Make sure you used a 'Freestyle project' for your job. If you use (for example) the Maven Plugin for your job, create a 'Freestyle project' then add Maven configuration build step.

Changelog

Version 2.2.6 (released October 03, 2019)

FIXED: Passwords storage has been changed for security reasons.

Version 2.2.5 (released May 20, 2019)

• IMPROVED: Allow usage of YAML or JSON local project.

Version 2.2.4 (released July 26, 2018)

• FIXED: Graph with ' character in name was not available.

Version 2.2.3 (released July 25, 2018)

• IMPROVED: The log on NeoLoad error.

Version 2.2.2 (released July 12, 2018)

• FIXED: Confusion of launching method when Master and Slave are running in different operating system.

Version 2.2.1 (released July 10, 2018)

• FIXED: Process is interrupted on Windows Slave.

Version 2.2.0 (released June 26, 2018)

• ADD: Pipeline as code support

• ADD: SAP license capability

Version 2.1.1 (released May 23, 2018)

• FIXED: Runtime Exception 'Issue executing password scrambler' was thrown when NeoLoad path executable could not be found. (#11841)

Version 2.1.0 (released February 1, 2018)

• IMPROVED: The trends are now generated at end of Neoload Job with new post-build Action

Version 2.0.2 (released March 17, 2017)

• FIXED: Randomly, the password scrambler could not be executed, so usage of shared project of shared license could fail. (#10743)
• FIXED: Avoid scanning result files if project does not have NeoLoad configuration.

Version 2.0.1 (released September 1, 2016)

• FIXED: Job could not be executed if Jenkins master and slave were on different OS. (#10588)
• FIXED: Job configuration warned about missing executable and nlp file on master even though it could be executed on slave. (#10635)
• FIXED: Graphs trend did not work when job was executed on Windows. (#10629)
• IMPROVED: To install the plugin, Java version 8 was required. Now Java version 7 and higher are supported. (#10610)

Version 2.0.0 (released June 20, 2016)

• FIXED: When creating a new Neoload job there is no choice selected by default for Report File Details. (#10272)
• FIXED: PDF report option is not kept in Project details after a save or apply. (#10253)
• FIXED: Exception: Fail to convert sharedProject. (#10278)
• FIXED: When using NTS server, job failed due to NTS password 'PASSWORD' . Input length must be multiple of 16 when decrypting. (#10248)
• FIXED: Label have to be reviewed for Neoload servers. (#10246)
• FIXED: Project details field ordering need to be review for better understanding. (#10244)
• FIXED: Help mention VU ( User path) when field label mention User Path (VU). (#10243)
• FIXED: Even Customize Report ... settings are not set they are filled in executed command. (#10236)
• FIXED: Jenkins 2.0: Deleting a server from the main config page makes the job fail. (#10260)
• FIXED: Job can't be run if there is no Test description: Invalid argument ', Expected value of type 'STRING'. (#10235)
• FIXED: Use new NL logo for link to Performance result. (#10249)

Version 1.0.2 (released February 9, 2016)

• FIXED: HTML report files sometimes didn't display. (#9257)

Version 1.0.1 (released February 6, 2014)

• FIXED: Performance results links were lost when a job was renamed. (#5308)
• FIXED: Performance results linked to a report artifact that was not related to the current build after renaming a job. (#5309)
• FIXED: Displayed graph values did not match displayed html report values. (#5813)

Version 1.0 (released November 5, 2013)

• Initial release.

This command-line interface helps you launch and observe performance tests on the Neotys Platform. Since NeoLoad is very flexible to many deployment models (SaaS, self-hosted, cloud or local containers, etc.), configuration and test execution parameters depend on your licensing and infrastructure provisioning options. Please read the following instructions carefully.

• Additional Options

Prerequisites

The examples below assume that you have Python3 and Git command line tools installed.

For Windows 10 users, see:

• Installing Python in WindowsIn short:
  • Just install via Python.org Downloads, then
  • Open a command prompt and install pip:
• Install Docker with Chocolatey
  • Open a command prompt and install Docker Desktop for Windows 10

For Mac OS X:

For Docker builds:

Installation

To install, simply run the following command. As of Jan 2020, Python 2 will be permanently deprecated, therefore this utility is written for Python 3.

Interactive CLI Help

For usage and CLI argument examples, simply run:

Configuration

Before running a load test, you will need to initialize a profile to contain connection information to NeoLoad Web. Subsequent commands will use this profile unless switched to another profile, as there can be multiple. This is so that you only need to specify these details once only, then you can focus on validating and running load tests.

Your profile details depend on your license model and how you want to obtain load testing resources (controller + load generators). See below:

Create a SaaS-Based License Profile

If you have an enterprise license activated in NeoLoad Web SaaS, you do not need to specify license details in your profile. When a controller begins to run a test, it will reach out to NeoLoad Web SaaS to lease an appropriate license.

Create a Self-Hosted Enterprise License Profile

If you do not have a SaaS-based license, you will need to specify additional licensing server url and credentials that refer to your own NeoLoad Team License Server.

Viewing saved profiles

You can always list profiles that have saved using the following command:

You can also view the raw/complete stored JSON representation of saved profile details using the following command:

Future Plans to Execute Local Tests on a Free Trial License

If you do not already have an enterprise license, such as if you only have NeoLoad installed on your local workstation (from a Free Trial download or professional license), NeoLoad CLI will eventually support using your installation as an attached controller and load generator.

Additionally, if you do not have access to a dynamic infrastructure provider (i.e. OpenShift, etc.) and if you do not have Docker installed where the NeoLoad CLI is executed, your local workstation can serve as load testing resources similar to dynamic and containerized resource use cases.

This is planned to be delivered in Dec 2019.

Running Load Tests

Once a profile is established, NeoLoad CLI makes it very easy to execute load tests. All you need is to provide an existing test suite or set of as-code file(s), then specify a scenario.

NeoLoad CLI defaults to using the NeoLoad Web APIs for Runtime operations, which means that your project assets will be zipped up together and uploaded to the NeoLoad Web deployment you specified in your profile (SaaS or self-hosted).

Once a test is initialized, if you are running in interactive console mode, the NeoLoad CLI will automatically open the system default browser to your live test results.

Obtain Basic Examples

Some basic examples are in our Git repository for this utility, under the directory ./tests/. To get them, simply clone the repo:

Then, underneath this directory, you can run them simply by typing in:

Additionally, you can specify multiple files, such as additional SLA, variables, or servers overriding files. This works for both [.nlp] and [.yaml] files.

Non-blocking Execution Workflow

You may want to manage individual steps of attaching resources, running, waiting, and detatching resources in a parallel pipeline. This is particularly useful in combination with other parallel steps that dynamically analyze SLA data and fast-fail the test (using API commands) using a known custom real-time analysis process.

The general process can be seen in the NeoLoad CLI E2E PyTest suite, but also abstracted below:

• Initialize a profile

• (Optional for static zone) Attach dynamic resources

  (Note: this assumes that the host you run the above command is a Docker host or is connected to one, DinD)

• Kick off a test via NeoLoad Web Runtime

• Grab the ID of the test just executed

• (Start some other parallel operations, such as monitoring for early advanced SLA failures)

• Wait (blocking) for test to complete

• (Always) Detatch Optional Docker resources

Additional Options

There are many other arguments for test summarization, modification, and exporting results.

Exporting SLA Results to JUnit

Verbose and Debug Mode

Verbose mode changes logging level to INFO. When executing from an interactive console, it also opens a browser window to the test logs immediately after they're available (useful for monitoring the infrastructure initialization process).

Debug mode changes logging level to DEBUG. This is an extreme amount of internal information which infers verbose/INFO mode and provides manual pausing on critical events (such as after Docker attach but before test execution).

Quiet Mode

Adding the --quiet flag generally produces only the relevant output (i.e. JSON, test id, etc.) and no other informational messages. This is useful when piping the structured data output into another process, such as obtaining the most recent test id from stdout file (--quiet is inferred by some other combinations of flags such as --infile AND --query testid).

Test Summary

Specifying both a --testid value and the --summary flag produces a JSON result set that shows both the high level test status metadata and overall statistics.

Produces:

TODO: This summary will also include SLA summaries in Jan 2020

Test Modifications

Metadata on a test can be modified after the test is complete, such as name, description, and status. With the required --testid argument as well, these modification arguments can be used individually or in combination with each other.

Additionally, plus and minus (+/-) operators can be used to append or remove the text specified from the existing name or description data. This is useful in combination with hashtag keywords to flag a test as baseline, candidate, or with an issue tracking id.

Test Result listings and queries

TODO: will be coming in Jan 2020

Contributing

Feel free to fork this repo, make changes, test locally, and create a pull request. As part of your testing, you should run the built-in test suite with the following command:

Neoload 7 0 20

*NOTE: omitting the --skipslow and --skipslas arguments also runs Docker-related attaching tests, which you will need to set environment variables up for in order to successfully run the test suite. An example of these variables can be found in example.bash_profile and can be addapted for Windows execution as well.