Tags

Tags give the ability to mark specific points in history as being important

0.9.0

797b1735 · Merge "add release notes for 0.9.0" · Mar 20, 2017

Rally v0.9.0
============

Overview
--------

+------------------+-----------------------+
| Release date     |      **3/20/2017**    |
+------------------+-----------------------+

Details
-------

Command Line Interface
~~~~~~~~~~~~~~~~~~~~~~

* `rally plugin list` now does not contain hidden plugins.

Task component
~~~~~~~~~~~~~~

* Added check for duplicated keys in task files.

* The order of subtasks (scenarios/workloads) is not ignored any more. You can
  generate whatever you want load or use that feature for up the cloud (put
  small scenario to the start of task to wake up the cloud before the real
  load).

* Information about workload creation is added to HTML-reports.

* Task statuses is changed to be more clear and cover more cases:

 - ``verifying`` is renamed to ``validating``.
 - ``failed`` is divided for 2 statuses - ``validation_failed``, which means
   that task did not pass validation step, and ``crashed``, which means that
   something went wrong in rally engine.

* Our awesome cleanup become more awesome! The filter mechanism is improved to
  discover resources in projects created only by Rally (it works for most of
  resources, except several network-related ). It makes possible to run Rally
  with existing users in real tenants without fear to remove something
  important.

Verification component
~~~~~~~~~~~~~~~~~~~~~~

* Fixed an issue with missed tests while listing all supported tests of
  specified verifier.

* Fixed an issue with displaying the wrong version of verifier in case of
  cloning from the local directory.

* Extend `rally verify rerun
  <http://rally.readthedocs.io/en/0.9.0/verification/cli_reference.html#rally-verify-rerun>`_
  with ``--detailed``, ``--no-use``, ``--tag`` and ``--concurrency`` arguments.

* Add output examples for `JSON
  <http://rally.readthedocs.io/en/0.9.0/verification/reports.html#json>`_ and
  `JUnit-XML
  <http://rally.readthedocs.io/en/0.9.0/verification/reports.html#junit-xml>`_
  reporters.

Plugins
~~~~~~~

**Contexts**

* Extend cinder quotas to support ``backups`` and ``backup_gigabytes``.

**Deployment Engines**:

*Updated* Extend `DevstackEngine
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#devstackengine-engine>`_
with ``enable_plugin`` option.

**OpenStack clients**:

* Extend support for auth urls like ``https://example.com:35357/foo/bar/v3``

* Pass endpoint type to heatclient

**Scenarios**:

* *NEW!!*

 - `CinderVolumeTypes.create_and_delete_encryption_type
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#cindervolumetypes-create-and-delete-encryption-type-scenario>`_

 - `CinderVolumeTypes.create_and_set_volume_type_keys
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#cindervolumetypes-create-and-set-volume-type-keys-scenario>`_

 - `KeystoneBasic.create_and_list_roles
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#keystonebasic-create-and-list-roles-scenario>`_

 - `KeystoneBasic.create_and_update_user
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#keystonebasic-create-and-update-user-scenario>`_

 - `NovaKeypair.create_and_get_keypair
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novakeypair-create-and-get-keypair-scenario>`_

 - `NovaServers.resize_shutoff_server
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#novaservers-resize-shutoff-server-scenario>`_

 - `VMTasks.dd_load_test
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#vmtasks-dd-load-test-scenario>`_

* *UPDATED!!*

 - Extend `VMTasks.boot_runcommand_delete
   <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#vmtasks-boot-runcommand-delete-scenario>`_
   to display just raw text output of executed command.

* *DELETED*

  Scenario `VMTasks.boot_runcommand_delete_custom_image
  <http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#vmtasks-boot-runcommand-delete-custom-image-scenario>`_
  is removed since `VMTasks.boot_runcommand_delete
  <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#vmtasks-boot-runcommand-delete-scenario>`_
  covers the case of that particular scenario without adding any complexity.

**Validators**:

* Extend ``required_contexts`` validator to support ``at least one of the``
  logic.

* Fix a bunch of JSON schemas which are used for validation of all plugins.

Documentation
~~~~~~~~~~~~~

We totally reworked `Plugins Reference
<http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html>`_ page.
Now it looks more like `Command Line Interface
<http://rally.readthedocs.io/en/0.9.0/cli_reference.html>`_, which means that
you can get links for particular parameter of particular plugin.

Also, you can find expected parameters and their types of all contexts, hooks,
SLAs and so on! Most of them still miss descriptions, but we are working on
adding them.

Fixed bugs
~~~~~~~~~~

* [osclients] Custom auth mechanism was used for zaqarclient instead of unified
  keystonesession, which led to auth errors at some envs.

* [plugins] During running
  `CinderVolumes.create_and_restore_volume_backup
  <http://rally.readthedocs.io/en/0.9.0/plugins/plugin_reference.html#cindervolumes-create-and-restore-volume-backup-scenario>`_
  scenario we had a race problem with backup deleting due to wrong check of
  backup status.

* [plugins][verifications] Jenkins expexts "classname" JUnitXML attribute
  instead of "class_name".

Thanks
~~~~~~

 2 Everybody!

Unverified

0.8.1

2ff6423b · Release 0.8.1 · Jan 28, 2017

Rally v0.8.1
============

Overview
--------

+------------------+-----------------------+
| Release date     |      **1/27/2017**    |
+------------------+-----------------------+

Details
-------

Fix for python requirements list.

Plugins
~~~~~~

**Scenarios**:

* *Updated*:
 - Use new network for each subnet at
   `NeutronNetworks.create_and_list_subnets
    <http://rally.readthedocs.io/en/0.8.1/plugins/plugin_reference.html#neutronnetworks-create-and-list-subnets-scenario>`_
   scenario.

* *NEW!!*:
 - `CinderVolumeTypes.create_and_list_encryption_type
   <http://rally.readthedocs.io/en/0.8.1/plugins/plugin_reference.html#cindervolumetypes-create-and-list-encryption-type-scenario>`_
 - `Quotas.cinder_get
   <http://rally.readthedocs.io/en/0.8.1/plugins/plugin_reference.html#quotas-cinder-get-scenario>`_
Thanks
~~~~~

 2 Everybody!

Unverified

0.8.0

ed15d27b · Merge "[doc][verify] Some more changes in docs before release" · Jan 26, 2017

Rally v0.8.0
============

Overview
--------

+------------------+-----------------------+
| Release date     |      **1/25/2017**    |
+------------------+-----------------------+

Details
-------

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* `[Implemented] Refactor Verification Component
  <https://github.com/openstack/rally/blob/0.8.0/doc/specs/implemented/verification_refactoring.rst>`_

* `[Implemented] Scaling & Refactoring Rally DB
  <https://github.com/openstack/rally/blob/0.8.0/doc/specs/implemented/db_refactoring.rst>`_

Installation
~~~~~~~~~~~~

We switched to use bindep library for checking required system packages.
All our dependencies moved to separate file (like requirements.txt for python
packages) `bindep.txt
<https://github.com/openstack/rally/blob/0.8.0/bindep.txt>`_.

Database
~~~~~~~~

.. warning:: Database schema is changed, you must run
     `rally-manage db upgrade <http://rally.readthedocs.io/en/0.8.0/cli/cli_reference.html#rally-manage-db-upgrade>`_
     to be able to use old Rally installation with latest release.
* change structure of database to be more flexible
* save raw task results in chunks (see raw_result_chunk_size option of
  [DEFAULT] rally configuration section)
* add db revision check in rally API, so it is impossible to use rally with
  wrong db now.

Rally API
~~~~~~~~~

Single entry point for Rally API is added - rally.api.API . Old API classes
(``rally.api.Task``, ``rally.api.Verification``, ``rally.api.Deployment``) are
deprecated now.

Rally CLI
~~~~~~~~~

* ``rally task sla_check`` is deprecated now in favor of
  ``rally task sla-check``

* Deprecated category ``rally show`` was removed.

* `rally plugin list` is extended with plugin base column

Task Component
~~~~~~~~~~~~~~

- [Random names] scenario for checking performance of generate_random_name
  method is added to our CI with proper SLA. Be sure, whatever number of random
  names you need, it will not affect performance of Rally at all, we checked.

- [atomic actions] scenario for checking performance of calculating atomic
  actions is added to our CI with proper SLA. Be sure, whatever number atomics
  you have in scenarios, it will not affect performance of Rally at all, we
  checked.

- [services] new entity is introduced for helping to provide compatibility
  layer between different API versions of one service.

Verification component
~~~~~~~~~~~~~~~~~~~~~~

We completely redesign the whole Verification component. For more details see
`our new docs for that component
<http://rally.readthedocs.ioen/0.8.0/verification/index.html>`_

Unfortunately, such big change could not be done in backward compatible way,
so old code is not compatible with new one. See `HowTo migrate from
Verification component 0.7.0 to 0.8.0
<http://rally.readthedocs.ioen/0.8.0/verification/howto/migrate_from_old_design.html>`_

Plugins
~~~~~~~

**Services**:

* Glance:

  Switched from V1 to V2 API by default.

* Keystone:

 - Transmit endpoint_type to keystoneclient
 - Full keystone V3 support

**Scenarios**:

* *Updated*:
 - The meaning of the volume_type argument is changes in
   `CinderVolumes.create_snapshot_and_attach_volume
    <http://rally.readthedocs.io/en/0.8.0/plugins/plugin_reference.html#cindervolumes-create-snapshot-and-attach-volume-scenario>`_
   scenario. It should contain actual volume type instead of boolean value to
   choose random volume type.
 - Extend `GlanceImages.create_image_and_boot_instances
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#glanceimages-create-image-and-boot-instances-scenario>`_
   with create_image_kwargs and boot_server_kwargs arguments.

* *NEW!!*:
 - `CeilometerAlarms.create_and_get_alarm
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#ceilometeralarms-create-and-get-alarm-scenario>`_
 - `CinderVolumeBackups.create_incremental_volume_backup
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumebackups-create-incremental-volume-backup-scenario>`_
 - `CinderVolumeTypes.create_and_delete_volume_type
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumetypes-create-and-delete-volume-type-scenario>`_
 - `CinderVolumeTypes.create_volume_type_and_encryption_type
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumetypes-create-volume-type-and-encryption-type-scenario>`_
 - `CinderVolumes.create_and_accept_transfer
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumes-create-and-accept-transfer-scenario>`_
 - `CinderVolumes.create_and_get_volume
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumes-create-and-get-volume-scenario>`_
 - `CinderVolumes.create_volume_and_update_readonly_flag
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumes-create-volume-and-update-readonly-flag-scenario>`_
 - `CinderVolumes.list_transfers
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumes-list-transfers-scenario>`_
 - `CinderVolumes.list_types
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#cindervolumes-list-types-scenario>`_
 - `KeystoneBasic.create_and_get_role
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#keystonebasic-create-and-get-role-scenario>`_
 - `ManilaShares.create_and_list_share
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#manilashares-create-and-list-share-scenario>`_
 - `ManilaShares.set_and_delete_metadata
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#manilashares-set-and-delete-metadata-scenario>`_
 - `MistralExecutions.create_execution_from_workbook
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#mistralexecutions-create-execution-from-workbook-scenario>`_
 - `MistralExecutions.list_executions
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#mistralexecutions-list-executions-scenario>`_
 - `NeutronLoadbalancerV2.create_and_list_loadbalancers
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#neutronloadbalancerv2-create-and-list-loadbalancers-scenario>`_
 - `NeutronNetworks.create_and_show_network
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#neutronnetworks-create-and-show-network-scenario>`_
 - `NeutronNetworks.list_agents
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#neutronnetworks-list-agents-scenario>`_
 - `NovaAggregates.create_aggregate_add_host_and_boot_server
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novaaggregates-create-aggregate-add-host-and-boot-server-scenario>`_
 - `NovaAggregates.create_and_get_aggregate_details
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novaaggregates-create-and-get-aggregate-details-scenario>`_
 - `NovaFlavors.create_and_delete_flavor
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novaflavors-create-and-delete-flavor-scenario>`_
 - `NovaFlavors.create_flavor_and_add_tenant_access
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novaflavors-create-flavor-and-add-tenant-access-scenario>`_
 - `NovaHosts.list_and_get_hosts
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novahosts-list-and-get-hosts-scenario>`_
 - `NovaHypervisors.list_and_get_uptime_hypervisors
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novahypervisors-list-and-get-uptime-hypervisors-scenario>`_
 - `NovaHypervisors.list_and_search_hypervisors
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novahypervisors-list-and-search-hypervisors-scenario>`_
 - `NovaHypervisors.statistics_hypervisors
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novahypervisors-statistics-hypervisors-scenario>`_
 - `NovaSecGroup.boot_server_and_add_secgroups
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novasecgroup-boot-server-and-add-secgroups-scenario>`_
 - `NovaServerGroups.create_and_list_server_groups
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#novaservergroups-create-and-list-server-groups-scenario>`_
 - `Quotas.nova_get
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#quotas-nova-get-scenario>`_

**Hooks**:

* *NEW!!*:
 - `fault_injection
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#fault-injection-hook>`_

**Runners**

* *Updated*:
 - `RPS runner
   <http://rally.readthedocs.ioen/0.8.0/plugins/plugin_reference.html#rps-scenario-runner>`_
   is extended with ability to increase 'rps' value by arithmetic progression
   across certain duration. Now it can be also a dict specifying progression
   parameters:

   .. code-block:: json

        rps": {
            "start": 1,
            "end": 10,
            "step": 1,
            "duration": 2
        }

   This will generate rps value: ``start, start + step, start + 2 * step, ..,
   end`` across certain 'duration' seconds each step. If iteration count not
   ended at the last step of progression, then rps will continue to generate
   with "end" value. Note that the last rps could be generated smaller.

Fixed bugs
~~~~~~~~~~

* [hooks] incorrect encoding of stdout/stderr streams opened by sys_call hook
  for py3

* [hooks] sorting Hook column at HTML report doesn't work

* [tasks][scenarios][neutron] L3 HA: Unable to complete operation on subnet

  `Launchpad bug-report #1562878 <https://bugs.launchpad.net/rally/+bug/1562878>`_

* [tasks] JSON report doesn't save order of atomics

* [tasks][cleanup][nova] Failed to remove aggregate which has hosts in it

* [tasks] `--abort-on-sla-failure
  <http://rally.readthedocs.ioen/0.8.0/cli_reference.html#task-start-abortonslafailure>`_
  mechanism works only for current workload, but does not stop the next ones.

* [hooks] hooks section isn't displayed in HTML report

Thanks
~~~~~~

 2 Everybody!

Unverified

0.7.0

edb093d9 · Merge "Remove pycparser restriction from requirements" · Oct 12, 2016

Rally v0.7.0
============

Overview
--------

+------------------+-----------------------+
| Release date     |     **10/11/2016**    |
+------------------+-----------------------+

Details
-------

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* [Used] Ported all rally scenarios to class base

  `Spec reference <https://github.com/openstack/rally/blob/0.7.0/doc/specs/implemented/class-based-scenarios.rst>`_

* `[Implemented] New Plugins Type - Hook <https://github.com/openstack/rally/blob/0.7.0/doc/specs/implemented/hook_plugins.rst>`_

Database
~~~~~~~~

.. warning:: Database schema is changed, you must run
     `rally-manage db upgrade <http://rally.readthedocs.io/en/0.7.0/cli/cli_reference.html#rally-manage-db-upgrade>`_
     to be able to use old Rally installation with latest release.

* [require migration] fix for wrong format of "verification_log" of tasks
* [require migration] remove admin_domain_name from OpenStack deployments

Rally Deployment
~~~~~~~~~~~~~~~~

* Remove admin_domain_name from openstack deployment
  Reason: admin_domain_name parameter is absent in Keystone Credentials.

Rally Task
~~~~~~~~~~

* [Trends][Reports] Use timestamps on X axis in trends report

* [Reports] Add new OutputTextArea chart plugin

  New chart plugin can show arbitrary textual data on
  "Scenario Stata -> Per iteration" tab.

  This finally allows to show non-numeric data like IP addresses, notes and
  even long comments.

  Plugin `Dummy.dummy_output <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#dummy-dummy-output-scenario>`_
  is also updated to provide demonstration.

* [cli] Add version info to *rally task start* output

* [api] Allow to delete stopped tasks without force=True

  It is reasonable to protect deletion of running tasks (statuses INIT,
  VERIFYING, RUNNING, ABORTING and so on...) but it is strange to protect
  deletion for stopped tasks (statuses FAILED and ABORTED). Also this is
  annoying in CLI usage.

* Added hooks and triggers.

  Hook is a new entity which can be launched on specific events. Trigger is
  another new entity which processes events and launches hooks.
  For example, hook can launch specific destructive action - just execute cli
  command(we have sys_call hook for this task) and it can be launched by
  simple trigger on specific iteration(s) or time (there is event trigger).

Rally Verify
~~~~~~~~~~~~

Scenario tests in Tempest require an image file. Logic of obtaining this image
is changed:

* If CONF.tempest.img_name_regex is set, Rally tries to find an image matching
  to the regex in Glance and download it for the tests.
* If CONF.tempest.img_name_regex is not set (or Rally didn't find the image
  matching to CONF.tempest.img_name_regex), Rally downloads the image by the
  link specified in CONF.tempest.img_url.

Plugins
~~~~~~~

**Scenarios**:

* *Removed*: `Dummy.dummy_with_scenario_output <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#dummy-dummy-with-scenario-output-scenario>`_

  It was deprecated in 0.5.0

  .. warning:: This plugin is not available anymore in 0.7.0

* *NEW!!*:
 - `MagnumClusterTemplates.list_cluster_templates <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#magnumclustertemplates-list-cluster-templates-scenario>`_
 - `MagnumClusters.list_clusters <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#magnumclusters-list-clusters-scenario>`_
 - `MagnumClusters.create_and_list_clusters <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#magnumclusters-create-and-list-clusters-scenario>`_
 - `NovaAggregates.create_aggregate_add_and_remove_host <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-aggregate-add-and-remove-host-scenario>`_
 - `NovaAggregates.create_and_list_aggregates <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-and-list-aggregates-scenario>`_
 - `NovaAggregates.create_and_delete_aggregate <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-and-delete-aggregate-scenario>`_
 - `NovaAggregates.create_and_update_aggregate <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaaggregates-create-and-update-aggregate-scenario>`_
 - `NovaFlavors.create_and_get_flavor <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaflavors-create-and-get-flavor-scenario>`_
 - `NovaFlavors.create_flavor_and_set_keys <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaflavors-create-flavor-and-set-keys-scenario>`_
 - `NovaHypervisors.list_and_get_hypervisors <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novahypervisors-list-and-get-hypervisors-scenario>`_
 - `NovaServers.boot_server_associate_and_dissociate_floating_ip <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#novaservers-boot-server-associate-and-dissociate-floating-ip-scenario>`_
 - `KeystoneBasic.authenticate_user_and_validate_token <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#keystonebasic-authenticate-user-and-validate-token-scenario>`_

**Contexts**:

* *NEW!!*:
 - `Manila manila_security_services <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#manila-security-services-context>`_
 - `Magnum cluster_templates <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#cluster-templates-context>`_
 - `Magnum clusters <http://rally.readthedocs.io/en/0.7.0/plugin/plugin_reference.html#clusters-context>`_

**OSClients**:

Port all openstack clients to use keystone session.

Bug fixes
~~~~~~~~~

* [tasks] rally task detailed incorrect / inconsistent output

  `Launchpad bug-report #1562713 <https://bugs.launchpad.net/rally/+bug/1562713>`_

Thanks
~~~~~~

 2 Everybody!

Unverified

0.6.0

b6bedc5b · Merge "[osclients] Fix zaqar client" · Sep 06, 2016

============
Rally v0.6.0
============

Overview
--------

+------------------+-----------------------+
| Release date     |     **9/05/2016**     |
+------------------+-----------------------+

Details
-------

Common
~~~~~~

* Added Python 3.5 support
* Sync requirements with OpenStack global-requirements
* Start using latest way of authentication - keystoneauth library
* Start porting all scenario plugins to class-based view.

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* `[Implemented] SLA Performance degradation plugin <https://github.com/openstack/rally/blob/0.6.0/doc/specs/implemented/sla_pd_plugin.rst>`_
* `[Proposed] New Tasks Configuration section - hook <https://github.com/openstack/rally/blob/0.6.0/doc/specs/in-progress/hook_section.rst>`_

Database
~~~~~~~~

* disable db downgrade api
* [require migration] upgrade deployment config

Docker image
~~~~~~~~~~~~

* Add sudo rights to rally user
  Rally is a pluggable framework. External plugins can require installation of
  additional python or system packages, so we decided to add sudo rights.

* Move from ubuntu:14.04 base image to ubuntu:16.04 .
  Ubuntu 16.04 is current/latest LTS release. Let's use it.

* pre-install vim
  Since there are a lot of users who like to experiment and modify samples
  inside container, rally team decided to pre-install vim

* configure/pre-install bash-completion
  Rally provides bash-completion script, but it doesn't work without installed
  `bash-completion` package and now it is included in our image.

Rally Deployment
~~~~~~~~~~~~~~~~

* Add strict jsonschema validation for ExistingCloud deployments. All incorrect
  and unexpected properties will not be ignored anymore. If you need to store
  some extra parameters, you can use new "extra" property.
* Fix an issue with endpoint_type.
  Previously, endpoint type was not transmitted to keystone client. In this
  case, keystoneclient used default endpoint type (for different API calls it
  can differ). Behaviour after the fix:

   - None endpoint type -> Rally will initialize all clients without setting
     endpoint type. It means that clients will choose what default values for
     endpoint type use by itself. Most of clients have "public" as default
     values. Keystone use "admin" or "internal" by default.
   - Not none endpoint type -> Rally will initialize all clients with this
     endpoint. Be careful, by default most of keystone v2 api calls do not work
     with public endpoint type.

Rally Task
~~~~~~~~~~

* [core] Iterations numbers in logging and reports must be synchronized. Now
  they start from 1 .

* [config] users_context.keystone_default_role is a new config option
  (Defaults to "member") for setting default user role for new users in case
  of Keystone V3.

* [Reports] Embed Rally version into HTML reports
  This adds Rally version via meta tag into HTML reports:

    <meta name="generator" content="Rally version {{ version }}">

* [Reports] Expand menu if there is only one menu group

* [logging] Remove deprecated rally.common.log module

* [Trends][Reports] Add success rate chart to trends report

* [Reports] Hide menu list if there is no data at all

Rally Verify
~~~~~~~~~~~~

* Updating Tempest config file
 - Some tests (for boto, horizon, etc.) were removed from Tempest and now there
   is no need to keep the corresponding  options in Tempest config file.

 - Some options in Tempest were moved from one section to another and we should
   to do the corresponding changes in Rally to be up to date with the latest
   Tempest version.

* Adding '--skip-list' arg to `rally verify start` cmd

  `CLI argument for --skip-list <http://rally.readthedocs.io/en/0.6.0/cli/cli_reference.html#verify-start-skiplist>`_

* *NEW!!*:

 - `Command for plugin listing <http://rally.readthedocs.io/en/0.6.0/cli/cli_reference.html#rally-verify-listplugins>`_
 - `Command to uninstall plugins <http://rally.readthedocs.io/en/0.6.0/cli/cli_reference.html#rally-verify-uninstallplugin>`_

* Rename and deprecated several arguments for `rally verify start` cmd:

 - tests-file  -> load-list
 - xfails-file -> xfail-list

Plugins
~~~~~~~

**Scenarios**:

* Extend Sahara scenarios with autoconfig param

  Affected plugins:

 - `SaharaClusters.create_and_delete_cluster <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharaclusters-create-and-delete-cluster-scenario>`_
 - `SaharaClusters.create_scale_delete_cluster <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharaclusters-create-scale-delete-cluster-scenario>`_
 - `SaharaNodeGroupTemplates.create_and_list_node_group_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharanodegrouptemplates-create-and-list-node-group-templates-scenario>`_
 - `SaharaNodeGroupTemplates.create_delete_node_group_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#saharanodegrouptemplates-create-delete-node-group-templates-scenario>`_

* *NEW!!*:
 - `MonascaMetrics.list_metrics <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#monascametrics-list-metrics-scenario>`_
 - `SenlinClusters.create_and_delete_cluster <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#senlinclusters-create-and-delete-cluster-scenario>`_
 - `Watcher.create_audit_template_and_delete <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#watcher-create-audit-template-and-delete-scenario>`_
 - `Watcher.create_audit_and_delete <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#watcher-create-audit-and-delete-scenario>`_
 - `Watcher.list_audit_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#watcher-list-audit-templates-scenario>`_

* Rename **murano.create_service** to **murano.create_services** atomic action

**SLA**:

*NEW!!*: `performance degradation plugin <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#performance-degradation-sla>`_

**Contexts**:

* *NEW!!*:
 - `Monasca monasca_metrics <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#monasca-metrics-context>`_
 - `Senlin profiles <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#profiles-context>`_
 - `Watcher audit_templates <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#audit-templates-context>`_

* Extend `manila_share_networks <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#manila-share-networks-context>`_
  context with share-network autocreation support.

* Extend `volumes <http://rally.readthedocs.io/en/0.6.0/plugin/plugin_reference.html#volumes-context>`_
  context to allow volume_type to be None to allow using default value

Bug fixes
~~~~~~~~~

* [existing users]  Quota context does not restore original settings on exit

  `Launchpad bug-report #1595578 <https://bugs.launchpad.net/rally/+bug/1595578>`_

* [keystone v3] Rally task's test user role setting failed

  `Launchpad bug-report #1595081 <https://bugs.launchpad.net/rally/+bug/1595081>`_

* [existing users] context cannot fetch 'tenant' and 'user' details from cloud
  deployment

  `Launchpad bug-report #1602157 <https://bugs.launchpad.net/rally/+bug/1602157>`_

* UnboundLocalError: local variable 'cmd' referenced before assignment

  `Launchpad bug-report #1587941 <https://bugs.launchpad.net/rally/+bug/1587941>`_

* [Reports] Fix trends report generation if there are n/a results

Documentation
~~~~~~~~~~~~~

* Add page about task reports

  `RTD page for reports <http://rally.readthedocs.io/en/0.6.0/reports.html>`_

Thanks
~~~~~~

 2 Everybody!

Unverified

0.5.0

41b26f4a · Merge "[Verify] Improvements in `rally verify installplugin` cmd" · Jul 21, 2016

============
Rally v0.5.0
============

Information
-----------

+------------------+-----------------------+
| Commits          |        **175**        |
+------------------+-----------------------+
| Bug fixes        |        **19**         |
+------------------+-----------------------+
| Dev cycle        |      **93 days**      |
+------------------+-----------------------+
| Release date     |     **7/20/2016**     |
+------------------+-----------------------+

Details
-------

This release took much more time than we expected, but we have a lot of
reasons for such delay and if you look at our change-log, you will understand
them.:)

Here is a quick introduction:

* To make our releases as much as possible stable, we added upper limits for
  each of our requirements;
* A lot of deprecated lines of code were removed, so be careful;
* Statistics trends for given tasks were introduced;
* Support for tempest plugins was added;
* Several new pages at docs.

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* `[Introduced && implemented] Introduce class-based scenario implementation <https://github.com/openstack/rally/blob/0.5.0/doc/specs/implemented/class-based-scenarios.rst>`_

* `[Introduced] Rally Task Validation refactoring <https://github.com/openstack/rally/blob/0.5.0/doc/specs/in-progress/pluggable_validators.rst>`_

* `[Introduced] Scaling & Refactoring Rally DB <https://github.com/openstack/rally/blob/0.5.0/doc/specs/in-progress/db_refactoring.rst>`_

* `[Introduced] SLA Performance degradation plugin <https://github.com/openstack/rally/blob/0.5.0/doc/specs/in-progress/sla_pd_plugin.rst>`_

Logging
~~~~~~~

* disable urllib3 warnings only if the library provide them

Database
~~~~~~~~

[doesn't require migration]
Transform DB layer to return dicts, not SQLAlchemy models

Rally Deployment
~~~~~~~~~~~~~~~~

* Support single-AZ deployment

  This supports the case where OpenStack is deployed with a single AZ for both
  controller(s) and compute(s), and not all hosts in the AZ that contains an
  instance are guaranteed to have the nova-compute service.

* Extend creation from environment with several new vars

  - OS_ENDPOINT_TYPE/OS_INTERFACE
  - OS_USER_DOMAIN_NAME
  - OS_PROJECT_DOMAIN_NAME

* Improve devstack plugin for Keystone V3

Rally Task
~~~~~~~~~~

*NEW!!* Statistics trends for given tasks.

Rally Verify
~~~~~~~~~~~~

* Remove '--tempest-config' arg from 'reinstall' command

  .. warning:: Using `--tempest-config` is became an error from this release.
      Use `rally verify genconfig` cmd for all config related stuff.

* Don't install Tempest when `rally verify start`

  .. warning:: Use should use `rally verify install` cmd to install tempest now

* Add ability to setup version of Tempest to install

  `CLI argument to setup version <http://rally.readthedocs.io/en/0.5.0/cli/cli_reference.html#verify-install-version>`_

* Configure 'aodh' service in 'service_available' section

* Check existence of Tempest-tree in `rally verify discover` cmd

* Make Tempest work with auth url which doesn't include keystone version

  Tempest needs /v2.0 and /v3 at the end of URLs. Actually, we can't fix
  Tempest, so we extend our configuration module with workaround which allow
  to specify auth_url without version in rally deployment config.

* Use default list of plugins for sahara

* Move tempest related options of rally configuration to separate section.

* *NEW!!* Support for tempest plugins.

  `CLI argument to install them <http://rally.readthedocs.io/en/0.5.0/cli/cli_reference.html#verify-installplugin>`_

Plugins
~~~~~~~

In this release we are happy to introduce new entity - plugins Base classes

We have a lot of base plugin entities: Context, Scenario, SLA and etc.
Sometimes plugins of different bases can have equal names(i.e ceilometer
OSClient and ceilometer Context). It is normal and we should allow such
conflicts. To support such cases we introduced new entity - plugin base.
Statements of plugin bases:
 - Each plugin base is unique entity;
 - Names of plugin bases can't conflict with each other;
 - Names of two or more plugins in one plugin base can't conflict with each
   other(in case of same namespace).
 - Names of two or more plugins in different plugin base can conflict

Current list of plugin bases:
 - rally.task.context.Context
 - rally.task.scenario.Scenario
 - rally.task.types.ResourceType
 - rally.task.exporter.TaskExporter
 - rally.task.processing.charts.Chart
 - rally.task.runner.ScenarioRunner
 - rally.task.sla.SLA
 - rally.deployment.serverprovider.provider.ProviderFactory
 - rally.deployment.engine.Engine
 - rally.osclients.OSClient

**OSClients**

* *NEW!!* Support for Senlin client

* *NEW!!* Support for Gnocchi client

* *NEW!!* Support for Magnum client

* *NEW!!* Support for Watcher client

* Transmit endpoint_type to saharaclient

**Scenarios**:

* *NEW!!*:

 - `Authenticate.validate_ceilometer <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#authenticate-validate-ceilometer-scenario>`_
 - `CinderVolumes.create_volume_from_snapshot <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-volume-from-snapshot>`_
 - `CinderVolumes.create_volume_and_clone <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-volume-and-clone>`_
 - `NovaFlavors.create_and_list_flavor_access <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaflavors-create-and-list-flavor-access-scenario>`_
 - `NovaFlavors.create_flavor <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaflavors-create-flavor-scenario>`_
 - `NovaServers.boot_and_update_server <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaservers-boot-and-update-server>`_
 - `NovaServers.boot_server_from_volume_snapshot <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#novaservers-boot-server-from-volume-snapshot>`_

* [Sahara] Add configs to MapR plugin

* Extend CinderVolumes.create_and_upload_volume_to_image with "image" argument

  `Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-and-upload-volume-to-image-scenario>`_

* Deprecate Dummy.dummy_with_scenario_output scenario in favor of Dummy.dummy_output

  .. warning:: Dummy.dummy_with_scenario_output scenario will be removed after
      several releases

  `Deprecated Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#dummy-dummy-with-scenario-output-scenario>`_
  `New Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#dummy-dummy-output-scenario>`_

* Extend CinderVolumes.create_volume_and_clone with nested_level

  Add nested_level argument for nested cloning volume to new volume

  `Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-volume-and-clone>`_

* Extend CinderVolumes.create_nested_snapshots_and_attach_volume

  Two new arguments were added: create_volume_kwargs and create_snapshot_kwargs

  .. warning:: All arguments related to snapshot creation should be transmitted
        only via create_snapshot_kwargs.

  `Plugin Reference <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#cindervolumes-create-nested-snapshots-and-attach-volume>`_

* Introduce new style of scenarios - class based.

  `Spec Reference <https://github.com/openstack/rally/blob/0.5.0/doc/specs/implemented/class-based-scenarios.rst>`_

* Improve report for VMTasks.boot_runcommand_delete

* [Sahara] Added 5.5.0 version for cdh-plugin and 1.6.0 version for spark

* Extend boot_server_from_volume_and_delete, boot_server_from_volume,
  boot_server_from_volume_and_live_migrate, boot_server_from_volume_snapshot
  scenarios of NovaServers class with "volume_type" parameter.

**Contexts**:

* *NEW!!*:

  - `Cinder volume_types <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#volume-types-context>`_
  - `Murano environments <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#murano-environments-context>`_
  - `Heat dataplane <http://rally.readthedocs.io/en/0.5.0/plugin/plugin_reference.html#heat-dataplane-context>`_

* Use Broker Pattern in Keystone roles context

* Use immutable types for locking context configuration

  Since context configuration passed to Context.__init__() was a mutable type
  (dict or list), sometimes we had unexpected changes done by unpredictable
  code (for example, in wrappers).

* Add possibility to balance usage of users

  For the moment all users for tasks were taken randomly and there was no way
  to balance them between tasks. It may be very useful when we have difference
  between first usage of tenant/user and all consecutive. In this case we get
  different load results.

  Therefore, "users" context was extended with new config option
  'user_choice_method' that defines approach for picking up users.

  Two values are available:
  - random
  - round_robin

  Default one is compatible with old approach - "random".

* Make sahara_image and custom_image contexts glance v2 compatible

* Extend servers context with "nics" parameter

* Extend network context with "dns_nameservers" parameter

* Extend volume context with "volume_type" parameter

**Cleanup**:

* Mark several cleanup resources as tenant_resource

  Nova servers and security groups are tenant related resources, but resource
  decorator missed that fact which makes cleanup tries to delete one resources
  several times.

* Turn off redundant nova servers cleanup for NovaFlavors.list_flavors scenario

* Add neutron cleanup for NeutronSecurityGroup.create_and_delete_security_groups

**Exporter**:

Rename task-exporter "file-exporter" to "file".

.. warning:: "file-exporter" is deprecated and will be removed in further
    releases.

**Types**:

Remove deprecated types.

.. warning:: you should use rally.task.types.convert instead of
    rally.task.types.set decorator

**Validators**

* Add a required_api_version validator
* Add validators for scenario arguments

**Utils**:

Use glance wrapper where appropriate to support compatibility between V1 and V2

Bug fixes
~~~~~~~~~

**19 bugs were fixed**:

* Wrong arguments order of Keystone wrapper in case of V2 and V3

* AttributeError while disabling urllib3 warnings on old installations

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1573650>`_

* install_rally.sh script is failed while obtaining setuptools

* "-inf" load duration in case of wrong runner plugin and failed start of
  contexts

* Strange input task in the report

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1570328>`_

* Wrong behaviour of boot_server_from_volume scenarios in case of booting
  server from image.

  The arg of image must be None, when booting server from volume. Otherwise
  still boot server from image.

  Affected scenarios:
    NovaServers.boot_server_from_volume
    NovaServers.boot_server_from_volume_and_delete
    NovaServers.boot_server_from_volume_and_resize
    NovaServers.boot_server_from_volume_and_live_migrate

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1578556>`_

* Weak validation of json schema of RPS runner

  JSON Schema of RPS runner doesn't have "required" field. It means that
  users are able to pass wrong configs and we will have runtime error while
  running task.

* Rally doesn't take cacert setting while creating keystone session

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1577360>`_

* Heat scenarios fail when API uses TLS

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1585456>`_

* Example in comment of context manila_share_networks wrong

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1587164>`_

* There is no way to get UUID of a verification after it is created by
  "rally verify start" or "rally verify import_results" when --no-use is set

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1587034>`_

* Exposed ssh timeout and interval in vm scenario

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1587728>`_

* Ceilometer scenario doesn't require "ceilometer" ctx

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1557642>`_

* "servers" context requires setting network id for multiple possible networks
  found.

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1592292>`_

* nested_level data type incorrect in create_nested_snapshots_and_attach_volume

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1594656>`_

* Rally cleanup servers raises exception

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1584104>`_

* Stopping server is redundant before cold-migrating server

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1594730>`_

* existing_users context doesn't work in case of Keystone v3

* Whether validates flavor's disk or not depends on booting type of the instance

  `Launchpad bug-report <https://bugs.launchpad.net/rally/+bug/1596756>`_

Documentation
~~~~~~~~~~~~~

* Re-use openstack theme for building docs outside rtd.

  `Rally Docs at docs.openstack.org <http://docs.openstack.org/developer/rally/>`_

* Add page for Verification component

  `RTD page <http://rally.readthedocs.io/en/0.5.0/tutorial/step_10_verifying_cloud_via_tempest.html>`_

* Add glossary page

  `RTD page <http://rally.readthedocs.io/en/0.5.0/tutorial/glossary.html>`_

* Adjust docs reference to "KeystoneBasic.authenticate" scenario

  `RTD page <http://rally.readthedocs.io/en/0.5.0/tutorial/step_6_aborting_load_generation_on_sla_failure.html>`_

Thanks
~~~~~~

 2 Everybody!

Unverified

0.4.0

b6a203d9 · fix up the formatting of the readme for the announce script · Apr 19, 2016

rally 0.4.0 release

meta:version: 0.4.0
meta:series: independent
meta:release-type: release
meta:announce: openstack-announce@lists.openstack.org
meta:pypi: yes
meta:first: no

Unverified

0.3.3

b8d76c16 · Release notes for 0.3.3 · Mar 24, 2016

============
Rally v0.3.3
============

Information
-----------

+------------------+-----------------------+
| Commits          |        **20**         |
+------------------+-----------------------+
| Dev cycle        |       **10 days**     |
+------------------+-----------------------+
| Release date     |     **3/24/2016**     |
+------------------+-----------------------+

Details
-------

A half of patches relate to Cleanup. We have once again proved that ideal
stuff can be improved. :)

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* `[Spec][Introduced] Improve atomic actions format`__

__ https://github.com/openstack/rally/blob/master/doc/specs/in-progress/improve_atomic_actions_format.rst

Plugins
~~~~~~~

* **Cleanups**:

 * Use proper attribute to get heat stack name

 * Always assign a name to created images.

   This is necessary for name-based cleanup. If a name is not specified, one
   will be generated automatically.

 * Improve filtering glance images in case of V2 API

 * Delete only images created by images context

   Since the images context allows creating images with arbitrary names,
   name-based cleanup won't work for it, so we have to delete the exact list
   of images that it created instead.

 * New config option to set cleanup threads

   Allow the user to change the number of cleanup threads via the rally
   config. When scaling out to thousands of instances, the cleanup can take
   forever with the static 20 threads.

 * Add inexact matching to name_matches_object

   This will support places where we create resources with names that start
   with a given name pattern, but include some additional identifier
   afterwards. For instance, when bulk creating instances, Nova appends a UUID
   to each instance name.

* **Scenarios**:

 * Add sample of template for testing for testing heat caching.

 * Introduced new scenario `Dummy.dummy_random_action`__. It is suitable for
   demonstration of upcoming trends report.

__ http://rally.readthedocs.org/en/latest/plugin/plugin_reference.html#dummy-dummy-random-action-scenario

* **Contexts**:

 `api_versions`__ context was extended to support switch between Keystone V2
 and V3 API versions. Now it is possible to use one Rally deployment to check
 both Keystone APIs.

__ http://rally.readthedocs.org/en/latest/plugin/plugin_reference.html#api-versions-context

* **Newcomer in the family**:

 All ResourceType classes are pluggable now and it is much easier to use and
 extend them.

 .. warning:: Decorator ``rally.task.types.set`` is deprecated now in favor of
    ``rally.task.types.convert``.

Bug fixes
~~~~~~~~~

* #1536172: rally deployment destroy failed with traceback for failed
  deployments. At current moment it is impossible to delete deployment if for
  some reason deployment engine plugin cannot be found, because exception will
  be thrown.

Documentation
~~~~~~~~~~~~~

* Remove extra link in `All release notes`

  Previously, two links for latest release were presented.

* Update release notes for 0.3.2

  * Fixed indents for warning messages
  * Fixed all references

Thanks
~~~~~~

 To Everybody!

Unverified

0.3.2

17f71d20 · Release notes for 0.3.2 · Mar 14, 2016

============
Rally v0.3.2
============

Information
-----------

+------------------+-----------------------+
| Commits          |         **55**        |
+------------------+-----------------------+
| Dev cycle        |       **25 days**     |
+------------------+-----------------------+
| Release date     |     **3/14/2016**     |
+------------------+-----------------------+

Details
-------

This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.

CLI changes
~~~~~~~~~~~

* .. warning:: [Modified] Option '--tempest-config' for 'rally verify
  reinstall' command was deprecated for removal.

* .. warning:: [Removed] Option `--system-wide-install` was removed from
  `rally verify` commands in favor of `--system-wide` option.

* .. warning:: [Modified] Step of installation of Tempest during execution of
  the `rally verify start` command was deprecated and will be removed in the
  future. Please use `rally verify install` instead.

* Rework commands.task.TaskCommands.detailed. Now output of the command
  contains the same results as in HTML report.

Rally Verify
~~~~~~~~~~~~

* Re-run failed Tempest tests

  Add the ability to re-run the tempest tests that failed in the last test
  execution. Sometimes Tempest tests fail due to a special temporary condition
  in the environment, in such cases it is very useful to be able to re-execute
  those tests.

  Running the following command will re-run all the test that failed during
  the last test execution regardless of what test suite was run.

  .. code-block:: bash

      rally verify start --failing

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* [Spec][Introduced] Refactoring scenario utils

  `https://github.com/openstack/rally/blob/master/doc/specs/in-progress
  /refactor_scenario_utils.rst`_

* [Spec] Deployment unification

Plugins
~~~~~~~

* **Scenarios**:

 * [updated] Fix flavor for cloudera manager

   Cloudera manager need master-node flavor

 * [added] Expand Nova API benchmark in Rally

   Add support for listing nova hosts, agents, availability-zones
   and aggregates.

 * [updated] Make sure VolumeGenerator uses the api version info while cleanup

 * Designate V2 - Add recordset scenarios

   Add create_and_(list|delete)_recordset scenarios
   Remove the test also that checks the allowed methods, this is in order for
   us to be able to have a private method _walk_pages that will do fetching of
   pages for us vs attempting to fetch 1 giant list at once.

 * unify *_kwargs name in scenarios

   When running a scenario, `kwargs` is used as default key-word arguments.
   But in some scenarios, there are more and one services being called, and
   we use xxx_kwargs for this case.

   However, some xxx_kwargs are not unifed for same usage[0]. Unifying
   these could avoid midleading for end users. Another improvement is to
   add xxx_kwargs with empty settings for scenario config files.

   [0] http://paste.openstack.org/show/489505/

 * .. warning:: Deprecated arguments 'script' and 'interpreter' were removed
     in favor of 'command' argument.

   VM task scenarios executes a script with a interpreter provided through a
   formatted argument called 'command' which expects the remote_path or
   local_path of the script and optionally an interpreter with which the
   script has to be executed.

Miscellaneous
~~~~~~~~~~~~~

* Avoid using `len(x)` to check if x is empty

  This cases are using `len()` to check if collection has items. As
  collections have a boolean representation too, directly check for true /
  false. And fix the wrong mock in its unit test.

* Fix install_rally.sh to get it to work on MacOSX

  On MacOSX, `mktemp` requires being passed a template. This change modifies
  the calls to `mktemp` to explicitly pass a template so that the code works
  on both MacOSX and linux.

* Use new-style Python classes

  There are some classes in the code that didn't inherited from
  nothing and this is an old-style classes. A "New Class" is the
  recommended way to create a class in modern Python.A "New Class"
  should always inherit from `object` or another new-style class.

  Hacking rule added as well.

* Make Rally cope with unversioned keystone URL

  With the change, the client version that's returned is now determined by
  the keystoneclient library itself based on whether you supply a URL with a
  version in it or not.

* Fix rally-mos job to work with mos-8.0

  Also remove hardcoded values for some other jobs.

* Add name() to ResourceManager

  This will allow us to perform cleanup based on the name.

* Add task_id argument to name_matches_object

  This will be used to ensure that we are only deleting resources for a
  particular Rally task.

* Extend api.Task.get_detailed

  Extend api.Task.get_detailed with ability to return task data as dict with
  extended results.

Bug fixes
~~~~~~~~~

**The most critical fixed bugs are**:

* #1547624: Wrong configuration for baremetal(ironic) tempest tests

* #1536800: openrc values are not quoted

  The openrc file created after rally deployment --fromenv did not quote the
  values for environment variables that will be exported.

* #1509027: Heat delete_stack never exits if status is DELETE_FAILED

* #1540545: Refactored atomic action in authenticate scenario

* #1469897: Incompatible with Keystone v3 argument in service create scenario

* #1550262: Different results in ``rally task detailed``, ``rally task report``
 and ``rally task status`` commands.

* #1553024: Backward incompatible change in neutronclient(release 4.1.0) broke
  Tempest config generation to support latest neutronclient.

Documentation
~~~~~~~~~~~~~

* Add documentation for DB migration

  Changes:
    * Add descriptive docstrings for plugins based on OutputChart
    * Register these plugins in doc/ext/plugin_reference.py
      so plugin/plugin_reference.htmlwill have a documentation
      chapter based on added docstrings

* Documentation tox fix

  Added information about debugging unit test with tox. Replace 3 references
  to py26 with py34 to reflect current rally tox configuration.

* Change structure of rally plugin and plugin references page

* Update the scenario development, runner and context sections

* The design of 'Rally Plugins Reference' page was improved (see
  `http://docs.openstack.org/developer/rally/plugin/plugin_reference.html`_
  with new design)

* New page was added - CLI references

  `http://docs.openstack.org/developer/rally/cli/cli_reference.html`_

Thanks
~~~~~~

 To Everybody!

Unverified

0.3.1

1b6e9e92 · Release notes for 0.3.1 · Feb 18, 2016

Rally v0.3.1

Information
-----------

+------------------+-----------------------+
| Commits          |         **9**         |
+------------------+-----------------------+
| Bug fixes        |         **6**         |
+------------------+-----------------------+
| Dev cycle        |       **2 days**      |
+------------------+-----------------------+
| Release date     |     **2/18/2016**     |
+------------------+-----------------------+

Details
-------

This release is more about bug-fixes than features.

.. warning:: Please, update 0.3.0 to latest one.

Features
~~~~~~~~

*  Pass api_versions info to glance images context

* [Verify] Don't create new flavor when flavor already exists

Bug fixes
~~~~~~~~~

**6 bugs were fixed, the most critical are**:

* #1545889: Existing deployment with given endpoint doesn't work anymore

* #1547092: Insecure doesn't work with Rally 0.3.0

* #1547083: Rally Cleanup failed with api_versions context in 0.3.0 release

* #1544839: Job gate-rally-dsvm-zaqar-zaqar fails since the recent Rally patch

* #1544522: Non-existing "called_once_with" method of Mock library is used

Unverified

0.3.0

60b671c5 · Merge "Add v0.3.0 release notes" · Feb 16, 2016

Rally v0.3.0

Information
-----------

+------------------+-----------------------+
| Commits          |        **69**         |
+------------------+-----------------------+
| Bug fixes        |        **7**          |
+------------------+-----------------------+
| Dev cycle        |       **29 days**     |
+------------------+-----------------------+
| Release date     |     **2/16/2016**     |
+------------------+-----------------------+

Details
-------

This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.

.. warning:: In this release Rally DB schema migration is introduced. While
    upgrading Rally from previous versions it is required now to run
    ``rally-manade db upgrade``. Please see 'Documentation' section for details.

CLI changes
~~~~~~~~~~~

* [Removed] ``rally info`` in favor of ``rally plugin *``.
  It was deprecated for a long time.

* [Modified] ``rally deployment check`` now prints services, which don't have
  names, since such services can be used via api_versions context.

* [Modified] ``rally verify [re]install``  option --no-tempest-venv for was
  deprecated in favor of --system-wide

* [Added] ``rally-manage db revision`` displays current revision of
  Rally database schema

* [Added] ``rally-manage db upgrade`` upgrades pre-existing Rally
  database schema to the latest revision

* [Added] ``rally-manage db downgrade`` to downgrades existing Rally
  database schema to previous revision

* [Added] ``rally task export`` exports task results to external
  services (only CLI command introduced, no real service support
  implemented yet, however one could write own plugins)

* [Added] ``rally verify export`` exports verification results to
  external services (only CLI command introduced, no real service support
  implemented yet, however one could write own plugins)

Rally Deployment
~~~~~~~~~~~~~~~~

* ``fuel`` deployment engine is removed since it was outdated and lacked
  both usage and support

Rally Task
~~~~~~~~~~

Add custom labels for "Scenario Output" charts

* X-axis label can be specified to add_output() by
  "axis_label" key of chart options dict.
  The key is named "axis_label" but not "x_label"
  because chart can be displayed as table, so we explicitly
  mention "axis" in option name to make this parameter
  useless for tables

* Y-axis label can be specified to add_output() by
  "label" key of chart options dict
  In some cases this parameter can be used for rendering
  tables - it becomes column name in case if chart with
  single iteration is transformed into table

* As mentiones above, if we have output chart
  with single iteration, then it is transformed to table,
  because chart with single value is useless

* OutputLinesChart is added, it is displayed by
  NVD3 lineChart()

* Chart "description" is optional now. Description is
  not shown if it is not specified explicitly

* Scenario Dummy.add_output is improved to display labels
  and OutputLinesChart

* Fix: If Y-values are too long and overlaps chart box,
  then JavaScript updates chart width in runtime to fit
  width of chart graphs + Y values to their DOM container

Rally Certification
~~~~~~~~~~~~~~~~~~~

 None.

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* [Spec][Introduced] Export task and verification results to external services

  https://github.com/openstack/rally/blob/master/doc/specs/in-progress/task_and_verification_export.rst

* [Spec][Implemented]  Consistent resource names

  https://github.com/openstack/rally/blob/master/doc/specs/implemented/consistent_resource_names.rst

* [Feature request][Implemented]  Tempest concurrency

  https://github.com/openstack/rally/blob/master/doc/feature_request/implemented/add_possibility_to_specify_concurrency_for_tempest.rst

Plugins
~~~~~~~

* **Scenarios**:

 - [added] VMTasks.workload_heat

 - [added] NovaFlavors.list_flavors

 - [updated] Flavors for Master and Worker node groups are now
   configured separately for SaharaCluster.* scenarios

* **Context**:

 - [improved] sahara_cluster

   Flavors for Master and Worker node groups are now
   configured separately in ``sahara_cluster`` context

Miscellaneous
~~~~~~~~~~~~~

* Cinder version 2 is used by default

* Keystone API v3 compatibility improved
  - Auth URL in both formats http://foo.rally:5000/v3
    and http://foo.rally:5000 is supported for Keystone API v3
  - Tempest configuration file is created properly according
    to Keystone API version used

* ``install_rally.sh --branch`` now accepts all git tree-ish,
  not just branches or tags

* VM console logs are now printed when Rally fails to connect to VM

* Add support for Rally database schema migration (see 'Documentation' section)

Bug fixes
~~~~~~~~~

**7 bugs were fixed**:

* #1540563: Rally is incompatible with liberty Neutron client

  The root cause is that in Neutron Liberty client,
  the _fx function doesn't take any explicit keyword parameter
  but Rally is passing one (tenant_id).

* #1543414: The `rally verify start` command fails when running
  a verification against Kilo OpenStack

* #1538341: Error in logic to retrieve image details in image_valid_on_flavor

Documentation
~~~~~~~~~~~~~

* Add documentation for DB migration

  https://github.com/openstack/rally/blob/master/rally/common/db/sqlalchemy/migrations/README.rst

Thanks
~~~~~~

 2 Everybody!

Unverified

0.2.0

57efc532 · Merge "Add release notes for 0.2.0" · Jan 11, 2016

Rally v0.0.2

Information
-----------

+------------------+-----------------+
| Commits          |     **100**     |
+------------------+-----------------+
| Bug fixes        |     **18**      |
+------------------+-----------------+
| Dev cycle        |   **45 days**   |
+------------------+-----------------+
| Release date     | **12/Mar/2015** |
+------------------+-----------------+

Details
-------

This release contains new features, new benchmark plugins, bug fixes,
various code and API improvements.

New Features
~~~~~~~~~~~~

* rally task start **--abort-on-sla-failure**

    Stopping load before things go wrong.
    Load generation will be interrupted if SLA criteria stop passing.

* Rally verify command supports multiple Tempest sources now.

* python34 support

* postgres DB backend support

API changes
~~~~~~~~~~~

* [new] **rally [deployment | verify | task] use** subcommand

    It should be used instead of root command **rally use**

* [new] Rally as a Lib API

    To avoid code duplication between Rally as CLI tool and Rally as a Service
    we decide to make Rally as a Lib as a common part between these 2 modes.

    Rally as a Service will be a daemon that just maps HTTP request to Rally
    as a Lib API.

* [deprecated] **rally use** CLI command

* [deprecated] Old Rally as a Lib API

    Old Rally API was quite mixed up so we decide to deprecate it

Plugins
~~~~~~~

* **Benchmark Scenario Runners**:

    [improved] Improved algorithm of generation load in **constant runner**

        Before we used processes to generate load, now it creates pool of
        processes (amount of processes is equal to CPU count) after that in
        each process use threads to generate load. So now you can easily
        generate load of 1k concurrent scenarios.

    [improved] Unify code of **constant** and **rps** runners

    [interface] Added **abort()** to runner's plugin interface

        New method **abort()** is used to immediately interrupt execution.

* **Benchmark Scenarios**:

    [new] DesignateBasic.create_and_delete_server

    [new] DesignateBasic.create_and_list_servers

    [new] DesignateBasic.list_servers

    [new] MistralWorkbooks.list_workbooks

    [new] MistralWorkbooks.create_workbook

    [new] Quotas.neutron_update

    [new] HeatStacks.create_update_delete_stack

    [new] HeatStacks.list_stacks_and_resources

    [new] HeatStacks.create_suspend_resume_delete_stac

    [new] HeatStacks.create_check_delete_stack

    [new] NeutronNetworks.create_and_delete_routers

    [new] NovaKeypair.create_and_delete_keypair

    [new] NovaKeypair.create_and_list_keypairs

    [new] NovaKeypair.boot_and_delete_server_with_keypair

    [new] NovaServers.boot_server_from_volume_and_live_migrate

    [new] NovaServers.boot_server_attach_created_volume_and_live_migrate

    [new] CinderVolumes.create_and_upload_volume_to_image

    [fix] CinderVolumes.create_and_attach_volume

        Pass optional \*\*kwargs only to create server command

    [fix]  GlanceImages.create_image_and_boot_instances

        Pass optional \*\*kwargs only to create server command

    [fix] TempestScenario.\* removed stress cleanup.

        Major issue is that tempest stress cleanup cleans whole OpenStack.
        This is very dangerous, so it's better to remove it and leave some
        extra resources.

    [improved] NovaSecGroup.boot_and_delete_server_with_secgroups

        Add optional \*\*kwargs that are passed to boot server comment

* **Benchmark Context**:

    [new] **stacks**

        Generates passed amount of heat stacks for all tenants.

    [new] **custom_image**

        Prepares images for benchmarks in VMs.

        To Support generating workloads in VMs by existing tools like: IPerf,
        Blogbench, HPCC and others we have to have prepared images, with
        already installed and configured tools.

        Rally team decide to generate such images on fly from passed to avoid
        requirements of having big repository with a lot of images.

        This context is abstract context that allows to automate next steps:

        1) runs VM with passed image (with floating ip and other stuff)
        2) execute abstract method that has access to VM
        3) snapshot this image

        In future we are going to use this as a base for making context that
        prepares images.

    [improved] **allow_ssh**

        Automatically disable it if security group are disabled in neutron.

    [improved] **keypair**

        Key pairs are stored in "users" space it means that accessing keypair
        from scenario is simpler now:

            self.context["user"]["keypair"]["private"]

    [fix] **users**

        Pass proper EndpointType for newly created users

    [fix] **sahara_edp**

        The Job Binaries data should be treated as a binary content

* **Benchmark SLA**:

    [interface] SLA calculations is done in additive way now

        Resolves scale issues, because now we don't need to have whole
        array of iterations in memory to process SLA.

        This is required to implement **--abort-on-sla-failure** feature

    [all] SLA plugins were rewritten to implement new interface

Bug fixes
~~~~~~~~~

**18 bugs were fixed, the most critical are**:

* Fix **rally task detailed --iterations-data**

  It didn't work in case of missing atomic actions. Such situation can occur
  if scenario method raises exceptions

* Add user-friendly message if the task cannot be deleted

  In case of trying to delete task that is not in "finished" status users get
  traces instead of user-friendly message try to run it with --force key.

* Network context cleanups networks properly now

Documentation
~~~~~~~~~~~~~

* Image sizes are fixed

* New tutorial in "Step by Step" relate to **--abort-on-sla-failure**

* Various fixes

Unverified

0.1.2

2c34d189 · Merge "Relese notes 0.1.2" · Dec 23, 2015

Rally v0.1.2

Information
-----------

+------------------+-----------------------+
| Commits          |        **208**        |
+------------------+-----------------------+
| Bug fixes        |        **37**         |
+------------------+-----------------------+
| Dev cycle        |      **77 days**      |
+------------------+-----------------------+
| Release date     |  **23/December/2015** |
+------------------+-----------------------+

Details
-------

This release, as well as all previous ones, includes a lot of internal and
external changes. Most important of them are listed below.

.. warning:: Release 0.1.2 is the last release with Python 2.6 support.

Deprecations
~~~~~~~~~~~~

* Class `rally.common.objects.Endpoint` was renamed to `Credentials`. Old
  class is kept for backward compatibility. Please, stop using the old class
  in your plugins.

  .. warning:: dict key was changed too in user context from "endpoint" to "credential"

* rally.task.utils: wait_is_ready(), wait_for(), wait_for_delete() deprecated
  you should use wait_for_status() instead.

Rally Verify
~~~~~~~~~~~~

* Added possibility to run Tempest tests listed in a file(--tests-file argument in ``verify start``)

* Added possibility to upload Tempest subunit stream logs into data base

* Improvements in generating Tempest config file

* Reworked subunit stream parser

* Don't install Tempest when `rally verify [gen/show]config`

* Rally team tries to simplify usage of each our component.
  Now Rally verification has some kind of a context like in Tasks.
  Before launching each verification, Rally checks existence of required
  resources(networks, images, flavours, etc) in Tempest configuration file and
  pre-creates them. Do not worry, all these resources will not be forgotten
  and left, Rally will clean them after verification.

Rally Task
~~~~~~~~~~

* Add --html-static argument to ``rally task report`` which allows to
  generate HTML reports that doesn't require Internet.

* Rally supports different API versions now via api_versions context:

.. code:: none

  CinderVolumes.create_and_delete_volume:
    -
      args:
        size: 1
      runner:
        type: "constant"
        times: 2
        concurrency: 2
      context:
        users:
          tenants: 2
          users_per_tenant: 2
        api_versions:
          cinder:
            version: 2
            service_name: cinderv2

* Move rally.osclients.Clients to plugin base

  Rally OSclients is plugable now and it is very easy to extend OSClients for
  your cloud out of Rally tree.

* Add 'merge' functionality to SLA

  All SLA plugins  should implement merge() method now.
  In future this will be used for distributed load generation.
  Where SLA results from different runners will be merged together.

* New optional_action_timer decorator

  Allows to make the methods that can be both atomic_action or regular
  method. Method changes behavior based on value in extra key "atomic_action"

Rally Certification
~~~~~~~~~~~~~~~~~~~

* Fix Glance certification arguments

* Add Neutron Quotas only if Neutron service is available

Specs & Feature Requests
~~~~~~~~~~~~~~~~~~~~~~~~

* Spec consistent-resource-names:

  Resource name is based on Task id now. It is a huge step to persistence
  and disaster cleanups.

* Add a spec for distributed load generation:

  https://github.com/openstack/rally/blob/master/doc/specs/in-progress/distributed_runner.rst

* Improvements for scenario output format

  https://github.com/openstack/rally/blob/master/doc/specs/in-progress/improve_scenario_output_format.rst

* Task and Verify results export command

  https://github.com/openstack/rally/blob/master/doc/specs/in-progress/task_and_verification_export.rst

Plugins
~~~~~~~

* **Scenarios**:

 * [new] NovaServers.boot_and_get_console_output
 * [new] NovaServers.boot_and_show_server
 * [new] NovaServers.boot_server_attach_created_volume_and_resize
 * [new] NovaServers.boot_server_from_volume_and_resize
 * [new] NeutronSecurityGroup.create_and_delete_security_groups
 * [new] NeutronSecurityGroup.create_and_list_security_groups
 * [new] NeutronSecurityGroup.create_and_update_security_groups
 * [new] NeutronLoadbalancerV1.create_and_delete_healthmonitors
 * [new] NeutronLoadbalancerV1.create_and_list_healthmonitors
 * [new] NeutronLoadbalancerV1.create_and_update_healthmonitors
 * [new] SwiftObjects.list_and_download_objects_in_containers
 * [new] SwiftObjects.list_objects_in_containers
 * [new] FuelNodes.add_and_remove_node
 * [new] CeilometerMeters.list_matched_meters
 * [new] CeilometerResource.list_matched_resources
 * [new] CeilometerSamples.list_matched_samples
 * [new] CeilometerStats.get_stats
 * [new] Authenticate.validate_monasca
 * [new] DesignateBasic.create_and_delete_zone
 * [new] DesignateBasic.create_and_list_zones
 * [new] DesignateBasic.list_recordsets
 * [new] DesignateBasic.list_zones
 * [fix] CinderVolumes.create_nested_snapshots_and_attach_volume
    Remove random nested level which produce different amount of atomic
    actions and bad reports.
 * Support for Designate V2 api
 * A lot of improvements in Sahara scenarios

* **Context**:

 * [new] api_versions

   Context allows us to setup client to communicate to specific service.

 * [new] swift_objects

   Context pre creates swift objects for future usage in scenarios

 * [update] sahara_cluster

   It supports proxy server which allows to use single floating IP for
   whole cluster.

 * [fix] cleanup

   Fix cleanup of networks remove vip before port.

Bug fixes
~~~~~~~~~

**37 bugs were fixed, the most critical are**:

* Follow symlinks in plugin discovery
* Use sed without -i option for portability (install_rally.sh)
* Fixed race in rally.common.broker
* Fixed incorrect iteration number on "Failures" Tab
* Fixing issue with create_isolated_networks = False
* Fix docker build command

Documentation
~~~~~~~~~~~~~

Fixed some minor typos and inaccuracies.

Thanks
~~~~~~

We would like to thank Andreas Jaeger for ability to provide Python 2.6 support in this release.

Unverified

0.1.1

3011fa6e · Release notes v0.1.1 · Oct 06, 2015

Rally v0.1.1

Information
-----------

+------------------+-----------------------+
| Commits          |         **32**        |
+------------------+-----------------------+
| Bug fixes        |         **9**         |
+------------------+-----------------------+
| Dev cycle        |       **11 days**     |
+------------------+-----------------------+
| Release date     |  **6/October/2015**   |
+------------------+-----------------------+

Details
-------

This release contains new features, new 6 plugins, 9 bug fixes,
various code and API improvements.

New Features
~~~~~~~~~~~~

* **Rally verify generates proper tempest.conf file now**

  Improved script that generates tempest.conf, now it works out of box for
  most of the clouds and most of Tempest tests will pass without hacking it.

* **Import Tempest results to Rally DB**

  ``rally verify import`` command allows you to import already existing Tempest
  results and work with them as regular "rally verify start" results:
  generate HTML/CSV reports & compare different runs.

API Changes
~~~~~~~~~~~~

**Rally CLI changes**

  * [add] ``rally verify import`` imports raw Tempest results to Rally

Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~

  There is no new specs and feature requests.

Plugins
~~~~~~~

* **Scenarios**:

  [new] NeutronNetworks.create_and_list_floating_ips

  [new] NeutronNetworks.create_and_delete_floating_ips

  [new] MuranoPackages.import_and_list_packages

  [new] MuranoPackages.import_and_delete_package

  [new] MuranoPackages.import_and_filter_applications

  [new] MuranoPackages.package_lifecycle

  [improved] NovaKeypair.boot_and_delete_server_with_keypair

    New argument ``server_kwargs``, these kwargs are used to boot server.

  [fix] NeutronLoadbalancerV1.create_and_delete_vips

      Now it works in case of concurrency > 1

* **Contexts**:

  [improved] network

      Network context accepts two new arguments:
      ``subnets_per_network`` and ``network_create_args``.

  [fix] network

      Fix cleanup if nova-network is used. Networks should be dissociate from
      project before deletion

  [fix] custom_image

      Nova server that is used to create custom image was not deleted if
      script that prepares server failed.

Bug fixes
~~~~~~~~~

**9 bugs were fixed, the most critical are**:

* Fix install_rally.sh script

  Set 777 access to /var/lib/rally/database file if system-wide method of
  installation is used.

* Rally HTML reports Overview table had few mistakes

    * Success rate was always 100%

    * Percentiles were wrongly calculated

* Missing Ironic, Murano and Workload(vm) options in default config file

* ``rally verify start`` failed while getting network_id

* ``rally verify genconfig`` hangs forever if Horizon is not available

Documentation
~~~~~~~~~~~~~

* **Fix project maintainers page**

  Update the information about Rally maintainers

* **Document rally --plugin-paths CLI argument**

* **Code blocks in documentation looks prettier now**

Unverified

0.1.0

785c05ea · Merge "Release notes v0.1.0" · Sep 25, 2015

Rally v0.1.0
============

Information
-----------

+------------------+-----------------------+
| Commits          |        **355**        |
+------------------+-----------------------+
| Bug fixes        |         **90**        |
+------------------+-----------------------+
| Dev cycle        |      **132 days**     |
+------------------+-----------------------+
| Release date     | **25/September/2015** |
+------------------+-----------------------+

Details
-------

This release contains new features, new 42 plugins, 90 bug fixes,
various code and API improvements.

New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
* **Improved installation script**

  * Add parameters:

    * ``--develop`` parameter to install rally in editable (develop) mode

    * ``--no-color`` to switch off output colorizing
      useful for automated output parsing and terminals that don't
      support colors.

  * Puts rally.conf under virtualenv etc/rally/ so you can have several
    rally installations in virtualenv

  * Many fixes related to access of different file, like: rally.conf,
    rally db file in case of sqlite

  * Update pip before Rally installation

  * Fix reinstallation

* **Separated Rally plugins & framework**

  Now plugins are here:
  https://github.com/openstack/rally/tree/master/rally/plugins

  Plugins are as well separated common/* for common plugins
  that can be use no matter what is tested and OpenStack related
  plugins

* **New Rally Task framework**

  * All plugins has the same Plugin base:
    rally.common.plugin.pluing.Plugin They have the same mechanisms for:
    discovering, providing information based on docstrings, and in future
    they will use the same deprecation/rename mechanism.

  * Some of files are moved:

    * rally/benchmark ->  rally/task

      *This was done to unify naming of rally task command and
      actually code that implements it.*

    * rally/benchmark/sla/base.py        -> rally/task/sla.py

    * rally/benchmark/context/base.py    -> rally/task/context.py

    * rally/benchmark/scenarios/base.py  -> rally/task/scenario.py

    * rally/benchmark/runners/base.py    -> rally/task/runner.py

    * rally/benchmark/scenarios/utils.py -> rally/task/utils.py

    This was done to:

    * avoid doing rally.benchamrk.scenarios import base as scenario_base

    * remove one level of nesting

    * simplify framework structure

  * Some of classes and methods were renamed

    * Plugin configuration:

      * context.context() -> context.configure()

      * scenario.scenario() -> scenario.configure()

      * Introduced runner.configure()

      * Introduced sla.configure()

      This resolves 3 problems:

      * Unifies configuration of different types of plugins

      * Simplifies plugin interface

      * Looks nice with new modules path:
          >>> from rally.task import scenario
          >>> @scenario.configure()

  * Atomic Actions were changed:

    * New rally.task.atomic module

      This  allow us in future to reuse atomic actions in Context plugins

    * Renames:

       rally.benchmark.scenarios.base.AtomicAction
       -> rally.task.atomic.ActionTimer

       rally.benchmark.scenarios.base.atomic_action()
       -> rally.task.atomic.action_timer()

  * **Context plugins decide how to map their data for scenario**

    Now Context.map_for_scenario method can be override to decide
    how to pass context object to each iteration of scenario.

  * Samples of NEW vs OLD context, sla, scenario and runner plugins:

    * Context
    .. code-block:: python

        # Old
        from rally.benchmark.context import base

        @base.context(name="users", order=100)
        class YourContext(base.Context):

            def setup(self):
                # ...

            def cleanup(self):
                # ...

        # New
        from rally.task import context

        @context.configure(name="users", order=100)
        class YourContext(context.Context):

            def setup(self):
                # ...

            def cleanup(self):
                # ...

            def map_for_scenario(self):
                # Maps context object to the scenario context object
                # like context["users"] -> context["user"] and so on.
    ..

    * Scenario
    .. code-block:: python

        # Old Scenario

        from rally.benchmark.scenarios import base
        from rally.benchmark import validation

        class ScenarioPlugin(base.Scenario):

            @base.scenario()
            def some(self):
                self._do_some_action()

            @base.atomic_action_timer("some_timer")
            def _do_some_action(self):
                # ...

        # New Scenario

        from rally.task import atomic
        from rally.task import scenario
        from rally.task import validation

        # OpenStack scenario has different base now:
        # rally.plugins.openstack.scenario.OpenStackScenario
        class ScenarioPlugin(scenario.Scenario):

            @scenario.configure()
            def some(self):
                self._do_some_action()

            @atomic.action_timer("some_action")
            def _do_some_action(self):
                # ...
    ..

    * Runner
    .. code-block:: python

        ## Old

        from rally.benchmark.runners import base

        class SomeRunner(base.ScenarioRunner):

             __execution_type__ = "some_runner"

            def _run_scenario(self, cls, method_name, context, args)
              # Load generation

            def abort(self):
              # Method that aborts load generation

        ## New

        from rally.task import runner

        @runner.configure(name="some_runner")
        class SomeRunner(runner.ScenarioRunner):

            def _run_scenario(self, cls, method_name, context, args)
                # Load generation

            def abort(self):
                # Method that aborts load generation

    ..

    * SLA
    .. code-block:: python

        # Old

        from rally.benchmark import sla

        class FailureRate(sla.SLA):
            # ...

        # New

        from rally.task import sla

        @sla.configure(name="failure_rate")
        class FailureRate(sla.SLA):
            # ...
    ..

* **Rally Task aborted command**

  Finally you can gracefully shutdown running task by calling:

  .. code:: bash

    rally task abort <task_uuid>
  ..

* **Rally CLI changes**

  * [add] ``rally --plugin-paths`` specify the list of directories with plugins

  * [add] ``rally task report --junit`` - generate a JUnit report
    This allows users to feed reports to tools such as Jenkins.

  * [add] ``rally task abort`` - aborts running Rally task
    when run with the ``--soft`` key, the ``rally task abort`` command is
    waiting until the currently running subtask is finished, otherwise the
    command interrupts subtask immediately after current scenario iterations
    are finished.

  * [add] ``rally plugin show`` prints detailed information about plugin

  * [add] ``rally plugin list`` prints table with rally plugin names and titles

  * [add] ``rally verify genconfig`` generates tempest.conf without running it.

  * [add] ``rally verify install`` install tempest for specified deployment

  * [add] ``rally verify reinstall`` removes tempest for specified deployment

  * [add] ``rally verify uninstall`` uninstall tempest of specified deployment

  * [fix] ``rally verify start --no-use`` --no-use was always turned on

  * [remove] ``rally use`` now each command has subcommand ``use``

  * [remove] ``rally info``

  * [remove] ``rally-manage tempest`` now it is covered by ``rally verify``

* **New Rally task reports**

  * New code is based on OOP style which is base step to make plugable Reports

  * Reports are now generated for only one iteration over the resulting data
    which resolves scalability issues when we are working with large
    amount of iterations.

  * New Load profiler plot that shows amount of iterations that are working
    in parallel

  * Failed iterations are shown as a red areas on stacked are graphic.

Non backward compatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* [remove] ``rally use`` cli command

* [remove] ``rally info`` cli command

* [remove] ``--uuid`` parameter from  ``rally deployment <any>``

* [remove ``--deploy-id`` parameter from:
  ``rally task <any>``, ``rally verify <any>``, ``rally show <any>``

Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~

  [feature request] Explicitly specify existing users for scenarios

  [feature request] Improve install script and add --unistall and --version

  [feature request] Allows specific repos & packages in install-rally.sh

  [feature request] Add abbility to caputre logs from tested services

  [feature request] Check RPC queue perfdata

  [spec] Refactoring Rally cleanup

  [spec] Consistent resource names

Plugins
~~~~~~~

* **Scenarios**:

  [new] CinderVolumes.create_volume_backup

  [new] CinderVolumes.create_and_restore_volume_backup

  [new] KeystoneBasic.add_and_remove_user_role

  [new] KeystoneBasic.create_and_delete_role

  [new] KeystoneBasic.create_add_and_list_user_roles

  [new] FuelEnvironments.list_environments

  [new] CinderVolumes.modify_volume_metadata

  [new] NovaServers.boot_and_delete_multiple_servers

  [new] NeutronLoadbalancerV1.create_and_list_pool

  [new] ManilaShares.list_shares

  [new] CeilometerEvents.create_user_and_get_event

  [new] CeilometerEvents.create_user_and_list_event_types

  [new] CeilometerEvents.create_user_and_list_events

  [new] CeilometerTraits.create_user_and_list_trait_descriptions

  [new] CeilometerTraits.create_user_and_list_traits

  [new] NeutronLoadbalancerV1.create_and_delete_pools

  [new] NeutronLoadbalancerV1.create_and_update_pools

  [new] ManilaShares.create_and_delete_share

  [new] ManilaShares.create_share_network_and_delete

  [new] ManilaShares.create_share_network_and_list

  [new] HeatStacks.create_and_delete_stack

  [new] ManilaShares.list_share_servers

  [new] HeatStacks.create_snapshot_restore_delete_stack

  [new] KeystoneBasic.create_and_delete_ec2credential

  [new] KeystoneBasic.create_and_list_ec2credentials

  [new] HeatStacks.create_stack_and_scale

  [new] ManilaShares.create_security_service_and_delete

  [new] KeystoneBasic.create_user_set_enabled_and_delete

  [new] ManilaShares.attach_security_service_to_share_network

  [new] IronicNodes.create_and_delete_node

  [new] IronicNodes.create_and_list_node

  [new] CinderVolumes.create_and_list_volume_backups

  [new] NovaNetworks.create_and_list_networks

  [new] NovaNetworks.create_and_delete_network

  [new] EC2Servers.list_servers

  [new] VMTasks.boot_runcommand_delete_custom_imagea

  [new] CinderVolumes.create_and_update_volume

* **Contexts**:

  [new] ManilaQuotas

        Add context for setting up Manila quotas:
        shares, gigabytes, snapshots, snapshot_gigabytes, share_networks

  [new] ManilaShareNetworks

        Context for share networks that will be used in case of usage
        deployment with existing users. Provided share networks via context
        option "share_networks" will be balanced between all share creations
        of scenarios.

  [new] Lbaas

        Context to create LBaaS-v1 resources

  [new] ImageCommandCustomizerContext

        Allows image customization using side effects of a command execution.
        E.g. one can install an application to the image and use these image
        for 'boot_runcommand_delete' scenario afterwards.

  [new] EC2ServerGenerator

        Context that creates servers using EC2 api

  [new] ExistingNetwork

        This context lets you use existing networks that have already been
        created instead of creating new networks with Rally. This is useful
        when, for instance, you are using Neutron with a dumb router that is
        not capable of creating new networks on the fly.

* **SLA**:

  [remove] max_failure_rate - use failure_rate instead

Bug fixes
~~~~~~~~~

**90 bugs were fixed, the most critical are**:

* Many fixes related that fixes access of rally.conf and DB files

* Incorrect apt-get "-yes" parameter in install_rally.sh script

* Rally bash completion doesn't exist in a virtualenv

* Rally show networks CLI command worked only with nova networks

* RPS runner was not properly generating load

* Check is dhcp_agent_scheduler support or not in network cleanup

* NetworkContext doesn't work with Nova V2.1

* Rally task input file was not able to use jinja2 include directive

* Rally in docker image was not able to

* Rally docker image didn't contain samples

* Do not update the average duration when iteration failed

Documentation
~~~~~~~~~~~~~

* **Add plugin reference page**

  :ref:`Rally Plugins Reference page <plugin_reference>` page contains a
  full list with

* **Add maintainers section on project info page**

  :ref:`Rally Maintainers section <project_info>` contains information
  about core contributors of OpenStack Rally their responsibilities and
  contacts. This will help us to make our community more transparent and open
  for newbies.

* **Added who is using section in docs**

* **Many small fixes**

Unverified

0.0.4

95713192 · Merge "Release 0.0.4" · May 14, 2015

============
Rally v0.0.4
============

Information
-----------

+------------------+-----------------+
| Commits          |     **87**      |
+------------------+-----------------+
| Bug fixes        |     **21**      |
+------------------+-----------------+
| Dev cycle        |   **30 days**   |
+------------------+-----------------+
| Release date     | **14/May/2015** |
+------------------+-----------------+

Details
-------

This release contains new features, new benchmark plugins, bug fixes, various code and API improvements.

New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~

* Rally now can generate load with users that already exist

  Now one can use Rally for benchmarking OpenStack clouds that are using LDAP, AD or any other read-only keystone backend where it is not possible to create any users. To do this, one should set up the "users" section of the deployment configuration of the ExistingCloud type. This feature also makes it safer to run Rally against production clouds: when run from an isolated group of users, Rally won’t affect rest of the cloud users if something goes wrong.

* New decorator *@osclients.Clients.register* can add new OpenStack clients at runtime

  It is now possible to add a new OpenStack client dynamically at runtime. The added client will be available from osclients.Clients at the module level and cached. Example:

.. code-block:: none

   >>> from rally import osclients
   >>> @osclients.Clients.register("supernova")
   ... def another_nova_client(self):
   ...   from novaclient import client as nova
   ...   return nova.Client("2", auth_token=self.keystone().auth_token,
   ...                      **self._get_auth_info(password_key="key"))
   ...
   >>> clients = osclients.Clients.create_from_env()
   >>> clients.supernova().services.list()[:2]
   [<Service: nova-conductor>, <Service: nova-cert>]

* Assert methods now available for scenarios and contexts

  There is now a new *FunctionalMixin* class that implements basic unittest assert methods. The *base.Context* and *base.Scenario* classes inherit from this mixin, so now it is possible to use *base.assertX()* methods in scenarios and contexts.

* Improved installation script

  The installation script has been almost completely rewritten. After this change, it can be run from an unprivileged user, supports different database types, allows to specify a custom python binary, always asks confirmation before doing potentially dangerous actions, automatically install needed software if run as root, and also automatically cleans up the virtualenv and/or the downloaded repository if interrupted.

Specs & Feature requests
~~~~~~~~~~~~~~~~~~~~~~~~

* [Spec] Reorder plugins

  The spec describes how to split Rally framework and plugins codebase to make it simpler for newbies to understand how Rally code is organized and how it works.

* [Feature request] Specify what benchmarks to execute in task

  This feature request proposes to add the ability to specify benchmark(s) to be executed when the user runs the *rally task start* command. A possible solution would be to add a special flag to the *rally task start* command.

Plugins
~~~~~~~

* **Benchmark Scenario Runners**:

    * Add limits for maximum Core usage to constant and rps runners

      The new 'max_cpu_usage' parameter can be used to avoid possible 100% usage of all available CPU cores by reducing the number of CPU cores available for processes started by the corresponding runner.

* **Benchmark Scenarios**:

    * [new] KeystoneBasic.create_update_and_delete_tenant

    * [new] KeystoneBasic.create_user_update_password

    * [new] NovaServers.shelve_and_unshelve_server

    * [new] NovaServers.boot_and_associate_floating_ip

    * [new] NovaServers.boot_lock_unlock_and_delete

    * [new] NovaHypervisors.list_hypervisors

    * [new] CeilometerSamples.list_samples

    * [new] CeilometerResource.get_resources_on_tenant

    * [new] SwiftObjects.create_container_and_object_then_delete_all

    * [new] SwiftObjects.create_container_and_object_then_download_object

    * [new] SwiftObjects.create_container_and_object_then_list_objects

    * [new] MuranoEnvironments.create_and_deploy_environment

    * [new] HttpRequests.check_random_request

    * [new] HttpRequests.check_request

    * [improved] NovaServers live migrate benchmarks

        add 'min_sleep' and 'max_sleep' parameters to simulate a pause between VM booting and running live migration

    * [improved] NovaServers.boot_and_live_migrate_server

        add a usage sample to samples/tasks

    * [improved] CinderVolumes benchmarks

        support size range to be passed to the 'size' argument as a dictionary
        *{"min": <minimum_size>, "max": <maximum_size>}*

* **Benchmark Contexts**:

    * [new] MuranoPackage

      This new context can upload a package to Murano from some specified path.

    * [new] CeilometerSampleGenerator

      Context that can be used for creating samples and collecting resources for benchmarks in a list.

* **Benchmark SLA**:

    * [new] outliers

      This new SLA checks that the number of outliers (calculated from the mean and standard deviation of the iteration durations) does not exceed some maximum value. The SLA is highly configurable: the parameters used for outliers threshold calculation can be set by the user.

Bug fixes
~~~~~~~~~

**21 bugs were fixed, the most critical are**:

* Make it possible to use relative imports for plugins that are outside of rally package.

* Fix heat stacks cleanup by deleting them only 1 time per tenant (get rid of "stack not found" errors in logs).

* Fix the wrong behaviour of 'rally task detailed --iterations-data' (it lacked the iteration info before).

* Fix security groups cleanup: a security group called "default", created automatically by Neutron, did not get deleted for each tenant.

Other changes
~~~~~~~~~~~~~~~~~~~~~~~~~~

* Streaming algorithms that scale

  This release introduces the common/streaming_algorithms.py module. This module is going to contain implementations of benchmark data processing algorithms that scale: these algorithms do not store exhaustive information about every single benchmark iteration duration processed. For now, the module contains implementations of algorithms for computation of mean & standard deviation.

* Coverage job to check that new patches come with unit tests

  Rally now has a coverage job that checks that every patch submitted for review does not decrease the number of lines covered by unit tests (at least too much). This job allows to mark most patches with no unit tests with '-1'.

* Splitting the plugins code (Runners & SLA) into common/openstack plugins

  According to the spec "Reorder plugins" (see above), the plugins code for runners and SLA has been moved to the *plugins/common/* directory. Only base classes now remain in the *benchmark/* directory.

Documentation
~~~~~~~~~~~~~

* Various fixes

    * Remove obsolete *.rst* files (*deploy_engines.rst* / *server_providers.rst* / ...)
    * Restructure the docs files to make them easier to natigate through
    * Move the chapter on task templates to the 4th step in the tutorial
    * Update the information about meetings (new release meeting & time changes)

Unverified

0.0.3

6d0fc0d5 · Merge "Release 0.0.3" · Apr 14, 2015

============
Rally v0.0.3
============

Information
-----------

+------------------+-----------------+
| Commits          |     **53**      |
+------------------+-----------------+
| Bug fixes        |     **14**      |
+------------------+-----------------+
| Dev cycle        |   **33 days**   |
+------------------+-----------------+
| Release date     | **14/Apr/2015** |
+------------------+-----------------+

Details
-------

This release contains new features, new benchmark plugins, bug fixes,
various code and API improvements.

New Features & API changes
~~~~~~~~~~~~~~~~~~~~~~~~~~

    * Add the ability to specify versions for clients in benchmark scenarios

      You can call self.clients("glance", "2") and get any client for
      specific version.

    * Add API for tempest uninstall

      $ rally-manage tempest uninstall
      # removes fully tempest for active deployment

    * Add a --uuids-only option to rally task list

        $ rally task list --uuids-only   # returns list with only task uuids

    * Adds endpoint to --fromenv deployment creation

        $ rally deployment create --fromenv
        # recognizes standard OS_ENDPOINT environment variable

    * Configure SSL per deployment

        Now SSL information is deployment specific not Rally specific and
        rally.conf option is deprecated

        Like in this sample
        https://github.com/stackforge/rally/blob/14d0b5ba0c75ececfdb6a6c121d9cf2810571f77/samples/deployments/existing.json#L11-L12

Specs
~~~~~

    * [spec] Proposal for new task input file format

      This spec describes new task input format that will allow us to generate
      multi scenario load which is crucial for HA and more real life testing:

      https://github.com/stackforge/rally/blob/master/doc/specs/in-progress/new_rally_input_task_format.rst

Plugins
~~~~~~~

* **Benchmark Scenario Runners**:

    * Add a maximum concurrency option to rps runner

      To avoid running to heavy load you can set 'concurrency' to configuration
      and in case if cloud is not able to process all requests it won't start
      more parallel requests then 'concurrency' value.

* **Benchmark Scenarios**:

    [new] CeilometerAlarms.create_alarm_and_get_history

    [new] KeystoneBasic.get_entities

    [new] EC2Servers.boot_server

    [new] KeystoneBasic.create_and_delete_service

    [new] MuranoEnvironments.list_environments

    [new] MuranoEnvironments.create_and_delete_environment

    [new] NovaServers.suspend_and_resume_server

    [new] NovaServers.pause_and_unpause_server

    [new] NovaServers.boot_and_rebuild_server

    [new] KeystoneBasic.create_and_list_services

    [new] HeatStacks.list_stacks_and_events

    [improved] VMTask.boot_runcommand_delete

        restore ability to use fixed IP and floating IP to connect to VM
        via ssh

    [fix] NovaServers.boot_server_attach_created_volume_and_live_migrate

        Kwargs in nova scenario were wrongly passed

* **Benchmark SLA**:

    * [new] aborted_on_sla

    This is internal SLA criteria, that is added if task was aborted

    * [new] something_went_wrong

    This is internal SLA criteria, that is added if something went wrong,
    context failed to create or runner raised some exceptions

Bug fixes
~~~~~~~~~

**14 bugs were fixed, the most critical are**:

    * Set default task uuid to running task. Before it was set only after
      task was fully finished.

    * The "rally task results" command showed a disorienting "task not found"
      message for a task that is currently running.

    * Rally didn't know how to reconnect to OpenStack in case if token
      expired.

Documentation
~~~~~~~~~~~~~

* New tutorial **task templates**

https://rally.readthedocs.org/en/latest/tutorial/step_8_task_templates.html

* Various fixes

Unverified

0.0.2

df2475d2 · Release 0.0.2 · Mar 12, 2015

============
Rally v0.0.2
============

Information
-----------

+------------------+-----------------+
| Commits          |     **100**     |
+------------------+-----------------+
| Bug fixes        |     **18**      |
+------------------+-----------------+
| Dev cycle        |   **45 days**   |
+------------------+-----------------+
| Release date     | **12/Mar/2015** |
+------------------+-----------------+

Details
-------

This release contains new features, new benchmark plugins, bug fixes,
various code and API improvements.

New Features
~~~~~~~~~~~~

* rally task start **--abort-on-sla-failure**

    Stopping load before things go wrong.
    Load generation will be interrupted if SLA criteria stop passing.

* Rally verify command supports multiple Tempest sources now.

* python34 support

* postgres DB backend support

API changes
~~~~~~~~~~~

* [new] **rally [deployment | verify | task] use** subcommand

    It should be used instead of root command **rally use**

* [new] Rally as a Lib API

    To avoid code duplication between Rally as CLI tool and Rally as a Service
    we decide to make Rally as a Lib as a common part between these 2 modes.

    Rally as a Service will be a daemon that just maps HTTP request to Rally
    as a Lib API.

* [deprecated] **rally use** CLI command

* [deprecated] Old Rally as a Lib API

    Old Rally API was quite mixed up so we decide to deprecate it

Plugins
~~~~~~~

* **Benchmark Scenario Runners**:

    [improved] Improved algorithm of generation load in **constant runner**

        Before we used processes to generate load, now it creates pool of
        processes (amount of processes is equal to CPU count) after that in
        each process use threads to generate load. So now you can easily
        generate load of 1k concurrent scenarios.

    [improved] Unify code of **constant** and **rps** runners

    [interface] Added **abort()** to runner's plugin interface

        New method **abort()** is used to immediately interrupt execution.

* **Benchmark Scenarios**:

    [new] DesignateBasic.create_and_delete_server

    [new] DesignateBasic.create_and_list_servers

    [new] DesignateBasic.list_servers

    [new] MistralWorkbooks.list_workbooks

    [new] MistralWorkbooks.create_workbook

    [new] Quotas.neutron_update

    [new] HeatStacks.create_update_delete_stack

    [new] HeatStacks.list_stacks_and_resources

    [new] HeatStacks.create_suspend_resume_delete_stac

    [new] HeatStacks.create_check_delete_stack

    [new] NeutronNetworks.create_and_delete_routers

    [new] NovaKeypair.create_and_delete_keypair

    [new] NovaKeypair.create_and_list_keypairs

    [new] NovaKeypair.boot_and_delete_server_with_keypair

    [new] NovaServers.boot_server_from_volume_and_live_migrate

    [new] NovaServers.boot_server_attach_created_volume_and_live_migrate

    [new] CinderVolumes.create_and_upload_volume_to_image

    [fix] CinderVolumes.create_and_attach_volume

        Pass optional **kwargs only to create server command

    [fix]  GlanceImages.create_image_and_boot_instances

        Pass optional **kwargs only to create server command

    [fix] TempestScenario.* removed stress cleanup.

        Major issue is that tempest stress cleanup cleans whole OpenStack.
        This is very dangerous, so it's better to remove it and leave some
        extra resources.

    [improved] NovaSecGroup.boot_and_delete_server_with_secgroups

        Add optional **kwargs that are passed to boot server comment

* **Benchmark Context**:

    [new] **stacks**

        Generates passed amount of heat stacks for all tenants.

    [new] **custom_image**

        Prepares images for benchmarks in VMs.

        To Support generating workloads in VMs by existing tools like: IPerf,
        Blogbench, HPCC and others we have to have prepared images, with
        already installed and configured tools.

        Rally team decide to generate such images on fly from passed to avoid
        requirements of having big repository with a lot of images.

        This context is abstract context that allows to automate next steps:

        1) runs VM with passed image (with floating ip and other stuff)
        2) execute abstract method that has access to VM
        3) snapshot this image

        In future we are going to use this as a base for making context that
        prepares images.

    [improved] **allow_ssh**

        Automatically disable it if security group are disabled in neutron.

    [improved] **keypair**

        Key pairs are stored in "users" space it means that accessing keypair
        from scenario is simpler now:

            self.context["user"]["keypair"]["private"]

    [fix] **users**

        Pass proper EndpointType for newly created users

    [fix] **sahara_edp**

        The Job Binaries data should be treated as a binary content

* **Benchmark SLA**:

    [interface] SLA calculations is done in additive way now

        Resolves scale issues, because now we don't need to have whole
        array of iterations in memory to process SLA.

        This is required to implement **--abort-on-sla-failure** feature

    [all] SLA plugins were rewritten to implement new interface

Bug fixes
~~~~~~~~~

**18 bugs were fixed, the most critical are**:

* Fix **rally task detailed --iterations-data**

  It didn't work in case of missing atomic actions. Such situation can occur
  if scenario method raises exceptions

* Add user-friendly message if the task cannot be deleted

  In case of trying to delete task that is not in "finished" status users get
  traces instead of user-friendly message try to run it with --force key.

* Network context cleanups networks properly now

Documentation
~~~~~~~~~~~~~

* Image sizes are fixed

* New tutorial in "Step by Step" relate to **--abort-on-sla-failure**

* Various fixes

Unverified

0.0.1

d431a0b7 · Merge "Fix cinder scenario: create nested snapshots" · Jan 26, 2015

This is the first Rally release 0.0.1

Rally is awesome tool for testing verifying and benchmarking
OpenStack clouds.

A lot of people started using Rally in their CI/CD
so we should make Rally more stable with clear strategy
of deprecation and upgrades.

More details about Rally:
https://rally.readthedocs.org/en/

Unverified