Azure SDK For Python EngSys Revamp

Existing Progress At

Currently works, but has organically grown into two separate stacks that I believe best be merged.

We have some repeated code, mostly present around:

• Requirements Parsing
• Versioning Comparisons
• Various setup.py parsing

Which is split between two primary stacks:

eng/tox/
    <various scripts related to the tox environments>

scripts/
    devops_tasks/
        <various scripts ALSO related to various ci needs>
        <building packages, invoking our primary test stack>
        <invoking tox>
        <other necessary pieces>

Also included are the following directories:

eng/versioning/
    <various scripts related to azure-sdk specific versioning>
    <this includes incrementing the version of a package after release>
    <and parsing the current version information>

scripts/
    pylint_plugin/

tools/
    azure-devtools/
    azure-sdk-tools/

This revamp is purely intented to give the way we interact with the azure sdk engineering system a revamp. Stuff like:

• Friendlier to interact with
  • Instead of dealing with frustrating script calls, wrapping in friendly ci prompts with custom help prompts that offer good examples right in the help text.
• Easier to diagnose and debug errors within
  • Reproduce an error in a specific environment and build. (Input is a build url from devops)
    • Use a new default Azure credential to talk to devops!
• Easier to maintain
• Finally delete common code that is not easily shareable between stacks
• Still compatible with python 2.7 (at least maintain current code)
• Centralize all documentation under azure-sdk-tools. Publish documentation under service "tools"
  • azure-sdk-tools
• Minimize the amount of installation that is actually happening. Always preseed. Everything is easily reproable locally.

Lower priority, but may get to

• Unit tests?

Logging the world,

Part 2 logging, the loggening of this revamp will revolve around what is visible in CI, and what is quieted/redirected into attached build logs. This is purely restructing how the engsys talks to itself.

0. Difference between Debug and Information Mode. Should be set via command line or environment variable.
   1. `Respond to both our custom one, as well as the microsoft logging default variable.
1. Test Runs should be quiet. Package installation redirected to file in all cases except debug mode.

Test Scenarios

• Minimum Dependency
  • Standard installation, but with a constraints file for minimum
• Maximum Dependency
  • Standard installation, but with a constraints file for maximum
• With or without specific extras that are not identitifed in the packages?
  • Can we maybe get them officially identified?
  • Alternatively pull from a negative requirements. These are short outlines of necessary and unnecessary packages.

Installation process.

0. Set packages to dev mode, or not. This includes local requirements strings and package versions.
1. If in CI, regenerate all local references. Replace all relative paths with compiled wheels.
2. If a build URL is presented, or if we're in a CI build, download build artifact and pip wheel directory environment variable.
3. pip install the requirements. pip install the target package.
4. pip uninstall the negative packages
5. Confirm pip freeze against scenario.txt
6. Invoke tests or other process

Generating documentation

<updates to generating documentation? Leave this alone for now?>

Updates to repro-ing failures in CI

Reproing a nightly failure

1. Pull down artifacts folder
2. Set environment variable for said artifacts folder location
3. Set any other environment variables
4. Invoke tests

Reproing a nightly `regression failure

1. Pull down artifacts folder
2. Set environment variables for said artifact folder location
3. Other environment variables
4. Clone target azure sdk SHA
5. Install required packages
6. Run tests from SHA

How to expand test environments by optional dependencies

An optional dependency is one that exists under the extras label. We need CI to process any additional requirements files as additional tox invocations.

Current/Future Interface

Each entrypoint should support either A environment variable or B command line input from the service level.

The additional text for arguments is what is there now simplified as much as possible.

Invoking Tests

azci test <pkgname>

azci 

# content of myinvoke.py
import pytest
import sys


class MyPlugin:
    def pytest_sessionfinish(self):
        print("*** test run reporting finishing")


if __name__ == "__main__":
    sys.exit(pytest.main(["-qq"], plugins=[MyPlugin()]))

Building Packages

azci build <glob>

azci build service <glob>

azci build package <glob>

Setting Versioning

azci version info <packagename> 

azci version increment <packagename>

azci version alpha <packagename>

Testing packages

azci test <service>

azci test <package>

Plan of attack

• Can we log where all the code is being called from? https://stackoverflow.com/a/8315566
• Add CLI interface
  • CLI Interface should understand how to pass all arguments it doesn't understand directly to pytest or the like
  • CLI Interface should dump environment configuration and how to enter the same environment on retry
• Move everything into the new places. As code is moved, ensure that:
  • Logging honors debug vs standard information.
  • Functions should accept an CIEnvironment config that states the settings for the current run. A CIEnvironment is created at the "CLI" level of
  • De-duplicated
  • Code Typed and documented
• Adjust tox.ini as best as possible
• Follow the CI and CLI experience, ensure that everything still works with the new entrypoints and changed relative paths.
• Set up local PAT to azure sdk CI system
  • Reproduce failure locally
  • _logs folder contains output by default
• tox.ini post_run cleanup script
• Start deleting code
• Simplify the methodology for generating a test requirements file